AT WHICH TRAINING STAGE DOES CODE DATA HELP LLMS REASONING?

Y Ma, Y Liu, Y Yu, Y Zhang, Y Jiang, C Wang, S Li

… For example, LLaMA was trained with 1.4 trillion tokens consisting of texts (CommonCrawl, C4) and codes (GitHub). These large-scale data with … To improve the data quality, we adopt the following rule-based text cleaning strategies over the raw web pages …


ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for Interdisciplinary Science

S Munikoti, A Acharya, S Wagle, S Horawalavithana - arXiv preprint arXiv:2311.12289, 2023

… We took the original ATLAS model as a baseline, which is pretrained with common crawl (CC) and Wikipedia on top of the T5 model. In addition to this pretrained ATLAS, we also leverage ATLAS-Science model from scratch with the …


Attribute Ambiguity Discovery: A Deep Learning Approach via Unsupervised Learning

E Veltri, G Badaro, M Saeed, P Papotti - 2023

… Typically, these models are pre-trained on large text corpora such as articles from Wikipedia, news articles, or Common Crawl. The model is pre-trained in an unsupervised manner, for example by predicting a missing token or the next-sentence …


Audio is all in one: speech-driven gesture synthetics using WavLM pre-trained model

F Zhang, N Ji, F Gao, S Zhao, Z Wang, S Li - arXiv preprint arXiv:2308.05995, 2023

… [30] obtain the word embeddings using the GloVe model pre-trained on the Common Crawl corpus [31]. This method marginally outperformed other similar-dimensional embedding models, such as Word2Vec [32] and FastText [33], and had similar …


Augmenters at SemEval-2023 Task 1: Enhancing CLIP in Handling Compositionality and Ambiguity for Zero-Shot Visual WSD through Prompt Augmentation and Text …

J Li, YT Shiue, YS Shih, J Geiping - Proceedings of the The 17th International …, 2023

This paper describes our zero-shot approachesfor the Visual Word Sense Disambiguation (VWSD) Task in English. Our preliminarystudy shows that the simple approach of match-ing candidate images with the phrase usingCLIP suffers …


Augmenting interpretable models with large language models during training

C Singh, A Askari, R Caruana, J Gao - Nature Communications, 2023

… We compare Aug-Linear to four interpretable baseline models: Bag of ngrams, TF-IDF (Term frequency-inverse document frequency) 23 , GloVE 24 (we use the pre-trained Glove embeddings trained on Common Crawl containing 840 billion tokens, 2.2 …


Augmenting product knowledge graphs with subjective information

… The authors scale the pre-training of the models either by increasing the number of parameters (up to 11 Billion) and with a large heuristically cleaned corpus from the Common Crawl web dump3. This final pre-trained dataset, called Colossal …


AugTriever: Unsupervised Dense Retrieval by Scalable Data Augmentation

R Meng, Y Liu, S Yavuz, D Agarwal, L Tu, N Yu…

… We apply those methods on Wikipedia passages and CommonCrawl web documents to obtain two large augmented AUGQ datasets. Then we train bi-encoder dense retrievers (using either InBatch or MoCo architecture) using AUGQ and we …


AUTOCAST++: ENHANCING WORLD EVENT PREDIC

Q Yan, R Seraj, J He, L Meng, T Sylvain - arXiv preprint arXiv:2310.01880, 2023

… news articles from the Common Crawl corpus1 spanning 2016 to 2022 for retrieval purpose. For performance evaluation, we use accuracy metrics for T/F and MCQ questions and absolute 1Common Crawl - Open Repository of Web Crawl Data …


Autoencoder for fraudulent transactions data feature engineering

D Breskuvienė, G Dzemyda - DAMSS: 13th conference on data analysis methods for …, 2022

DAMSS-2022 is the 13th International Conference on Data Analysis Methods for Software Systems, held in Druskininkai, Lithuania. Every year at the same place and time. The exception was in 2020, when the world was gripped by the Covid-19 …


Automated Interpretation of Place Descriptions: Determining Entity Types for Querying OSM

M Yousaf, T Schwartz, D Wolter - KI-Künstliche Intelligenz, 2023

This paper is concerned with interpretation of natural language place descriptions, as they are a rich source of geographic information. A place description is interpreted by matching geographic entities occurring in the text against the …


Automated programming, symbolic computation, machine learning: my personal view

B Buchberger - Annals of Mathematics and Artificial Intelligence, 2023

In this note, I present my personal view on the interaction of the three areas Automated Programming, Symbolic Computation, and Machine Learning. Programming is the activity of finding a (hopefully) correct program (algorithm) for a …


Automated Short Answer Grading using BERT on German datasets

Owing to a growth in online learning, the interest in automated grading has intensified, especially due to scoring time reduction. The aim of automated grading is to approximate human level scoring. Automated Short Answer Grading (ASAG) is …


Automated Testing and Improvement of Named Entity Recognition Systems

B Yu, Y Hu, Q Mang, W Hu, P He - arXiv preprint arXiv:2308.07937, 2023

… Common Crawl,1 a nonprofit organization that crawls the web and freely provides its archives and datasets to the public, has collected 380 TB of data and 3.15 billion pages by October 2022. As of 30 December 2022, there are 6,594,544 articles in the …


Automated Text Identification: Multilingual Transformer-based Models Approach

G Gritsay, A Grabovoy, A Kildyakov, Y Chekhovich - 2023

This paper describes our solution approach for the AuTexTification (Automated Text Identification) competition held as part of the IberLEF 2023 conference. Generated text is an increasing problem nowadays. Due to the spread of large volumes of …


Automatic Analysis of Peer Feedback using Machine Learning and Explainable Artificial Intelligence

Peer assessment is a process where learners evaluate and provide feedback on one another’s performance, which is critical to the student learning process. Earlier research has shown that it can improve student learning outcomes in various settings …


Automatic Anonymization of Swiss Federal Supreme Court Rulings

J Niklaus, R Mamié, M Stürmer, D Brunner, M Gygli - arXiv preprint arXiv:2310.04632, 2023

Releasing court decisions to the public relies on proper anonymization to protect all involved parties, where necessary. The Swiss Federal Supreme Court relies on an existing system that combines different traditional computational methods with …


Automatic classification and prioritisation of actionable BI-RADS categories using natural language processing models

P López-Úbeda, T Martín-Noguerol, A Luna - Clinical Radiology, 2023

Aim To facilitate the routine tasks performed by radiologists in their evaluation of breast radiology reports, by enhancing the communication of relevant results to referring physicians via a natural language processing (NLP)-based system to …


Automatic Construction Hazard Identification Integrating On-Site Scene Graphs with Information Extraction in Outfield Test

X Liu, X Jing, Q Zhu, W Du, X Wang - Buildings, 2023

… We initialize the word embeddings for the objects and predicates with pre-trained two million word vectors fastText learned on Common Crawl [45]. The word embedding is computed by representing a word as a bag of n-grams [46]. This word-internal …


Automatic Detection in Twitter of Non-Traumatic Grief due to Deaths by COVID-19. A Deep Learning Approach

J Mata-Vázquez, V Pachón-Álvarez, E Gualda… - IEEE Access, 2023

Non-traumatic grief can be defined as, a complex process that includes emotional, physical, spiritual, social, and intellectual behaviors and responses through which individuals, families, and communities incorporate actual, anticipated, or perceived …


Automatic Extraction of Conversation Flows from Human Dialogues: Understanding Their Impact to Refine NLP Models

MF Sanches, JMC de Sá, RR de Souza, AM de Souza… - SN Computer Science, 2023

… Similarly, Common Crawl [23] is a continuously updated collection of Web pages. Due to its similarity to the BrWaC, the Common Crawl is … Smith JR, Saint-Amand H, Plamada M, Koehn P, Callison-Burch C, Lopez A, Dirt cheap web-scale parallel text …


Automatic Extractive Text Summarization for Text in Nepali Language with Bidirectional Encoder Representation Transformers and K-Mean Clustering

… Pre-trained in over 104 diffiferent language of 2.5TB of filtered common crawl data corpus, XLMRoBERTa or XLM-R is a multilingual version of RoBERTa. It is a transformer based multilingual model which has been pretrained on large corpus of …


Automatic generation of definitions: Exploring if GPT is useful for defining words

When reading a text, it is common to get stuck on unfamiliar words that are difficult to understand in the local context. In these cases, we use dictionaries or similar online resources to find the general meaning of the word. However, maintaining a …


Automatic Generation of German Drama Texts Using Fine Tuned GPT-2 Models

M Bangura, K Barabashova, A Karnysheva, S Semczuk… - arXiv preprint arXiv …, 2023

This study is devoted to the automatic generation of German drama texts. We suggest an approach consisting of two key steps: fine-tuning a GPT-2 model (the outline model) to generate outlines of scenes based on keywords and fine-tuning a …


Automatic Genre Identification for Robust Enrichment of Massive Text Collections: Investigation of Classification Methods in the Era of Large Language Models

T Kuzman, I Mozetič, N Ljubešić - Machine Learning and Knowledge Extraction, 2023

… The XLM-RoBERTa model was pre-trained on the CommonCrawl multilingual data [52], which comprise 167 billion tokens in 100 languages, out of which Slovenian is represented by 1.7 billion tokens. We use the base-sized model that …


Automatic information extraction in business document

… This model was pre-trained with 2.5 TB of text data from the Common Crawl corpus [67], which includes text in 100 different languages. This extensive pre-training has allowed RoBERTa to compute text representations in multiple languages …


Automatic login form leakage detection for website administrators

… We crawled a list of 30K web pages with login forms from a list composed by Roefs [56] using the the Tranco top 100K domains from a Common Crawl snapshot from November/December 2021. The Common Crawl dataset consists of a large …


Automatic Methods for Extracting Taxonomic Relationships from Texts

NV Loukachevitch - Pattern Recognition and Image Analysis, 2023

… Fasttext embeddings [5], which were calculated on the Internet corpus of Common Crawl and Wikipedia, Footnote 1 turned out to be the best vector representation for predicting hypernyms. The best method in this testing was the …


Automatic Multilingual Question Generation for Health Data Using LLMs

R Ackerman, R Balyan - International Conference on AI-generated Content, 2023

Question Generation (QG) involves automatic generation of yes/no, factual and Wh-questions created from data sources such as a database, raw text or semantic representation. QG can be used in an adaptive intelligent tutoring system or a dialog system for …


Automatic pull request title generation.(2022)

T ZHANG, IC IRSAN, F THUNG, DG HAN, D LO… - 2022 IEEE International …

… Unlike BART, T5 was pre-trained with the Colossal Clean Crawled Corpus (C4) dataset, which consists of 750GB of English text from the public Common Crawl web scrape. T5 was reported to achieve state-of-the-art results on many benchmarks …


Automatic Scoring of Creative Problem-Solving with Large Language Models: A Comparison of Originality and Quality Ratings

S Luchini, NT Maliakkal, PV DiStefano, JD Patterson… - 2023

… a collection of data from the Toronto BookCorpus (800M words), the English portion of Wikipedia (2,500M words), the CCNews English dataset (63M news articles), the OpenWebText dataset of Reddit posts, and the Stories dataset which …


Automatic Subjective Answer Grading Software Using Machine Learning

R Kothari, B Rangwala, K Patel - 2023 7th International Conference on Trends in …, 2023

One of the major challenges during online examinations is the assessment of answers, particularly of the subjective type. Subjective answers test a student's ability to retain information and express it in natural language. While objective …


Automatic Summarization of Validated Intelligence Events

… Fortunately there exists one very large dataset, which is the Common Crawl dataset that consist of nearly a trillion words. Common Crawl is a dataset that has been obtained by crawling the web and therefore the quality of data is quite low …


Automatically Assembling a Custom-Built Training Corpus for Improving the Learning of In-Domain Word/Document Embeddings

Y Blanco-Fernández, A Gil-Solla, JJ Pazos-Arias… - Informatica, 2023

Embedding models turn words/documents into real-number vectors via co-occurrence data from unrelated texts. Crafting domain-specific embeddings from general corpora with limited domain vocabulary is challenging. Existing solutions retrain …


Automatically Preventing, Detecting and Repairing Crucial Errors in Programs

Today, high-impact errors in programs result in huge money losses, are notoriously expensive for programmers to repair, and affect millions of real-world users. This dissertation highlights the need to move beyond our post-mortem manual error …


Automating democracy: Generative AI, journalism, and the future of democracy

Sophisticated AI systems are increasingly everywhere. In many ways, we have already been affected by the rollout of AI systems into more and more areas of life, from insurance and law to healthcare and the media – often without really noticing …


AutoPCF: Efficient Product Carbon Footprint Accounting with Large Language Models

Z Deng, J Liu, B Luo, C Yuan, Q Yang, L Xiao, W Zhou… - arXiv preprint arXiv …, 2023

… The GPT3.5 and GPT-4 training datasets include the Common Crawl dataset, the Webtext2 dataset, two internet-based book corpora, and Wikipedia7. The Common Crawl dataset, which contributes 60% of the total weight in the training mix, contains …


B\"{u} y\"{u} k dil modellerinin T\"{u} rk\c {c} e verisetleri ile e\u {g} itilmesi ve ince ayarlanmas\i

AT Arslan - arXiv preprint arXiv:2306.03978, 2023

Large language models have advanced enormously, gained vast attraction and are having a phase of intensed research. Some of the developed models and training datasets have been made open-accessible. Hence these may be further fine-tuned …


Bachelor Thesis Emergent Theory of Mind in Large Language Models

… The emergence of transformers, known for their reduced training time compared to older long short-term memory (LSTM) models, has enabled utilizing large language datasets like the Wikipedia Corpus and Common Crawl for training …


Bachelor's Thesis Assignment

When it comes to language learning apps that allow sentence-like answers, accurately estimating the correctness score is crucial. A possible approach is to compute the semantic similarity of input sentences and predefined correct answers …


Back to common sense: Oxford dictionary descriptive knowledge augmentation for aspect-based sentiment analysis

W Jin, B Zhao, L Zhang, C Liu, H Yu - Information Processing & Management, 2023

Aspect-based Sentiment Analysis (ABSA) is a crucial natural language understanding (NLU) research field which aims to accurately recognize reviewers’ opinions on different aspects of products and services. Despite the prominence of …


Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors

M Kubis, P Skórzewski, M Sowański, T Ziętkiewicz - arXiv preprint arXiv:2310.16609, 2023

In a spoken dialogue system, an NLU model is preceded by a speech recognition system that can deteriorate the performance of natural language understanding. This paper proposes a method for investigating the impact of speech recognition …


Back Translation for Speech-to-text Translation Without Transcripts

Q Fang, Y Feng - arXiv preprint arXiv:2305.08709, 2023

… Specifically, we use europarl v78, commoncrawl9, news commentary v1210 subsets for all three languages. The source text is never used … 8http://statmt.org/wmt13/ training-parallel-europarl-v7.tgz 9http://statmt.org/wmt13/ training-parallel-commoncrawl.tgz …


Backdoor Learning on Sequence to Sequence Models

L Chen, M Cheng, H Huang - arXiv preprint arXiv:2305.02424, 2023

Backdoor learning has become an emerging research area towards building a trustworthy machine learning system. While a lot of works have studied the hidden danger of backdoor attacks in image or text classification, there is a limited …


BacklinkDB: A Purpose-Built Backlink Database Management System

… Common Crawl is a non-profit organization that periodically crawls the web and publicizes data. For the experiments … Common Crawl provides data on all the indexable webpages. This data is provided in a series of warc files found in their …


Bag of Words and Embedding Text Representation Methods for Medical Article Classification

P Cichosz - International Journal of Applied Mathematics and …, 2023

… vectors pre-trained on the Common Crawl corpus were used.For training custom fastText vectors, the fasttext Python library10 was used (Bojanowski et al., 2020), with vector dimensionality set to 100, SG training mode, and other parameters left at …


Baichuan 2: Open Large-scale Language Models

A Yang, B Xiao, B Wang, B Zhang, C Yin, C Lv, D Pan… - arXiv preprint arXiv …, 2023

… For instance, the main data source for LLaMA is Common Crawl1, which comprises 67% of LLaMA’s pre-training data but is filtered to English content only. Other open source LLMs such as MPT (MosaicML… 1https://commoncrawl.org/ …


Balancing of tourist opinions for sentiment analysis task

AB García-Gutiérrez, PE López-Ávila… - 2023

This article presents a proposal for the treatment of an unbalanced tourist database with emphasis on minority classes for its classification, in this case, one based on BERT, called BETO. This methodology originally forms part of the thesis project of …


BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla

A Bhattacharjee, T Hasan, W Ahmad, R Shahriyar - Findings of the Association for …, 2023

This work presents ‘BanglaNLG,’a comprehensive benchmark for evaluating natural language generation (NLG) models in Bangla, a widely spoken yet low-resource language. We aggregate six challenging conditional text generation tasks under the …


Basic Principles of Cross-Lingual Models

S Sharoff, R Rapp, P Zweigenbaum - Building and Using Comparable Corpora for …, 2023

When we start working across languages, we need to determine ways of measuring similarity of entities (such as a word, a phrase, a sentence or a link between entities) within each language and across languages. Modern approaches usually rely on …


Behind the Scenes: Examining Efficacy of Race Classification Models with a Focus on TV

This thesis explores the use of web scraping and machine learning to enhance the predictive power of current race classification models. With the rise of online data as an important source for research, the availability of race labels typically acquired …


BELT: Bootstrapping Electroencephalography-to-Language Decoding and Zero-Shot Sentiment Classification by Natural Language Supervision

J Zhou, Y Duan, YC Chang, YK Wang, CT Lin - arXiv preprint arXiv:2309.12056, 2023

This paper presents BELT, a novel model and learning framework for the pivotal topic of brain-to-language translation research. The translation from noninvasive brain signals into readable natural language has the potential to promote the …


Benchmark for Evaluation of Danish Clinical Word Embeddings

MS Laursen, JS Pedersen, PJ Vinholt, RS Hansen… - Northern European Journal …, 2023

In natural language processing, benchmarks are used to track progress and identify useful models. Currently, no benchmark for Danish clinical word embeddings exists. This paper describes the development of a Danish benchmark for clinical word …


Benchmark tasks for evaluation of language models

Language models (LMs) are an integral part of NLP, with their importance sharply increasing in recent years with the advent of large generalized LMs (such as OpenAI GPT and BERT) that reach and in some cases surpass the level of nonexpert …


Benchmarking and Analyzing In-context Learning, Fine-tuning and Supervised Learning for Biomedical Knowledge Curation: a focused study on chemical entities of …

E Groves, M Wang, Y Abdulle, H Kunz… - arXiv preprint arXiv …, 2023

Automated knowledge curation for biomedical ontologies is key to ensure that they remain comprehensive, high-quality and up-to-date. In the era of foundational language models, this study compares and analyzes three NLP paradigms for …


Benchmarking Arabic AI with Large Language Models

A Abdelali, H Mubarak, SA Chowdhury, M Hasanain… - arXiv preprint arXiv …, 2023

… GPT-3 is trained mainly using a filtered version of Common Crawl dataset. The filtering is based on similarity to high-quality reference corpora. Fuzzy deduplication at document level is performed to remove redundancy and avoid contamination of …


Benchmarking Large Language Model Capabilities for Conditional Generation

J Maynez, P Agrawal, S Gehrmann - arXiv preprint arXiv:2306.16793, 2023

Pre-trained large language models (PLMs) underlie most new developments in natural language processing. They have shifted the field from application-specific model pipelines to a single model that is adapted to a wide range of tasks …


Benchmarking Multilabel Topic Classification in the Kyrgyz Language

A Alekseev, SI Nikolenko, G Kabaeva - arXiv preprint arXiv:2308.15952, 2023

Kyrgyz is a very underrepresented language in terms of modern natural language processing resources. In this work, we present a new public benchmark for topic classification in Kyrgyz, introducing a dataset based on collected and annotated …


Benchmarking Procedural Language Understanding for Low-Resource Languages: A Case Study on Turkish

A Uzunoğlu, GG Şahin - arXiv preprint arXiv:2309.06698, 2023

Understanding procedural natural language (eg, step-by-step instructions) is a crucial step to execution and planning. However, while there are ample corpora and downstream tasks available in English, the field lacks such resources for most …


Bengali Text Classification: A New multi-class Dataset and Performance Evaluation of Machine Learning and Deep Learning Models

This study focuses on Bengali text classification using machine learning and deep learning techniques. Text classification is a fundamental task in natural language processing (NLP) that involves categorizing text documents into predefined …


BERT Goes Off-Topic: Investigating the Domain Transfer Challenge using Genre Classification

D Roussinov, S Sharoff - arXiv preprint arXiv:2311.16083, 2023

While performance of many text classification tasks has been recently improved due to Pre-trained Language Models (PLMs), in this paper we show that they still suffer from a performance gap when the underlying distribution of topics changes. For …


BERT-based ensemble learning for multi-aspect hate speech detection

AC Mazari, N Boudoukhani, A Djeffal - Cluster Computing, 2023

… After testing FastText embedding pre-trained on Wikipedia texts and those pre-trained on Common Crawl corpus, Footnote 2 we choose to use the FastText vectors pre-trained on Common Crawl data because they provide us the best results for all the …


BERTinchamps: Cost-Effective Training of Large Language Models for Medical Tasks in French

… The OSCAR dataset is an extensive multilingual corpus acquired through language classification and filtering of the Common Crawl corpus employing the goclassy architecture. A subset of 17GB of the French part of the OSCAR dataset …


BERTuit: Understanding Spanish language in Twitter with transformers

J Huertas‐Tato, A Martín, D Camacho - Expert Systems, 2023

The appearance of complex attention‐based language models such as BERT, RoBERTa or GPT‐3 has allowed to address highly complex tasks in a plethora of scenarios. However, when applied to specific domains, these models encounter …


Better conditioning on context for natural language processing

… T5 introduces a new pre-training dataset, Colossal Clean Crawled Corpus by cleaning the web pages from Common Crawl. T5 also systematically compares previous methods in terms of pre-training objectives, architectures, pre-training …


Better Language Support and Pragmatic System Designs for Automatic Text Summarization

… Common approaches to acquiring large amounts of text data usually involve filtering large snapshots of the internet distributed by online archives like CommonCrawl3. Such approaches provide virtually endless amounts of text data …


Better Quality Pre-training Data and T5 Models for African Languages

A Oladipo, M Adeyemi, O Ahia, A Owodunni… - Proceedings of the 2023 …, 2023

… 2019), Wikipedia and Common Crawl archive. The latter often has significant quality issues (Kreutzer et al.… noisier common crawl data. Obtaining such highquality data is tedious since it involved curating several verified sources manually. Thus …


Better Translation+ Split and Generate for Multilingual RDF-to-Text (WebNLG 2023)

N Kumar, SOU Islam, O Dušek - Proceedings of the Workshop on Multimodal …, 2023

This paper presents system descriptions of our submitted outputs for WebNLG Challenge 2023. We use mT5 in multi-task and multilingual settings to generate more fluent and reliable verbalizations of the given RDF triples. Furthermore, we …


Beyond Document Page Classification: Design, Datasets, and Challenges

J Van Landeghem, S Biswas, MB Blaschko, MF Moens - arXiv preprint arXiv …, 2023

This paper highlights the need to bring document classification benchmarking closer to real-world applications, both in the nature of data tested ($X$: multi-channel, multi-paged, multi-industry; $Y$: class distributions and label set variety) and in classification …


Beyond Extractive: Advancing Abstractive Automatic Text Summarization in Norwegian with Transformers

Automatic summarization is a key area in natural language processing (NLP) and machine learning which attempts to generate informative summaries of articles and documents. Despite its evolution since the 1950s, research on automatically …


Beyond Fish and Bicycles: Exploring the Varieties of Online Women's Ideological Spaces

U Balci, C Ling, E De Cristofaro, M Squire, G Stringhini… - arXiv preprint arXiv …, 2023

The Internet has been instrumental in connecting under-represented and vulnerable groups of people. Platforms built to foster social interaction and engagement have enabled historically disenfranchised groups to have a voice. One such vulnerable …


Beyond quantity of experience: Exploring the role of semantic consistency in Chinese character knowledge.

CY Hsieh, M Marelli, K Rastle - … of Experimental Psychology: Learning, Memory, and …, 2023

Most printed Chinese words are compounds built from the combination of meaningful characters. Yet, there is a poor understanding of how individual characters contribute to the recognition of compounds. Using a megastudy of …


Beyond Scale: the Diversity Coefficient as a Data Quality Metric Demonstrates LLMs are Pre-trained on Formally Diverse Data

… more stringent preprocessing method applied to the Common Crawl corpus for Pile-CC, which contributes to enhanced data diversity. … Since many canonical datasets already exist and are publicly available (eg Common Crawl, Wikipedia) …


bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark

M Hardalov, P Atanasova, T Mihaylov, G Angelova… - arXiv preprint arXiv …, 2023

We present bgGLUE (Bulgarian General Language Understanding Evaluation), a benchmark for evaluating language models on Natural Language Understanding (NLU) tasks in Bulgarian. Our benchmark includes NLU tasks targeting a variety of NLP …


Bhagavad Geeta Based ChatBot

A Karekar, S Limaye, A Nara, S Panchal - 2023 3rd International Conference on …, 2023

… By merging data from Common Crawl and WebText, a significantly larger and more varied dataset was used to train the model. … A wide variety of data sources, including BookCorpus, Common Crawl, and Wikipedia, are used to train the GPT-3 …


Bias Against 93 Stigmatized Groups in Masked Language Models and Downstream Sentiment Classification Tasks

KX Mei, S Fereidooni, A Caliskan - arXiv preprint arXiv:2306.05550, 2023

The rapid deployment of artificial intelligence (AI) models demands a thorough investigation of biases and risks inherent in these models to understand their impact on individuals and society. This study extends the focus of bias evaluation in extant …


Bias Detection for Customer Interaction Data: A Survey on Datasets, Methods, and Tools

A Donald, A Galanopoulos, E Curry, E Muñoz, I Ullah… - IEEE Access, 2023

… It is important to highlight that these embedders, though often built on Wikipedia or CommonCrawl, can be trained on a vast variety of text … on CommonCrawl, and trained on Twitter tweets. They found Twitter embeddings to produce the greatest …


Biases in Large Language Models: Origins, Inventory and Discussion

R Navigli, S Conia, B Ross - ACM Journal of Data and Information Quality, 2023

In this paper, we introduce and discuss the pervasive issue of bias in the large language models that are currently at the core of mainstream approaches to Natural Language Processing (NLP). We first introduce data selection bias, that is, the bias …


Bidirectional Contrastive Split Learning for Visual Question Answering

Visual Question Answering (VQA) based on multi-modal data facilitates real-life applications such as home robots and medical diagnoses. One significant challenge is to devise a robust decentralized learning framework for various client models …


Bilingual Terminology Alignment Using Contextualized Embeddings

I Setha, H Aliane - Proceedings of the Workshop on Computational …, 2023

S Imene, A Algeria, A Hassina - Computational Terminology in NLP and Translation …, 2023

… In our work, we choose to use the Multilingual Elmo embeddings2, which was pre-trained on 20 million words data randomly sampled from the raw text released by the shared task wiki dump + common crawl, (github…


Binary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models

X Jin, J Larson, W Yang, Z Lin - arXiv preprint arXiv:2312.09601, 2023

Binary code summarization, while invaluable for understanding code semantics, is challenging due to its labor-intensive nature. This study delves into the potential of large language models (LLMs) for binary code comprehension. To this end, we …


Biologically Plausible Sparse Temporal Word Representations

Y Liu, W Chen, H Liu, Y Zhang, M Zhang, H Qu - IEEE Transactions on Neural …, 2023

… In our experiment, we use the pretrained GloVe10(“Common Crawl” version with 2.2M vocabulary) and Word2Vec11(“Google News” version with 3.0M vocabulary) of 300 dimensions, as well as BERT-mini(256 dimensions and four layers) and BERT-base(768 …


BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition

P Vera, M Makhlouf - arXiv preprint arXiv:2308.08625, 2023

V Pavlova, M Makhlouf - The 22nd Workshop on Biomedical Natural Language …, 2023

Using language models (LMs) pre-trained in a self-supervised setting on large corpora and then fine-tuning for a downstream task has helped to deal with the problem of limited label data for supervised learning tasks such as Named Entity …


Bipolar Sentiment Analysis of Japanese Social Media Posts: A Semantic Similarity Based Approach

MFF Khan, N Oi, K Sakamura - 2023 Sixth International Symposium on Computer …, 2023

… These vectors are trained on Common Crawl and Wikipedia using fastText with the following configuration: CBOW (continuous bag of words) with position-weights, in dimension 300, with character n-grams of length 5, a window of size 5 and 10 …


BiSync: A Bilingual Editor for Synchronized Monolingual Texts

J Crego, J Xu, F Yvon - arXiv preprint arXiv:2306.00400, 2023

In our globalized world, a growing number of situations arise where people are required to communicate in one or several foreign languages. In the case of written communication, users with a good command of a foreign language may find …


Bit Cipher--A Simple yet Powerful Word Representation System that Integrates Efficiently with Language Models

H Zhao, JR Williams - arXiv preprint arXiv:2311.11012, 2023

While Large Language Models (LLMs) become ever more dominant, classic pre-trained word embeddings sustain their relevance through computational efficiency and nuanced linguistic interpretation. Drawing from recent studies demonstrating that the …


BiTimeBERT: Extending Pre-Trained Language Representations with Bi-Temporal Information

J Wang, A Jatowt, M Yoshikawa, Y Cai - Proceedings of the 46th International ACM …, 2023

Time is an important aspect of documents and is used in a range of NLP and IR tasks. In this work, we investigate methods for incorporating temporal information during pre-training to further improve the performance on time-related tasks. Compared …


BitNet: Scaling 1-bit Transformers for Large Language Models

H Wang, S Ma, L Dong, S Huang, H Wang, L Ma… - arXiv preprint arXiv …, 2023

The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption. In this work, we introduce BitNet, a scalable and stable 1-bit Transformer …


Black Ostrich: Web Application Scanning with String Solvers

B Eriksson, A Stjerna, R De Masellis, P Rüemmer… - Proceedings of the 2023 …, 2023

… We evaluate Black Ostrich on a set of 8,820 unique validation patterns gathered from over 21,667,978 forms from a combination of the July 2021 Common~Crawl and Tranco top 100K. For these forms and reconstructions of input elements …


Blacks is to Anger as Whites is to Joy? Understanding Latent Affective Bias in Large Pre-trained Neural Language Models

A Kadan, S Bhadra, MP Gangan - arXiv preprint arXiv:2301.09003, 2023

… Details of training corpora used for corpus level affective bias analysis (†Giga5, ClueWeb, & Common Crawl used to pre-train XLNet are … We omit Giga5 and ClueWeb due to their unavailability as open-source corpora and Common Crawl as …


BloombergGPT: A Large Language Model for Finance

S Wu, O Irsoy, S Lu, V Dabravolski, M Dredze… - arXiv preprint arXiv …, 2023

The use of NLP in the realm of financial technology is broad and complex, with applications ranging from sentiment analysis and named entity recognition to question answering. Large Language Models (LLMs) have been shown to be …


BongHope: An Annotated Corpus for Bengali Hope Speech Detection

… It is pretrained on 2.5TB data from CommonCrawl corpus, consisting of 100 languages. XLMRoberta tokenizer is used to tokenize the tweets, which are then fed to the ‘xlm-robertabase’ encoder to obtain the word embeddings (Conneau et al …


Bootstrapped nDCG Estimation in the Presence of Unjudged Documents

M Fröbe, L Gienapp, M Potthast, M Hagen - … , ECIR 2023, Dublin, Ireland, April 2–6 …, 2023

Retrieval studies often reuse TREC collections after the corresponding tracks have passed. Yet, a fair evaluation of new systems that retrieve documents outside the original judgment pool is not straightforward. Two common ways of dealing with …


Bots Behaving Badly: A Products Liability Approach to Chatbot-Generated Defamation

Within two months of its launch, ChatGPT became the fastest-growing consumer application in history with more than 100 million monthly active users. 1 Created by OpenAI, a private company backed by Microsoft, ChatGPT is just one * Associate …


Breaking the Bias: Gender Fairness in LLMs Using Prompt Engineering and In-Context Learning

Large Language Models (LLMs) have been identified as carriers of societal biases, particularly in gender representation. This study introduces an innovative approach employing prompt engineering and incontext learning to rectify these biases in LLMs …


Bridging the Resource Gap in Cross-Lingual Embedding Space

K Bhowmik, A Ralescu - Modelling and Development of Intelligent Systems: 8th …, 2023

… XLM-R uses CommonCrawl as the source of data instead of Wikipedia. The larger amount of data available through CommonCrawl has resulted in performance improvement for low-resource languages. We intend to explore these models in the future. …


Bringing legal knowledge to the public by constructing a legal question bank using large-scale pre-trained language model

M Yuan, B Kao, TH Wu, MMK Cheung, HWH Chan… - Artificial Intelligence and …, 2023

Access to legal information is fundamental to access to justice. Yet accessibility refers not only to making legal documents available to the public, but also rendering legal information comprehensible to them. A vexing problem in bringing legal …


Bringing order into the realm of Transformer-based language models for artificial intelligence and law

CM Greco, A Tagarelli - arXiv preprint arXiv:2308.05502, 2023

… The training data is obtained from a filtered and cleaned version of CommonCrawl and consists of more than 2TB of data in 100 languages. Sub-word tokenization is directly applied on the raw text for all languages using a …


Broader and Deeper: A Multi-Features with Latent Relations BERT Knowledge Tracing Model

J Wang - Responsive and Sustainable Educational Futures: 18th …, 2023

Knowledge tracing aims to estimate students’ knowledge state or skill mastering level over time, which is evolving into an essential task in educational technology. Traditional knowledge tracing algorithms generally use one or a few features to …


BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model

N Dey, D Soboleva, F Al-Khateeb, B Yang, R Pathria… - arXiv preprint arXiv …, 2023

We introduce the Bittensor Language Model, called "BTLM-3B-8K", a new state-of-the-art 3 billion parameter open-source language model. BTLM-3B-8K was trained on 627B tokens from the SlimPajama dataset with a mixture of 2,048 and 8,192 context …


Bug Localization Model in Source Code using Ontologies

AS Da Silva, RE Garcia, LC Botega - IEEE Access, 2023

… For this step, the GloVe model is pre-trained with 300dimensional vectors using the Common Crawl knowledge base [39]. As the vectors are pre-trained, the similarity is only checked for decomposed concept patterns that generate a token in …


Building and Using Comparable Corpora for Multilingual Natural Language Processing

… [25] or the use of filtered Common Crawl data [26] with the assumption that different cultures use the Web for broadly similar purposes. … For example, the multilingual BERT is trained on Wikipedias, while XLM-Roberta is trained on the …


Building Comparable Corpora

S Sharoff, R Rapp, P Zweigenbaum - Building and Using Comparable Corpora for …, 2023

In a parallel corpus we know which document is a translation of what by design. If the link between documents in different languages is not known, it needs to be established. In this chapter we will discuss methods for measuring document …


BUILDING FLEXIBLE, SCALABLE, AND MACHINE LEARNING-READY MULTIMODAL ONCOLOGY DATASETS

A Tripathi, A Waqas, K Venkatesan, Y Yilmaz, G Rasool - arXiv preprint arXiv …, 2023

… For example, the rise of large general-purpose datasets like Common Crawl [6] for natural language processing (NLP) has fueled advances in language models and Artificial Intelligence (AI) assistants. One may hope that extensive, standardized …


Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models (Vision Paper)

… For example, GPT-3 was pre-trained on a huge language corpus consisting of around 500 billion tokens from Web resources (eg, CommonCrawl, Wikipedia) and books. BLIP-2 and DINOv2 were pre-trained on 129 million images (with captions) …


Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models

J Rao, S Gao, G Mai, K Janowicz - arXiv preprint arXiv:2309.17319, 2023

In recent years we have seen substantial advances in foundation models for artificial intelligence, including language, vision, and multimodal models. Recent studies have highlighted the potential of using foundation models in geospatial artificial …


Building Programmable Commons

First, we asked ‘who shall govern the Internet?’as its decentralised and open nature promised a public sphere free from the manipulation and fragmentation of the ad-driven mass media.‘Commons’ was an omnipresent theme in this discourse. These …


Building Web-Based Subject-Specific Corpora on the Desktop: Evaluation of Search Metrics

JP Van Belle - … Intelligence: Select Proceedings of InCITe 2022, 2023

Building subject-specific or domain corpora from Web data is well-researched. However, most approaches start by using seed articles as inputs to Web crawlers and take document similarity algorithms for selection. We take a different lean …


BureauBERTo: adapting UmBERTo to the Italian bureaucratic language

S Auriemma, M Madeddu, M Miliani, A Bondielli… - 2023

In this work, we introduce BureauBERTo, the first transformer-based language model adapted to the Italian Public Administration (PA) and technical-bureaucratic domains. We further pre-trained the general-purpose Italian model UmBERTo on a …


calamanCy: A Tagalog Natural Language Processing Toolkit

LJV Miranda - arXiv preprint arXiv:2311.07171, 2023

We introduce calamanCy, an open-source toolkit for constructing natural language processing (NLP) pipelines for Tagalog. It is built on top of spaCy, enabling easy experimentation and integration with other frameworks. calamanCy addresses the …


Calibrating a Transformer-Based Model's Confidence on Community-Engaged Research Studies: Decision Support Evaluation Study

B Ferrell, SE Raskin, EB Zimmerman - JMIR Formative Research, 2023

… This type of model was made for cross-lingual transfer learning tasks and is trained on >2 terabytes of the Common Crawl corpus. It differs from BERT in terms of its tokenization and masking pattern, thus making it an interesting model with which …


CamPros at CASE 2022 Task 1: Transformer-based Multilingual Protest News Detection

N Kumari, M Anand, T Mohan, P Kumaraguru… - Proceedings of the 5th …, 2022

Socio-political protests often lead to grave consequences when they occur. The early detection of such protests is very important for taking early precautionary measures. However, the main shortcoming of protest event detection is the scarcity …


Can ChatGPT Replace Traditional KBQA Models? An In-Depth Analysis of the Question Answering Performance of the GPT LLM Family

Y Tan, D Min, Y Li, W Li, N Hu, Y Chen, G Qi - International Semantic Web …, 2023

ChatGPT is a powerful large language model (LLM) that covers knowledge resources such as Wikipedia and supports natural language question answering using its own knowledge. Therefore, there is growing interest in exploring whether …


Can Large Language Models Capture Dissenting Human Voices?

N Lee, N An, J Thorne - Proceedings of the 2023 Conference on Empirical …, 2023

Large language models (LLMs) have shown impressive achievements in solving a broad range of tasks. Augmented by instruction fine-tuning, LLMs have also been shown to generalize in zero-shot settings as well. However, whether LLMs closely …


Can Large Language Models Generate Outpatient Clinic Letters at First Consultation That Incorporate Complication Profiles From UK and USA Aesthetic Plastic …

RHR Roberts, SR Ali, TD Dobbs, IS Whitaker - Aesthetic Surgery Journal Open …, 2023

TD Dobbs, IS Whitaker, MA Cantab - Aesthetic Surgery Journal, 2024

Background The importance of written communication between clinicians and patients, especially in the wake of the Supreme Court case of Montgomery vs Lanarkshire, has led to a shift towards patient-centric care in the UK. This study …


Can Peanuts Fall in Love with Distributional Semantics?

JA Michaelov, S Coulson, BK Bergen - arXiv preprint arXiv:2301.08731, 2023

The context in which a sentence appears can drastically alter our expectations about upcoming words - for example, following a short story involving an anthropomorphic peanut, experimental participants are more likely to expect the sentence 'the peanut …


Can we Debunk Disinformation by Leveraging SpeakerCredibility and Perplexity Measures?

AFUR Khilji, A Sachan, D Lachi, AV Singh, TD Singh - 2023

In the present age, őghting disinformation is the main concern after pandemic. The exponential growth of fake news and its role in deteriorating general public trust and democratic standards certainly calls for counter-combat approaches. The prediction …


cantnlp@ LT-EDI-2023: Homophobia/Transphobia Detection in Social Media Comments using Spatio-Temporally Retrained Language Models

S Wong, M Durward, B Adams, J Dunn - Proceedings of the Third Workshop on …, 2023

This paper describes our multiclass classification system developed as part of the LT-EDI@ RANLP-2023 shared task. We used a BERT-based language model to detect homophobic and transphobic content in social media comments across five …


cantnlp@ LT-EDI@ RANLP-2023: Homophobia/Transphobia Detection in Social Media Comments using Spatio-Temporally Retrained Language Models

SGJ Wong, M Durward, B Adams, J Dunn - arXiv preprint arXiv:2308.10370, 2023

This paper describes our multiclass classification system developed as part of the LTEDI@RANLP-2023 shared task. We used a BERT-based language model to detect homophobic and transphobic content in social media comments across five …


CAPS: A Practical Partition Index for Filtered Similarity Search

G Gupta, J Yi, B Coleman, C Luo, V Lakshman… - arXiv preprint arXiv …, 2023

With the surging popularity of approximate near-neighbor search (ANNS), driven by advances in neural representation learning, the ability to serve queries accompanied by a set of constraints has become an area of intense interest. While …


Cataloging of Internet resources

… Common crawl en A large resource that contains a lot of information of a high scientific level, covering not only human biology, but also a number of issues of general biology (in particular, molecular and cellular biology). Relatively …


CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Data

M Turski, T Stanisławek, K Kaczmarek, P Dyda… - arXiv preprint arXiv …, 2023

… The spike for the year 2021 was probably caused by the use of the Common Crawl dump from May 2022. We assume that crawlers from Common Crawl usually tend to find files that are a few months old, which often means that they are from the …


CEIA-NLP at CASE 2022 Task 1: Protest News Detection for Portuguese

D Fernandes, A Junior, G Marques, A Soares… - Proceedings of the 5th …, 2022

This paper summarizes our work on the document classification subtask of Multilingual protest news detection of the CASE@ ACL-IJCNLP 2022 workshok. In this context, we investigate the performance of monolingual and multilingual …


Center for Artificial Intelligence Challenge on Conversational AI Correctness

M Kubis, P Skórzewski, M Sowanski, T Zietkiewicz - 2023

This paper describes a challenge on Conversational AI correctness with the goal to develop Natural Language Understanding models that are robust against speech recognition errors. The data for the competition consist of natural language …


Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster

N Dey, G Gosal, H Khachane, W Marshall, R Pathria… - arXiv preprint arXiv …, 2023

We study recent research advances that improve large language models through efficient pre-training and scaling, and open datasets and tools. We combine these advances to introduce Cerebras-GPT, a family of open compute-optimal language …


CFDA & CLIP at TREC 2022 NeuCLIR Track

JH Ju, WC Chen, HT Chang, CW Lin, MF Tsai…

In this notebook paper, we report our methods for NeuCLIR track in TREC 2022. We adopt the common multistage pipeline for cross-language information retrieval task (CLIR). The pipeline includes the machine translation, sparse passage retrieval and the …


Challenges and Applications of Large Language Models

J Kaddour, J Harris, M Mozes, H Bradley, R Raileanu… - arXiv preprint arXiv …, 2023

Large Language Models (LLMs) went from non-existent to ubiquitous in the machine learning discourse within a few years. Due to the fast pace of the field, it is difficult to identify the remaining challenges and already fruitful application areas. In this paper …


Challenges of Large Language Models for Mental Health Counseling

NC Chung, G Dyer, L Brocki - arXiv preprint arXiv:2311.13857, 2023

… One of the largest data sources for LLMs is the CommonCrawl [19, 20] which contains 390 TB with more than 3.1 billion pages of websites. 46% of documents are in English followed by German and Russian. This focus on sources emanating from …


Challenging News Media Narratives on Refugees

… The model used for this project is the GloVe embeddings model that has been trained on the Common Crawl dataset. It consists of a vast collection of web pages and related metadata. (Common Crawl Foundation, nd). Training GloVe …


Challenging specialized transformers on zero-shot classification

S Auriemma, M Madeddu, M Miliani, A Bondielli… - 2023

… We may suppose that this is due to the fact that UmBERTo was trained on Common-Crawl, which also contains legal and administrative texts in its Italian section. Very high results are obtained by UmBERTo for PER entities, reaching …


Change is Hard: A Closer Look at Subpopulation Shift

Y Yang, H Zhang, D Katabi, M Ghassemi - arXiv preprint arXiv:2302.12254, 2023

Machine learning models often perform poorly on subgroups that are underrepresented in the training data. Yet, little is understood on the variation in mechanisms that cause subpopulation shifts, and how algorithms generalize across …


ChaPat at SemEval-2023 Task 9: Text Intimacy Analysis using Ensembles of Multilingual Transformers

T Chavan, V Patwardhan - Proceedings of the The 17th International Workshop on …, 2023

Intimacy estimation of a given text has recently gained importance due to the increase in direct interaction of NLP systems with humans. Intimacy is an important aspect of natural language and has a substantial impact on our everyday …


Character, Word, or Both? Revisiting the Segmentation Granularity for Chinese Pre-trained Language Models

X Liang, Z Zhou, H Huang, S Wu, T Xiao, M Yang, Z Li… - arXiv preprint arXiv …, 2023

Pretrained language models (PLMs) have shown marvelous improvements across various NLP tasks. Most Chinese PLMs simply treat an input text as a sequence of characters, and completely ignore word information. Although Whole Word Masking …


Characterizing Bias in Word Embeddings Towards Analyzing Gender Associations in Philippine Texts

LCL Gamboa, MRJE Estuar - 2023 IEEE World Conference on Applied Intelligence …, 2023

The steady increase in computational gender bias research has been mostly done on languages for which reliable NLP packages are readily available—such as English, Chinese, and Spanish. This study expands on this area of research by …


Chat GPT: A magical tool or the end of traditional language learning and teaching approach

HDA Chi - KHOA HỌC VÀ CÔNG NGHỆ GIAO THÔNG VẬN TẢI

… GPT-3 was trained on hundreds of billions of words around 45 terabytes of text and its dataset mostly comes from Common Crawl, WebText2, and Wikipedia [7],[4]. Furthermore, GPT3has more than 175 billion machine learning parameters that are …


Chatbots to ChatGPT in a Cybersecurity Space: Evolution, Vulnerabilities, Attacks, Challenges, and Future Recommendations

A Qammar, H Wang, J Ding, A Naouri, M Daneshmand… - arXiv preprint arXiv …, 2023

Chatbots shifted from rule-based to artificial intelligence techniques and gained traction in medicine, shopping, customer services, food delivery, education, and research. OpenAI developed ChatGPT blizzard on the Internet as it crossed one …


ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge

Y Li, Z Li, K Zhang, R Dan, S Jiang, Y Zhang - Cureus, 2023

Objective The primary aim of this research was to address the limitations observed in the medical knowledge of prevalent large language models (LLMs) such as ChatGPT, by creating a specialized language model with enhanced accuracy in …


ChatDoctor: A Medical Chat Model Fine-tuned on LLaMA Model using Medical Domain Knowledge

L Yunxiang, L Zihan, Z Kai, D Ruilong, Z You - arXiv preprint arXiv:2303.14070, 2023

Recent large language models (LLMs) in the general domain, such as ChatGPT, have shown remarkable success in following instructions and producing human-like responses. However, such language models have not been learned individually and …


ChatGPT 101: Introduction to Generative AI

The slides accompany a ChatGPT workshop that is intended for university students to learn more about generative AI and in the context of ChatGPT. This work is licensed under a Creative Commons license so that others may share and adapt the …


ChatGPT and Big Data: Enhancing Text-to-Speech Conversion

HA Dida, DSK Chakravarthy, F Rabbi - Mesopotamian Journal of Big Data, 2023

Text-to-speech (TTS) conversion is a crucial technology for various applications, including accessibility, education, and entertainment. With the rapid growth of big data, TTS conversion systems face new challenges in terms of data size and diversity …


ChatGPT and Environmental Research

JJ Zhu, J Jiang, M Yang, ZJ Ren - Environmental Science & Technology, 2023

ChatGPT, the latest text-based artificial intelligence (AI) tool, has quickly gained popularity and is poised to revolutionize various aspects of our lives, including education and research. With its advanced natural language processing (NLP) …


ChatGPT backend: A comprehensive analysis

A Belgacem, A BRADAI, K Beghdad-Bey - 2023 International Symposium on …, 2023

… Some of the significant datasets used in training different versions of ChatGPT models include booksCorpus, common crawl, english … Some of the most popular types of such datasets are omageNet, common crawl, openstreetMap, million song …


ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning

VD Lai, NT Ngo, APB Veyseh, H Man, F Dernoncourt… - arXiv preprint arXiv …, 2023

… Regarding the classification of high-, medium-, low-, and extremely low-resource languages, our work currently relies on data ratios for the languages in the CommonCrawl corpus. According to our experiments, it is interesting that the performance …


ChatGPT in Education

• ChatGPT is a Large Language Model (LLM) that is finetuned from a Generative Pre-Trained Transformer-3.5 (GPT-3.5) series LLM, produced by OpenAI.• An LLM is a Deep Neural Network (DNN) trained to generate text similar to human language.• The fine-tuning …


ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports

K Jeblick, B Schachtner, J Dexl, A Mittermeier… - arXiv preprint arXiv …, 2022

The release of ChatGPT, a language model capable of generating text that appears human-like and authentic, has gained significant attention beyond the research community. We expect that the convincing performance of ChatGPT incentivizes …


ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Review.

ChatGPT is another large language model (LLM) inline but due to its performance and ability to converse effectively, it has gained a huge popularity amongst research as well as industrial community. Recently, many studies have been published to …


ChatGPT Prompting Cannot Estimate Predictive Uncertainty in High-Resource Languages

M Pelucchi, M Valdenegro-Toro - arXiv preprint arXiv:2311.06427, 2023

… in web content [19] or in the Common Crawl dataset [7] can give an estimation of the distribution of language in ChatGPT’s training data. Table 1 gives an overview of the percentage of web content and of the Common Crawl dataset in different …


ChatGPT versus the neurosurgical written boards: a comparative analysis of artificial intelligence/machine learning performance on neurosurgical board–style …

BS Hopkins, VN Nguyen, J Dallas, P Texakalidis… - Journal of Neurosurgery, 2023

… These same models are further trained on extremely large databases of books, articles, websites, and more through common crawl data extraction, a method of broadly web scraping the internet for incredibly wide variety and depth of available …


ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots

R Omar, O Mangukiya, P Kalnis, E Mansour - arXiv preprint arXiv:2302.06466, 2023

… It was trained on massive datasets from different open-access scientific sources, such as papers and filtered common crawl. Its training datasets also included some general knowledge, such as Wikipedia. Traditional QASs for KGs divide the question-answering …


ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?

H Chen, F Jiao, X Li, C Qin, M Ravaut, R Zhao, C Xiong… - arXiv preprint arXiv …, 2023

Upon its release in late 2022, ChatGPT has brought a seismic shift in the entire landscape of AI, both in research and commerce. Through instruction-tuning a large language model (LLM) with supervised fine-tuning and reinforcement learning from …


ChatGPT, Let Us Chat Sign Language: Experiments, Architectural Elements, Challenges and Research Directions

N Shahin, L Ismail - … International Symposium on Networks, Computers and …, 2023

ChatGPT is a language model based on Generative AI. Existing research work on ChatGPT focused on its use in various domains. However, its potential for Sign Language Translation (SLT) is yet to be explored. This paper addresses this void …


ChatGPT/AIGC and Educational Innovation: Opportunities, Challenges, and the Future

Y Zhu, F Yang - Journal of East China Normal University (Educational …, 2023

… 尤其到 GPT-3 时,数 据已经空前庞大,综合了 Common Crawl(约占 60%),WebText2(约 占 21%),Books1(约占 8%), Books2(约占 8%),Wikipedia(约占 3%)等不同来源的资料,囊括 了基于 3 000 亿单词语料的 1 750 亿参 数量和 45TB 数据量.在海量的数据支撑之下,ChatGPT …


ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope

PP Ray - Internet of Things and Cyber-Physical Systems, 2023

… C4 Corpus: T5 is pretrained on a large-scale dataset called Colossal Clean Crawled Corpus (C4), which is a cleaned and deduplicated version of the Common Crawl dataset. This large-scale pretraining helps T5 learn general language …


ChatGPT: promise and challenges for deployment in low-and middle-income countries

X Wang, HM Sanders, Y Liu, K Seang, BX Tran… - The Lancet Regional Health …, 2023

In lowand middle-income countries (LMICs), the fields of medicine and public health grapple with numerous challenges that continue to hinder patients' access to healthcare services. ChatGPT, a publicly accessible chatbot, has emerged as a …


ChatGPT: towards an AI subjectivity

Motivated by the question of responsible AI and value alignment, I seek to offer a uniquely Foucauldian reconstruction of the problem as the emergence of an ethical subject in a disciplinary setting. This reconstruction contrasts with the strictly human-oriented …


Andronchik, DS ChatGPT–the quintessence of neural networks/DS Andronchik, NS Denisyuk; sci. sup. IY Vanik//Знание иностранного языка как основной фактор для работы в инновационных условиях [Электронный ресурс]: сборник …


ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human

J Tian, H Chen, G Xu, M Yan, X Gao, J Zhang, C Li… - arXiv preprint arXiv …, 2023

In this paper, we present ChatPLUG, a Chinese open-domain dialogue system for digital human applications that instruction finetunes on a wide range of dialogue tasks in a unified internet-augmented format. Different from other open-domain …


Check for Adaptation of Enterprise Modeling Methods for Large Language Models Balbir S. Barn¹, Souvik Barat2, and Kurt Sandkuhl³ (~) 2

BS Barn¹, S Barat, K Sandkuhl - The Practice of Enterprise Modeling: 16th IFIP Working …

… GPT-3 uses 175 billion parameters and is trained on data from the Common Crawl data set1 comprising nearly a trillion words. The development in large language models and their evolution has been widely documented and the reader is …


Check for Matching Production and Test Files: A Comparison of Filename and Statement-Based Approaches Gerald Kipruto Kirui and Stephen Phillip Levitt ()℗

GK Kirui - South African Institute of Computer Scientists and …

Accurate matching of test and production files plays a critical role in the analysis and evaluation of Test-Driven Development (TDD). However, current approaches often yield unsatisfactory results due to their reliance on filename-based matching. The …


Check for updates

D Chernyshev¹, B Dobrov - Analysis of Images, Social Networks and Texts: 11th …

F de Wet¹, R Eiselen, E Schillack¹, M Puttkammer - … South Africa, December 4–8, 2023 …, 2023

… The Common Crawl (CC) data set is likely the largest text data set currently available consisting of (bi-) monthly crawls of the web, and is … Part of the common crawl collection process is language identification to categorise the predominant …


Check for

F Deimling, M Fazzolari - Data and Applications Security and Privacy XXXVII …, 2023

… These models are trained with data from Wikipedia and Common Crawl and evaluated on word analogy tasks. For a triplet of words A: B:: C, the goal is to guess D. For example, for the triplet Paris France:: Berlin:?, the answer would be Germany. …


CheckIT!: A Corpus of Expert Fact-checked Claims for Italian

This paper introduces CheckIT!, a resource of expert fact-checked claims, filling a gap for the development of fact-checking pipelines in Italian. We further investigate the use of three state-of-the-art generative text models to create variations of claims …


ChessGPT: Bridging Policy Learning and Language Modeling

X Feng, Y Luo, Z Wang, H Tang, M Yang, K Shao… - arXiv preprint arXiv …, 2023

… Existing dataset Numerous existing datasets comprise general internet crawl data from platforms like CommonCrawl or Wikipedia. We establish a filtering pipeline to extract only chess-related language corpus from pre-existing language corpus …


Chinesewebtext: Large-scale high-quality Chinese web text extracted with effective evaluation model

J Chen, P Jian, T Xi, Y Yi, C Ding, Q Du, G Zhu, C Zong… - arXiv preprint arXiv …, 2023

… Training Data Composition While the evaluation in our current experiment targets CommonCrawl data, we believe the positive training samples … Since CommonCrawl data has a relatively high noise level overall, we directly sampled …


CHORUS: Foundation Models for Unified Data Discovery and Exploration

M Kayali, A Lykov, I Fountalis, N Vasiloglou, D Olteanu… - arXiv preprint arXiv …, 2023

We explore the application of foundation models to data discovery and exploration tasks. Foundation models are large language models (LLMs) that show promising performance on a range of diverse tasks unrelated to their training. We show that …


Chuweb21D: A Deduped English Document Collection for Web Search Tasks

… The Chuweb21 collection is constructed based on the 2021-17 (April, 2021) block of the Common Crawl dataset 3. We choose Common Crawl as the cornerstone corpus mainly due to its timeliness and accessibility. As a non-profit project …


Cicognini at ACTI: Analysis of techniques for conspiracies individuation in Italian

This report illustrates methods and results for solving SubtaskA (conspiracy detection) and SubtaskB (conspiracy topic classification) of EVALITA 2023 ACTI challenge. We employed different transformer-based models and an original method based on tf-idf …


CiT: Curation in Training for Effective Vision-Language Data

H Xu, S Xie, PY Huang, L Yu, R Howes, G Ghosh… - arXiv preprint arXiv …, 2023

Large vision-language models are generally applicable to many downstream tasks, but come at an exorbitant training cost that only large institutions can afford. This paper trades generality for efficiency and presents Curation in Training (CiT), a …


CKingCoder at SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis

B Harish, D Naveen, P Balasubramanian, S Aarthi - Proceedings of the The 17th …, 2023

The SemEval 2023 Task 9 Multilingual Tweet Intimacy Analysis, is a shared task for analysing the intimacy in the tweets posted on Twitter. The dataset was provided by Pei and Jurgens, who are part of the task organisers, for this task consists of tweets …


CLAD-ST: Contrastive Learning with Adversarial Data for Robust Speech Translation

SR Indurthi, S Chollampatt, R Agrawal, M Turchi - Proceedings of the 2023 …, 2023

The cascaded approach continues to be the most popular choice for speech translation (ST). This approach consists of an automatic speech recognition (ASR) model and a machine translation (MT) model that are used in a pipeline to translate …


CLAMP: Contrastive LAnguage Model Prompt-tuning

P Teterwak, X Sun, BA Plummer, K Saenko, SN Lim - arXiv preprint arXiv:2312.01629, 2023

Large language models (LLMs) have emerged as powerful general-purpose interfaces for many machine learning problems. Recent work has adapted LLMs to generative visual tasks like image captioning, visual question answering, and visual …


Clarifying the Dialogue-Level Performance of GPT-3.5 and GPT-4 in Task-Oriented and Non-Task-Oriented Dialogue Systems

S Iizuka, S Mochizuki, A Ohashi, S Yamashita, A Guo… - 2023

… It has been pre-trained on the Wikipedia and Common Crawl datasets. In this experiment, we utilized a model with 3B parameters4, which is more than the Japanese-dialog-transformers model. We built the system by LoRA-tuning (Hu et al …


Class Cardinality Comparison as a Fermi Problem

S Ghosh, S Razniewski, G Weikum - arXiv preprint arXiv:2303.04532, 2023

… The common crawl [3] is another massive dataset of textual information, almost 50 times greater than Wikipedia (around 5.6 TB) and the BookCorpus (6 GB). It requires significant processing before it can be used for pre-training large LMs [18]. A case …


ClassBases at CASE-2022 Multilingual Protest Event Detection Tasks: Multilingual Protest News Detection and Automatically Replicating Manually Created Event …

P Wiriyathammabhum - arXiv preprint arXiv:2301.06617, 2023

In this report, we describe our ClassBases submissions to a shared task on multilingual protest event detection. For the multilingual protest news detection, we participated in subtask-1, subtask-2, and subtask-4, which are document …


Classifying European Court of Human Rights Cases Using Transformer-Based Techniques

AS Imran, H Hodnefjeld, Z Kastrati, N Fatima… - IEEE Access, 2023

In the field of text classification, researchers have repeatedly shown the value of transformer-based models such as Bidirectional Encoder Representation from Transformers (BERT) and its variants. Nonetheless, these models are expensive in …


Clever little tricks: A socio-technical history of text-to-image generative models

K Steinfeld - International Journal of Architectural Computing, 2023

The emergence of text-to-image generative models (eg, Midjourney, DALL-E 2, Stable Diffusion) in the summer of 2022 impacted architectural visual culture suddenly, severely, and seemingly out of nowhere. To contextualize this …


Clustering of Monolingual Embedding Spaces

Suboptimal performance of cross-lingual word embeddings for distant and low-resource languages calls into question the isomorphic assumption integral to the mapping-based methods of obtaining such embeddings. This paper investigates the comparative …


CNN-Fusion: An Effective and Lightweight Phishing Detection Method Based on Multi-Variant ConvNet

M Hussain, C Cheng, R Xu, M Afzal - Information Sciences, 2023

Phishing scams are increasing as the technical skills and costs of phishing attacks diminish, emphasizing the need for rapid, precise, and low-cost prevention measures. Based on a character-level convolutional neural network (CNN), we present CNN-Fusion …


Co $^ 2$ PT: Mitigating Bias in Pre-trained Language Models through Counterfactual Contrastive Prompt Tuning

X Dong, Z Zhu, Z Wang, M Teleki, J Caverlee - arXiv preprint arXiv:2310.12490, 2023

… cupation classification from the Common Crawl corpus. We report the overall accuracy of the task as well as the accuracy breakdown based on gender. To quantify gender bias, we compute the difference in true positive rates (TPR) between …


COCKATIEL: COntinuous Concept ranKed ATtribution with Interpretable ELements for explaining neural net classifiers on NLP tasks

F Jourdan, A Picard, T Fel, L Risser, JM Loubes… - arXiv preprint arXiv …, 2023

Transformer architectures are complex and their use in NLP, while it has engendered many successes, makes their interpretability or explainability challenging. Recent debates have shown that attention maps and attribution …


CODAMOSA: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models

C Lemieux, JP Inala, SK Lahiri, S Sen - 45th International Conference on Software …, 2023

… For instance, GPT-3 [16] was trained on almost 700 GB of data collected from CommonCrawl [21]. Following the success of LLMs at natural language tasks, there has been tremendous interest in applying LLMs to code. This led to the development …


Code-switching input for machine translation: a case study of Vietnamese–English data

L Nguyen, O Mayeux, Z Yuan - International Journal of Multilingualism, 2023

Multilingualism presents both a challenge and an opportunity for Natural Language Processing, with code-switching representing a particularly interesting problem for computational models trained on monolingual datasets. In this paper, we explore …


CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Model

P Di, J Li, H Yu, W Jiang, W Cai, Y Cao, C Chen… - arXiv preprint arXiv …, 2023

… The Chinese corpus is sourced from CommonCrawl, computer-related websites, documentation of programming languages and their third-party libraries, etc. The English corpus is sampled from various categories in Pile including StackExchange, …


CogAgent: A Visual Language Model for GUI Agents

W Hong, W Wang, Q Lv, J Xu, W Yu, J Ji, Y Wang… - arXiv preprint arXiv …, 2023

… To facilitate robust training in GUI grounding, we have constructed the CCS400K (Common Crawl Screenshot 400K) dataset. This extensive dataset is formed by extracting URLs from the latest Common Crawl data, followed by capturing 400,000 web page …


Collaborative Graph Neural Networks for Attributed Network Embedding

Q Tan, X Zhang, X Huang, H Chen, J Li, X Hu - arXiv preprint arXiv:2307.11981, 2023

… The posts are preprocessed into 602-dimensional feature vectors via Glove CommonCrawl world embedding [122]. Hence, node attributes refer to the 602 latent dimensions. We use the communities or ‘subreddit’ that the post belongs to a target label. …


Collecting and Predicting Neurocognitive Norms for Mandarin Chinese

… The results in Table 4 reveal that embedding models based on FastText and Skip Gram had highly significant correlations with human scores, and that the FastText vectors trained on Common Crawl achieved higher scores than did any of the ones …


COLLIE: Systematic Construction of Constrained Text Generation Tasks

S Yao, H Chen, AW Hanjie, R Yang, K Narasimhan - arXiv preprint arXiv:2307.08689, 2023

Text generation under constraints have seen increasing interests in natural language processing, especially with the rapidly improving capabilities of large language models. However, existing benchmarks for constrained generation usually …


Combating Hate: How Multilingual Transformers Can Help Detect Topical Hate Speech

T Srikissoon, V Marivate - EPiC Series in Computing, 2023

Automated hate speech detection is important to protecting people’s dignity, online experiences, and physical safety in Society 5.0. Transformers are sophisticated pretrained language models that can be fine-tuned for multilingual hate speech …


Common Crawl : Data Collection and Use Cases for NLP

Crawler Politeness Implications •(with well-written robots. txt) less of• private/personal content• duplicated content• significant parts of the web (eg. social media) are not included• links in disallowed content are not visible to the crawler• easy-to-adapt …


CommonCanvas: An Open Diffusion Model Trained with Creative-Commons Images

A Gokaslan, AF Cooper, J Collins, L Seguin… - arXiv preprint arXiv …, 2023

We assemble a dataset of Creative-Commons-licensed (CC) images, which we use to train a set of open diffusion models that are qualitatively competitive with Stable Diffusion 2 (SD2). This task presents two challenges: (1) high-resolution CC images …


Community Competition and Political Extremism

… Second, the LLaMa 1 foundational models are trained on publicly available data sources including the CommonCrawl. The … the 30 seed users used to build the community dataset is present in the CommonCrawl corpus. Accuracy, precision, and …


Company Similarity using Large Language Models

D Vamvourellis, M Toth, S Bhagat, D Desai, D Mehta… - arXiv preprint arXiv …, 2023

… It has been trained on multiple data sources like Common crawl dataset (around 600 billion words of text), GitHub dataset (100 million code repository), Stack overflow dataset (170 million questions and answers) on the task of next word …


Comparative Analysis of Balanced Code Smell Detection Using Machine Learning Check for updates

M Sabharwal, A Gupta, R Gandhi, I Khan - … : Proceedings of the International Conference on …

… Any website which enables API calls to it can be used to collect data or for a more comprehensive analysis, Common Crawl by AWS can be used [5]. The scraped data will be extracted using a local script ran using the python requests library from the …


Comparative Analysis of Machine Learning and Deep Learning Models for Sentiment Analysis in Somali

AA Abdirahman, AO Hashi, MA Elmi, OER Rodriguez

Understanding and analysing sentiment in user-generated content has become crucial with the increasing use of social media and online platforms. However, sentiment analysis in less-resourced languages like Somali poses unique …


Comparing different search methods for the open access journal recommendation tool B! SON

E Entrup, A Eppelin, R Ewerth, J Hartwig, M Tullney… - International Journal on …, 2023

Finding a suitable open access journal to publish academic work is a complex task: Researchers have to navigate a constantly growing number of journals, institutional agreements with publishers, funders’ conditions and the risk of predatory publishers …


Comparing Different Transformer Models' Performance for Identifying Toxic Language Online

There is a growing use of the internet and alongside that, there has been an increase in the use of toxic language towards other people that can be harmful to those that it targets. The usefulness of artificial intelligence has exploded in recent …


Comparing the Similarity of OpenAPI-Based Microservices

Microservices constitute the state of the art for implementing distributed systems and have been seen as a potential solution towards open systems. The characteristics of open systems require structured microservice management, including grouping …


Comparison of Pre-trained vs Custom-trained Word Embedding Models for Word Sense Disambiguation

MF Ullah, A Saeed, N Hussain - ADCAIJ: Advances in Distributed Computing and …, 2023

… First, the pre-trained model by Facebook1, trained on Common Crawl with vector dimension size 300, and a context window of size 5. Second, pre-trained by Khurram (Kanwal et al., 2019) with a vector dimension size of 300, and a context …


Comparison of the Three Algorithms for Concreteness Rating Estimation of English Words

VV Bochkarev, SV Khristoforov, AV Shevlyakova… - Acta Polytechnica …, 2022

… In our case, the size of the CommonCrawl corpus that was used to obtain fastText vector … If we used a corpus which 3 times exceeds the size of CommonCrawl, we would probably … and the use of fastText vectors obtained on the CommonCrawl …


Comparison of Websites Employing Search Engine Optimization and Live Data

S Maitra, L Sahoo, S Sen, K Tiwary - Journal of Computer Science Research, 2023

This study compares websites that take live data into account using search engine optimization (SEO). A series of steps called search engine optimization can help a website rank highly in search engine results. Static websites and dynamic websites …


Complementing Scale: Novel Guidance Methods for Improving Language Models

Abstract Language models (LMs) are at the core of almost all state of the art natural language processing systems. Recent papers, such as Brown et al.[2020] and Hoffmann et al.[2022] have shown that scaling up the size of these models leads to …


Complex business ecosystem intelligence using AI-powered visual analytics

RC Basole, H Park, CD Seuss - Decision Support Systems, 2023

Business ecosystems are complex, dynamic systems characterized by a multitude of entities, including companies, ventures, and technologies, as well as activities and trends. Understanding the state of business ecosystems is an increasingly critical …


Comprehensive Evaluation of ChatGPT Reliability Through Multilingual Inquiries

PCR Puttaparthi, SS Deo, H Gul, Y Tang, W Shang… - arXiv preprint arXiv …, 2023

… First, we recorded languages included in the Common Crawl [8] documentation—a pivotal open-source dataset widely used in the training of ChatGPT. Then, we built a list of languages supported by Google Cloud Translation, which is our chosen …


Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt

Z Xu, Z Liu, B Chen, Y Tang, J Wang, K Zhou, X Hu… - arXiv preprint arXiv …, 2023

While the numerous parameters in Large Language Models (LLMs) contribute to their superior performance, this massive scale makes them inefficient and memory-hungry. Thus, they are hard to deploy on the commodity hardware, such as one single GPU …


Computational Models for Understanding Narrative

N Montfort, RP y Pérez - Revista de Comunicação e Linguagens, 2023

Descrevemos como a modelagem computacional da narrativa serve como um método de investigação e ajuda a aprofundar a compreensão humanística neste domínio. O nosso foco está em nossos próprios sistemas, MEXICA e Curveship …


Computational Study of News Systems: Embracing the Complexity Paradigm

The production and spread of digital news involves a wide range of actors: journalists and the organizations that employ them, social media platforms, audiences, and myriad commentators, citizen journalists, bloggers, and other actors …


Conclusions and Future Research

S Sharoff, R Rapp, P Zweigenbaum - Building and Using Comparable Corpora for …, 2023

… The developers of XLM-Roberta and mT5 used different strategies to create “clean” subsets of the Common Crawl, while the two models are also based on different architectures. What is the impact of the difference in the architectures versus the …


Configuration Validation with Large Language Models

X Lian, Y Chen, R Cheng, J Huang, P Thakkar, T Xu

Misconfigurations are the major causes of software failures. Existing configuration validation techniques rely on manually written rules or test cases, which are expensive to implement and maintain, and are hard to be comprehensive …


Constructing Multilingual Code Search Dataset Using Neural Machine Translation

R Sekizawa, N Duan, S Lu, H Yanaka - arXiv preprint arXiv:2306.15604, 2023

Code search is a task to find programming codes that semantically match the given natural language queries. Even though some of the existing datasets for this task are multilingual on the programming language side, their query data are only in English …


Constructing Temporal Dynamic Knowledge Graphs from Interactive Text-based Games

KP Yu - arXiv preprint arXiv:2311.01928, 2023

… embedding layer, which is initialized using the 300-dimensional fastText [6] word embeddings pretrained on Common Crawl (600B tokens). Lintext is a linear layer that reduces the dimension of the word embeddings to H, which is the dimension of …


Content still matters. A machine learning model for predicting news longevity from textual and context features

K Rybinski - Information Processing & Management, 2023

There is an ongoing debate about what is more important in the modern online media newsroom, whether it is the news content and worthiness, or the audience clicks. Using a dataset of over one million articles from five countries (Belarus …


CONTENT-BASED CHARACTERIZATION OF THE END OF TERM WEB ARCHIVE

ME Phillips, KK Phillips, S Alam - iPRES 2023, 2023

… This work is expected to continue to leverage existing tools and processes developed by Common Crawl for graph generation. With the … Likewise, this project leaned heavily upon the prior work of the Common Crawl team and adopted their …


Context Sensitive Tamil Language Spellchecker Using RoBERTa

R Rajalakshmi, V Sharma - Speech and Language Technologies for Low …, 2023

… In our spellchecker model, we have used xlm-roberta-base model which is a pre-trained model on 2.5 TB of filtered CommonCrawl data containing 100 languages. Our spellchecker takes an input text, finds out the misspelled word and lists out the …


CONTEXT-AND/OR FREQUENCY

… We used BERT – with the Italian model taken from UWAC corpus, Umberto-commoncrawl - and examined the output of the first or projection layer3. In this way we intended to check the predicting ability of BERT on the masked word, by selecting in turn one …


Context-aware Swedish Lexical Simplification: Using pre-trained language models to propose contextually fitting synonyms

This thesis presents the development and evaluation of context-aware Lexical Simplification (LS) systems for the Swedish language. In total three versions of LS models, LäsBERT, LäsBERT-baseline, and LäsGPT, were created and evaluated on …


Contextualizing the Limits of Model & Evaluation Dataset Curation on Semantic Similarity Classification Tasks

D Theron - arXiv preprint arXiv:2311.04927, 2023

This paper demonstrates how the limitations of pre-trained models and open evaluation datasets factor into assessing the performance of binary semantic similarity classification tasks. As (1) end-user-facing documentation around the …


Contextually Enriched Meta-Learning Ensemble Model for Urdu Sentiment Analysis

K Ahmed, MI Nadeem, D Li, Z Zheng, N Al-Kahtani… - Symmetry, 2023

… On the other hand, this model makes use of pre-trained word vectors that were trained on “common crawl” and “Wikipedia” through the use of the fastText model [82]. The CBOW algorithm with position-weighting is used to train this word vector. After …


Continual Pre-Training of Large Language Models: How to (re) warm your model?

K Gupta, B Thérien, A Ibrahim, ML Richter, Q Anthony…

Large language models (LLMs) are routinely pretrained on billions of tokens, only to restart the process over again once new data becomes available. A much cheaper and more efficient solution would be to enable the continual pre-training of these …


Contributions to Neural Theorem Proving

In this dissertation we present several contributions to the nascent field of neural theorem proving, deep learning-driven automated theorem proving over large libraries of formalized mathematics. We work primarily with the Lean theorem prover …


Contributions to the video captioning in an open-world scenario using deep learning techniques

Video captioning poses a significant challenge within the Computer Vision and Artificial Intelligence domains. It involves the challenging task of translating the visual content of videos into natural language descriptions. Despite significant …


Controllable sentence simplification in Swedish: Automatic simplification of sentences using control prefixes and mined Swedish paraphrases

… snapshots using CC-Net [62], an open-source repository with tools to download and clean Common Crawl snapshots. Common Crawl is a non-profit organization that collects and freely provides vast amounts of data from the web, including web …


Conversation Analysis for Computational Modelling of Task-oriented Dialogue

Current methods of dialogue modelling for Conversational AI (CAI) bear little resemblance to the manner in which humans organise conversational interactions. The way utterances are represented, interpreted, and generated are determined by …


Convolutional neural network approach for anomaly-based intrusion detection on IoT-enabled smart space orchestration system

V Upman, N Goranin, A Čenys - DAMSS 2022: 13th conference on data analysis …, 2022

DAMSS-2022 is the 13th International Conference on Data Analysis Methods for Software Systems, held in Druskininkai, Lithuania. Every year at the same place and time. The exception was in 2020, when the world was gripped by the Covid-19 …


Corpus Modeling and the Geometries of Text: Meaning Spaces as Metaphor and Method

We explore the theoretical implications of spatial metaphors in the field of computational text analysis and inspect how the properties of topologies aid and inhibit our theories of textual meaning. Rather than mining for “ground truth” …


Countering Malicious Content Moderation Evasion in Online Social Networks: Simulation and Detection of Word Camouflage

Á Huertas-García, A Martín, JH Tato, D Camacho - arXiv preprint arXiv:2212.14727, 2022

Content moderation is the process of screening and monitoring user-generated content online. It plays a crucial role in stopping content resulting from unacceptable behaviors such as hate speech, harassment, violence against specific groups …


CPG-LS: Causal Perception Guided Linguistic Steganography

L Xiang, J Xia, Y Liu, Y Gui - IEEE Signal Processing Letters, 2023

… Experimental Settings 1) Dataset: A total of 10000 original English texts are randomly selected from the CC-100[27] corpus, which comprises a vast collection of high-quality datasets extracted from the Common Crawl. Each text treated as a cover …


Creating Web-Scale, Annotated Document Understanding Datasets Tailored to Downstream Tasks: An Extension and Validation Study of the WordScape Data …

… As the WordScape pipeline utilizes CommonCrawl purely to find URLs to downloadable Word documents, only .WAT files (containing … ) are extracted from the CommonCrawl metadata. This initial phase requires the downloading of up to …


Creative Use of OpenAI in Education: Case Studies from Game Development

F French, D Levi, C Maczo, A Simonaityte… - Multimodal Technologies …, 2023

… A similar problem occurs with text output because many large generative AI models (LGAIMs), including GPT, have been trained on a vast amount of historical Internet data drawn from the ‘Common Crawl’ publicly available dataset [25], as well …


CreoleVal: Multilingual Multitask Benchmarks for Creoles

H Lent, K Tatariya, R Dabre, Y Chen, M Fekete… - arXiv preprint arXiv …, 2023

Creoles represent an under-explored and marginalized group of languages, with few available resources for NLP research. While the genealogical ties between Creoles and other highly-resourced languages imply a significant potential for …


Critical Perspectives: A Benchmark Revealing Pitfalls in PerspectiveAPI

L Piedras, L Rosenblatt, J Wilkins - arXiv preprint arXiv:2301.01874, 2023

L Rosenblatt, L Piedras, J Wilkins - Proceedings of the Second Workshop on NLP for …, 2022

Detecting “toxic” language in internet content is a pressing social and technical challenge. In this work, we focus on Perspective API from Jigsaw, a state-of-the-art tool that promises to score the “toxicity” of text, with a recent model update that …


Critical Theory of AI

We live in an age of artificial intelligence. Machines think and act in ever more complex ways, making suggestions and decisions on our behalf. While AI might be seen as practical and profitable, issues of data surveillance, algorithmic control, and …


Cross-Genre Argument Mining: Can Language Models Automatically Fill in Missing Discourse Markers?

G Rocha, HL Cardoso, J Belouadi, S Eger - arXiv preprint arXiv:2306.04314, 2023

Available corpora for Argument Mining differ along several axes, and one of the key differences is the presence (or absence) of discourse markers to signal argumentative content. Exploring effective ways to use discourse markers has …


Cross-Lingual Approaches for Text Generation Tasks in Low-Resource Languages

… A challenge to this approach is that such web content is itself very sparse in low-resource languages, as can be observed in publicly available large dumps like CommonCrawl [69]. Hence, it is impossible to build monolingual parallel datasets in …


Cross-lingual Aspect-level Sentiment Classification with Graph Neural Network

鲍小异， 姜晓彤， 王中卿， 周国栋 - Journal of Software, 2023

: 目前, 在属性级情感分类任务上较为成熟的有标注数据集均为英文数据集, 而有标注的 中文数据集较少. 为了能够更好地利用规模庞大但却缺乏成熟标注数据的中文语言数据 集, 针对跨语言属性级情感分类任务进行了研究. 在跨语言属性级情感分类中, 一个核心 …


Cross-lingual Classification of Crisis-related Tweets Using Machine Translation

S Al Amer, M Lee, P Smith - Proceedings of the 14th International Conference on …, 2023

Utilisation of multilingual language models such as mBERT and XLM-RoBERTa has increasingly gained attention in recent work by exploiting the multilingualism of such models in different downstream tasks across different languages. However …


Cross-lingual Machine Translation

GB Mohan, RP Kumar, NL Keerthana, D Mukesh… - Proceedings of Third …, 2023

… There are 2.5 TB of unlabeled text in 100 languages extracted from the Common Crawl databases. Based on the ROBERTa methodology, … language model, and it has 2.5 TB of freshly constructed clean Common Crawl data in 100 languages. It …


Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages

L Qin, Q Chen, F Wei, S Huang, W Che - arXiv preprint arXiv:2310.14799, 2023

… Therefore, we examine the language distribution (refer to Figure 8) in the widely used multilingual pretraining dataset, Common Crawl 2021. Based on the proportions, we incrementally integrated languages in descending and ascending …


Cross-lingual Sentence Embedding for Low-resource Chinese-Vietnamese Based on Contrastive Learning

Y Huang, Y Liang, Z Wu, E Zhu, Z Yu - ACM Transactions on Asian and Low …, 2023

… And it was pre-trained on a new Common Crawl-based dataset covering 101 languages. mT5 inherits all of the benefits of T5 , such as its general-purpose text-to-text format, its design based on insights from a large-scale empirical study, and its scale. …


Cross-Lingual Sentiment Analysis: A Survey

X Yuemei, C Han, W Wenqing, D Wanze, X Chengyang - Data Analysis and …, 2023

[Objective] This paper teases out the research context of cross-lingual sentiment analysis (CLSA).[Coverage] We searched “TS= cross lingual sentiment OR cross lingual word embedding” in Web of Science database and 90 representative papers …


Cross-Lingual Text Reuse Detection at Document Level for English-Urdu Language Pair

M Sharjeel, I Muneer, S Nosheen, RMA Nawab… - ACM Transactions on Asian …, 2023

In recent years, the problem of Cross-Lingual Text Reuse Detection (CLTRD) has gained the interest of the research community due to the availability of large digital repositories and automatic Machine Translation (MT) systems. These systems are …


Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis

S Goldfarb-Tarrant, B Ross, A Lopez - arXiv preprint arXiv:2305.12709, 2023

… Since our pre-training data is from Wikipedia and CommonCrawl, Paracrawl, or the target language equivalent, there is a domain shift between pre-training and fine-tuning data, and between fine-tuning and evaluation data, which are more similar to the pretraining; …


Cross-Lingual Transfer Learning for Misinformation Detection: Investigating Performance Across Multiple Languages

O Ozcelik, AS Yenicesu, O Yildirim, DS Haliloglu… - Proceedings of the 4th …, 2023

… Indeed, 100 languages from 2.5TB of filtered CommonCrawl data were used as its pre-training material. In order to learn embedded representations of multilingual texts, XLM-R employs an unsupervised learning technique. This makes it possible to …


Cross-lingual Transfer Learning with Persian

The success of cross-lingual transfer learning for POS tagging has been shown to be strongly dependent, among other factors, on the (typological and/or genetic) similarity of the lowresource language used for testing and the language (s) used in …


Cross-Lingual Transfer of Cognitive Processing Complexity

C Pouw, N Hollenstein, L Beinborn - arXiv preprint arXiv:2302.12695, 2023

When humans read a text, their eye movements are influenced by the structural complexity of the input sentences. This cognitive phenomenon holds across languages and recent studies indicate that multilingual language models utilize …


Cross-Modal Reasoning with Event Correlation for Video Question Answering

C Yin, Z Che, K Wu, Z Xu, Q Qiu, J Tang - arXiv preprint arXiv:2312.12721, 2023

Video Question Answering (VideoQA) is a very attractive and challenging research direction aiming to understand complex semantics of heterogeneous data from two domains, ie, the spatio-temporal video content and the word sequence in question …


Cross-modal Transformer with Language Query for Referring Image Segmentation

W Zhang, Q Tan, P Li, Q Zhang, R Wang - Neurocomputing, 2023

… As shown in row 2 in Table 5, we use Glove [65] pre-trained on Common Crawl 840B tokens instead of the randomly initialized word embeddings. We found that Glove improves the performance of CMT more than LSTM. …


Cross-Multilingual, Cross-Lingual and Monolingual Transfer Learning For Arabic Dialect Sentiment Classification

Transfer learning have recently proven to be very powerful in diverse Natural language processing (NLP) tasks such as Machine translation, Sentiment Analysis, Question/Answering. In this work, we investigate the use of transfer learning (TL) in …


Cross-portal metadata alignment-connecting open data portals through means of formal concept analysis

M Bogdanović, MF Gligorijević, N Veljković, D Puflović… - Information Sciences, 2023

… The GloVe model we used in this research contains word vector space with meaningful substructure generated and trained on Common Crawl data. Training resulted in 840 billion generated tokens and vocabulary containing 2.2 million …


Cross‐domain sentiment classification using decoding‐enhanced bidirectional encoder representations from transformers with disentangled attention

RK Singh, MK Sachan, RB Patel - Concurrency and Computation: Practice and …

Cross‐domain sentiment classification is a significant task of sentiment analysis that objectives to predict the opinion orientation of text documents in the target domain by using the source domain's learned classifier. Most of the existing approaches of …


Crystal Ball for the Impact of Anthropogenic Climate Change on Global Air Quality PM 2.5

Air pollution caused by PM2. 5 particles is a major global concern, particularly for human health. 19 This study used machine learning tools to uncover the social factors influencing PM2. 5 emissions. 20 Text mining techniques were employed to …


CS 5968/6968: Data Str Alg Scalable Comp Spring 2023

In this lecture, we will focus only on static graphs, where no insertions/deletions to the vertices would be made. An example of such a graph would be the SRA graph of a reference human genome. In the next lecture, we will extend our discussion to …


Cultural Adaptation of Recipes

Y Cao, Y Kementchedjhieva, R Cui, A Karamolegkou… - arXiv preprint arXiv …, 2023

Building upon the considerable advances in Large Language Models (LLMs), we are now equipped to address more sophisticated tasks demanding a nuanced understanding of cross-cultural contexts. A key example is recipe adaptation, which …


CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages

T Nguyen, C Van Nguyen, VD Lai, H Man, NT Ngo… - arXiv preprint arXiv …, 2023

The driving factors behind the development of large language models (LLMs) with impressive learning capabilities are their colossal model sizes and extensive training datasets. Along with the progress in natural language processing, LLMs …


Curricular Transfer Learning for Sentence Encoded Tasks

JMC de Sá, MF Sanches, RR de Souza, JC Reis… - arXiv preprint arXiv …, 2023

Fine-tuning language models in a downstream task is the standard approach for many state-of-the-art methodologies in the field of NLP. However, when the distribution between the source task and target task drifts, \textit{eg}, conversational …


Cyberbullying Detection in Social Networks: A Comparison between Machine Learning and Transfer Learning Approaches

TH Teng, KD Varathan - IEEE Access, 2023

… For comparison purposes, this research employed pre-trained word vectors trained from tokens in Common Crawl, Wikipedia, and Twitter which come with several dimensions (100, 200). The word vectors are publicly available in the …


Czech-ing the News: Article Trustworthiness Dataset for Czech

M Bohacek, M Bravansky, F Trhlík, V Moravec - Proceedings of the 13th Workshop on …, 2023

We present the Verifee dataset: a multimodal dataset of news articles with fine-grained trustworthiness annotations. We bring a diverse set of researchers from social, media, and computer sciences aboard to study this interdisciplinary problem holistically and …


D4: Improving LLM Pretraining via Document De-Duplication and Diversification

K Tirumala, D Simig, A Aghajanyan, A Morcos

… We start with 5 CommonCrawl dumps 3 which range from 2017 to 2020. We then use CC-net (Wenzek … We note that since our dataset is curated from CommonCrawl dumps, there is risk that our training set … (2023), since we use the …


Danish Foundation Models

K Enevoldsen, L Hansen, DS Nielsen, RAF Egebæk… - arXiv preprint arXiv …, 2023

Large language models, sometimes referred to as foundation models, have transformed multiple fields of research. However, smaller languages risk falling behind due to high training costs and small incentives for large companies to train …


Data Acquisition: A New Frontier in Data-centric AI

L Chen, B Acun, N Ardalani, Y Sun, F Kang, H Lyu… - arXiv preprint arXiv …, 2023

As Machine Learning (ML) systems continue to grow, the demand for relevant and comprehensive datasets becomes imperative. There is limited study on the challenges of data acquisition due to ad-hoc processes and lack of consistent …