''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT Generated English Text

R Hada, A Seth, H Diddee, K Bali - arXiv preprint arXiv:2310.17428, 2023

Language serves as a powerful tool for the manifestation of societal belief systems. In doing so, it also perpetuates the prevalent biases in our society. Gender bias is one of the most pervasive biases in our society and is seen in online and offline …


12 The Future of

P Chundi, V Bommanapally, V Gadhamshetty - Machine Learning in 2D Materials …, 2023

… GPT-3 is an autoregressive model built with 96 layers [31], increased embedding vector dimension of 12888, trained with 175B parameters on five datasets including Common crawl, WebText2, Books1, Books2, and Wikipedia. Though GPT-3 has …


14 Augmenting and Informing the Translation Process through Workflow-Enabled CALT Tools

SM Rudan, E Kelbert, L Kovacevic, M Reynolds… - Computer-Assisted Literary …, 2023

CALT is an emerging field in the process of defining itself, but it is already, we feel, a misnomer. Adding an ‘L’(Literary) to the pre-existing notion of Computer-Assisted Translation (CAT) creates continuity with CAT research; Machine Translation (MT) is …


19. Assessing collective identity (non-) verification with social media data through web scraping, sentiment analysis, and qualitative coding

TP Love, JL Davis, RE Davis, WG Fisher, RM Barczak - Handbook of Research …, 2023

… Perhaps the most prolific example of these techniques is Common Crawl, a non-profit organization that employs a web crawling and web scraping program to harvest information across the Internet. Common Crawl has been web crawling and web …


22 Forward Thinking on GeoAI

S Newsam - Handbook of Geospatial Artificial Intelligence, 2023

… Second, is the unsupervised pre-training on large corpora of text such as the Common Crawl archive of the Internet. This pre-training does not require manually labeled data since it largely involves simply determining whether the model correctly …


3.3 On Information and Interactions on the Web

… In natural language processing, large language models such as BERT or GPT are trained on large corpi found on the Web, including Wikipedia and CommonCrawl 4 data. … php 4 https://commoncrawl. org/ 5 https://pedantic-web. org/ …


6 Author-Tailored Neural Machine Translation Systems for Literary Works

A Oliver - Computer-Assisted Literary Translation, 2023

Literature is one of the areas where machine translation (MT) has not been used until very recent times. Nevertheless, the impressive increase in quality obtained with neural machine translation (NMT) systems has led some authors to start …


6 Distributed computational models of intervention effects: A study on cleft structures in French

G Samo, P Merlo - It-Clefts: Empirical and Theoretical Surveys and …, 2023

… Briefly, the uncompressed 270 GB of French text corpus consists of 24 sub-corpora gathered from different sources,(eg Wikipedia and books, random text crawled from the Internet (eg Common Crawl). 6 We use the code discussed in Renaud (2020) …


6 Persian Named Entity Recognition with Structural Prediction Methods

H Poostchi, M Piccardi - Persian Computational Linguistics and NLP, 2023

… As the standard word embedding we have downloaded and used the 300D Farsi fastText embeddings trained on Common Crawl and … Our initial experiments showed that the combination of the fastText embeddings trained on Common Crawl …


" Get a Higher Return on Your Savings!": Comparing Adverts for Cryptocurrency Investment Scams Across Platforms

GA Siu, A Hutchings - 2023 IEEE European Symposium on Security and …, 2023

This work compares machine learning methods using supervised, semi-supervised and unsupervised learning, to classify advertisements for cryptocurrency related investment scams found in the online forum Bitcointalk, and the social media …


" I wouldn't say offensive but...": Disability-Centered Perspectives on Large Language Models

V Gadiraju, S Kane, S Dev, A Taylor, D Wang… - 2023

Large language models (LLMs) trained on real-world data can inadvertently reflect harmful societal biases, particularly toward historically marginalized communities. While previous work has primarily focused on harms related to age and race …


" Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters

Y Wan, G Pu, J Sun, A Garimella, KW Chang, N Peng - arXiv preprint arXiv …, 2023

… 2019a), which is sourced from online biographies in the Common Crawl corpus. The dataset also includes metadata across several occupations and gender indicators. We prompt ChatGPT to rephrase this initial biography into two versions …


" Paraphrasing The Original Text" Makes High Accuracy Long-Context QA

Y Yu - arXiv preprint arXiv:2312.11193, 2023

Although LLMs continue to iterate and improve, most open-source models still have a context window of no more than 4k, limiting their ability to handle long-context problems. Most existing open-source models for long-context chat still lack …


\texttt {IngesTables}: Scalable and Efficient Training of LLM-Enabled Tabular Foundation Models

S Yak, Y Dong, J Gonzalvo, S Arik - NeurIPS 2023 Second Table Representation …, 2023

There is a massive amount of tabular data that can be taken advantage of via `foundation models' to improve prediction performance for downstream tabular prediction tasks. However, numerous challenges constitute bottlenecks in building tabular foundation …


Özgeçmişlerde varlık isimlerinin tanınması Named entity recognition in resumes

E Kesim, S Tanberk - 2023 Innovations in Intelligent Systems and …, 2023

Named entity recognition (NER) is used to extract information from various documents and texts such as names and dates. It is important to extract education and work experience information from resumes in order to filter them. Considering …


“Source?”“I Made It Up”: The Ethics of Citing ChatGPT in Academia

… This should not be necessarily surprising as we do know that most LLMs are trained using the CommonCrawl dataset, a collection of twelve years’ worth of publicly available Internet pages, where sites like AO3 with over 11 million freely …


A Benchmark Analysis of Few-Shot Biomedical Named Entity Recognition

… This led to the development of pretrained systems such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer), which were trained with large language datasets, such as Wikipedia …


A benchmark for evaluating Arabic contextualized word embedding models

A Elnagar, S Yagi, Y Mansour, L Lulu, S Fareh - Information Processing & …, 2023

… It was pre-trained on a 2.5 TB of clean CommonCrawl dataset in 100 languages. … Amassing huge amounts of non-financial data as in the Common Crawl dilutes the patterns found in financial texts. That is the most likely explanation for why so many …


A Benchmark for Text Expansion: Datasets, Metrics, and Baselines

Y Chen, H Jiang, W Bi, R Wang, L Wang, S Shi, R Xu - arXiv preprint arXiv …, 2023

… We use CommonCrawl2 as the data source. Based on the four complementary approaches in §3, we further conduct data filtering (Appendix D) and construct a large-scale TE corpus with 12M pairs for both English and Chinese. Detailed …


A benchmark for toxic comment classification on Civil Comments dataset

C Duchene, H Jamet, P Guillaume, R Dehak - arXiv preprint arXiv:2301.11125, 2023

Toxic comment detection on social media has proven to be essential for content moderation. This paper compares a wide set of different models on a highly skewed multi-label hate speech dataset. We consider inference time and several metrics to …


A BERT-Based Model for Financial Social Media Sentiment Analysis

J Delgadillo, J Kinyua, C Mutigwe - International Journal of Cognitive and Language …, 2023

The purpose of sentiment analysis is to determine the sentiment strength (eg, positive, negative, neutral) from a textual source for good decision-making. Natural Language Processing (NLP) in domains such as financial markets requires …


A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development

T Wu, S He, J Liu, S Sun, K Liu, QL Han, Y Tang - IEEE/CAA Journal of Automatica …, 2023

ChatGPT, an artificial intelligence generated content (AIGC) model developed by OpenAI, has attracted world-wide attention for its capability of dealing with challenging language understanding and generation tasks in the form of …


A Case Study Analysis of Google Smart Compose and Its Effects on the Student Writing Process From the Student and Teacher Perspectives

This qualitative case study addresses the rise in popularity of the use of predictive text programs within the K-12 educational environment. A problem exists in the discrepancy between the widespread availability of Google Smart Compose within …


A Cohesive Distillation Architecture for Neural Language Models

JP Wahle - arXiv preprint arXiv:2301.08130, 2023

A recent trend in Natural Language Processing is the exponential growth in Language Model (LM) size, which prevents research groups without a necessary hardware infrastructure from participating in the development process. This study …


A Comparative Performance Evaluation of Algorithms for the Analysis and Recognition of Emotional Content

K Kyritsis, N Spatiotis, I Perikos, M Paraskevas - 2023

Sentiment Analysis is highly valuable in Natural Language Processing (NLP) across domains, processing and evaluating sentiment in text for emotional understanding. This technology has diverse applications, including social media monitoring, brand …


A Comparative Study of Code Generation using ChatGPT 3.5 across 10 Programming Languages

A Buscemi - arXiv preprint arXiv:2308.04477, 2023

… OpenAI employed a dataset known as the Common Crawl [24], a publicly accessible collection of billions of web pages, making it as one of the most extended text databases currently accessible. It is to be noted that the selection of the dataset …


A Comparative Study of Pre-trained Language Models to Filter Informative Code-mixed Data on Social Media during Disasters

… A multi-lingual language model trained using masked language modeling on 2.5 TB of newly created and cleaned CommonCrawl data. … XLM-R: A multilingual version of RoBERTa that is pre-trained on 2.5TB of filtered CommonCrawl data …


A Comparative Study of Sentence Embedding Models for Assessing Semantic Variation

DM Mistry, AA Minai - arXiv preprint arXiv:2308.04625, 2023

… XLM-R [14] is a transformer trained using masked language modeling on one hundred languages using over two terabytes of filtered CommonCrawl data. The trained model shows significant performance improvement over multilingual BERT (mBERT) …


A Comparative Study of Text Embedding Models for Semantic Text Similarity in Bug Reports

A Patil, K Han, S Mukhopadhyay - arXiv preprint arXiv:2308.09193, 2023

… For Fasttext, we employed the ”crawl-300d2M-subword” model, which consisted of 2 million word vectors trained with subword information on the Common Crawl dataset, encompassing 600 billion tokens. In the case of Doc2Vec, we used the ”GoogleNews-vectors-negative300…


A Comparative Study of Text Representations for French Real-Estate Classified Advertisements Information Extraction

Text representations are widely used in NLP tasks such as text classification. Very powerful models have emerged and been trained on huge corpora for different languages. However, most of the pre-trained models are domain-agnostic and fail …


A Comparative Study of Transformer Based Pretrained AI Models for Content Summarization

ASA Rasheed, MM Masud, M Abduljabbar - 2023 15th International Conference on …, 2023

In this study, we examine different transformer based pretrained Artificial Intelligence (AI) models on their ability to summarize text content from different sources. AI has emerged as a powerful tool in this context, offering the potential to automate and …


A Comparitive Study on Abstractive Text Summarization Techniques Using Deep Learning (ATS-DL)

S Adhithyan, AR Nirupama, S Sri Akshya… - Soft Computing: Theories …, 2023

… It is trained using Common Crawl’s web crawl corpus (C4), a very large corpus of data for all NLP tasks. T5 comes in 5 variants such as T5-small, T5-big, T5-large, T5-3B and T5-11B, where each is trained with 60 million, 220 million, 770 million, 3 billion …


A compendium of data sources for data science, machine learning, and artificial intelligence

P Bilokon, O Bilokon, S Amen - arXiv preprint arXiv:2309.05682, 2023

Recent advances in data science, machine learning, and artificial intelligence, such as the emergence of large language models, are leading to an increasing demand for data that can be processed by such models. While data sources are application-specific …


A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators

C Zhang, LF D'Haro, Y Chen, M Zhang, H Li - arXiv preprint arXiv:2312.15407, 2023

Automatic evaluation is an integral aspect of dialogue system research. The traditional reference-based NLG metrics are generally found to be unsuitable for dialogue assessment. Consequently, recent studies have suggested various unique …


A Comprehensive Overview of Large Language Models

H Naveed, AU Khan, S Qiu, M Saqib, S Anwar… - arXiv preprint arXiv …, 2023

… The dataset is extracted from the public common crawl scrape. The model uses GeGLU activation and trains with a vocab size of 250,000 … 3) PanGu-α [78]: An autoregressive model trained on 1.1TB Chinese data collected from Common Crawl …


A Comprehensive Study of ChatGPT: Advancements, Limitations, and Ethical Considerations in Natural Language Processing and Cybersecurity

M Alawida, S Mejri, A Mehmood, B Chikhaoui… - Information, 2023

This paper presents an in-depth study of ChatGPT, a state-of-the-art language model that is revolutionizing generative text. We provide a comprehensive analysis of its architecture, training data, and evaluation metrics and explore its …


A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT

Y Cao, S Li, Y Liu, Z Yan, Y Dai, PS Yu, L Sun - arXiv preprint arXiv:2303.04226, 2023

Recently, ChatGPT, along with DALL-E-2 and Codex,has been gaining significant attention from society. As a result, many individuals have become interested in related resources and are seeking to uncover the background and secrets behind its …


A Comprehensive Survey of ChatGPT: Advancements, Applications, Prospects, and Challenges

Large Language Models (LLMs) especially when combined with Generative Pre-trained Transformers (GPT) represent a groundbreaking in natural language processing. In particular, ChatGPT, a state-of-the-art conversational language model with a user-friendly …


A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks

S Islam, H Elmekki, A Elsebai, J Bentahar, N Drawel… - arXiv preprint arXiv …, 2023

Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. Unlike conventional neural networks or updated versions of Recurrent Neural Networks (RNNs) such as …


A Computational Analysis on the Role of Social Relationships in Online Communication and Information Diffusion

… 2014] embeddings with 300 dimensions, trained on the Common Crawl corpus (42B tokens) performed best in the tasks we addressed. In addition to considering a word’s local context, GloVe uses also global co-occurrence statistics across the whole text …


A Conversational Breakdown Detector for a Motivational Interviewing Conversational Agent

ZL Qin - The iJournal: Student Journal of the University of …, 2023

… The first is pre-training a large language model on a general corpus, such as Wikipedia or Common Crawl. The next step is adding a classification layer and fine-tuning the model on new data so this model can be used in downstream tasks. They …


A Cross-Attention Augmented Model for Event-Triggered Context-Aware Story Generation

C Tang, T Loakman, C Lin - arXiv preprint arXiv:2311.11271, 2023

Despite recent advancements, existing story generation systems continue to encounter difficulties in effectively incorporating contextual and event features, which greatly influence the quality of generated narratives. To tackle these …


A Data-driven Understanding of Left-Wing Extremists on Social Media

U Balcı, M Sirivianos, J Blackburn - arXiv preprint arXiv:2307.06981, 2023

Social media's role in the spread and evolution of extremism is a focus of intense study. Online extremists have been involved in the spread of online hate, mis/disinformation, and real-world violence. However, the overwhelming majority of existing work has …


A Dataset and Strong Baselines for Classification of Czech News Texts

H Kydlíček, J Libovický - arXiv preprint arXiv:2307.10666, 2023

… We create the CZE-NEC by crawling Czech news websites from CommonCrawl (§ 2.1) and use the available metadata to define classification tasks (§ 2.3). … Instead of crawling the pages directly, we used the CommonCrawl archive to extract the articles. …


A deep learning framework for the detection of Malay hate speech

K Maity, S Bhattacharya, S Saha, M Seera - IEEE Access, 2023

Although social media can efficiently disseminate information, they also facilitate the dissemination of online abuse, harassment, and hate speech. In 2019, United Nations Secretary-General introduced the United Nations Strategy and Plan of …


A Deep Learning-Based Phishing Detection System Using CNN, LSTM, and LSTM-CNN

Z Alshingiti, R Alaqel, J Al-Muhtadi, QEU Haq… - Electronics, 2023

… Around one million legitimate and phishing URLs were used on the dataset collected from PhishTank and Common Crawl. To build the IPDS, the LSTM and CNN classifier used over 10,000 images and one million URLs for training. The …


A Deep Neural Network Architecture for Extracting Contextual Information

Z Alami Merrouni, B Frikh, B Ouhbi - The 3rd International Conference on Artificial …, 2023

… However, existing datasets for AKE are quite small, so we use Stanford's GloVe Embeddings, which are trained on either Wikipedia 2014 + Gigaword 5 (approximately 6 billion tokens) or Common Crawl (approximately 6 billion tokens) (about 840 …


A deep neural network-based data-driven model for evaluating the recognition of ADR mentions in the texts of the PsyTAR corpus

A Sboev, G Rylkov, A Selivanov, I Moloshnikov… - AIP Conference …, 2023

In this paper, for the first time, the accuracy of solving the problem of recognizing named entities of the ADR type for the PsyTAR corpus was evaluated. Initially, the corpus contained markup for solving the problems of detecting entities, classifying …


A Detailed Study on Preventing the Malicious URLs from Cyber Attacks

S Adnaan, G Tejasri, AVP Krishna, MGS Reddy - 2023 Third International …, 2023

Today, there are more malicious URLs and cyber-attacks than ever. Cybercrimes are developing along with the software system’s progress. If we are able to stop one attack, then another one will follow in a different direction. These URLs were given …


A Fast Method to Filter Noisy Parallel Data WMT2023 Shared Task on Parallel Data Curation

The effectiveness of a machine translation (MT) system is intricately linked to the quality of its training dataset. In an era where websites offer an extensive repository of translations such as movie subtitles, stories, and TED Talks, the fundamental …


A Few Words about ChatGPT

P Nag, S Mondal, A Sinha, S Mullick, R Mondal…

ChatGPT is an artificial intelligence chatbot that uses natural language processing to generate human-like conversational talk. The language model can respond to inquiries and generate a wide range of written materials, including articles, social …


A Formal Perspective on Byte-Pair Encoding

V Zouhar, C Meister, JL Gastaldi, L Du, T Vieira… - arXiv preprint arXiv …, 2023

Byte-Pair Encoding (BPE) is a popular algorithm used for tokenizing data in NLP, despite being devised initially as a compression method. BPE appears to be a greedy algorithm at face value, but the underlying optimization problem that BPE …


A Framework for Responsible Development of Automated Student Feedback with Generative AI

ED Lindsay, A Johri, J Bjerva - arXiv preprint arXiv:2308.15334, 2023

… roughly made up of 500 billion words of training data from Common Crawl, constituting a good chunk of the internet [7], rather than on a curated sample of similar assignments, with corresponding expert feedback. It is a generalized model …


A Golden Age: Conspiracy Theories' Relationship with Misinformation Outlets, News Media, and the Wider Internet

HWA Hanley, D Kumar, Z Durumeric - arXiv preprint arXiv:2301.10880, 2023

… For each website in our dataset, we collect all the domain’s HTML pages that were indexed by Common Crawl before August 2021. In addition to Common Crawl data, we further utilize our own website scrapes. We utilize our own crawls, in …


A Graphical Approach to Document Layout Analysis

J Wang, M Krumdick, B Tong, H Halim, M Sokolov… - arXiv preprint arXiv …, 2023

Document layout analysis (DLA) is the task of detecting the distinct, semantic content within a document and correctly classifying these items into an appropriate category (eg, text, title, figure). DLA pipelines enable users to convert documents into structured …


A Hierarchical Encoding-Decoding Scheme for Abstractive Multi-document Summarization

C Shen, L Cheng, Y You, L Bing - arXiv preprint arXiv:2305.08503, 2023

Pre-trained language models (PLMs) have accomplished impressive achievements in abstractive single-document summarization (SDS). However, such benefits may not be readily extended to muti-document summarization (MDS), where the …


A Hybrid Extractive-Abstractive Framework with Pre & Post-Processing Techniques To Enhance Text Summarization

R Habu, R Ratnaparkhi, A Askhedkar, S Kulkarni - 2023 13th International …, 2023

The goal of this paper is to enhance text summarization using a hybrid methodology. The process of producing a condensed version of a text while keeping its essential details is known as text summarization. In this study, we have presented a method …


A Hybrid Method on Emotion Detection for Indonesian Tweets of COVID-19

D Purwitasari, ASS Ansyah, AP Kurniawan… - Jurnal RESTI (Rekayasa …, 2023

As a result of the COVID-19 pandemic, there have been restrictions on activities outside the home which has caused people to interact more and express their emotions through social media platforms, one of which is Twitter. Previous studies …


A Hybrid Model for Multilingual OCR

D Etter, C Carpenter, N King - International Conference on Document Analysis and …, 2023

… This large multilingual set is filtered from the Common Crawl corpus and is often used for the pretraining of large language models. The fonts to render the line images are selected from Google Fonts, Noto Fonts, and GNU Unifont. Table 1 …


A Hybrid Transformer Ensemble Approach for Phishing Website Detection

KS Mandapati, S Meesala, D Maddela, K Ponnada… - … International Conference on …, 2023

The internet has developed into a veritable informational gold mine in the era of the digital age, and URLs (Uniform Resource Locator) act as access points to enormous volumes of material. For a variety of applications, including web content classification …


A Large Language Model Classification Framework (LLMCF)

The rapid advancement of Large Language Models (LLMs) has necessitated a comprehensive framework for categorizing and understanding their diverse characteristics. In response, this paper introduces the" Large Language Model …


A LARGE-SCALE BENCHMARK TWITTER DATASETS FOR COVID-19 SENTIMENT ANALYSIS

… Similarly, for word embeddings, pretrained Word2Vec, GloVe, and fastText embeddings trained on Common Crawl and Wikipedia are used and have 300-D vectors. In addition, we used hybrid models, such as hybrid ranking (HyRank) and …


A Large-Scale Pretrained Deep Model for Phishing URL Detection

Y Wang, W Zhu, H Xu, Z Qin, K Ren, W Ma - ICASSP 2023-2023 IEEE International …, 2023

Phishing attacks have always been a security issue that has attracted great attention in the cyber security community. Recently, the famous pre-trained models is being used as an anti-phishing solution. However, existing studies either simply transfer …


A large-scale web accessibility analysis considering technology adoption

B Martins, C Duarte - Universal Access in the Information Society, 2023

… To collect a sample fit for a large-scale accessibility evaluation, we obtained individual URLs from the CommonCrawl Footnote 6 set. We used the crawl data from November and December 2020. We removed possibly duplicated URLs, as …


A lightweight deep learning architecture for text embedding: Comparison between the usage of Transformers and Mixers for textual embedding

… These datasets contain all kinds of data, from Wikipedia which contains factual information to Common Crawl which is a collection of diverse websites. However, this demands considerable computing resources and long training times. To keep …


A linear time approximation of Wasserstein distance with word embedding selection

S Otao, M Yamada - Proceedings of the 2023 Conference on Empirical …, 2023

Wasserstein distance, which can be computed by solving the optimal transport problem, is a powerful method for measuring the dissimilarity between documents. In the NLP community, it is referred to as word mover’s distance (WMD). One of the …


A low-resource approach to the grammatical error correction of Ukrainian

F Gomez, A Rozovskaya, D Roth - Proceedings of the Second Ukrainian Natural …, 2023

We present our system that participated in the shared task on the grammatical error correction of Ukrainian. We have implemented two approaches that make use of large pre-trained language models and synthetic data, that have been used for error …


A Machine Learning Approach to Government Business Process Re-engineering

A Riyadi, M Kovacs, U Serdült, V Kryssanov - … Conference on Big Data and Smart …, 2023

Governments around the world accumulate large amounts of data but rarely use them to make their daily work more effective. For example, data classification tasks are typically performed manually or with systems that utilize rules created by humans …


A map of words: Retrieving the spatial layout of underground stations from natural

G Anceresi, D Gatti, T Vecchi, M Marelli, L Rinaldi

Recent evidence has indicated that spatial representations, such as large-scale geographical maps, can be retrieved from natural language alone through cognitively plausible distributional-semantic models based on non-spatial …


A meaningful learning method for zero-shot semantic segmentation

X Liu, S Bai, S An, S Wang, W Liu, X Zhao, Y Ma - Science China Information …, 2023

Zero-shot semantic segmentation, which is developed to segment unseen categories, has attracted increasing attention due to its strong practicability. Previous approaches usually applied semantic-visual mapping based on seen categories to …


A Measurement-Based Quantum-Like Language Model for Text Matching

W Zhang, G Gan, H Gao, P Zhang, W Hui, Z Fan - … 22–26, 2022, Proceedings, Part III, 2023

… The initialization method of word embeddings in the word representation utilizes the 300-dimensional GloVe word vectors pretrained from the 840B Common Crawl corpus [20]. Meanwhile, the word embeddings for the out-of-vocabulary words were …


A Multi-Modal Multilingual Benchmark for Document Image Classification

Y Fujinuma, S Varia, N Sankaran, S Appalaraju, B Min… - arXiv preprint arXiv …, 2023

… LayoutXLM is also pretrained on documents from Common Crawl (CC) and Donut on synthetic Wikipedia documents (Syn. Wiki). InfoXLM … LayoutXLM is pretrained on PDFs from 53 languages extracted from Common Crawl6, thus …


A Multi-solution Study on GDPR AI-enabled Completeness Checking of DPAs

MI Azeem, S Abualhaija - arXiv preprint arXiv:2311.13881, 2023

Specifying legal requirements for software systems to ensure their compliance with the applicable regulations is a major concern to requirements engineering (RE). Personal data which is collected by an organization is often shared with other …


A Multi-Task Multi-Stage Transitional Training Framework for Neural Chat Translation

C Zhou, Y Liang, F Meng, J Zhou, J Xu, H Wang… - IEEE Transactions on …, 2022

Neural chat translation (NCT) aims to translate a cross-lingual chat between speakers of different languages. Existing context-aware NMT models cannot achieve satisfactory performances due to the following inherent problems: 1) limited …


A multimodal turn in Digital Humanities. Using contrastive machine learning models to explore, enrich, and analyze digital visual historical collections

T Smits, M Wevers - Digital Scholarship in the Humanities, 2023

Until recently, most research in the Digital Humanities (DH) was monomodal, meaning that the object of analysis was either textual or visual. Seeking to integrate multimodality theory into the DH, this article demonstrates that recently developed …


A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity

Y Bang, S Cahyawijaya, N Lee, W Dai, D Su, B Wilie… - arXiv preprint arXiv …, 2023

… Based on the percentage of data in the CommonCrawl8, we group languages into 3 8CommonCrawl (http://commoncrawl.org) is the primary source of language pre-training data used in GPT3 … The languages are sorted based on the language …


A Neighbourhood-Aware Differential Privacy Mechanism for Static Word Embeddings

D Bollegala, S Otake, T Machide, K Kawarabayashi - arXiv preprint arXiv:2309.10551, 2023

We propose a Neighbourhood-Aware Differential Privacy (NADP) mechanism considering the neighbourhood of a word in a pretrained static word embedding space to determine the minimal amount of noise required to guarantee a specified …


A New Era of Artificial Intelligence in Education: A Multifaceted Revolution

F Kamalov, I Gurrib - arXiv preprint arXiv:2305.18303, 2023

The recent high performance of ChatGPT on several standardized academic test has thrust the topic of artificial intelligence (AI) into the mainstream conversation about the future of education. The objective of the present study is to investigate the …


A New Frontier: AI and Ancient Language Pedagogy

EAS Ross - Journal of Classics Teaching, 2023

… OpenAI notes, and ChatGPT also reiterates when prompted, that the AI model was trained using a blend of filtered Common Crawl data, WebText, ebooks, and Wikipedia articles from before Q4 2021, so it is unable to answer questions or …


A new phishing-website detection framework using ensemble classification and clustering

M Alsharaiah, A Abu-Shareha, M Abualhaj, L Baniata… - International Journal of Data …, 2023

… The phishing samples were collected from the OpenPhish and Phish Tank, while the legitimate samples were gathered from different resources such as Common Crawl and Alexa. These samples were collected from January to May 2015 and May …


A novel application of machine learning and zero-shot classification methods for automated abstract screening in systematic reviews

CF Moreno-Garcia, C Jayne, E Elyan… - Decision Analytics Journal, 2023

… examples of pre-trained transformer-based NLP systems are Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformers (GPT), which were trained with large language datasets, such as the …


A novel Data and Model Centric artificial intelligence based approach in developing high-performance Named Entity Recognition for Bengali Language

KA Lima, K Md Hasib, S Azam, A Karim, S Montaha… - Plos one, 2023

Named Entity Recognition (NER) plays a significant role in enhancing the performance of all types of domain specific applications in Natural Language Processing (NLP). According to the type of application, the goal of NER is to identify …


A PhD Student's Perspective on Research in NLP in the Era of Very Large Language Models

O Ignat, Z Jin, A Abzaliev, L Biester, S Castro, N Deng… - arXiv preprint arXiv …, 2023

Recent progress in large language models has enabled the deployment of many generative NLP applications. At the same time, it has also led to a misleading public discourse that ``it's all been solved.'' Not surprisingly, this has in turn made many …


A PHISHING DETECTION SYSTEM BASED ON INTELLIGENT DEEP LEARNING TECHNIQUES

GC Uzoaru, G Nwamuruamu, C Johnson-Okoronkwo - Int'l Journal of Education …, 2023

… Approximately one million authentic and fraudulent URLs were employed in the dataset gathered from PhishTank and Common Crawl. Over 10,000 images and one million URLs were used in training for the CNN classifier and LSTM to create the …


A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models

Y Zhou, J Camacho-Collados, D Bollegala - arXiv preprint arXiv:2310.12936, 2023

… Domain: Full-text biomedical papers from the Semantic Scholar Open Research Corpus (S2ORC), PubMed Abstracts (PMed), PMC Fulltext articles and the Medical Information Mart for Intensive Care III (MIMIC3); Finally, we also consider a …


A Preference Judgment Interface for Authoritative Assessment

… of Common Crawl4. There is more information about document collection on the Track website. Each document in the collection that is provided for the 2https://huggingface.co/datasets/allenai/c4 3https://www.tensorflow.org/datasets/catalog/c4 4https://commoncrawl.org …


A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity

S Longpre, G Yauney, E Reif, K Lee, A Roberts, B Zoph… - arXiv preprint arXiv …, 2023

… For each version we begin with Common Crawl data and remove all data that was scraped after the cutoff year. Following Luu et al. (2021), we measure the effect of temporal misalignment by using evaluation tasks (from News, Twitter, and …


A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

I Gur, H Furuta, A Huang, M Safdari, Y Matsuo, D Eck… - arXiv preprint arXiv …, 2023

… For HTML-denoising, we prepare the corpus from CommonCrawl with (Extracted) or without (Raw) subtree extraction around label elements on the documents. We also compare the initialization of base architectures before HTML-denoising; from …


A RelEntLess Benchmark for Modelling Graded Relations between Named Entities

A Ushio, JC Collados, S Schockaert - arXiv preprint arXiv:2305.15002, 2023

Relations such as "is influenced by", "is known for" or "is a competitor of" are inherently graded: we can rank entity pairs based on how well they satisfy these relations, but it is hard to draw a line between those pairs that satisfy them and those …


A Research On The New Generation Artificial Intelligence Technology Generative Pretraining Transformer 3

N Aydın, OA Erdem - 2022 3rd International Informatics and Software …, 2022

In the digitalizing world, Artificial Intelligence (AI) paves the way for the automation of routine work done by humans and makes life easier. Recently, there is no area where AI and its applications are not used in daily life, from health to education, from …


A Review of Conversational Agents in Education

C Rodrigues, A Reis, R Pereira, P Martins, J Sousa… - International Conference on …, 2022

The use of mobile conversations is increasing all around the world. A conversational agent (CA) is mostly useful due to the fast response times and their simple nature. Recently, we have seen the development and increasing use of dialog systems on …


A Review of Generative AI from Historical Perspectives

Many applications of Generative AI (such as DALLE, GPT-3, ChatGPT, etc.) are making headline news in recent months and have been receiving both praise and criticism for their far reaching implications. Some of these applications include query …


A review of the approaches to neural machine translation

PK Buttar, MK Sachan - Natural Language Processing and Information …, 2023

… (2019b) is a collection of 4.5 billion parallel sentences in 576 language pairs pulled from the snapshots of the CommonCrawl public dataset. WikiMatrix (Schwenk et al., 2019a) is another parallel collection of 135 million parallel sentences for 1,620 …


A review on big data based on deep neural network approaches

M Rithani, RP Kumar, S Doss - Artificial Intelligence Review, 2023

… The highest recall and accuracy scores for Common Crawl (94.98 and 97.97, respectively) were linked with the highest F-measure score (96.46). There is no denying that this is an exciting piece of research, however the strategy used to …


A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges

MAK Raiaan, MSH Mukta, K Fatema, NM Fahad… - 2023

… Besides, several corpus including the Common Crawl web corpus, the BooksCorpus dataset, and the English Wikipedia are also used … ) Books (Dec 2015) Books (Dec 2021) CommonCrawl (Sep 2019) CommonCrawl (Feb 2019) …


A Review on Malicious URLs Detection Using Machine Learning Methods

T Tabassum, MM Alam, MS Ejaz, MK Hasan - Journal of Engineering Research and …, 2023

Malicious URLs are a serious threat to cybersecurity because they can compromise user security and inflict large financial losses. The extensiveness and adaptability of traditional detection approaches which rely on blacklists are limited when it comes to …


A Sentence Alignment Approach to Document Alignment and Multi-faceted Filtering for Curating Parallel Sentence Pairs from Web-crawled Data

This paper describes the AST submission to the WMT23 Shared Task on Parallel Data Curation. We experiment with two approaches for curating data from the provided web-scraped texts. We use sentence alignment to identify document …


A Simple Baseline for Knowledge-Based Visual Question Answering

A Xenos, T Stafylakis, I Patras, G Tzimiropoulos - arXiv preprint arXiv:2310.13570, 2023

This paper is on the problem of Knowledge-Based Visual Question Answering (KB-VQA). Recent works have emphasized the significance of incorporating both explicit (through external databases) and implicit (through LLMs) knowledge to answer questions …


A Study of Continual Learning Under Language Shift

E Gogoulou, T Lesort, M Boman, J Nivre - arXiv preprint arXiv:2311.01200, 2023

The recent increase in data and model scale for language model pre-training has led to huge training costs. In scenarios where new data become available over time, updating a model instead of fully retraining it would therefore provide significant …


A Study of Multilingual versus Meta-Learning for Language Model Pre-Training for Adaptation to Unseen Low Resource Languages

J Khatri, R Murthy, AP Azad, P Bhattacharyya

In this paper, we compare two approaches to train a multilingual language model:(i) simple multilingual learning using data-mixing, and (ii) meta-learning. We examine the performance of these models by extending them to unseen language pairs and …


A Study on Bias Detection and Classification in Natural Language Processing

… An example of this is the work done by Luccioni and Viviano [19] on the Common Crawl Corpus1, with a focus on finding instances of Hate Speech and sexually explicit content. The Common Crawl is a multilingual corpus, composed of 200 to …


A Study on the Utilization of OpenAI ChatGPT as a Second Language Learning Tool

S Kim, J Shim, J Shim - Journal of Multimedia Information System, 2023

In November 2022, ChatGPT was introduced and caused a sensation, gathering 100 million users only within two months. ChatGPT’s capability of generating high-quality sentences demonstrated potential to be applied in a wide range of fields. In this …


A Survey of Large Language Models in Medicine: Progress, Application, and Challenge

H Zhou, B Gu, X Zou, Y Li, SS Chen, P Zhou, J Liu… - arXiv preprint arXiv …, 2023

Large language models (LLMs), such as ChatGPT, have achieved substantial attention due to their impressive human language understanding and generation capabilities. Therefore, the application of LLMs in medicine to assist physicians and …


A Survey of Large Language Models

WX Zhao, K Zhou, J Li, T Tang, X Wang, Y Hou, Y Min… - arXiv preprint arXiv …, 2023

… Based on CommonCrawl, there are four filtered datasets that are commonly used in existing work: C4 [71], CCStories [102], CC-News [27], and RealNews [103]. The Colossal Clean Crawled Corpus (C4) includes five variants10, namely en (806G), en.noclean …


A Survey of Text Representation and Embedding Techniques in NLP

R Patil, S Boit, V Gudivada, J Nandigam - IEEE Access, 2023

Natural Language Processing (NLP) is a research field where a language in consideration is processed to understand its syntactic, semantic, and sentimental aspects. The advancement in the NLP area has helped solve problems in the …


A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trends

X Qu, Y Gu, Q Xia, Z Li, Z Wang, B Huai - arXiv preprint arXiv:2302.03512, 2023

As more and more Arabic texts emerged on the Internet, extracting important information from these Arabic texts is especially useful. As a fundamental technology, Named entity recognition (NER) serves as the core component in information …


A Survey on Graph Neural Network Acceleration: Algorithms, Systems, and Customized Hardware

S Zhang, A Sohrabizadeh, C Wan, Z Huang, Z Hu… - arXiv preprint arXiv …, 2023

… -dimensional vectors concatenating the average embedding of the post title and that of all the post’s comments, the post’s score, and the number of comments made on the post, where the average embedding is obtained by averaging the 300-dimensional …


A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage

MU Hadi, R Qureshi, A Shah, M Irfan, A Zafar… - 2023

Within the vast expanse of computerized language processing, a revolutionary entity known as Large Language Models (LLMs) has emerged, wielding immense power in its capacity to comprehend intricate linguistic patterns and conjure coherent and …


A Survey on LLM-gernerated Text Detection: Necessity, Methods, and Future Directions

J Wu, S Yang, R Zhan, Y Yuan, DF Wong, LS Chao - arXiv preprint arXiv:2310.14724, 2023

The powerful ability to understand, follow, and generate complex language emerging from large language models (LLMs) makes LLM-generated text flood many areas of our daily lives at an incredible speed and is widely accepted by …


A Systematic Literature Review on Phishing Website Detection Techniques

A Safi, S Singh - Journal of King Saud University-Computer and …, 2023

Phishing is a fraud attempt in which an attacker acts as a trusted person or entity to obtain sensitive information from an internet user. In this Systematic Literature Survey (SLR), different phishing detection approaches, namely Lists Based, Visual …


A systematic review of applications of natural language processing and future challenges with special emphasis in text-based emotion detection

S Kusal, S Patil, J Choudrie, K Kotecha, D Vora… - Artificial Intelligence Review, 2023

Artificial Intelligence (AI) has been used for processing data to make decisions, Interact with humans, and understand their feelings and emotions. With the advent of the Internet, people share and express their thoughts on day-to-day activities and …


A Systematic Review of Transformer-Based Pre-Trained Language Models through Self-Supervised Learning

E Kotei, R Thirunavukarasu - Information, 2023

… A typical example is the CommonCrawl corpus [75]. Models such as IndoNLG [76], MuRIL [77], IndicNLPSuite [78], mT5 [79], mT6 [80], XLM… derived from Common Crawl corpus [75] Introduced the mT5 multilingual variant of the T5 model pretrained …


A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets

MTR Laskar, MS Bari, M Rahman, MAH Bhuiyan… - arXiv preprint arXiv …, 2023

The development of large language models (LLMs) such as ChatGPT has brought a lot of attention recently. However, their evaluation in the benchmark academic datasets remains under-explored due to the difficulty of evaluating the generative …


A T5-based interpretable reading comprehension model with more accurate evidence training

B Guan, X Zhu, S Yuan - Information Processing & Management, 2024

Pre-trained language models (PLMs) have achieved outstanding performance on Machine Reading Comprehension (MRC) tasks, but these models’ interpretability remains uncertain. In this paper, we exploit the strengths of the pre-trained T5 (Text-to-Text …


A Thorough Analysis of E-commerce Customer Reviews in Arabic Language Using Deep Learning Techniques for Successful Marketing Decisions

… AraVec provides six-word embedding models that have already been pretrained, and these models use data from Twitter, Wikipedia, and Common Crawl. The overall quantity of tokens that were utilized in the creation of the models is more remarkable …


A Tool Kit for Relation Induction in Text Analysis

Distances derived from word embeddings can measure a range of gradational relations—similarity, hierarchy, entailment, and stereotype—and can be used at the document-and author-level in ways that overcome some of the limitations of …


A transformer-based deep learning model for Persian moral sentiment analysis

B Karami, F Bakouie, S Gharibzadeh - Journal of Information Science, 2023

Moral expressions in online communications can have a serious impact on framing discussions and subsequent online behaviours. Despite research on extracting moral sentiment from English text, other low-resource languages, such as Persian …


A Transformer-based Generative Adversarial Learning to Detect Sarcasm from Bengali Text with Correct Classification of Confusing Text

SK Lora, I Jahan, R Hussain, R Shahriyar… - Heliyon

Sarcasm detection research in Bengali is still limited due to a lack of relevant resources. In this context, getting high-quality annotated data is costly and time-consuming. Therefore, in this paper, we present a transformer-based generative adversarial …


A transformer-based multi-task framework for joint detection of aggression and hate on social media data

S Ghosh, A Priyankar, A Ekbal, P Bhattacharyya - Natural Language Engineering, 2023

… It is a massive multi-lingual model that was trained on 2.5 TB of CommonCrawl data in 100 distinct languages. It outperforms other transformer models, such as Bidirectional Encoder Representations from Transformers (BERT) and Multi-lingual …


A User-Centered Evaluation of Spanish Text Simplification

A de Wynter, A Hevia, SQ Chen - arXiv preprint arXiv:2308.07556, 2023

We present an evaluation of text simplification (TS) in Spanish for a production system, by means of two corpora focused in both complex-sentence and complex-word identification. We compare the most prevalent Spanish-specific readability scores …


A Vision for Semantically Enriched Data Science

U Khurana, K Srinivas, S Galhotra, H Samulowitz - arXiv preprint arXiv:2303.01378, 2023

The recent efforts in automation of machine learning or data science has achieved success in various tasks such as hyper-parameter optimization or model selection. However, key areas such as utilizing domain knowledge and data semantics are …


ABCD Team at FinancES 2023: An Unified Generative Framework for the Financial Targeted Sentiment Analysis in Spanish

This paper presents our participation in the IBERLEF 2023 Task-FinancES in Spanish, focusing on two sub-tasks: Financial targeted sentiment analysis and Financial Sentiment Analysis at the document level for companies and consumers …


Abstractive summarization with deep reinforcement learning using semantic similarity rewards

FB Fikri, K Oflazer, B Yanıkoğlu - Natural Language Engineering, 2023

Abstractive summarization is an approach to document summarization that is not limited to selecting sentences from the document but can generate new sentences as well. We address the two main challenges in abstractive summarization: how to …


Abstractive Text Summarization based on Transformer Deep Neural Networks

… GPT-3 (Generative Pre-trained Transformers) [32] is large autoregressive language model (LLM) with 175 billion of parameters that is trained on 300 billion tokens of text (570GB in total Common Crawl dataset) and tested in the fewshot …


ABSTRACTIVE TEXT SUMMARIZATION FOR MORPHOLOGICALLY RICH LANGUAGES

The exponential growth in the number of documents available on the Web has turned finding the relevant piece of information into a challenging, tedious, and timeconsuming activity. Accordingly, automatic text summarization has become an …


Abstractive Text Summarization for Tamil Language Using m-T5

C Saraswathi, V Prinitha, J Briskilal - Inventive Systems and Control: Proceedings of …, 2023

Summarization is the act of reducing a long form of text to a shorter version by lowering the length of the original text while retaining and keeping the summary’s core and the informational parts which conveys the content’s meaning. Summarizing …


Abstractive Text Summarization of Hindi Corpus Using Transformer Encoder-Decoder Model

R Bhansali, A Bhave, G Bharat, V Mahajan, ML Dhore - International Symposium on …, 2023

… FastText has been trained on Hindi text procured from Wikipedia and Common Crawl using positional weights with a Continuous Bag-of-Words (CBOW) model giving word representations of dimension 300. This model is known to give efficient …


Abusive Span Detection for Vietnamese Narrative Texts

NT Nguyen, K Thi-Kim Phan, DV Nguyen… - Proceedings of the 12th …, 2023

Abuse in its various forms, including physical, psychological, verbal, sexual, financial, and cultural, has a negative impact on mental health. However, there are limited studies on applying natural language processing (NLP) in this field in Vietnam …


AcademicGPT: Empowering Academic Research

S Wei, X Xu, X Qi, X Yin, J Xia, J Ren, P Tang, Y Zhong… - arXiv preprint arXiv …, 2023

… for classification using the obtained 200K data; (4) we conduct classification on the common crawl data and clean out 12.7B tokens from CC. … Please assess the provided CommonCrawl sample based on the following criteria and return the …


Accelerating the integration of ChatGPT and other large‐scale AI models into biomedical research and healthcare

DQ Wang, LY Feng, JG Ye, JG Zou, YF Zheng - MedComm–Future Medicine, 2023

… For example, GPT-3 was trained on 300B tokens, with 60% coming from the filtered Common Crawl data set, and the rest from webtext2 (used to train GPT-2), Books1, Books2, Wikipedia, and code datasets (such as GitHub Code). The …


Accessing Higher Dimensions for Unsupervised Word Translation

SI Wang - arXiv preprint arXiv:2305.14200, 2023

The striking ability of unsupervised word translation has been demonstrated with the help of word vectors / pretraining; however, they require large amounts of data and usually fails if the data come from different domains. We propose coocmap, a …


Accurate and Reliable Classification of Unstructured Reports on Their Diagnostic Goal Using BERT Models

MT Rietberg, VB Nguyen, J Geerdink, O Vijlbrief… - Diagnostics, 2023

Understanding the diagnostic goal of medical reports is valuable information for understanding patient flows. This work focuses on extracting the reason for taking an MRI scan of Multiple Sclerosis (MS) patients using the attached free-form reports …


Accurate Knowledge Distillation with n-best Reranking

H Setiawan - arXiv preprint arXiv:2305.12057, 2023

We propose extending the Sequence-level Knowledge Distillation (Kim and Rush, 2016) with n-best reranking to consider not only the top-1 hypotheses but also the top n-best hypotheses of teacher models. Our approach leverages a diverse set of …


Acoustic and linguistic representations for speech continuous emotion recognition in call center conversations

M Macary, M Tahon, Y Estève, D Luzzati - arXiv preprint arXiv:2310.04481, 2023

The goal of our research is to automatically retrieve the satisfaction and the frustration in real-life call-center conversations. This study focuses an industrial application in which the customer satisfaction is continuously tracked down to …


ADA: An Attention-Based Data Augmentation Approach to Handle Imbalanced Textual Datasets

AK Sah, M Abulaish - … : 29th International Conference, ICONIP 2022, Virtual …, 2023

… For classification tasks throughout this work, we have used \(300-\)dimensional GloVe embeddings trained on the Common Crawl dataset with 840B tokens. For BERT-related tasks, we have used the BERT base uncased pre-trained model …


Adaptation of Enterprise Modeling Methods for Large Language Models

BS Barn, S Barat, K Sandkuhl - IFIP Working Conference on The Practice of …, 2023

… GPT-3 uses 175 billion parameters and is trained on data from the Common Crawl data set Footnote 1 comprising nearly a trillion words. The development in large language models and their evolution has been widely documented and the …


Adapting an English Corpus and a Question Answering System for Slovene

U Šmajdek, M Zupanič, M Zirkelbach, M Jazbinšek - Slovenščina 2.0: empirične …, 2023

Pomanjkanje ustreznih podatkov za učenje je ena od ključnih težav pri razvoju slovenskih modelov za odgovarjanje na vprašanja (QA). Sodobna orodja za strojno prevajanje lahko to težavo rešijo, vendar pa se pri njihovi uporabi soočimo z novih …


Adapting Grounded Visual Question Answering Models to Low Resource Languages

Y Wang, J Pfeiffer, N Carion, Y LeCun, A Kamath - … of the IEEE/CVF Conference on …, 2023

… by classifying and filtering the Common Crawl dataset by languages. The statistics for different languages in OSCAR are shown in Table 1. Compared to unimodal datasets, multilingual multimodal datasets are much harder to obtain and …


Adaptive Training Distributions with Scalable Online Bilevel Optimization

D Grangier, P Ablin, A Hannun - arXiv preprint arXiv:2311.11973, 2023

Large neural networks pretrained on web-scale corpora are central to modern machine learning. In this paradigm, the distribution of the large, heterogeneous pretraining data rarely matches that of the application domain. This work considers …


Adding guardrails to advanced chatbots

Y Wang, L Singh - arXiv preprint arXiv:2306.07500, 2023

… According to OpenAI, 60% of the training data come from Common Crawl, a large data set consisting of web pages, extracted metadata and text extractions through a big web crawler since 2008. Another 22% of data are from WebText2, containing all …


Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models

S Prabhumoye, M Patwary, M Shoeybi, B Catanzaro - arXiv preprint arXiv …, 2023

Pretrained large language models have become indispensable for solving various natural language processing (NLP) tasks. However, safely deploying them in real world applications is challenging because they generate toxic content. To address …


Addressing Controversial Topics in Search Engines

… top 500 documents from the Common Crawl Index1 on eight controversial topics. Then, several sequence-to-sequence classifiers were developed to perform the task both in a cross-topic and in-topic settings. The results of the experiments show very …


ADM+ S submission to the Safe and responsible AI in Australia discussion paper

The ADM+ S is pleased to have this opportunity to engage with an important and complex question which confronts Australia: how should the Australian federal government take actionregulatory or otherwise-to promote artificial intelligence (and …


Adopting machine translation in the healthcare sector: A methodological multi-criteria review

M Zappatore, G Ruggieri - Computer Speech & Language, 2023

Background: The recent advances in machine translation (MT) offer an appealing and low-cost solution to overcome language barriers in multiple contexts (eg, travelling, cultural interaction, digital content localisation). However, highly-technical …


Advanced Methods to Audit Online Web Services

Online web services have grown dramatically in size and diversity in the last years, becoming essential components of our daily life and allowing us to conduct elementary tasks like working, getting informed, or keeping in contact with relatives …


Advanced Network Technologies and Intelligent Computing: Second International Conference, ANTIC 2022, Varanasi, India, December 22–24, 2022, Proceedings …

I Woungang, SK Dhurandher, KK Pattanaik, A Verma… - 2023

The 2nd International Conference on Advanced Network Technologies and Intelligent Computing (ANTIC-2022) was organized by Department of Computer Science, Institute of Science, Banaras Hindu University, Varanasi, India in hybrid …


Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers

S Bengesi, H El-Sayed, MK Sarker, Y Houkpati… - arXiv preprint arXiv …, 2023

… CommonCrawl was used to train the model, which was tested on all domains of NLP, and it had promising few-short and zero-shot performance. This version was further improved to GPT 3.5, which was used to develop ChatGPT. Considerable …


Advances in apparent conceptual physics reasoning in ChatGPT-4

CG West - arXiv preprint arXiv:2303.17012, 2023

… While the exact details of the text which was used to “train” ChatGPT are a proprietary secret, it is known that its reading material was largely drawn from the “Common Crawl” corpus [18], an open repository of data scraped from text found on the public …


Advances in monolingual and crosslingual automatic disability annotation in Spanish

I Goenaga Azcarate, E Andrés Santamaría… - 2023

Background Unlike diseases, automatic recognition of disabilities has not received the same attention in the area of medical NLP. Progress in this direction is hampered by obstacles like the lack of annotated corpus. Neural architectures learn …


Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPT

M Hu, S Pan, Y Li, X Yang - arXiv preprint arXiv:2304.04920, 2023

In this paper, we aimed to provide a review and tutorial for researchers in the field of medical imaging using language models to improve their tasks at hand. We began by providing an overview of the history and concepts of language models, with a …


Advancing Neural Encoding of Portuguese with Transformer Albertina PT

J Rodrigues, L Gomes, J Silva, A Branco, R Santos… - arXiv preprint arXiv …, 2023

To advance the neural encoding of Portuguese (PT), and a fortiori the technological preparation of this language for the digital age, we developed a Transformer-based foundation model that sets a new state of the art in this respect for two of its variants …


Adversarial Capsule Networks for Romanian Satire Detection and Sentiment Analysis

SV Echim, RA Smădu, AM Avram, DC Cercel, F Pop - International Conference on …, 2023

Satire detection and sentiment analysis are intensively explored natural language processing (NLP) tasks that study the identification of the satirical tone from texts and extracting sentiments in relationship with their targets. In languages with fewer …


AfriCLIRMatrix: Enabling Cross-Lingual Information Retrieval for African Languages

O Ogundepo, X Zhang, S Sun, K Duh, J Lin - Proceedings of the 2022 Conference on …, 2022

… is to exploit existing large multilingual corpora, eg, the Common Crawl1 and Wikipedia. For example, the HC4 corpus for cross-lingual information retrieval was created from Common Crawl data (Lawrie et al.… 1https://commoncrawl.org 2https://www.elastic.co/elasticsearch/ …


Aggregating Users' Online Opinions Attributes and News Influence for Cryptocurrencies Reputation Generation

A Boumhidi, A Benlahbib - Journal of Universal Computer Science, 2023

… The model was pre-trained on the Common Crawl’s web crawl corpus (C4) and it achieved state-of-the-art results on many natural language processing tasks while being flexible enough to be fine-tuned to a variety of important downstream tasks …


Agile Modeling: From Concept to Classifier in Minutes

O Stretcu, E Vendrow, K Hata, K Viswanathan, V Ferrari… - Proceedings of the IEEE …, 2023

… Since our prototype requires an unlabeled source of images from which to source training labels, we use the LAION-400M dataset [53], due to its large size and comprehensive construction based on the large Common Crawl web corpus. We …


Agile Modeling: Image Classification with Domain Experts in the Loop

O Stretcu, E Vendrow, K Hata, K Viswanathan, V Ferrari… - arXiv preprint arXiv …, 2023

Machine learning is not readily accessible to domain experts from many fields, blocked by issues ranging from data mining to model training. We argue that domain experts should be at the center of the modeling process, and we introduce the "Agile …


AGNETHA FLORE AND RÜDIGER LANG

APMKIN LARGE, H NUHN, A OSWALD - … Future of Project Management: Adapting to …, 2023

Digital Transformation seems to be ubiquitous, and it is radically changing humans’ lives. Artificial Intelligence (AI) and Machine Learning (ML) are two very effective drivers of this transformation. Following Vial (2019), we understand Digital …


AI and Law: The Next Generation

K Lee, AF Cooper, J Grimmelman, D Ippolito - 2023

It’s impossible to read about technological innovation right now without hearing the term “generative AI.” We are in a moment of seemingly nonstop excitement (and seemingly nonstop lawsuits) about the future of AI-assisted content creation, and the …


AI and the Creative Process: Part Two

… Stable Difusion by Stability AI, for example, was developed using image and caption pairs from the LAION-5B dataset, a publicly accessible collection derived from Common Crawl data gathered from the web. This vas dataset consiss of 5 …


AI and the FCI: Can ChatGPT Project an Understanding of Introductory Physics?

CG West - arXiv preprint arXiv:2303.01067, 2023

… While the exact details of the text which was used to “train” ChatGPT are a proprietary secret, it is known that its reading material was largely drawn from the “Common Crawl” corpus [13], an open repository of data scraped from text found on the public …


AI as a License Review Assistant

I will present the steps we have taken to develop a prototype AI assistant for license review. I’ll explain our criteria for the selection of an AI tool for this project. We reviewed ChatGPT, Claude 2, Bard, and PDF readers. My goal was to develop an …


AI Design Issues in Education

J Adams, KL Riddle - ITP Research Symposium 2022 30 November–2 …

… CommonCOW: Massively huge web corpora from CommonCrawl data and a method to distribute them freely under restrictive EU copyright laws. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)(pp …


AI Ethics and Critique for Robotics

In this thesis I consider using 3D computer vision for social good. In particular, I present a broad and deep array of AI ethics methodologies and practices necessary to assess the harms and benefits of a particular AI technology or application. Many …


AI for social good: social media mining of migration discourse

The number of international migrants has steadily increased over the years, and it has become one of the pressing issues in today’s globalized world. Our bibliometric review of around 400 articles on Scopus platform indicates an increased interest in …


AI model GPT-3 (dis) informs us better than humans

G Spitale, N Biller-Andorno, F Germani - arXiv preprint arXiv:2301.11924, 2023

Artificial intelligence is changing the way we create and evaluate information, and this is happening during an infodemic, which has been having dramatic effects on global health. In this paper we evaluate whether recruited individuals can …


AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap

QV Liao, JW Vaughan - arXiv preprint arXiv:2306.01941, 2023

The rise of powerful large language models (LLMs) brings about tremendous opportunities for innovation but also looming risks for individuals and society at large. We have reached a pivotal moment for ensuring that LLMs and LLM-infused …


AI, Large Language Models and University Education

D Psarras, Z Sotireli, I Valsamara, I Pitas

… [FOU] The Common Crawl Foundation. Common crawl. URL http://commoncrawl.org. [ALE2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. …


AI-Assisted Tasks and Materials Development for Language Instruction

X Kou - Center for Language Technology, 2023

• Multilingual compacity• Comprehension of Text• Generating Text• Engaging in Conversation with ChatGPT• ChatGPT-based Writing Task• Prompt engineering• An example of Lesson Plan generation• Cautions (errors, etc.)• Alternative tools• Role of …


AI-Augmented Surveys: Leveraging Large Language Models for Opinion Prediction in Nationally Representative Surveys

J Kim, B Lee - arXiv preprint arXiv:2305.09620, 2023

How can we use large language models (LLMs) to augment surveys? This paper investigates three distinct applications of LLMs fine-tuned by nationally representative surveys for opinion prediction -- missing data imputation, retrodiction …


AI-based novelty detection in crowdsourced idea spaces

J Just, T Ströhle, J Füller, K Hutter - Innovation, 2023

Processing large and heterogeneous numbers of ideas submitted to crowdsourcing contests is a regular challenge for idea evaluators. The aim of this study is to investigate a potential use case for AI-based innovation management and to extend …


AI-Driven Confidential Computing across Edge-to-Cloud Continuum

S Zobaed - arXiv preprint arXiv:2301.00928, 2023

With the meteoric growth of technology, individuals and organizations are widely adopting cloud services to mitigate the burdens of maintenance. Despite its scalability and ease of use, many users who own sensitive data refrain from fully …


AI-Generated Content (AIGC) for Various Data Modalities: A Survey

Amidst the rapid advancement of artificial intelligence (AI), the development of content generation techniques stands out as one of the most captivating and widely discussed topics in the field. AI-generated content (AIGC) encompasses the …


AI-generated vs human-authored texts: A Multidimensional comparison

TB Sardinha - Applied Corpus Linguistics, 2023

The goal of this study is to assess the degree of resemblance between texts generated by artificial intelligence (GPT) and (written and spoken) texts produced by human individuals in real-world settings. A comparative analysis was conducted …


AI-POWERED TEXT ANALYSIS TOOL FOR SENTIMENT ANALYSIS

In today’s digital era, text data plays a ubiquitous role across various domains. This bachelor’s thesis focuses on the field of sentiment analysis, specifically addressing the task of classifying text into positive, negative, or neutral sentiments with the help …


AIGC Empowering Telecom Sector White Paper

Y Ouyang, Y Zhang, X Ye, Y Liu, Y Song, Y Liu, S Bian… - arXiv preprint arXiv …, 2023

In the global craze of GPT, people have deeply realized that AI, as a transformative technology and key force in economic and social development, will bring great leaps and breakthroughs to the global industry and profoundly influence the future world …


ALEXSIS+: Improving Substitute Generation and Selection for Lexical Simplification with Information Retrieval

K North, A Dmonte, T Ranasinghe, M Shardlow… - … on Innovative Use of NLP for …, 2023

… We retrieve instances from the CommonCrawl News (CC-News) dataset2 by searching for the 386 English, 381 Spanish, and 386 Portuguese complex words given to the original participants of TSAR-2022. The CC-News dataset contains …


Algorithmic bias and discrimination through digitalisation in education

R Eynon - World Yearbook of Education 2024: Digitalisation of …, 2023

… One example is the Common-Crawl data set, created by a San Francisco-based non-profit organisation that regularly crawls the web. The size of this and similar data sets is vast: as of April 2021, the Common-Crawl archive was around 320 TB in …


Algorithms learn and propagate gender‐biased representations of consumers

S Rathee, S Banker, A Mishra, H Mishra - Journal of Consumer Psychology

… In an analysis based on billions of documents from Common Crawl, we build on prior research to illustrate that algorithms can learn gender-biased associations with marketplace-relevant psychographic attributes even when there is no psychological …


Algorithms propagate gender bias in the marketplace—with consumers' cooperation

… In an analysis based on billions of documents from Common Crawl, we build on prior research to illustrate that algorithms can learn genderbiased associations with marketplacerelevant psychographic attributes even when there is no psychological …


Alzheimer Disease Classification through ASR-based Transcriptions: Exploring the Impact of Punctuation and Pauses

L Gómez-Zaragozá, S Wills, C Tejedor-Garcia… - arXiv preprint arXiv …, 2023

… Then, the pre-trained FastText embedding trained on Common Crawl [32] was used to convert the transcriptions of the participants’ responses into word vectors. These representations then were fed into a neural network (NN), composed of a …


AMOE: A Tool to Automatically Extract and Assess Organizational Evidence for Continuous Cloud Audit

F Deimling, M Fazzolari - IFIP Annual Conference on Data and Applications …, 2023

The recent spread of cloud services has enabled many companies to take advantage of them. Nevertheless, the main concern about the adoption of cloud services remains the lack of transparency perceived by customers regarding security …


Amplifying Limitations, Harms and Risks of Large Language Models

M O'Neill, M Connor - arXiv preprint arXiv:2307.04821, 2023

… We know that GPT-3 was trained on a combination of sources drawn from the wilds of the internet, including Common Crawl data from 2016 to 2019, Webtext, Books and Wikipedia [10]. Analysis of Common Crawl data has shown that it …


An Ambient Intelligence-based Approach For Longitudinal Monitoring of Verbal and Vocal Depression Symptoms

A Othmani, M Muzammel - arXiv preprint arXiv:2308.08472, 2023

… For word embedding, we utilize the fastText pretrained network [15], which was trained on Common Crawl 1 and incorporates sub-word information. This network produces word vectors of size 300. In cases where certain words are not present in …


An Analysis on the Basic Technologies of ChatGPT

Q Li, L Yi, Z Zhixiong, L Xuesi, X Jing, X Qinya, L Yang… - Data Analysis and …, 2023

[Objective] Review and analyze the corpus, algorithms and models related to ChatGPT, and provide a systematic reference for peer research.[Methods] This paper systematically reviewed the relevant literature and materials since the release …


An Empirical Comparison of Web Content Extraction Algorithms

J Bevendorff, S Gupta, J Kiesel, B Stein - 2023

… Ungoliant uses a FastText [19] language classifier to identify natural-language text in large amounts of web data from the Common Crawl. Although extraction precision is a concern, the focus is clearly on recall. A more specific task than main …


An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation

P Gao, R Zhang, Z He, H Wu, H Wang - arXiv preprint arXiv:2308.14482, 2023

Consistency regularization methods, such as R-Drop (Liang et al., 2021) and CrossConST (Gao et al., 2023), have achieved impressive supervised and zero-shot performance in the neural machine translation (NMT) field. Can we also boost end-to-end …


An Ensemble Method to Classify Telugu Idiomatic Sentences using Deep Learning Models

J Briskilal, CVMS Praneeth, C Chaitanya, MJ Karthik… - … International Conference on …, 2023

… the English Wikipedia that is utilized in BERT, The extra data included the Common Crawl News dataset, which has 63 million items and 76 gigabytes, the Web text corpus, and the Common Crawl Stories dataset (31 GB) makes the simple …


An Ensemble-Based Approach for Generative Language Model Attribution

H Abburi, M Suesserman, N Pudota, B Veeramani… - International Conference on …, 2023

… xlm-roberta-large-finetuned-conll03-english is XLM-RoBERTa based model [2] which is a large multi-lingual language model trained on 2.5TB of filtered Common Crawl data. The conll03-english model is fine-tuned on the XLM-RoBERTa model …


An Evaluation of Persian-English Machine Translation Datasets with Transformers

A Sartipi, M Dehghan, A Fatemi - arXiv preprint arXiv:2302.00321, 2023

Nowadays, many researchers are focusing their attention on the subject of machine translation (MT). However, Persian machine translation has remained unexplored despite a vast amount of research being conducted in languages with high resources …


An Evaluation on Large Language Model Outputs: Discourse and Memorization

A de Wynter, X Wang, A Sokolov, Q Gu, SQ Chen - arXiv preprint arXiv:2304.08637, 2023

We present an empirical evaluation of various outputs generated by nine of the most widely-available large language models (LLMs). Our analysis is done with off-the-shelf, readily-available tools. We find a correlation between percentage of memorized text …


An Explainable Ensemble of Multi-View Deep Learning Model for Fake Review Detection

R Mohawesh, S Xu, M Springer, Y Jararweh… - Journal of King Saud …, 2023

Online reviews significantly impact consumers who are purchasing or seeking services via the Internet. Businesses and review platforms need to manage these online reviews to avoid misleading customers through fake ones. This necessitates …


An Extractive Question Answering System for the Tamil Language

A Krishnan, SR Sriram, BVR Ganesan, S Sridhar - Advances in Science and …, 2023

… This model has been trained on a huge amount of cleaned common crawl data. For low resource languages, new unlabeled corpora were generated. XLM-R is a variant of RoBERTa, which in turn is a variant of BERT [6]. …


An Improved Framework for Scaling Party Positions from Texts using Transformer and Supervised Dimension Reduction

Previous research in both Natural Language Processing (NLP) and political science has proven the superiority of the Transformer architecture compared to word frequencies and word embeddings on multiple text analysis tasks. In this article, I …


An improved sentiment classification model based on data quality and word embeddings

A Siagh, FZ Laallam, O Kazar, H Salem - The Journal of Supercomputing, 2023

User-generated content on social media platforms has reached big data levels. Sentiment analysis of this data provides opportunities to gain valuable insights into any domain. However, analyzing real-world data may confront the challenge of class …


An Integrated Analysis of Multilingual Texts Spanning Dual Alphabets

FT Аdilova, RR Davronov, RA Safarov - INTERNATIONAL JOURNAL OF …, 2023

Abstract Language recognition in natural language processing (NLP) aims to determine the specific language of a text or document. As the number of languages increases, this task becomes more complex. This study introduces a detailed model …


An Investigation Into Supervision for Seq2Seq Techniques for Natural Language to Code Translation

… The dataset that was used to train this was called “Common Crawl”, an open source dataset consisting of nearly a trillion words. On top of this, the authors filtered a version of Common Crawl that was most similar to a range of high-quality corpora …


An Investigation of Preference Judging Consistency

Preference judging has been proposed as an effective method to identify the most relevant documents for a given search query. In this thesis, we investigate the degree to which assessors using a preference judging system are able to …


An Investigation of Representation and Allocation Harms in Contrastive Learning

S Maity, M Agarwal, M Yurochkin, Y Sun - arXiv preprint arXiv:2310.01583, 2023

… In this experiment, we study the potential harms of CL applied to data obtained from Common Crawl, a popular source of text data for self-… 2019) which consists of around 400k online biographies in English extracted from the Common Crawl data …


An Open Source Data Contamination Report for Llama Series Models

Y Li - arXiv preprint arXiv:2310.17589, 2023

… Our approach utilises a search engine and the Common Crawl index, avoiding the need to host the full 2017-2020 Common Crawl dumps locally. This massive training data would incur prohibitive computational requirements. However, relying …


An Urgency for Inclusivity: Redesigning Datasets for Improved Representation of LGBTQ+ Identity Terms in Artificial Intelligence (AI)

… The Common Crawl Dataset stands as a vital resource in the realm of AI model training, … However, despite the Common Crawl's prominence, handling LGBTQ+ identity terms in AI … ’s C4 dataset, a filtered version of the Common Crawl.The …


Analogical Proportions and Creativity: A Preliminary Study

S Afantenos, H Prade, LC Bernardes - arXiv preprint arXiv:2310.13500, 2023

Analogical proportions are statements of the form "$a$ is to $b$ as $c$ is to $d$", which expresses that the comparisons of the elements in pair $(a, b)$ and in pair $(c, d)$ yield similar results. Analogical proportions are creative in the sense that given 3 …


ANALOGICAL--A New Benchmark for Analogy of Long Text for Large Language Models

T Wijesiriwardene, R Wickramarachchi, BG Gajera… - arXiv preprint arXiv …, 2023

… In addition three other corpora containing news articles, web content, and a filtered subset of the CommonCrawl corpus were used. The training approach of RoBERTa differs from BERT as follows. RoBERTa modifies the MLM task by moving …


Analysing Cross-Lingual Transfer in Low-Resourced African Named Entity Recognition

M Beukman, M Fokam - arXiv preprint arXiv:2309.05311, 2023

Transfer learning has led to large gains in performance for nearly all NLP tasks while making downstream models easier and faster to train. This has also been extended to low-resourced languages, with some success. We investigate the …


ANALYSIS OF A DECISION SUPPORT SYSTEM USING AHP FOR FOOD AND RESTAURANT SELECTION BASED ON THE USER'S FOOD CRAVINGS AND …

D Dyondra, J Purnama, M Galinium - SGU Online Thesis Submission, 2023

The purpose of this research is to develop a decision support system (DSS) using the AHP algorithm for selecting restaurants based on the user's food cravings and location in Jakarta. The data for the DSS was gathered by scraping restaurant data …


Analysis of ChatGPT as a Question-Answering Tool

P Pichappan, M Krishnamurthy, P Vijayakumar - Journal of Digital Information …, 2023

… Its data sources are drawn from Common Crawl (a non-profit organisation that crawls the web data and makes it available to open the stored information and data volume open), WebText2, Books and Wikipedia. WebText2 is the extraction of the …


Analysis of Deep Learning Model Combinations and Tokenization Approaches in Sentiment Classification

Sentiment classification is a natural language processing task to identify opinions expressed in texts such as product or service reviews. In this work, we analyze the effects of different deep-learning model combinations, embedding methods, and …


Analysis of Digital Information in Storage Devices Using Supervised and Unsupervised Natural Language Processing Techniques

LA Martínez Hernández, AL Sandoval Orozco… - Future Internet, 2023

… Sources supplementing the training data include news data (CommonCrawl news, 76 GB), text from websites (38 GB), and stories (31 GB). The training of this data is performed over the course of a day using the latest generation graphics card. This …


Analysis of Fake News Detection Methods

M Amjad, O Vitman, G Sidorov, A Zhila, A Gelbukh - … : Selected Papers of the 8th World …, 2023

The fact that it might be difficult to distinguish between real news and fake news is of the utmost significance. It is difficult, time-consuming, and expensive to manually review enormous volumes of digital data in order to detect false news, which is …


Analysis of Machine Learning and Deep Learning Algorithms for Text Sentiment Detection

Sentiment analysis or opinion mining is the task of automatically extracting and classifying the sentiment of the text. It can be applied in numerous fields such as marketing, customer service, etc. Sentiment analysis has gained much attention in …


Analysis of the Evolution of Advanced Transformer-Based Language Models: Experiments on Opinion Mining

NE Zekaoui, S Yousfi, M Rhanoui, M Mikram - arXiv preprint arXiv:2308.03235, 2023

… However, the multilingual version of RoBERTa [15] called XLM-RoBERTa [21], pre-trained on the newly created 2.5TB multilingual CommonCrawl corpus containing 100 different languages, has further pushed the performance. It has shown strong …


Analysis of the Performance Impact of Fine-Tuned Machine Learning Model for Phishing URL Detection

SR Abdul Samad, S Balasubaramanian, AS Al-Kaabi… - Electronics, 2023

… Alexa and Common Crawl are used to compile lists of legitimate websites; whereas Phish-Tank and OpenPhish are used to compile lists of malicious ones. Binary labels, such as 0 for legitimate and 1 for phishing, are present in this dataset …


Analysis of the Semantic Shift in Diachronic Word Embeddings for Spanish Before and After COVID-19

ER Betancourt, EC Murillo - CLEI Electronic Journal, 2023

… In this work, we analyzed millions of Spanish web pages collected by the CommonCrawl … The documents were downloaded from CommonCrawl, discarding the documents with less … 2018 and week 25 of 2021 were downloaded …


Analysis of the Technical Principles of ChatGPT and Prospects for Pre-trained Large Models

Z Jin - 2023 IEEE 3rd International Conference on Information …, 2023

ChatGPT is a pre-trained model in the field of natural language processing. As a generative model, the technical foundation of ChatGPT is a deep learning model called the "Generative Adversarial Network". The pre-trained large model …


Analyzing and Improving the Scalability of In-Memory Indices for Managed Search Engines

… We create indices of various sizes using the January 2022 CommonCrawl data set [43], a large repository of web crawl data. Index size in Lucene is difficult to control precisely. We, therefore, index … https://commoncrawl.org/2022/06/may-2022-crawl-archivenow-available/ …


Analyzing Failure Modes of Inscrutable Machine Learning Models

ABSTRACT Machine learning models and in specific, neural networks, are well known for being inscrutable in nature. From image classification tasks and generative techniques for data augmentation, to general purpose natural language …


Analyzing Syntactic Generalization Capacity of Pre-trained Language Models on Japanese Honorific Conversion

R Sekizawa, H Yanaka - arXiv preprint arXiv:2306.03055, 2023

Using Japanese honorifics is challenging because it requires not only knowledge of the grammatical rules but also contextual information, such as social relationships. It remains unclear whether pre-trained large language models (LLMs) can flexibly …


Analyzing Text Representations by Measuring Task Alignment

C Gonzalez-Gutierrez, A Primadhanty, F Cazzaro… - arXiv preprint arXiv …, 2023

Textual representations based on pre-trained language models are key, especially in few-shot learning scenarios. What makes a representation good for text classification? Is it due to the geometric properties of the space or because it is well …


Analyzing the Impact of Tokenization on Multilingual Epidemic Surveillance in Low-Resource Languages

S Mutuvi, E Boros, A Doucet, G Lejeune, A Jatowt… - International Conference on …, 2023

… On the other hand, XLM-RoBERTa is pre-trained on 2.5TB of CommonCrawl data covering 100 languages. As a result, when compared to Wikipedia data, CommonCrawl data is significantly larger, effectively increasing the available …


Analyzing the unrestricted web: The finnish corpus of online registers

V Skantsi, V Laippala - Nordic Journal of Linguistics, 2023

… Second, Finnish documents were identified and retrieved from Common Crawl, an organization that crawls the Internet providing its archives and datasets for public use (https://commoncrawl.org/). Texts were cleaned from menus and listings with …


and JR Orozco-ArroyaveD

D Escobar-Grisales, SA Moreno-Acevedo… - Applied Computer Sciences in …

… This model was pre-trained with 2.5 TB of text data from the Common Crawl corpus [17], which includes text in 100 different languages. This extensive pre-training has allowed RoBERTa to compute text representations in multiple languages …


Annotation for computational argumentation analysis: Issues and perspectives

A Lindahl, L Borin - Language and Linguistics Compass, 2024

Abstract Argumentation has long been studied in a number of disciplines, including several branches of linguistics. In recent years, computational processing of argumentation has been added to the list, reflecting a general interest from the field …


Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Models

A Nyffenegger, M Stürmer, J Niklaus - arXiv preprint arXiv:2308.11103, 2023

Anonymity of both natural and legal persons in court rulings is a critical aspect of privacy protection in the European Union and Switzerland. With the advent of LLMs, concerns about large-scale re-identification of anonymized persons are growing. In …


ANSEL Photobot: A Robot Event Photographer with Semantic Intelligence

D Rivkin, G Dudek, N Kakodkar, D Meger, O Limoyo… - arXiv preprint arXiv …, 2023

Our work examines the way in which large language models can be used for robotic planning and sampling, specifically the context of automated photographic documentation. Specifically, we illustrate how to produce a photo-taking robot with …


App2Check at EMit: Large Language Models for Multilabel Emotion Classification

In this paper we compare the performance of three state-of-the-art LLM-based approaches for multilabel emotion classification: fine-tuned multilingual T5 and two few shot prompting approaches: plain FLAN and ChatGPT. In our experimental …


Applications of Natural Language Processing for the Management of Stroke Disorders: Scoping Review

H De Rosario, S Pitarch-Corresa, I Pedrosa… - JMIR Medical Informatics, 2023

… maturity of more complex representations of language data, such as the word embeddings into large-dimensional numeric vectors and their effective processing through deep neural networks, as well as the exploitation of huge databases of texts …


Ar-PuFi: A Short-Text Dataset to Identify the Offensive Messages Towards Public Figures in the Arabian Community

M Abdelhakim, B Liu, C Sun - Expert Systems with Applications, 2023

The fight against offensive speech on the Internet necessitates increased efforts from linguistic analysis and artificial intelligence perspectives to develop countermeasures and preventive methods. Reliable predictions can only be …


Arabic Mini-ClimateGPT: A Climate Change and Sustainability Tailored Arabic LLM

S Mullappilly, A Shaker, O Thawakar, H Cholakkal… - Findings of the Association …, 2023

Climate change is one of the most significant challenges we face together as a society. Creating awareness and educating policy makers the wide-ranging impact of climate change is an essential step towards a sustainable future. Recently, Large …


Arabic Tweets-Based Sentiment Analysis to Investigate the Impact of COVID-19 in KSA: A Deep Learning Approach

A Alqarni, A Rahman - Big Data and Cognitive Computing, 2023

… The FastText model was also trained by [68] on Wikipedia and common crawl used the CBOW architecture. The results of the comparison between all pre-trained word embeddings for all datasets in two periods are shown in Table 7 and Table 8. …


AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing

A Alghamdi, X Duan, W Jiang, Z Wang, Y Wu, Q Xia… - arXiv preprint arXiv …, 2023

Developing monolingual large Pre-trained Language Models (PLMs) is shown to be very successful in handling different tasks in Natural Language Processing (NLP). In this work, we present AraMUS, the largest Arabic PLM with 11B parameters trained …


AraProp at WANLP 2022 Shared Task: Leveraging Pre-Trained Language Models for Arabic Propaganda Detection

GS Tomar - Proceedings of the The Seventh Arabic Natural …, 2022

This paper presents the approach taken for the shared task on Propaganda Detection in Arabic at the Seventh Arabic Natural Language Processing Workshop (WANLP 2022). We participated in Sub-task 1 where the text of a tweet is provided, and the …


Are Large Language Models Fit For Guided Reading?

P Ochieng - arXiv preprint arXiv:2305.10645, 2023

This paper looks at the ability of large language models to participate in educational guided reading. We specifically, evaluate their ability to generate meaningful questions from the input text, generate diverse questions both in terms of content …


Are Large Language Models Geospatially Knowledgeable?

P Bhandari, A Anastasopoulos, D Pfoser - arXiv preprint arXiv:2310.13002, 2023

… [29] by processing Commoncrawl 6 snapshots. Given that the LLaMA model was trained directly on Commoncrawl snapshots, we believe that CC100 is a good approximation of the training data. We conduct an analysis similar to Elazar et al. [7] …


Argot as a Trust Signal: Slang, Jargon & Reputation on a Large Cybercrime Forum

… Using the FastText models trained on CommonCrawl and HackForums data, we obtain word embeddings for all tokens in the HackForums dataset which have over 100 occurrences across all posts (including multiple usage in a post). We threshold …


ArgU: A Controllable Factual Argument Generator

S Saha, R Srihari - arXiv preprint arXiv:2305.05334, 2023

Effective argumentation is essential towards a purposeful conversation with a satisfactory outcome. For example, persuading someone to reconsider smoking might involve empathetic, well founded arguments based on facts and expert …


ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models

Y Liu, Z Zhang, W Zhang, S Yue, X Zhao, X Cheng… - arXiv preprint arXiv …, 2023

AI generated content (AIGC) presents considerable challenge to educators around the world. Instructors need to be able to detect such text generated by large language models, either with the naked eye or with the help of some tools. There is …


Argumentation Element Annotation Modeling using XLNet

C Ormerod, A Burkhardt, M Young, S Lottridge - arXiv preprint arXiv:2311.06239, 2023

This study demonstrates the effectiveness of XLNet, a transformer-based language model, for annotating argumentative elements in persuasive essays. XLNet's architecture incorporates a recurrent mechanism that allows it to model long-term …


Artificial cheerleading in IEO: Marketing campaign or pump and dump scheme

Y Tian, Y Xie - Information Processing & Management, 2024

… XLM-R, on the other hand, was trained on a newly curated and clean CommonCrawl dataset spanning 2.5 TB in 100 different languages. The model has surpassed previous multilingual models like mBERT and XLM in tasks such as …


Artificial Intelligence (AI) Art Generators in the Architectural Design Curricula

KE Hedges - 2023 ASEE Annual Conference & Exposition, 2023

… A web crawler, such as the non-profit Common Crawl, is a bot that trolls and scrapes the web for … Schäfer, “CommonCOW: Massively Huge Web Corpora from CommonCrawl Data and a Method to Distribute them Freely under Restrictive EU …


Artificial Intelligence and Librarianship: Page 1 Martin Frické Artificial Intelligence and Librarianship: Notes for Teaching Page 2 2 Martin Frické, Professor Emeritus School of Information The University of Arizona, Tucson, AZ, USA mfricke@arizona.edu …


Artificial intelligence applications in technical communication for the optimization of content management processes

KB Hanczaryk, W Ziegler - AIP Conference Proceedings, 2023

… The training was done on texts from Wikipedia and Common Crawl. The vocabulary of the model includes nearly 4,947,000 words, each represented by a vector of length 100 [21, 22]. The similarity of words can be determined by the …


Artificial intelligence as a tool in social media content moderation

On social media, users engage with each other through consuming and creating user-generated content, the amount of which has increased alongside the growth of the userbase. This presents challenges to social media companies as some …


Artificial Intelligence for Business Creativity provides an in-depth examination of the integration of Artificial Intelligence (AI) into the business sector to foster creativity. The book explores the interplay between micro-level individual creativity and macro-level …


Artificial intelligence literacy for the language industry–with particular emphasis on recent large language models such as GPT-4

This article explores the concept of artificial intelligence (AI) literacy in the context of the language industry, placing particular emphasis on recent large language models such as GPT-4. After a brief introduction in which the relevance of AI literacy in the …


Artificial Intelligent based Models for Event Extraction using Customer Support Applications

M Mayuranathan, G Akilandasowmya, B Jayaram… - 2023 Second International …, 2023

… text corpora like Wikipedia and Common Crawl are popular options for computer vision and natural language processing jobs, respectively. Prepare for pretrained models by designing the architecture of the base network. Common choices include …


Artificial Neural Networks: A Noticeable Evolution in AI

P Sarang - Thinking Data Science: A Data Science Practitioner's …, 2023

Artificial neural networks (ANN) surely brought a noticeable evolution to the AI field. A data scientist has now a choice between the two techniques—GOFAI and ANN, while designing a machine learning model. In this chapter, I introduce you to this …


Artificially Intelligent Billing in Spine Surgery: An Analysis of a Large Language Model

B Zaidat, YS Lahoti, A Yu, KS Mohamed, SK Cho… - Global Spine Journal, 2023

Study Design Retrospective cohort study. Objectives This study assessed the effectiveness of a popular large language model, ChatGPT-4, in predicting Current Procedural Terminology (CPT) codes from surgical operative notes. By employing a …


Asking Questions: an Innovative Way to Interact with Oral History Archives

A Frémund, M Bulín, J Švec, F Polák - 2023

… As a speech recognizer, we trained the recent Wav2Vec 2.0 end-to-end model with a lower-case n-gram language model estimated from CommonCrawl data. In addition, the raw output of the speech recognizer was post-processed using …


Aspect-Category based Sentiment Analysis with Unified Sequence-To-Sequence Transfer Transformers

D Van Thin, NLT Nguyen - VNU Journal of Science: Computer Science and …, 2023

… While the BARTPho and its variants were trained on 20GB of news data, the viT5 models were trained on 71GB of Common Crawl data (a subset of CC100 corpus [19]) for Vietnamese language. Moreover, the text in the CC100 corpus is crawled in the …


Assessing LLMs for Moral Value Pluralism

N Benkler, D Mosaphir, S Friedman, A Smart… - arXiv preprint arXiv …, 2023

The fields of AI current lacks methods to quantitatively assess and potentially alter the moral values inherent in the output of large language models (LLMs). However, decades of social science research has developed and refined widely-accepted …


Assessing the Ability of ChatGPT to Screen Articles for Systematic Reviews

E Syriani, I David, G Kumar - arXiv preprint arXiv:2307.06464, 2023

… ChatGPT is a Generative Pretrained Transformer that employs an auto-regressive language model trained on large datasets with billions of tokens from CommonCrawl, Wikipedia, and other publicly available text sources. It relies on a deep neural …


Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT

B Wolleb, R Silvestri, G Vernikos, LDA Popescu-Belis - arXiv preprint arXiv …, 2023

Subword tokenization is the de facto standard for tokenization in neural language models and machine translation systems. Three advantages are frequently cited in favor of subwords: shorter encoding of frequent tokens, compositionality of subwords …


Assessing the research landscape and clinical utility of large language models: A scoping review

YJ Park, A Pillai, J Deng, E Guo, M Gupta, M Paget… - 2023

… Introduced in November 2022, ChatGPT was trained using large web corpora, including CommonCrawl, WebText, and Wikipedia, as well as internet-based book corpora spanning multiple languages [3]. GPT, along with other popular LLMs such …


Assessing the Value of ChatGPT for Clinical Decision Support Optimization

S Liu, AP Wright, BL Patterson, JP Wanderer, RW Turer… - medRxiv, 2023

Objective: To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. Methods: We supplied summaries of CDS logic to …


Assessing Translation capabilities of Large Language Models involving English and Indian Languages

V Mujadia, A Urlana, Y Bhaskar, PA Pavani, K Shravya… - arXiv preprint arXiv …, 2023

Generative Large Language Models (LLMs) have achieved remarkable advancements in various NLP tasks. In this work, our aim is to explore the multilingual capabilities of large language models by using machine translation as a …


Assessing Vulnerability from Its Description

Z Zhang, V Kumar, M Mayo, A Bifet - … , UbiSec 2022, Zhangjiajie, China, December 28 …, 2023

This paper shows an end-to-end Artificial Intelligence (AI) system to estimate the severity level and the various Common Vulnerability Scoring System (CVSS) components from natural language descriptions without reproducing the vulnerability …


Assessment on trial? ChatGPT and the new frontiers of learning and assessment in higher education

ChatGPT, a state-of-the-art chatbot built upon Open AI’s generative pre-trained transformer (GPT-3), has generated a major public interest and caused quite a stir in the higher education sector, where reactions have ranged from excitement to …


At the intersection of humanity and technology: a technofeminist intersectional critical discourse analysis of gender and race biases in the natural language processing …

P Barea, D Boeren, JF Ferreira Goncalves - AI & SOCIETY, 2023

Algorithmic biases, or algorithmic unfairness, have been a topic of public and scientific scrutiny for the past years, as increasing evidence suggests the pervasive assimilation of human cognitive biases and stereotypes in such systems. This …