The Three Terms Task-an open benchmark to compare human and artificial semantic representations

V Borghesani, J Armoza, MN Hebart, P Bellec… - Scientific Data, 2023

… A fasttext 32 model trained on Common Crawl with subword information. Finally, three additional fasttext 32,39,40 models trained on … First, fasttext models outperform glove ones, except for the fasttext model trained on Common Crawl. As a …


The Use of a Large Language Model for Cyberbullying Detection. Analytics 2023, 2, 694–707

… It was trained on 100 languages from 2.5 TB of filtered common crawl data. The “RoBERTa” part in XLM-RoBERTa originates from the fact that it uses the identical training procedures as the monolingual RoBERTa model, with the Masked Language Model …


The Utility of Large Language Models and Generative AI for Education Research

A Katz, U Shakir, B Chambers - arXiv preprint arXiv:2305.18125, 2023

… Typically pre-trained on extensive text corpora (eg, a common crawl of the web consisting of billions of tokens of text) and fine-tuned for tasks such as question-answering or text classification [1], LLMs are distinguished by their capacity to seemingly …


The Value of Web Data Scraping: An Application to TripAdvisor

G Barbera, L Araujo, S Fernandes - Big Data and Cognitive Computing, 2023

Social Media Analytics (SMA) is more and more relevant in today’s market dynamics. However, it is necessary to use it wisely, either in promoting any kind of product/brand, or interacting with customers. This requires its effective understanding and …


The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation

DN Manh, NL Hai, ATV Dau, AM Nguyen, K Nghiem… - arXiv preprint arXiv …, 2023

We present The Vault, an open-source, large-scale code-text dataset designed to enhance the training of code-focused large language models (LLMs). Existing open-source datasets for training code-based LLMs often face challenges in terms of size, quality (due …


The Web Can Be Your Oyster for Improving Language Models

J Li, T Tang, WX Zhao, J Wang, JY Nie, JR Wen - Findings of the Association for …, 2023

… Common Crawl snapshot which covers a wide range of 134M web documents and finally yields 906M passages of 100 tokens. CCNet processes Common Crawl … an analysis of undesirable content in the common crawl corpus. In Proceedings of …


The Web Can Be Your Oyster for Improving Large Language Models

J Li, T Tang, WX Zhao, J Wang, JY Nie, JR Wen - arXiv preprint arXiv:2305.10998, 2023


The Web Data Commons Schema. org Data Set Series

A Brinkmann, A Primpeli, C Bizer - Companion Proceedings of the ACM Web …, 2023

… The Web Data Commons project has been extracting schema.org data from the Common Crawl every year since 2013 and offers the extracted data for public download in the form of the schema.org data set series. The latest release in the …


Theoretical Considerations on AI-based Business Models for Lexicography

H Køhler Simonsen - Lexicographica, 2023

AI-generated text production is on the rise Zandan (2020), and AI writers seem to be playing an increasingly important role in marketing, L2 text production, lexicography and language teaching, cf. Simonsen (2020b; 2021a; 2022a; 2022b; Sharples/Pérez …


There's no Data Like Better Data: Using QE Metrics for MT Data Filtering

JT Peter, D Vilar, D Deutsch, M Finkelstein, J Juraska… - arXiv preprint arXiv …, 2023

Quality Estimation (QE), the evaluation of machine translation output without the need of explicit references, has seen big improvements in the last years with the use of neural metrics. In this paper we analyze the viability of using QE metrics for …


This Is a Local Domain: On Amassing Country-Code Top-Level Domains from Public Data

R Sommese, R van Rijswijk-Deij, M Jonker - arXiv preprint arXiv:2309.01441, 2023

… For Common Crawl we consider data for crawl snapshots dated between June 2017 and June 2023 (inclusive). There are 58 such snapshots, collectively accounting for 127 million registered domain names. The combined total number of …


Thou Shalt Not Reject: Analyzing Accept-Or-Pay Cookie Banners on the Web

Privacy regulations have led to many websites showing cookie banners to their users. Usually, cookie banners present the user with the option to “accept” or “reject” cookies. Recently, a new form of paywall-like cookie banner has taken hold on the …


TiBERT: A Non-autoregressive Pre-trained Model for Text Editing

B Wang, Z Wang, W Che, D Wu, R Zhang, B Wang… - … International Conference on …, 2023

Text editing refers to the task of creating new sentences by altering existing text through methods such as replacing, inserting, or deleting. Two commonly used techniques for text editing are Seq2Seq and sequence labeling. The Seq2Seq …


TiC-CLIP: Continual Training of CLIP Models

S Garg, M Farajtabar, H Pouransari, R Vemulapalli… - arXiv preprint arXiv …, 2023

… OpenCLIP models (eg, models trained on Datacomp and LAION-5B) have been trained on data curated from Common Crawl. Since the retrieval tasks we constructed are built on top of data curated from Common Crawl, one may argue …


TIL TA'LIMIDA RAQAMLI TEXNOLOGIY VA SUN'IY INTELLEKT VOSITALARIDAN FOYDALANISH

M Shahista - Innovations in Technology and Science Education, 2023

… Algoritmni o’qitish uchun ingliz tilidagi 570 Gb hajmdagi matndan foydalanigan – matnlar Vikipediyadan, Common Crawl loyihasidan, ikkita kitoblar datasetidan va WebText2 datasetdagi web-sahifalardan olingan. Lekin bunda rus tilidagi matnlar …


TIL TA'LIMIDA RAQAMLI TEXNOLOGIYALARDAN FOYDALANISH USUL VA VOSITALARI

R Ayupov, S Maksimkulava, G Baltabaeva - Евразийский журнал академических …, 2023


TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

R Eldan, Y Li - arXiv preprint arXiv:2305.07759, 2023

Language models (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small. Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely …


TKG: Telecom Knowledge Governance Framework for LLM Application

… 15], such as Common Crawl and Wikipedia, which offer limited exposure to domain-specific resources. In telecommunications, there exist numerous manuals, delivery documents, and fault diagnosis cases related to communication equipment …


TMD-BERT: A Transformer-Based Model for Transportation Mode Detection

I Drosouli, A Voulodimos, P Mastorocostas, G Miaoulis… - Electronics, 2023

… This feature led to the development of pre-trained systems such as BERT (Bidirectional Encoder Representations from Transformers) [23] which was trained with large language datasets, such as the Wikipedia Corpus and Common Crawl. Thus …


TMD-NER: Turkish multi-domain named entity recognition for informal texts

SF Yilmaz, FB Mutlu, I Balaban, SS Kozat - Signal, Image and Video Processing, 2023

We examine named entity recognition (NER), an essential and commonly used first step in many natural language processing tasks, including chatbots and language translation. We focus on the application of NER to texts that have a lot of noise, such …


tmn at SemEval-2023 Task 9: Multilingual Tweet Intimacy Detection using XLM-T, Google Translate, and Ensemble Learning

A Glazkova - arXiv preprint arXiv:2304.04054, 2023

The paper describes a transformer-based system designed for SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis. The purpose of the task was to predict the intimacy of tweets in a range from 1 (not intimate at all) to 5 (very intimate). The …


To token or not to token: A Comparative Study of Text Representations for Cross-Lingual Transfer

MM Rahman, FA Sakib, F Faisal, A Anastasopoulos - arXiv preprint arXiv:2310.08078, 2023

Choosing an appropriate tokenization scheme is often a bottleneck in low-resource cross-lingual transfer. To understand the downstream implications of text representation choices, we perform a comparative analysis on language models …


Token-level Identification of Multiword Expressions using Pre-trained Multilingual Language Models

R Swaminathan, P Cook - Proceedings of the 19th Workshop on Multiword …, 2023

In this paper, we consider novel cross-lingual settings for multiword expression (MWE) identification (Ramisch et al., 2020) and idiomaticity prediction (Tayyar Madabushi et al., 2022) in which systems are tested on languages that are unseen during training …


Tokenization and the Noiseless Channel

V Zouhar, C Meister, JL Gastaldi, L Du, M Sachan… - arXiv preprint arXiv …, 2023

… of 0.78 with BLEU on German → English MT (1M parallel sentences from CommonCrawl). This stands in contrast to Shannon entropy or expected … We use the English→German CommonCrawl dataset in all experiments. The specifics of the …


Tokenizer Choice For LLM Training: Negligible or Crucial?

The recent success of Large Language Models (LLMs) has been predominantly driven by curating the training dataset composition, scaling of model architectures and dataset sizes and advancements in pretraining objectives, leaving tokenizer …


Too Legal; Didn't Read (TLDR): Summarization of Court Opinions

A Ghimire, R Shrestha, J Edwards - 2023 Intermountain Engineering, Technology and …, 2023

… The majority of data in the PEGASUS project come from web common crawl, social media, and news. It however also includes the BillSum dataset [17]. BillSum (Kornilova & Eidelman, 2019) contains 23k US Congressional bills and human-written …


Topical Clustering of Unlabeled Transformer-Encoded Researcher Activity

Z Bettouche, A Fischer - Bavarian Journal of Applied Sciences, 2023

Transformer models have the ability to understand the meaning of text efficiently through the use of self-attention mechanisms. We investigate the bundled meanings in clusters of transformer-generated embeddings by evaluating the topical clustering …


Topological Interpretations of GPT-3

T Sun, B Nelson - arXiv preprint arXiv:2308.03565, 2023

This is an experiential study of investigating a consistent method for deriving the correlation between sentence vector and semantic meaning of a sentence. We first used three state-of-the-art word/sentence embedding methods including GPT-3 …


TorchicTab: Semantic Table Annotation with Wikidata and Language Models

I Dasoulas, D Yang, X Duan, A Dimou - CEUR Workshop Proceedings, 2023

… (3) The SOTAB Tables refer to the WDC Schema.org Table Annotation Benchmark3, generated by extracting Schema.org data from the Common Crawl. The data is grouped into separate tables for each class/host combination with the label spaces …


TourismNLG: A Multi-lingual Generative Benchmark for the Tourism Domain

SM Bhatt, S Agarwal, O Gurjar, M Gupta, M Shrivastava - Advances in Information …, 2023

… We use the mC4-pretrained mT5-base and CommonCrawl-pretrained mBART-large models, and perform domain adaptive pretraining to adapt them to the tourism domain. These are then further finetuned using task-specific labeled data. We …


Toward automatic generation of control structures for process flow diagrams with large language models

E Hirtreiter, L Schulze Balhorn, AM Schweidtmann - AIChE Journal, 2023

Developing Piping and Instrumentation Diagrams (P&IDs) is a crucial step during process development. We propose a data‐driven method for the prediction of control structures. Our methodology is inspired by end‐to‐end transformer‐based human …


Toward Detection of Arabic Cyberbullying on Online Social Networks using Arabic BERT Models

ME AlFarah, I Kamel, Z Al Aghbari - 2023 International Symposium on Networks …, 2023

Cyberbullying is one of the serious threats on social networks particularly toward children and teenagers. Cyberbullying can cause many harmful consequences towards victims such as anxiety, depression and even suicide. This research studies …


Toward Semi-supervised Transcription of NAKO+ ILSE: Influence of Auto-matic Speech Recognition Performance on Manual Transcription Effort

… The best performance was achieved using flat start HMMGaussian Mixture Model initialization in acoustic modeling and an RNN-LM trained on the ILSE training text data with additional, semantically similar Common Crawl text data. The mismatch …


Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review

F Philippy, S Guo, S Haddadan - arXiv preprint arXiv:2305.16768, 2023

In recent years, pre-trained Multilingual Language Models (MLLMs) have shown a strong ability to transfer knowledge across different languages. However, given that the aspiration for such an ability has not been explicitly incorporated in the design of …


Towards Auditing Large Language Models: Improving Text-based Stereotype Detection

W Zekun, S Bulathwela, AS Koshiyama - arXiv preprint arXiv:2311.14126, 2023

… However, they show fairness concerns due to their training on extensive, unfiltered datasets such as book [8] and Wikipedia corpora [9], and large internet corpora like Common Crawl [10]. This training data often exhibits systemic biases …


Towards Automatic Evaluation of NLG Tasks Using Conversational Large Language Models

M Riyadh, MO Shafiq - Artificial Intelligence Applications and Innovations: 19th …, 2023

Evaluating the quality of machine generated open-ended texts is a long-standing challenge in Natural Language Processing (NLP). Even though there have been dramatic advancements in the machine learning technologies that propelled the …


Towards Automatic Identification of Violation Symptoms of Architecture Erosion

R Li, P Liang, P Avgeriou - arXiv preprint arXiv:2306.08616, 2023

… Based on the fastText model, Meta published pre-trained word vectors with 300 dimensions for 157 languages constructed on Common Crawl and Wikipedia5. In this study, we employed the pre-trained fastText model, which can be transformed …


Towards Better Evaluation of Instruction-Following: A Case-Study in Summarization

O Skopek, R Aralikatte, S Gooding, V Carbune - arXiv preprint arXiv:2310.08394, 2023

Despite recent advances, evaluating how well large language models (LLMs) follow user instructions remains an open problem. While evaluation methods of language models have seen a rise in prompt-based approaches, limited work on the …


Towards Comparable Active Learning

T Werner, J Burchert, L Schmidt-Thieme - arXiv preprint arXiv:2311.18356, 2023

Active Learning has received significant attention in the field of machine learning for its potential in selecting the most informative samples for labeling, thereby reducing data annotation costs. However, we show that the reported lifts in recent literature …


Towards End-User Development for IoT: A Case Study on Semantic Parsing of Cooking Recipes for Programming Kitchen Devices

F Ventirozos, S Clinch, R Batista-Navarro - arXiv preprint arXiv:2309.14165, 2023

Semantic parsing of user-generated instructional text, in the way of enabling end-users to program the Internet of Things (IoT), is an underexplored area. In this study, we provide a unique annotated corpus which aims to support the transformation of …


Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learning

Z Charles, N Mitchell, K Pillutla, M Reneer, Z Garrett - arXiv preprint arXiv:2307.09619, 2023

We introduce a library, Dataset Grouper, to create large-scale group-structured (eg, federated) datasets, enabling federated learning simulation at the scale of foundation models. This library allows the creation of group-structured versions of …


Towards General Text Embeddings with Multi-stage Contrastive Learning

Z Li, X Zhang, Y Zhang, D Long, P Xie, M Zhang - arXiv preprint arXiv:2308.03281, 2023

We present GTE, a general-purpose text embedding model trained with multi-stage contrastive learning. In line with recent advancements in unifying various NLP tasks into a single format, we train a unified text embedding model by employing …


Towards Informed Pre-Training for Critical Error Detection in English-German

This paper presents two data augmentation methods for pre-training, to find critical errors in machine translations. This includes an alignment approach used in traditional machine translation and an imitation method, mimicking the structure of …


Towards Massively Multi-domain Multilingual Readability Assessment

T Naous, MJ Ryan, M Chandra, W Xu - arXiv preprint arXiv:2305.14463, 2023

We present ReadMe++, a massively multi-domain multilingual dataset for automatic readability assessment. Prior work on readability assessment has been mostly restricted to the English language and one or two text domains. Additionally, the …


Towards Multilingual Automatic Dialogue Evaluation

J Mendonça, A Lavie, I Trancoso - arXiv preprint arXiv:2308.16795, 2023

The main limiting factor in the development of robust multilingual dialogue evaluation metrics is the lack of multilingual data and the limited availability of open sourced multilingual dialogue systems. In this work, we propose a workaround for …


Towards multilingual automatic open-domain dialogue evaluation

J Mendonça, A Lavie, I Trancoso - Proceedings of the 24th Meeting of the Special …, 2023

The main limiting factor in the development of robust multilingual open-domain dialogue evaluation metrics is the lack of multilingual data and the limited availability of open-sourced multilingual dialogue systems. In this work, we propose a …


Towards Scalable Structured Data from Clinical Text

The adoption of electronic health records (EHRs) presents an incredible opportunity to improve medicine both at the point-of-care and through retrospective research. Unfortunately, many pertinent variables are trapped in unstructured clinical note text …


Towards Sentence-level Text Readability Assessment for French

D Van Ngo, Y Parmentier - Second Workshop on Text Simplification, Accessibility …, 2023

In this paper, we report on some experiments aimed at exploring the relation between document-level and sentence-level readability assessment for French. These were run on an open-source tailored corpus, which was automatically created …


Towards Understanding the Generalization of Medical Text-to-SQL Models and Datasets

R Tarbell, KKR Choo, G Dietrich, A Rios - arXiv preprint arXiv:2303.12898, 2023

… To train T5, Google used a diverse corpus of text data from sources such as Wikipedia, Common Crawl, and the web pages crawled by Google. We apply the T5 model to the medical text-to-SQL task. The input to the model includes the database …


Toxic Content Recognition in Conversational Systems

Guidelines: The goal of the thesis is to devise a method of recognition of toxic content in conversational systems. 1) Review the current methods used for recognizing toxic content, with focus on methods based on embedding clustering …


Traces of Human Attitudes in Contemporary and Historical Word Embeddings (1800-2000)

K Morehouse, V Rouduri, W Cunningham… - 2023

… When inspecting the 36 separate models (each measuring the relationship between implicit attitudes and an evaluative language representation eg, Glove:Common Crawl WEAT-long), results showed that > 95% of the posterior fell outside the ROPE …


Tracking the Newsworthiness of Public Documents

A Spangher, E Ferrara, B Welsh, N Peng, S Tumgoren… - arXiv preprint arXiv …, 2023

… 2013–2023 and via the Common Crawl4. We parse article text5 and deduplicate based on text, and ultimately are left with a set of 202,644 SFChron articles6. We also scrape the public meeting calendar on the SFBOS website7 to collect all …


Tracking User Web Browsing Behavior: Privacy Harms and Security Benefits

The presence of web tracking technology has grown to a near-ubiquitous state as web pages contain a growing number of trackers, representing progressively more and more third parties, that employ an increasingly diverse set of tracking techniques …


TrackSign-labeled web tracking dataset

I Castell-Uroz, P Barlet-Ros - Computer Networks, 2023

Recent studies [8] show that more than 95% of the websites available on the Internet contain at least one of the so-called web tracking systems. These systems are specialized in identifying their users by means of a plethora of different methods …


Training CLIP models on Data from Scientific Papers

C Metzger - arXiv preprint arXiv:2311.04711, 2023

Contrastive Language-Image Pretraining (CLIP) models are able to capture the semantic relationship of images and texts and have enabled a wide range of applications, from image retrieval to classification. These models are trained with …


Transfer Learning across Several Centuries: Machine and Historian Integrated Method to Decipher Royal Secretary's Diary

SL Kim, T Jang, J Ahn, H Lee, J Lee - arXiv preprint arXiv:2306.14592, 2023

A named entity recognition and classification plays the first and foremost important role in capturing semantics in data and anchoring in translation as well as downstream study for history. However, NER in historical text has faced challenges …


Transfer Learning for Code-Mixed Data: Do Pretraining Languages Matter?

K Tatariya, H Lent, M De Lhoneux - Proceedings of the 13th Workshop on …, 2023

… 2020) is the multilingual version of RoBERTa, pretrained on 100 languages from the CommonCrawl corpus. The Hindi included in the pretraining is … , with data sourced from the BBC news and the Common Crawl Corpus. It is trained with the …


Transfer Learning of Transformer-based Speech Recognition Models from Czech to Slovak

J Lehečka, JV Psutka, J Psutka - arXiv preprint arXiv:2306.04399, 2023

… As training data, we used web pages from the Common Crawl project3. We downloaded and processed 34 crawls from August 2018 to October 2021 following the same cleaning and deduplicating rules as in the English C4 dataset [14] …


Transformer based Answer-Aware Bengali Question Generation

JF Ruma, TT Mayeesha, RM Rahman - International Journal of Cognitive Computing …, 2023

… or T5 (Raffel et al., 2020) trained on a Common-Crawl based dataset called mC4 covering 101 languages. T5 model was introduced during … MC4 dataset contains 101 language variants drawn from the Common Crawl web scrape. MT5 did not …


Transformer Based Image-Text Consistency Analysis for Infographic Articles

Y Chen, MC Chang - 2023 IEEE 6th International Conference on Multimedia …, 2023

… We start with a T5 model that is pre-trained on both the common crawl-based dataset of Colossal Clean Crawled Corpus (C4) [8] and the Multi-NLI [13] datasets. Thereafter, we fine-tuned the Transformer further by using the Stanford Natural …


Transformer Based Implementation for Automatic Book Summarization

S Porwal, L Bewoor, V Deshpande - International Journal of Intelligent Systems and …, 2022

Document Summarization is the procedure of generating a meaningful and concise summary of a given document with the inclusion of relevant and topic-important points. There are two approaches-one is picking up the most relevant statements …


Transformer Models for Machine Translation and Streaming Automatic Speech Recognition

[EN] Natural language processing (NLP) is a set of fundamental computing problems with immense applicability, as language is the natural communication vehicle for people. NLP, along with many other computer technologies, has been …


Transformer models for mining intents and predicting activities from emails in knowledge-intensive processes

F Khandaker, A Senderovich, J Zhao, E Cohen, E Yu… - Engineering Applications of …, 2024

Process mining is an interdisciplinary field that combines Artificial Intelligence and Business Process Management to extract insights from historical event data. Knowledge-intensive processes, which predominantly involve knowledge work, are …


Transformer models: an introduction and catalog

X Amatriain - arXiv preprint arXiv:2302.07730, 2023

In the past few years we have seen the meteoric appearance of dozens of models of the Transformer family, all of which have funny, but not self-explanatory, names. The goal of this paper is to offer a somewhat comprehensive but simple catalog and …


Transformers in Healthcare: A Survey

S Nerella, S Bandyopadhyay, J Zhang, M Contreras… - arXiv preprint arXiv …, 2023

With Artificial Intelligence (AI) increasingly permeating various aspects of society, including healthcare, the adoption of the Transformers neural network architecture is rapidly changing many applications. Transformer is a type of deep learning …


Translating Ancient Chinese to Modern Chinese at Scale: A Large Language Model-based Approach

J Cao, D Peng, Y Shi, Z Jiang, L Jin - These articles are licensed under a Creative …, 2023

Recently, the emergence of large language models (LLMs) has provided powerful foundation models for a wide range of natural language processing (NLP) tasks. However, the vast majority of the pre-training corpus for most existing LLMs is in …


Trigger Warning Assignment as a Multi-Label Document Classification Problem

M Wiegmann, M Wolska, C Schröder, O Borchardt…

A trigger warning is used to warn people about potentially disturbing content. We introduce trigger warning assignment as a multi-label classification task, create the Webis Trigger Warning Corpus 2022, and with it the first dataset of 1 million …


Triple-Entry Accounting as a Means of Auditing Large Language Models

K Sgantzos, MA Hemairy, P Tzavaras, S Stelios - Journal of Risk and Financial …, 2023

The usage of Large Language Models (LMMs) and their exponential progress has created a Cambrian Explosion in the development of new tools for almost every field of science and technology, but also presented significant concerns regarding the AI …


Trustworthy Digital Repository Certification: A Longitudinal Study

DR Donaldson, SV Russell - Information for a Better World: Normality, Virtuality …, 2023

Increasingly, government policies are directing federal agencies to make the results of federally funded scientific research publicly available in data repositories. Additionally, academic journal policies are progressively recommending that …


Tryage: Real-time, intelligent Routing of User Prompts to Large Language Model

SN Hari, M Thomson - arXiv preprint arXiv:2308.11601, 2023

… Language models are trained on data sampled from both generalized data sources like CommonCrawl as well as domain specific data (Pile sub-sets, financial data, clinical data), and also trained and fined tuned using different paradigms (Masked …


Tsetlin Machine for Fake News Detection: Enhancing Accuracy and Reliability

This thesis aims to improve the accuracy of fake news detection by using Tsetlin Machines (TM). TMs are well suited for noisy and complex relations within the provided data, which on initial analysis, overlaps nicely with characteristics found in …


Tsinghua Issue-Generative AI, Learning And New Literacies

SY Chen - Journal of Educational Technology Development and …, 2023

Launched in November 2022, OpenAI's ChatGPT garnered over 100 million users within two months, sparking a surge in research and concern over potential risks of extensive AI experiments. The article, originating from a conference presentation by …


TTIC's Submission to WMT-SLT 23

M Sandoval-Castañeda, Y Li, B Shi, D Brentari… - Proceedings of the Eighth …, 2023

… on the German Colossal Cleaned Common Crawl (GC4) corpus, which is a cleaned and pre-processed Germanonly corpus based on Common Crawl. We take pre-… This is a subset of Common Crawl where the primary language is German …


TUDublin at CheckThat! 2023: ChatGPT for Data Augmentation

This work describes the approach of the TUDublin team at the CheckThat! 2023 Task 2: Subjectivity in News Articles. This task is aimed to discern whether a sentence in a news article conveys the subjective perspective of its author or …


TunBERT: Pretraining BERT for Tunisian Dialect Understanding

H Haddad, AC Rouhou, A Messaoudi, A Korched… - SN Computer Science, 2023

… We describe the process of creating a training dataset from collecting a Common-Crawl-based dataset, filtering and pre-processing the … Our models results indicate that a proportionately small sized Common-Crawl-based dataset (500K sentences, 67.2MB) …


TurkishBERTweet: Fast and Reliable Large Language Model for Social Media Analysis

A Najafi, O Varol - arXiv preprint arXiv:2311.18063, 2023

Turkish is one of the most popular languages in the world. Wide us of this language on social media platforms such as Twitter, Instagram, or Tiktok and strategic position of the country in the world politics makes it appealing for the social network …


TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter

X Zhang, Y Malkov, O Florez, S Park, B McWilliams… - Proceedings of the 29th …, 2023

… It is trained on over two terabytes of CommonCrawl data. … pre-training objectives, the vast majority of the work has focused on text-only training objectives applied to general domain corpora, eg, Wikipedia and CommonCrawl. In this paper, we …


TWIGMA: A dataset of AI-Generated Images with Metadata From Twitter

Y Chen, J Zou - arXiv preprint arXiv:2306.08310, 2023

… ArtBench features 60,000 high-quality, genre-annotated images of artwork spanning ten different art genres from the 14th century to the 21st century; and LAION encompasses CLIP-similarity-filtered image-text pairs sourced from the Common Crawl. …


Twitter Accounts Suggestion: Pipeline Technique SpaCy Entity Recognition

S Algamdi, A Albanyan, SK Shah, Z Tariq - … International Conference on Big Data (Big …, 2022

Twitter Accounts Suggestion is concerned with recommending accounts for the users according to their tweet contents. Twitter contains a massive amount of data that can be useful for knowing each user’s preferences. This paper uses Named …


Two Neural Models for Multilingual Grammatical Error Detection

P Le-Hong, TMH Nguyen - Swedish Language Technology Conference and …, 2023

This paper presents two neural models for multilingual grammatical error detection and their results in the MultiGED-2023 shared task. The first model uses a simple, purely supervised character-based approach. The second model uses a large …


Typhoon: Thai Large Language Models

K Pipatanakul, P Jirabovonvisut, P Manakul… - arXiv preprint arXiv …, 2023

… of the Common Crawl data [13], making Thai the 26th rank in terms of size in Common Crawl. … the Oscar dataset extracted a single package of the Common Crawl (CC) data and MC4 only … (1) Scaling the Data: we initiate the process by …


UBERT22: Unsupervised Pre-training of BERT for Low Resource Urdu Language

B Tahir, MA Mehmood - 2022 16th International Conference on Open Source …, 2022

Natural Language Understanding (NLU) tools have enabled the development of sophisticated and powerful Natural Language Processing (NLP) models. However, this progress is limited to English and European languages and low resource …


UCAS-IIE-NLP at SemEval-2023 Task 12: Enhancing Generalization of Multilingual BERT for Low-resource Sentiment Analysis

D Hu, L Wei, Y Liu, W Zhou, S Hu - arXiv preprint arXiv:2306.01093, 2023

… It is pre-trained on filtered CommonCrawl data containing 100 languages. We use xlm-roberta-base3 to initialize XLM-R. … It is trained on an aggregation of datasets from the BBC news website and Common Crawl. We use castorini/afriberta_large3 …


UD-MULTIGENRE–a UD-Based Dataset Enriched with Instance-Level Genre Annotations

V Danilova, S Stymne - Proceedings of the 3rd Workshop on Multi-lingual …, 2023

Prior research on the impact of genre on crosslingual dependency parsing has suggested that genre is an important signal. However, these studies suffer from a scarcity of reliable data for multiple genres and languages. While Universal …


UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation

X Liang, S Song, S Niu, Z Li, F Xiong, B Tang, Z Wy… - arXiv preprint arXiv …, 2023

… For example, the Chinese data incorporated in GPT’s training from the Common Crawl corpus comprises less than 5%4. Evaluations by Type. Given the categorization of news into four types, we can … 4https://commoncrawl.github.io/cc-crawl-statistics/plots/languages.html …


UM6P at SemEval-2023 Task 3: News genre classification based on transformers, graph convolution networks and number of sentences

H Alami, A Benlahbib, A El Mahdaouy, I Berrada - Proceedings of the The 17th …, 2023

This paper presents our proposed method for english documents genre classification in the context of SemEval 2023 task 3, subtask 1. Our method use ensemble technique to combine four distinct models predictions: Longformer …


UMUTeam at SemEval-2023 Task 12: Ensemble Learning of LLMs applied to Sentiment Analysis for Low-resource African Languages

JA García-Díaz, C Caparros-Laiz, Á Almela… - Proceedings of the The 17th …, 2023

These working notes summarize the participation of the UMUTeam in the SemEval 2023 shared task: AfriSenti, focused on Sentiment Analysis in several African languages. Two subtasks are proposed, one in which each language is considered …


Uncovering and Categorizing Social Biases in Text-to-SQL

Y Liu, Y Gao, Z Su, X Chen, E Ash, JG Lou - arXiv preprint arXiv:2305.16253, 2023

Content Warning: This work contains examples that potentially implicate stereotypes, associations, and other harms that could be offensive to individuals in certain social groups.} Large pre-trained language models are acknowledged to carry social …


UNDERSTANDING AND MITIGATING THE LABEL NOISE IN PRE-TRAINING ON DOWNSTREAM TASKS

H Chen, J Wang, A Shah, R Tao, H Wei, X Xie… - arXiv preprint arXiv …, 2023

Pre-training on large-scale datasets and then fine-tuning on downstream tasks have become a standard practice in deep learning. However, pre-training data often contain label noise that may adversely affect the generalization of the model. This …


Understanding Catastrophic Forgetting in Language Models via Implicit Inference

S Kotha, JM Springer, A Raghunathan - arXiv preprint arXiv:2309.10105, 2023

Fine-tuning (via methods such as instruction-tuning or reinforcement learning from human feedback) is a crucial step in training language models to robustly carry out tasks of interest. However, we lack a systematic understanding of the effects of fine-tuning …


Understanding Finetuning for Factual Knowledge Extraction from Language Models

M Kazemi, S Mittal, D Ramachandran - arXiv preprint arXiv:2301.11293, 2023

… Recently, Language Models (LMs) pre-trained on large corpora of web documents such as CommonCrawl1 have achieved impressive results on multiple NLP tasks. In their pioneering work, Petroni et al. (2019) showed that LMs also …


Understanding Multilingual Language Models: Training, Representation and Architecture

The field of natural language processing has seen numerous advancements since its inception during the Cold War, when attempts were made to automate translating Russian text to English. The 2010s have been a particularly fertile decade, going by …


Understanding the Effectiveness of Very Large Language Models on Dialog Evaluation

J Huynh, C Jiao, P Gupta, S Mehri, P Bajaj… - arXiv preprint arXiv …, 2023

… TNLGv2 has not seen datasets explicitly categorized as having dialog, but elements of casual language may be included in the Common Crawl snapshots and other internet-based corpora. Symbols: means that the category is included and × …


Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations

V Chen, QV Liao, JW Vaughan, G Bansal - arXiv preprint arXiv:2301.07255, 2023

… The AI system used in this task was trained on the BIOS dataset [23], which contains hundreds of thousands of online biographies from the CommonCrawl corpus. While the original dataset consisted individuals with 29 professions, we …


Understanding Website Privacy Policies—A Longitudinal Analysis Using Natural Language Processing

V Belcheva, T Ermakova, B Fabian - Information, 2023

Privacy policies are the main method for informing Internet users of how their data are collected and shared. This study aims to analyze the deficiencies of privacy policies in terms of readability, vague statements, and the use of pacifying phrases …


UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning

A Masry, P Kavehzadeh, XL Do, E Hoque, S Joty - arXiv preprint arXiv:2305.14761, 2023

Charts are very popular for analyzing data, visualizing key insights and answering complex reasoning questions about data. To facilitate chart-based data analysis using natural language, several downstream tasks have been introduced recently …


UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding

H Feng, Z Wang, J Tang, J Lu, W Zhou, H Li, C Huang - arXiv preprint arXiv …, 2023

… The data were collected from the “Common Crawl” dataset, a vast web corpus containing publicly available web page. We opt for PowerPoint files based on two primary considerations. On one hand, PowerPoint presentations are characterized …


Unifying Corroborative and Contributive Attributions in Large Language Models

T Worledge, JH Shen, N Meister, C Winston, C Guestrin - arXiv preprint arXiv …, 2023

As businesses, products, and services spring up around large language models, the trustworthiness of these models hinges on the verifiability of their outputs. However, methods for explaining language model outputs largely fall across two distinct fields …


UniManc at NADI 2023 Shared Task: A Comparison of Various T5-based Models for Translating Arabic Dialectical Text to Modern Standard Arabic

A Khered, I Abdelhalim, N Abdelhalim, A Soliman… - Proceedings of ArabicNLP …, 2023

This paper presents the methods we developed for the Nuanced Arabic Dialect Identification (NADI) 2023 shared task, specifically targeting the two subtasks focussed on sentence-level machine translation (MT) of text written in any of four …


Universal Multimodal Representation for Language Understanding

Z Zhang, K Chen, R Wang, M Utiyama, E Sumita, Z Li… - IEEE Transactions on …, 2023

Abstract Representation learning is the foundation of natural language processing (NLP). This work presents new methods to employ visual information as assistant signals to general NLP tasks. For each sentence, we first retrieve a flexible number of images …


University of Pennsylvania Working Paper s in Linguistic s

As speech datasets used in sociolinguistic research increase in size, laborious and time-intensive manual orthographic transcription is a challenge, limiting the amount of (transcribed) data which can be analysed. In this paper, I discuss the use of (commercial) …


Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services

M Xu, H Du, D Niyato, J Kang, Z Xiong, S Mao, Z Han… - arXiv preprint arXiv …, 2023

Artificial Intelligence-Generated Content (AIGC) is an automated method for generating, manipulating, and modifying valuable and diverse data using AI algorithms creatively. This survey paper focuses on the deployment of AIGC …


Unlocking Model Insights: A Dataset for Automated Model Card Generation

S Singh, H Lodwal, H Malwat, R Thakur, M Singh - arXiv preprint arXiv:2309.12616, 2023

… LLaMa is trained on trillions of tokens, using publicly available datasets such as C4, CommonCrawl, Github, Wikipedia, Gutenberg and … , Wikipedia, StackExchange, scientific knowledge bases, scientific and academic Common Crawl …


Unlocking the Potential of Electronic Health Records With Danish Clinical Language Models for Text Mining

This PhD dissertation focuses on the development of language technology that can be used to extract clinical information from Danish electronic health records (EHRs). EHRs contain important health-related information that can be used to guide the …


Unmaking AI Imagemaking: A Methodological Toolkit for Critical Investigation

L Munn, L Magee, V Arora - arXiv preprint arXiv:2307.09753, 2023

… To create the dataset, developers drew on Common Crawl, an immense repository of web pages scraped from the internet over twelve years which is now petabytes in size. Developers identified all the image tags in each webpage along …


Unsupervised Anomaly Detection in Multi-Topic Short-Text Corpora

M Ait-Saada, M Nadif - Proceedings of the 17th Conference of the European …, 2023

Unsupervised anomaly detection seeks to identify deviant data samples in a dataset without using labels and constitutes a challenging task, particularly when the majority class is heterogeneous. This paper addresses this topic for textual data and …


Unsupervised ASR via Cross-Lingual Pseudo-Labeling

T Likhomanenko, L Lugosch, R Collobert - arXiv preprint arXiv:2305.13330, 2023

… For African languages, we use Kinyarwanda only as a source language (as text data are not available in the Common Crawl dataset), and Swahili and Hausa only as target languages (as they are low resource). Finally, to compare with wav2vec-U …


Unsupervised Deep Representation Learning for Low-Resourced Languages and Applications

Artificial intelligence and Natural Language Processing (NLP) are becoming integral parts of modern technologies and amenities starting from the adaptation of Amazon Alexa to automatic chatbots in different industries. Though earlier NLP algorithms …


Unsupervised Story Discovery from Continuous News Streams via Scalable Thematic Embedding

S Yoon, D Lee, Y Zhang, J Han - arXiv preprint arXiv:2304.04099, 2023

… WCEP [11] is a benchmark news data set collected from Wikipedia Current Event Portal and the Common Crawl archive. We used articles in the stories of at least 50 articles and published in 2018 and 2019 (ie, WCEP18 and WCEP19). For a …


Unsupervised Ultra-Fine Entity Typing with Distributionally Induced Word Senses

The lack of annotated data is one of the challenging issues in an ultra-fine entity typing, which is the task to assign semantic types for a given entity mention. Hence, automatic type generation is receiving increased interest, typically to be used as …


Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networks

S Bhattacharya, O Bojar - arXiv preprint arXiv:2310.15552, 2023

… The model description of the XGLM model states that the model was trained on CommonCrawl data of various languages. CzEng heavily relies on various freely accessible web sources and a part of the data included in CzEng is also drawn from …


Unveiling the Black Box: Investigating the Interplay between AI Technologies, Explainability, and Legal Implications

C Erdoğanyılmaz, B Mengünoğul, M Balci - 2023 8th International Conference on …, 2023

Discovering patterns is the process of understanding, while explainability refers to the ability to represent the discovered patterns in a way that the target audience can comprehend. If we adapt the relevant proposition to explainability in law, jurists …


Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City Environment

F Xu, J Zhang, C Gao, J Feng, Y Li - arXiv preprint arXiv:2312.11813, 2023

… As claimed in open source models [159, 139, 8], most of the training corpus are from the public web text like common crawl project and only limited common data processing methods are applied due to the high-cost of handling the large volume …


Urdu paraphrase detection: A novel DNN-based implementation using a semi-automatically generated corpus

HR Iqbal, R Maqsood, AA Raza, SU Hassan - Natural Language Engineering, 2023

Automatic paraphrase detection is the task of measuring the semantic overlap between two given texts. A major hurdle in the development and evaluation of paraphrase detection approaches, particularly for South Asian languages like Urdu …


Use of Large Language Model for Cyberbullying Detection

… It was trained on 100 languages from 2.5TB of filtered common crawl data. The "RoBERTa" part in XLM-RoBERTa originates from the fact that it uses the identical training procedures as the monolingual RoBERTa model, with the Masked Language Model …


USE OF SCHEMA. ORG MICRO-MARKUP IN E-COMMERCE PROJECTS

O Belz - Three Seas Economic Journal, 2022

… were removed (Common Crawl, 2022). Both scientists and practitioners are working with the obtained data sets of the Common Crawl project. … SOTAB is based on data from schema.org and Common Crawl. SOTAB has two annotation …


User-defined Content Detection Framework

This paper presents UCDF, a simple interactive framework for training a classifier that can detect user-defined content. User-defined content signifies texts that a user wants to detect. UCDF receives examples of user-defined content as queries to …


Using AI-based detectors to control AI-assisted plagiarism in ESL writing:“The Terminator Versus the Machines”

K Ibrahim - Language Testing in Asia, 2023

The release of ChatGPT marked the beginning of a new era of AI-assisted plagiarism that disrupts traditional assessment practices in ESL composition. In the face of this challenge, educators are left with little guidance in controlling AI-assisted …


Using AI-generated suggestions from ChatGPT to optimize clinical decision support

S Liu, AP Wright, BL Patterson, JP Wanderer, RW Turer… - Journal of the American …, 2023

Objective To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. Methods We supplied summaries of CDS logic to …


Using artificial intelligence in day-to-day practice

… Most of the data used for ChatGPT’s dataset come from the Common Crawl, a collection of over 12 years of raw webpage data, metadata, and text extracts from websites starting in 2008. It does not have access to medical journal articles or raw …


Using Automated Procedures to Score Written Essays in Persian: An Application of the Multilingual BERT System

The automated scoring of student essays is now recognized as a significant development in both the research and practice of educational testing. The majority of the published studies on automated essay scoring (AES) focus on outcomes in …


USING CHATGPT TO INVESTIGATE TRENDS IN DIGITAL STORYTELLING AND KNOWLEDGE SHARING IN HIGHER EDUCATION

D Cranfield, I Venter, A Daniels - EDULEARN23 Proceedings, 2023

Literature reviews are traditionally used by researchers to consider large bodies of knowledge within their field of research and to synthesize new knowledge. The principal objective of this research was to do a state-of-the-art review using ChatGPT …


Using Domain-Specific Word Embeddings to Examine the Demand for Skills

S Chaturvedi, K Mahajan, Z Siddique - 2023

… We examine and compare domain-specific word embeddings with word embeddings trained on Common Crawl on … Common Crawl on the web and Wikipedia demonstrates the benefits of using the former.As opposed to pre-trained …


Using Language Models to Detect Alarming Student Responses

CM Ormerod, M Patel, H Wang - arXiv preprint arXiv:2305.07709, 2023

This article details the advances made to a system that uses artificial intelligence to identify alarming student responses. This system is built into our assessment platform to assess whether a student's response indicates they are a threat to …


Using Large Pre-Trained Language Model to Assist FDA in Premarket Medical Device Classification

… This paper selects the FastText embeddings with subword information pre-trained on Common Crawl which has a dimension of 300. When constructing the embeddings for the device descriptions, pre-trained embeddings are found for each …


Using machine learning to extract information and predict outcomes from reports of randomised trials of smoking cessation interventions in the Human Behaviour …

R West, F Bonin, J Thomas, AJ Wright… - 2023

… The vectors are generated based on the cooccurrence statistics of words in a large corpus of text, such as Wikipedia or Common Crawl. The algorithm is trained to learn a weighted average of the co-occurrence probabilities between words …


Using the web to explain claims

An important part of human interaction is argumentation. Arguments can be found anywhere and have been studied in various disciplines, going all the way back to ancient times. With the rise of the world wide web, and more specifically the social …


Using Transformers to Extract Date Expressions From Receipts

The extraction of date expressions in documents is a time-consuming task if done manually. For computers, automating this process can be achieved through various methods. In this thesis, I compare a regular expression system that extracts date …


Using Vector Embeddings and Feature Vectors to Humor Identification

MC Aguirre-Delgado, AE Cadena-Bautista - 2023

Expressing prejudice is the most common strategy to harm minority groups. Prejudice is defined as" the formation of a negative concept or judgment in advance about members of a race, religion, or any other significant social group, despite facts …


Using word embeddings to investigate human psychology: Methods and applications

HWS BAO, ZX WANG, X CHENG, Z SU, Y YANG… - Advances in Psychological …

… 词向量通常是根据大 规模文本语料训练的(比如 Common Crawl 语料库 覆盖了多种 来源,万亿级规模的网页链接), 分析 结果更能代表人群总体.而传统方法中, 样本量 一般 比较有限, 且以学生样本居多, 只有经过严 格,系统的抽样才能保证样本代表性. 第三 …


UT5: Pretraining Non autoregressive T5 with unrolled denoising

MG Salem, J Ye, CC Lin, F Liu - arXiv preprint arXiv:2311.08552, 2023

… For our pretraining experiments, we use the C4 dataset, which is a large-scale web document corpus created by scraping the Common Crawl data. The C4 dataset contains over 750GB of text data and includes a diverse range of topics, such as …


Vacaspati: A Diverse Corpus of Bangla Literature

P Bhattacharyya, J Mondal, S Maji, A Bhattacharya - arXiv preprint arXiv:2307.05083, 2023

Bangla (or Bengali) is the fifth most spoken language globally; yet, the state-of-the-art NLP in Bangla is lagging for even simple tasks such as lemmatization, POS tagging, etc. This is partly due to lack of a varied quality corpus. To alleviate this need, we …


Valence without meaning: investigating form and semantic components in pseudowords valence

D Gatti, L Raveling, A Petrenco, F Günther - 2023

Valence is a dominant semantic dimension, and it is fundamentally linked to basic approach-avoidance behavior within a broad range of contexts. Previous studies have shown that it is possible to approximate the valence of existing words based on …


Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing

Existing regression models tend to fall short in both accuracy and uncertainty estimation when the label distribution is imbalanced. In this paper, we propose a probabilistic deep learning model, dubbed variational imbalanced regression (VIR) …


Vashantor: A Large-scale Multilingual Benchmark Dataset for Automated Translation of Bangla Regional Dialects to Bangla Language

FTJ Faria, MB Moin, AA Wase, M Ahmmed, MR Sani… - arXiv preprint arXiv …, 2023

The Bangla linguistic variety is a fascinating mix of regional dialects that adds to the cultural diversity of the Bangla-speaking community. Despite extensive study into translating Bangla to English, English to Bangla, and Banglish to Bangla in the past …


Vax-Culture: A Dataset for Studying Vaccine Discourse on Twitter

MR Zarei, M Christensen, S Everts, M Komeili - arXiv preprint arXiv:2304.06858, 2023

Vaccine hesitancy continues to be a main challenge for public health officials during the COVID-19 pandemic. As this hesitancy undermines vaccine campaigns, many researchers have sought to identify its root causes, finding that the increasing …


VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning

ZR Zhang, C Tan, S Huang, F Huang - arXiv preprint arXiv:2304.08205, 2023

Recent studies have demonstrated the potential of cross-lingual transferability by training a unified Transformer encoder for multiple languages. In addition to involving the masked language model objective, existing cross-lingual pre-training …


Vega-mt: The jd explore academy machine translation system for wmt22

C Zan, K Peng, L Ding, B Qiu, B Liu, S He, Q Lu… - Proceedings of the Seventh …, 2022

We describe the JD Explore Academy’s submission of the WMT 2022 shared general translation task. We participated in all high-resource tracks and one medium-resource track, including Chinese-English, German-English, Czech-English, Russian-English …


Verbalising Query Results to Text

… C4 is a colossal, cleaned version of Common Crawl’s web crawl corpus. It was based on Common Crawl dataset which consists of all the websites on the internet. In our case, C4 has a regulatory effect on our pretraining process and we use a …


ViCGCN: Graph Convolutional Network with Contextualized Language Models for Social Media Mining in Vietnamese

CT Phan, QN Nguyen, CT Dang, TH Do, K Van Nguyen - arXiv preprint arXiv …, 2023

… XLM-R particularly impressive is the extensive and careful curation of over 2.5TB of data from CommonCrawl. Among its notable contributions are the improvements made for low-resource languages through specialized training and vocabulary …


Video question answering supported by a multi-task learning objective

A Falcon, G Serra, O Lanz - Multimedia Tools and Applications, 2023

… By using GloVe, pretrained on the Common Crawl dataset , a vector of size E = 300 is computed for each word in both question and answer. Since GloVe is not contextual, question and answer can be given in input to it either jointly or separately …


VideoCon: Robust Video-Language Alignment via Contrast Captions

H Bansal, Y Bitton, I Szpektor, KW Chang, A Grover - arXiv preprint arXiv:2311.10111, 2023

Despite being (pre)trained on a massive amount of data, state-of-the-art video-language alignment models are not robust to semantically-plausible contrastive changes in the video captions. Our work addresses this by identifying a broad spectrum of …


Vision-Aware Language Reasoning for Referring Image Segmentation

F Xu, B Luo, C Zhang, L Xu, M Pu, B Li - Neural Processing Letters, 2023

… Like [20, 28], we use the Glove word embedding [48] pretrained on Common Crawl 840B tokens as the language initial encoding. We default the length of expressions to 20 on all datasets and perform language feature extraction …


Vision-Language Models for Vision Tasks: A Survey

J Zhang, J Huang, S Jin, S Lu - arXiv preprint arXiv:2304.00685, 2023

Most visual recognition studies rely heavily on crowd-labelled data in deep neural networks (DNNs) training, and they usually train a DNN for each single visual recognition task, leading to a laborious and time-consuming visual recognition …


ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing

QN Nguyen, TC Phan, DV Nguyen, K Van Nguyen - arXiv preprint arXiv:2310.11166, 2023

English and Chinese, known as resource-rich languages, have witnessed the strong development of transformer-based language models for natural language processing tasks. Although Vietnam has approximately 100M people speaking …


Visual experience modulates the sensitivity to the distributional history of words in natural language

G Anceresi, D Gatti, M Marelli, T Vecchi, L Rinaldi - 2023

Different experiential traces (ie, linguistic, motor and perceptual) are likely contributing to the organization of human semantic knowledge. Here, we aimed to address this issue by investigating whether visual experience may affect the …


Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literature

ACAM de Faria, FC Bastos, JVNA da Silva, VL Fabris… - arXiv preprint arXiv …, 2023

CFG dos Sants, F de Castro Bastos, ACAM de Faria… - 2023

… More technically, this new architecture has a language model based on Text-toText transformer and uses the base of T5 [78] because of its extensive pre-training data using Common Crawl, that is 750GB of cleaned English text data. To complement, a …


Visual-Semantic Learning

… of 15 words, while the questions with length smaller than 15 were padded with zeros to the length of 15 (10 for the MSVD-QA dataset), and each word in the questions was represented as a 300D vectors using the GloVe word embedding [214] …


Visualisation and Classification of Phishing URL using Ensemble Learning Algorithms and Hyper-Parameter Tuning

G Agarwal, C Goel, K Jindal, T Subbulakshmi - 2023 Third International Conference …, 2023

… Alexa and Common Crawl were used to gather legitimate URLs. Lexical features, host-based features, and correlated feature groups are the three categories used to classify the features. Lexical features are textual aspects of the URL rather than the …


Vocabulary-free Image Classification

A Conti, E Fini, M Mancini, P Rota, Y Wang, E Ricci - arXiv preprint arXiv:2306.00917, 2023

Recent advances in large vision-language models have revolutionized the image classification paradigm. Despite showing impressive zero-shot capabilities, a pre-defined set of categories, aka the vocabulary, is assumed at test time for composing the …


WADER at SemEval-2023 Task 9: A Weak-labelling framework for Data augmentation in tExt Regression Tasks

M Suri, A Garg, D Chaudhary, I Gorton, B Kumar - arXiv preprint arXiv:2303.02758, 2023

… This model is pre-trained on a massive dataset of 2.5 terabytes of CommonCrawl data filtered for 100 different languages. By training on such a large and diverse dataset, XLM-RoBERTa is able to capture the linguistic nuances and patterns that …


WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models

C He, Z Jin, C Xu, J Qiu, B Wang, W Li, H Yan… - arXiv preprint arXiv …, 2023

The rise in popularity of ChatGPT and GPT-4 has significantly accelerated the development of large models, leading to the creation of numerous impressive large language models(LLMs) and multimodal large language models (MLLMs). These …


WDC Products: A Multi-Dimensional Entity Matching Benchmark

R Peeters, RC Der, C Bizer - arXiv preprint arXiv:2301.09521, 2023

… The first step of the pipeline is the extraction of large amounts of product offers from the Common Crawl4 using schema.org annotations. Some product offers contain product identifiers like MPNs and GTINs which allow us to group offers into …


Weak supervision and label noise handling for Natural language processing in low-resource scenarios

The lack of large amounts of labeled data is a significant factor blocking many low-resource languages and domains from catching up with recent advancements in natural language processing. To reduce this dependency on labeled instances, weak …


Web content classification analysis

This dissertation was written as a part of the MSc in Data Science at the International Hellenic University. Web analytics is a way for companies to learn more about the people who visit their websites. This information may include things like how people …


Web Page Prediction Model using Machine Learning Approaches: A Review

PA Omosebi, AP Adewole, O Sennaike - … on Science, Engineering and Business for …, 2023

Webpage classification and prefetching are essential techniques to reduce internet access latency. However, insufficient server resources and network bandwidth can lead to inefficient usage when the majority of the prefetched web pages are not …


Web Readability Challenges

E Elahi, AMI Maqueda, JLM Lara - Proceedings of the Computational Methods in …, 2023

… The datasets called Common Crawl comprised more than sixty million domain names, seven million Word documents, and more than ninety million PDF documents. Around sixty percent of the information is created from the organization, commercial …


Web Structure Mining and Social Network Analysis

… Covers 3.5 billion web pages and 128 billion hyperlinks, extracted from Common Crawl 2012 …


WebCPM: Interactive Web Search for Chinese Long-form Question Answering

Y Qin, Z Cai, D Jin, L Yan, S Liang, K Zhu, Y Lin, X Han… - arXiv preprint arXiv …, 2023

… 2021) covers 101 languages collected from the public Common Crawl web scrape. mT5 achieves superior performance in various … Mengzi-T5 was trained on Chinese Wikipedia, Chinese News, and Common Crawl and the total size of the pre-training …


WEBIE: Faithful and Robust Information Extraction on the Web

C Whitehouse, C Vania, AF Aji, C Christodoulopoulos…

… We present WEBIE, the first large-scale, entity-linked closed IE dataset consisting of 1.6M sentences automatically collected from the English Common Crawl corpus. WEBIE also includes negative examples, ie sentences without fact triples, to better …


wGMU: A Novel Fusion Strategy to Identify the Important Parts of Sentence for Relation Classification

D Escobar-Grisales, SA Moreno-Acevedo… - Workshop on Engineering …, 2023

… This model was pre-trained with 2.5 TB of text data from the Common Crawl corpus [17], which includes text in 100 different languages. This extensive pre-training has allowed RoBERTa to compute text representations in multiple languages …


What can we learn from Data Leakage and Unlearning for Law?

J Borkar - arXiv preprint arXiv:2307.10476, 2023

… the model with the start-of-the-sequence token (2) Prompting the model with random ten tokens from the Common Crawl3 for each … 5since we use tokens from common crawl for prompts we assume that the attacker has access to a dataset with …


What ChatGPT Tells Us about Gender: A Cautionary Tale about Performativity and Gender Biases in AI

Large language models and generative AI, such as ChatGPT, have gained influence over people’s personal lives and work since their launch, and are expected to scale even further. While the promises of generative artificial intelligence are compelling …


What comes next?: Investigating the neural correlates of predictability during conversation with fMRI

The notion that prediction plays a role in language processing is getting less controversial, however research and discussion is ongoing as to the nature and extent of its involvement. Previous studies have mainly focused on prediction during …


What do text-to-image models know about the languages of the world?

V Firsanova - Записки научных семинаров ПОМИ, 2023

… Another way is to use statistics of a large multilingual dataset such as Common Crawl [6]. We could try using statistics of the most spoken languages in the world [13]. But it would not be representative for our purposes because widespread spoken …


What is (and was) a person? Evidence on historical mind perceptions from natural language

E Ash, D Stammbach, K Tobia - Cognition, 2023

… Embeddings trained on a broader, less-well-curated corpus (Common Crawl) get worse performance in the human validation. Performance with our newly trained embeddings using recent decades of the Corpus of Historical American English (Davies …


What is a Fair Diffusion Model? Designing Generative Text-To-Image Models to Incorporate Various Worldviews

Z De Simone, A Boggust, A Satyanarayan, A Wilson - arXiv preprint arXiv:2309.09944, 2023

… For example, Stable Diffusion [37] is trained on the image-text pairs from LAION-5B [41], a dataset synthesized from a repository of webcrawled data (Common Crawl [17]) and filtered using a CLIP [35] image classification model. This data represents a …


What Makes ImageNet Look Unlike LAION

A Shirali, M Hardt - arXiv preprint arXiv:2306.15769, 2023

ImageNet was famously created from Flickr image search results. What if we recreated ImageNet instead by searching the massive LAION dataset based on image captions alone? In this work, we carry out this counterfactual investigation …


What Makes LLMs Large?

T Amaratunga - … Large Language Models: Learning Their Underlying …, 2023

By now you should have a high-level understanding of the concepts of natural language processing and how the transformer architecture and attention mechanisms revolutionized the NLP field and how it changed the way we look at …


WHAT MAKES PRE-TRAINED VISUAL REPRESENTATIONS SUCCESSFUL FOR ROBUST MANIPULATION?

K Burns, Z Witzel, JI Hamid, T Yu, C Finn, K Hausman

Inspired by the success of transfer learning in computer vision, roboticists have investigated visual pre-training as a means to improve the learning efficiency and generalization ability of policies learned from pixels. To that end, past work has …


What Works When in Context-aware Neural Machine Translation?

H Gete, T Etchegoyhen, G Labaka - Proceedings of the 24th Annual Conference of …, 2023

Document-level Machine Translation has emerged as a promising means to enhance automated translation quality, but it is currently unclear how effectively context-aware models use the available context during translation. This paper aims …


What's in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models through User-Provided Names in CAD Files

P Meltzer, JG Lambourne, D Grandi - arXiv preprint arXiv:2304.14275, 2023

Semantic knowledge of part-part and part-whole relationships in assemblies is useful for a variety of tasks from searching design repositories to the construction of engineering knowledge bases. In this work we propose that the natural language …


What's in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models Through User-Provided Names in Computer Aided Design Files

P Meltzer, JG Lambourne, D Grandi - Journal of Computing and Information Science …, 2024

Semantic knowledge of part-part and part-whole relationships in assemblies is useful for a variety of tasks from searching design repositories to the construction of engineering knowledge bases. In this work, we propose that the natural language …


What's In My Big Data?

Y Elazar, A Bhagia, I Magnusson, A Ravichander… - arXiv preprint arXiv …, 2023

… The dataset: The Colossal Clean Crawled Corpus (C4 in short) is based on Common Crawl as a source of text that was scraped from the … We use the latest version (v.3.1.0) which was used to train umT5, containing documents collected from …


What's in the News? Towards Identification of Bias by Commission, Omission, and Source Selection (COSS)

A Zhukova, TL Ruas, F Hamborg, K Donnay, B Gipp - 2023

In a world overwhelmed with news, determining which information comes from reliable sources or how neutral is the reported information in the news articles poses a challenge to news readers. In this paper, we propose a methodology for …


When AI Moves Downstream

F Grodzinsky, K Miller, M Wolf - International Conference on Computer Ethics, 2023

… The training set for GPT-3 filtered a version of Common Crawl used to develop GPT-2. These were deliberate developer choices. In … Developers should be aware of the harm that uncurated and unanalyzed large data sets gathered from web …


When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTs

M Bevilacqua, K Oketch, R Qin, W Stamey, X Zhang… - arXiv preprint arXiv …, 2023

… RoBERTa also uses BookCorpus and Wikipedia, but in addition to common crawl news (CC-News), open web text, and stories, resulting in over 30 million tokens used for pre-training [50]. Both BERT and RoBERTa have performed well on text …


When Giant Language Brains Just Aren't Enough! Domain Pizzazz with Knowledge Sparkle Dust

MT Nguyen, DH Nguyen, S Sabahi, H Le, J Yang… - arXiv preprint arXiv …, 2023

Large language models (LLMs) have significantly advanced the field of natural language processing, with GPT models at the forefront. While their remarkable performance spans a range of tasks, adapting LLMs for real-world business …


When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities

J Chen, Z Liu, X Huang, C Wu, Q Liu, G Jiang, Y Pu… - arXiv preprint arXiv …, 2023

… For another thing, the pre-training is conducted based on much more unsupervised corpora, with hundreds of billions or trillions of tokens carefully filtered from sources like Common Crawl, GitHub, Wikipedia, Books, ArXiv, etc. The impact …


When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale

M Marion, A Üstün, L Pozzobon, A Wang, M Fadaee… - arXiv preprint arXiv …, 2023

… test set from the same CommonCrawl snapshot with identical … CommonCrawl A reference model trained on Wikipedia (an example of a clean noise-free corpus) achieves consistently lower validation perplexity compared to a reference model …


When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming

H Mozannar, G Bansal, A Fourney, E Horvitz - arXiv preprint arXiv:2306.04930, 2023

AI powered code-recommendation systems, such as Copilot and CodeWhisperer, provide code suggestions inside a programmer's environment (eg, an IDE) with the aim to improve their productivity. Since, in these scenarios, programmers accept and …


Where Did the News Come From? Detection of News Agency Releases in Historical Newspapers

Since their beginnings in the 1830s and 1840s, news agencies have played an important role in the national and international news market, aiming to deliver news as fast and as reliable as possible. While we know that newspapers have been …


Where's the Liability in Harmful AI Speech?

P Henderson, T Hashimoto, M Lemley - arXiv preprint arXiv:2308.04635, 2023

… The underlying datasets can range from crawls of the web (eg, CommonCrawl) to court opinions from CourtListener and even to books downloaded from BitTorrent trackers.This input data may be lightly filtered for hate speech and private information …


Who can verify this? Finding authorities for rumor verification in Twitter

F Haouari, T Elsayed, W Mansour - Information Processing & Management, 2023

A large body of research work has proposed verification techniques for rumors spreading in social media that mainly relied on subjective evidence, eg, propagation networks or user interactions. Alternatively, in this work, we introduce the task of …


Who Killed the Winograd Schema Challenge?

H Neri, FG Cozman - Brazilian Conference on Intelligent Systems, 2023

In which we investigate the technical issues surrounding the defeat, or perhaps the sudden assassination, of the Winograd Schema Challenge. We argue that, while the obvious suspect is the WinoGrande-based solution, the real cause of death was the …


Why do Words with Negative Connotations Still Exist? A Corpus-Based Analysis of the Words 'Handicapped','Diffable', and 'Disability'

This corpus-based study examines the persistence of negatively connoted words in the Indonesian, particularly focusing on cacat (handicapped). Cacat is compared to its synonyms, namely difabel (difable) and disabilitas (disability). The study employs …


Why You Should Give Your Students Automatic Process Feedback on Their Collaboration: Evidence from a Randomized Experiment

L Menzel, S Gombert, J Weidlich, A Fink, A Frey… - European Conference on …, 2023

In Computer-Supported Collaborative Learning (CSCL), students learn in small groups to achieve learning benefits outside what would be possible for individual students. As in other forms of learning, students need feedback on the quality of their …


WiCE: Real-World Entailment for Claims in Wikipedia

R Kamoi, T Goyal, JD Rodriguez, G Durrett - arXiv preprint arXiv:2303.01432, 2023

… For each claim c, we re-retrieve the cited web articles from Common Crawl.We automatically parse the cited articles’ (note that there might be more than one cited article) HTML to extract the article text. This process is often quite noisy and may …


Witscript 3: A Hybrid AI System for Improvising Jokes in a Conversation

J Toplyn - arXiv preprint arXiv:2301.02695, 2023

Previous papers presented Witscript and Witscript 2, AI systems for improvising jokes in a conversation. Witscript generates jokes that rely on wordplay, whereas the jokes generated by Witscript 2 rely on common sense. This paper extends that …


WizardLM: Empowering Large Language Models to Follow Complex Instructions

C Xu, Q Sun, K Zheng, X Geng, P Zhao, J Feng, C Tao… - arXiv preprint arXiv …, 2023

Training large language models (LLM) with open-domain instruction following data brings colossal success. However, manually creating such instruction data is very time-consuming and labor-intensive. Moreover, humans may struggle to produce …


Word Embedding for Bengali Language using Domain-related Corpus

A Bandyopadhyay, J Nair - 2023 International Conference on Inventive …, 2023

… Of late, word embeddings from general corpus such as Wikipedia dump, and common crawl corpus are very famous to make word vectors. In the case of Indian languages also such type of corpus is used to make word vectors. But it suffers in …


Word length and frequency effects on text reading are highly similar in 12 alphabetic languages

V Kuperman, S Schroeder, D Gnetov - 2023

One of the most robust findings in research on eye-movement control in reading is that shorter and more frequent words are recognized faster and skipped more often than longer and less frequent words. These benchmark effects of word length and …


WordScape: a Pipeline to extract multilingual, visually rich Documents with Layout Annotations from Web Crawl Data

M Weber, C Siebenschuh, RM Butler, A Alexandrov… - Thirty-seventh Conference …, 2023

… Similar to large-scale NLP dataset creation pipelines like CCNet [33], we use the Common Crawl web corpus 2 as our primary source of documents. We parse Common Crawl to collect urls pointing to MS Word documents embedded in …


Working with Corpora in Translation Technology Teaching: Enhancing Aspects of Course Design

M Shuttleworth - Corpora and Translation Education: Advances and …, 2023

This chapter offers a discussion of two linked areas that are relevant to the translation technology curriculum. Firstly, in the absence of a suitable cost-effective industry-standard terminology extraction utility—as well as the relative under-representation …


X-ABI: Toward Parameter-Efficient Multilingual Adapter-Based Inference for Cross-Lingual Transfer

AS Menon, K Anand - … Management, Analytics and Innovation: Proceedings of …, 2023

… XLM-RoBERTa is a multilingual self-supervised transformers model that is pre-trained on 2.5 TB of filtered CommonCrawl data containing 100 … Table 6 is a comparison between XNLI accuracy and number of tokens present in the 2.5 TB of filtered …


xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark

C Zhang, LF D'Haro, C Tang, K Shi, G Tang, H Li

Recent advancements in reference-free learned metrics for open-domain dialogue evaluation have been driven by the progress in pre-trained language models and the availability of dialogue data with high-quality human annotations. However …


XFEVER: Exploring Fact Verification across Languages

YC Chang, C Kruengkrai, J Yamagishi - arXiv preprint arXiv:2310.16278, 2023

This paper introduces the Cross-lingual Fact Extraction and VERification (XFEVER) dataset designed for benchmarking the fact verification models across different languages. We constructed it by translating the claim and evidence texts of the Fact …


XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models

D Liang, H Gonen, Y Mao, R Hou, N Goyal… - arXiv preprint arXiv …, 2023

Large multilingual language models typically rely on a single vocabulary shared across 100+ languages. As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged. This vocabulary bottleneck …


XNLI 2.0: Improving XNLI dataset and performance on Cross Lingual Understanding (XLU)

AK Upadhyay, HK Upadhya - arXiv preprint arXiv:2301.06527, 2023

Natural Language Processing systems are heavily dependent on the availability of annotated data to train practical models. Primarily, models are trained on English datasets. In recent times, significant advances have been made in multilingual …


xSIM++: An Improved Proxy to Bitext Mining Performance for Low-Resource Languages

M Chen, K Heffernan, O Çelebi, A Mourachko… - arXiv preprint arXiv …, 2023

We introduce a new proxy score for evaluating bitext mining based on similarity in a multilingual embedding space: xSIM++. In comparison to xSIM, this improved proxy leverages rule-based approaches to extend English sentences in any evaluation set …


XWikiGen: Cross-lingual Summarization for Encyclopedic Text Generation in Low Resource Languages

D Taunk, S Sagare, A Patil, S Subramanian, M Gupta… - arXiv preprint arXiv …, 2023

… 2020) was pretrained on CommonCrawl corpus using the BART objective where the input texts are noised by masking phrases and permuting sentences, and a single Transformer model is learned to recover the texts. Specifically, our mT5-base …


Yes but.. Can ChatGPT Identify Entities in Historical Documents?

CE González-Gallardo, E Boros, N Girdhar, A Hamdi… - arXiv preprint arXiv …, 2023

E Boros, N Girdhar, A Hamdi, JG Moreno, A Doucet

Large language models (LLMs) have been leveraged for several years now, obtaining state-of-the-art performance in recognizing entities from modern documents. For the last few months, the conversational agent ChatGPT has "prompted" …


You Call This Archaeology? Evaluating Web Archives for Reproducible Web Security Measurements

F Hantke, S Calzavara, M Wilhelm, A Rabitti, B Stock - 2023

S Calzavara, F Hantke, M Wilhelm, A Rabitti, B Stock - 2023

… Besides Memento-based archives, we also consider Common Crawl as a possible alternative source of archival data. Common Crawl … Besides Common Crawl, we consider a public list4 of web archives supporting the Memento protocol …


YUAN 2.0: A Large Language Model with Localized Filtering-based Attention

S Wu, X Zhao, S Wang, J Luo, L Li, X Chen, B Zhao… - arXiv preprint arXiv …, 2023

In this work, the Localized Filtering-based Attention (LFA) is introduced to incorporate prior knowledge of local dependencies of natural language into Attention. Based on LFA, we develop and release Yuan 2.0, a large language model with …


Zero-shot cross-lingual transfer language selection using linguistic similarity

J Eronen, M Ptaszynski, F Masui - Information Processing & Management, 2023

We study the selection of transfer languages for different Natural Language Processing tasks, specifically sentiment analysis, named entity recognition and dependency parsing. In order to select an optimal transfer language, we propose to …


Zero-shot Cross-lingual Transfer With Learned Projections Using Unlabeled Target-Language Data

U Deb, R Parab, P Jyothi - The 61st Annual Meeting Of The Association For …, 2023

Adapters have emerged as a parameter-efficient Transformer-based framework for cross-lingual transfer by inserting lightweight languagespecific modules (language adapters) and taskspecific modules (task adapters) within pretrained multilingual …


Zero-shot Cross-lingual Transfer without Parallel Corpus

Y Zhang, X Han, B Wang - arXiv preprint arXiv:2310.04726, 2023

Recently, although pre-trained language models have achieved great success on multilingual NLP (Natural Language Processing) tasks, the lack of training data on many tasks in low-resource languages still limits their performance. One effective …


Zero-Shot ECG Diagnosis with Large Language Models and Retrieval-Augmented Generation

H Yu, P Guo, A Sano - Machine Learning for Health (ML4H), 2023

Recently, Large Language Models (LLMs) have become essential players in the deep learning domain. While their capabilities are evident across various textual tasks, this study aims to bridge the gap and explore the potential of leveraging LLMs …


Zero-Shot Learning on Low-Resource Languages by Cross-Lingual Retrieval

Research on zero-shot learning on low-resource languages in NLP is motivated by inherent data scarcity of low-resource languages. On the contrary, languages with a large amount of both labeled and unlabeled resources are not fully utilized …


Zero-Shot Slot and Intent Detection in Low-Resource Languages

SY Kwon, G Bhatia, EMB Nagoudi, AA Inciarte… - arXiv preprint arXiv …, 2023

Intent detection and slot filling are critical tasks in spoken and natural language understanding for task-oriented dialog systems. In this work we describe our participation in the slot and intent detection for low-resource language varieties (SID4LR; …


Zero-Shot Speech Emotion Recognition Using Generative Learning with Reconstructed Prototypes

X Xu, J Deng, Z Zhang, Z Yang, BW Schuller - ICASSP 2023-2023 IEEE International …, 2023

… For the prototype-reconstruction step, we employ the 300-dimensional fastText model considering 2 million word vectors (600 billion tokens) trained on Common Crawl [31]. Then, a three-fold emotion-independent Cross-Validation (CV) is …


Zero-Shot Transfer Learning using Affix and Correlated Cross-Lingual Embeddings.

Learning morphologically supplemented embedding spaces using cross-lingual models has become an active area of research and facilitated many research breakthroughs in various applications such as machine translation, named entity …


Zhanglin Wu, Zhengzhe Yu, Zongyao Li, Daimeng Wei, Yuhao Xie, Xiaoyu Chen, Hengchao Shang, Jiaxin Guo, Zhiqiang Rao, Shaojun Li, Song peng, Lizhi Lei, Hao …

X Chen - Machine Translation: 19th China Conference, CCMT …, 2023

This paper presents Huawei Translation Service Center (HW-TSC)'s submission to the machine translation tasks of the 19th China Conference on Machine Translation (CCMT 2023). We participate in all machine translation tasks, including five bilingual …


Ziya2: Data-centric Learning is All LLMs Need

R Gan, Z Wu, R Sun, J Lu, X Wu, D Zhang, K Pan… - arXiv preprint arXiv …, 2023

… First, we find that Common Crawl and other open-source datasets contain substantial duplication of web pages and thus use Bloomfilter5 to de-duplicate URLs, which significantly reduces the computational load required for subsequent content …


Zoology: Measuring and Improving Recall in Efficient Language Models

S Arora, S Eyuboglu, A Timalsina, I Johnson, M Poli… - arXiv preprint arXiv …, 2023

Attention-free language models that combine gating and convolutions are growing in popularity due to their efficiency and increasingly competitive performance. To better understand these architectures, we pretrain a suite of 17 attention and "gated-convolution" …


Zootopi at HOPE2023IberLEF: Is Zero-Shot Chat-GPT the Future of Hope Speech Detection

A Ngo, HTH Tran - Proceedings of the Iberian Languages Evaluation …, 2023

Hope Speech Detection is a Natural Language Processing (NLP) task where we aim to detect any message or text that can relax a hostile environment and inspire people in a time of suffering illness, stress, loneliness or depression with optimism …