Question and Answer Generation from Text Using Transformers

C Srihari, S Sunagar, RK Kamat, KS Raghavendra… - International Symposium on …, 2023

Natural language generation is one of the major tasks that comes under Natural Language Processing and, Question Generation (QG) and Question Answering (QA) are two NLP-based tasks that have found widespread application. The QA and QG …


Question Answer Generation in Bengali: Mitigating the scarcity of QA datasets in a low-resource language

… For fastText word embeddings, we use the pretrained word vector model for Bengali, trained on Common Crawl7 and Wikipedia. For back-translation, we use the "csebuetnlp/banglat5_nmt_bn_en" model checkpoint. To train our answer span …


Question Answering Chatbots for Biomedical Research using Transformers

E Xygi, AD Andriopoulos, DA Koutsomitropoulos

Professionals as well as the general public need effective help to access, understand and consume complex biomedical concepts. The existence of an interaction environment capable of automatically processing such information-thus …


Question Answering System for Tamil Using Deep Learning

B Antony, NRR Paul - Speech and Language Technologies for Low …, 2023

Tamil, a Dravidian language family member, is widely spoken in numerous Indian states. But languages like Tamil, are underrepresented on the web. Many NLP models perform worse with these languages when compared to English, the effects …


QUESTION GENERATION AGENT

According to recent studies that highlight the impact of technology on modern students’ learning, it came to light that modern tools increase learners’ interactivity. However, there is still an absence of modern technological tools that can be helpful …


Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities

J Bai, S Bai, S Yang, S Wang, S Tan, P Wang, J Lin… - arXiv preprint arXiv …, 2023

… In order to improve the text-oriented tasks, we collect pdf and HTML format data from Common Crawl3 and generate synthetic OCR data in English and Chinese language with natural scenery background, following Kim et al. (2022). Finally, we …


Rapidgzip: Parallel Decompression and Seeking in Gzip Files Using Cache Prefetching

M Knespel, H Brunst - Proceedings of the 32nd International Symposium on …, 2023

… 吀栀e multiple petabytes large Common Crawl [8] dataset is also distributed as a set of gzip-compressed 昀椀les. A pipeline for decompressing and preprocessing such data, eg, for use in machine learning context, would bene昀椀t from being …


RAVEN: Stateless Rapid IP Address Variation for Enterprise Networks.

L Wang, H Kim, P Mittal, J Rexford - Proc. Priv. Enhancing Technol., 2023

Enterprise networks face increasing threats against the privacy of their clients. Existing enterprise services like Network Address Translation (NAT) offer limited privacy protection, at the cost of requiring per-flow state. In this paper, we introduce …


Re-examining cross-cultural similarity judgments using language statistics

KN Le, S Gao, MC Frank, A Carstensen - Proceedings of the Annual Meeting of the …, 2023

… These models were trained on Common Crawl and Wikipedia using fastText and use character n-grams of length 5, and 10 negative examples. The training used a Continuous Bag of Words with position-weights and a window of size 5. From these …


Realistic but Non-Identifiable Synthetic User Data Generation

I Noble, I Vendrov, X Xu, D Ramachandran - 2023

… Recent revolutions in Computer Vision and Natural Language Processing are often credited to the use of massive datasets such as ImageNet [1] and Common Crawl [2], which dramatically improved performance across a multitude of tasks, and …


Reclaiming Data Agency in the Age of Ubiquitous Machine Learning

… For example, text generation models like ChatGPT are trained on datasets like Common Crawl [5], which is an open repository containing … Or at least this is the operating assumption of ML developers who create large-scale internet datasets like …


Reclaiming the Digital Commons: A Public Data Trust for Training Data

A Chan, H Bradley, N Rajkumar - arXiv preprint arXiv:2303.09001, 2023

… The data trust could also start from existing efforts, such as the Common Crawl We emphasize that the process of scraping data should be a continual, iterative process given the continual growth in the amount of internet data [100]. The data …


Recommending Answers to Math Questions Using KL-Divergence and the Approximate XML Tree Matching Approach

… The dataset used by ChatGPT to train the language model is Common Crawl dataset that includes nearly a trillion of words. ChatGPT, which uses a large unsupervised corpus of tokens to train the language model, applies alternating …


Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models

T Ahmed, S Ghosh, C Bansal, T Zimmermann, X Zhang… - arXiv preprint arXiv …, 2023

Incident management for cloud services is a complex process involving several steps and has a huge impact on both service health and developer productivity. On-call engineers require significant amount of domain knowledge and manual effort for …


Red AI? Inconsistent Responses from GPT3. 5 Models on Political Issues in the US and China

D Zhou, Y Zhang - arXiv preprint arXiv:2312.09917, 2023

The rising popularity of ChatGPT and other AI-powered large language models (LLMs) has led to increasing studies highlighting their susceptibility to mistakes and biases. However, most of these studies focus on models trained on English texts. Taking an …


ReDASPersuasion at SemEval-2023 Task 3: Persuasion Detection using Multilingual Transformers and Language Agnostic Features

FZ Qachfar, R Verma - Proceedings of the The 17th International Workshop on …, 2023

This paper describes a multilingual persuasion detection system that incorporates persuasion technique attributes for a multi-label classification task. The proposed method has two advantages. First, it combines persuasion features with a sequence …


Regret and Hope on Transformers: An Analysis of Transformers on Regret and Hope Speech Detection Datasets

G Sidorov, F Balouchzahi, S Butt, A Gelbukh - Applied Sciences, 2023

… than BERT, including additional data from CommonCrawl and the WebText dataset. The training … XLNet was trained on a large corpus of text from Wikipedia and the CommonCrawl dataset. … The model was trained on a large corpus of text …


Regulating ChatGPT and other Large Generative AI Models

P Hacker, A Engel, M Mauer - arXiv preprint arXiv:2302.02337, 2023

Large generative AI models (LGAIMs), such as ChatGPT or Stable Diffusion, are rapidly transforming the way we communicate, illustrate, and create. However, AI regulation, in the EU and beyond, has primarily focused on conventional AI models …


Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API

Z Zhang, W Xie, X Zhang, Y Lu - arXiv preprint arXiv:2310.04716, 2023

… As for the experiments with desktop data, we collect about 37K UI images from Common Crawl2, an open repository of web crawl data. We follow the practices in the open repository3 of (Burns et al., 2022) to generate 0.5M image-instruction pairs …


RelBERT: Embedding Relations with Language Models

A Ushio, J Camacho-Collados, S Schockaert - arXiv preprint arXiv:2310.00299, 2023

Many applications need access to background knowledge about how different concepts and entities are related. Although Knowledge Graphs (KG) and Large Language Models (LLM) can address this need to some extent, KGs are inevitably …


ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models

I Mirzadeh, K Alizadeh, S Mehta, CC Del Mundo… - arXiv preprint arXiv …, 2023

… We chose RefinedWeb because it is a high-quality subset of Common Crawl, which is often used in the pretraining phase of LLMs, including Llama, Falcon, and OPT. We also use the validation split of WikiText [54] for measuring the sparsity and …


Remodelling internet infrastructure: A first look at platform governance in the era of ChatGPT

… nature and scope of OpenAI’s filtering of the Common Crawl set, rather mysterious. Common Crawl is a nonprofit organization that relies on a broad … Ostensibly, Common Crawl operates on a commons-based production model. At its launch in …


RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model

F Bie, Y Yang, Z Zhou, A Ghanem, M Zhang, Z Yao… - arXiv preprint arXiv …, 2023

… The image-text pairs are retrieved from random web pages with Common Crawl [224] and are filtered by CLIP score to ensure image-text alignment. Beyond the textual descriptions, LAION400M also equips users with CLIP embeddings and KNN …


Report of the 1st Workshop on Generative AI and Law

AF Cooper, K Lee, J Grimmelmann, D Ippolito… - arXiv preprint arXiv …, 2023

This report presents the takeaways of the inaugural Workshop on Generative AI and Law (GenLaw), held in July 2023. A cross-disciplinary group of practitioners and scholars from computer science and law convened to discuss the technical, doctrinal …


Report on the 16th Round of NII Testbeds and Community for Information Access Research (NTCIR-16)

T Yamamoto, Z Dou, N Kando, CLA Clarke, MP Kato… - ACM SIGIR Forum, 2023

… Firstly, Chuweb21, a subset of the Common Crawl dataset, is introduced for this task. Chuweb21 contains 3,402,457 domains and 858,616,203 English web pages. Secondly, two types of relevance assessment are introduced: the Gold version given …


Requirements Classification for Smart Allocation: A Case Study in the Railway Industry

S Bashir, M Abbas, A Ferrari, M Saadatmand…

Allocation of requirements to different teams is a typical preliminary task in large-scale system development projects. This critical activity is often performed manually and can benefit from automated requirements classification techniques. To date, limited …


Research on The Evaluation of Token Imbalance Degree in NMT Corpus

H WANG, L YU, H WANG - ACTA ELECTONICA SINICA, 2023

… 1 实验数据与参数设置 本文选择国际机器翻译大赛 WMT15 的 NewsCommentary-v10 和 Common Crawl corpus 以及 WMT21 … 其中,News-Commentary-v10 和 NewsCommentary-v16 语料库的词表规模设置为 30K,Com⁃ mon Crawl corpus语料库的词表规模设置为200K …


Research proposal content extraction using natural language processing and semi-supervised clustering: A demonstration and comparative analysis

BM Knisely, HH Pavliscsak - Scientometrics, 2023

Funding institutions often solicit text-based research proposals to evaluate potential recipients. Leveraging the information contained in these documents could help institutions understand the supply of research within their domain. In this work, an …


Resolving Elliptical Compounds in German Medical Text

N Kammer, F Borchert, S Winkler, G De Melo… - The 22nd Workshop on …, 2023

… mT5 closely follows the original T5 architecture, but was trained on a multilingual Common Crawl-based dataset called mC4 with 101 languages, including German. We performed some preliminary experiments with other models. These included a …


Responsible Task Automation: Empowering Large Language Models as Responsible Task Automators

Z Zhang, X Zhang, W Xie, Y Lu - arXiv preprint arXiv:2306.01242, 2023

The recent success of Large Language Models (LLMs) signifies an impressive stride towards artificial general intelligence. They have shown a promising prospect in automatically completing tasks upon user instructions, functioning as brain-like …


Rethinking Homework in the Age of Artificial Intelligence

H Ibrahim, R Asim, F Zaffar, T Rahwan, Y Zaki - IEEE Intelligent Systems, 2023

… ChatGPT, for instance, is trained on a curated version of the common crawl dataset, built upon text available on the Internet. However, if this content is not properly cited, it could be considered a violation of copyright laws. Indeed, this has …


Retrieving Webpages Using Online Discussions

K Ros, M Jin, J Levine, CX Zhai - Proceedings of the 2023 ACM SIGIR International …, 2023

… query search over all training, validation, and testing splits, it could be expanded by collecting more webpages (eg, from the Common Crawl corpus [6]). … http://web.archive.org/web/ 20221002030422/https://campuswire.com/chatrooms [6] Common Crawl. 2022 …


RetVec: Resilient and Efficient Text Vectorizer

E Bursztein, M Zhang, O Vallis, X Jia, A Kurakin - arXiv preprint arXiv:2302.09207, 2023

This paper describes RetVec, a resilient multilingual embedding scheme designed for neural-based text processing, including small-text classification and large-language models. RetVec combines a novel character encoding with an optional small model …


Reveal the Unknown: Out-of-Knowledge-Base Mention Discovery with Entity Linking

H Dong, J Chen, Y He, Y Liu, I Horrocks - arXiv preprint arXiv:2302.07189, 2023

Discovering entity mentions that are out of a Knowledge Base (KB) from texts plays a critical role in KB maintenance, but has not yet been fully explored. The current methods are mostly limited to the simple threshold-based approach and feature-based …


Revealing Media Bias in News Articles: NLP Techniques for Automated Frame Analysis

This open access book presents an interdisciplinary approach to reveal biases in English news articles reporting on a given political event. The approach named person-oriented framing analysis identifies the coverage’s different perspectives on …


Review on Query focused Multi-Document Summarization (QMDS) with Comparative Analysis

P Roy, S Kundu - ACM Computing Surveys, 2023

… AQUAMUSE is one of the recent question-answering datasets generated using the Google Natural Questions (NQ) dataset [98] and Common Crawl corpus [160]. The inputs are matched documents from the Common Crawl corpus, whereas the …


REVIEWTAG: TAGGING AMAZON NEGATIVE PRODUCT REVIEW WITH DEEP LEARNING

… It is trained on a massive dataset of text, Common Crawl, consisting of over 600 billion tokens from various sources, including web pages, news articles, and social media posts [31]. The model outputs 2 million word vectors, each with a …


Revisiting and Improving Retrieval-Augmented Deep Assertion Generation

W Sun, H Li, M Yan, Y Lei, H Zhang - arXiv preprint arXiv:2309.10264, 2023

Unit testing validates the correctness of the unit under test and has become an essential activity in software development process. A unit test consists of a test prefix that drives the unit under test into a particular state, and a test oracle (eg, assertion) …


Revisiting Entropy Rate Constancy in Text

V Verma, N Tomlin, D Klein - arXiv preprint arXiv:2305.12084, 2023

… 3.2 Common Crawl News We include a subset of the Common Crawl News Dataset due to its chronological diversity. In particular, we run the majority of our experiments on GPT-2; because articles in the NYT … To address this concern, we …


Revisiting non-English Text Simplification: A Unified Multilingual Benchmark

MJ Ryan, T Naous, W Xu - arXiv preprint arXiv:2305.15678, 2023

Recent advancements in high-quality, large-scale English resources have pushed the frontier of English Automatic Text Simplification (ATS) research. However, less work has been done on multilingual text simplification due to the lack of a diverse …


Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing

A Ghaddar, Y Wu, S Bagga, A Rashid, K Bibi… - Proceedings of the 2022 …, 2022

There is a growing body of work in recent years to develop pre-trained language models (PLMs) for the Arabic language. This work addresses two major problems in existing Arabic PLMs that limit the progress of the Arabic NLU and NLG fields. First …


Revolutionizing subjective assessments: A three-pronged comprehensive approach with NLP and deep learning

R Agrawal, H Mishra, I Kandasamy, SR Terni… - Expert Systems with …, 2024

… The model also uses the CCNet dataset, extracted from a common crawl with a different filtering process than for OSCAR. The model’s performance measure was impossible for NER due to the limited availability of annotated corpora. The model’s …


RIATIG: Reliable and Imperceptible Adversarial Text-to-Image Generation With Natural Prompts

H Liu, Y Wu, S Zhai, B Yuan, N Zhang - Proceedings of the IEEE/CVF Conference on …, 2023

The field of text-to-image generation has made remarkable strides in creating high-fidelity and photorealistic images. As this technology gains popularity, there is a growing concern about its potential security risks. However, there has been limited …


Right to be Forgotten in the Era of Large Language Models: Implications, Challenges, and Solutions

D Zhang, P Finckenberg-Broman, T Hoang, S Pan… - arXiv preprint arXiv …, 2023

… For example, Common Crawl data is 60% of the training data used in GPT-3 [3]; 50% of PaLM’s training dataset is social media conversations [4]; and OpenAI and Google have extensively used Reddit user posts in their large language models [5, 6 …


ROBBIE: Robust Bias Evaluation of Large Generative Language Models

D Esiobu, X Tan, S Hosseini, M Ung, Y Zhang… - arXiv preprint arXiv …, 2023

As generative large language models (LLMs) grow more performant and prevalent, we must develop comprehensive enough tools to measure and improve their fairness. Different prompt-based datasets can be used to measure social bias across …


RoBERTa-CoA: RoBERTa-Based Effective Finetuning Method Using Co-Attention

JH Kim, SW Park, JY Kim, J Park, SH Jung, CB Sim - IEEE Access, 2023

… has improved upon BERT by adding Common CrawlNews (CC-News), open web text, and story data, applying dynamic masking with multiple masked words generated per sentence, removing NSP, and increasing the batch size from 256 to …


Robust Information Extraction From Unstructured Documents

In computer science, robustness can be thought of as the ability of a system to handle erroneous or nonstandard input during execution. This thesis studies the robustness of the methods that extract structured information from unstructured …


ROZAM at SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis

M Rostamkhani, G Zamaninejad, S Eetemadi - … of the The 17th International Workshop …, 2023

We build a model using large multilingual pretrained language model XLM-T for regression task and fine-tune it on the MINT (Multilingual INTmacy) analysis dataset which covers 6 languages for training and 4 languages for testing zero-shot …


RTQ: Rethinking Video-language Understanding Based on Image-text Model

X Wang, Y Li, T Gan, Z Zhang, J Lv, L Nie - Proceedings of the 31st ACM International …, 2023

… Besides, Flamingo uses 2 billion image-text pairs and 27 million video-text pairs, while mPLUG-B uses 2 million videotext pairs and a large natural language corpus (WikiCorpus and common crawl) as the pre-training data. It is noteworthy that HiTeA [50] …


RuSentNE-2023: Evaluating Entity-Oriented Sentiment Analysis on Russian News Texts

A Golubev, N Rusnachenko, N Loukachevitch - arXiv preprint arXiv:2305.17679, 2023

… The authors selected articles of 14 US newspapers from the Common Crawl news crawl (CC-NEWS). Mentions of named entities such as PERSON or ORG were automatically identified, and corresponding sentences were extracted. The …


s-elBat: a Semantic Interpretation Approach for Messy taBle-s

M Cremaschi, R Avogadro, D Chieregato - Semantic Web Challenge on Tabular Data …, 2022

This paper describes s-elBat, a Semantic Table Interpretation approach. The approach inherits and improves the part of the techniques belonging to the MantisTable, an approach used and tested in previous editions of the SemTab …


Sailing the Data Sea to Advance Research on the Sustainable Development Goals

A Spezzatti, E Kheradmand, K Gupta, M Peras… - The Ethics of Artificial …, 2023

The Sustainable Development Goals (SDGs) are the framework adopted by the global community to encourage taking actions on the multiple challenges facing the world today to ensure environmental protection, health and well-being, and …


Saisiyat Is Where It Is At! Insights Into Backdoors And Debiasing Of Cross Lingual Transformers For Named Entity Recognition

RA Calix, J Ben-Joseph, N Lopatina, R Ashley… - 2022 IEEE International …, 2022

… This RoBERTa XLM model was pre-trained on text from 100 languages which came from a Common crawl and had a size of around 2.5 terabytes. Not all languages in this corpus are equally represented in terms of amount of text and …


Salogni at GeoLingIt: Geolocalization by Fine-tuning BERT

I Salogni - Proceedings of the Eighth Evaluation Campaign of …, 2023

… In this work, the models pre-trained on a miscellaneous corpus (umberto-commoncrawl-… work the best performing ones are the 32k umberto-commoncrawl-cased-v1 and the 31k bert-base-… However it is necessary to remember that only umberto-commoncrawl-cased-v1 …


SAND: Semantic Annotation of Numeric Data in Web Tables

Y Su, D Rafiei, BK Nazari - Proceedings of the 32nd ACM International …, 2023

A large portion of quantitative information about entities is expressed as Web tables, and these tables often lack proper schema and annotation, which introduces challenges for the purpose of querying and analysis. In this paper, we introduce …


Sarcasm Detection in English and Arabic Tweets Using Transformer Models

… XLM-RoBERTa is pre-trained on 2.5TB of filtered CommonCrawl data containing text from 100 languages [6]. RoBERTa, on the other hand… , in gigabytes, between CommonCrawl and Wikipedia data across a plethora of languages. CommonCrawl …


SC-Block: Supervised Contrastive Blocking within Entity Resolution Pipelines

A Brinkmann, R Shraga, C Bizer - arXiv preprint arXiv:2303.03132, 2023

… In total, 502 million records describing products and 55 million records describing local businesses were extracted from the Common Crawl. Due to the shallow coverage of the Common Crawl, the extracted data only represents a fraction of the …


Scalable 3D Captioning with Pretrained Models

T Luo, C Rockwell, H Lee, J Johnson - arXiv preprint arXiv:2306.07279, 2023

We introduce Cap3D, an automatic approach for generating descriptive text for 3D objects. This approach utilizes pretrained models from image captioning, image-text alignment, and LLM to consolidate captions from multiple views of a 3D asset …


Scalable and Accurate Self-Supervised Multimodal Representation Learning Without Aligned Video and Text Data

V Lialin, S Rawls, D Chan, S Ghosh, A Rumshisky… - Proceedings of the IEEE …, 2023

… Using Wikipedia, Common Crawl, and other text sources for dataset mining was essential in the recent NLP progress [17, 6]. CLIP [52] and ALIGN [27] demonstrated the importance of the scale of weakly-supervised data for images. …


Scalable Extraction of Training Data from (Production) Language Models

M Nasr, N Carlini, J Hayase, M Jagielski, AF Cooper… - arXiv preprint arXiv …, 2023

This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset. We show an adversary can extract gigabytes of training data …


SCALE: Scaling up the Complexity for Advanced Language Model Evaluation

V Rasiah, R Stern, V Matoshi, M Stürmer, I Chalkidis… - arXiv preprint arXiv …, 2023

Recent strides in Large Language Models (LLMs) have saturated many NLP benchmarks (even professional domain-specific ones), emphasizing the need for novel, more challenging novel ones to properly assess LLM capabilities. In this …


Scaling Expert Language Models with Unsupervised Domain Discovery

S Gururangan, M Li, M Lewis, W Shi, T Althoff… - arXiv preprint arXiv …, 2023

Large language models are typically trained densely: all parameters are updated with respect to all inputs. This requires synchronization of billions of parameters across thousands of GPUs. We introduce a simple but effective method to …


Scaling Laws for Generative Mixed-Modal Language Models

A Aghajanyan, L Yu, A Conneau, WN Hsu… - arXiv preprint arXiv …, 2023

… CommonCrawl). This also applies to auditory or visual modalities, which are captured with different sensors. Yet the decisions are not … (2022), and a custom image-text dataset scraped from Common Crawl. We remove all NSFW images and …


Scaling Speech Technology to 1,000+ Languages

V Pratap, A Tjandra, B Shi, P Tomasello, A Babu… - arXiv preprint arXiv …, 2023

… MMS-lab compared to Common Crawl, then we add it to the subset of words we include in our study. This enables us to evaluate on 51 languages of the FLEURS corpus since not all languages are covered by MMS-lab and we also need to find …


Schema. org: How is it used?

MH Dang, A Gaignard, H Skaf-Molli, P Molli - International Semantic Web Conference …, 2023

Schema.org defines a shared vocabulary for semantically annotating web pages. Due to the vast and diverse nature of the contributed annotations, it is not easy to understand the widespread use of Schema.org. In this poster, we rely on the …


Science Fact vs Science Fiction: A ChatGPT Immunological Review Experiment Gone Awry

Artificial intelligence (AI) has made great progress in recent years. The latest chatbot to make a splash is ChatGPT. To see if this type of AI could also be helpful in creating an immunological review article, I put a planned review on different classes …


SciFoodNER: Food Named Entity Recognition for Scientific Text

G Cenikj, G Petelin, BK Seljak, T Eftimov - 2022 IEEE International Conference on Big …, 2022

… Apart from the data used for the pre-training of BERT model, RoBERTa is trained on data from 3 additional sources: the CommonCrawl News dataset [24], the OpenWebText corpus [25] and the Common Crawl Stories dataset [26]. BioBERT is …


SciOL and MuLMS-Img: Introducing a Large-Scale Multimodal Scientific Dataset and Models for Image-Text Tasks in the Scientific Domain

T Tarsi, H Adel, JH Metzen, D Zhang, M Finco… - Proceedings of the IEEE …, 2024

In scientific publications, a substantial part of the information is expressed via figures containing images and diagrams. Hence, the retrieval of relevant figures given a natural language query is an important real-world task. However, due to the lack of …


Scraping Data from Web Pages Using SPARQL Queries

R Burget - International Conference on Web Engineering, 2023

… According to Web Data Commons statistics, structured data was available in 42% of the 33.8 million domains included in the Common Crawl corpus [3]. However, the use of structured data is growing slowly and is limited to certain domains. A large …


Scrapism: A Manifesto

Web scraping is a technique for automatically downloading and processing web content or converting online text and other media into structured data. This article describes the role that web scraping plays for web businesses and machine …


SeaLLMs--Large Language Models for Southeast Asia

XP Nguyen, W Zhang, X Li, M Aljunied, Q Tan, L Cheng… - arXiv preprint arXiv …, 2023

… data, we include web-based corpora such as Common Crawl2, journalistic content such as CC-News, text corpora with expertly-curated knowledge such as Wikipedia [9], and some scholarly publications. After collecting the data, we employ …


Seals_Lab at SemEval-2023 Task 12: Sentiment Analysis for Low-resource African Languages, Hausa and Igbo

N Raychawdhary, A Das, G Dozier, CD Seals - … of the The 17th International Workshop …, 2023

… The pretraining data includes 2.5 TB of filtered CommonCrawl data with 100 languages. … and also This particular model was trained on Common Crawl Corpus and the BBC news website. After our initial experiments, our task of sentiment …


Search This Blog

The California Effect occurs when California’s large market, its capacity to successfully regulate, its preference for stringent standards, the inability of the regulatory target to simply move beyond California’s jurisdiction, and non-divisibility …


Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capability

E Briakou, C Cherry, G Foster - arXiv preprint arXiv:2305.10266, 2023

Large, multilingual language models exhibit surprisingly good zero- or few-shot machine translation capabilities, despite having never seen the intentionally-included translation examples provided to typical neural translation systems. We investigate …


SecureBERT and LLAMA 2 Empowered Control Area Network Intrusion Detection and Classification

X Li, H Fu - arXiv preprint arXiv:2311.12074, 2023

… These models were trained on an extensive dataset comprising 2 trillion tokens sourced from various outlets such as web pages (CommonCrawl), open-source repository code (GitHub), Wikipedia content (in 20 different languages), public …


Segmentation-Free Streaming Machine Translation

J Iranzo-Sánchez, J Iranzo-Sánchez, A Giménez… - arXiv preprint arXiv …, 2023

Streaming Machine Translation (MT) is the task of translating an unbounded input text stream in real-time. The traditional cascade approach, which combines an Automatic Speech Recognition (ASR) and an MT system, relies on an intermediate …


Selecting Shots for Demographic Fairness in Few-Shot Learning with Large Language Models

C Aguirre, K Sasse, I Cachola, M Dredze - arXiv preprint arXiv:2311.08472, 2023

Recently, work in NLP has shifted to few-shot (in-context) learning, with large language models (LLMs) performing well across a range of tasks. However, while fairness evaluations have become a standard for supervised methods, little is known …


Self-Attention-Based Bi-LSTM Model for Sentiment Analysis on Tweets about Distance Learning in Higher Education.

I Lasri, A Riadsolh, M Elbelkacemi - … Journal of Emerging Technologies in Learning, 2023

For limiting the COVID-19 spread, countries around the world have implemented prevention measures such as lockdowns, social distancing, and the closers of educational institutions. Therefore, most academic activities are shifted to distance …


Self-Deception: Reverse Penetrating the Semantic Firewall of Large Language Models

Z Wang, W Xie, K Chen, B Wang, Z Gui, E Wang - arXiv preprint arXiv:2308.11521, 2023

Large language models (LLMs), such as ChatGPT, have emerged with astonishing capabilities approaching artificial general intelligence. While providing convenience for various societal needs, LLMs have also lowered the cost of generating harmful …


Self-Influence Guided Data Reweighting for Language Model Pre-training

M Thakkar, T Bolukbasi, S Ganapathy, S Vashishth… - arXiv preprint arXiv …, 2023

Language Models (LMs) pre-trained with self-supervision on large text corpora have become the default starting point for developing models for various NLP tasks. Once the pre-training corpus has been assembled, all data samples in the corpus are …


Self-Supervised and Weakly-Supervised Domain-Adaptive Continual Pre-Training in Low-Resource Settings

… Using this we process 3 Common Crawl dumps and extract all the available Bulgarian text from the .doc and .docx files they contain. Dump 2023-14 contains 52k documents and 75M words, dump 2022-49 contains 24k documents and 35M …


Self-supervised learning in natural language processing

Most natural language processing (NLP) learning algorithms require labeled data. While this is given for a select number of (mostly English) tasks, the availability of labeled data is sparse or non-existent for the vast majority of use-cases. To alleviate …


Self-supervised learning of multi-omics embeddings in the low-label, high-data regime

CJ Hurry, E Slade - arXiv preprint arXiv:2311.09962, 2023

Contrastive, self-supervised learning (SSL) is used to train a model that predicts cancer type from miRNA, mRNA or RPPA expression data. This model, a pretrained FT-Transformer, is shown to outperform XGBoost and CatBoost, standard …


Self-training Reduces Flicker in Retranslation-based Simultaneous Translation

S Sen, R Sennrich, B Zhang, B Haddow - Proceedings of the 17th Conference of the …, 2023

In simultaneous translation, the retranslation approach has the advantage of requiring no modifications to the inference engine. However, in order to reduce the undesirable flicker in the output, previous work has resorted to increasing the …


Semantic Information Extraction From Natural Language Using a Learning and Rule-Based Approach

Open Information Extraction (OIE) is a subset of Natural Language Processing (NLP) that constitutes the processing of natural language into structured and machinereadable data. This thesis uses data in Resource Description Framework (RDF) …


SEMANTIC TEXT ANALYSIS USING ARTIFICIAL NEURAL NETWORKS BASED ON NEURAL-LIKE ELEMENTS WITH TEMPORAL SIGNAL SUMMATION

K Alexander, S Eugeny, K Dmitry, P Dmitry - Проблемы искусственного интеллекта, 2023

… • and, of course, Internet web-pages collected, for example, in the CommonCrawl database [9] containing about a petabyte of data, although of low quality. Filtered versions of this base can already be used for training language models, such as the …


SemDeDup: Data-efficient learning at web-scale through semantic deduplication

A Abbas, K Tirumala, D Simig, S Ganguli, AS Morcos - arXiv preprint arXiv …, 2023

Progress in machine learning has been driven in large part by massive increases in data. However, large web-scale datasets such as LAION are largely uncurated beyond searches for exact duplicates, potentially leaving much redundancy. Here …


Semi-Supervised Dimensional Media Sentiment Analysis via Exploring Sample Relationships

P Liu, W Qian, H Li, J Cao - IEEE Transactions on Computational Social Systems, 2023

Dimensional sentiment analysis (DSA) aims to recognize continuous real-valued annotations in multidimensional spaces such as valence-arousal space. It can serve as a sentiment analysis that is more fine-grained than traditional polarity …


Semi-supervised learning and bidirectional decoding for effective grammar correction in low-resource scenarios

Z Mahmoud, C Li, M Zappatore, A Solyman, A Alfatemi… - PeerJ Computer Science, 2023

The correction of grammatical errors in natural language processing is a crucial task as it aims to enhance the accuracy and intelligibility of written language. However, developing a grammatical error correction (GEC) framework for low-resource …


SentAlign: Accurate and Scalable Sentence Alignment

S Steingrímsson, H Loftsson, A Way - arXiv preprint arXiv:2311.08982, 2023

We present SentAlign, an accurate sentence alignment tool designed to handle very large parallel document pairs. Given user-defined parameters, the alignment algorithm evaluates all possible alignment paths in fairly large documents of …


Sentence-Level Multimodal and Language-Agnostic Representations

PA Duquenne, H Schwenk, B Sagot - arXiv preprint arXiv:2308.11466, 2023

We introduce SONAR, a new multilingual and multimodal fixed-size sentence embedding space. Our single text encoder, covering 200 languages, substantially outperforms existing sentence embeddings such as LASER3 and LabSE on the …


SENTENCES SIMILARITY DETECTION IN INDONESIAN POETRY COMPARISON USING SIAMESE MALSTM

… It contains 2 million common crawl words with 300 dimensions, providing 600 billion word-vectors. FastText is different from Google word embedding because it does the analysis in the sub word context by providing the n-gram character level …


Sentiment analysis in Portuguese tweets: an evaluation of diverse word representation models

D Vianna, F Carneiro, J Carvalho, A Plastino, A Paes - Language Resources and …, 2023

During the past years, we have seen a steady increase in the number of social networks worldwide. Among them, Twitter has consolidated its position as one of the most influential social platforms, with Brazilian Portuguese speakers holding the fifth …


Sentiment analysis in Tourism: Fine-tuning BERT or sentence embeddings concatenation?

I Bouabdallaoui, F Guerouate, S Bouhaddour, C Saadi… - arXiv preprint arXiv …, 2023

Undoubtedly that the Bidirectional Encoder representations from Transformers is the most powerful technique in making Natural Language Processing tasks such as Named Entity Recognition, Question & Answers or Sentiment Analysis, however, the …


Sentiment Analysis of Forest Fires on Social Media Networks Twitter Using the Long Short Term Memory (LSTM) Method

AA Aziz, W Maharani - KLIK: Kajian Ilmiah Informatika dan Komputer, 2023

… This research will also build two corpus, namely the tweet corpus for the GloVe extension feature and the common crawl corpus for the FastText extension feature, to increase the model's accuracy. Classification of reviews through this process will …


Sentiment analysis on Hindi tweets during COVID‐19 pandemic

A Saroj, A Thakur, S Pal - Computational Intelligence

… The CBOW model learns by predicting the current word based on its context, and it was trained on Common Crawl and Wikipedia. Fasttext has given remarkable results in the English language. Still, in a regional language like Hindi, it is found …


Sentiment Analysis with Neural Models for Hungarian

LJ Laki, ZG Yang - Acta Polytechnica Hungarica, 2023

Sentiment analysis is a powerful tool to gain insight into the emotional polarity of opinionated texts. Computerized applications can contribute to the establishment of nextgeneration models that can provide us with data of unprecedented quantity and …


Separating the Wheat from the Chaff with BREAD: An open-source benchmark and metrics to detect redundancy in text

I Caswell, L Wang, I Papadimitriou - arXiv preprint arXiv:2311.06440, 2023

Data quality is a problem that perpetually resurfaces throughout the field of NLP, regardless of task, domain, or architecture, and remains especially severe for lower-resource languages. A typical and insidious issue, affecting both training data and model …


Sequence to Sequence Pre-Trained Model for Natural Language Processing

G Dhasmana, PK HR, G Prasad - … International Conference on Computer Science and …, 2023

… Text available in wikipedia is of good quality but it is small for pre –trianed model On the other side common crawl web scrapes are massive but relatively low quality. Colossal clean crawled corpus which is called as C4 it can serve the purpose of pre …


Sequence‐to‐sequence transfer transformer network for automatic flight plan generation

Y Yang, S Qian, M Zhang, K Cai - IET Intelligent Transport Systems, 2023

In this work, a machine translation framework is proposed to tackle the flight plan generation in the air transport field. Diverging from the traditional human expert‐based way, a novel sequence‐to‐sequence transfer transformer network to automatic flight …


Sequential Modeling Enables Scalable Learning for Large Vision Models

Y Bai, X Geng, K Mangalam, A Bar, A Yuille, T Darrell… - arXiv preprint arXiv …, 2023

We introduce a novel sequential modeling approach which enables learning a Large Vision Model (LVM) without making use of any linguistic data. To do this, we define a common format, "visual sentences", in which we can represent raw images …


Sequential Text-based Knowledge Update with Self-Supervised Learning for Generative Language Models

HR Sung, YJ Tang, YC Cheng, PL Chen, TY Li… - Proceedings of the 32nd …, 2023

… articles from Common Crawl3 that contain the information of ⟨𝑒1,𝑒2, ..., 𝑒𝑇 ⟩. The details are described as follows. Inspired by the concept in Gholipour Ghalandari et al. [11], we first collect the news events that are listed in Wiki Current …


Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study

B Wang, W Ping, P Xu, L McAfee, Z Liu, M Shoeybi… - arXiv preprint arXiv …, 2023

Large decoder-only language models (LMs) can be largely improved in terms of perplexity by retrieval (eg, RETRO), but its impact on text generation quality and downstream task accuracy is unclear. Thus, it is still an open question: shall we …


Shared Language: Linguistic Similarity in an Algebra Discussion Forum

MP Banawan, J Shin, T Arner, R Balyan, WL Leite… - Computers, 2023

Academic discourse communities and learning circles are characterized by collaboration, sharing commonalities in terms of social interactions and language. The discourse of these communities is composed of jargon, common terminologies …


Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

M Xia, T Gao, Z Zeng, D Chen - arXiv preprint arXiv:2310.06694, 2023

The popularity of LLaMA (Touvron et al., 2023a;b) and other recently emerged moderate-sized large language models (LLMs) highlights the potential of building smaller yet powerful LLMs. Regardless, the cost of training such models from scratch …


Short-Answer Grading for German: Addressing the Challenges

U Padó, Y Eryilmaz, L Kirschner - International Journal of Artificial Intelligence in …, 2023

Short-Answer Grading (SAG) is a time-consuming task for teachers that automated SAG models have long promised to make easier. However, there are three challenges for their broad-scale adoption: A technical challenge regarding the need …


Should ChatGPT and Bard Share Revenue with Their Data Providers? A New Business Model for the AI Era Advances in Artificial Intelligence and Machine Learning …

… of data sources, including the CommonCrawl data, WebText, two internet-based books corpora, as well as English-language Wikipedia. The CommonCrawl data contains 410 billion tokens originally from billions of web pages collected by the …


Should ChatGPT and Bard Share Revenue with Their Data Providers? A New Business Model for the AI Era

D Zhang - arXiv preprint arXiv:2305.02555, 2023

… sources, including the CommonCrawl data, WebText, two internetbased books corpora, as well as English-language Wikipedia. The CommonCrawl data contains 410 billion tokens originally from billions of web pages collected by the …


Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models

E Ferrara - arXiv preprint arXiv:2304.03738, 2023

As the capabilities of generative language models continue to advance, the implications of biases ingrained within these models have garnered increasing attention from researchers, practitioners, and the broader public. This article …


Sidlr: Slot and intent detection models for low-resource language varieties

SY Kwon, G Bhatia, EMB Nagoudi, AA Inciarte… - Tenth Workshop on NLP for …, 2023

Intent detection and slot filling are two critical tasks in spoken and natural language understandingfor task-oriented dialog systems. In this work, we describe our participation in slot and intent detection for low-resource language varieties (SID4LR)(Aepli …


SIEVE: MULTIMODAL DATASET PRUNING USING IMAGE CAPTIONING MODELS

A Mahmoud, M Elhoushi, A Abbas, Y Yang, N Ardalani… - arXiv preprint arXiv …, 2023

… To pretrain CLIP, billions of image-text pairs are collected using common crawl The raw data is highly diverse but contains many noisy image-text pairs, including low quality images, low quality alternative text (alt-text), and misaligned image-text …


SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore

S Min, S Gururangan, E Wallace, H Hajishirzi… - arXiv preprint arXiv …, 2023

The legality of training language models (LMs) on copyrighted or otherwise restricted data is under intense debate. However, as we show, model performance significantly degrades if trained only on low-risk text (eg, out-of-copyright books or …


Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation

J Mendonça, P Pereira, JP Carvalho, A Lavie… - arXiv preprint arXiv …, 2023

… This model is the multilingual version of RoBERTa, pretrained on CommonCrawl data containing 100 languages. We used a single Quadro RTX 6000 24GB GPU for the encoder experiments, and accessed ChatGPT (gpt-3.5-turbo) in late March using …


Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrieval

J Lin, D Alfonso-Hermelo, V Jeronymo, E Kamalloo… - arXiv preprint arXiv …, 2023

The advent of multilingual language models has generated a resurgence of interest in cross-lingual information retrieval (CLIR), which is the task of searching documents in one language with queries from another. However, the rapid pace of …


Simulating Users in Interactive Web Table Retrieval

B Engelmann, T Breuer, P Schaer - arXiv preprint arXiv:2310.11931, 2023

… Here, tables, with their context, are added from the Common Crawl dataset. Tables are thus present in different modalities, such as page titles, entities, text before/after, and the table itself. Modality relevance assessments were created by crowdsourcing. …


Sinhala Part of Speech Tagger using Deep Learning Techniques

M Sathsarani, T Thalawaththa, NK Galappaththi… - 2022 6th International …, 2022

Natural Language Processing (NLP) is a sub-field of Artificial Intelligence (AI) that consists of a collection of computational methods motivated by theory for the automated classification and reflection of human languages. The foundation for …


Sinhala-English Word Embedding Alignment: Introducing Datasets and Benchmark for a Low Resource Language

K Wickramasinghe, N de Silva - arXiv preprint arXiv:2311.10436, 2023

Since their inception, embeddings have become a primary ingredient in many flavours of Natural Language Processing (NLP) tasks supplanting earlier types of representation. Even though multilingual embeddings have been used for the …


Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models

MF Chen, N Roberts, K Bhatia, J Wang, C Zhang… - arXiv preprint arXiv …, 2023

The quality of training data impacts the performance of pre-trained large language models (LMs). Given a fixed budget of tokens, we study how to best select data that leads to good downstream model performance across tasks. We develop a new …


Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models

D Yu, S Kaur, A Gupta, J Brown-Cohen, A Goyal… - arXiv preprint arXiv …, 2023

… We identified 17 skills that appeared with rather high frequency in the common crawl corpus. Omitting them from the evaluation appeared to make it much harder for most models (see Section 5.2). This second version is recommended. …


SKIM at WMT 2023 general translation task

K Kudo, T Ito, M Morishita, J Suzuki - Proceedings of the Eighth Conference on …, 2023

… Monolingual Corpus We also used the following provided monolingual data: News Crawl, News Commentary, and Common Crawl. We backtranslated the monolingual sentences using a targetto-source model (ie, an initial translation model) …


SLDT: Sequential Latent Document Transformer for Multilingual Document-based Dialogue

… XLM-R is trained with a cross-lingual masked language modeling objective on data in 100 languages from Common Crawl. To improve the pre-training data quality, pages from Common Crawl were filtered by an n-gram language model trained on …


Slice-and-Forge: Making Better Use of Caches for Graph Convolutional Network Accelerators

M Yoo, J Song, H Lee, J Lee, N Kim, Y Kim, J Lee - arXiv preprint arXiv:2301.09813, 2023

Graph convolutional networks (GCNs) are becoming increasingly popular as they can process a wide variety of data formats that prior deep neural networks cannot easily support. One key challenge in designing hardware accelerators for GCNs is …


SlimPajama-DC: Understanding Data Combinations for LLM Training

Z Shen, T Tao, L Ma, W Neiswanger, J Hestness… - arXiv preprint arXiv …, 2023

… We tested different sampling strategies for different domains of our training data: (1) each token is trained only once during training, such as Commoncrawl, and (2) we perform more than one epoch for training on particular sources, such as the …


Smaller Language Models are Better Black-box Machine-Generated Text Detectors

F Mireshghallah, J Mattern, S Gao, R Shokri… - arXiv preprint arXiv …, 2023

… The GPT-2 family is reportedly trained on the WebText dataset, GPT-3 is trained on a combination of the Common Crawl 1, WebText2, books and Wikipedia, and there is not any information released about the training data of ChatGPT. … 1https://commoncrawl.org …


Smart Home Notifications in Croatian Language: A Transformer-Based Approach

M Simunec, R Soic - 2023 17th International Conference on …, 2023

Natural language generation is one of the essential components in achieving spoken interaction between a human user and a computer system. In this paper, we describe the process of generating notifications in Croatian language by employing …


Smart Knowledge Submission System Based on Natural Language Processing (NLP) Leveraging on Language Modelling Approach

MA Idris, H Alsultan - Gas & Oil Technology Showcase and Conference, 2023

… This led to the development of pre-trained systems such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer), which were trained with large language datasets, such as the …


Smart Subs Subtitling App for Watching Live Virtual Dome Performances

D Christopoulos, E Chatzi, G Sofianopoulos… - 2023 18th International …, 2023

… from the Greek Wikipedia, (b) the europarl database which contains textual data from the European Parliament, including eleven different languages, including Greek which we used, (c) the oscar database created by language classification and …


Smart Video Retrieval and Question Answering System

M Dubey, M Dashora, VK Shree, R Jayashree - 2023 International Conference for …, 2023

More and more video content is being produced on daily basis and utilizing it to its fullest has never been easier. In this paper we propose a new solution for this problem as a smart video retrieval system, leveraging the progress made in …


SMS Spam Classification–Simple Deep Learning Models With Higher Accuracy Using BUNOW And GloVe Word Embedding

S Giri, S Das, SB Das, S Banerjee - Journal of Applied Science and Engineering, 2023

… tokens; 2014 Wikipedia dump with 1.6 billion tokens; Gigaword5 which has 4.3 billion tokens; the combination Gigaword5 and Wikipedia2014, which has 6 billion tokens; and 42 billion tokens of web data, from Common Crawl. Each corpus was …


So2al-wa-Gwab: A new Arabic Question-Answering Dataset Trained on Answer Extraction Models

H Al-Omari, R Duwairi - ACM Transactions on Asian and Low-Resource …

Question answering (QA) is the task of responding to questions posed by users automatically. A question-answering system is divided into three main components: question analysis, information retrieval, and answer extraction; where this paper has …


Socially Aware Natural Language Processing with Commonsense Reasoning and Fairness in Intelligent Systems

Although Artificial Intelligence (AI) promises to deliver ever more user-friendly consumer applications, recent mishaps involving fake information and biased treatment serve as vivid reminders of the pitfalls of AI. AI can harbor latent biases …


Soft Language Clustering for Multilingual Model Pre-training

J Zeng, Y Jiang, Y Yin, Y Jing, F Meng, B Lin, Y Cao… - arXiv preprint arXiv …, 2023

Multilingual pre-trained language models have demonstrated impressive (zero-shot) cross-lingual transfer abilities, however, their performance is hindered when the target language has distant typology from source languages or when pre-training …


SOTAB: The WDC Schema. org Table Annotation Benchmark

K Korini, R Peeters, C Bizer - Semantic Web Challenge on Tabular Data to …, 2022

Understanding the semantics of table elements is a prerequisite for many data integration and data discovery tasks. Table annotation is the task of labeling table elements with terms from a given vocabulary. This paper presents the WDC Schema …


SoUnD Framework: Analyzing (So) cial Representation in (Un) structured (D) ata

M Díaz, S Dev, E Reif, R Denton, V Prabhakaran - arXiv preprint arXiv:2311.17259, 2023

… We apply the framework in two toy examples using the Common Crawl web text corpus (C4) and LAION-400M. We also propose a set of … LAION-400M is a multimodal text and image dataset that features over 400 million image-text pairs …


Source Prompt: Coordinated Pre-training of Language Models on Diverse Corpora from Multiple Sources

Y Xu, D Lu, J Liang, X Wang, Y Geng, Y Xin, H Wu… - arXiv preprint arXiv …, 2023

… For common crawl based corpora such as C4, their source information is largely unusable because their data is crawled from millions of web pages. Therefore, our current method is limited to the scenario where a certain number of small corpora …


Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models

Y Huang, Z Meng, F Liu, Y Su, N Collier, Y Lu - arXiv preprint arXiv:2308.16463, 2023

Large language models exhibit enhanced zero-shot performance on various tasks when fine-tuned with instruction-following data. Multimodal instruction-following models extend these capabilities by integrating both text and images. However …


SPARSITY IN LARGE LANGUAGE MODELS

… We chose RefinedWeb because it is a high-quality subset of Common Crawl, which is often used in the pretraining phase of LLMs, including Llama, Falcon, and OPT. We also use the validation split of WikiText (Merity et al., 2017) for measuring …


Spatiotemportal and multilingual Semantic Machine Learning Analysis of Social Media Data for the recent protests in Europe–based on Twitter data–

The perception of inherent tensions between justice and injustice (or the disproportion of good and bad) often press a group of people (or even the whole society) to seek change concerning politics and power, for example in the form of …


Speak While You Think: Streaming Speech Synthesis During Text Generation

A Dekel, S Shechtman, R Fernandez, D Haws, Z Kons… - arXiv preprint arXiv …, 2023

… We construct our training dataset based on the C4 (Common Crawl Cleaned Corpus) dataset [20], which was also used to train T5. As C4 contains 365M samples, we consider only a random fraction of the dataset containing 3M training and 130K …


Specious Sites: Tracking the Spread and Sway of Spurious News Stories at Scale

HWA Hanley, D Kumar, Z Durumeric - arXiv preprint arXiv:2308.02068, 2023

… To ensure full coverage of each site’s published articles, we additionally gather the HTML pages indexed by Common Crawl [48] for each site during this same period. We emphasize that under 1% of articles were only found in the Common …


Spelling convention sensitivity in neural language models

E Nielsen, C Kirov, B Roark - arXiv preprint arXiv:2303.03457, 2023

We examine whether large neural language models, trained on very large collections of varied English text, learn the potentially long-distance dependency of British versus American spelling conventions, ie, whether spelling is consistently …


Spot: A Natural Language Interface for Geospatial Searches in OSM

L Khellaf, IB Schlicht, J Bayer, R Bouwmeester, T Miraß… - arXiv preprint arXiv …, 2023

Investigative journalists and fact-checkers have found OpenStreetMap (OSM) to be an invaluable resource for their work due to its extensive coverage and intricate details of various locations, which play a crucial role in investigating news scenes …


Sprinter: Speeding Up High-Fidelity Crawling of the Modern Web

A Goel, J Zhu, R Netravali, HV Madhyastha

Crawling the web at scale forms the basis of many important systems: web search engines, smart assistants, generative AI, web archives, and so on. Yet, the research community has paid little attention to this workload in the last decade. In this paper …


SRBerta-BERT transformer language model for Serbian legal texts

… OSCAR is a large set of open data created using linguistic classification over data from the Common Crawl corpus [3]. The dataset we used consisted of 645,747 texts. The evaluation of the SRBert network was performed using 10of which consists of …


SSL-GAN-RoBERTa: A robust semi-supervised model for detecting Anti-Asian COVID-19 hate speech on social media

X Su, Y Li, P Branco, D Inkpen - Natural Language Engineering, 2023

Anti-Asian speech during the COVID-19 pandemic has been a serious problem with severe consequences. A hate speech wave swept social media platforms. The timely detection of Anti-Asian COVID-19-related hate speech is of utmost importance …


Stable Bias: Analyzing Societal Representations in Diffusion Models

AS Luccioni, C Akiki, M Mitchell, Y Jernite - arXiv preprint arXiv:2303.11408, 2023

As machine learning-enabled Text-to-Image (TTI) systems are becoming increasingly prevalent and seeing growing adoption as commercial services, characterizing the social biases they exhibit is a necessary first step to lowering their …


StarCoder: may the source be with you!

R Li, LB Allal, Y Zi, N Muennighoff, D Kocetkov, C Mou… - arXiv preprint arXiv …, 2023

The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context …


stopes-Modular Machine Translation Pipelines

P Andrews, G Wenzek, K Heffernan, O Çelebi, A Sun… - Proceedings of the The …, 2022

Neural machine translation, as other natural language deep learning applications, is hungry for data. As research evolves, the data pipelines supporting that research evolve too, oftentimes re-implementing the same core components. Despite the …


Strengthening Small and Medium-Sized Businesses' Cybersecurity: A Machine Learning-based Phishing Classification Model

… The study used 5,000 URLs from the PhishTank, OpenPhish, Alexa, and Common Crawl archives. The best model used a Random Forest algorithm and had an … The phishing URLs were extracted from phishtank.org while the legitimate …


STRESS TEST FOR BERT AND DEEP MODELS: PREDICTING WORDS FROM ITALIAN POETRY

In this paper we present a set of experiments carried out with BERT on a number of Italian sentences taken from poetry domain. The experiments are organized on the hypothesis of a very high level of difficulty in predictability at the three levels of …


string2string: A Modern Python Library for String-to-String Algorithms

M Suzgun, SM Shieber, D Jurafsky - arXiv preprint arXiv:2304.14395, 2023

We introduce string2string, an open-source library that offers a comprehensive suite of efficient algorithms for a broad range of string-to-string problems. It includes traditional algorithmic solutions as well as recent advanced neural approaches to …


Strong Prediction: Language model surprisal explains multiple N400 effects

JA Michaelov, MD Bardolph, CK Van Petten…

Theoretical accounts of the N400 are divided as to whether the amplitude of the N400 response to a stimulus reflects the extent to which the stimulus was predicted, the extent to which the stimulus is semantically similar to its preceding context, or …


Structural Self-Supervised Objectives for Transformers

L Di Liello - arXiv preprint arXiv:2309.08272, 2023

This thesis focuses on improving the pre-training of natural language models using unsupervised raw data to make them more efficient and aligned with downstream applications. In the first part, we introduce three alternative pre-training objectives to …


Students' perceptions of using ChatGPT in a physics class as a virtual tutor

L Ding, T Li, S Jiang, A Gapud - International Journal of Educational Technology in …, 2023

The latest development of Generative Artificial Intelligence (GenAI), particularly ChatGPT, has drawn the attention of educational researchers and practitioners. We have witnessed many innovative uses of ChatGPT in STEM classrooms. However …


Study of ChatGPT and its Comparison with Other Mainstream Large Language Models

… T5 was pre-trained on 34B tokens, which filtered the publicly crawled web dataset Common Crawl to remove some duplicates, low-quality, code-looking texts, etc., and finally kept only English texts to obtain dataset C4: the Colossal Clean Crawled …


Sub-Standards and Mal-Practices: Misinformation's Role in Insular, Polarized, and Toxic Interactions

… pages that were indexed by Common Crawl before August 2021. For each HTML page indexed by Common Crawl, we parse the HTML … Altogether we gather the available Common Crawl pages and scrape the HTML for 541 misinformation and …


Subject-verb Agreement with Seq2Seq Transformers: Bigger Is Better, but Still Not Best

MA Wilson, Z Zhou, R Frank - Proceedings of the Society for Computation in …, 2023

Past work (Linzen et al., 2016; Goldberg, 2019, ao) has used the performance of neural network language models on subject-verb agreement to argue that such models possess structure-sensitive grammatical knowledge. We investigate what …


Submission of USTC's system for the IWSLT 2023-Offline Speech Translation Track

X Zhou, J Cui, Z Ye, Y Wang, L Xu, H Zhang, W Zhang… - Proceedings of the 20th …, 2023

This paper describes the submissions of the research group USTC-NELSLIP to the 2023 IWSLT Offline Speech Translation competition, which involves translating spoken English into written Chinese. We utilize both cascaded models and end-to-end …


Subversion of the Human Aura: A Crisis in Representation

The human aura is now being subverted by a variety of simulacra. OpenAI’s language-generation program GPT-3 illustrates the challenges of interpreting algorithmic-generated texts. This article advocates interpretive strategies that …


Subword-Based Neural Machine Translation for Low-Resource Fusion Languages

A Nürnberger, EW De Luca, M Gasser
AM Gezmu - 2023

Neural approaches, which are currently state-of-the-art in many areas, have contributed significantly to the exciting advancements in machine translation. However, Neural Machine Translation (NMT) requires a substantial quantity and …


Suicidal Text Detection in Social Media

MP Karthikeyan, I Ajay, R Magesh, G Saran - 2023

This system is developed with the aim of providing and insight information of people who are personally disturbed by the factors of either their personal life or family background or bully at school or work pressure. With the help of people’s online …


Suicide risk assessment using word-level model with dictionary-based risky posts selection

YS Tsai, ALP Chen - Multimedia Tools and Applications, 2023

Suicide is a serious issue around the world and is a leading cause of death in US. In the past 20 years, the suicide rate has seen a significant increase of 35%. With the rapid development of information technology, more and more people begin to use …


SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation

J Jiang, C Dong, A Aizawa, S Kurohashi - arXiv preprint arXiv:2305.08371, 2023

… For TextTiling+Glove, we used the version pretrained with 42 billion tokens of web data from Common Crawl.For GreedySeg and CSM, we corrected some inconsistencies in their open-sourced codes with respect to their original published …


Superlim: A Swedish Language Understanding Evaluation Benchmark

A Berdičevskis, G Bouma, R Kurtz, F Morger, J Öhman… - Proceedings of the 2023 …, 2023

We present Superlim, a multi-task NLP benchmark and analysis platform for evaluating Swedish language models, a counterpart to the English-language (Super) GLUE suite. We describe the dataset, the tasks, the leaderboard and report the …


Supervised Feature Selection to Improve the Accuracy for Malware Detection

Malware is becoming increasingly sophisticated and difficult to detect with traditional monitoring tools and antivirus software. As a result, machine learning has become a popular approach for classifying and detecting malware-related data. In this study …


Supervised Feature-based Classification Approach to Bilingual Lexicon Induction from Specialised Comparable Corpora

This study, submitted to the BUCC2023 shared task on bilingual term alignment in comparable specialised corpora, introduces a supervised, feature-based classification approach. The approach employs both static cross-lingual …


Supervised Knowledge Makes Large Language Models Better In-context Learners

L Yang, S Zhang, Z Yu, G Bao, Y Wang, J Wang, R Xu… - arXiv preprint arXiv …, 2023

Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The recent progress in large-scale generative models has further expanded their use in real-world language applications. However, the …


Supervised Machine Generated Text Detection Using LLM Encoders In Various Data Resource Scenarios

With the recent innovation in Large Language Models, the world has been taken by storm by the vast implications and applications of including these effective new creations into our everyday lives. However, as with most things, these new …


Support of Sparse Tensor Computing for MLIR HLS

GM Liang, CL Lee, R Lai, JK Lee - Proceedings of the 52nd International Conference …, 2023

Nowadays, sparse tensor computations are widely used in machine learning. Without the multiplications in zero values, sparse tensor computation can significantly reduce the latency and power consumption. Famous frameworks like …


Supporting Account-Based Queries for Archived Instagram Posts

Social media has become one of the primary modes of communication in recent times, with popular platforms such as Facebook, Twitter, and Instagram leading the way. Despite its popularity, Instagram has not received as much attention in …


Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models

E Doostmohammadi, T Norlund, M Kuhlmann… - arXiv preprint arXiv …, 2023

Augmenting language models with a retrieval mechanism has been shown to significantly improve their performance while keeping the number of parameters low. Retrieval-augmented models commonly rely on a semantic retrieval mechanism …


Surgicberta: a pre-trained language model for procedural surgical language

M Bombieri, M Rospocher, SP Ponzetto, P Fiorini - International Journal of Data …, 2023

… of the CommonCrawl News, from OpenWebText, and some stories from CommonCrawl data. RoBERTa has been trained via MLM with dynamic masking: ie, each time a sequence is input to the model, a new masking pattern is created …


SurreyAI 2023 Submission for the Quality Estimation Shared Task

A Sindhujan, D Kanojia, C Orasan, T Ranasinghe - Proceedings of the Eigth …, 2023

Quality Estimation (QE) systems are important in situations where it is necessary to assess the quality of translations, but there is no reference available. This paper describes the approach adopted by the SurreyAI team for addressing the Sentence-Level …


Survey On Machine Learning Paradigms For Phishing Website Detection

Phishing attacks continue to be a major security threat for individuals and organizations alike. It causes billions of dollars in losses annually. Machine learning (ML) has shown great promise in detecting such attacks by identifying patterns and …


Survey Paper: Automatic Title Generation for Text with RNN and Pre-trained Transformer Language Model

Nowadays huge amounts of text data are available due to the evolution of the Internet. Although search engines are used to select text data, it is unfeasible to go through the entire search results of text that are related to search intent. Therefore …


Syntactic-Semantic Similarity Based on Dependency Tree Kernel

M Alian, A Awajan - Arabian Journal for Science and Engineering, 2023

The representation of words in the vector space generated using the Word2Vec model does not capture the syntactic similarity between sentences, but only the semantic similarity by considering the context of words. To address this problem, we …


Synthetic Cross-language Information Retrieval Training Data

J Mayfield, E Yang, D Lawrie, S Barham, O Weller… - arXiv preprint arXiv …, 2023

… Documents in NeuCLIR 1 are news articles extracted from Common Crawl News. The HC3 dataset consists of Chinese and Persian Tweet reply … criterion used for the NeuCLIR 1 was dropped since Twitter conversations are less coherent than …


System for Enhancing Accuracy of Noisy Text using Deep Network Language Models

R Rohit, SA Gandheesh, KS Suriya, PB Pati - 2023 IEEE 8th International Conference …, 2023

… C4 dataset is a collection of English language text sourced from an open-source Common Crawl web scrape. Each record consists of an input erroneous test along with its target text. While the dataset contains many millions of sentences, for this …


Systematic Rectification of Language Models via Dead-end Analysis

M Cao, M Fatemi, JCK Cheung, S Shabanian - arXiv preprint arXiv:2302.14003, 2023

With adversarial or otherwise normal prompts, existing large language models (LLM) can be pushed to generate toxic discourses. One way to reduce the risk of LLMs generating undesired discourses is to alter the training of the LLM. This can be very …


Systems and Algorithms for Dynamic Graph Processing

Data generated from human and systems interactions could be naturally represented as graph data. Several emerging applications rely on graph data, such as the semantic web, social networks, bioinformatics, finance, and trading among …


Systems and Resources for Telugu: Question Answering and Summarization

Natural language processing (NLP) is a bridge between the computer and human interactions in their natural language. NLP has wide variety of applications such as machine translation, text summarization, question-answering, sentiment analysis, etc …


T-MARS: Improving Visual Representations by Circumventing Text Feature Learning

P Maini, S Goyal, ZC Lipton, JZ Kolter, A Raghunathan - arXiv preprint arXiv …, 2023

… 46] datasets, there has been a growing interest in exploring improved strategies for selecting subsets of the common crawl that help learn better … Most recently, DataComp [14] was introduced as a benchmark challenge for subset selection from …


T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification

IJ Unanue, G Haffari, M Piccardi - arXiv preprint arXiv:2306.04996, 2023

… 2021), all typically using hundreds of gigabytes or terabytes of pretraining data (eg, 2.5 TB of Common Crawl data for XLM-R). However, such large, pretrained multilingual models still suffer from significant performance limitations, particularly in …


T5G2P: Multilingual Grapheme-to-Phoneme Conversion with Text-to-Text Transfer Transformer

M Řezáčková, A Frémund, J Švec, J Matoušek - Asian Conference on Pattern …, 2023

… The original Google’s T5-base English model Footnote 4 (labeled as t5-EN in our paper) was trained from Common Crawl data Footnote 5 . We replicated the same pre-processing procedure to obtain the data for other languages tested in the …


TabLib: A Dataset of 627M Tables with Context

G Eggert, K Huo, M Biven, J Waugh - arXiv preprint arXiv:2310.07875, 2023

… An important caveat is that there may be a selection bias affecting this analysis, due to factors such as our exclusion of tables larger than 1 GB, Common Crawl’s truncation of large responses, parsing bugs and limitations, etc. We leave a more …


Tackling Fake News in Bengali: Unraveling the Impact of Summarization vs. Augmentation on Pre-trained Language Models

AS Chowdhury, GM Shahariar, AT Aziz, SM Alam… - arXiv preprint arXiv …, 2023

With the rise of social media and online news sources, fake news has become a significant issue globally. However, the detection of fake news in low resource languages like Bengali has received limited attention in research. In this paper, we …


Tackling Hallucinations in Chart Summarization

… Pre-training models on large corpora like Common Crawl results in models learning language in its parameters. This is called parametric knowledge which helps improve the performance when the model is finetuned on a downstream task …


Tackling the multilingual and heterogeneous documents with the pre-trained language identifiers

MR Kanfoud, A Bouramoul - International Journal of Computers and Applications, 2023

… The authors collected data from a non-profit foundation, Common Crawl, which explores the Web and provides data freely to the public. The collected datasets are heterogeneous and multilingual. The authors proposed a multithreaded pipelined …


TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes

B Upadhayay, V Behzadan - arXiv preprint arXiv:2311.10797, 2023

… Authors trained XLM-R model by increasing the training examples and training the cross-lingual presentations from more than 2 terabytes of Common Crawl data. Chi et al. [28] further improved these models using two discriminative pretraining …


TaCo: Targeted Concept Removal in Output Embeddings for NLP via Information Theory and Explainability

F Jourdan, L Béthune, A Picard, L Risser, N Asher - arXiv preprint arXiv:2312.06499, 2023

… 2018) a dataset containing a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas. Pre-training was performed on these data by randomly masking 15% of the words in each of the input sentences and then trying …


Taiwan LLM: Bridging the Linguistic Divide with a Culturally Aligned Language Model

YT Lin, YN Chen - arXiv preprint arXiv:2311.17487, 2023

… When we added approximately 9 billion tokens of CommonCrawl data, specifically filtered for Traditional Chinese content (zh-tw), the performance of TAIWAN-LLM unexpectedly declined. For the 13 billion parameter model, the Exact …


Take a Closer Look at Multilinguality! Improve Multilingual Pre-Training Using Monolingual Corpora Only

J Lu, Y Lu, J Zhang - Findings of the Association for Computational …, 2023

… Pre-training Data We collect monolingual corpora from Common Crawl Corpus, which contains about 890GB data for 50 languages. Different from previous studies, we do not use any bilingual corpus. Following (Conneau and Lample…


Taken out of context: On measuring situational awareness in LLMs

L Berglund, AC Stickland, M Balesni, M Kaufmann… - arXiv preprint arXiv …, 2023

We aim to better understand the emergence of `situational awareness' in large language models (LLMs). A model is situationally aware if it's aware that it's a model and can recognize whether it's currently in testing or deployment. Today's LLMs are …


Targeted and Troublesome: Tracking and Advertising on Children's Websites

Z Moti, A Senol, H Bostani, FZ Borgesius, V Moonsamy… - arXiv preprint arXiv …, 2023

… Applying this classifier to over two million pages from the Common Crawl dataset, we compile a list of two thousand child-directed websites. Crawling these sites from five vantage points, we measure the prevalence of trackers, fingerprinting scripts …


Targeted Data Augmentation Improves Context-aware Neural Machine Translation

H Gete, T Etchegoyhen, G Labaka - Proceedings of Machine Translation Summit XIX …, 2023

Progress in document-level Machine Translation is hindered by the lack of parallel training data that include context information. In this work, we evaluate the potential of data augmentation techniques to circumvent these limitations, showing that …


TASTA: Text‐Assisted Spatial and Temporal Attention Network for Video Question Answering

T Wang, B Hou, J Li, P Shi, B Zhang, H Snoussi - Advanced Intelligent Systems, 2022

… GLoVe[34] is pretrained on the Common Crawl dataset and it maps each word (eg, the ith word) into a 300D feature vector (eg, ti ∈ ℝ300). Considering that the length of the video and the image size are not fixed, the video is preprocessed to a uniform …


Taxi1500: A Multilingual Dataset for Text Classification in 1500 Languages

C Ma, A ImaniGooghari, H Ye, E Asgari, H Schütze - arXiv preprint arXiv:2305.08487, 2023

… massive data from Common Crawl in 104 languages, significantly boosting the performance and outperforming mBERT. Compared with XLM, XLM-R has a larger vocabulary size of 250K. Besides, the training data scales from Wikipedia to a larger …


Teach LLMs to Personalize--An Approach inspired by Writing Education

C Li, M Zhang, Q Mei, Y Wang, SA Hombaiah, Y Liang… - arXiv preprint arXiv …, 2023

Personalized text generation is an emerging research area that has attracted much attention in recent years. Most studies in this direction focus on a particular domain by designing bespoke features or models. In this work, we propose a general …


TeamAmpa at SemEval-2023 Task 3: Exploring Multilabel and Multilingual RoBERTa Models for Persuasion and Framing Detection

A Pauli, R Sarabia, L Derczynski, I Assent - Proceedings of the The 17th International …, 2023

This paper describes our submission to theSemEval 2023 Task 3 on two subtasks: detectingpersuasion techniques and framing. Bothsubtasks are multi-label classification problems. We present a set of experiments, exploring howto get robust …


Tech tapas-one byte at a time more about AI, machine learning and ChatGPT

F MacDowall - Australian Law Librarian, 2023

… The Common Crawl corpus13 contains petabytes of data collected over 12 years of web … While it can be argued that the Common Crawl corpus is an accurate portrayal of the discourse of … Notably, the Common Crawl overrepresents those …


Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks

… , which we illustrate both theoretically and empirically on simple networks as well as Megatron transformer trained on Common Crawl. … We demonstrate this pedagogically on resnet with MLP blocks but also on Megatron transformer trained …


Testing the Depth of ChatGPT's Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3. 5's Abilities in Regard to Recognizing and Generating ASCII-Art …

D Bayani - arXiv preprint arXiv:2307.16806, 2023

… Thus, this is a form of ASCII-art for which GPT3.5 may have a substantial amount of varied training data as compiled in the common-crawl dataset11 and potentially other locations. Additionally, owing to its use in technical areas often as an aid to …


Testing the Limits of Unified Sequence to Sequence LLM Pretraining on Diverse Table Data Tasks

S Sarkar, L Lausen - arXiv preprint arXiv:2310.00789, 2023

… . 2015) is a collection of about 125 million data tables extracted from the Common Crawl. 3. TAPEX: w use the dataset used in training the … The tables were extracted from the Common Crawl although the timeline of the pages of the common crawl …


TEXILA INTERNATIONAL JOURNAL OF ACADEMIC RESEARCH

RAT-SQL is among the popular framework used in the Text-To-SQL challenges for jointly encoding the database relations and questions in a way to improve the semantic parser. In this work, we propose a light version of the RAT-SQL where we …


Text Analytics Solutions for the Control of Fake News: Materials and Methods

E Ogbuju, T Abiodun, F Oladipo - International Journal of Open Information …, 2023

The increase in the rate of internet and social media use has given rise to a lot of fake news and misinformation available online. The internet and social media have made information and communications flow to be faster and easier. On the other …


Text augmentation for semantic frame induction and parsing

S Anwar, A Shelmanov, N Arefyev, A Panchenko… - Language Resources and …, 2023

Semantic frames are formal structures describing situations, actions or events, eg, Commerce buy, Kidnapping, or Exchange. Each frame provides a set of frame elements or semantic roles corresponding to participants of the situation and lexical …


Text Content Moderation Model to Detect Sexually Explicit Content

Our objective is to build a Transformer based classification model for the recognition of Sexually Explicit Speech and do a comparative analysis with other (fine-tuned) models in the Sexually Explicit domain. Our approach was to take an open-source …


Text Intimacy Analysis using Ensembles of Multilingual Transformers

T Chavan, V Patwardhan - arXiv preprint arXiv:2312.02590, 2023

Intimacy estimation of a given text has recently gained importance due to the increase in direct interaction of NLP systems with humans. Intimacy is an important aspect of natural language and has a substantial impact on our everyday …


Text mining of scientific literature and the need for open access

… We need a scientific analogue to Common-Crawl, an open repository of scientific articles for use in exploratory data analysis. Ironically, this argument is not new, and indeed, was anticipated as part of the original definition of open access given at the …


Text Summarization for Resource-Poor Languages: Datasets and Models for Multiple Indian Languages

… model pre-trained on Common Crawl [7] data in 100 languages. XLM-R is created by training a Transformer-based masked language model on a hundred languages. • mT5: mT5 [77] is a multilingual variant of T5 model pre-trained on …


Text Summarization of Medical Documents using Abstractive Techniques

E Lalitha, K Ramani, D Shahida, EVS Deepak… - 2023 2nd International …, 2023

Medical researchers are exposed to enormous amounts of medical information in the form of medical news, clinical trial reports, research articles, etc. Researchers would need the documents' summaries that help them decide to do an in-depth study …


Text, Speech, and Dialogue: 26th International Conference, TSD 2023, Pilsen, Czech Republic, September 4–6, 2023, Proceedings

The annual International Conference on Text, Speech and Dialogue (TSD), which emerged in 1998, constitutes a recognized platform for presenting and discussing stateof-the-art technology and recent achievements in the computer processing of …


Text-based Patent-Quality Prediction Using Multi-Section Attention

The number of patents has increased tremendously in recent years and statistics derived from patents have become the standard measurement for innovation. Patents statistics are widely available and correlate well with patent valuation and …


Text-Blueprint: An Interactive Platform for Plan-based Conditional Generation

F Huot, J Maynez, S Narayan, RK Amplayo, K Ganchev… - Proceedings of the 17th …, 2023

… -sourced answer spans from Wikipedia, and matched with passages from web documents from Common Crawl. AQuaMuse uses the answer passages as summaries with the passages extracted from Common Crawl as the input documents …


Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attacks

X Zhang, H Hong, Y Hong, P Huang, B Wang, Z Ba… - arXiv preprint arXiv …, 2023

The language models, especially the basic text classification models, have been shown to be susceptible to textual adversarial attacks such as synonym substitution and word insertion attacks. To defend against such attacks, a growing body of …


Text-Guided Image Generation for Railway Intrusion Anomaly Detection

J Chen, C Zhang, X Feng, Y Liu - 2023 IEEE International Conference on Unmanned …, 2023

Railway intrusion are characterized by strong suddenness, high unpredictability and many disturbing factors, which are key factors affecting railway safety. Currently, the deep neural network-based intrusion object detection algorithm relies on the …


Text-guided Image-and-Shape Editing and Generation: A Short Survey

CKT Chao, Y Gingold - arXiv preprint arXiv:2304.09244, 2023

… CLIP is trained on 400 million text-image pairs from common-crawl dataset with contrastive objectives to learn image representations from text. The overall framework can be seen in Figure 1. To be more specific, the image encoder is …


That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?

J Lee, A Liu, O Ahia, H Gonen, NA Smith - arXiv preprint arXiv:2310.14610, 2023

… Moreover, we observe that the differences between LMs and MT-specific models become less pronounced for more under-resourced languages (the languages are ordered left to right based on count of pages in Common Crawl9). 9https://commoncrawl.github.io …


The (Undesired) Attenuation of Human Biases by Multilinguality

C España-Bonet, A Barrón-Cedeño - Proceedings of the 2022 Conference on …, 2022

… CCWP: Models trained on Common Crawl and Wikipedia using CBOW with position weights and subword information (Grave et al.… We also build 5 static in-house word embeddings on Common Crawl using a subset of the CC-100 corpus (Conneau et al.…


The Algorithmic Internet: Culture, capture, corruption

The internet is haunted by ailing social media platforms with declining engagement and perpetual feeds of quick dopamine hits; large machine learning models are contaminated with an insurmountable sea of toxicity and cul-de-sacs of gibberish …


The Axiom of Choice and Its Influence on LLM Hallucinations: An Exploration

The Axiom of Choice (AoC), a foundational proposition in set theory, allows for the selection of elements from a collection of non-empty sets without specifying the selection method. Surprisingly, this mathematical principle offers insights into the …


The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants

L Bandarkar, D Liang, B Muller, M Artetxe, SN Shukla… - arXiv preprint arXiv …, 2023

We present Belebele, a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. Significantly expanding the language coverage of natural language understanding (NLU) benchmarks, this dataset …


The BETTER Cross-Language Datasets

I Soboroff - Proceedings of the 46th International ACM SIGIR …, 2023

… The document collections are taken from CommonCrawl.The phase 1 documents were collected by MITRE, and for phases 2 and 3 by ARLIS. Care was taken to identify subsets of the CommonCrawl collection that were likely to contain topical …


The BETTER Cross-Language Information Retrieval Datasets


The Claire French Dialogue Dataset

J Hunter, J Louradour, V Rennard, I Harrando… - arXiv preprint arXiv …, 2023

We present the Claire French Dialogue Dataset (CFDD), a resource created by members of LINAGORA Labs in the context of the OpenLLM France initiative. CFDD is a corpus containing roughly 160 million words from transcripts and stage plays in …


The Cultivated Practices of Text-to-Image Generation

J Oppenlaender - arXiv preprint arXiv:2306.11393, 2023

Humankind is entering a novel creative era in which anybody can synthesize digital information using generative artificial intelligence (AI). Text-to-image generation, in particular, has become vastly popular and millions of practitioners produce AI-generated …


The Cultural Devaluation of Feminized Work: The Evolution of Occupational Prestige and Gender Typing in the United States, 1900-2019

… to head and visual modality from the Common Crawl embedding are positively associated with an occupation’s survey-based prestige. … Index (ISEI) and prestige measures from the embeddings of the Common Crawl (CC) corpus. Table 1 …


The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model

DM Levine, R Tuwani, B Kompa, A Varma… - medRxiv, 2023

Importance: Artificial intelligence (AI) applications in health care have been effective in many areas of medicine, but they are often trained for a single task using labeled data, making deployment and generalizability challenging. Whether a general-purpose …


The Dimensions of Data Labor: A Road Map for Researchers, Activists, and Policymakers to Empower Data Producers

H Li, N Vincent, S Chancellor, B Hecht - arXiv preprint arXiv:2305.13238, 2023

Many recent technological advances (eg ChatGPT and search engines) are possible only because of massive amounts of user-generated data produced through user interactions with computing systems or scraped from the web (eg …


The Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Models

L Hagström, D Saynova, T Norlund, M Johansson… - arXiv preprint arXiv …, 2023

Large Language Models (LLMs) make natural interfaces to factual knowledge, but their usefulness is limited by their tendency to deliver inconsistent answers to semantically equivalent questions. For example, a model might predict both "Anne …


The Effects of Corpus Choice and Morphosyntax on Multilingual Space Induction

V Ravishankar, J Nivre - Findings of the Association for Computational …, 2022

… There is also a clear difference between Wikipedia and Common Crawl; in general, we find that correlations tend to be either weaker or less significant with Common Crawl than with Wikipedia. We hypothesise that this is due to Wikipedia …


The emergence of autolography: the 'magical'invocation of images from text through AI

C Chesher, C Albarrán-Torres - Media International Australia, 2023

In mid-2022, AI systems automatically translating text into evocative original images became an internet sensation. People compared it to magic: invoking an uncannily competent artist–magician. We call it ‘autolography’ from the Greek ‘automatos + …


The Falcon Series of Open Language Models

E Almazrouei, H Alobeidli, A Alshamsi, A Cappelli… - arXiv preprint arXiv …, 2023

… CommonCrawl), and evaluate on English tasks. Each data split uses a dedicated tokenizer. … Latin alphabet, and sufficient presence in CommonCrawl to collect at least 10 billion tokens. … Since training of the models was started before we had …


The Future of AI-Assisted Writing

CA Pereira, T Komarlu, W Mobeirek - arXiv preprint arXiv:2306.16641, 2023

The development of Natural Language Generation models has led to the creation of powerful Artificial Intelligence-assisted writing tools. These tools are capable of predicting users' needs and actively providing suggestions as they write. In this work …


The Future of the Human–Machine Interface (HMI) in Society 5.0

D Mourtzis, J Angelopoulos, N Panopoulos - Future Internet, 2023

The blending of human and mechanical capabilities has become a reality in the realm of Industry 4.0. Enterprises are encouraged to design frameworks capable of harnessing the power of human and technological resources to enhance the era of …


The Gender-GAP Pipeline: A Gender-Aware Polyglot Pipeline for Gender Characterisation in 55 Languages

B Muller, B Alastruey, P Hansanti, E Kalbassi… - arXiv preprint arXiv …, 2023

… level for Common Crawl. We segment each sample at the word level using Stanza tokenizer available in the given language (Qi et al.… In addition, we run the pipeline on a sample of Common CrawlCommon Crawl is a snapshot of crawlable …


The generative art community bridging between UNESCO Heritage and AI-generated works: an interview with Celestino Soddu and Enrica Colabella

S Park - Proceedings of GA2023, XXVI Generative Art …, 2023

… Around 60% of ChatGPT-3’s dataset was based on af iltered version of the web-crawled data of Common Crawl [1], a non -profit organisation that scraps publicly available textual data on the internet such as books, web pages and ar ticles. Most of the tasks …


The Ghost in the Machine: Generating Beliefs with Large Language Models

I introduce a methodology to generate economic expectations by applying large language models to historical news. Leveraging this methodology, I make three key contributions.(1) I show generated expectations closely match existing survey …


The Governance Challenge Posed by Large Learning Models

On August 24, 2023, the Office of the Australian Information Commissioner (OAIC) and 11 of its international data protection and privacy counterparts released a joint statement on web scraping (data collected by a bot from a wide range of websites) …


The Great Awokening as a Global Phenomenon

D Rozado - arXiv preprint arXiv:2304.01596, 2023

… The text content of the analyzed articles is available on the web domains of the respective media outlets and often also in Internet cache repositories such as Common Crawl, Google cache or the Internet Archive. Our content analysis is limited …


The History and Future of Human–Robot Communication

F Shkurti - The SAGE Handbook of Human–Machine …, 2023


The House that Looked Like It Should Collapse. Natural Language Processing for Architectural Design

N Gaudillière-Jami - … Conference on Computer-Aided Architectural Design …, 2023

… It has been trained on billions of online texts, coming from databases such as Common Crawl, Books2 or Wikipedia. The texts it generates are therefore based on a very large range of information, including about architecture. While only being word-based …


The Impact of Arabic Diacritization on Word Embeddings

M Abbache, A Abbache, J Xu, F Meziane, X Wen - ACM Transactions on Asian and …, 2023

Word embedding is used to represent words for text analysis. It plays an essential role in many Natural Language Processing (NLP) studies and has hugely contributed to the extraordinary developments in the field in the last few years. In …


The Impact of Depth and Width on Transformer Language Model Generalization

J Petty, S van Steenkiste, I Dasgupta, F Sha, D Garrette… - arXiv preprint arXiv …, 2023

To process novel sentences, language models (LMs) must generalize compositionally -- combine familiar elements in new ways. What aspects of a model's structure promote compositional generalization? Focusing on transformers …


The Impact of GloVe and Word2Vec Word-Embedding Technologies on Bug Localization with Convolutional Neural Network

In the field of software engineering, software quality assurance faces many challenges, including overcoming the problem of identifying errors in the source code. Finding the location of the error in the source code is a very important process …


The Importance of Context in the Evaluation of Word Embeddings: The Effects of Antonymy and Polysemy

Word embeddings are widely used for diverse applications in natural language processing. Despite extensive research, it is unclear when they succeed or fail to capture human judgements of semantic relatedness and similarity. In this study, we …


The Less the Merrier? Investigating Language Representation in Multilingual Models

HH Nigatu, AL Tonja, J Kalita - arXiv preprint arXiv:2310.13228, 2023

… 2019) is a multilingual Transformer-based MLM trained on the Common Crawl data for 100 languages. To balance data between English and other languages, the XLM-R uses one dump for English and 12 dumps for all other languages. … Filtered …


The Library & Generative AI

N Gustafson-Sundell, M McCullough - 2023

A demonstration of several AI tools, including ChatGPT, ChatPDF, Consensus, and more. The focus of the session is on potential student uses of the tools and related library initiatives, so we address the limits of ChatGPT as an information source …


The Mental Representation of Structured Auditory Sequences in Memory

The capacity to remember and retrieve sequences of items or everyday experiences is fundamental to human intelligence. The goal of this research is to understand how the episodic memory system supports the integration, representation, and retrieval of …


The more

KS Yong, JSY Liew - Journal of Intelligent Information Systems, 2023

… The main difference between the two InferSent model is one uses GloVe word embeddings trained on Common Crawl 840B (InferSent-GloVe) and the other uses fastText word embeddings trained on Common Crawl 600B (InferSent-fastText). …


The Nordic Pile: A 1.2 TB Nordic Dataset for Language Modeling

J Öhman, S Verlinden, A Ekgren, AC Gyllensten… - arXiv preprint arXiv …, 2023

… We decided that our Nordic-language datasets were more prone to include many duplicate documents since we included several data sources from Common Crawl that may overlap for these languages. So, our English data was deduplicated intra-shard …


The Ouroboros Threat

JM Vukov, TL Joseph, G Lebkuecher, M Ramirez… - The American Journal of …, 2023

Jorge Luis Borges introduces the mythical ouroboros as follows:“A third-century Greek amulet, to be found today in the British Museum, gives us an image that can better illustrate that infinitude: the serpent that bites its own tail”(Borges 2005). In this …


The Perils & Promises of Fact-checking with Large Language Models

D Quelle, A Bovet - arXiv preprint arXiv:2310.13549, 2023

… The GPT models are trained on vast amounts of unstructured textual data like the common crawl data-set, which is the largest data-set to be included in the training. Common Crawl is a web archive that consists of terabytes of data collected since …


The perpetual motion machine of AI-generated data and the distraction of ChatGPT-as-scientist

J Listgarten - arXiv preprint arXiv:2312.00818, 2023

Since ChatGPT works so well, are we on the cusp of solving science with AI? Is not AlphaFold2 suggestive that the potential of LLMs in biology and the sciences more broadly is limitless? Can we use AI itself to bridge the lack of data in the sciences in …


The Photographic Pipeline of Machine Vision; or, Machine Vision's Latent Photographic Theory

Despite computer vision's extensive mobilization of cameras, photographers, and viewing subjects, photography's place in machine vision remains undertheorized. This article illuminates an operative theory of photography that exists in a latent form …


The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only

G Penedo, Q Malartic, D Hesslow, R Cojocaru… - Thirty-seventh Conference …, 2023

… Although this is only a 135 narrow subset of the kind of pages making up CommonCrawl, we found this finding to hold more 136 broadly. We use trafilatura for text extraction, and apply extra formatting via regular expressions: 137 we limit new …


The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

G Penedo, Q Malartic, D Hesslow, R Cojocaru…

… Although this is only a narrow subset of the kind of pages making up CommonCrawl, we found this finding to hold more broadly. We use trafilatura … in the processed CommonCrawl data were identified as English. We find the …


The Role of Codeword-to-Class Assignments in Error-Correcting Codes: An Empirical Study

I Evron, O Onn, TW Orzech, H Azeroual, D Soudry - arXiv preprint arXiv:2302.05334, 2023

Error-correcting codes (ECC) are used to reduce multiclass classification tasks to multiple binary classification subproblems. In ECC, classes are represented by the rows of a binary matrix, corresponding to codewords in a codebook. Codebooks are …


The Role of Pre-training Data in Transfer Learning

R Entezari, M Wortsman, O Saukh, MM Shariatnia… - arXiv preprint arXiv …, 2023

… Common crawl. https://commoncraw .org/. Accessed: 2022-09-20. … 2021): The images and corresponding alt-texts come from web pages collected by Common Crawl (Com) between 2014 and 2021. We randomly select a subset of 2.7M and …


The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages

C Zhang, KD Doan, Q Liao, M Abdul-Mageed - arXiv preprint arXiv:2310.14557, 2023

Instruction tuned large language models (LLMs), such as ChatGPT, demonstrate remarkable performance in a wide range of tasks. Despite numerous recent studies that examine the performance of instruction-tuned LLMs on various NLP benchmarks …


The social construction of datasets: On the practices, processes and challenges of dataset creation for machine learning

Despite the critical role that datasets play in how systems make predictions and interpret the world, the dynamics of their construction are not well understood. Drawing on a corpus of interviews with dataset creators, we uncover the messy and …