From Text to Source: Results in Detecting Large Language Model-Generated Content

W Antoun, B Sagot, D Seddah - arXiv preprint arXiv:2309.13322, 2023

The widespread use of Large Language Models (LLMs), celebrated for their ability to generate human-like text, has raised concerns about misinformation and ethical implications. Addressing these concerns necessitates the development of robust …


From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference

S Samsi, D Zhao, J McDonald, B Li, A Michaleas… - arXiv preprint arXiv …, 2023

… Like other LLMs, LLaMA was pre-trained on a large collection of data including but not limited to CommonCrawl, Github, Wikipedia, etc. As of spring 2023, alongside other recently timed releases of state-of-the-art LLMs such as Google’s …


From Zero to Hero: Examining the Power of Symbolic Tasks in Instruction Tuning

Q Liu, F Zhou, Z Jiang, L Dou, M Lin - arXiv preprint arXiv:2304.07995, 2023

Fine-tuning language models on tasks with instructions has demonstrated potential in facilitating zero-shot generalization to unseen tasks. In this paper, we introduce a straightforward yet effective method for enhancing instruction tuning by employing …


FSTP Project Report AID2030–ArtificialIn-telligenceDataKit2030

… CommonCrawl29 creates and maintains an open web crawl data collection. Petabytes of data have been collected by Common Crawl since 2008, including … RealNews33 is a dataset of news articles from Common Crawl. The text and the …


Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defense

A Kucharavy, Z Schillaci, L Maréchal, M Würsch… - arXiv preprint arXiv …, 2023

… to train GPT-2, along with a filtered and deduplicated version of the Common Crawl, a dataset of all the text that can be found by a crawl on … from Wikipedia is likely to be seen on average 3.4 times during the training compared to 0.44 times for …


GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Exploration

A Piktus, O Ogundepo, C Akiki, A Oladipo, X Zhang… - arXiv preprint arXiv …, 2023

… Rather than investing in scraping the Web on their own, dataset creators typically turn to Common Crawl1 as the main source of text to include in their corpora. A repository of Web snapshots dating back to 2011, Common Crawl contains various …


GEM: Gestalt Enhanced Markup Language Model for Web Understanding via Render Tree

Z Shao, F Gao, Z Qi, H Xing, J Bu, Z Yu, Q Zheng, X Liu - Proceedings of the 2023 …, 2023

… Our corpora are built from the Common Crawl12 dataset. We derive approximately 2 million training samples from 100k renderable web pages by pre-processing. Details of the data pre-processing are available in Section 3.1. … 12https://commoncrawl.org/ …


Gender stereotypes embedded in natural language are stronger in more economically developed and individualistic countries

… Relying on two different sources (Wikipedia and Common Crawl), we found that these gender stereotypes are all significantly more pronounced in the text corpora of more economically developed and more individualistic countries. Our analysis …


Gender stereotypes in neural sentence representations

Neural networks have seen a spike in popularity in natural language processing in re- cent years. They consistently outperform the traditional methods and require less human labor to perfect as they are trained unsupervised on large text corpora …


Gender Stereotypes in User-Generated Content

Gender stereotypes pose an important hurdle on the way to gender equality. It is difficult to quantify the problem, though, as stereotypical beliefs are often subconscious or not openly expressed. User-generated content (UGC) opens up …


Generating a Large Web Traffic Dataset

… Resource Locators) from the Common Crawl dataset as a foundational … the Common Crawl dataset, by requesting each website of the collection to measure the required server response time. Nonetheless, this solution has its drawbacks …


Generating abstract art using artificial neural networks

Articial intelligence (AI) tools are slowly becoming integral to our everyday lives. Today we already rely on AI for much of our navigation and security. In the near future, much like the internet today, AI and machine learning tools are likely to assist …


Generating plumitifs descriptions using neural networks

Comme dans de nombreuses autres démocraties, il existe au Canada un droit d'accès à l'information judiciaire. Il s' agit d'un élément fondamental de tout processus judiciaire. Ce droit a deux objectifs principaux: offrir une fenêtre sur le système de …


Generating Query Focused Summaries without Fine-tuning the Transformer-based Pre-trained Models

D Abdullah, S Nayak, G Suri, Y Chali - arXiv preprint arXiv:2303.06230, 2023

Fine-tuning the Natural Language Processing (NLP) models for each new data set requires higher computational time associated with increased carbon footprint and cost. However, fine-tuning helps the pre-trained models adapt to the latest data sets; …


Generating Synthetic Data from Large Language Models

S Choenni, T Busker, MS Bargh - 2023 15th International Conference on Innovations …, 2023

Data collection for studying social phenomena is not only costly but is also, at best, a time-consuming and tedious task. Therefore, tools that may ease the task of data collection will speed up these studies and improve their efficiency. In this contribution …


Generation of Synthetic Responses to Survey Questions Using GPT-3: A Case of Hard-to-Reach Members of Russian Elites (based on the Survey of Russian Elites)

The goal of this study is to use a large language model, such as GPT-3, to generate responses from hard-to-reach elite members to multiple-choice questions. The most relevant or data-consistent responses can then be used to infer potential responses …


“Generations: Creative Computation, Community, and the Rhetorical Canon” investigates how computational poets and artists use the intrinsic rhetoricity of generative computational processes for social critique and community-building …


Generative AI and Lexicography: The Current State of the Art Using ChatGPT

GM de Schryver - International Journal of Lexicography, 2023

In this article, all ten papers and talks that have been devoted to the use of ChatGPT in lexicography so far are critically analysed, their results tabulated and cross-compared, from which the leading trends are determined. Extrapolating from the trendlines, a …


Generative AI and the Digital Commons

S Huang, D Siddarth - arXiv preprint arXiv:2303.11074, 2023

… They also rely on nonprofits like Common Crawl (which build and maintain open repositories of web crawl data), Creative Commons (for open licenses for the data used), open source libraries, and other digital infrastructure. They also take …


Generative AI Considered Harmful

The recent months have seen an explosion of interest, hype, and concern about generative AI, driven by the release of ChatGPT. In this article I seek to explicate some potential and actual harms of the engineering and use of generative AI such …


Generative AI for Medical Imaging: extending the MONAI Framework

WHL Pinaya, MS Graham, E Kerfoot, PD Tudosiu… - arXiv preprint arXiv …, 2023

… -hungry, they have thrived in fields with large publicly available datasets, such as computer vision, with datasets like ImageNet [8] and LAION [43], as well as natural language processing (with access to large textual corpora, like the Wikipedia Text …


Generative AI for Semantic Communication: Architecture, Challenges, and Outlook

L Xia, Y Sun, C Liang, L Zhang, MA Imran, D Niyato - arXiv preprint arXiv:2308.15483, 2023

… Through pretraining on large datasets (like Wikipedia and Common Crawl) and fine-tuning on user data (to be more personalized and customized), GPT-Neo is capable to be directly installed into TDs to precisely extract keywords of source …


Generative AI in Academic Writing, Ethical Recommendations

As the use of Artificial Intelligence (AI) tools such as Generative Pre-trained Transformers (GPTs) becomes more prevalent in academic writing, there are growing concerns about the ethical implications of their use. This article addresses …


Generative approach for gender-rewriting task with ArabicT5

S Alrowili, K Vijay-Shanker - Proceedings of the The Seventh Arabic Natural …, 2022

… 2021) is a multilingual variant of T5, which was pre-trained on the new Common Crawl-based dataset that consists of 6.3T tokens covering 101 languages. The mT5 model also uses a large vocabulary file that consists of 250K tokens. …


Generative Artificial Intelligence through ChatGPT and Other Large Language Models in Ophthalmology: Clinical Applications and Challenges

TF Tan, AJ Thirunavukarasu, JP Campbell, PA Keane… - Ophthalmology Science

The rapid progress of large language models (LLMs) driving generative artificial intelligence applications heralds the potential of opportunities in healthcare. We conducted a review up to April 2023 on Google Scholar, Embase, MEDLINE, and …


Generative Artificial Intelligence: A Double-Edged Sword

K Kuck - 2023 World Engineering Education Forum-Global …, 2023

… Stable Diffusion was trained with three massive image datasets collected by the nonprofit LAION, which built their datasets from CommonCrawl, another nonprofit that scrapes through the internet monthly and releases datasets of what they …


Generative Artificial Intelligence: Fundamentals

JM Corchado, S López, R Garcia, P Chamoso - ADCAIJ: Advances in Distributed …, 2023

Generative language models have witnessed substantial traction, notably with the introduction of refined models aimed at more coherent user-AI interactions—principally conversational models. The epitome of this public attention has arguably been the …


Generative Benchmark Creation for Table Union Search

K Pal, A Khatiwada, R Shraga, RJ Miller - arXiv preprint arXiv:2308.03883, 2023

Data management has traditionally relied on synthetic data generators to generate structured benchmarks, like the TPC suite, where we can control important parameters like data size and its distribution precisely. These benchmarks were …


Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations By admin 6 Comments

JA Goldstein, G Sastry, M Musser, R DiResta… - 2023

In recent years, artificial intelligence (AI) systems have significantly improved and their capabilities have expanded. In particular, AI systems called “generative models” have made great progress in automated content creation, such as images generated …


Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations

JA Goldstein, G Sastry, M Musser, R DiResta… - arXiv preprint arXiv …, 2023

Generative language models have improved drastically, and can now produce realistic text outputs that are difficult to distinguish from human-written content. For malicious actors, these language models bring the promise of automating the …


Generative Language Models Exhibit Social Identity Biases

T Hu, Y Kyrychenko, S Rathje, N Collier… - arXiv preprint arXiv …, 2023

The surge in popularity of large language models has given rise to concerns about biases that these models could learn from humans. In this study, we investigate whether ingroup solidarity and outgroup hostility, fundamental social biases known …


Generative Language Models for Automated Programming Feedback

In recent years, Generative Language Models have exploded into the mainstream with household names like BERT and ChatGPT, proving that text generation could have the potential to solve a variety of tasks. As the number of students enrolled into …


Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions

G Yenduri, G Srivastava, PKR Maddikunta, RH Jhaveri… - arXiv preprint arXiv …, 2023

… 500 billion words called ”Common Crawl” that was gathered from a sizable content archive and the internet [30]. Its other noteworthy and unexpected capability is its ability to carry out basic mathematical operations, write bits of code, and carry …


Generative Pretraining in Multimodality

Q Sun, Q Yu, Y Cui, F Zhang, X Zhang, Y Wang, H Gao… - arXiv preprint arXiv …, 2023

We present Emu, a Transformer-based multimodal foundation model, which can seamlessly generate images and texts in multimodal context. This omnivore model can take in any single-modality or multimodal data input indiscriminately (eg …


Geoparsing at Web-scale-Challenges and Opportunities

The increasing amount of web data being generated and stored along with geographic information is of great importance to enrich future search applications in science, news, economics, etc.. In addition to location information provided by users …


Geopolitical Forecasting Analysis of the Russia-Ukraine War Using the Expert's Survey, Predictioneer's Game and GPT-3

K Kalinin - Predictioneer's Game and GPT-3 (April 8, 2023), 2023

The Russia-Ukraine war that began on February 24th has resulted in significant geopolitical upheaval, sending shockwaves that reverberated around the globe. This extended and highly destructive war continues to rage on without any apparent …


German Compounds in Transformer Models

German is known for its highly productive word formation processes, particularly in the area of compounding and derivation. In this thesis, we focus on German nominal compounds and their representation in machine translation (MT) outputs. Despite …


German Question Tags: A Computational Analysis

Y Clausen - Proceedings http://ceur-ws. org ISSN, 2023

The German language exhibits a range of question tags that can typically, but not always, be substituted for one another. Moreover, the same words can have other meanings while occurring in the sentence-昀椀 nal position. The tags’ felicity …


GERNERMED++: Semantic annotation in German medical NLP through transfer-learning, translation and word alignment

J Frei, L Frei-Stuber, F Kramer - Journal of Biomedical Informatics, 2023

We present a statistical model, GERNERMED++, for German medical natural language processing trained for named entity recognition (NER) as an open, publicly available model. We demonstrate the effectiveness of combining multiple …


Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs

F Kang, HA Just, Y Sun, H Jahagirdar, Y Zhang, R Du…

This work focuses on leveraging and selecting from vast, unlabeled, open data to pre-fine-tune a pre-trained language model. The goal is to minimize the need for costly domain-specific data for subsequent fine-tuning while achieving desired performance levels …


GetPt: Graph-enhanced General Table Pre-training with Alternate Attention Network

R Jia, H Guo, X Jin, C Yan, L Du, X Ma, T Stankovic… - Proceedings of the 29th …, 2023

… However, the dataset mainly contains small and medium size tables [16] from the Common Crawl, and we observe that the tables have a relatively simple structure. Additionally, the dataset only contains 386 samples, leading to high variance in its …


GIO: Gradient Information Optimization for Training Dataset Selection

D Everaert, C Potts - arXiv preprint arXiv:2306.11670, 2023

… [4] develop a heuristic method to filter CommonCrawl based on a trained classifier’s probability that datapoints are high quality. Similarly, … an analysis of undesirable content in the Common Crawl corpus. In Proceedings of the 59th Annual Meeting of …


GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks

S Gupta, C Rosenbaum, ER Elenberg - arXiv preprint arXiv:2311.09606, 2023

Large language models (LLMs) have the ability to perform in-context learning (ICL) of new tasks by conditioning on prompts comprising a few task examples. This work studies the problem of selecting the best examples given a candidate pool to …


GiusBERTo: Italy's AI-Based Judicial Transformation: A Teaching Case

P Datta, BJ Zahn, L Attias, G Salierno, R Bertè… - Communications of the …, 2023

In an age when open access to law enforcement files and judicial documents can erode individual privacy and confidentiality, miscreants can abuse this open access to personal information for blackmail, misinformation, and even social engineering …


Gloss Alignment using Word Embeddings

H Walsh, OM Sincan, B Saunders, R Bowden - 2023 IEEE International Conference …, 2023

… The models are trained on the Common Crawl and Wikipedia datasets and have an output dimension of 300. For the following experiments we use the English implementation when testing our approach on the BOBSL dataset and the German …


GlotScript: A Resource and Tool for Low Resource Writing System Identification

AH Kargaran, F Yvon, H Schütze - arXiv preprint arXiv:2309.13320, 2023

We present GlotScript, an open resource and tool for low resource writing system identification. GlotScript-R is a resource that provides the attested writing systems for more than 7,000 languages. It is compiled by aggregating information from existing …


GMNLP at SemEval-2023 Task 12: Sentiment Analysis with Phylogeny-Based Adapters

MMI Alam, R Xie, F Faisal, A Anastasopoulos - arXiv preprint arXiv:2304.12979, 2023

This report describes GMU's sentiment analysis system for the SemEval-2023 shared task AfriSenti-SemEval. We participated in all three sub-tasks: Monolingual, Multilingual, and Zero-Shot. Our approach uses models initialized with AfroXLMR-large …


GNN-based retrieval and recommadation system: A semantic enhenced graph model

L Wang, X Li, H Zhang, Y Dai, S Zhang - 2022 IEEE 5th Advanced Information …, 2022

As a general method that captures knowledge from relation, graph modeling has been widely used in many fields including recommendation in Internet applications. A typical graph based system must be provided with a nodes and edges. To get …


Google Bard Artificial Intelligence versus the 2022 Self-Assessment Study Program for Urology

LM Huynh, BT Bonebrake, K Schultis, A Quach… - Urology Practice, 2023

… While Bard’s training dataset has been largely undisclosed by its Google developers, a 2022 paper discussing the underlying LaMDA model [5] found its training set composition to be 50% from online public forums, 12.5% from Wikipedia …


GPT (Generative Pre-trained Transformer)–A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions

G Yenduri, M Ramalingam, G Chemmalar Selvi…

… It is taught using a corpus of 500 billion words called ”Common Crawl” that was gathered from a sizable content archive and the internet [26]. Its other noteworthy and unexpected capability is its ability to carry out basic mathematical operations …


GPT is an effective tool for multilingual psychological text analysis

S Rathje, DM Mirea, I Sucholutsky, R Marjieh… - 2023

… Our analyses so far have focused on widely-spoken languages that are relatively highly represented in CommonCrawl – a large repository … eight of these languages, some of which had less than 20 million speakers (Table 1) and relatively little …


GPT takes the Bar Exam

… As reported in [19], OpenAI’s models are trained on a combination of curated CommonCrawl data and high-quality reference data that, if we consider The Pile V1 as reference [24], may have included some material from public legal sources …


GPT-3 for Education; Benefits and Concerns

MIZ Abidin, A Alkhalidi, F Mustaffa - … Conference on Creative Multimedia 2023 (ICCM …, 2023

… It was trained on publicly available data, including the Common Crawl dataset, which contains a vast amount of text from the web. GPT-3 excels in language translation, question answering, and human-like text generation. Compared to …


GPT-3 Models are Few-Shot Financial Reasoners

RS de Padua, I Qureshi, MU Karakaplan - arXiv preprint arXiv:2307.13617, 2023

… GPT-3 is a 175B parameter autoregressive language model trained on 45TB from CommonCrawl, Wikipedia and others, showing state-of-the-art performance on many NLP-related tasks. Often these tasks can be performed by providing a relevant …


GPT-3 vs Object Oriented Programming Assignments: An Experience Report

BP Cipriano, P Alves - Proceedings of the 2023 Conference on Innovation …, 2023

… We were also curious about GPT’s capacity to reply to prompts written in a language other than English, since it gathers information from the Common Crawl dataset which is primarily comprised of English content (46% as of 2022) [9]. All of …


GPT-4 can pass the Korean National Licensing Examination for Korean Medicine Doctors

D Jang, TR Yun, CY Lee, YK Kwon, CE Kim - PLOS Digital Health, 2023

… For example, the pre-training dataset for GPT-3, a model released in 2020 that GPT-3.5 is based on and the technical details have been officially reported, is mainly derived from a modified version of Common Crawl [19] in which English-based data …


GPT-4 For Developers

This resource is designed to bridge the gap between theoretical understanding and practical application, making it a useful tool for software developers, data scientists, AI researchers, and tech enthusiasts interested in harnessing the power of GPT-4 in …


Gradient Ascent Post-training Enhances Language Model Generalization

D Yoon, J Jang, S Kim, M Seo - arXiv preprint arXiv:2306.07052, 2023

… We consider OPT LMs to be familiar (in-domain) to Common Crawl, as it was included in their pretraining corpora. As OPT LMs were not explicitly trained on the Github corpora we consider OPT to be unfamiliar (out-ofdistribution) with Github …


Grammatical Error Correction with byte-level language models

… It is pre-trained on over a hundred languages on the multilingual dataset named mC4: multilingual Colossal Cleaned Common Crawl. The training of the model mirrors that of t5 closely, but differs both in that is was trained on a multilingual …


GreekBART: The First Pretrained Greek Sequence-to-Sequence Model

I Evdaimon, H Abdine, C Xypolopoulos, S Outsios… - arXiv preprint arXiv …, 2023

The era of transfer learning has revolutionized the fields of Computer Vision and Natural Language Processing, bringing powerful pretrained models with exceptional performance across a variety of tasks. Specifically, Natural Language Processing …


GreekT5: A Series of Greek Sequence-to-Sequence Models for News Summarization

N Giarelis, C Mastrokostas, N Karacapilidis - arXiv preprint arXiv:2311.07767, 2023

Text summarization (TS) is a natural language processing (NLP) subtask pertaining to the automatic formulation of a concise and coherent summary that covers the major concepts and topics from one or multiple documents. Recent advancements in …


GRI: Graph-based Relative Isomorphism of Word Embedding Spaces

MA Ali, Y Hu, J Qin, D Wang - arXiv preprint arXiv:2310.12360, 2023

… In order to validate these claims for GRI, we re-run the experiments using target embeddings trained on 33.8 million lines of web-crawl data from the English Common Crawl data. The embeddings for the source languages (“bn", “uk" and “ta") …


GRILLBot-v2: Generative Models for Multi-Modal Task-Oriented Assistance

S Fischer, N Tecklenburg, P Zubel, E Kupcova…

… Figure 2 shows how our pipeline converts raw textual task sources (Common Crawl, Wikipedia, WikiHow) into an augmented TaskGraph corpus using LLMs, external knowledge sources, and multi-modal content. Specifically for Common …


GTCOM and DLUT's Neural Machine Translation Systems for WMT23

H Zong, C Bei, C Yuan, W Chen, H Liu, D Huang

This paper presents the submission by Global Tone Communication Co., Ltd. and Dalian Univeristy of Technology for the WMT23 shared general Machine Translation (MT) task at the Conference on Empirical Methods in Natural Language Processing (EMNLP) …


Hallucinating Machines: Exploring the ethical implications of generative language models

… OpenAI utilised a filtered version of the Common Crawl dataset as the foundation of the training dataset for GPT-3 (Brown et al., 2020). The Common Crawl dataset comprises petabytes of web data collected since 2008 (So You’re Ready to Get …


Hallucinations in Large Multilingual Translation Models

NM Guerreiro, D Alves, J Waldendorf, B Haddow… - arXiv preprint arXiv …, 2023

Large-scale multilingual machine translation systems have demonstrated remarkable ability to translate directly between numerous languages, making them increasingly appealing for real-world applications. However, when deployed in the …


HARMONY@ DravidianLangTech: Transformer-based Ensemble Learning for Abusive Comment Detection

A Murugappan, LP RS, M Deivamani - Proceedings of the Third Workshop on …, 2023

Millions of posts and comments are created every minute as a result of the widespread use of social media and easy access to the internet. It is essential to create an inclusive environment and forbid the use of abusive language against any …


Hate Speech Detection in Hindi

Social media is a great place to share one’s thoughts and to express oneself. Very often the same social media platforms become a means for spewing hatred. The large amount of data being shared on these platforms make it difficult to moderate …


Hate speech: semi-automatic recognition in Slovak texts

… This could be caused by the data on which was the FastText model trained (texts from Wikipedia – which primarily represent scientific language style – and Common Crawl). Hate speech is likely underrepresented in the training data; perhaps the data …


HatEmoTweet: Low Level Emotion Classifications and Spatio-Temporal Trends of Hate and Offensive COVID-19 Tweets

Social media platforms (like Twitter) positively and negatively impact users in diverse societies; one of Twitter’s negative effects is the usage of hate and offensive language. Hate speech fosters prejudice; it also harms the vulnerable. There are …


HC3: A Suite of Test Collections for CLIR Evaluation over Informal Text

D Lawrie, J Mayfield, DW Oard, E Yang, S Nair… - Proceedings of the 46th …, 2023

While there are many test collections for Cross-Language Information Retrieval (CLIR), none of the large public test collections focus on short informal text documents. This paper introduces a new pair of CLIR test collections with millions of Chinese or …


HeRo: RoBERTa and Longformer Hebrew Language Models

V Shalumov, H Haskey - arXiv preprint arXiv:2304.11077, 2023

In this paper, we fill in an existing gap in resources available to the Hebrew NLP community by providing it with the largest so far pre-train dataset HeDC4, a state-of-the-art pre-trained language model HeRo for standard length inputs and an efficient …


HHLD: Hateful posts Identification in Hindi Language leveraging multi task learning

P Kapil, G Kumari, A Ekbal, S Pal, A Chatterjee… - IEEE Access, 2023

… 2.5 TB of common crawl data in 100 languages are used to train it. It outperforms M-BERT across cross-lingual classification, especially for low-resource languages. To apply sub-word tokenization on the raw input, sentence piece [50] with unigram …


Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpus

A Piergentili, B Savoldi, D Fucci, M Negri, L Bentivogli - arXiv preprint arXiv …, 2023

Gender inequality is embedded in our communication practices and perpetuated in translation technologies. This becomes particularly apparent when translating into grammatical gender languages, where machine translation (MT) often defaults to …


Hierarchical Representations From Large Mathematical Corpora

… Examples of common corpora used in NLP include Project Gutenberg1, which is an online library of more that 60,000 free ebooks [83]; the Common Crawl data set2 is a large collection of data from internet websites. And, of particular importance to …


High-Efficiency Device-Cloud Collaborative Transformer Model

P Jiang, K Xin, C Li, Y Zhou - Proceedings of the IEEE/CVF Conference on Computer …, 2023

Natural Language Processing (NLP) experts have had significant success with unsupervised language pre-training techniques. However, compared to typical NLP models, modern self-attention models require far more computational and memory …


HODI at EVALITA 2023: Overview of the first Shared Task on Homotransphobia Detection in Italian

D Nozza, AT Cignarella, G Damo, T Caselli, V Patti - Proceedings of EVALITA, 2023

HODI is a new shared task for the automatic detection of homotransphobia in Italian presented at EVALITA 2023. The challenge is organized into two subtasks: Subtask A focuses on the binary textual classification of homotransphobic tweets, while …


Home Evolving Standards of Care Demystifying AI: Current Insights on Artificial Intelligence in Thoracic Oncology

… ChatGPT was trained on a diverse mix of datasets, including a dataset known as Common Crawl,4 which alone contains nearly a trillion words before filtering. It is worth noting that this model learns in an unsupervised manner—it does not rely on …


HOMOCHAR: A novel adversarial attack framework for exposing the vulnerability of text based neural sentiment classifiers

A Bajaj, DK Vishwakarma - Engineering Applications of Artificial Intelligence, 2023

… In addition, RoBERTa had also been trained on datasets which include CC-News (Common Crawl-News), Open WebText, and others. These datasets have a combined size of about 160 GB. RoBERTa used a batch size of 8000 with 300,000 …


Homophobia and transphobia detection for low-resourced languages in social media comments

PK Kumaresan, R Ponnusamy, R Priyadharshini… - Natural Language …, 2023

… It’sa multilingual language model with 2.5 TB of filtered Common Crawl data that was trained. XLM-RoBERTa was fine-tuned for evaluation and inference on a range of downstream tasks using Wikipedia data in over 100 languages. XLM-RoBERTa …


Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale

V Deshpande, D Pechi, S Thatte, V Lialin, A Rumshisky - arXiv preprint arXiv …, 2023

In recent years, language models have drastically grown in size, and the abilities of these models have been shown to improve with scale. The majority of recent scaling laws studies focused on high-compute high-parameter count settings, leaving the …


HOW CHATGPT CAN BE USED FOR A CONTRACT DRAFTING CLASS

… • The Common Crawl dataset (https://commoncrawl.org), …


How Deep Contextualized Representation and Attention Mechanism Justifies Explainable Cross-lingual Sentiment Analysis

R Ghasemi, S Momtazi - ACM Transactions on Asian and Low-Resource …, 2023

… It has been pre-trained on the CommonCrawl Corpus, containing 100 languages with 2.5TB of data. Considering the advantages of LSTM models in capturing words’ sequential behavior, we build an LSTM-based classiier on the top of XLM-RoBERTa …


How Does Calibration Data Affect the Post-training Pruning and Quantization of Large Language Models?

M Williams, N Aletras - arXiv preprint arXiv:2311.09755, 2023

… This consists of English web text from Common Crawl, filtered with heuristics that form a “… This mainly consists of web text (Common Crawl and C4), in addition to high-quality sources … was created from stringent filtering and deduplication of …


How far can we get in creating a digital replica of a philosopher?

A Strasser, E Schwitzgebel, M Crosby - 2023

Can we build machines with which we can have interesting conversations? Observing the new optimism of AI regarding deep learning and new language models, we set ourselves an ambitious goal: We want to find out how far we can get …


How Language-Dependent is Emotion Detection? Evidence from Multilingual BERT

L De Bruyne, P Singh, O De Clercq, E Lefever, V Hoste - 2nd Workshop on Multi …, 2022

As emotion analysis in text has gained a lot of attention in the field of natural language processing, differences in emotion expression across languages could have consequences for how emotion detection models work. We evaluate the …


How optimal transport can tackle gender biases in multi-class neural-network classifiers for job recommendations?

F Jourdan, TT Kaninku, N Asher, JM Loubes, L Risser - arXiv preprint arXiv …, 2023

… Note that to build this dataset, its authors used Common Crawl and identified online biographies written in English. Then, they filtered the biographies starting with a name-like pattern followed by the string “is a(n) (xxx) title,” where title is an …


How Prevalent is Gender Bias in ChatGPT?--Exploring German and English ChatGPT Responses

S Urchs, V Thurner, M Aßenmacher, C Heumann… - arXiv preprint arXiv …, 2023

… It is unclear from the documentation on which data the system was trained exactly, but since it includes training data from CommonCrawl4 it is likely to reflect many of the biases and stereotypes common to internet content. Furthermore, the model is …


How to deploy security mechanisms online (consistently)

To mitigate a myriad of Web attacks, modern browsers support client-side security policies shipped through HTTP response headers. To enforce these policies, the operator can set response headers that the server then communicates to the client …


How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases

A Mueller, T Linzen - arXiv preprint arXiv:2305.19905, 2023

… All of these models are trained on approximately 34B words from the Colossal Cleaned Common Crawl (C4) web text corpus. … , which we included in the previous experiment, we also pre-train models on the Colossal Cleaned Common …


How User Language Affects Conflict Fatality Estimates in ChatGPT

D Kazenwadel, CV Steinert - arXiv preprint arXiv:2308.00072, 2023

OpenAI's ChatGPT language model has gained popularity as a powerful tool for complex problem-solving and information retrieval. However, concerns arise about the reproduction of biases present in the language-specific training data. In this study …


How well do language models understand grammar?: a case study on Japanese

Modern attention-based language models such as BERT and GPT have been shown to outperform previous state-of-the-art models on many NLP tasks. This performance implies a level of understanding of grammatical structures. This work …


HPLT: High Performance Language Technologies

M Aulamo, N Bogoychev, S Ji, G Nail… - Proceedings of the 24th …, 2023

We describe the High Performance Language Technologies project (HPLT), a 3-year EU-funded project started in September 2022. HPLT will build a space combining petabytes of natural language data with large-scale model training. It will derive …


HTTP header based phishing attack detection using machine learning

S Shukla, M Misra, G Varshney - Transactions on Emerging Telecommunications …

In the past, many techniques like blacklisting/whitelisting, third‐party, search engine, visual similarity, heuristic, URL features, and website content were used for anti‐phishing. Search engine‐based, third‐party assisted tools and blacklist/whitelist fail to identify …


HUAWEI NOAH'S ARK LAB AT TREC NEUCLIR 2022

E Kamalloo, D Alfonso-Hermelo, M Rezagholizadeh

… Both corpora are collected from the Common Crawl news collection. To identify documents in Chinese, Persian, and Russian, the language of documents were determined via automated language identification. HC4 comprises documents within …


HULAT at SemEval-2023 Task 10: Data augmentation for pre-trained transformers applied to the detection of sexism in social media

I Segura-Bedmar - arXiv preprint arXiv:2302.12840, 2023

This paper describes our participation in SemEval-2023 Task 10, whose goal is the detection of sexism in social media. We explore some of the most popular transformer models such as BERT, DistilBERT, RoBERTa, and XLNet. We also study …


HULAT at SemEval-2023 Task 9: Data augmentation for pre-trained transformers applied to Multilingual Tweet Intimacy Analysis

I Segura-Bedmar - arXiv preprint arXiv:2302.12794, 2023

This paper describes our participation in SemEval-2023 Task 9, Intimacy Analysis of Multilingual Tweets. We fine-tune some of the most popular transformer models with the training dataset and synthetic data generated by different data augmentation …


Human intelligence-based metaverse for co-learning of students and smart machines

CS Lee, MH Wang, M Reformat, SH Huang - Journal of Ambient Intelligence and …, 2023

This paper proposes a Human Intelligence (HI)-based Computational Intelligence (CI) and Artificial Intelligence (AI) Fuzzy Markup Language (CI&AI-FML) Metaverse as an educational environment for co-learning of students and machines. The HI-based …


Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitment

A Peña, I Serna, A Morales, J Fierrez, A Ortega… - arXiv preprint arXiv …, 2023

… In the study of demographic bias in NLP technologies,4 we can cite the Common Crawl Bios dataset [31], which contains nearly 400K short biographies collected from Common Crawl. The goal of the dataset is to predict the occupation from these …


Humanoid Robot as a Debate Partner

HS Yun, H Hübert, A Sardogan, N Pinkwart, VV Hafner… - International Conference on …, 2023

In this paper, we describe our design and development process of a humanoid robot, Pepper, as a debate partner by integrating open-source, offline Generative Pre-trained Transformer models, namely GPT-J 6B and BLOOM. We used our tool flow, which …


HunSum-1: an Abstractive Summarization Dataset for Hungarian

B Barta, D Lakatos, A Nagy, MK Nyist, J Ács - arXiv preprint arXiv:2302.00455, 2023

… In Section 3 we discuss in detail how we extracted the data from CommonCrawl and performed a number of preprocessing steps to arrive at the current version of HunSum-1. We also train three baseline models on the dataset and evaluate the results both …


HW-TSC's Neural Machine Translation System for CCMT 2023

Z Wu, Z Yu, Z Li, D Wei, Y Xie, X Chen, H Shang, J Guo… - China Conference on …, 2023

This paper presents Huawei Translation Service Center (HW-TSC)’s submission to the machine translation tasks of the 19th China Conference on Machine Translation (CCMT 2023). We participate in all machine translation tasks, including five bilingual …


HyperGraphDis: Leveraging Hypergraphs for Contextual and Social-Based Disinformation Detection

N Salamanos, P Leonidou, N Laoutaris, M Sirivianos… - 2023

In light of the growing impact of disinformation on social, economic, and political landscapes, accurate and efficient identification methods are increasingly critical. This paper introduces HyperGraphDis, a novel approach for detecting disinformation …


HYTREL: Hypergraph-enhanced Tabular Data Representation Learning

P Chen, S Sarkar, L Lausen, B Srinivasan, S Zha… - arXiv preprint arXiv …, 2023

… HYTREL pretrained with tables from Wikipedia and Common Crawl could not transfer well to the medical domain PMC dataset. As for the improvement observed from the contrastive objective, the reason could be that contrastive learning that uses …


I-WAS: a Data Augmentation Method with GPT-2 for Simile Detection

Y Chang, R Zhang, J Pu - arXiv preprint arXiv:2308.04109, 2023

… For the sentence completion stage, we apply the GPT-2 [35] model, which is trained on a large corpus of novels and common crawl and has 5 billion parameters. The model uses a transformer architecture and consists of a 45-layer decoder with …


ICON: A Linguistically-Motivated Large-Scale Benchmark Indonesian Constituency Treebank

ES Lim, WQ Leong, NT Nguyen, WM Kng, WC Tjhi… - ACM Transactions on Asian …, 2023

Constituency parsing is an important task of informing how words are combined to form sentences. While constituency parsing in English has seen significant progress in the last few years, tools for constituency parsing in Indonesian remain few and far …


Identification and Correction of Grammatical Errors in Ukrainian Texts Based on Machine Learning Technology

V Lytvyn, P Pukach, V Vysotska, M Vovk, N Kholodna - Mathematics, 2023

A machine learning model for correcting errors in Ukrainian texts has been developed. It was established that the neural network has the ability to correct simple sentences written in Ukrainian; however, the development of a full-fledged …


Identifying and Eliminating CSAM in Generative ML Training Data and Models

… Thisdataset was built by taking a snapshot of the Common Crawl5 repository, downloading images referenced in the HTML, reading the “alt” attributes of the images and using CLIP6 interrogation to discard images that did not sufficiently …


Identifying and Mitigating Privacy Risks Stemming from Language Models: A Survey

V Smith, AS Shamsabadi, C Ashurst, A Weller - arXiv preprint arXiv:2310.01424, 2023

Rapid advancements in language models (LMs) have led to their adoption across many sectors. Alongside the potential benefits, such models present a range of risks, including around privacy. In particular, as LMs have grown in size, the potential to …


Identifying and predicting stereotype change in large language corpora: 72 groups, 115 years (1900–2015), and four text sources.

TES Charlesworth, N Sanjeev, ML Hatzenbuehler… - Journal of Personality and …, 2023

… yearly corpus of New York Times (NYT) articles since 1990; and (d) although not diachronic or time-stamped, the most widely used of any corpus, the Common Crawl (CC), a massive corpus argued to reflect all internet text from the 2000s and before …


Identifying Duplicate Questions Leveraging Recurrent Neural Network

MB Baby, B Ankhari, M Shajalal, M Atabuzzaman… - Proceedings of the Fourth …, 2023

Community Question Answering (CQA) forums are the predominant platform where the users can respond to others’ questions and share acquainted insights. The influx of new questions with linguistic expression variability and ambiguity leads to a …


Idioms, Probing and Dangerous Things: Towards Structural Probing for Idiomaticity in Vector Space

F Klubička, V Nedumpozhimana, JD Kelleher - arXiv preprint arXiv:2304.14333, 2023

The goal of this paper is to learn more about how idiomatic information is structurally encoded in embeddings, using a structural probing method. We repurpose an existing English verbal multi-word expression (MWE) dataset to suit the probing …


If I Say the Word Out Loud, It Will Be More Real

… The training set for GPT-3 was a filtered version of the Common Crawl dataset, developed by training a classifier to pick out those documents most similar to the ones used in GPT-2’s training data, ie documents linked to from Reddit, plus …


Iimas-unam team entry: Transformers adapters for the sentiment analysis rest-mex 2023

EY Baez-Reyes, I Barrón-Jiménez, H Becerril-Pizarro… - IberLEF@ SEPLN, 2023

… The transformers model we used is the "xlm-roberta-base" model, which is pre-trained with 2.5 TB of data filtered by CommonCrawl. And as an adapter and fine-tuned an adapter. When training a pre-trained model of Huggingface, the first thing we did …


Image Synthesis from Themes Captured in Poems using Latent Diffusion Models

M Virmani, AM Michael, M Pathak, KS Pai, VRB Prasad - 2023 2nd International …, 2023

… Additionally, MUSS offers an approach that does not require labelled data and uses semantic sentence embeddings to mine paraphrase data from Common Crawl in any language. As a result, MUSS can shorten phrases in any language without …


Imitating Task and Motion Planning with Visuomotor Transformers

M Dalal, A Mandlekar, C Garrett, A Handa… - arXiv preprint arXiv …, 2023

… Large models at the scale of billions of parameters, trained on massive corpi such as Common Crawl, WebText2 [2], JFT-300M [3] and LAION-5B [4], exhibit powerful capabilities. These models can write coherently [2…


Impact and development of an Open Web Index for open web search

M Granitzer, S Voigt, NA Fathima, M Golasowski… - Journal of the Association …, 2023

Web search is a crucial technology for the digital economy. Dominated by a few gatekeepers focused on commercial success, however, web publishers have to optimize their content for these gatekeepers, resulting in a closed ecosystem of …


Impact of ChatGPT on Interdisciplinary Nursing Education and Research

H Miao, H Ahn - Asian/Pacific Island Nursing Journal, 2023

… To train GPT-3, which laid the foundation for ChatGPT, the 2016-2019 Common Crawl data set [6] of 45 TB of compressed plain text was used. Nowadays the data set used for training ChatGPT consists of more than 145 million dialogues scraped …


Impact of Transformers on Multilingual Fake News Detection for Tamil and Malayalam

RILA Hariharan, M Anand Kumar - Speech and Language Technologies for Low …, 2023

… XLM-RoBERTa is a multilingual RoBERTa model, an improved version of BERT, and pre-trained on 2.5 TB of filtered CommonCrawl data containing 100 languages. … They trained on monolingual data too, which are publicly available corpora from …


Implications of the Convergence of Language and Vision Model Geometries

J Li, Y Kementchedjhieva, A Søgaard - arXiv preprint arXiv:2302.06555, 2023

Large-scale pretrained language models (LMs) are said to ``lack the ability to connect [their] utterances to the world'' (Bender and Koller, 2020). If so, we would expect LM representations to be unrelated to representations in computer vision …


Implicit information extraction from news stories

… We used Common Crawl2 as a data source, as crawling live websites would be infeasible. For extraction, we developed a custom tool C’monCrawl3, which allows end-to-end extraction of Common Crawl data. We then deployed it in distributed …


Improvements in Language Modeling, Voice Activity Detection, and Lexicon in OpenASR21 Low Resource Languages

V Gupta, G Boulianne - International Conference on Speech and Computer, 2023

OpenASR21 evaluation was on 15 low resource languages and 3 case sensitive languages. During the evaluation, participants got significant reduction in word error rates (WER) with text downloaded from the internet for only the case sensitive …


Improving accuracy of GPT-3/4 results on biomedical data using a retrieval-augmented language model

D Soong, S Sridhar, H Si, JS Wagner, ACC Sá, CY Yu… - arXiv preprint arXiv …, 2023

… Popular LLMs with billions of parameters such as GPT-3 [4] , PaLM [15], OPT [16], and LLaMA [17] are typically trained on vast amounts of information collected from the Internet (eg the Common Crawl dataset [18]) and capture a diverse range of …


Improving Bug Severity Prediction with Domain-Specific Representation Learning

Y Wei, C Zhang, T Ren - IEEE Access, 2023

… Typically, these language models are trained on colossal datasets, such as Wikipedia or Common Crawl, which facilitates their understanding of the intricacies and complexities of natural language. Once trained, these models can be fine-tuned …


Improving Cascaded Systems in Spoken Language Processing

Spoken language processing encompasses a broad range of speech production and perception tasks. One of the central challenges in building spoken language systems is the lack of end-to-end training corpora. For example in spoken language …


Improving CLIP Training with Language Rewrites

L Fan, D Krishnan, P Isola, D Katabi, Y Tian - arXiv preprint arXiv:2305.20088, 2023

Contrastive Language-Image Pre-training (CLIP) stands as one of the most effective and scalable methods for training transferable vision models using paired image and text data. CLIP models are trained using contrastive loss, which typically relies …


Improving Domain-Specific Retrieval by NLI Fine-Tuning

R Dušek, A Wawer, C Galias, L Wojciechowska - arXiv preprint arXiv:2308.03103, 2023

The aim of this article is to investigate the fine-tuning potential of natural language inference (NLI) data to improve information retrieval and ranking. We demonstrate this for both English and Polish languages, using data from one of the largest Polish …


Improving Gender-Related Fairness in Sentence Encoders: A Semantics-Based Approach

T Dolci, F Azzalini, M Tanelli - Data Science and Engineering, 2023

… Finally, as underlying word-level representations, we select the biggest and more powerful GloVe model available, trained on Common Crawl 840B. Footnote 5 For the retraining, we used a Linux machine with Ubuntu 18.04, 78GB of RAM and a …


Improving Generalization of Adapter-Based Cross-lingual Transfer with Scheduled Unfreezing

CC Liu, J Pfeiffer, I Vulić, I Gurevych - arXiv preprint arXiv:2301.05487, 2023

Standard fine-tuning of language models typically performs well on in-distribution data, but suffers with generalization to distribution shifts. In this work, we aim to improve generalization of adapter-based cross-lingual task transfer where such …


Improving Log-Based Anomaly Detection by Pre-Training Hierarchical Transformers

S Huang, Y Liu, C Fung, H Wang, H Yang, Z Luan - IEEE Transactions on Computers, 2023

… For example, LogRobust [8] leverages word vectors pre-trained on the Common Crawl Corpus [10] dataset to extract the log event’s semantic information and transforms each log event into a semantic vector. It then utilizes an attention-based …


Improving Low-Resource Languages in Pre-Trained Multilingual Language Models

V Hangya, HS Saadi, A Fraser - Proceedings of the 2022 Conference on Empirical …, 2022

… However, multilingual LMs mainly focus on high resource languages, eg, mBERT supports the top 104 languages based on Wikipedia sizes, while XLM-R supports the top 100 based on CommonCrawl data. Additionally, many of these languages …


Improving Machine Translation in the E-commerce Luxury Space. A case study

JM De-la-Torre-Vilariño, JL García-Mendoza… - Proceedings of the 24th …, 2023

This case study presents a Multilingual e-commerce Project, which principal aim is to create an improved system that translates product titles and descriptions, plus other content in multiple languages. The project consisted of two main phases; a …


Improving Multiclass Classification of Fake News Using BERT-Based Models and ChatGPT-Augmented Data

E Shushkevich, M Alexandrov, J Cardiff - Inventions, 2023

Given the widespread accessibility of content creation and sharing, false information proliferation is a growing concern. Researchers typically tackle fake news detection (FND) in specific topics using binary classification. Our study addresses a more practical …


Improving Multimodal Datasets with Image Captioning

T Nguyen, SY Gadre, G Ilharco, S Oh, L Schmidt - arXiv preprint arXiv:2307.10350, 2023

… We also work with raw Common Crawl data instead of preprocessed datasets to study the trade-offs between raw and generated captions in a systematic manner. Finally, Gadre et al. [18] introduces DataComp, a benchmark for designing better pre-training …


Improving Natural Language Inference in Arabic using Transformer Models and Linguistically Informed Pre-Training

MMSA Deen, M Pielka, J Hees, BS Abdou, R Sifa - arXiv preprint arXiv:2307.14666, 2023

This paper addresses the classification of Arabic text data in the field of Natural Language Processing (NLP), with a particular focus on Natural Language Inference (NLI) and Contradiction Detection (CD). Arabic is considered a resource-poor language …


Improving network intrusion detection applying hybrid machine learning algorithms

K Čiurlienė, D Stankevičius - DAMSS 2022: 13th conference on data analysis …, 2022

DAMSS-2022 is the 13th International Conference on Data Analysis Methods for Software Systems, held in Druskininkai, Lithuania. Every year at the same place and time. The exception was in 2020, when the world was gripped by the Covid-19 …


Improving predictive models for rate of penetration in real drilling operations through transfer learning

FJ Pacis, A Ambrus, S Alyaev, R Khosravanian… - Journal of Computational …, 2023

The rate of penetration (ROP) is a key performance indicator in the oil and gas drilling industry as it directly translates to cost savings and emission reductions. A prerequisite for a drilling optimization algorithm is a predictive model that provides …


Improving Subword Tokenization Methods for Multilingual Models

… This unlabeled, multilingual dataset was created from the Common Crawl corpus using an automatic pipeline. The data was deduplicated and languageidentified. Then for each monolingual corpus the data was filtered using KneserNey language …


Improving Word Embedding Using Variational Dropout

Z Albujasim, D Inkpen, X Han, Y Guo - The International FLAIRS Conference …, 2023

Pre-trained word embeddings are essential in natural language processing (NLP). In recent years, many post-processing algorithms have been proposed to improve the pre-trained word embeddings. We present a novel method-Orthogonal Auto …


Improving Word Sense Disambiguation in Neural Machine Translation with Salient Document Context

E Rippeth, M Carpuat, K Duh, M Post - arXiv preprint arXiv:2311.15507, 2023

Lexical ambiguity is a challenging and pervasive problem in machine translation (\mt). We introduce a simple and scalable approach to resolve translation ambiguity by incorporating a small amount of extra-sentential context in neural \mt. Our approach …


In search of founding era registers: automatic modeling of registers from the corpus of Founding Era American English

L Repo, B Hashimoto, V Laippala - Digital Scholarship in the Humanities, 2023

Registers are situationally defined text varieties, such as letters, essays, or news articles, that are considered to be one of the most important predictors of linguistic variation. Often historical databases of language lack register information, which …


In What Languages are Generative Language Models the Most Formal? Analyzing Formality Distribution across Languages

A Ersoy, G Vizcarra, TT Mayeesha, B Muller - arXiv preprint arXiv:2302.12299, 2023

… BLOOM was trained from a collection of multiple sources such as Huggingface datasets, Github code, and Web Common Crawl. The data sources were then preprocessed to reduce non-natural language and anonymize personal identifiable …


In-Context Pretraining: Language Modeling Beyond Document Boundaries

W Shi, S Min, M Lomeli, C Zhou, M Li, V Lin, NA Smith… - arXiv preprint arXiv …, 2023

… For example, less than 5% of documents in CommonCrawl have longer than 2k tokens. In this work, we focus on constructing meaningful long-context data, making language models better leverage its context-window. Our sorted data can be used …


IN-ORDER TRANSITION-BASED PARSING FOR VIETNAMESE

J Bauer, H Bui, V Thai, C Manning - Journal of Computer Science and Cybernetics, 2023

In this paper, we implement a general neural constituency parser based on an in-order parser. We apply this parser to the VLSP 2022 Vietnamese treebank, obtaining a test score of. 8393 F1, top of the private test leaderboard. Earlier versions of the …


Incorporating Natural Language Processing into Virtual Assistants: An Intelligent Assessment Strategy for Enhancing Language Comprehension

F Antonius, PR Alapati, M Ritonga, I Patra… - International Journal of …, 2023

… The process starts with the selection of diverse pretraining datasets, including the multilingual Colossal Clean Common Crawl (mC4), CC100 dataset, and Wikipedia data. These datasets encompass various domains, languages, and tones …


IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages

J Gala, PA Chitale, R AK, S Doddapaneni, V Gumma… - arXiv preprint arXiv …, 2023

India has a rich linguistic landscape with languages from 4 major language families spoken by over a billion people. 22 of these languages are listed in the Constitution of India (referred to as scheduled languages) are the focus of this work. Given the …


IndiSocialFT: Multilingual Word Representation for Indian languages in code-mixed environment

S Kumar, R Sanasam, S Nandi - Findings of the Association for Computational …, 2023

The increasing number of Indian language users on the internet necessitates the development of Indian language technologies. In response to this demand, our paper presents a generalized representation vector for diverse text characteristics …


IndoToD: A Multi-Domain Indonesian Benchmark For End-to-End Task-Oriented Dialogue Systems

MDA Kautsar, RK Nurdini, S Cahyawijaya, GI Winata… - arXiv preprint arXiv …, 2023

… Specifically, we incorporate common crawl4 English and Indonesian fastText word vectors for monolingual English and Indonesian experiment settings respectively. Meanwhile, we use bilingual English-Indonesian word vectors (Conneau et al.…


Influence of the duration of training a deep neural network model on the quality of text summarization task

A Gryaznov, R Rybka, I Moloshnikov, A Selivanov… - AIP Conference …, 2023

In this paper we apply the generative deep learning language model to text summarization task. Because such large language models require a lot of resources to train, it is interesting to study how much long training affects the final result and at …


Information Access Using Neural Networks For Diverse Domains And Sources

The ever-increasing volume of web-based documents poses a challenge in efficiently accessing specialized knowledge from domain-specific sources, requiring a profound understanding of the domain and substantial comprehension effort …


Information Extraction and Knowledge Graph Development for Manufacturing Science Domain using Natural Language Processing.

KUMAR, AMAN. Information Extraction and Knowledge Graph Development for Manufacturing Science Domain using Natural Lang Page 1 ABSTRACT KUMAR, AMAN. Information Extraction and Knowledge Graph Development for …


Information Extraction from Polish Radiology Reports using Language Models

A Obuchowski, B Klaudel, P Jasik - Proceedings of the 9th Workshop on Slavic …, 2023

… All the models were pre-trained using a Polish subset of the Common Crawl corpus. The model’s pre-training details are shown in (Dadas … 2019) embeddings for the Polish language trained on the Polish part of the Common Crawl dataset …


Information resource management researchers' thinking about the opportunities and challenges of AIGC

Z ZHANG, Z Jianxun, XIA Cuijuan, W Dongbo… - Journal of Library and …, 2023

With the explosive popularity of ChatGPT and the development of AI generated content (AIGC), the new generation of artificial intelligence (AI) technology has triggered people's imagination and discussion about the production mode of digital …


infoVerse: A Universal Framework for Dataset Characterization with Multidimensional Meta-information

J Kim, Y Kim, K de Langis, J Shin, D Kang - arXiv preprint arXiv:2305.19344, 2023

The success of NLP systems often relies on the availability of large, high-quality datasets. However, not all samples in these datasets are equally valuable for learning, as some may be redundant or noisy. Several methods for characterizing …


INGENIOUS: Using Informative Data Subsets for Efficient Pre-Training of Large Language Models

HK Renduchintala, K Killamsetty, S Bhatia, M Aggarwal… - arXiv preprint arXiv …, 2023

A salient characteristic of large pre-trained language models (PTLMs) is a remarkable improvement in their generalization capability and emergence of new capabilities with increasing model capacity and pre-training dataset size …


Inria-ALMAnaCH at WMT 2022: Does Transcription Help Cross-Script Machine Translation?

J Alabi, L Nishimwe, B Muller, C Rey, B Sagot… - Proceedings of the Seventh …, 2022

This paper describes the Inria ALMAnaCH team submission to the WMT 2022 general translation shared task. Participating in the language directions {cs, ru, uk}→ en and cs↔ uk, we experiment with the use of a dedicated Latin-script transcription …


Insincere Questions Classification Using CNN with Increased Vocabulary Coverage of GloVe Embedding

S Mishra, N Kumar - Journal of The Institution of Engineers (India): Series B, 2023

… word embedding used in training the model is the GloVe word embedding from Stanford NLP [9], the GloVe stands for “Global Vectors for Word Representation”, it is a pre-trained embedding and has different options, so for this study, the authors …


Instance-Based Domain Adaptation for Improving Terminology Translation

P Nayak, J Kelleher, R Haque, A Way - … of Machine Translation Summit XIX, Vol. 1 …, 2023

Terms are essential indicators of a domain, and domain term translation is dealt with priority in any translation workflow. Translation service providers who use machine translation (MT) expect term translation to be unambiguous and consistent with the …


InstructIE: A Chinese Instruction-based Information Extraction Dataset

H Gui, J Zhang, H Ye, N Zhang - arXiv preprint arXiv:2305.11527, 2023

We introduce a new Information Extraction (IE) task dubbed Instruction-based IE, which aims to ask the system to follow specific instructions or guidelines to extract information. To facilitate research in this area, we construct a dataset called InstructIE …


Instruction-tuning Aligns LLMs to the Human Brain

KL Aw, S Montariol, B AlKhamissi, M Schrimpf… - arXiv preprint arXiv …, 2023

… T5 models are encoder-decoder LLMs pretrained on the Colossal Common Crawl Corpus (C4), a corpus of 356 billion tokens, using a masked infilling objective, and then further finetuned on multi-task mixture of unsupervised and supervised …


InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining

B Wang, W Ping, L McAfee, P Xu, B Li, M Shoeybi… - arXiv preprint arXiv …, 2023

Pretraining auto-regressive large language models (LLMs) with retrieval demonstrates better perplexity and factual accuracy by leveraging external databases. However, the size of existing pretrained retrieval-augmented LLM is still …


Integrating artificial intelligence-based methods into qualitative research in physics education research: A case for computational grounded theory

P Tschisgale, P Wulff, M Kubsch - Physical Review Physics Education Research, 2023

[This paper is part of the Focused Collection on Qualitative Methods in PER: A Critical Examination.] Qualitative research methods have provided key insights in physics education research (PER) by drawing on non-numerical data such as text or …


Integrating Offline Reinforcement Learning with Transformers for Sequential Recommendation

X Xi, Y Zhao, Q Liu, L Ouyang, Y Wu - arXiv preprint arXiv:2307.14450, 2023

We consider the problem of sequential recommendation, where the current recommendation is made based on past interactions. This recommendation task requires efficient processing of the sequential data and aims to provide …


Intelligent Phishing Website Detection before and after Multiple Informative Feature Selection Techniques: Machine Learning Approach

K Adane, B Beyene, M Abebe - International Journal of Information, 2024

Individuals and Organizations that rely on the Internet for communication, collaboration, and daily tasks regularly encounter security and privacy issues unless interventions of intelligent Cybersecurity defense systems have been made to …


Intent Detection in Urdu Queries using Fine-tuned BERT models

S Shams, B Sadia, M Aslam - 2022 16th International Conference on Open Source …, 2022

User's intent detection provides essential cues in query understanding and accurate information retrieval through search engines and task-oriented dialogue systems. Intent detection from user queries is challenging due to short query length and lack …


Interactive Natural Language Processing

Z Wang, G Zhang, K Yang, N Shi, W Zhou, S Hao… - arXiv preprint arXiv …, 2023

… 2008), PubMed Corpus4, and CommonCrawl Corpus5, among others. Most previous research has focused on corpus knowledge due to its controllability and efficiency. RetrievalAugmented Language Models (Guu et … 4https://pubmed.ncbi.nlm.nih.gov …


Interactive writing systems and why small (er) could be more beautiful

ABSTRACT Machine learning models can support human creativity, including tasks such as writing. This position paper explorers and critiques the design of writing systems based on big data. The paper proposes small data as way to think about …


Intercity networks and urban performance: a geographical text mining approach

W Tongjing, E Meijers, Z Bao, H Wang - International Journal of Urban Sciences, 2023

… While it is possible to search for toponym occurrences using a search engine like google, results returned are vulnerable to bias (Meijers & Peris, Citation2019), and like Meijers and Peris we prefer using corpora from the CommonCrawl Archive of …


Interconnecting Advanced Networks with AI Applications

A Luntovskyy - Preprints of Position Papers of the 18th Conference on …, 2023

… org as a text’s source can also be specified as the so-called C4 (Common Crawl Cultural Context). Furthermore, the sources of texts can be differentiated as follows: search engines, online forums and communities, social media, blogs, newspapers …


INTERPRETING THE RELEVANCE OF READABILITY PREDICTION FEATURES

S Berrichi, N Nassiri, A Mazroui, A Lakhouaja - Jordanian Journal of Computers and …, 2023

… from “Common Crawl” and a dump of Arabic Wikipedia totaling 8.2 billion words. Different pretrained BERT models are used. We have selected in this study, the bert-base-Arabic model. 3) XLM-R [29]: is a multi-lingual version of BERT model, trained on “Common …


Into the LAION's Den: Investigating Hate in Multimodal Datasets

A Birhane, S Han, V Boddeti, S Luccioni - Thirty-seventh Conference on Neural …, 2023

… -2B-en are extracted from the Common Crawl dataset, we hypothesize that during the race to expand the dataset to 5 billion samples, the dataset scraping module might have sampled from the lower-quality sub-parts of the Common Crawl at a rate …


Intriguing Properties of Quantization at Scale

A Ahmadian, S Dash, H Chen, B Venkitesh, S Gou… - arXiv preprint arXiv …, 2023

Emergent properties have been widely adopted as a term to describe behavior not present in smaller models but observed in larger models. Recent work suggests that the trade-off incurred by quantization is also an emergent property, with sharp drops …


Introducing UberText 2.0: a corpus of modern Ukrainian at scale

D Chaplynskyi - Proceedings of the Second Ukrainian Natural …, 2023

This paper addresses the need for massive corpora for a low-resource language and presents the publicly available UberText 2.0 corpus for the Ukrainian language and discusses the methodology of its construction. While the collection and …


Introduction to ChatGPT: A new revolution of artificial intelligence with machine learning algorithms and cybersecurity

ChatGPTis an innovative natural language processing (NLP) technology developed by OpenAI. It has revolutionized humancomputer interactions by utilizing Machine Learning (ML) and Deep Learning (DL) algorithms. ChatGPT generates a …


Investigating Biases in Rules Extracted from Language Models

We investigate an approach for extracting occupational gender bias in the form of logical rules from Large Language Models (LLM)s based on Angluin's exact learning model with membership and equivalence queries to an oracle. In our …


Investigating parallelograms inside word embedding space using various analogy test sets in various languages

The idea of using analogy to assess the quality of word embedding spaces implies the existence of parallelograms between the four terms of an analogy. We investigate the presence of analogy parallelograms in various word embedding …


Investigating the Extent and Usability of Webtext Available in South Africa's Official Languages

F de Wet, R Eiselen, E Schillack, M Puttkammer - Southern African Conference for …, 2023

… The Common Crawl (CC) data set is likely the largest text data set currently available consisting of (bi-)monthly crawls of the web, and is … Part of the common crawl collection process is language identification to categorise the predominant …


Investigating Vision Foundational Models for Tactile Representation Learning

B Zandonati, R Wang, R Gao, Y Wu - arXiv preprint arXiv:2305.00596, 2023

… In particular, foundational models [11] are trained on massive datasets such as ImageNet [12] and CommonCrawl [13] to derive general representational knowledge, which can be specialized to diverse downstream tasks, such as semantic …


Investigation of quantum-inspired modelling in interactive search based on information foraging theory

This thesis investigates the use of the mathematical formalism of quantum mechanics for modelling users’ information needs from the viewpoint of Information Foraging Theory (IFT). IFT has been successfully applied to model user behaviours …


IOL Research Machine Translation Systems for WMT23 General Machine Translation Shared Task

W Zhang, Z Yan, Q Deng, J Cai, H Mao - Proceedings of the Eighth Conference on …, 2023

This paper describes the IOL Research team’s submission systems for the WMT23 general machine translation shared task. We participated in two language translation directions, including English→ Chinese and Chinese→ English. Our final …


IPARO: INTERPLANETARY ARCHIVAL RECORD OBJECT FOR DECENTRALIZED WEB ARCHIVING AND REPLAY

We proposed a decentralized version tracking system using the existing primitives of IPFS and IPNS. While our description talks primarily about archived web pages, we proposed the concept of IPMT and namespacing so that it can be used in other …


Irreducible Curriculum for Language Model Pretraining

S Fan, M Jaggi - arXiv preprint arXiv:2310.15389, 2023

… 5, IRREDUCIBLE CURRICULUM reduce the validation perplexity on 4 domains (C4, CommonCrawl, Book and Wikipedia), while showing the … Figure 4 shows that most of the globally top-ranked samples are from Wikipedia, C4 and CommonCrawl …


Is GPT-3 all you need for low-data discovery in chemistry?

KM Jablonka, P Schwaller, A Ortega-Guerrero, B Smit - 2023

… The version of GPT-3 we utilized in this work has been trained on data up to Oct 2019 that mostly comes from web scraping (Common Crawl47 and WebText48) along with books corpora and Wikipedia. Structured datasets, however, have not …


IS IT POSSIBLE TO RE-EDUCATE ROBERTA? EXPERT-DRIVEN MACHINE LEARNING FOR PUNCTUATION CORRECTION

JMH ŽIŽKOVÁ, AFJAN ŠVEC - JAZYKOVEDNÝ ČASOPIS

… As the training data set used for fine-tuning, we used 10 GB of raw text extracted from the Czech CommonCrawl data set. Because RoBERTa’s output is related to input tokens (not words), we assigned the target label (‟,” for comma,“0” for no …


Is More Always Better? Testing the Addition Bias for German Language Statistics

This replication study aims to investigate a potential bias toward addition in the German language, building upon previous findings of Winter and colleagues who identified a similar bias in English. Our results confirm a bias in word frequencies …


Issues Surrounding the Use of ChatGPT in Similar Languages: The Case of Malay and Indonesian

We report a problem that one faces when using ChatGPT in similar languages, taking Malay and Indonesian as examples: ChatGPT often responds to prompts in Malay (the language with fewer speakers) in Indonesian (the language with more …


Its All Graph To Me: Foundational Topology Models with Contrastive Learning on Multiple Domains

AO Davies, R Green, NS Ajmeri, TM Silva Filho - arXiv preprint arXiv:2311.03976, 2023

Abstract Representations and embeddings of graph data have been essential in many domains of research. The principle benefit of learning such representations is that the pretrained model can be fine-tuned on smaller datasets where data or labels …


IussNets at DisCo-Tex: A fine-tuned approach to coherence

E Zanoli, M Barbini, C Chesi - Proceedings of the Eighth Evaluation Campaign of …, 2023

We present our submission to the DisCoTex shared task of the EVALITA 2023 evaluation campaign, which focuses on modeling discourse coherence for Italian texts. We highlight the importance of coherence modeling in natural language …


Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models

N Sengupta, SK Sahu, B Jia, S Katipomu, H Li, F Koto… - arXiv preprint arXiv …, 2023

… • Pile-CC: A subset of The Pile dataset, derived from the Common Crawl, a collection of website crawls from 2008 onwards. The dataset includes raw web pages, metadata, and text extractions from diverse domains. Due to the varying …


Jake Glendenning

IIM LEARNING - AIPLA QUARTERLY JOURNAL, 2023

… sentence and understand that machine learning models like GPT-3 don't "read" like humans do, but rather ingest data sets like Common Crawl, which is made up of many written works … 2020) (explaining the use of the Common Crawl dataset in …


Japanese SimCSE Technical Report

H Tsukagoshi, R Sasano, K Takeda - arXiv preprint arXiv:2310.19349, 2023

We report the development of Japanese SimCSE, Japanese sentence embedding models fine-tuned with SimCSE. Since there is a lack of sentence embedding models for Japanese that can be used as a baseline in sentence embedding …


JCoLA: Japanese Corpus of Linguistic Acceptability

T Someya, Y Sugimoto, Y Oseki - arXiv preprint arXiv:2309.12676, 2023

Neural language models have exhibited outstanding performance in a range of downstream tasks. However, there is limited understanding regarding the extent to which these models internalize syntactic knowledge, so that various datasets have …


Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents

M Günther, J Ong, I Mohr, A Abdessalem, T Abel… - arXiv preprint arXiv …, 2023

Text embedding models have emerged as powerful tools for transforming sentences into fixed-sized feature vectors that encapsulate semantic information. While these models are essential for tasks like information retrieval, semantic clustering, and text …


KASYS at the TREC 2022 NeuCLIR Track

This paper describes the KASYS team’s participation in the TREC 2022 NeuCLIR track. Our approach is One-for-All, which employs a single multilingual pre-trained language model to retrieve documents of any languages in response to an English …


KaustubhSharedTask@ LT-EDI 2023: Homophobia-Transphobia Detection in Social Media Comments with NLPAUG-driven Data Augmentation

K Lande, R Ponnusamy, PK Kumaresan… - Proceedings of the Third …, 2023

Our research in Natural Language Processing (NLP) aims to detect hate speech comments specifically targeted at the LGBTQ+ community within the YouTube platform shared task conducted by LTEDI workshop. The dataset provided by the …


Keyword Embeddings for Query Suggestion

J Gabín, ME Ares, J Parapar - arXiv preprint arXiv:2301.08006, 2023

Nowadays, search engine users commonly rely on query suggestions to improve their initial inputs. Current systems are very good at recommending lexical adaptations or spelling corrections to users' queries. However, they often struggle to …


Know Where to Go: Make LLM a Relevant, Responsible, and Trustworthy Searcher

X Shi, J Liu, Y Liu, Q Cheng, W Lu - arXiv preprint arXiv:2310.12443, 2023

The advent of Large Language Models (LLMs) has shown the potential to improve relevance and provide direct answers in web searches. However, challenges arise in validating the reliability of generated results and the credibility of contributing …


Knowledge and Pre-trained Language Models: a deep-dive into datasets and external knowledge

Recent years have witnessed the emergence of Pre-trained Language Models (PLMs), such as ELMo, GPT, BERT, XLNet, GPT-3 and InstructGPT [11, 12, 13, 14, 15, 16, 17, 18, 19], which have been widely used in many NLP tasks and have shown superior …


Knowledge Informed Fake News Detection Using Large Language Models

The spread of false or misleading information as news has been a significant threat to governments, organizations and the economy for a long time. However, it has become more prevalent and influential in recent years due to the growing popularity …


Knowledge of cultural moral norms in large language models

A Ramezani, Y Xu - arXiv preprint arXiv:2306.01857, 2023

Moral norms vary across cultures. A recent line of work suggests that English large language models contain human-like moral biases, but these studies typically do not examine moral variation in a diverse cultural setting. We investigate the extent to …


Knowledge-Based and Generative-AI-Driven Pedagogical Conversational Agents: A Comparative Study of Grice's Cooperative Principles and Trust

M Wölfel, MB Shirzad, A Reich, K Anderer - Big Data and Cognitive Computing, 2023

The emergence of generative language models (GLMs), such as OpenAI’s ChatGPT, is changing the way we communicate with computers and has a major impact on the educational landscape. While GLMs have great potential to support education, their …


KoBigBird-large: Transformation of Transformer for Korean Language Understanding

K Yang, Y Jang, T Lee, J Seong, H Lee, H Jang, H Lim - arXiv preprint arXiv …, 2023

This work presents KoBigBird-large, a large size of Korean BigBird that achieves state-of-the-art performance and allows long sequence processing for Korean language understanding. Without further pretraining, we only transform the …


Kosmos-2.5: A Multimodal Literate Model

T Lv, Y Huang, J Chen, L Cui, S Ma, Y Chang, S Huang… - arXiv preprint arXiv …, 2023

We present Kosmos-2.5, a multimodal literate model for machine reading of text-intensive images. Pre-trained on large-scale text-intensive images, Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1) generating spatially-aware text blocks …


Kosmos-G: Generating Images in Context with Multimodal Large Language Models

X Pan, L Dong, S Huang, Z Peng, W Chen, F Wei - arXiv preprint arXiv:2310.02992, 2023

Recent advancements in text-to-image (T2I) and vision-language-to-image (VL2I) generation have made significant strides. However, the generation from generalized vision-language inputs, especially involving multiple images, remains under-explored …


KTRL+ F: Knowledge-Augmented In-Document Search

H Oh, H Shin, M Ko, H Lee, M Seo - arXiv preprint arXiv:2311.08329, 2023

We introduce a new problem KTRL+F, a knowledge-augmented in-document search task that necessitates real-time identification of all semantic targets within a document with the awareness of external sources through a single natural query …


Lagniappe: Exploring the Pros and Cons of ChatGPT

J Daugherty - North Carolina Libraries, 2023

… It trained on a dataset taken from Common Crawl, a nonprofit that scrapes text from websites and offers it for free download, Wikipedia, and other web texts, all from 2021.The ChatGPT Plus paid subscription offered web searching capabilities with a …


Language Aligned Visual Representations Predict Human Behavior in Naturalistic Learning Tasks

C Demircan, T Saanum, L Pettini, M Binz… - arXiv preprint arXiv …, 2023

Humans possess the ability to identify and generalize relevant features of natural objects, which aids them in various situations. To investigate this phenomenon and determine the most effective representations for predicting human behavior, we …


Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization

A Chronopoulou, J Pfeiffer, J Maynez, X Wang… - arXiv preprint arXiv …, 2023

… This corpus has been created using a Common Crawl-based dataset covering 101 languages. All languages considered in our experiments are covered by mC4. For the language vectors, we finetune the LLM using LoRA on prefix-LM for only 5k …


Language Is Not All You Need: Aligning Perception with Language Models

S Huang, L Dong, W Wang, Y Hao, S Singhal, S Ma… - arXiv preprint arXiv …, 2023

… We also include the Common Crawl snapshots (2020-50 and 2021-04) datasets, CC-Stories… are collected from web pages of the Common Crawl web data by extracting image sources … Interleaved Image-Text Data We collect interleaved …


Language Models and Power Laws

… Training data: Common Crawl (410 bln, 60%), WebText2 (19 bln, 22%), books (67 bln, 16%), Wikipedia (3 bln, 3%). …


Language Models are Drummers: Drum Composition with Natural Language Pre-Training

L Zhang, C Callison-Burch - arXiv preprint arXiv:2301.01162, 2023

Automatic music generation with artificial intelligence typically requires a large amount of data which is hard to obtain for many less common genres and musical instruments. To tackle this issue, we present ongoing work and preliminary findings …


Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-training

M Anschütz, J Oehms, T Wimmer, B Jezierski, G Groh - arXiv preprint arXiv …, 2023

Automatic text simplification systems help to reduce textual information barriers on the internet. However, for languages other than English, only few parallel data to train these systems exists. We propose a two-step approach to overcome this data …


Language Report French

G Adda, I Vasilescu, F Yvon - European Language Equality: A Strategic Agenda for …, 2023

… The CommonCrawl project aggregates Web data that is orders of magnitude larger than these resources; and it is updated on a regular basis. Using French subsets of CommonCrawl, it has been possible to train large language models (LMs) …


Language Report Slovenian

S Krek - European Language Equality: A Strategic Agenda for …, 2023

Around 2.5 million people around the world speak or understand Slovene, with the vast majority of them living in the Republic of Slovenia where it is the official language. The constitution grants the right to use their mother tongue to Italian and …


Language Resources for Dutch Large Language Modelling

B Vanroy - arXiv preprint arXiv:2312.12852, 2023

Despite the rapid expansion of types of large language models, there remains a notable gap in models specifically designed for the Dutch language. This gap is not only a shortage in terms of pretrained Dutch models but also in terms of data, and …


Language Variety Identification with True Labels

M Zampieri, K North, T Jauhiainen, M Felice, N Kumari… - arXiv preprint arXiv …, 2023

Language identification is an important first step in many IR and NLP applications. Most publicly available language identification datasets, however, are compiled under the assumption that the gold label of each instance is determined by where …


Language-Aware Spatial-Temporal Collaboration for Referring Video Segmentation

T Hui, S Liu, Z Ding, S Huang, G Li, W Wang, L Liu…

… The GloVe word embedding [66] pretrained on the Common Crawl with 840B tokens is used to embed input words. The maximum sequence length of the input referring expression is set as 20. The height and width of the input frames are …


Language-Family Adapters for Low-Resource Multilingual Neural Machine Translation

A Chronopoulou, D Stojanovski, A Fraser - Proceedings of the The Sixth Workshop …, 2023

Large multilingual models trained with self-supervision achieve state-of-the-art results in a wide range of natural language processing tasks. Self-supervised pretrained models are often fine-tuned on parallel data from one or multiple …


Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior

A Khandelwal, A Agrawal, A Bhattacharyya, YK Singla… - arXiv preprint arXiv …, 2023

… Consider the Common Crawl project1, one common source of data included in most language models; it produces more than 20TB of text … Similarly, while a lot of public blogs and user forums are included in major text corpora used in training …


Large Data-to-Text Generation

This thesis presents a domain-driven approach to sports game summarization, a specific instance of large data-to-text generation (DTG). We first address the data fidelity issue in the Rotowire dataset by supplementing existing input records and …


Large Graph Models: A Perspective

Z Zhang, H Li, Z Zhang, Y Qin, X Wang, W Zhu - arXiv preprint arXiv:2308.14522, 2023

… This massive amount of data for NLP and CV tasks is typically sourced from publicly accessible human-generated content, such as web pages in CommonCrawl or user-posted photos in social media, which are easily collected from the web. …


Large Language Models (LLM): Need, Methods, And Research Trends

… Pre-training and fine-tuning: This technique involves pre-training a large model on a massive dataset, such as Wikipedia or Common Crawl, and then fine-tuning the model on a specific task, such as sentiment analysis or question answering. This …


Large Language Models and Control Mechanisms Improve Text Readability of Biomedical Abstracts

Z Li, S Belkadi, N Micheletti, L Han, M Shardlow… - arXiv preprint arXiv …, 2023

… The authors used the common crawl corpus and filtered to keep only natural text and de-duplication processing. They extracted 750GB of clean English data to feed into the model for multi-task pre-training. Different masking strategies are integrated …


Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset

S Alavi Naeini, R Saqur, M Saeidi, J Giorgi, B Taati - Advances in Neural Information …, 2024

S Naeini, R Saqur, M Saeidi, J Giorgi, B Taati - arXiv preprint arXiv:2306.11167, 2023

… We used two FastText models, one pre-trained on the Common Crawl corpus and another on Wikipedia. Approximately 10% of the total clues encountered in the dataset were out-of-vocabulary (OOV). A significant portion (~80%) of the OOV …


Large Language Models Can Be Used to Estimate the Ideologies of Politicians in a Zero-Shot Learning Setting

PY Wu, JA Tucker, J Nagler, S Messing - arXiv preprint arXiv:2303.12057, 2023

The mass aggregation of knowledge embedded in large language models (LLMs) holds the promise of new solutions to problems of observability and measurement in the social sciences. We examine the utility of one such model for a particularly …


Large Language Models Can Be Used to Estimate the Latent Positions of Politicians

Existing approaches to estimating politicians’ latent positions along specific dimensions often fail when relevant data is limited. We leverage the embedded knowledge in generative large language models (LLMs) to address this challenge …


Large language models in medicine

AJ Thirunavukarasu, DSJ Ting, K Elangovan… - Nature Medicine, 2023

Large language models (LLMs) can respond to free-text queries without being specifically trained in the task in question, causing excitement and concern about their use in healthcare settings. ChatGPT is a generative artificial intelligence (AI) …


Large Language Models Need Symbolic AI

… GPT-3 was trained on an extensive dataset, based on a version of the CommonCrawl dataset (with almost a trillion words) and additional reference sources. Given tasks and few-shot demonstrations provided to the system as text …


Large Language Models

M McTear, M Ashurkina - Transforming Conversational AI: Exploring the Power …, 2024

MR Douglas - arXiv preprint arXiv:2307.05782, 2023

… The model was trained on the Common Crawl, a dataset of billions of words from web pages, and the Bookcorpus dataset, consisting of … The model was trained on a diverse dataset, including Common Crawl and WebText. GPT-2 was able to …


Large Language Models' Understanding of Math: Source Criticism and Extrapolation

R Yousefzadeh, X Cao - arXiv preprint arXiv:2311.07618, 2023

… Common Crawl is particularly interesting. The GPT-f model developed for mathematical learning was trained on 300 billion tokens from CommonCrawl… The size of the most recent CommonCrawl is 390 TiB including the contents of 3.1 billion …


Large Language Models, scientific knowledge and factuality: A systematic analysis in antibiotic discovery

M Wysocka, O Wysocki, M Delmas, V Mutel, A Freitas - arXiv preprint arXiv …, 2023

Inferring over and extracting information from Large Language Models (LLMs) trained on a large corpus of scientific literature can potentially drive a new era in biomedical research, reducing the barriers for accessing existing medical evidence …


Large Scale Fine-Tuned Transformers Models Application for Business Names Generation

M Lukauskas, T Rasymas, M Minelga, D Vaitmonas - Computing and Informatics, 2023

… on larger datasets, leading to pre-trained systems such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer), which have been trained on large language datasets such as the …


Large Sentiment Dictionary of Russian Words

VV Bochkarev, AA Achkeev, AV Savinkov… - … International Conference on …, 2023

… Vector sets for 157 languages, trained on the Common Crawl corpus using the most advanced word embedding algorithms at the time, were made publicly available in 2018. The great advantage of this dataset in terms of the tasks set in our …


Large Web Archive Collection Infrastructure and Services

… We use Common Crawl’s web archiving data crawled from May 20 to 23, 2018. The data set consists of 1219 Gzip compressed WARC files totaling 0.98 TB, and contains 53,324,440 records. The WARC files are organized by crawling time, each …


Large-scale Security Analysis of HTTP Responses

… In order to find vulnerabilities on a large scale, this thesis utilizes the crawl data from Common Crawl, an open repository of web crawl data. In this data, indicators of specific programs, software, or libraries are visible, allowing us to find vulnerable …


LASTING IMPACT: WHY YOU SHOULD CARE ABOUT COURT OF CUSTOMS AND PATENT APPEAL CASES

… sentence and understand that machine learning models like GPT-3 don't "read" like humans do, but rather ingest data sets like Common Crawl, which is made up of many written works … 2020) (explaining the use of the Common Crawl dataset in …


Lawyer LLaMA Technical Report

Q Huang, M Tao, Z An, C Zhang, C Jiang, Z Chen… - arXiv preprint arXiv …, 2023

… On one hand, current LLMs are primarily trained on general corpora such as Common Crawl and Wikipedia, with limited exposure to domainspecific resources. Therefore, they lack the necessary knowledge required for specific domains. On the …


LCTs at HODI: Homotransphobic Speech Detection on Italian Tweets

D Locatelli, L Locatelli - Proceedings of the Eighth Evaluation Campaign of …, 2023

… -Commoncrawl-Cased version. Using the HuggingFace Transformers library [11], we applied a classification head on top of the model outputs, which enabled us to fine-tune the base model on the HODI data for Subtask A. The selection of the …


Learning Human-Human Interactions in Images from Weak Textual Supervision

M Alper, H Averbuch-Elor - arXiv preprint arXiv:2304.14104, 2023

Interactions between humans are diverse and context-dependent, but previous works have treated them as categorical, disregarding the heavy tail of possible interactions. We propose a new paradigm of learning human-human interactions as …


Learning Language-Specific Layers for Multilingual Machine Translation

TP Pires, RM Schmidt, YH Liao, S Peitz - arXiv preprint arXiv:2305.02665, 2023

Multilingual Machine Translation promises to improve translation quality between non-English languages. This is advantageous for several reasons, namely lower latency (no need to translate twice), and reduced error cascades (eg, avoiding …


Learning word embeddings for Ukrainian: A comparative study of fastText hyperparameters

N Romanyshyn, D Chaplynskyi, K Zakharov - Proceedings of the Second Ukrainian …, 2023

… noisy data from the Common Crawl (CC) project4 for learning their word vectors. Another shortcoming is that they did not optimize subword … It was a deliberate decision not to include Common Crawl or Oscar13 corpora data into UberText because …


Legal Governance of Internet Information Content in AIGC Era Taking Large Language Model as an Example

LI Mingxuan, WEN Jirong - Journal of Beijing Institute of Technology (Social …, 2023

… 以 GPT-3,PaLM 和 LLaMa 这三个具有代表性的大语言模型为例,GPT-3 的预训练 数据包含 Common Crawl,WebText 2 和维基百科,PaLM 的预训练数据包含社交媒体 对话,过滤后的网页,Github, 多语言维基百科和新闻,LLaMa 的预训练数据包含 Common …


Lemur: Harmonizing Natural Language and Code for Language Agents

Y Xu, H Su, C Xing, B Mi, Q Liu, W Shi, B Hui, F Zhou… - arXiv preprint arXiv …, 2023

We introduce Lemur and Lemur-Chat, openly accessible language models optimized for both natural language and coding capabilities to serve as the backbone of versatile language agents. The evolution from language chat models to …


Let's Chat to Find the APIs: Connecting Human, LLM and Knowledge Graph through AI Chain

Q Huang, Z Wan, Z Xing, C Wang, J Chen, X Xu, Q Lu - arXiv preprint arXiv …, 2023

… an analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


Let's Discover More API Relations: A Large Language Model-based AI Chain for Unsupervised API Relation Inference

Q Huang, Y Sun, Z Xing, Y Cao, J Chen, X Xu, H Jin… - arXiv preprint arXiv …, 2023

… are pre-trained on the corpus of the entire Internet (Common Crawl [11]), which are referred to as foundation models [12]. As such foundation LLMs pack the knowledge of the entire Web, we are no longer limited to the scope of API text corpus …


Leveraging Auxiliary Domain Parallel Data in Intermediate Task Fine-tuning for Low-resource Translation

S Nayak, S Ranathunga, S Thillainathan, R Hung… - arXiv preprint arXiv …, 2023

NMT systems trained on Pre-trained Multilingual Sequence-Sequence (PMSS) models flounder when sufficient amounts of parallel data is not available for fine-tuning. This specifically holds for languages missing/under-represented in these models …


Leveraging Closed-Access Multilingual Embedding for Automatic Sentence Alignment in Low Resource Languages

I Abdulmumin, AA Khalid, SH Muhammad, IS Ahmad… - arXiv preprint arXiv …, 2023

… Using a similarity metric in that multimodal embedding space, they performed mining of audio in German, French, Spanish, and English from Librivox against billions of sentences from Common Crawl. This yielded more than twenty thousand …


Leveraging Diffusion Perturbations for Measuring Fairness in Computer Vision

N Lui, B Chia, W Berrios, C Ross, D Kiela - arXiv preprint arXiv:2311.15108, 2023

Computer vision models have been known to encode harmful biases, leading to the potentially unfair treatment of historically marginalized groups, such as people of color. However, there remains a lack of datasets balanced along demographic traits …


Leveraging Domain Adaptation and Data Augmentation to Improve Qur'anic IR in English and Arabic

V Pavlova - arXiv preprint arXiv:2312.02803, 2023

In this work, we approach the problem of Qur'anic information retrieval (IR) in Arabic and English. Using the latest state-of-the-art methods in neural IR, we research what helps to tackle this task more efficiently. Training retrieval models requires a lot of …


Leveraging Feature Selection to Improve the Accuracy for Malware Detection

Malware is becoming increasingly sophisticated and difficult to detect with traditional monitoring tools and antivirus software. As a result, machine learning has become a popular approach for classifying and detecting malware-related data. In this study …


Leveraging Function Space Aggregation for Federated Learning at Scale

N Dhawan, N Mitchell, Z Charles, Z Garrett… - arXiv preprint arXiv …, 2023

… C4 and CC-News are hosted by commoncrawl.org and we access both through HuggingFace datasets. … C4 The Colossal Clean Crawled Corpus (C4) dataset is a cleaned version of Common Crawl’s web crawl corpus (Raffel et al., 2020). We use …


Leveraging Image-Text Similarity and Caption Modification for the DataComp Challenge: Filtering Track and BYOD Track

S Yokoo, P Zhu, Y Ishikawa, M Tanaka, M Kondo… - arXiv preprint arXiv …, 2023

… DataComp provides a new candidate pool of 12.8B image-text pairs collected from Common Crawl. This challenge contains four distinct compute and data scales. In this work, we mainly focus on the small scale (12.8M samples). …


Leveraging Prior Knowledge and Structure for Data-Efficient Machine Learning

Building high-performing end-to-end machine learning systems primarily consists of developing the machine learning model and gathering high-quality training data for the application of interest, assuming one has access to the right hardware. Although …


Leveraging supplementary text data to kick-start automatic speech recognition system development with limited transcriptions

N San, M Bartelds, B Billings, E de Falco, H Feriza… - arXiv preprint arXiv …, 2023

Recent research using pre-trained transformer models suggests that just 10 minutes of transcribed speech may be enough to fine-tune such a model for automatic speech recognition (ASR) -- at least if we can also leverage vast amounts of text …


LFWE: L inguistic F eature Based W ord E mbedding for Hindi Fake News Detection

R Sharma, A Arya - ACM Transactions on Asian and Low-Resource …, 2023

It is essential for the research communities to investigate ways for authenticating news. The use of linguistic feature-based analysis to automatically detect false news is gaining popularity among the scientific community. However, such techniques are …


LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models

A Meister, M Novikov, N Karpov, E Bakhturina… - arXiv preprint arXiv …, 2023

… To enrich our LM model with more diverse examples of punctuation and capitalization, we download a random subset from the Common Crawl dataset [20]. The Common Crawl subset is 100 times bigger than LibriSpeech-PC and is normalized …


Life-long phishing attack detection using continual learning

A Ejaz, AN Mian, S Manzoor - Scientific Reports, 2023

… Also, FastText is trained on a common crawl dataset consisting of web pages related to our problem. Word2Vec and Glove are small and used for non-complex tasks, while ELMo and BERT are very complex models used for complex tasks. …


Lightweb: Private web browsing without all the baggage

This paper proposes lightweb, a new system for private browsing. A lightweb client can browse a web of textbased pages without revealing to anyone—not the network, not the servers hosting the pages—which pages it is reading. Unlike Tor and other …


Limitations of democratic rights during the Covid-19 pandemic—exploring the citizens' perception and discussions on dangers to democracy in Germany

A Katsanidou, M Kneuer, F Bensmann, D Dimitrov… - Zeitschrift für Vergleichende …, 2023

The governments’ mitigation measures to fight the COVID-19 pandemic are unprecedented in our post-war history. For overcoming this crisis, citizens were expected to act in compliance with these measures in order to control the spread of …


Linear Mappings: Semantic Transfer from Transformer Models for Cognate Detection and Coreference Resolution

Embeddings or vector representations of language and their properties are useful for under-standing how Natural Language Processing technology works. The usefulness of embeddings, however, depends on how contextualized or information-rich …