Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlook

X Zou, Y Yan, X Hao, Y Hu, H Wen, E Liu, J Zhang, Y Li… - arXiv preprint arXiv …, 2024

As cities continue to burgeon, Urban Computing emerges as a pivotal discipline for sustainable development by harnessing the power of cross-domain data fusion from diverse sources (eg, geographical, traffic, social media, and environmental data) …


Deep Learning Model for Tamil Part-of-Speech Tagging

H Visuwalingam, R Sakuntharaj, J Alawatugoda… - The Computer Journal, 2024

… These pre-trained word embeddings are trained on Common Crawl and Wikipedia. This model was trained using CBOW with position-weights, in dimension 300, with character n-grams of length 5, a window of size 5 and 10 negatives. …


DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

X Bi, D Chen, G Chen, S Chen, D Dai, C Deng, H Ding… - arXiv preprint arXiv …, 2024

… Our analysis revealed that deduplicating the entire Common Crawl corpus results in higher removal of duplicate instances compared to deduplicating within a single dump. Table 1 illustrates that deduplicating across 91 dumps eliminates four times …


DeepSeek-VL: Towards Real-World Vision-Language Understanding

H Lu, W Liu, B Zhang, B Wang, K Dong, B Liu, J Sun… - arXiv preprint arXiv …, 2024

We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. Our approach is structured around three key dimensions: We strive to ensure our data is diverse …


DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Z Shao, P Wang, Q Zhu, R Xu, J Song, M Zhang, YK Li… - arXiv preprint arXiv …, 2024

… the DeepSeekMath Corpus from Common Crawl. As depicted in Figure 2, … Common Crawl, we employ URL-based deduplication and near-deduplication techniques, resulting in 40B HTML web pages. We then recall mathematical web …


Deliverable 3.1: WP3 1st Interim technical report

1.1. Background This document is the first Interim Technical Report (deliverable 3.1) prepared within the WP3 New usecases for the ESSnet Trusted Smart Statistics–Web Intelligence Network project (ESSnet TSS-WIN). The report covers the period from …


Design of Artificial Intelligence Companion Chatbot

With the development of cities and the prevalence of networks, interpersonal relationships have become increasingly distant. When people crave communication, they hope to find someone to confide in. With the rapid advancement of deep …


Design, Implementation and Evaluation of a Chatbot for Accounting Firm: A Fine-Tuning Approach With Two Novel Dataset

Artificial intelligence, particularly in the field of Chatbots, is fundamentally reshaping learning, communication, and work paradigms. This phenomenon has sparked growing interest among businesses, viewing Chatbots as a means to streamline …


Designing a course for pre-service science teachers using ChatGPT: what ChatGPT brings to the table

HZ Okulu, N Muslu - Interactive Learning Environments, 2024

ChatGPT holds significant potential for enhancing learning through integration into education as an advanced chatbot. With the goal of harnessing this potential, our research focused on exploring the utilization of ChatGPT in designing a course plan …


Designing an Intelligent System to Map Global Connections

E Bellamy, K Farrell, A Hopping, J Pinter, M Saju… - 2024

… Extended Abstract: This study develops a knowledge graph from the Common Crawl News Dataset to provide situational awareness and … We developed a data pipeline to extract semantic content from the Common Crawl News feed and filter it …


Designing and developing a dedicated Natural Language Processing Framework for Healthcare Information Technology Management and Assessment

The escalating complexity of the hospital environment, propelled by technological advancements, necessitates a comprehensive exploration of the integration and management of diverse tools and technologies in healthcare settings. In this context …


Designing Heterogeneous LLM Agents for Financial Sentiment Analysis

F Xing - arXiv preprint arXiv:2401.05799, 2024

… GPT-3.5 was trained mainly on the Common Crawl corpus [2], which archives the web. BLOOMZ was trained on an even larger Open-science Open-collaboration Text Sources corpus [19], which is mainly crowd-sourced scientific datasets. The five …


Detecting Dementia from Transcribed Speech in Slovak using Pre-trained BERT Models

J Staš, D Hládek, A Kopnický - 2024 34th International Conference Radioelektronika …, 2024

… TB of texts written in 100 languages from the Common Crawl dataset [14]; • RemBERT model6: a Rebalanced multilingual BERT model pre-trained on a large unlabeled text created by the combination of mC4 Common Crawl and Wikipedia …


Detecting Hidden Meaning in Stock Images

OBJECTIVE − Automated extraction of hidden meaning in stock images− Is it possible to distinguish what is shown from what is meant?− Examine the divergence of text and image− Analyze the usage of stock images and textual descriptions on the web …


Detection of Hate Speech and Offensive Language CodeMix Text in Dravidian Languages using Cost-Sensitive Learning Approach

K Sreelakshmi, B Premjith, BR Chakravarthi… - IEEE Access, 2024

… It was trained using more than two terabytes of filtered CommonCrawl data. It has significantly improved performance on various cross-lingual transfer tasks and outperformed the mBERT model. XLM-R has 24 layers and 16 attention heads …


Detection of phishing addresses and pages with a data set balancing approach by generative adversarial network (GAN) and convolutional neural network (CNN) …

S Jafari, N Aghaee‐Maybodi - Concurrency and Computation: Practice and …

Phishing attacks have a remarkable ability to steal user information by using simple techniques. Phishing attacks steal valuable information, such as user names and passwords. The loss caused by phishing attacks is significant, and every year …


Detection of Sarcasm in Urdu Tweets using Deep Learning and Transformer based Hybrid Approaches

ME Hassan, M Hussain, I Maab, U Habib, MA Khan… - IEEE Access, 2024

Sarcasm has a significant role in human communication especially on social media platforms where users express their sentiments through humor, satire, and criticism. The identification of sarcasm is crucial in comprehending the sentiment and the …


Detection of Twitter Spam Using GLoVe Vocabulary Features, Bidirectional LSTM and Convolution Neural Network

P Manasa, A Malik, I Batra - SN Computer Science, 2024

… The training of the GloVe model involves a substantial text corpus, like Wikipedia or Common Crawl, which facilitates the learning of connections between words and phrases. For every word in the corpus, the model creates a vector, wherein each …


Developing ChatGPT for Biology and Medicine: A Complete Review of Biomedical Question Answering

Q Li, L Li, Y Li - arXiv preprint arXiv:2401.07510, 2024

ChatGPT explores a strategic blueprint of question answering (QA) in delivering medical diagnosis, treatment recommendations, and other healthcare support. This is achieved through the increasing incorporation of medical domain data via natural …


Development and Evaluation of a German Language Model for the Financial Domain

N Kozaeva, S Hamotskyi, C Hänig - Proceedings of the Joint Workshop of the 7th …, 2024

… The German colossal, cleaned Common Crawl corpus15 was employed, comprising texts of varying lengths. Further the results of LM performance for mixed datasets (financial corpus mixed with common language sentences and financial …


Development and Evaluation of Pre-trained Language Models for Historical Danish and Norwegian Literary Texts

A Al-Laith, A Conroy, J Bjerring-Hansen… - Proceedings of the 2024 …, 2024

We develop and evaluate the first pre-trained language models specifically tailored for historical Danish and Norwegian texts. Three models are trained on a corpus of 19th-century Danish and Norwegian literature: two directly on the corpus with no …


Diagonalization & Forcing FLEX: From Cantor to Cohen and Beyond

Background: Maybe very high energy cosmic radiation started a synaptic butterfly cascade 2... who knows? For one reason or another, I decided to embark on a journey of exploring and discussing the possible role of Artificial Intelligence (AI) on …


Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning

XY Liu, R Zhu, D Zha, J Gao, S Zhong, M Qiu - arXiv preprint arXiv:2312.17493, 2023

The surge in interest and application of large language models (LLMs) has sparked a drive to fine-tune these models to suit specific applications, such as finance and medical science. However, concerns regarding data privacy have emerged …


Digital Threats

S Samtani, E Raff, H Anderson, E Domschot… - 2024

… First, we used the common crawl GloVe embedding, created by Stanford [28]. This embedding was trained over 42 billion tokens, with a 300-dimensional vector space, and was used as the embedding layer of the BiLSTM model. Results are …


DiPaCo: Distributed Path Composition

A Douillard, Q Feng, AA Rusu, A Kuncoro, Y Donchev… - arXiv preprint arXiv …, 2024

Progress in machine learning (ML) has been fueled by scaling neural network models. This scaling has been enabled by ever more heroic feats of engineering, necessary for accommodating ML approaches that require high bandwidth …


Disambiguating Homographs and Homophones Simultaneously: A Regrouping Method for Japanese

Y Sato - Proceedings of the 2024 Joint International Conference …, 2024

We present a method that re-groups surface forms into clusters representing synonyms, and help disambiguate homographs as well as homophone. The method is applied post-hoc to trained contextual word embeddings. It is beneficial to …


Disambiguating natural language via aligning meaningful descriptions

Artificial Intelligence (AI) technologies are increasingly pervading aspects of our lives. Because people use natural language to communicate with each other, computers should also use natural language to communicate with us. One of the principal …


Display options

Chemical patents are a commonly used channel for disclosing novel compounds and reactions, and hence represent important resources for chemical and pharmaceutical research. Key chemical data in patents is often presented in tables …


Dissecting Whiteness: consistencies and differences in the stereotypes of lower-and upper-class White US Americans

T Morgenroth, CT Begeny, TA Kirby, B Paaßen, Y Zeng - Self and Identity, 2024

Economic inequality is increasing in the United States, making categorization and stereotyping based on social class more likely. Yet, social class stereotypes have received relatively little attention. Focusing on spontaneously generated stereotypes …


Distance Comparison Operators for Approximate Nearest Neighbor Search: Exploration and Benchmark

Z Wang, H Xiong, Z He, P Wang - arXiv preprint arXiv:2403.13491, 2024

Approximate nearest neighbor search (ANNS) on high-dimensional vectors has become a fundamental and essential component in various machine learning tasks. Prior research has shown that the distance comparison operation is the bottleneck of …


Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages

R van Noord, T Kuzman, P Rupnik, N Ljubešić… - arXiv preprint arXiv …, 2024

… rest of corpora compared in this paper, MaCoCu corpora are not obtained by processing Common Crawl data. Instead, a strategy consisting of crawling relevant internet top-level domains directly for the targeted languages is followed (eg, .al for …


Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks

The concept of localization in LLMs is often mentioned in prior work; however, methods for localization have never been systematically and directly evaluated. We propose two complementary benchmarks that evaluate the ability of localization …


Do not shut up and do dribble: social media and TV consumption

M Pazzona, N Spagnolo - Journal of Population Economics, 2024

… Footnote 11 The NLP model utilized in this study, XLM-RoBERTa, undergoes pre-training on 2.5TB of filtered CommonCrawl data encompassing 100 languages. For tweet classification, we employed the general-purpose Python library TweetNLP (Camacho-Collados …


Does Lack of Knowledge and Hardship of Information Access Signify Powerful AI? A Large Language Model Perspective

IA Zahid, SS Joudar - Applied Data Science and Analysis, 2023

Large Language Models (LLMs) are evolving and expanding enormously. With the consistent improvement of LLMs, more complex and sophisticated tasks will be tackled. Handling various tasks and fulfilling different queries will be more precise …


Does the Language Matter? Curriculum Learning over Neo-Latin Languages

G Pucci, L Ranaldi - Proceedings of the 2024 Joint International Conference …, 2024

Curriculum Learning (CL) is emerging as a relevant technique to reduce the cost of pre-training Large Language Models. The idea, tested for the English language, is to train LLMs by organizing training examples from the simplest to the most complex …


Does your data spark joy? Performance gains from domain upsampling at the end of training

C Blakeney, M Paul, BW Larsen, S Owen, J Frankle - arXiv e-prints, 2024

Pretraining datasets for large language models (LLMs) have grown to trillions of tokens composed of large amounts of CommonCrawl (CC) web scrape along with smaller, domain-specific datasets. It is expensive to understand the impact of these …


Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

L Soldaini, R Kinney, A Bhagia, D Schwenk, D Atkinson… - arXiv preprint arXiv …, 2024

… In total, we find 0.02% of documents in the 25 Common Crawl snapshots match this filter. … Head, Middle, and Tail parts of our Common Crawl data. The correlation is computed for 24M, … The correlation among the documents flagged for removal …


DPHANet: Discriminative Parallel and Hierarchical Attention Network for Natural Language Video Localization

R Chen, J Tan, Z Yang, X Yang, Q Dai, Y Cheng, L Lin - IEEE Transactions on …, 2024

Natural Language Video Localization (NLVL) has recently attracted much attention because of its practical significance. However, the existing methods still face the following challenges: 1) When the models learn intra-modal semantic association …


Dr. Nahid Ebrahimi Majd

The issue of web phishing attacks has been more prevalent in recent years, and phishing is one of the riskiest online crimes that can have disastrous consequences. The aim of phishing is to collect confidential information by tricking a target and …


DrawL: Understanding the Effects of Non-Mainstream Dialects in Prompted Image Generation

JN Williams, M FitzMorris, O Aka, S Laszlo - arXiv preprint arXiv:2405.05382, 2024

… Stable Diffusion is trained in part using the LAION-5B dataset [50], which consists of text-image pairs from Common CrawlCommon Crawl includes archived data from a large variety of sources, including sites such as Reddit and a variety of blogs …


DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domain

Y Labrak, A Bazoge, OE Khettari, M Rouvier, N Grabar… - arXiv preprint arXiv …, 2024

The biomedical domain has sparked a significant interest in the field of Natural Language Processing (NLP), which has seen substantial advancements with pre-trained language models (PLMs). However, comparing these models has proven …


DsDm: Model-Aware Dataset Selection with Datamodels

L Engstrom, A Feldmann, A Madry - arXiv preprint arXiv:2401.12926, 2024

When selecting data for training large-scale models, standard practice is to filter for examples that match human notions of data quality. Such filtering yields qualitatively clean datapoints that intuitively should improve model behavior. However, in …


Dual Modalities of Text: Visual and Textual Generative Pre-training

Y Chai, Q Liu, J Xiao, S Wang, Y Sun, H Wu - arXiv preprint arXiv:2404.10710, 2024

… Meanwhile, The C4 dataset represents a substantial refinement of the Common Crawl corpus. This dataset, derived from the extensive Common Crawl web scrape, undergoes rigorous cleaning and preprocessing to ensure the quality and relevance …


Dynamic Few-shot Learning for Computational Social Science

R Malla, TG Coan, V Srinivasan, C Boussalis

Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP) and have recently demonstrated remarkable potential in social science research through their capacity to efficiently perform a wide range of text-as-data tasks …


Easy Problems That LLMs Get Wrong

S Williams, J Huckle - arXiv preprint arXiv:2405.19616, 2024

We introduce a comprehensive Linguistic Benchmark designed to evaluate the limitations of Large Language Models (LLMs) in domains such as logical reasoning, spatial intelligence, and linguistic understanding, among others. Through a series of …


Echoes of culture: Relationships of implicit and explicit attitudes with contemporary English, historical English, and 53 non-English languages

TES Charlesworth, K Morehouse, V Rouduri… - Social Psychological and …, 2024

Attitudes are intertwined with culture and language. But to what extent? Emerging perspectives in attitude research suggest that cultural representations in language are more related to implicitly measured (vs. explicitly measured) attitudes, and that …


Efficiency Comparison of Dataset Generated by LLMs using Machine Learning Algorithms

P Pawade, M Kulkarni, S Naik, A Raut, KS Wagh - 2024 International Conference on …, 2024

… • Bing Chat was trained on web text data, including Common Crawl, Wikipedia, news articles, social media posts, and other publicly available text, via a technique called denoising autoencoder (DAE). In DAE, the model is given a noisy version of a …


Efficient learning in spiking neural networks

A Rast, MA Aoun, EG Elia, N Crook - Neurocomputing, 2024

… Training data required is similarly massive: the CommonCrawl dataset is 242 TB in size, and hyperparameter tuning remains an expensive, search-based process that can require ∼ 1000 − 10 , 000 + sweeps. After evaluating a number of possible …


Efficient Model-Relational Data Management: Challenges and Opportunities

V Sanca, A Ailamaki - IEEE Transactions on Knowledge and Data …, 2024

As modern data pipelines continue to collect, produce, and store various data formats, extracting and combining value from traditional and context-rich sources becomes unsuitable for RDBMS. To tap into the dark data, domain experts analyze …


Efficient Training and Inference: Techniques for Large Language Models Using Llama

… The training dataset consisted of a mixture of the Common Crawl, Wikipedia, and BooksCorpus datasets, encompassing a diverse range of linguistic structures and vocabularies. For evaluation, the GLUE benchmark suite was utilized, providing a …


Eliciting Big Five Personality Traits in Large Language Models: A Textual Analysis with Classifier-Driven Approach

A Hilliard, C Munoz, Z Wu, AS Koshiyama - arXiv preprint arXiv:2402.08341, 2024

… GPT-3 followed, with 175 billion parameters and training on datasets like Common Crawl [15]. GPT-3.5 Turbo was introduced to enhance real-time performance, and the latest, GPT-4, boasts 1.76 trillion parameters and capabilities …


EMBA: Entity Matching using Multi-Task Learning of BERT with Attention-over-Attention

… The WDC Product Data Corpus for Large-scale Product Matching [34], was built by extracting product offers from Common Crawl. The WDC datasets serve as a popular benchmark and have been used for evaluation in DITTO, JointBERT, and …


Embedding-based Query Spelling Correction

For many retrieval systems, correcting spelling errors in the queries that searchers submit is an essential step of query understanding. Inspired by a blog post on spelling correction from 2018, we implement and analyze a simple embedding-based …


Emergent Abilities in Reduced-Scale Generative Language Models

S Muckatira, V Deshpande, V Lialin, A Rumshisky - arXiv preprint arXiv:2404.02204, 2024

Large language models can solve new tasks without task-specific fine-tuning. This ability, also known as in-context learning (ICL), is considered an emergent ability and is primarily seen in large language models with billions of parameters. This …


Emerging AI Tools for Education and Research: Perspective and Policies for IISc

The Director, IISc, constituted a committee on August 21, 2023 with the following mandate:(a) Explore the challenges and benefits of emerging AI tools in the context of academic teaching and learning,(b) provide guidance and recommendations to …


Emotional significance in Cross-Cultural Semantic Crossmodal Correspondences

J Alvarado - Proceedings of the Annual Meeting of the Cognitive …, 2024

Crossmodal correspondences are associations between perceptual features from different senses that aid in crossmodal binding. The semantic coding of these correspondences is expected to capture and mediate the emergence of perceptual …


Employing Siamese MaLSTM Model and ELMO Word Embedding for Quora Duplicate Questions Detection

A Altamimi, M Umer, D Hanif, S Alsubai, TH Kim… - IEEE Access, 2024

… 3) FastText Subword FastText Subword comprises a collection of 2 million word vectors trained on the Common Crawl dataset, which consists of a massive 600 billion tokens. In contrast to traditional word embeddings, sub-word embeddings …


Empowering hate speech detection: leveraging linguistic richness and deep learning

IGBJ Abasan, EB Setiawan - Bulletin of Electrical Engineering and Informatics, 2024

… What they don’t realize is the use of pre-trained models, namely CommonCrawl and Wiki, in extraction using FastText or GloVe. The pre-trained model used is not specific for hate speech detection and, of course, consists of many languages, and …


Enabling action crossmodality for a pretrained large language model

A Caesar, O Özdemir, C Weber, S Wermter - Natural Language Processing Journal, 2024

Natural language processing and vision tasks have seen large improvements recently through the rise of Transformer architectures. The high performing large language models (LLMs) benefit from large textual datasets that are numerously …


Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment

A Agarwalla, A Gupta, A Marques, S Pandit, M Goin… - arXiv preprint arXiv …, 2024

Large language models (LLMs) have revolutionized Natural Language Processing (NLP), but their size creates computational bottlenecks. We introduce a novel approach to create accurate, sparse foundational versions of performant LLMs that achieve full …


Enabling self-identification in intelligent agent: insights from computational psychoanalysis

L Li, C Li - arXiv preprint arXiv:2403.07664, 2024

Building upon prior framework of computational Lacanian psychoanalysis with the theory of active inference, this paper aims to further explore the concept of self-identification and its potential applications. Beginning with two classic paradigms in psychology …


End to End Urdu Abstractive Text Summarization with Dataset and Improvement in Evaluation Metric

Urdu, being a common language in South Asia, has not received significant attention in terms of language processing compared to more advanced languages. In the field of Natural Language Processing (NLP), the task of text summarization …


EnhancedBERT: A Feature-rich Ensemble Model for Arabic Word Sense Disambiguation with Statistical Analysis and Optimized Data Collection

S Kaddoura, R Nassar - Journal of King Saud University-Computer and …, 2024

Accurate assignment of meaning to a word based on its context, known as Word Sense Disambiguation (WSD), remains challenging across languages. Extensive research aims to develop automated methods for determining word senses in …


Enhancing 3D Asset Retrieval with Semantic Search

Semantic search goes beyond exact keyword matching and instead focuses on the comprehension of search queries' purpose and the surrounding context to deliver relevant search results. In this work, we focus on integrating hybrid search, which …


ENHANCING AUTOMATIC INVOICE CODING PERFORMANCE WITH UNSTRUCTURED DATA

… The published word vectors are 300 dimensional and trained on the internet data from Common Crawl3. The pretrained word vectors can be further fine-tuned using the provided code and datasets more similar to the final usage scenario to increase …


Enhancing Cybersecurity: A Review and Comparative Analysis of Convolutional Neural Network Approaches for Detecting URL-Based Phishing Attacks

M Nanda, M Saraswat, PK Sharma - e-Prime-Advances in Electrical Engineering …, 2024

Phishing attempts to mimic the official websites of businesses, including banks, e-commerce, government offices, and financial institutions. Phishing websites aim to collect and retrieve sensitive data from users, including passwords, credit card numbers, email …


Enhancing Ecological Knowledge Discovery Using Large Language Models

Earth is home to an astonishing diversity of life forms. Among these, vascular plants stand as one of the most vital and ubiquitous groups, accounting for approximately 80% of global biomass [3]. The census of vascular plants, which presently exceeds …


Enhancing English Translation Quality Assessment through Knowledge Transfer in Artificial Intelligence Context

Abstract Machine translation technology, which employs computers to autonomously convert text between source and target languages, represents a pivotal realm within artificial intelligence and natural language processing research. This paper …


Enhancing knowledge graphs with microdata and LLMs: the case of Schema. org and Wikidata in touristic information

L Gonzalez-Garcia, G González-Carreño… - The Electronic Library, 2024

… We have used the Common Crawl (CC) corpus of crawled Web pages to extract metadata published in the wild, and then used a mapping of Schema.org annotations to Wikidata to get estimates of the added value (in terms of additional …


Enhancing Parameter Efficiency in Model Inference Using an Ultralight Inter-Transformer Linear Structure

… It was constructed from the Chuweb21 generated based on the April 2021 block of the Common Crawl dataset. The WWW-4 dataset comprises 50 queries. Robust04 dataset is also an English web corpus generated from TREC disks 4 and 5, and …


Enhancing Phishing Detection, Leveraging Deep Learning Techniques

A Ullah, RA Shah, SA Nawaz, N Ahmad, MH Malik - Journal of Computing & …, 2024

… To achieve this, we utilized the Common Crawl Corpus, which aggregates extensive data collected over seven years. This corpus comprises raw web page data, metadata, and textual content. Legitimate domains with substantial backlink …


Enhancing Product Design through AI-Driven Sentiment Analysis of Amazon Reviews Using BERT

MK Shaik Vadla, MA Suresh, VK Viswanathan - Algorithms, 2024

Understanding customer emotions and preferences is paramount for success in the dynamic product design landscape. This paper presents a study to develop a prediction pipeline to detect the aspect and perform sentiment analysis on review …


Enhancing Sentiment Analysis Accuracy in Borobudur Temple Visitor Reviews through Semi-Supervised Learning and SMOTE Upsampling

C Agustina, P Purwanto, F Farikhin - Journal of Advances in Information Technology, 2024

The level of visitor satisfaction with tourist destinations can be known from reviews on social media. One method used is to carry out sentiment analysis on comments given by visitors on social media or related websites. This study was envisioned as a …


Enhancing Sentiment Analysis on Social Media Data with Advanced Deep Learning Techniques.

HH Nguyen - International Journal of Advanced Computer Science & …, 2024

This paper introduces a comprehensive methodology for conducting sentiment analysis on social media using advanced deep learning techniques to address the unique challenges of this domain. As digital platforms play an increasingly pivotal …


Enhancing Vision-Language Pre-training with Rich Supervisions

Y Gao, K Shi, P Zhu, E Belval, O Nuriel, S Appalaraju… - arXiv preprint arXiv …, 2024

… This is expected as our data collected from Common Crawl has different distribution against the original Pix2struct data due to the data size (2M vs. 80M) and potentially different website filtering strategies. Specifically, we filter out website …


Enriching Interactive Explanations with Fuzzy Temporal Constraint Networks

M Canabal-Juanatey, JM Alonso-Moral, A Catala… - International Journal of …, 2024

… such as BERT [2] (Bidirectional Encoder Representations from Transformers), GPT [26] (Generative Pre-trained Transformer) and their successors (ALBERT [4], RoBERTa [3], BART [7], GPT-3 [5] or GPT-4 [27]), which were trained with huge …


EpilepsyLLM: Domain-Specific Large Language Model Fine-tuned with Epilepsy Medical Knowledge

X Zhao, Q Zhao, T Tanaka - arXiv preprint arXiv:2401.05908, 2024

With large training datasets and massive amounts of computing sources, large language models (LLMs) achieve remarkable performance in comprehensive and generative ability. Based on those powerful LLMs, the model fine-tuned with domain-specific …


Ethics of AI in the Teaching of English

New technologies have always impacted literacy, and artificial intelligence (AI) is no exception (Tyner, 1998). English teachers find themselves having to consider how AI may impact how they teach. NCTE’s Beliefs for Integrating Technology Into the …


EthioLLM: Multilingual Large Language Models for Ethiopian Languages with Task Evaluation

AL Tonja, IA Azime, TD Belay, MG Yigezu… - arXiv preprint arXiv …, 2024

Large language models (LLMs) have gained popularity recently due to their outstanding performance in various downstream Natural Language Processing (NLP) tasks. However, low-resource languages are still lagging behind current state-of-the-art …


EvalQuiz: self-assessment generated through language transformer models

This thesis explores the constraints of self-assessment creation in higher education, focusing on the lack of tools, standardization, and time. We conduct didactic field expert interviews to investigate how teaching evolves towards learning management …


Evaluating and improving lexical language understanding in neural machine translation

Lexical understanding is an inalienable component of the translation process. In order to correctly map the meaning of a linguistic unit to the appropriate target language expression, the meaning of its constituent words has first to be identified …


Evaluating and Mitigating Limitations of Large Language Models in Clinical Decision Making

P Hager, F Jungmann, K Bhagat, I Hubrecht, M Knauer… - medRxiv, 2024

Clinical decision making is one of the most impactful parts of a physician's responsibilities and stands to benefit greatly from AI solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent …


Evaluating Differential Privacy Approaches for Query Obfuscation in Information Retrieval

Protecting the privacy of a user while they interact with an Information Retrieval (IR) system is crucial. This becomes more challenging when the IR system is not cooperative in satisfying the user’s privacy needs. Recent advancements in Natural …


Evaluating embedded semantics for accessibility description of web crawl data

R Navarrete, D Martinez-Mosquera, L Recalde… - AHFE (2023) International …, 2023

… The dataset for analysis is provided by Web Data Commons (WDC), an organization that releases extracted data from Common Crawl (CC), the largest web corpus available to the public. The dataset was released in October 2021 …


Evaluating English-language morphological awareness assessments

CL Hudson Kam, E Sadlier-Brown, S Clark, C Jang… - First Language, 2024

… These corpora were selected because they contain language that children are exposed to, and so, provide a more realistic picture of children’s potential knowledge than an adult-language based corpus would (eg the Enron email dataset …


Evaluating Large Language Model Performance on Haskell

I introduce HaskellEval, a Haskell evaluation benchmark for Large Language Models. HaskellEval’s curation leverages a novel synthetic generation framework, streamlining the process of dataset curation by minimizing manual intervention. The …


Evaluating Large Language Models for Generalization and Robustness via Data Compression

Y Li, Y Guo, F Guerin, C Lin - arXiv preprint arXiv:2402.00861, 2024

Existing methods for evaluating large language models face challenges such as data contamination, sensitivity to prompts, and the high cost of benchmark creation. To address this, we propose a lossless data compression based evaluation …


Evaluating Large Language Models on Academic Literature Understanding and Review: An Empirical Study among Early-stage Scholars

J Wang, H Hu, Z Wang, Y Song, Y Sheng, D He - 2024

The rapid advancement of large language models (LLMs) such as ChatGPT makes LLM-based academic tools possible. However, little research has empirically evaluated how scholars perform different types of academic tasks with LLMs …


Evaluating Multilingual Abstractive Dialogue Summarization in Indian Languages using mT5-small & IndicBART

M Sharma, G Goyal, A Gupta, R Rani, A Sharma, A Dev - 2024 IEEE 9th International …, 2024

… The mC4 dataset consists of natural text in 101 languages including Indian languages like Hindi, Mararti, Punjabi and others, that was collected from the public Common Crawl web scrape. Introduced as a multilingual, sequence-to-sequence pre-trained …


Evaluating Shortest Edit Script Methods for Contextual Lemmatization

O Toporkov, R Agerri - arXiv preprint arXiv:2403.16968, 2024

Modern contextual lemmatizers often rely on automatically induced Shortest Edit Scripts (SES), namely, the number of edit operations to transform a word form into its lemma. In fact, different methods of computing SES have been proposed as an …


Evaluating the Experience of LGBTQ+ People Using Large Language Model Based Chatbots for Mental Health Support

Z Ma, Y Mei, Y Long, Z Su, KZ Gajos - arXiv preprint arXiv:2402.09260, 2024

LGBTQ+ individuals are increasingly turning to chatbots powered by large language models (LLMs) to meet their mental health needs. However, little research has explored whether these chatbots can adequately and safely provide tailored support …


Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains

S Ramprasad, K Krishna, ZC Lipton, BC Wallace - arXiv preprint arXiv:2402.03509, 2024

Recent work has shown that large language models (LLMs) are capable of generating summaries zero-shot (ie, without explicit supervision) that, under human assessment, are often comparable or even preferred to manually composed …


Evaluating the impact of design decisions on passive DNS-based domain rankings

V Le Pochat, S Fernandez, T Van Goethem…

… They recommend that researchers select a random sample of websites among Common Crawl hosts. Ruth et al. [6] compared top lists to Cloudflare traffic data, concluding that the Chrome User Experience Report [11] most accurately represents …


Evaluating the Quality of AI-Generated Items for a Certification Exam

AD Mead, C Zhou - Journal of Applied Testing Technology, 2024

OpenAI’s GPT-3 model can write multiple-choice exam items. This paper reviewed the literature on automatic item generation and then described the recent history of OpenAI GPT models and their operation, and then described a methodology for …


Evaluating Transformers and Linguistic Features integration for Author Profiling tasks in Spanish

JA García-Díaz, G Beydoun, R Valencia-García - Data & Knowledge Engineering, 2024

… This model is a training from scratch of RoBERTa, trained on the Spanish texts from mC4, compiled from the public Common Crawl web … This is a multilingual version of RoBERTa, pre-trained with about 2.5 TB of data from the 100 different …


Evaluation and Adaptation of Neural Language Models for Under-Resourced Languages

Abstract Language models are now commonly used by researchers, industry, and anyone interested. However, language models of all sizes and types are primarily developed for the English language while efforts on other languages lag behind …


Evaluation of AI-generated Responses by Different Artificial Intelligence Chatbots to the Clinical Decision-Making Case-Based Questions in Oral and Maxillofacial …

A Azadi, F Gorjinejad, H Mohammad-Rahimi, R Tabrizi… - Oral Surgery, Oral Medicine …, 2024

Objectives This study aims to evaluate the correctness of the generated answers by Google Bard, GPT-3.5, GPT-4, Claude-Instant, and Bing chatbots to decision-making clinical questions in the oral and maxillofacial surgery (OMFS) area. Study Design A …


Evaluation of Geographical Distortions in Language Models: A Crucial Step Towards Equitable Representations

R Decoupes, R Interdonato, M Roche, M Teisseire… - arXiv preprint arXiv …, 2024

Language models now constitute essential tools for improving efficiency for many professional tasks such as writing, coding, or learning. For this reason, it is imperative to identify inherent biases. In the field of Natural Language Processing …


Evaluation of word embedding models used for diachronic semantic change analysis

Y Maslennikova, V Bochkarev - Journal of Physics: Conference Series, 2024

… As a reference word2vec model, we used pre-trained word vectors that were trained on English versions of Common Crawl dataset using fastText library [14]. fastText is a library for efficient learning of word representations and sentence …


Evidence for a digital divide? Measuring DNS dependencies in the context of the indigenous population of Australia

R Holz, N Nazemi, O Tavallaie, AY Zomaya

We recently presented a work-in-progress paper at the Workshop on Transparency, Accountability and User Control for a Responsible Internet (TAURIN 2023). Our paper investigates the relationship between the digital divide, Internet transparency …


EXAMINING ACCURACY HETEROGENEITIES IN CLASSIFICATION OF MULTILINGUAL AI-GENERATED TEXT

Tools for detection of AI-generated texts are used globally, however, the nature of the apparent accuracy disparities between languages must be further observed. This paper aims to examine the nature of these differences through testing OpenAI’s …


Experimental Evaluation of Possible Feature Combinations for the Detection of Fraudulent Online Shops

A Janavičiūtė, A Liutkevičius, G Dabužinskas… - Applied Sciences, 2024

Online shopping has become a common and popular form of shopping, so online attackers try to extract money from customers by creating online shops whose purpose is to compel the buyer to disclose credit card details or to pay money for …


Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda

J Schneider - arXiv preprint arXiv:2404.09554, 2024

Generative AI (GenAI) marked a shift from AI being able to recognize to AI being able to generate solutions for a wide variety of tasks. As the generated solutions and applications become increasingly more complex and multi-faceted, novel needs …


Exploring LLMs' Capabilities for Error Detection in Dutch L1 and L2 Writing Products

J Kruijsbergen, S Van Geertruyen, V Hoste… - Computational Linguistics in …, 2024

This research examines the capabilities of Large Language Models for writing error detection, which can be seen as a first step towards automated writing support. Our work focuses on Dutch writing error detection, targeting two envisaged end-users …


Exploring Machine Learning for Malware Detection With Feature Selection, Explainable AI, and Generative Adversarial Networks

This research focuses on the detection of malware in sample datasets using machine learning algorithms. As malware becomes more advanced against detection in the cybersecurity sector, researchers have looked for other methods to …


Exploring the competence of ChatGPT for customer and patient service management

A Haleem, M Javaid, RP Singh - Intelligent Pharmacy, 2024

The modern language generation model ChatGPT, created by Open Artificial Intelligence (AI), is recognised for its capacity to comprehend context and produce pertinent content. This model is built on the transformer architecture, which enables …


Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning

Y Wang, W Chen, X Han, X Lin, H Zhao, Y Liu, B Zhai… - arXiv preprint arXiv …, 2024

Strong Artificial Intelligence (Strong AI) or Artificial General Intelligence (AGI) with abstract reasoning ability is the goal of next-generation AI. Recent advancements in Large Language Models (LLMs), along with the emerging field of Multimodal Large …


Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties

E Artemova, V Blaschke, B Plank - arXiv preprint arXiv:2402.02078, 2024

Mainstream cross-lingual task-oriented dialogue (ToD) systems leverage the transfer learning paradigm by training a joint model for intent recognition and slot-filling in English and applying it, zero-shot, to other languages. We address a gap in prior …


Exploring the Role of ChatGPT in Cardiology: A Systematic Review of the Current Literature

A Sharma, T Medapalli, M Alexandrou, E Brilakis… - Cureus, 2024

Chat Generative Pre-Trained Transformer (ChatGPT) is a chatbot based on a large language model that has gained public interest since its release in November 2022. This systematic review examines the current literature on the potential applications …


Exploring the role of ChatGPT in higher education: opportunities, challenges and ethical considerations

A Zeb, R Ullah, R Karim - The International Journal of Information and Learning …, 2024

Purpose This paper aims to examine the opportunities and challenges of using ChatGPT in higher education. Furthermore, it is also discuss the potential risks and plunders of these tools. Design/methodology/approach The paper discuss the use of …


EXPRESS: Malay Lexicon Project 3: The Impact of Orthographic-Semantic Consistency on Lexical Decision Latencies

M Maziyah Mohamed, D Jared - Quarterly Journal of Experimental Psychology, 2024

Theories of word processing propose that readers are sensitive to statistical co-occurrences between spelling and meaning. Orthographic-Semantic Consistency (OSC) measures provide a continuous estimate of the statistical regularities between …


Extending the Comparative Argumentative Machine: Multilingualism and Stance Detection

I Nikishina, A Bondarenko, S Zaczek, OL Haag…

The comparative argumentative machine CAM can retrieve arguments that answer comparative questions—questions that ask which of several to-be-compared options should be favored in some scenario. In this paper, we describe how we equipped …


Extending Translate-Train for ColBERT-X to African Language CLIR

E Yang, DJ Lawrie, P McNamee, J Mayfield - arXiv preprint arXiv:2404.08134, 2024

… We used Common Crawl documents in Afriberta Corpus [6] for the four African languages and collected additional English Common Crawl documents to match the genre. We fine-tune the model for 200,000 update steps using a learning rate of 1×10-5 …


Extracting intersectional stereotypes from embeddings: Developing and validating the Flexible Intersectional Stereotype Extraction procedure

TES Charlesworth, K Ghate, A Caliskan, MR Banaji - PNAS Nexus, 2024

… text corpora, ranging from static GloVe embeddings trained on 840 billion words from the Common Crawl to contextualized BERT embeddings trained on a combination of Wikipedia and Common Crawl text. Additionally, in supplemental …


Factuality of Large Language Models in the Year 2024

Y Wang, M Wang, MA Manzoor, G Georgiev, RJ Das… - arXiv preprint arXiv …, 2024

Large language models (LLMs), especially when instruction-tuned for chat, have become part of our daily lives, freeing people from the process of searching, extracting, and integrating information from multiple sources by offering a …


FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication

E Slyman, S Lee, S Cohen, K Kafle - arXiv preprint arXiv:2404.16123, 2024

Recent dataset deduplication techniques have demonstrated that content-aware dataset pruning can dramatically reduce the cost of training Vision-Language Pretrained (VLP) models without significant performance losses compared to …


Fake news detection in low-resource languages: A novel hybrid summarization approach

J Alghamdi, Y Lin, S Luo - Knowledge-Based Systems, 2024

… Pre-training encompasses a vast dataset exceeding 2TB of CommonCrawl data. The overarching concept involves mapping any given language to a language-agnostic vector space, ensuring that inputs in various languages converge to corresponding …


Feature-augmented model for multilingual discourse relation classification

E Metheniti, C Braud, P Muller - Proceedings of the 5th Workshop on Computational …, 2024

Discourse relation classification within a multilingual, cross-framework setting is a challenging task, and the best-performing systems so far have relied on monolingual and mono-framework approaches. In this paper, we introduce transformer-based …


Federated learning for privacy-preserving depression detection with multilingual language models in social media posts

SS Khalil, NS Tawfik, M Spruit - Patterns, 2024

The incidences of mental health illnesses, such as suicidal ideation and depression, are increasing, which highlights the urgent need for early detection methods. There is a growing interest in using natural language processing (NLP) models to analyze …


Few shot clinical entity recognition in three languages: Masked language models outperform LLM prompting

M Naguib, X Tannier, A Névéol - arXiv preprint arXiv:2402.12801, 2024

Large Language Models are becoming the go-to solution for many natural language processing tasks, including in specialized domains where their few-shot capacities are expected to yield high performance in low-resource settings. Herein, we aim to …


Final Thoughts: Digital Humanities Looking at Generative AI

M Aguiar, S Araújo - Digital Humanities Looking at the World: Exploring …, 2024

In this chapter, we examine generative artificial intelligence in the context of Digital Humanities. We commence by providing a concise overview of this technology and its prevalent models. Following that, we provide a brief survey of the existing …


Final Words

M McTear, M Ashurkina - Transforming Conversational AI: Exploring the Power …, 2024

Conversational AI is a dynamic and fast moving field, and a lot has happened in the six months or so since we began writing this book. We have tried to ensure that what we have covered in the preceding chapters provides a sufficiently general and …


Fine grain emotion analysis in Spanish using linguistic features and transformers

A Salmerón-Ríos, JA García-Díaz, R Pan… - PeerJ Computer Science, 2024

Mental health issues are a global concern, with a particular focus on the rise of depression. Depression affects millions of people worldwide and is a leading cause of suicide, particularly among young people. Recent surveys indicate an increase in …


Fine Tuning LLM for Enterprise: Practical Guidelines and Recommendations

K VM, H Warrier, Y Gupta - arXiv preprint arXiv:2404.10779, 2024

… The datasets are massive text corpus like Common crawl, The Pile and code repositories from Github. Further the model can be fine tuned for specific tasks using specialised datasets. One such series of datasets are called Instruction datasets like …


Fine-Tuned Large Language Models for Symptom Recognition from Spanish Clinical Text

MA Shaaban, A Akkasi, A Khan, M Komeili, M Yaqub - arXiv preprint arXiv …, 2024

… XLM-RL (561M parameters) and XLM-RB (279M parameters) are multilingual masked language models, trained on a filtered CommonCrawl dataset of over two terabytes. We leverage BBS and BBES both pretrained on Spanish clinical data …


Fine-tuning BERT-based Models for Negative Content Identification on Indonesian Tweets

AF Hidayatullah, K Kalinaki, MM Aslam, RY Zakari…

Social media platforms like Twitter have become substantial sources of user-generated content, enabling people to easily express their emotions and opinions. However, this freedom has increased the spread of harmful content, such as abusive language …


Fine-tuning Large Language Models for Multigenerator, Multidomain, and Multilingual Machine-Generated Text Detection

F Xiong, T Markchom, Z Zheng, S Jung, V Ojha… - arXiv preprint arXiv …, 2024

SemEval-2024 Task 8 introduces the challenge of identifying machine-generated texts from diverse Large Language Models (LLMs) in various languages and domains. The task comprises three subtasks: binary classification in monolingual …


FINETUNING COMMERCIAL LARGE LANGUAGE MODELS WITH LORA FOR ENHANCED ITALIAN LANGUAGE UNDERSTANDING

J Hartsuiker, P Torroni, AE Ziri, DF Alise, F Ruggeri

The world of Artificial intelligence is booming, especially the fields of natural language processing (NLP) and Computer Vision (CV). Recently the chatGPT models from OpenAI have caused another shock-wave. This was a clear sign to the …


Finite State Automata on Multi-Word Units for Efficient Text-Mining

Text mining is crucial for analyzing unstructured and semi-structured textual documents. This paper introduces a fast and precise text mining method based on a finite automaton to extract knowledge domains. Unlike simple words, multi-word …


FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models

G Bhatia, EMB Nagoudi, H Cavusoglu… - arXiv preprint arXiv …, 2024

We introduce FinTral, a suite of state-of-the-art multimodal large language models (LLMs) built upon the Mistral-7b model and tailored for financial analysis. FinTral integrates textual, numerical, tabular, and image data. We enhance FinTral with domain-specific …


Foregrounding Artist Opinions: A Survey Study on Transparency, Ownership, and Fairness in AI Generative Art

J Lovato, J Zimmerman, I Smith, P Dodds, J Karson - arXiv preprint arXiv:2401.15497, 2024

Generative Artificial Intelligence (AI) tools are used to create art-like outputs and aid in the creative process. While these tools have potential benefits for artists, they also have the potential to harm the art workforce and infringe upon artistic and intellectual …


Fostering the Ecosystem of Open Neural Encoders for Portuguese with Albertina PT* Family

R Santos, J Rodrigues, L Gomes, J Silva, A Branco… - arXiv preprint arXiv …, 2024

… The OSCAR subset for Portuguese we use is based on November/December 2022 version of Common Crawl, which is an automatic crawl from the web. Despite being a crawl, the final dataset is of relatively good quality due the filtering performed …


Frequently asked questions on erectile dysfunction: evaluating artificial intelligence answers with expert mentorship

M Baturu, M Solakhan, TG Kazaz, O Bayrak - International Journal of Impotence …, 2024

The present study assessed the accuracy of artificiaI intelligence-generated responses to frequently asked questions on erectile dysfunction. A cross-sectional analysis involved 56 erectile dysfunction-related questions searched on Google …


From Bytes to Biases: Investigating the Cultural Self-Perception of Large Language Models

W Messner, T Greene, J Matalone - arXiv preprint arXiv:2312.17256, 2023

Large language models (LLMs) are able to engage in natural-sounding conversations with humans, showcasing unprecedented capabilities for information retrieval and automated decision support. They have disrupted human-technology …


From Cosine Similarity to Likelihood Ratio: Coupling Representations From Machine Learning (and Other Sources) With Cognitive Models

Modern machine learning models yield vector representations that capture similarity relations between complex items like text and images. These representations can help explain and predict how individuals respond to those items in particular tasks …


From Data Creator to Data Reuser: Distance Matters

CL Borgman, PT Groth - arXiv preprint arXiv:2402.07926, 2024

… Common Crawl, launched in 2007, now spends roughly $200,000 per year to maintain petabytes of openly available web crawl data (Common Crawl, 2023; Roberts, 2023). The use of these data to train large language models has …


From Data Quality to Model Performance: Navigating the Landscape of Deep Learning Model Evaluation

M Akram, WH Moosa - Deep Learning for Multimedia Processing Applications, 2024

Deep learning (DL) technologies have revolutionized the analysis of multimedia data, with natural language processing, visual data analytics, and audio recognition being just a few examples of multimedia applications that have effectively utilized DL. This …


From Development to Dissemination: Social and Ethical Issues with Text-to-Image AI-Generated Art

… Model developed by the Computer Vision & Learning Group (CompVis) lab at Ludwig Maximilian University of Munich was trained with a subset of the LAION-5B dataset [6], comprising of about 2.3 billion CLIP-filtered image-text pairs by parsing …


From Form (s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency

X Ohmer, E Bruni, D Hupkes - arXiv preprint arXiv:2404.12145, 2024

… We use the current common crawl statistics6 to compute an estimate of how low- or high-resource these languages are in web-based corpora. Of this corpus, English constitutes 46% of the data, German 5.8%, Italian 2.7%, Dutch 2.2%, and Swedish …


From News to Summaries: Building a Hungarian Corpus for Extractive and Abstractive Summarization

B Barta, D Lakatos, A Nagy, MK Nyist, J Ács - arXiv preprint arXiv:2404.03555, 2024

… We use the freely available Common Crawl dataset4 as a basis for constructing the corpus. It contains petabytes of crawled web pages from the past 25 years and it is available on Amazon S3 in WARC format. Retrieval and deduplication of the raw …


From Text to Transformation: A Comprehensive Review of Large Language Models' Versatility

P Kaur, GS Kashyap, A Kumar, MT Nafis, S Kumar… - arXiv preprint arXiv …, 2024

This groundbreaking study explores the expanse of Large Language Models (LLMs), such as Generative Pre-Trained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT) across varied domains ranging from …


Frozen or Fine-tuned? Analyzing Deep Learning Models and Training Strategies for Optimizing Big Five Personality Traits Prediction from Text

M Soleimani, HB Kashani - 2024 10th International Conference on Artificial …, 2024

In today’s digital age, the comprehension and prediction of human personality traits have assumed paramount significance. This study embarks on the task of forecasting the Big Five personality traits through textual data, harnessing the …


Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions

M Dallabetta, C Dobberstein, A Breiding, A Akbik - arXiv preprint arXiv:2403.15279, 2024

… Specifically, we support the CC-NEWS3 dataset provided by the COMMONCRAWL initiative. At the time of writing, this dataset comprises around 40 terabytes of WARC-formatted data, containing millions of news articles dating back …


Fusing Domain-Specific Content from Large Language Models into Knowledge Graphs for Enhanced Zero Shot Object State Classification

F Gouidis, K Papantoniou, KPT Patkos, A Argyros… - arXiv preprint arXiv …, 2024

Domain-specific knowledge can significantly contribute to addressing a wide variety of vision tasks. However, the generation of such knowledge entails considerable human labor and time costs. This study investigates the potential of Large Language …


Fusion of Deep Learning with Advanced and Traditional Embeddings in Sentiment Analysis

K Kiran, PD Shenoy - 2024 IEEE 9th International Conference for …, 2024

In todays changing landscape the abundance of unprocessed data plays a vital role. It is crucial to curate this dataset and conduct a thorough analysis of the sentiments it captures. The aim of this research study is to explore learning techniques such as Bi …


GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

J Zhao, Z Zhang, B Chen, Z Wang, A Anandkumar… - arXiv preprint arXiv …, 2024

Training Large Language Models (LLMs) presents significant memory challenges, predominantly due to the growing size of weights and optimizer states. Common memory-reduction approaches, such as low-rank adaptation (LoRA), add a trainable …


Gauging Novelty in Crowdfunding Projects: A Theory-Driven Text Analysis Approach

The explosion of crowdfunding projects has called for new ways to screen for novel projects. Building on the multidimensional nature of novelty, we draw on the literature to propose two measures of project novelty that are theoretically sound and …


GenDec: A robust generative Question-decomposition method for Multi-hop reasoning

J Wu, L Yang, Y Ji, W Huang, BF Karlsson, M Okumura - arXiv preprint arXiv …, 2024

Multi-hop QA (MHQA) involves step-by-step reasoning to answer complex questions and find multiple relevant supporting facts. However, Existing large language models'(LLMs) reasoning ability in multi-hop question answering remains exploration, which is …


Gender Bias in Natural Gender Language and Grammatical Gender Language within Children's Literature

There has been much research on the connection between language and gender bias but there is little comparing natural gender language, grammatical gender language, and gender bias. This research is important because it can offer an …


Gender Identity and Representation in the Context of Economic Development in India

… It is a large multi-lingual language model, trained on 2.5TB of filtered CommonCrawl data. It represents some of the most sophisticated pre-trained models prevalently being used in natural language processing for text classification …


Gendered Grammar or Ingrained Bias? Exploring Gender Bias in Icelandic Language Models

SR Friðriksdóttir, H Einarsson - Proceedings of the 2024 Joint International …, 2024

Large language models, trained on vast datasets, exhibit increased output quality in proportion to the amount of data that is used to train them. This data-driven learning process has brought forth a pressing issue where these models may not only reflect …


Generating Apparel Images by Using Stable Diffusion with Prompt Recommendation

XZ Liu, HM Hung, LH Chen - 人工知能学会全国大会論文集 第 38 回 (2024), 2024

Fashion is a profound expression of personal identity, style, and culture in daily life. Making oneself look attractive, which often equates to being fashionable or stylish, is always a priority for many people. In this paper, we proposed a clothing …


Generating Character Lines in Four-Panel Manga

M Inaba - Proceedings of the 37th Pacific Asia Conference on …, 2023

Automatic content generation based on natural language processing is an active research area, especially for story generation. Research on story generation has focused on generating consistent text pertaining to characters’ actions and events; …


Generating Probabilistic Scenario Programs from Natural Language

K Elmaaroufi, D Shankar, A Cismaru… - arXiv preprint arXiv …, 2024

… Most of these model’s are fine-tuned variants of a foundational model that was trained on a wider body of text such as Common Crawl. For instance Codex is ”a GPT language model fine-tuned on publicly available code from GitHub” and it’s …


Generation of API Documentation using Large Language Models–Towards Self-explaining APIs

Application Programming Interfaces (APIs) play a crucial role in modern software development. They are key in not only in enabling software reuse and adaptation, which greatly increase productivity, but also in facilitating exploration of new …


Generative adversarial network-based phishing URL detection with variational autoencoder and transformer

JK Sasi, A Balakrishnan - Int J Artif Intell, 2024

Phishing attacks pose a constant threat to online security, necessitating the development of efficient tools for identifying malicious URLs. In this article, we propose a novel approach to detect phishing URLs employing a generative …


Generative AI and Large Language Models for Cyber Security: All Insights You Need

MA Ferrag, F Alwahedi, A Battah, B Cherif, A Mechri… - arXiv preprint arXiv …, 2024

This paper provides a comprehensive review of the future of cybersecurity through Generative AI and Large Language Models (LLMs). We explore LLM applications across various domains, including hardware design security, intrusion detection …


Generative AI and the Continuing Importance of Information Literacy

GAI models are embedded in a variety of applications. These models are trained using existing information, which is then formatted and delivered within the context of the GAI application. Bias, misand disinformation, and ethical use of information …


Generative AI for Math: Part I--MathPile: A Billion-Token-Scale Pretraining Corpus for Math

Z Wang, R Xia, P Liu - arXiv preprint arXiv:2312.17120, 2023

… expanse of Common Crawl snapshots, a venture we aim to pursue in our future work. … We provide a near-duplicate example found in Common Crawl, as shown in 2 (See Table 8 to … StackExchange that appeared in a document from Common …


Generative AI for Unmanned Vehicle Swarms: Challenges, Applications and Opportunities

G Liu, N Van Huynh, H Du, DT Hoang, D Niyato, K Zhu… - arXiv preprint arXiv …, 2024

With recent advances in artificial intelligence (AI) and robotics, unmanned vehicle swarms have received great attention from both academia and industry due to their potential to provide services that are difficult and dangerous to perform by humans …


Generative AI from Theory to Practice: A Case Study of Financial Advice

We identify some of the most pressing issues facing the adoption of large language models (LLMs) in practical settings and propose a research agenda to reach the next technological inflection point in generative AI. We focus on three challenges …


Generative AI in Education From the Perspective of Students, Educators, and Administrators

This research explores how advanced artificial intelligence (AI), like the technology that powers tools such as ChatGPT, is changing the way we teach and learn in schools and universities. Imagine AI helping to summarize thick legal documents …


Generative AI

… • CommonCrawl News: This is a dataset of news stories from websites across the globe. …


Generative AI, Plagiarism, and Copyright Infringement in Legal Documents

AB Cyphert - Minnesota Journal of Law, Science & Technology, 2024

… OpenAI acknowledged in the GPT-3 research paper that GPT-3 was trained on the Common Crawl dataset, “which includes everything from traditional news sites like the New York Times to sites like Reddit.” Cyphert, supra note 1, at 407, citing Liz …


Generative AI-Based Text Generation Methods Using Pre-Trained GPT-2 Model

R Pandey, H Waghela, S Rakshit, A Rangari, A Singh… - arXiv preprint arXiv …, 2024

This work delved into the realm of automatic text generation, exploring a variety of techniques ranging from traditional deterministic approaches to more modern stochastic methods. Through analysis of greedy search, beam search, top-k …


Generative Artificial Intelligence: Models, Benefits, Dangers and Detection of AI-Generated Text on Specialized Domains

… However, Radford et al. found that a large amount of document within the hashed pages of Common Crawl was unintelligible. A new approach could be to simply use a curated subset of Common Crawl, but they also created a web scrape from Reddit …


Generative Knowledge Management for Financial Inclusion Through Financial Literacy: A Systematic Review.

S Vadari, C Malladi - IUP Journal of Knowledge Management, 2024

… OpenAI utilized an openly available corpus of webpages called Common Crawl, which in turn derives from a bot that crawls the entire Internet. This Common Crawl dataset contains billions of webpages and is one of the colossal textual datasets …


GeniL: A Multilingual Dataset on Generalizing Language

AM Davani, S Gubbi, S Dev, S Dave, V Prabhakaran - arXiv preprint arXiv …, 2024

… For each language, we queried the Multilingual Common Crawl (mC4) language corpus to collect naturally occurring sentences that contain both the terms in the tuples present in SGM. To ensure a diverse representation of different associations …


German Text Embedding Clustering Benchmark

S Wehrli, B Arnrich, C Irrgang - arXiv preprint arXiv:2401.02709, 2024

This work introduces a benchmark assessing the performance of clustering German text embeddings in different domains. This benchmark is driven by the increasing use of clustering neural text embeddings in tasks that require the grouping of texts (such …


GFWeb: Measuring the Great Firewall's Web Censorship at Scale

NP Hoang, J Dalek, M Crete-Nishihata, N Christin…

… Aggregating the ranking information provided by the Tranco list [62] and the Common Crawl dataset [1], we analyze the popularity of the base censored domains discovered by GFWeb for HTTP(S) filters in comparison to the GFWatch (DNS filter) …


Ghost Sentence: A Tool for Everyday Users to Copyright Data from Large Language Models

S Zhao, L Zhu, R Quan, Y Yang - arXiv preprint arXiv:2403.15740, 2024

Web user data plays a central role in the ecosystem of pre-trained large language models (LLMs) and their fine-tuned variants. Billions of data are crawled from the web and fed to LLMs. How can \textit{\textbf{everyday web users}} confirm if LLMs …


GPTs: Concerns, Limitations and (Some) Responses

… The ones of these that are large and containing substantial source material from the Internet (eg from Common Crawl) will have some harms of representation bias. Then, if the training data has bias then so too will LLMs. To address this, there seem …


Grammatical versus Spelling Error Correction: An Investigation into the Responsiveness of Transformer-Based Language Models Using BART and MarianMT

R Raju, PB Pati, SA Gandheesh, GS Sannala… - Journal of Information & …, 2024

Text continues to remain a relevant form of representation for information. Text documents are created either in digital native platforms or through the conversion of other media files such as images and speech. While the digital native text is …


Grammatical vs Spelling Error Correction: An Investigation into the Responsiveness of Transformer-based Language Models using BART and MarianMT

R Raju, PB Pati, SA Gandheesh, GS Sannala, S KS - arXiv preprint arXiv:2403.16655, 2024

… C4 Dataset is an Open-Source dataset obtained with Common Crawl web scrape [30]. This dataset contains millions of collected sentences along with their target sentences. The collected statements, referred to as input sentences in this work …


Granite Code Models: A Family of Open Foundation Models for Code Intelligence

M Mishra, M Stallone, G Zhang, Y Shen, A Prasad… - arXiv preprint arXiv …, 2024

Large Language Models (LLMs) trained on code are revolutionizing the software development process. Increasingly, code LLMs are being integrated into software development environments to improve the productivity of human programmers, and …


Hallucinating (or poorly fed) LLMs? The problem of data accuracy

… This is the case of “Common Crawl”, which consists in a massive collection of web pages and … To give a picture, the Common Crawl dataset constitutes nearly a trillion words. … were publicly available online and memorised in the Common …


Harder Tasks Need More Experts: Dynamic Routing in MoE Models

Q Huang, Z An, N Zhuang, M Tao, C Zhang, Y Jin, K Xu… - arXiv preprint arXiv …, 2024

In this paper, we introduce a novel dynamic expert selection framework for Mixture of Experts (MoE) models, aiming to enhance computational efficiency and model performance by adjusting the number of activated experts based on input difficulty …


Hardware Phi-1.5 B: A Large Language Model Encodes Hardware Domain Specific Knowledge

W Fu, S Li, Y Zhao, H Ma, R Dutta, X Zhang, K Yang…

… The first segment includes data from specialized platforms such as Arxiv , Books, Wikipedia, and StackExchange; the second segment is derived from broader internet content via CommonCrawl and C4. Table II provides a comprehensive …


Harmony in the Australian Domain Space

X Gong, PX McCarthy, MA Rizoiu, P Boldi - arXiv preprint arXiv:2404.10006, 2024

… Though not focused on localized networks, the extensive and all-encompassing web graph from Common Crawl utilized in our study enables the extraction of implicit information on a broader scale. … We utilize the datasets mainly from …


Harnessing Artificial Intelligence in Bariatric Surgery: Comparative Analysis of ChatGPT-4, Bing, and Bard in Generating Clinician-Level Bariatric Surgery …

Y Lee, T Shin, L Tessier, A Javidan, J Jung, D Hong… - Surgery for Obesity and Related …

Background The formulation of clinical recommendations pertaining to bariatric surgery is essential in guiding healthcare professionals. However, the extensive and continuously evolving body of literature in bariatric surgery presents considerable …


Harnessing generative AI for overcoming labeled data challenges in social media NLP

With the introduction of Transformers and Large Language Models, the field of NLP has significantly evolved. Generative AI, a prominent transformer-based technology for crafting human-like content, has proven powerful skills across numerous NLP …


Harnessing the Power of Metadata for Enhanced Question Retrieval in Community Question Answering

S Ghasemi, A Shakery - IEEE Access, 2024

… GPT-3 was trained on an open source dataset called ‘CommonCrawl’ and other texts from OpenAI such as Wikipedia entries. GPT-4 is the newest version of OpenAI’s language model systems. In this article, we fine-tune GPT-2 and GPT-3 pretrained …


HCI International 2024 Posters: 26th International Conference on Human-Computer Interaction, HCII 2024, Washington, DC, USA, June 29–July 4, 2024, Proceedings …

Preliminary scientific results, professional news, or work in progress, described in the form of short research papers (4–11 pages long), constitute a popular submission type among the International Conference on Human-Computer …


HGT: Leveraging Heterogeneous Graph-enhanced Large Language Models for Few-shot Complex Table Understanding

R Jin, Y Li, G Qi, N Hu, YF Li, J Chen, J Wang, Y Chen… - arXiv preprint arXiv …, 2024

Table understanding (TU) has achieved promising advancements, but it faces the challenges of the scarcity of manually labeled tables and the presence of complex table structures.To address these challenges, we propose HGT, a framework with a …


Hierarchical Multimodal Pre-training for Visually Rich Webpage Understanding

H Xu, L Chen, Z Zhao, D Ma, R Cao, Z Zhu, K Yu - arXiv preprint arXiv:2402.18262, 2024

… Common Crawl is a publicly available web crawl dataset that collects webpages from the internet. Instead of using the source code, we collect webpage links from Common Crawl. We traverse all links in a dataset snapshot 4 and categorize and …


Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania

J Rudolph, MF Ismail, S Popenici - Journal of University Teaching and Learning …, 2024

Higher education is currently under a significant transformation due to the emergence of generative artificial intelligence (GenAI) technologies, the hype surrounding GenAI and the increasing influence of educational technology business …


Historical Dutch Spelling Normalization with Pretrained Language Models

A Wolters, A Van Cranenburgh - Computational Linguistics in the Netherlands …, 2024

… These texts are from the common crawl corpus, which contains petabytes of data scraped from the internet. Further pretraining on in-domain data can make the model more suited for the downstream task it will be finetuned for, and since our task …


History, Development, and Principles of Large Language Models-An Introductory Survey

Z Chu, S Ni, Z Wang, X Feng, C Li, X Hu, R Xu, M Yang… - arXiv preprint arXiv …, 2024

Language models serve as a cornerstone in natural language processing (NLP), utilizing mathematical methods to generalize language laws and knowledge for prediction and generation. Over extensive research spanning decades, language …


Holographic Embeddings for Text and Graphs A Master's Thesis Presented to The Faculty of the Graduate School of Arts and Sciences

… Static word embeddings such as word2vec, Global Vectors for Word Representation (GloVe), and fastText are trained on large amounts of natural language text sources like Common Crawl and Wikipedia dumps. These models …


Homonym Sense Disambiguation in the Georgian Language

D Melikidze, A Gamkrelidze - arXiv preprint arXiv:2405.00710, 2024

This research proposes a novel approach to the Word Sense Disambiguation (WSD) task in the Georgian language, based on supervised fine-tuning of a pre-trained Large Language Model (LLM) on a dataset formed by filtering the Georgian …


How Do We Learn What We Cannot Say?

The contributions of this thesis are two-fold. First, this thesis presents UDTube, an easily usable software developed to perform morphological analysis in a multi-task fashion. This work shows the strong performance of UDTube versus the current state-of-the-art …


How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites

Z Chen, W Wang, H Tian, S Ye, Z Gao, E Cui, W Tong… - arXiv preprint arXiv …, 2024

In this report, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple …


How Gender Interacts with Political Values: A Case Study on Czech BERT Models

AA Ali, J Libovický - arXiv preprint arXiv:2403.13514, 2024

Neural language models, which reach state-of-the-art results on most natural language processing tasks, are trained on large text corpora that inevitably contain value-burdened content and often capture undesirable biases, which the models …


How Large Corpora Sizes Influence the Distribution of Low Frequency Text n-grams

JF Silva, JC Cunha - Pacific-Asia Conference on Knowledge Discovery and …, 2024

The prediction of the numbers of distinct word n-grams and their frequency distributions in text corpora is important in domains like information processing and language modelling. With big data corpora, there is an increased application …


How Lexical is Bilingual Lexicon Induction?

H Kohli, H Feng, N Dronen, C McCarter, S Moeini… - arXiv preprint arXiv …, 2024

… derived from Common Crawl and Wikipedia. The plot visualizes the Spearman’s Rank correlation of term frequencies between each of the source (by row) and target (by column) language pairs in the 5k vocabularies in the XLING corpus derived from …


How many news websites block AI crawlers?

In this factsheet we describe the proportion of news websites in ten countries that block AI (artificial intelligence) crawlers. We find that, (i) by the end 2023, 48% of the most widely used news websites across ten countries were blocking OpenAI’s (ChatGPT) …


How Much are LLMs Contaminated? A Comprehensive Survey and the LLMSanitize Library

M Ravaut, B Ding, F Jiao, H Chen, X Li, R Zhao, C Qin… - arXiv preprint arXiv …, 2024

… This includes policies and protocols for data privacy, consent, and use that help prevent the incorporation of contaminated data from unethical sources and the contamination of widely used pre-training data sources (eg, CommonCrawl) …


How to Train Data-Efficient LLMs

N Sachdeva, B Coleman, WC Kang, J Ni, L Hong… - arXiv preprint arXiv …, 2024

The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, ie, techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We …


HPLT's First Release of Data and Models

N Arefyev, M Aulamo, P Chen, O de Gibert, B Haddow… - 2024

… Datasets For the first release, we processed 1.85 petabytes of the Internet Archive and CommonCrawl to create monolingual and parallel corpora. We release them under the permissive CC0 licence1 through our project website2, OPUS3, and Hugging …


Human-Centered Interaction in Virtual Worlds: A New Era of Generative Artificial Intelligence and Metaverse

Y Wang, L Wang, KL Siau - International Journal of Human–Computer Interaction, 2024

The metaverse has emerged as an exciting new paradigm for human-computer interaction (HCI) and virtual collaboration. This paper presents a comprehensive review of the metaverse to address the gap in the existing literature where there is a …


Human-Centered Natural Language Processing for Countering Misinformation

As curbing the spread of online misinformation has proven to be challenging, we look to artificial intelligence (AI) and natural language technology for helping individuals and society counter and limit it. Despite current advances, state-of-the-art …


Hybrid semantic models for building smart and robust home robots

The creation of home robots that can aid human beings in daily mundane chores has been a long-standing goal of robotics research. Some common indoor tasks that service robots can help with include retrieving objects from different locations …


HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts

W Kim, S Chun, T Kim, D Han, S Yun - arXiv preprint arXiv:2404.17507, 2024

In an era where the volume of data drives the effectiveness of self-supervised learning, the specificity and clarity of data semantics play a crucial role in model training. Addressing this, we introduce HYPerbolic Entailment filtering (HYPE), a …


Hypertext Entity Extraction in Webpage

Y Yang, T Liu, B Shao, H Zhao, L Shou, M Gong… - arXiv preprint arXiv …, 2024

… Common Crawl5 is a widely used large-scale dataset for IE. However, they all overlook rich hypertext features and contain too much noise. Moreover, their annotations are often determined by rules or automated tools, lacking accurate manual annotations. …


I'm In! A Comparative Analysis of Business Pitch Competition

E Barron, I Bernatovic, LE Ruiz - DIANA INTERNATIONAL RESEARCH …, 2023

Methods Based on a perceived theoretical gap in research, entrepreneurship scholars have called for increased research to understand how institutions (including governments and academia), influence the construction of intersectional …


Identification of patients' smoking status using an explainable AI approach: a Danish electronic health records case study

A Ebrahimi, MBH Henriksen, CL Brasen, O Hilberg… - BMC Medical Research …, 2024

Smoking is a critical risk factor responsible for over eight million annual deaths worldwide. It is essential to obtain information on smoking habits to advance research and implement preventive measures such as screening of high-risk …


iDRAMA-rumble-2024: A Dataset of Podcasts from Rumble Spanning 2020 to 2022

U Balci, J Patel, B Balci, J Blackburn - 2024

Rumble has emerged as a prominent platform hosting controversial figures facing restrictions on YouTube. Despite this, the academic community’s engagement with Rumble has been minimal. To help researchers address this gap, we introduce a …


IGOT: Information Gain Optimized Tokenizer on Domain Adaptive Pretraining

D Feng, Y Zhang, Z Xu - arXiv preprint arXiv:2405.09857, 2024

Pretrained Large Language Models (LLM) such as ChatGPT, Claude, etc. have demonstrated strong capabilities in various fields of natural language generation. However, there are still many problems when using LLM in specialized domain-specific …


Impact of COVID-19 Pandemic on Social Determinants of Health Issues of Marginalized Black and Asian Communities: A Social Media Analysis Empowered by …

C Whitfield, Y Liu, M Anwar - Journal of Racial and Ethnic Health Disparities, 2024

Purpose This study aims to understand the impact of the COVID-19 pandemic on social determinants of health (SDOH) of marginalized racial/ethnic US population groups, specifically African Americans and Asians, by leveraging natural language …


Improved methodology for longitudinal Web analytics using Common Crawl

HS Thompson - 16th ACM Web Science Conference 2024, 2024

Common Crawl is a multi-petabyte longitudinal dataset containing over 100 billion web pages which is widely used as a source of language data for sequence model training and in web science research. Each of its constituent archives is on the order …


Improving Bengali and Hindi Large Language Models

A Shahriar, D Barbosa - Proceedings of the 2024 Joint International Conference …, 2024

Despite being widely spoken worldwide, Bengali and Hindi are low-resource languages. The state-of-the-art in modeling such languages uses BERT and the Wordpiece tokenizer. We observed that the Wordpiece tokenizer often breaks words …


Improving Cross-lingual Representation for Semantic Retrieval with Code-switching

M Maimaiti, Y Zheng, J Zhang, F Huang, Y Zhang… - arXiv preprint arXiv …, 2024

Semantic Retrieval (SR) has become an indispensable part of the FAQ system in the task-oriented question-answering (QA) dialogue scenario. The demands for a cross-lingual smart-customer-service system for an e-commerce platform or some particular …


INCLURE: a Dataset and Toolkit for Inclusive French Translation

P Lerner, C Grouin - The 17th Workshop on Building and Using …, 2024

… Indeed, this corpus mostly contains transcripts of political speeches, whose oral style differs from the text typically found in OSCAR/CommonCrawl. Exceptions are six examples used to illustrate the use of the inclusive neutralization process …


Incorporating Geo-Diverse Knowledge into Prompting for Increased Geographical Robustness in Object Recognition

K Buettner, S Malakouti, XL Li, A Kovashka - arXiv preprint arXiv:2401.01482, 2024

Existing object recognition models have been shown to lack robustness in diverse geographical scenarios due to significant domain shifts in design and context. Class representations need to be adapted to more accurately reflect an object concept …


IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context

NR Sahoo, PP Kulkarni, N Asad, A Ahmad, T Goyal… - arXiv preprint arXiv …, 2024

The pervasive influence of social biases in language data has sparked the need for benchmark datasets that capture and evaluate these biases in Large Language Models (LLMs). Existing efforts predominantly focus on English language and the …


IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages

MSUR Khan, P Mehta, A Sankar, U Kumaravelan… - arXiv preprint arXiv …, 2024

Despite the considerable advancements in English LLMs, the progress in building comparable models for other languages has been hindered due to the scarcity of tailored resources. Our work aims to bridge this divide by introducing an expansive …


Indiscriminate Data Poisoning Attacks on Pre-trained Feature Extractors

Y Lu, MYR Yang, G Kamath, Y Yu - arXiv preprint arXiv:2402.12626, 2024

… To collect such large-scale datasets, practitioners usually extract the desired data by crawling on the internet (eg, web pages using Common Crawl1). However, using outsourced data raises an imminent security risk (Kumar et al. 2020; Nelson et al …


Individual-vs. Multiple-Objective Strategies for Targeted Sentiment Analysis in Finances Using the Spanish MTSA 2023 Corpus

R Pan, JA García-Díaz, R Valencia-García - Electronics, 2024

… complexity; (vi) RoBERTuito [22], a pretrained language model for user-generated content in Spanish, trained following RoBERTa guidelines on 500 million tweets; and (vii) XLM-RoBERTa [17], a multilingual version of RoBERTa, trained with data …


IndoBerea: Evolving Semantic Search in Theological Context

FVP Samosir, S Mendrofa - 2023 Eighth International Conference on Informatics …, 2023

This paper presents IndoBerea, a semantic search model pre-trained on an Indonesian Bible dataset and based on SentenceTransformers and IndoBERT. It aims to enhance theological research by providing contextually relevant verses in …


INDUS: Effective and Efficient Language Models for Scientific Applications

B Bhattacharjee, A Trivedi, M Muraoka… - arXiv preprint arXiv …, 2024

Large language models (LLMs) trained on general domain corpora showed remarkable results on natural language processing (NLP) tasks. However, previous research demonstrated LLMs trained using domain-focused corpora perform better …


Inference-Based No-Learning Approach on Pre-Trained BERT Model Retrieval

HL Pham, R Mibayashi, T Yamamoto, MP Kato… - 2024 IEEE International …, 2024

In recent years, the practice of leveraging pre-trained machine learning models for specific tasks has gained traction. Instead of training models from the ground up, it is now common to fine-tune existing pre-trained models. However, users have the …


Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks

N Yax, PY Oudeyer, S Palminteri - arXiv preprint arXiv:2404.04671, 2024

This paper introduces PhyloLM, a method applying phylogenetic algorithms to Large Language Models to explore their finetuning relationships, and predict their performance characteristics. By leveraging the phylogenetic distance metric, we …


Information Retrieval Chatbot on Military Policies and Standards

C Gunasekara, A Sharafeldin, M Triff, Z Kabir…

In the Canadian Armed Forces (CAF), navigating through extensive policies and standards can be a challenging task. To address the need for streamlined access to these vital documents, this paper explores the usage of artificial intelligence (AI) and …


Information Retrieval with Dense and Sparse Representations

… However, they use 1TB of the training data from Common Crawl dumps while our model only use 115MB of the Wikipedia data for pretraining. We put their scores in Table 2.2 for reference. In SimCSE, the authors propose to use MLM as an auxiliary …


Innovative Use of Self-Attention-Based Ensemble Deep Learning for Suicide Risk Detection in Social Media Posts

HS Choi, J Yang - Applied Sciences, 2024

… Specifically, we employed the extensive corpus variant, known as “Common Crawl (840 B tokens, 2.2 M vocab, cased, 300 d vectors)”, which encompasses an impressive 840 billion tokens, a vocabulary of 2.2 million cased terms, and provides …


Instruction-tuned Language Models are Better Knowledge Learners

Z Jiang, Z Sun, W Shi, P Rodriguez, C Zhou, G Neubig… - arXiv preprint arXiv …, 2024

… However, its scope is limited to Wikipedia, which restricts the trained models’ adaptability to other sources like web pages from Common Crawl or scientific documents from arXiv. We focus on eliciting factual knowledge with instruction-tuning …


Integrating Generative AI Literacy into the Information Retrieval Course at a university in Canada: towards critical evaluation of online search results

… • Trained on information (billions of words) on the open web prior to 2021 • Among other open sources, dataset comes from Common Crawl (crawls the web) and Wikipedia • Length of answers limited - between 500 and 700 words, “leaving …


Intelligent Pharmacy

The modern language generation model ChatGPT, created by Open Artificial Intelligence (AI), is recognised for its capacity to comprehend context and produce pertinent content. This model is built on the transformer architecture, which enables …


Intelligent Phishing Website Detection Model Powered by Deep Learning Techniques

U Chetachi, O Henry, OA Agbugba - Asian Journal of Research in Computer …, 2024

… Approximately one million authentic and fraudulent URLs were employed in the dataset gathered from PhishTank and Common Crawl. Over 10,000 images and one million URLs were used in training for the CNN classifier and LSTM to create the …


Intercity relationships between 293 Chinese cities quantified based on toponym co-occurrence

W Tongjing, Z Yin, Z Bao, E Meijers - Cybergeo: European Journal of Geography, 2024

… 14The primary dataset of this study was obtained from the Common Crawl, a web archive that has periodically crawled the Internet since … We used the entire Common Crawl text archive from April 2019 for processing and conducting …


International Journal of Cognitive Computing in Engineering

… MC4 dataset contains 101 language variants drawn from the Common Crawl web scrape. MT5 did not deviates much from the original T5 model and attempted to extend the capacities of the T5 to multilingual settings. T5 is a pre-trained language …


Internet-scale Topic Modeling using Large Language Models

… This proportion is estimated from the Common Crawl dataset’s statistics [28]. In short, the model should have an excellent understanding of English, an adequate understanding of languages such as German and Spanish, and be satisfactory at …


InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning

H Ying, S Zhang, L Li, Z Zhou, Y Shao, Z Fei, Y Ma… - arXiv preprint arXiv …, 2024

The math abilities of large language models can represent their abstract reasoning ability. In this paper, we introduce and open-source our math reasoning LLMs InternLM-Math which is continue pre-trained from InternLM2. We unify chain-of-thought …


InternLM2 Technical Report

Z Cai, M Cao, H Chen, K Chen, K Chen, X Chen… - arXiv preprint arXiv …, 2024

… Our web page data mainly comes from Common Crawl1. Firstly, we need to decompress the original Warc format files and use Trafilatura (… The entire filtering process eliminates a large proportion of Web page data (Common Crawl) and Patent …


Interpretable Tensor Fusion

S Varshneya, A Ledent, P Liznerski, A Balinskyy… - arXiv preprint arXiv …, 2024

Conventional machine learning methods are predominantly designed to predict outcomes based on a single data type. However, practical applications may encompass data of diverse types, such as text, images, and audio. We introduce …


Interpreting Themes from Educational Stories

Y Zhang, FA González, T Solorio - arXiv preprint arXiv:2404.05250, 2024

Reading comprehension continues to be a crucial research focus in the NLP community. Recent advances in Machine Reading Comprehension (MRC) have mostly centered on literal comprehension, referring to the surface-level …


Intersectional Male-Centric and White-Centric Biases in Collective Concepts

AH Bailey, A Williams, A Poddar, A Cimpian - 2024

… Here we used NLP tools (specifically, static word embeddings trained on the Common Crawl corpus) to investigate collective understanding of three fundamental concepts: PERSON, WOMAN, and MAN.These three concepts organize much of …


Introducing the Djinni Recruitment Dataset: A Corpus of Anonymized CVs and Job Postings

N Drushchak, M Romanyshyn - Proceedings of the Third Ukrainian Natural …, 2024

This paper introduces the Djinni Recruitment Dataset, a large-scale open-source corpus of candidate profiles and job descriptions. With over 150,000 jobs and 230,000 candidates, the dataset includes samples in English and Ukrainian, thereby …


Investigating Complex Answer Attribution Approaches with Large Language Models

This master’s thesis explores the attribution of answers in complex question-answering scenarios utilizing large language models (LLMs). The research aims to assess and enhance the traceability of answers back to their source documents, a critical aspect …


Investigating Gender Bias in Turkish Language Models

O Caglidil, M Ostendorff, G Rehm - arXiv preprint arXiv:2404.11726, 2024

… an analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


Investigating the pre-training bias in low-resource abstractive summarization

D Chernyshev, B Dobrov - IEEE Access, 2024

Recent advances in low-resource abstractive summarization were largely made through the adoption of specialized pre-training, pseudo-summarization, that integrates the content selection knowledge through various centrality-based …


IPA Transcription of Bengali Texts

K Fatema, FD Haider, NF Turpa, T Azmal, S Ahmed… - arXiv preprint arXiv …, 2024

The International Phonetic Alphabet (IPA) serves to systematize phonemes in language, enabling precise textual representation of pronunciation. In Bengali phonology and phonetics, ongoing scholarly deliberations persist concerning the …


IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models

DI Adelani, J Ojo, IA Azime, JY Zhuang, JO Alabi, X He… - arXiv preprint arXiv …, 2024

Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (eg African languages) are often …


Is AI changing learning and assessment as we know it? Evidence from a ChatGPT experiment and a conceptual framework

O Kolade, A Owoseni, A Egbetokun - Heliyon, 2024

ChatGPT, a state-of-the-art chatbot built upon Open AI's generative pre-trained transformer, has generated a major public interest and caused quite a stir in the higher education sector, where reactions have ranged from excitement to …


Is ChatGPT taking over the language classroom? How language ideologies of large language models impact teaching and learning

M Lau - Working papers in Applied Linguistics and Linguistics …, 2024

… Thus, using datasets such as the Common Crawl can lead to the over-representation of English and the perspectives of English-speaking content … This means that LLMs that use Common Crawl for training, such as ChatGPT, may perform with …


Is Google Getting Worse? A Longitudinal Investigation of SEO Spam in Search Engines

J Bevendorff, M Wiegmann, M Potthast, B Stein

Many users of web search engines have been complaining in recent years about the supposedly decreasing quality of search results. This is often attributed to an increasing amount of search-engineoptimized but low-quality content. Evidence for …


Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models

C Liu, W Zhang, Y Zhao, AT Luu, L Bing - arXiv preprint arXiv:2403.10258, 2024

… We categorize languages larger than 1% frequency in Common Crawl2 as high-resource languages (ie, de, ru, fr, zh, es, ja, it and vi), and the rest as low-resource languages. We exclude English since we want to evaluate the efficient prompting strategy for …


It's About Time: Incorporating Temporality in Retrieval Augmented Language Models

A Gade, J Jetcheva - arXiv preprint arXiv:2401.13222, 2024

… Additionally both models are fine-tuned jointly as a Fusion-in-Decoder [3] on common-crawl data using Wikipedia as a document index. The authors showed that Atlas performs well out-of-box and can be adapted to different tasks such as question-answering …


IUST at ClimateActivism 2024: Towards Optimal Stance Detection: A Systematic Study of Architectural Choices and Data Cleaning Techniques

G Mahmoudi, S Eetemadi - Proceedings of the 7th Workshop on Challenges and …, 2024

This work presents a systematic search of various model architecture configurations and data cleaning methods. The study evaluates the impact of data cleaning methods on the obtained results. Additionally, we demonstrate that a combination of …


Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agent

Q Gallouédec, E Beeching, C Romac, E Dellandréa - arXiv preprint arXiv:2402.09844, 2024

The search for a general model that can operate seamlessly across multiple domains remains a key goal in machine learning research. The prevailing methodology in Reinforcement Learning (RL) typically limits models to a single task …


Japanese Pointer Network based Entity Linker for Wikidata

Y Sawamura, T Morita, S Egami, T Ugai, K Fukuda - 2023

… Pre-trained fastText is a model trained on Common Crawl and Wikipedia and provides pre-computed vectors for a word. The length of the POS Tags was changed from 36 to 18. The other five embeddings, namely Entity Embedding, Text Match …


JetMoE: Reaching Llama2 Performance with 0.1 M Dollars

Y Shen, Z Guo, T Cai, Z Qin - arXiv preprint arXiv:2404.07413, 2024

Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to the development of powerful and accessible super-human intelligence. This report introduces JetMoE-8B …


Jina CLIP: Your CLIP Model Is Also Your Text Retriever

A Koukounas, G Mastrapas, M Günther, B Wang… - arXiv preprint arXiv …, 2024

Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information retrieval and related tasks …


JiuZhang3. 0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models

K Zhou, B Zhang, J Wang, Z Chen, WX Zhao, J Sha… - arXiv preprint arXiv …, 2024

Mathematical reasoning is an important capability of large language models~(LLMs) for real-world applications. To enhance this capability, existing work either collects large-scale math-related texts for pre-training, or relies on stronger LLMs (\eg GPT-4) …


JLBert: Japanese Light BERT for Cross-Domain Short Text Classification

C Kayal, S Chattopadhyay, A Gupta, S Abrol, A Gugol - Proceedings of the 2024 Joint …, 2024

… Second, there are limited models that have been trained on diverse datasets beyond Wikipedia and Common Crawl, limiting experimentation with other forms of text data. Moreover, with the English language being the focus of research interest …


Juru: Legal Brazilian Large Language Model from Reputable Sources

RM Junior, R Pires, R Romero, R Nogueira - arXiv preprint arXiv:2403.18140, 2024

The high computational cost associated with pretraining large language models limits their research. Two strategies have emerged to address this issue: domain specialization and pretraining with high-quality data. To explore these strategies, we …


Just Rewrite It Again: A Post-Processing Method for Enhanced Semantic Similarity and Privacy Preservation of Differentially Private Rewritten Text

S Meisenbacher, F Matthes - arXiv preprint arXiv:2405.19831, 2024

The study of Differential Privacy (DP) in Natural Language Processing often views the task of text privatization as a $\textit{rewriting}$ task, in which sensitive input texts are rewritten to hide explicit or implicit private information. In order to evaluate the …