Magnitude Pruning Outperforms Variational Dropout

Related work found that MP can outperform VD, especially for moderate sparsity ratios #REFR . Our results confirm that on population level.

MP outperforms VD for sparsification levels of up to 80% consistently, see Figure   5 , Appendix B and C.

At higher sparsification levels, MP shows steep drops in performance.

VD on some zoos is more stable and thus shows higher performance at higher sparsification levels, justifying the larger parameter count and computational load.

Sparsified Model Zoo Twins: Investigating Populations Of Sparsified Neural Network Models

Abstract: With growing size of Neural Networks (NNs), model sparsification to reduce the computational cost and memory demand for model inference has become of vital interest for both research and production. While many sparsification methods have been proposed and successfully applied on individual models, to the best of our knowledge their behavior and robustness has not yet been studied on large populations of models. With this paper, we address that gap by applying two popular sparsification methods on populations of models (so called model zoos) to create sparsified versions of the original zoos. We investigate the performance of these two methods for each zoo, compare sparsification layer-wise, and analyse agreement between original and sparsified populations. We find both methods to be very robust with magnitude pruning able outperform variational dropout with the exception of high sparsification ratios above 80%. Further, we find sparsified models agree to a high degree with their original non-sparsified counterpart, and that the performance of original and sparsified model is highly correlated. Finally, all models of the model zoos and their sparsified model twins are publicly available: modelzoos.cc.

The State Of Sparsity In Deep Neural Networks

Abstract: We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017; Louizos et al., 2017b) shown to yield high compression rates on smaller datasets perform inconsistently, and that simple magnitude pruning approaches achieve comparable or better results. Additionally, we replicate the experiments performed by (Frankle & Carbin, 2018) and (Liu et al., 2018) at scale and show that unstructured sparse architectures learned through pruning cannot be trained from scratch to the same test set performance as a model trained with joint sparsification and optimization. Together, these results highlight the need for large-scale benchmarks in the field of model compression. We open-source our code, top performing model checkpoints, and results of all hyperparameter configurations to establish rigorous baselines for future work on compression and sparsification.


B. Hardware And Hyperparameter Configurations

In addition, for a medium dataset like QM8, a medium discriminator dimension, such as [64, 128] , 64, [128, #REFR or slightly smaller, is preferred to generate the molecules that have considerable metric values.

On the other hand, training a higher complexity model over a small dataset may lead to mode collapse, which reduces the values of QED, Uniqueness, and LogP.

Thus, there is a tradeoff among Validity and QED/Uniqueness/LogP/Similarity, and the tradeoff can be adjusted by changing the discriminator For QM8, which is considered a medium dataset that has more compounds than ESOL but less than QM9, we observe the results for the IID setting in Table II and find that as the discriminator complexity increases, the values of QED, Diversity, and Validity decrease, i.e., (0.49, 0.46, 0.45), (1.00, 0.99, 0.99), and (78.4, 26.9, 13.7), respectively, but the values of Uniqueness, LogP, and similarity increases, i.e., (18.1, 51.4, 67.9), (0.54, 0.59, 0.62), and (0.0067, 0.0261, 0.0336), respectively.

The tradeoff changes from "between Validity and QED/Uniqueness/LogP" for ESOL into "between QED/Diversity/Validity and Uniqueness/LogP/Similarity" for QM8.

Also, if we observe the results for non-IID in Table III , we can find that as the discriminator complexity increases, only Uniqueness decreases, while QED, Validity, LogP, and Similarity increase.

Hence, we conclude that the tradeoff among different evaluation metrics may change based on the size of the dataset and the data sample distribution among clients.

For QM9, which is considered a large dataset, we can observe from the results that as we increase the discriminator complexity, QED, Validity, and Similarity increase, but Uniqueness reduces in the IID setting as shown in Table II .

In the non-IID setting, as shown in Table III , only Uniqueness decreases, but QED, Validity, LogP, and Similarity increase as we increase the discriminator dimension.

Hence, we can derive the same conclusion that the tradeoff among different metrics may change based on the size of the data set and the data sample distribution among clients.

In addition, for a large dataset like QM9, a large discriminator complexity is preferred to generate the molecules that have considerable metric values.

3) Effect of different client settings on the metrics: In this section, we evaluate how different numbers of clients affect the performance of GraphGANFed.

Graphganfed: A Federated Generative Framework For Graph-Structured Molecules Towards Efficient Drug Discovery

Abstract: Recent advances in deep learning have accelerated its use in various applications, such as cellular image analysis and molecular discovery. In molecular discovery, a generative adversarial network (GAN), which comprises a discriminator to distinguish generated molecules from existing molecules and a generator to generate new molecules, is one of the premier technologies due to its ability to learn from a large molecular data set efficiently and generate novel molecules that preserve similar properties. However, different pharmaceutical companies may be unwilling or unable to share their local data sets due to the geodistributed and sensitive nature of molecular data sets, making it impossible to train GANs in a centralized manner. In this paper, we propose a Graph convolutional network in Generative Adversarial Networks via Federated learning (GraphGANFed) framework, which integrates graph convolutional neural Network (GCN), GAN, and federated learning (FL) as a whole system to generate novel molecules without sharing local data sets. In GraphGANFed, the discriminator is implemented as a GCN to better capture features from molecules represented as molecular graphs, and FL is used to train both the discriminator and generator in a distributive manner to preserve data privacy. Extensive simulations are conducted based on the three benchmark data sets to demonstrate the feasibility and effectiveness of GraphGANFed. The molecules generated by GraphGANFed can achieve high novelty (≈ 100) and diversity (> 0.9). The simulation results also indicate that 1) a lower complexity discriminator model can better avoid mode collapse for a smaller data set, 2) there is a tradeoff among different evaluation metrics, and 3) having the right dropout ratio of the generator and discriminator can avoid mode collapse. Index Terms-Generative adversarial networks, Graph convolutional networks, Federated learning, Drug discovery I. INTRODUCTION The discovery of new organic and inorganic molecules remains a challenge in medicine, chemistry, and materials sciences. Traditional approaches to molecular discovery involve mathematical frameworks derived from related properties calculated from chemical structures with different physical or biological reactions [1, 2]. However, these mathematical frameworks may not fully capture the properties of the chemical structures, limiting the ability to fully explore novel D. Manu and X. Sun are with the

Crystalgan: Learning To Discover Crystallographic Structures With Generative Adversarial Networks

Abstract: Our main motivation is to propose an efficient approach to generate novel multi-element stable chemical compounds that can be used in real world applications. This task can be formulated as a combinatorial problem, and it takes many hours of human experts to construct, and to evaluate new data. Unsupervised learning methods such as Generative Adversarial Networks (GANs) can be efficiently used to produce new data. Cross-domain Generative Adversarial Networks were reported to achieve exciting results in image processing applications. However, in the domain of materials science, there is a need to synthesize data with higher order complexity compared to observed samples, and the state-of-the-art cross-domain GANs can not be adapted directly. In this contribution, we propose a novel GAN called CrystalGAN which generates new chemically stable crystallographic structures with increased domain complexity. We introduce an original architecture, we provide the corresponding loss functions, and we show that the CrystalGAN generates very reasonable data. We illustrate the efficiency of the proposed method on a real original problem of novel hydrides discovery that can be further used in development of hydrogen storage materials.


We further decreased the divergence of the selected clean source samples, which maximized the agreement of the multiple classifiers to achieve better results, similar to #REFR .

The probability that samples x i belong to class k predicted by each classifier are denoted as p k 1 (y|x i ) and p k 2 (y|x i ), respectively.

It is also possible to build these multiple independent classifiers by using dropout regularization #OTHEREFR , #OTHEREFR repeatedly to a single classifier. Section III-F goes into more detail about this.

To solve the noisy label problem of source samples, we calculated the divergence between the multiple classifier outputs for each source sample at the mini-batch level.

The multiple classifiers tend to output similar predictions on clean samples and different predictions on noisy data because the multiple classifiers that are trained individually have varying capacities to learn the noisy label.

In each mini-batch, we chose samples with small divergences in addition to the wellknown small-loss strategy to filter out the noisy samples.

We propose a divergence separation loss for the target samples to address the issues with the source and target private samples.

The target private samples will have larger divergences than the target common samples because they can also be thought of as noisy samples with wrong labels.

Therefore, we can filter out some target private samples to achieve stable performance by separating the divergences of the target samples.

Derived from the existing methods #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR that deploy multiple classifiers with different parameters to achieve domain adaptation, we further used the multiple classifiers as a discriminator to detect the target samples that are far from the cluster of the source domain.

After that, we trained the generator to minimize the divergence to prevent it from generating target features that are not supported by the source samples.

Noisy Universal Domain Adaptation Via Divergence Optimization For Visual Recognition

Abstract: To transfer the knowledge learned from a labeled source domain to an unlabeled target domain, many studies have worked on universal domain adaptation (UniDA), where there is no constraint on the label sets of the source domain and target domain. However, the existing UniDA methods rely on source samples with correct annotations. Due to the limited resources in the real world, it is difficult to obtain a large amount of perfectly clean labeled data in a source domain in some applications. As a result, we propose a novel realistic scenario named Noisy UniDA, in which classifiers are trained using noisy labeled data from the source domain as well as unlabeled domain data from the target domain that has an uncertain class distribution. A multi-head convolutional neural network framework is proposed in this paper to address all of the challenges faced in the Noisy UniDA at once. Our network comprises a single common feature generator and multiple classifiers with various decision bounds. We can detect noisy samples in the source domain, identify unknown classes in the target domain, and align the distribution of the source and target domains by optimizing the divergence between the outputs of the various classifiers. The proposed method outperformed the existing methods in most of the settings after a thorough analysis of the various domain adaption scenarios.

Combating Noisy Labels By Agreement: A Joint Training Method With Co-Regularization

Abstract: Deep Learning with noisy labels is a practically challenging problem in weakly-supervised learning. The state-of-the-art approaches"Decoupling"and"Co-teaching+"claim that the"disagreement"strategy is crucial for alleviating the problem of learning with noisy labels. In this paper, we start from a different perspective and propose a robust learning paradigm called JoCoR, which aims to reduce the diversity of two networks during training. Specifically, we first use two networks to make predictions on the same mini-batch data and calculate a joint loss with Co-Regularization for each training example. Then we select small-loss examples to update the parameters of both two networks simultaneously. Trained by the joint loss, these two networks would be more and more similar due to the effect of Co-Regularization. Extensive experimental results on corrupted data from benchmark datasets including MNIST, CIFAR-10, CIFAR-100 and Clothing1M demonstrate that JoCoR is superior to many state-of-the-art approaches for learning with noisy labels.


Because Of The Challenges Of Limitation In Network Bandwidth

Thanks to the improvements in storage and computing capabilities of edge intelligence devices, most computing tasks can now be completed directly at the edge, making the Mobile Edge Computing (MEC) paradigm the next-generation computing network #REFR .

Shirajum Munir is with the Virginia Modeling, Analysis, and Simulation Center, Department of Electrical and Computer Engineering, Old Dominion University, Suffolk, VA 23435, USA, and also with the Department of Computer Science and Engineering, Kyung Hee University, Yongin-si 17104, Republic of Korea (e-mail: munir@khu.ac.kr).

Le, Avi Deb Raha, and Choong Seon Hong are with the Department of Computer Science and Engineering, School of Computing, Kyung Hee University, Yongin-si 17104, Republic of Korea (e-mail: apurba@khu.ac.kr; quanghuy69@khu.ac.kr; avi@khu.ac.kr; cshong@khu.ac.kr).

Corresponding author: Choong Seon Hong (e-mail: cshong@khu.ac.kr) or the requirements for transmission delay, the traditional cloud computing paradigm that uploads such big data to a cloud centre for data processing can no longer meet these demands #OTHEREFR .

Further, collecting data from distributed devices poses risks and challenges due to the sensitive nature of a large amount of data, as well as regulations such as the General Data Protection Regulation (GDPR) #OTHEREFR in Europe.

Therefore, as edge devices' storage and computing power continue to grow, coupled with concerns about privacy issues, it becomes more attractive to implement edge intelligence in MEC systems in a distributed manner #OTHEREFR .

To this end, Federated learning (FL), as one application of edge computing in distributed machine learning, is first proposed by #OTHEREFR to simultaneously achieve edge intelligence and address privacy concerns.

It trains a global model through the cooperation between local clients and an edge server while keeping the clients' raw data within their respective local environments.

In general, the typical federated training process consists of the following four steps #OTHEREFR : (1) the server chooses a certain network architecture such as Convolutional Neural Network (CNN) as the global model to be optimized and sends it to local clients; (2) the clients update the received the model parameters of the global model based on their local data; #OTHEREFR all clients send their updated model parameters back to the edge server for aggregation; (4) the server averages all the sent parameters as the new global model parameters for the next global round, repeating these four steps until convergence.

Mp-Fedcl: Multi-Prototype Federated Contrastive Learning For Edge Intelligence

Abstract: Federated learning-assisted edge intelligence enables privacy protection in modern intelligent services. However, not Independent and Identically Distributed (non-IID) distribution among edge clients can impair the local model performance. The existing single prototype-based strategy represents a sample by using the mean of the feature space. However, feature spaces are usually not clustered, and a single prototype may not represent a sample well. Motivated by this, this paper proposes a multiprototype federated contrastive learning approach (MP-FedCL) which demonstrates the effectiveness of using a multi-prototype strategy over a single-prototype under non-IID settings, including both label and feature skewness. Specifically, a multi-prototype computation strategy based on k-means is first proposed to capture different embedding representations for each class space, using multiple prototypes (k centroids) to represent a class in the embedding space. In each global round, the computed multiple prototypes and their respective model parameters are sent to the edge server for aggregation into a global prototype pool, which is then sent back to all clients to guide their local training. Finally, local training for each client minimizes their own supervised learning tasks and learns from shared prototypes in the global prototype pool through supervised contrastive learning, which encourages them to learn knowledge related to their own class from others and reduces the absorption of unrelated knowledge in each global iteration. Experimental results on MNIST, Digit-5, Office-10, and DomainNet show that our method outperforms multiple baselines, with an average test accuracy improvement of about 4.6% and 10.4% under feature and label non-IID distributions, respectively.

Edge Intelligence: The Confluence Of Edge Computing And Artificial Intelligence

Abstract: Along with the deepening development in communication technologies and the surge of mobile devices, a brand-new computation paradigm, Edge Computing, is surging in popularity. Meanwhile, ::: Artificial Intelligence (AI) applications are thriving with the breakthroughs in deep learning and the upgrade of hardware architectures. Billions of bytes of data, generated at the network edge, put great demands on data processing and structural optimization. ::: Therefore, there exists a strong demand to integrate Edge Computing and AI, which gives birth to Edge Intelligence. ::: In this article, we divide Edge Intelligence into AI for edge (Intelligence-enabled Edge Computing) and AI on edge (Artificial Intelligence on Edge). ::: The former focuses on providing a more optimal solution to the key concerns in Edge Computing with the help of popular and effective AI technologies while the latter studies how to carry out the entire process of building AI models, i.e., model training and inference, on edge. This article focuses on giving insights into this new inter-disciplinary field from a broader vision and perspective. It discusses the core concepts and the research road-map, which should provide the necessary background for potential future research programs in Edge Intelligence.


ELBA also provides support for the GPU-based -Drop alignment called LOGAN #REFR . LOGAN does not support protein alignment.

Thus, the overlap detection phase has the form of ASA T .

Once the output matrix is formed, PASTIS computes an alignment step on each non-zero, similar to ELBA.

PASTIS has two alignment modes: seedand-extend with -Drop and Smith-Waterman alignment.

Using -Drop, PASTIS initiates the alignment from the -mer match.

Both PASTIS and ELBA defer implementation of -Drop to the Library for Sequence Analysis (SeqAn) C++ library for CPU #OTHEREFR .

Space Efficient Sequence Alignment For Sram-Based Computing: X-Drop On The Graphcore Ipu

Abstract: Dedicated accelerator hardware has become essential for processing AI-based workloads, leading to the rise of novel accelerator architectures. Furthermore, fundamental differences in memory architecture and parallelism have made these accelerators targets for scientific computing. The sequence alignment problem is fundamental in bioinformatics; we have implemented the-Drop algorithm, a heuristic method for pairwise alignment that reduces search space, on the Graphcore Intelligence Processor Unit (IPU) accelerator. The-Drop algorithm has an irregular computational pattern, which makes it difficult to accelerate due to load balancing. Here, we introduce a graph-based partitioning and queue-based batch system to improve load balancing. Our implementation achieves 10× speedup over a state-of-the-art GPU implementation and up to 4.65× compared to CPU. In addition, we introduce a memoryrestricted-Drop algorithm that reduces memory footprint by 55× and efficiently uses the IPU's limited low-latency SRAM. This optimization further improves the strong scaling performance by 3.6×.

Logan: High-Performance Gpu-Based X-Drop Long-Read Alignment

Abstract: Pairwise sequence alignment is one of the most computationally intensive kernels in genomic data analysis, accounting for more than 90% of the runtime for key bioinformatics applications. This method is particularly expensive for third-generation sequences due to the high computational cost of analyzing sequences of length between 1Kb and 1Mb. Given the quadratic overhead of exact pairwise algorithms for long alignments, the community primarily relies on approximate algorithms that search only for high-quality alignments and stop early when one is not found. In this work, we present the first GPU optimization of the popular X-drop alignment algorithm, that we named LOGAN. Results show that our high-performance multi-GPU implementation achieves up to 181.6 GCUPS and speed-ups up to 6.6x and 30.7x using 1 and 6 NVIDIA Tesla V100, respectively, over the state-of-the-art software running on two IBM Power9 processors using 168 CPU threads, with equivalent accuracy. We also demonstrate a 2.3x LOGAN speed-up versus ksw2, a state-of-art vectorized algorithm for sequence alignment implemented in minimap2, a long-read mapping software. To highlight the impact of our work on a real-world application, we couple LOGAN with a many-to-many long-read alignment software called BELLA, and demonstrate that our implementation improves the overall BELLA runtime by up to 10.6x. Finally, we adapt the Roofline model for LOGAN and demonstrate that our implementation is near-optimal on the NVIDIA Tesla V100s.


Many users center their concern around the threats from other users, e.g., as a bystander #REFR .

#OTHEREFR discovered that users are concerned about the security issues in losing the authentication token.

Lassak et al.'s study #OTHEREFR indicated that users have misconceptions about how biometric data is stored using FIDO2 biometric authentication.

Both studies showed that users' security perception of authentication does not necessarily match the inherent security.

Recent research also focused on users' security and privacy perception in VR.

VR developers and users felt the lack of privacy due to opaque data collection policies #OTHEREFR .

Users are also concerned about being deceived by the digital content in VR #OTHEREFR .

User authentication for payment. Authentication requirements can differ in different settings.

For example, using a chip card may suffice in a physical store, whereas additional one-time passwords (OTPs) may be required while shopping online #OTHEREFR .

The perceived security of authentication also impact users' adoption and use of payment services.

For example, there is a significant uptake of mobile payments because users associate perceived control and security with user authentication on their devices #OTHEREFR . Voskobojnikov et al.

"I Want The Payment Process To Be Cool": Understanding How Interaction Factors Into Security And Privacy Perception Of Authentication In Virtual Reality

Abstract: Users embrace the rapid development of virtual reality (VR) technology. We are witnessing a widespread adoption of VR technology in more routine settings, such as gaming, social interactions, shopping, and commerce. VR systems access sensitive user data and assets when handling these routine activities, including payment, which raises the need for user authentication in VR. However, there is a limited understanding of how users perceive user authentication in VR, in particular, how users' interaction experiences factor into their perception of security and privacy. Our work adopts a "technology probe" approach to understand this question. We design technology probes of authentication in VR based on existing authentication interactions in both VR and the physical world. Further, we embed these probes in the routine payment of a VR game. Our qualitative analysis reveals that users face unique usability challenges in VR authentication, e.g., in motion control. Such challenges also hinder users from accessing security and privacy accurately in VR authentication. Users' expectations for VR authentication mainly center on improvements in interaction. However, their expectations could appear nonspecific and conflicting. We provide recommendations to accommodate users' expectations and resolve conflicts between usability and security.

Security And Privacy Approaches In Mixed Reality: A Literature Survey

Abstract: Mixed reality (MR) technology development is now gaining momentum due to advances in computer vision, sensor fusion, and realistic display technologies. With most of the research and development focused on delivering the promise of MR, the privacy and security implications of this technology are yet to be thoroughly investigated. This survey article aims to put in to light these risks and to look into the latest security and privacy work on MR. Specifically, we list and review the different protection approaches that have been proposed to ensure user and data security and privacy in MR. We extend the scope to include work on related technologies such as augmented reality, virtual reality, and human-computer interaction as crucial components, if not the origins, of MR, as well as numerous related work from the larger area of mobile devices, wearables, and Internet-of-Things. We highlight the lack of investigation, implementation, and evaluation of data protection approaches in MR. Further challenges and directions on MR security and privacy are also discussed.


For labels, we use a character set for Mandarin and subword units #REFR of vocabulary size 1000 for all other languages.

Following #OTHEREFR , we fine-tune the pre-trained visual student encoder to perform visual speech recognition by attaching a linear layer and a Transformer decoder with attention dimension 256, 4 heads, 6 layers and 2048 linear units.

We apply a joint CTC / attention loss #OTHEREFR .

The beam size and CTC weight are fixed to 40 and 0.1 respectively, as in #OTHEREFR .

Learning Cross-Lingual Visual Speech Representations

Abstract: Cross-lingual self-supervised learning has been a growing research topic in the last few years. However, current works only explored the use of audio signals to create representations. In this work, we study cross-lingual self-supervised visual representation learning. We use the recently-proposed Raw AudioVisual Speech Encoders (RAVEn) framework to pre-train an audiovisual model with unlabelled multilingual data, and then fine-tune the visual model on labelled transcriptions. Our experiments show that: (1) multilingual models with more data outperform monolingual ones, but, when keeping the amount of data fixed, monolingual models tend to reach better performance; (2) multilingual outperforms English-only pre-training; (3) using languages which are more similar yields better results; and (4) fine-tuning on unseen languages is competitive to using the target language in the pre-training set. We hope our study inspires future research on non-English-only speech representation learning.

Sentencepiece: A Simple And Language Independent Subword Tokenizer And Detokenizer For Neural Text Processing

Abstract: This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation. It provides open-source C++ and Python implementations for subword units. While existing subword segmentation tools assume that the input is pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system. We perform a validation experiment of NMT on English-Japanese machine translation, and find that it is possible to achieve comparable accuracy to direct subword training from raw sentences. We also compare the performance of subword training and segmentation with various configurations. SentencePiece is available under the Apache 2 license at https://github.com/google/sentencepiece.


Without any training, Codex has shown near state-of-the-art performance that is comparable with a fine-tuned T5 model #REFR .

Some concurrent works also explored pretrained language models for data querying. Rajkumar et al.

#OTHEREFR The code editor displays SQL command generated by the pretrained code model.

It allows users to edit and re-run the SQL command. (Bottom right) The automatically generated visualization is interactive.

Users can easily export the generated visualization with the embedded menu.

GPT-3 #OTHEREFR and Codex #OTHEREFR on the Spider #OTHEREFR benchmark.

Notably, this work serves as a proof-of-concept and lays a solid foundation for our work.

Binder #OTHEREFR is a pipeline for few-shot data querying that generates SQL by utilizing the Codex API to fill in designated slots.

Note that different from this work, Mirror focuses on building a human-in-the-loop system for data querying and analysis.

Thus, Mirror is orthogonal to Binder and is designed to be generic and compatible with different prompting techniques with minimal modification.

Business intelligence (BI) Tools are a type of data analysis applications that have user-friendly interfaces which allow people to connect to and query data sources, build dashboards to visualize data and share the results.

Mirror: A Natural Language Interface For Data Querying, Summarization, And Visualization

Abstract: We present Mirror, an open-source platform for data exploration and analysis powered by large language models. Mirror offers an intuitive natural language interface for querying databases, and automatically generates executable SQL commands to retrieve relevant data and summarize it in natural language. In addition, users can preview and manually edit the generated SQL commands to ensure the accuracy of their queries. Mirror also generates visualizations to facilitate understanding of the data. Designed with flexibility and human input in mind, Mirror is suitable for both experienced data analysts and non-technical professionals looking to gain insights from their data. 1 CCS CONCEPTS • Information systems → Data mining; • Human-centered computing → Natural language interfaces; Visualization systems and tools; • Computing methodologies → Natural language processing.

Exploring The Limits Of Transfer Learning With A Unified Text-To-Text Transformer

Abstract: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.


Speech-Aware Dialog Systems Technology Challenge

The organizers released a new version of MultiWOZ 2.1 #REFR with user utterances voiced by crowdworkers, as illustrated in Figure 1 .

The lack of recent work on spoken dialogue can be attributed in part to the lack of available datasets.

Track 3 of the Dialog Systems Technology Challenge 11 4 seeks to promote work on spoken dialogue by releasing a spoken version of Multi-WOZ.

This Multi-domain (restaurant, hotel, attraction, taxi, train, hospital and police) Wizard-of-Oz dataset is a large-scale human-human task-oriented conversational corpus commonly used for training and evaluating dialogue state tracking (DST), policy optimization and end-to-end dialogue modeling systems.

The goal of this track is to characterize the performance of DST models in the presence of ASR errors and speech phenomena such as disfluencies.

Despite being widely used by the research community, MultiWOZ has been shown to exhibit an entity bias and a large overlap in the distribution of slot-values between the training and the evaluation sets which can lead to memorization in generative models (Qian et al., 2021).

To encourage generalization, the organizers introduced modifications in the dev and test sets: the values for the slots hotel-name, restaurant-name, train-departure and train-destination were replaced with unseen entities, and time mentions were offset by a constant amount.

User utterances in the dev and test sets are vocalized by crowdworkers.

A speech synthesized version of the training data is also provided in the aim of assessing the validity of such data to mitigate the lack of real spoken conversations.

Two verbatim versions of the dev set are provided to the participants, i.e.

Olisia: A Cascade System For Spoken Dialogue State Tracking

Abstract: Though Dialogue State Tracking (DST) is a core component of spoken dialogue systems, recent work on this task mostly deals with chat corpora, disregarding the discrepancies between spoken and written language. In this paper, we propose OLISIA, a cascade system which integrates an Automatic Speech Recognition (ASR) model and a DST model. We introduce several adaptations in the ASR and DST modules to improve integration and robustness to spoken conversations. With these adaptations, our system ranked first in DSTC11 Track 3, a benchmark to evaluate spoken DST. We conduct an in-depth analysis of the results and find that normalizing the ASR outputs and adapting the DST inputs through data augmentation, along with increasing the pre-trained models size all play an important role in reducing the performance discrepancy between written and spoken conversations. 1

Multiwoz 2.1: A Consolidated Multi-Domain Dialogue Dataset With State Corrections And State Tracking Baselines

Abstract: MultiWOZ 2.0 (Budzianowski et al., 2018) is a recently released multi-domain dialogue dataset spanning 7 distinct domains and containing over 10,000 dialogues. Though immensely useful and one of the largest resources of its kind to-date, MultiWOZ 2.0 has a few shortcomings. Firstly, there is substantial noise in the dialogue state annotations and dialogue utterances which negatively impact the performance of state-tracking models. Secondly, follow-up work (Lee et al., 2019) has augmented the original dataset with user dialogue acts. This leads to multiple co-existent versions of the same dataset with minor modifications. In this work we tackle the aforementioned issues by introducing MultiWOZ 2.1. To fix the noisy state annotations, we use crowdsourced workers to re-annotate state and utterances based on the original utterances in the dataset. This correction process results in changes to over 32% of state annotations across 40% of the dialogue turns. In addition, we fix 146 dialogue utterances by canonicalizing slot values in the utterances to the values in the dataset ontology. To address the second problem, we combined the contributions of the follow-up works into MultiWOZ 2.1. Hence, our dataset also includes user dialogue acts as well as multiple slot descriptions per dialogue state slot. We then benchmark a number of state-of-the-art dialogue state tracking models on the MultiWOZ 2.1 dataset and show the joint state tracking performance on the corrected state annotations. We are publicly releasing MultiWOZ 2.1 to the community, hoping that this dataset resource will allow for more effective models across various dialogue subproblems to be built in the future.


For example, traditional ML models can offer a better performance-to-complexity ratio on tabular data than DNNs #REFR . Third, DNNs act in a nontransparent black-box manner.

The popularity of FL caused the development of a plethora of FL frameworks, e.g., Intel ® OpenFL #OTHEREFR , Flower #OTHEREFR , TensorFlow Federated [1] , and HPE Swarm Learning #OTHEREFR to cite a few.

This software only supports one ML model type: Deep Neural Networks (DNNs).

While DNNs have shown unprecedented results across a wide range of applications, from image recognition #OTHEREFR to natural language processing #OTHEREFR , from drug discovery #OTHEREFR to fraud detection #OTHEREFR , they are not the best ML model for every use case.

First, DNNs require massive amounts of data, and collecting and labelling enough high-quality samples is often prohibitive.

Second, DNNs are not the best for all types of data.

This property makes them undesirable when the model's output has to be explained or justified #OTHEREFR .

Lastly, DNNs require high computational resources, and modern security-preserving approaches, e.g. #OTHEREFR , only exacerbate this issues #OTHEREFR .

An FL system leveraging lightweight models could open new possibilities for the industry and allow research where such resources are hardly available.

We propose the MAFL (Model-Agnostic Federated Learning) framework to alleviate these problems.

This framework leverages Ensemble Learning to work with all types of ML models.

Model-Agnostic Federated Learning

Abstract: Since its debut in 2016, Federated Learning (FL) has been tied to the inner workings of Deep Neural Networks (DNNs). On the one hand, this allowed its development and widespread use as DNNs proliferated. On the other hand, it neglected all those scenarios in which using DNNs is not possible or advantageous. The fact that most current FL frameworks only allow training DNNs reinforces this problem. To address the lack of FL solutions for non-DNN-based use cases, we propose MAFL (Model-Agnostic Federated Learning). MAFL marries a modelagnostic FL algorithm, AdaBoost.F, with an open industry-grade FL framework: Intel ® OpenFL. MAFL is the first FL system not tied to any specific type of machine learning model, allowing exploration of FL scenarios beyond DNNs and trees. We test MAFL from multiple points of view, assessing its correctness, flexibility and scaling properties up to 64 nodes. We optimised the base software achieving a 5.5x speedup on a standard FL scenario. MAFL is compatible with x86-64, ARM-v8, Power and RISC-V.

Deep Learning Vs. Traditional Computer Vision

Abstract: Deep Learning has pushed the limits of what was possible in the domain of Digital Image Processing. However, that is not to say that the traditional computer vision techniques which had been undergoing progressive development in years prior to the rise of DL have become obsolete. This paper will analyse the benefits and drawbacks of each approach. The aim of this paper is to promote a discussion on whether knowledge of classical computer vision techniques should be maintained. The paper will also explore how the two sides of computer vision can be combined. Several recent hybrid methodologies are reviewed which have demonstrated the ability to improve computer vision performance and to tackle problems not suited to Deep Learning. For example, combining traditional computer vision techniques with Deep Learning has been popular in emerging domains such as Panoramic Vision and 3D vision for which Deep Learning models have not yet been fully optimised.


However, it has been shown that in practice, minimizing the distortion does not result in perceptually satisfying output for human subjects #REFR .

Lossy compression plays a vital role in the communication and storage of images, videos, and audio data #OTHEREFR - #OTHEREFR .

As the cornerstone of lossy compression, the classical Rate-Distortion (RD) theory #OTHEREFR studies the tradeoff between the bit rate used for representing data and the distortion caused by compression #OTHEREFR .

The reconstruction quality is traditionally measured by a per-letter distortion metric, such as the mean-squared error.

Since high perceptual quality may come at the expense of distortion #OTHEREFR , #OTHEREFR , researchers are motivated to extend the RD theory by bringing perception into account #OTHEREFR - #OTHEREFR .

Blau and Michaeli first proposed and studied the information Rate-Distortion-Perception (RDP) functions in #OTHEREFR .

Theoretical solutions with closed form expressions to the RDP problem are often intractable, except for some special cases such as the Gaussian source with squared error distortion and Wasserstein-2 metric perception #OTHEREFR . Therefore, a computation method for RDP functions is desirable.

Traditionally, the Blahut-Arimoto (BA) algorithm #OTHEREFR , #OTHEREFR has been successful in the computation of capacities and RD functions.

However, to our best knowledge, we have not seen any generalization of the BA algorithm to computing RDP functions.

Computation Of Rate-Distortion-Perception Functions With Wasserstein Barycenter

Abstract: The nascent field of Rate-Distortion-Perception (RDP) theory is seeing a surge of research interest due to the application of machine learning techniques in the area of lossy compression. The information RDP function characterizes the three-way trade-off between description rate, average distortion, and perceptual quality measured by discrepancy between probability distributions. However, computing RDP functions has been a challenge due to the introduction of the perceptual constraint, and existing research often resorts to data-driven methods. In this paper, we show that the information RDP function can be transformed into a Wasserstein Barycenter problem. The nonstrictly convexity brought by the perceptual constraint can be regularized by an entropy regularization term. We prove that the entropy regularized model converges to the original problem. Furthermore, we propose an alternating iteration method based on the Sinkhorn algorithm to numerically solve the regularized optimization problem. Experimental results demonstrate the efficiency and accuracy of the proposed algorithm.

Generative Compression

Abstract: Traditional image and video compression algorithms rely on hand-crafted encoder/decoder pairs (codecs) that lack adaptability and are agnostic to the data being compressed. Here we describe the concept of generative compression, the compression of data using generative models, and show its potential to produce more accurate and visually pleasing reconstructions at much deeper compression levels for both image and video data. We also demonstrate that generative compression is orders-of-magnitude more resilient to bit error rates (e.g. from noisy wireless channels) than traditional variable-length entropy coding schemes.


Getting Started With Two Examples 4.1 A Standard Polynomial System

For inner RoA approximation #REFR , the commands are Here the last argument with value 1 is an optional argument that asks SOStab to also represent the target set in the figure.

It returns the volume of the calculated RoA appproximation and the coefficients of the polynomial variables v k and w k .

Once the optimization is done, the results can be plotted in two dimensions using:

where the first two arguments indicate the indices of the plotted variables (respectively in abscissa and ordinate).

The string "outer" indicates that the outer approximation RoA is plotted.

This gives the plot represented in Figure 1 , which reproduces results presented in [7, Figure 2 ] and [13, Figure 3] .

The 3d-plots of polynomials v k and w k can be displayed with:

VdP.plot_v(1, 2, 'outer'); VdP.plot_w(1, 2, 'outer');

of course, one can also represent the certificates v k and w k obtained in inner approximation, simply by setting the last argument at 'inner'.

Sostab: A Matlab Toolbox For Approximating Regions Of Attraction Of Nonlinear Systems

Abstract: This paper presents a novel Matlab toolbox, aimed at facilitating the use of polynomial optimization for stability analysis of nonlinear systems. Indeed, in the past decade several decisive contributions made it possible to recast the difficult problem of computing stability regions of nonlinear systems, under the form of convex optimization problems that are tractable in modest dimensions. However, these techniques combine sophisticated frameworks such as algebraic geometry, measure theory and mathematical programming, and existing software still requires their user to be fluent in Sum-of-Squares and Moment programming, preventing these techniques from being used more widely in the control community. To address this issue, SOStab entirely automates the writing and solving of optimization problems, and directly outputs relevant data for the user, while requiring minimal input. In particular, no specific knowledge of optimization is needed to use it.

Inner Approximations Of The Region Of Attraction For Polynomial Dynamical Systems

Abstract: In a previous work we developed a convex infinite dimensional linear programming (LP) approach to approximating the region of attraction (ROA) of polynomial dynamical systems subject to compact basic semialgebraic state constraints. Finite dimensional relaxations to the infinite-dimensional LP lead to a truncated moment problem in the primal and a polynomial sum-of-squares problem in the dual. This primal-dual linear matrix inequality (LMI) problem can be solved numerically with standard semidefinite programming solvers, producing a hierarchy of outer (i.e. exterior) approximations of the ROA by polynomial sublevel sets, with a guarantee of almost uniform and set-wise convergence. In this companion paper, we show that our approach is flexible enough to be modified so as to generate a hierarchy of polynomial inner (i.e.\,interior) approximations of the ROA with similar convergence guarantees.


In Figure 10 we show results only for the instance groups for which Hybroscale or TreeChild could output a solution within 1 hour, consistent with the experiments in #REFR .

For sufficiently small instances, we compared the results of our heuristics with the results of two existing tools for reconstructing networks from binary trees: TreeChild #OTHEREFR and Hybroscale #OTHEREFR .

Hybroscale is an exact method performing an exhaustive search on the networks displaying the input trees, therefore it can only handle reasonably small instances in terms of the number of input trees.

TreeChild is a fixed-parameter (in the number of reticulations of the output) exact algorithm that reconstructs the best tree-child network, a restricted class of phylogenetic networks, and due to its fast-growing computation time cannot handle large instances either.

We tested ML and TrivialRand against Hybroscale and TreeChild using the same dataset used in #OTHEREFR , in turn taken from #OTHEREFR .

The dataset consists of ten instances for each possible combination of the parameters L ∈ {10, 20, 30, 40, 50, 60, 80, 100, 150} and |T | ∈ #OTHEREFR .

As a consequence of Hybroscale and TreeChild being exact methods (TreeChild only for a restricted class of networks), they performed better than both ML and TrivialRand on all instances they could solve, although the best results of TrivialRand are often close (no worse than 15%) and sometimes match the optimal value.

The main advantage of our heuristics is that they can handle much larger instances than the exact methods.

In the conference version of this paper #OTHEREFR we showed the results of our heuristics on large real instances, using a ML model trained on 10 networks with at most 100 leaves each.

These results demonstrated that consistently with the simulated data, the machine-learned heuristics gave significantly better results than the randomised ones for the largest instances.

When we first repeated the experiments with the new models trained on 1000 networks with maxL = 100, however, we did not obtain similar results: instead, the results of the randomised heuristics were better or only marginally worse than the machine-learned ones on almost all the instance groups, including the largest.

Constructing Phylogenetic Networks Via Cherry Picking And Machine Learning *

Abstract: Combining a set of phylogenetic trees into a single phylogenetic network that explains all of them is a fundamental challenge in evolutionary studies. Existing methods are computationally expensive and can either handle only small numbers of phylogenetic trees or are limited to severely restricted classes of networks. In this paper, we apply the recently-introduced theoretical framework of cherry picking to design a class of efficient heuristics that are guaranteed to produce a network containing each of the input trees, for datasets consisting of binary trees. Some of the heuristics in this framework are based on the design and training of a machine learning model that captures essential information on the structure of the input trees and guides the algorithms towards better solutions. We also propose simple and fast randomised heuristics that prove to be very effective when run multiple times. Unlike the existing exact methods, our heuristics are applicable to datasets of practical size, and the experimental study we conducted on both simulated and real data shows that these solutions are qualitatively good, always within some small constant factor from the optimum. Moreover, our machine-learned heuristics are one of the first applications of machine learning to phylogenetics and show its promise.

A Practical Fixed-Parameter Algorithm For Constructing Tree-Child Networks From Multiple Binary Trees

Abstract: We present the first fixed-parameter algorithm for constructing a tree-child phylogenetic network that displays an arbitrary number of binary input trees and has the minimum number of reticulations among all such networks. The algorithm uses the recently introduced framework of cherry picking sequences and runs in $O((8k)^k \mathrm{poly}(n, t))$ time, where $n$ is the number of leaves of every tree, $t$ is the number of trees, and $k$ is the reticulation number of the constructed network. Moreover, we provide an efficient parallel implementation of the algorithm and show that it can deal with up to $100$ input trees on a standard desktop computer, thereby providing a major improvement over previous phylogenetic network construction methods.


It operates over a volumetric representation of the environment based on #REFR and functions in a bifurcated architecture of local-and global path planning.

SWAP implements its autonomous exploration functionality by interfacing our previous and open-sourced work on graph-based exploration (GBPlanner) #OTHEREFR .

The method, verified extensively in subterranean and industrial environments, offers efficient exploration within a volume of set bounds assuming no prior map knowledge.

At the local stage the method exploits a dense random graph G L E around the robot to identify collisionfree paths maximizing volumetric exploration.

Simultaneously, as such local steps take place, the algorithm builds a sparse global graph G G E , used by the global stage that is invoked when local exploration reports inability to find a path of significant gain or when the robot approaches its endurance limits.

Accordingly, the method offers repositioning to previously detected unexploited frontiers of the exploration space or timely auto-homing.

In SWAP, the autonomous exploration behavior is invoked for T e seconds, before the system possibly switches to its semantically-driven behaviors. Fig. 3 .

Viewpoint generation procedure for a boundary edge of a hole during Semantics Hole Coverage mode.

Semantics-Aware Exploration And Inspection Path Planning

Abstract: This paper contributes a novel strategy for semantics-aware autonomous exploration and inspection path planning. Attuned to the fact that environments that need to be explored often involve a sparse set of semantic entities of particular interest, the proposed method offers volumetric exploration combined with two new planning behaviors that together ensure that a complete mesh model is reconstructed for each semantic, while its surfaces are observed at appropriate resolution and through suitable viewing angles. Evaluated in extensive simulation studies and experimental results using a flying robot, the planner delivers efficient combined exploration and high-fidelity inspection planning that is focused on the semantics of interest. Comparisons against relevant methods of the state-of-the-art are further presented.

Voxblox: Incremental 3D Euclidean Signed Distance Fields For On-Board Mav Planning

Abstract: Micro Aerial Vehicles (MAVs) that operate in unstructured, unexplored environments require fast and flexible local planning, which can replan when new parts of the map are explored. Trajectory optimization methods fulfill these needs, but require obstacle distance information, which can be given by Euclidean Signed Distance Fields (ESDFs). We propose a method to incrementally build ESDFs from Truncated Signed Distance Fields (TSDFs), a common implicit surface representation used in computer graphics and vision. TSDFs are fast to build and smooth out sensor noise over many observations, and are designed to produce surface meshes. We show that we can build TSDFs faster than Octomaps, and that it is more accurate to build ESDFs out of TSDFs than occupancy maps. Our complete system, called voxblox, is available as open source and runs in real-time on a single CPU core. We validate our approach on-board an MAV, by using our system with a trajectory optimization local planner, entirely on-board and in real-time.


Like our implementation, the prototype of SA #REFR ran out of memory while determining similarity for vasy_18_73.

Our algorithm can deal with bigger examples than #OTHEREFR (which fails at peterson, vasy_10_56 and cwi_1_2, and takes more than 500 seconds for vasy_8_24). Even where #OTHEREFR has a smaller game graph (e.g. cwi_3_14), the exponential formula construction renders it slower.

Also, the clever game graph indeed is much smaller than for examples with a lot of nondeterminism such as peterson.

Of those terminating, the heavily nondeterministic cwi_1_2 is the most expensive example.

As many coarse notions must record the nondeterministic options, this blowup is to be expected.

If we compare to the best similarity algorithm by Ranzato and Tapparo #OTHEREFR , they report their algorithm SA to tackle cwi_1_2 single-handedly.

This is in spite of SA theoretically having optimal complexity and similarity being less complex (cubic) than trace equivalence, which we need to cover.

The benchmarks in #OTHEREFR failed at vasy_10_56, and vasy_25_25, which might be due to 2010's tighter memory requirements (they used 2 GB of RAM) or the degree to which bisimilarity and enabledness in the models is exploited.

Process Equivalence Problems As Energy Games

Abstract: We characterize all common notions of behavioral equivalence by one 6-dimensional energy game, where energies bound capabilities of an attacker trying to tell processes apart. The defender-winning initial credits determine exhaustively which preorders and equivalences from the (strong) linear-time-branching-time spectrum relate processes. The time complexity is exponential, which is optimal due to trace equivalence being covered. This complexity improves drastically on our recent approach for deciding groups of equivalences where exponential sets of distinguishing HML formulas are constructed on top of a super-exponential reachability game. In experiments using the VLTS benchmarks, the algorithm performs on par with the best similarity algorithm.

An Efficient Simulation Algorithm Based On Abstract Interpretation

Abstract: A number of algorithms for computing the simulation preorder are available. Let Sigma denote the state space, ->the transition relation and Psim the partition of Sigma induced by simulation equivalence. The algorithms by Henzinger, Henzinger, Kopke and by Bloom and Paige run in O(|Sigma||->|)-time and, as far as time-complexity is concerned, they are the best available algorithms. However, these algorithms have the drawback of a space complexity that is more than quadratic in the size of the state space. The algorithm by Gentilini, Piazza, Policriti--subsequently corrected by van Glabbeek and Ploeger--appears to provide the best compromise between time and space complexity. Gentilini et al.'s algorithm runs in O(|Psim|^2|->|)-time while the space complexity is in O(|Psim|^2 + |Sigma|log|Psim|). We present here a new efficient simulation algorithm that is obtained as a modification of Henzinger et al.'s algorithm and whose correctness is based on some techniques used in applications of abstract interpretation to model checking. Our algorithm runs in O(|Psim||->|)-time and O(|Psim||Sigma|log|Sigma|)-space. Thus, this algorithm improves the best known time bound while retaining an acceptable space complexity that is in general less than quadratic in the size of the state space. An experimental evaluation showed good comparative results with respect to Henzinger, Henzinger and Kopke's algorithm.


However, it is still challenging for the labeled nodes to propagate their information far away using a conventional message passing algorithm, since the influence of labeled nodes decays as the topological distance increases #REFR .

G RAPH or network is widely used for describing the interactions between elements of a complex system, such as those in social networks #OTHEREFR , knowledge graphs #OTHEREFR , molecular graphs #OTHEREFR , and recommender systems #OTHEREFR .

To deal with those non-Euclidean data for various graph analytical tasks such as node classification #OTHEREFR and link prediction #OTHEREFR , graph neural networks (GNNs) #OTHEREFR , #OTHEREFR have been developed and shown having superior performances.

The core of current GNNs such as GCN #OTHEREFR is message passing.

In message passing, feature representations are learned for each node by recursively performing aggregation and transformation on the representations of its immediate neighbors, revealing that information about long-distance neighbors can be captured this way.

Moreover, increasing message passing number will lead to oversmoothing #OTHEREFR , #OTHEREFR , i.e., the case where representations are determined by the graph structure.

While techniques like residual connections used in GCNII #OTHEREFR allow the network architecture to be deeper, they substantially increase the number of learnable parameters and computational complexity of the GNN.

Another shortcoming of message passing is its negative smoothing effect in the circumstances where the nodes of the same type discontinuously distributed in the topology space.

For instance, in heterophilious graphs, the immediate neighbors of a node come from different classes.

It has been revealed #OTHEREFR that in smoothing such nodes, message passing forcefully make the feature representations of the nodes with different labels approximate the average of the local neighborhood, thus deteriorating the representation learning.

Steering Graph Neural Networks With Pinning Control

Abstract: In the semi-supervised setting where labeled data are largely limited, it remains to be a big challenge for message passing based graph neural networks (GNNs) to learn feature representations for the nodes with the same class label that is distributed discontinuously over the graph. To resolve the discontinuous information transmission problem, we propose a control principle to supervise representation learning by leveraging the prototypes (i.e., class centers) of labeled data. Treating graph learning as a discrete dynamic process and the prototypes of labeled data as "desired" class representations, we borrow the pinning control idea from automatic control theory to design learning feedback controllers for the feature learning process, attempting to minimize the differences between message passing derived features and the class prototypes in every round so as to generate class-relevant features. Specifically, we equip every node with an optimal controller in each round through learning the matching relationships between nodes and the class prototypes, enabling nodes to rectify the aggregated information from incompatible neighbors in a graph with strong heterophily. Our experiments demonstrate that the proposed PCGCN model achieves better performances than deep GNNs and other competitive heterophily-oriented methods, especially when the graph has very few labels and strong heterophily.

Representation Learning On Graphs With Jumping Knowledge Networks

Abstract: Recent deep learning approaches for representation learning on graphs follow a neighborhood aggregation procedure. We analyze some important properties of these models, and propose a strategy to overcome those. In particular, the range of "neighboring" nodes that a node's representation draws from strongly depends on the graph structure, analogous to the spread of a random walk. To adapt to local neighborhood properties and tasks, we explore an architecture -- jumping knowledge (JK) networks -- that flexibly leverages, for each node, different neighborhood ranges to enable better structure-aware representation. In a number of experiments on social, bioinformatics and citation networks, we demonstrate that our model achieves state-of-the-art performance. Furthermore, combining the JK framework with models like Graph Convolutional Networks, GraphSAGE and Graph Attention Networks consistently improves those models' performance.


Second, what role does the particular distribution of post-activations in a given layer play in determining network performance? For example, the activation function considered in #REFR is hyperbolic tangent, which we adopt henceforth.

The boundary between these two phases is a critical line called the edge of chaos, 2 which is a continuous phase transition characterized by a diverging correlation length ξ for the layer-to-layer two-point function of the neurons.

Since the correlation length sets the depth scale at which information can propagate, this theoretically enables networks of arbitrary depth to be trained at criticality (more generally, networks are trainable provided their depth does not exceed the scale set by ξ).

In other words, the deeper the network, the closer one must lie to the edge of chaos; this was demonstrated in #OTHEREFR along a slice of parameter space at bias variance 0.05 and weight variance ranging from 1 to 4, and subsequently generalized/corroborated in, e.g., #OTHEREFR Several questions naturally arise from the above work.

First, given that the network parameters will evolve under training in order to minimize the specified cost function and, in particular, develop interdependencies, why does the choice of initialization have such a decisive effect on network performance? 3 Indeed, it was observed in #OTHEREFR that the hidden-layer pre-activation distributions (as quantified by their variance) rapidly approach some asymptotic value within 10 or fewer layers, and then remain relatively unchanged for arbitrarily many additional layers.

We corroborate this fact at the level of the post-activation in fig. 6 of appendix A.

When σ 2 b 1 and σ 2 w 1, the pre-activations z of the hidden layers are approximately Gaussian-distributed with small variance (cf. (8) ).

In this case, tanh(z) ≈ z, so the network behaves like a linear network.

These are quite restrictive, being incapable of representing functions whose output data are nonlinearly separable and cannot be generated by a combination of linearly separable data.

In the opposite extreme, for large values of σ 2 w and σ 2 b , the pre-activation variance becomes so large that the post-activation distribution becomes peaked at ±1.

In other words, large pre-activation variance saturates the tanh, causing it to behave like a discrete step-function.

Criticality Versus Uniformity In Deep Neural Networks

Abstract: Deep feedforward networks initialized along the edge of chaos exhibit exponentially superior training ability as quantified by maximum trainable depth. In this work, we explore the effect of saturation of the tanh activation function along the edge of chaos. In particular, we determine the line of uniformity in phase space along which the post-activation distribution has maximum entropy. This line intersects the edge of chaos, and indicates the regime beyond which saturation of the activation function begins to impede training efficiency. Our results suggest that initialization along the edge of chaos is a necessary but not sufficient condition for optimal trainability.

Deep Information Propagation

Abstract: We study the behavior of untrained neural networks whose weights and biases are randomly distributed using mean field theory. We show the existence of depth scales that naturally limit the maximum depth of signal propagation through these random networks. Our main practical result is to show that random networks may be trained precisely when information can travel through them. Thus, the depth scales that we identify provide bounds on how deep a network may be trained for a specific choice of hyperparameters. As a corollary to this, we argue that in networks at the edge of chaos, one of these depth scales diverges. Thus arbitrarily deep networks may be trained only sufficiently close to criticality. We show that the presence of dropout destroys the order-to-chaos critical point and therefore strongly limits the maximum trainable depth for random networks. Finally, we develop a mean field theory for backpropagation and we show that the ordered and chaotic phases correspond to regions of vanishing and exploding gradient respectively.


Table 11 : Comparing CSST(FixMatch) against other Semi-Supervised Learning Methods for long tailed data distribution for the objectives (4), #REFR .

We provide a detailed description of algorithms used for optimizing non decomposable objectives through CSST(FixMatch) ans CSST(UDA).

Algorithm 1 is used for experiments in Section 5 for maximizing worst-case recall (i.e. min recall using CSST(FixMatch) and CSST(UDA)).

Algorithm 2 is used for experiments in Section 5 for maximizing recall under coverage constraints (i.e. min coverage experiments on CIFAR10-LT, CIFAR100-LT and ImageNet100-LT).

Only CSST(FixMatch) is the closest to satisfying the constraint yet suffers very little on the avg. recall.

these methods on the long-tailed CIFAR-10 (ρ = 100) and CIFAR-100 (ρ = 10) datasets.

The objective for long-tailed CIFAR-10 dataset was to maximise the worst-case recall (2.1) and average recall, subject to a per-class coverage constraint (4).

For CIFAR-100 LT dataset, we compare these methods for the objectives maximizing HT recall #OTHEREFR and recall under HT coverage constraints #OTHEREFR .

For the objectives (2.1) and #OTHEREFR , DARP achieves the best average recall yet it suffers on the worst-case recall.

Cost-Sensitive Self-Training For Optimizing Non-Decomposable Metrics

Abstract: Self-training based semi-supervised learning algorithms have enabled the learning of highly accurate deep neural networks, using only a fraction of labeled data. However, the majority of work on self-training has focused on the objective of improving accuracy whereas practical machine learning systems can have complex goals (e.g. maximizing the minimum of recall across classes etc.) that are non-decomposable in nature. In this work, we introduce the Cost-Sensitive Self-Training (CSST) framework which generalizes the self-training-based methods for optimizing non-decomposable metrics. We prove that our framework can better optimize the desired non-decomposable metric utilizing unlabeled data, under similar data distribution assumptions made for the analysis of self-training. Using the proposed CSST framework we obtain practical self-training methods (for both vision and NLP tasks) for optimizing different non-decomposable metrics using deep neural networks. Our results demonstrate that CSST achieves an improvement over the state-of-the-art in majority of the cases across datasets and objectives.

Marian: Fast Neural Machine Translation In C++

Abstract: We present Marian, an efficient and self-contained Neural Machine Translation framework with an integrated automatic differentiation engine based on dynamic computation graphs. Marian is written entirely in C++. We describe the design of the encoder-decoder framework and demonstrate that a research-friendly toolkit can achieve high training and translation speed.


To better explore the unaligned and clear reference image, we are inspired by the contextual loss #REFR which aims to compute the cosine similar distance between the unaligned images for imageto-image translation tasks.

All layers use 4 × 4 Convolution-BatchNorm-LeakyReLU layers with stride 2, except that the first and last layers do not use the BatchNorm.

Furthermore, we extend the adversarial loss to a multi-scale adversarial loss:

where i represents the different scales, and J is generated by the dehazing network so that it is trained by using the above loss.

We extend it to a multi-scale contextual loss, which is defined as

where S is the contextual similarity between image features, and Φ l (J) and Φ l (J ref ) represent the feature maps extracted from layer l of the VGG-16 network Φ with the inputs J and J ref , respectively.

Our non-aligned supervision brings an important benefit to relaxing the strict alignment requirement on the hazy/clear image pair.

This leads us to easily collect the non-aligned hazy/clear image pairs in the same real scenes under some relaxed conditions, e.g., pixel misalignment, and shift views.

Non-Aligned Supervision For Real Image Dehazing

Abstract: Removing haze from real-world images is challenging due to unpredictable weather conditions, resulting in misaligned hazy and clear image pairs. In this paper, we propose a non-aligned supervision framework that consists of three networks-dehazing, airlight, and transmission. In particular, we explore a non-alignment setting by utilizing a clear reference image that is not aligned with the hazy input image to supervise the dehazing network through a multi-scale reference loss that compares the features of the two images. Our setting makes it easier to collect hazy/clear image pairs in real-world environments, even under conditions of misalignment and shift views. To demonstrate this, we have created a new hazy dataset called "Phone-Hazy", which was captured using mobile phones in both rural and urban areas. Additionally, we present a mean and variance self-attention network to model the infinite airlight using dark channel prior as position guidance, and employ a channel attention network to estimate the three-channel transmission. Experimental results show that our framework outperforms current state-of-the-art methods in the real-world image dehazing. Phone-Hazy and code will be available at here.

The Contextual Loss For Image Transformation With Non-Aligned Data

Abstract: Feed-forward CNNs trained for image transformation problems rely on loss functions that measure the similarity between the generated image and a target image. Most of the common loss functions assume that these images are spatially aligned and compare pixels at corresponding locations. However, for many tasks, aligned training pairs of images will not be available. We present an alternative loss function that does not require alignment, thus providing an effective and simple solution for a new space of problems. Our loss is based on both context and semantics -- it compares regions with similar semantic meaning, while considering the context of the entire image. Hence, for example, when transferring the style of one face to another, it will translate eyes-to-eyes and mouth-to-mouth. Our code can be found at https://www.github.com/roimehrez/contextualLoss


Our implementation is based on the sequence labelling implementation of the Huggingface open-source library #REFR .

Given unlabelled text as input, we predict named entity boundaries by analyzing the input sentence structure ( Figure 2 ).

We treat this task as a sequence labelling task in which the model predicts if a given token is part of an entity or not ("B-ENTITY", "I-ENTITY","O").

We use the multilingual XLM-RoBERTa-large model #OTHEREFR ) with a token classification layer (a linear layer) on top of each token representation.

We evaluate the model in the development set at the end of each epoch and then select the best performing checkpoint.

We train five independent models and then use majority vote as the ensembling strategy at inference time.

Ixa/Cogcomp At Semeval-2023 Task 2: Context-Enriched Multilingual Named Entity Recognition Using Knowledge Bases

Abstract: Named Entity Recognition (NER) is a core natural language processing task in which pretrained language models have shown remarkable performance. However, standard benchmarks like CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003) do not address many of the challenges that deployed NER systems face, such as having to classify emerging or complex entities in a fine-grained way. In this paper we present a novel NER cascade approach comprising three steps: first, identifying candidate entities in the input sentence; second, linking the each candidate to an existing knowledge base; third, predicting the fine-grained category for each entity candidate. We empirically demonstrate the significance of external knowledge bases in accurately classifying fine-grained and emerging entities. Our system exhibits robust performance in the MultiCoNER2 (Fetahu et al., 2023) shared task, even in the low-resource language setting where we leverage knowledge bases of high-resource languages.

Huggingface'S Transformers: State-Of-The-Art Natural Language Processing

Abstract: Recent advances in modern Natural Language Processing (NLP) research have been dominated by the combination of Transfer Learning methods with large-scale language models, in particular based on the Transformer architecture. With them came a paradigm shift in NLP with the starting point for training a model on a downstream task moving from a blank specific model to a general-purpose pretrained architecture. Still, creating these general-purpose models remains an expensive and time-consuming process restricting the use of these methods to a small sub-set of the wider NLP community. In this paper, we present HuggingFace's Transformers library, a library for state-of-the-art NLP, making these developments available to the community by gathering state-of-the-art general-purpose pretrained models under a unified API together with an ecosystem of libraries, examples, tutorials and scripts targeting many downstream NLP tasks. HuggingFace's Transformers library features carefully crafted model implementations and high-performance pretrained weights for two main deep learning frameworks, PyTorch and TensorFlow, while supporting all the necessary tools to analyze, evaluate and use these models in downstream tasks such as text/token classification, questions answering and language generation among others. The library has gained significant organic traction and adoption among both the researcher and practitioner communities. We are committed at HuggingFace to pursue the efforts to develop this toolkit with the ambition of creating the standard library for building NLP systems. HuggingFace's Transformers library is available at \url{https://github.com/huggingface/transformers}.

sequence labelling implementation
Huggingface open-source library


We run MoSh++ #REFR (henceforth referred to as MoSh) on the C3D files to acquire the SMPL-X parameters corresponding to each frame in the sequence (Fig. 3, right) . Synthetic sensor information.

After collection, CIRCLE data are passed through a variety of post-processing steps. Mocap data processing.

We use Shogun Post to process and export the captured clips to BVH and C3D formats.

Offline synchronization Due to latency between the headset and the webserver communicating with the Vicon machine, the start times of the headset and mocap data are misaligned and must be synchronized.

By assuming that the offset between the head bone and the headset remains constant during each sequence, this can be accomplished by solving for the time offset which maximizes the convolution between the velocity profiles of the head bone and headset.

With start times aligned, we then trim the sequences to the same length and linearly interpolate the headset poses such that every mocap frame has a corresponding headset pose. Human mesh fitting.

After synchronization, we load both the mocap data in BVH format and the VR trajectories in Habitat and extract synthetic sensor information such as ego-centric RGB-D videos (Fig. 3, middle) .

Additionally, we can use Blender to render first-person RGB-D videos with the SMPL-X meshes calculated by MoSh. Quality assurance.

Identifying and fixing sequences with artifacts is a demanding manual process.

We find that our pipeline has a very high yield of data that does not need to be fixed, so our focus is on identifying sequences with problems so that they can be collected again.

To help prioritize, we develop a suite of tools that automatically check for common problems, such as:

Circle: Capture In Rich Contextual Environments

Abstract: Figure 1. Example poses from CIRCLE captured from real human motion in a virtual environment.

Amass: Archive Of Motion Capture As Surface Shapes

Abstract: Large datasets are the cornerstone of recent advances in computer vision using deep learning. In contrast, existing human motion capture (mocap) datasets are small and the motions limited, hampering progress on learning models of human motion. While there are many different datasets available, they each use a different parameterization of the body, making it difficult to integrate them into a single meta dataset. To address this, we introduce AMASS, a large and varied database of human motion that unifies 15 different optical marker-based mocap datasets by representing them within a common framework and parameterization. We achieve this using a new method, MoSh++, that converts mocap data into realistic 3D human meshes represented by a rigged body model. Here we use SMPL [Loper et al., 2015], which is widely used and provides a standard skeletal representation as well as a fully rigged surface mesh. The method works for arbitrary marker sets, while recovering soft-tissue dynamics and realistic hand motion. We evaluate MoSh++ and tune its hyperparameters using a new dataset of 4D body scans that are jointly recorded with markerbased mocap. The consistent representation of AMASS makes it readily useful for animation, visualization, and generating training data for deep learning. Our dataset is significantly richer than previous human motion collections, having more than 40 hours of motion data, spanning over 300 subjects, more than 11000 motions, and will be publicly available to the research community.


In addition to the source model, MAS #REFR [324] also provides the estimated GMM of source features for the target domain.

TAN #OTHEREFR individually learns the feature encoder and the classifier in the source training stage in which the encoder is optimized based on a reconstruction objective.

StickerDA #OTHEREFR and TTT++ #OTHEREFR introduce an auxiliary self-supervised classification head in the source model, and GarDA #OTHEREFR and ADV-M #OTHEREFR need to learn a domain discriminator from the source domain to help the semantic model adapt to the target domain.

Besides, a few methods #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR employ the multi-head classifier learning strategy by learning multiple classification heads with one shared feature encoder in the source domain.

On top of the prior-enforcing auto-encoder, SoMAN-cPAE #OTHEREFR also learns a global classification head and multiple local classification heads for different data augmentations.

BUFR #OTHEREFR further requires the marginal feature distributions in softly-binned histograms for measurement shifts.

U-SFAN #OTHEREFR utilizes the distribution of the Bayesian source classifier to quantify the uncertainty for more accurate adaptation.

Moreover, Prototype-DA #OTHEREFR and OnDA #OTHEREFR need the existence of class prototypes calculated in the source domain.

FTA-FDA #OTHEREFR needs to retain some random spectrum maps of source data for the following data translation step, while CATTAn #OTHEREFR transfers the subspace learned from source features to the target domain.

Some approaches #OTHEREFR , #OTHEREFR achieve feature alignment using different orders of moments for features in the source domain.

A Comprehensive Survey On Test-Time Adaptation Under Distribution Shifts

Abstract: Machine learning methods strive to acquire a robust model during training that can generalize well to test samples, even under distribution shifts. However, these methods often suffer from a performance drop due to unknown test distributions. Test-time adaptation (TTA), an emerging paradigm, has the potential to adapt a pre-trained model to unlabeled data during testing, before making predictions. Recent progress in this paradigm highlights the significant benefits of utilizing unlabeled data for training self-adapted models prior to inference. In this survey, we divide TTA into several distinct categories, namely, test-time (source-free) domain adaptation, test-time batch adaptation, online test-time adaptation, and test-time prior adaptation. For each category, we provide a comprehensive taxonomy of advanced algorithms, followed by a discussion of different learning scenarios. Furthermore, we analyze relevant applications of TTA and discuss open challenges and promising areas for future research. A comprehensive list of TTA methods can be found at https://github.com/tim-learn/awesome-test-time-adaptation.

No More Discrimination: Cross City Adaptation Of Road Scene Segmenters

Abstract: Despite the recent success of deep-learning based semantic segmentation, deploying a pre-trained road scene segmenter to a city whose images are not presented in the training set would not achieve satisfactory performance due to dataset biases. Instead of collecting a large number of annotated images of each city of interest to train or refine the segmenter, we propose an unsupervised learning approach to adapt road scene segmenters across different cities. By utilizing Google Street View and its time-machine feature, we can collect unannotated images for each road scene at different times, so that the associated static-object priors can be extracted accordingly. By advancing a joint global and class-specific domain adversarial learning framework, adaptation of pre-trained segmenters to that city can be achieved without the need of any user annotation or interaction. We show that our method improves the performance of semantic segmentation in multiple cities across continents, while it performs favorably against state-of-the-art approaches requiring annotated training data.


Macrobase #REFR explains outliers by correlating them to some external attributes which are not used to detect anomalies such as location, time of occurrence, software version, etc.

If a system is able to summarize anomaly detection results into groups and explain why each group of objects is considered to be abnormal or normal, this will greatly reduce the effort of users in evaluating anomaly detection results. State-of-the-Art.

To the best of our knowledge, the problem of summarizing and interpreting outlier detection results is yet to be addressed.

Scorpion #OTHEREFR produces meaningful explanations for anomalies in aggregation queries when the 'cause' of an outlier is contained in its provenance.

Similar to Scorpion, Cape #OTHEREFR aims to explain the outliers in aggregation queries, but using the objects that counterbalance the outliers.

Both works do not tackle the problem of summarizing outliers.

However, Macrobase only targets explaining the outliers captured by its default density-based outlier detector and does not generalize to other outlier detection methods.

In the broader field of interpretable AI, LIME #OTHEREFR explains the predictions of a classifier by learning a linear model locally around the prediction with respect to each testing object and pointing out the attributes that are most important to the prediction of the linear model.

However, rather than use one model to represent a set of objects, LIME has to learn a linear model for each individual object.

Therefore, using LIME to explain a large number of prediction results will be prohibitively expensive.

Other methods #OTHEREFR explain classification results in the similar way to LIME. Challenges.

Interpretable Outlier Summarization

Abstract: Outlier detection is critical in real applications to prevent financial fraud, defend network intrusions, or detecting imminent device failures. To reduce the human effort in evaluating outlier detection results and effectively turn the outliers into actionable insights, the users often expect a system to automatically produce interpretable summarizations of subgroups of outlier detection results. Unfortunately, to date no such systems exist. To fill this gap, we propose STAIR which learns a compact set of human understandable rules to summarize and explain the anomaly detection results. Rather than use the classical decision tree algorithms to produce these rules, STAIR proposes a new optimization objective to produce a small number of rules with least complexity, hence strong interpretability, to accurately summarize the detection results. The learning algorithm of STAIR produces a rule set by iteratively splitting the large rules and is optimal in maximizing this objective in each iteration. Moreover, to effectively handle high dimensional, highly complex data sets which are hard to summarize with simple rules, we propose a localized STAIR approach, called L-STAIR. Taking data locality into consideration, it simultaneously partitions data and learns a set of localized rules for each partition. Our experimental study on many outlier benchmark datasets shows that STAIR significantly reduces the complexity of the rules required to summarize the outlier detection results, thus more amenable for humans to understand and evaluate, compared to the decision tree methods.

Macrobase: Prioritizing Attention In Fast Data

Abstract: As data volumes continue to rise, manual inspection is becoming increasingly untenable. In response, we present MacroBase, a data analytics engine that prioritizes end-user attention in high-volume fast data streams. MacroBase enables efficient, accurate, and modular analyses that highlight and aggregate important and unusual behavior, acting as a search engine for fast data. MacroBase is able to deliver order-of-magnitude speedups over alternatives by optimizing the combination of explanation and classification tasks and by leveraging a new reservoir sampler and heavy-hitters sketch specialized for fast data streams. As a result, MacroBase delivers accurate results at speeds of up to 2M events per second per query on a single core. The system has delivered meaningful results in production, including at a telematics company monitoring hundreds of thousands of vehicles.


Limitations, Challenges, And Future Directions

Since language models often encode biases and stereotypes derived from training corpora #REFR , they may have difficulty producing relevant individuals who are not prototypical (i.e., they do not have a particular stereotype).

Therefore, future work should investigate more diverse annotator pools or matching annotators to targeted groups, as well as examining how annotator's familiarity with essentialist beliefs and identities affect their judgements.

Furthermore, prior work in countering hatespeech has show that effective strategies can vary widely depending on the target group #OTHEREFR .

In our work, we consider results aggregated across all groups.

However, community-specific investigations are an important future step towards developing effective counter-statements.

Accuracy of generated exceptions The selection of specific individuals for direct exceptions presents an ongoing challenge, based on the high number of DIR-IND marked incorrect.

We illustrate incorrect individuals and subgroups in the bottom two examples of Table 1 .

Additionally, as mentioned in §4, many stereotypes are subjective (e.g., "women are vain").

Therefore, individuals who are counterexamples to the stereotype may be judged differently by different people (e.g., our system proposes that "taylor swift, sarah palin, and scarlett johansson" are not vain).

Producing accurate and relevant direct exceptions to a stereotype is important for understanding the role of such examples to counter essentialist beliefs.

Our results and discussion highlight the complexity of countering essentialist beliefs.

Towards Countering Essentialism Through Social Bias Reasoning

Abstract: Essentialist beliefs (i.e., believing that members of the same group are fundamentally alike) play a central role in social stereotypes and can lead to harm when left unchallenged. In our work, we conduct exploratory studies into the task of countering essentialist beliefs (e.g., "liberals are stupid"). Drawing on prior work from psychology and NLP, we construct five types of counterstatements and conduct human studies on the effectiveness of these different strategies. Our studies also investigate the role in choosing a counterstatement of the level of explicitness with which an essentialist belief is conveyed. We find that statements that broaden the scope of a stereotype (e.g., to other groups, as in "conservatives can also be stupid") are the most popular countering strategy. We conclude with a discussion of challenges and open questions for future work in this area (e.g., improving factuality, studying community-specific variation) and we emphasize the importance of work at the intersection of NLP and psychology.

The Woman Worked As A Babysitter: On Biases In Language Generation

Abstract: We present a systematic study of biases in natural language generation (NLG) by analyzing text generated from prompts that contain mentions of different demographic groups. In this work, we introduce the notion of the regard towards a demographic, use the varying levels of regard towards different demographics as a defining metric for bias in NLG, and analyze the extent to which sentiment scores are a relevant proxy metric for regard. To this end, we collect strategically-generated text from language models and manually annotate the text with both sentiment and regard scores. Additionally, we build an automatic regard classifier through transfer learning, so that we can analyze biases in unseen text. Together, these methods reveal the extent of the biased nature of language model generations. Our analysis provides a study of biases in NLG, bias metrics and correlated human judgments, and empirical evidence on the usefulness of our annotated dataset.


Notably, while ACNP is arguably more powerful than CNP and indeed reports better results as expected for GP data #REFR , it performs worse with sine and oscillator data, probably due to the number of functions being too limited. Also, CCNP does not improve CNP consistently either.

For sine waves, we vary the values of amplitude U[−1, 1] Results.

Table 1 presents the quantitative results of various baseline CNPs with and without adversarial training.

The baseline models include CNP #OTHEREFR , ACNP #OTHEREFR , and CCNP #OTHEREFR , where the first two are commonly adopted as per most CNPs literature.

We consider CCNP as a comparison to the previous NCE setting, where NCE is also applied but on the context set as a regularization term to CNPs.

We find that adversarial training consistently results in performance improvements to baselines in most cases.

Further, we observe that adversarial training can be adapted to CCNP, showing potential in improving generative CNPs by leveraging composite contrastive objectives.

Adversarially Contrastive Estimation Of Conditional Neural Processes

Abstract: Conditional Neural Processes (CNPs) formulate distributions over functions and generate function observations with exact conditional likelihoods. CNPs, however, have limited expressivity for high-dimensional observations, since their predictive distribution is factorized into a product of unconstrained (typically) Gaussian outputs. Previously, this could be handled using latent variables or autoregressive likelihood, but at the expense of intractable training and quadratically increased complexity. Instead, we propose calibrating CNPs with an adversarial training scheme besides regular maximum likelihood estimates. Specifically, we train an energy-based model (EBM) with noise contrastive estimation, which enforces EBM to identify true observations from the generations of CNP. In this way, CNP must generate predictions closer to the ground-truth to fool EBM, instead of merely optimizing with respect to the fixed-form likelihood. From generative function reconstruction to downstream regression and classification tasks, we demonstrate that our method fits mainstream CNP members, showing effectiveness when unconstrained Gaussian likelihood is defined, requiring minimal computation overhead while preserving foundation properties of CNPs.

Attentive Neural Processes

Abstract: Neural Processes (NPs) (Garnelo et al 2018a;b) approach regression by learning to map a context set of observed input-output pairs to a distribution over regression functions. Each function models the distribution of the output given an input, conditioned on the context. NPs have the benefit of fitting observed data efficiently with linear complexity in the number of context input-output pairs, and can learn a wide family of conditional distributions; they learn predictive distributions conditioned on context sets of arbitrary size. Nonetheless, we show that NPs suffer a fundamental drawback of underfitting, giving inaccurate predictions at the inputs of the observed data they condition on. We address this issue by incorporating attention into NPs, allowing each input location to attend to the relevant context points for the prediction. We show that this greatly improves the accuracy of predictions, results in noticeably faster training, and expands the range of functions that can be modelled.