The purpose of an encoding model is to predict brain activity given a stimulus.

In this contribution, we attempt at estimating a whole brain encoding model of auditory perception in a naturalistic stimulation setting.

We analyze data from an open dataset, in which 16 subjects watched a short movie while their brain activity was being measured using functional MRI.

We extracted feature vectors aligned with the timing of the audio from the movie, at different layers of a Deep Neural Network pretrained on the classification of auditory scenes.

fMRI data was parcellated using hierarchical clustering on 500 parcels, and encoding models were estimated using a fully connected neural network with one hidden layer, trained to predict the signals for each parcel from the DNN features.

Individual encoding models were successfully trained and predicted brain activity on unseen data, in parcels located in the superior temporal lobe, as well as dorsolateral prefrontal regions, which are usually considered as areas involved in auditory and language processing.

Taken together, this contribution extends previous attempts on estimating encoding models, by showing the ability to model brain activity using a generic DNN (ie not specifically trained for this purpose) to extract auditory features, suggesting a degree of similarity between internal DNN representations and brain activity in naturalistic settings.

Feature vectors from SoundNet can predict brain activity of subjects watching a movie in auditory and language related brain regions.

Estimating encoding models of cortical auditory processing using naturalistic stimuli and transfer learning


In this paper, we describe the "implicit autoencoder" (IAE), a generative autoencoder in which both the generative path and the recognition path are parametrized by implicit distributions.

We use two generative adversarial networks to define the reconstruction and the regularization cost functions of the implicit autoencoder, and derive the learning rules based on maximum-likelihood learning.

Using implicit distributions allows us to learn more expressive posterior and conditional likelihood distributions for the autoencoder.

Learning an expressive conditional likelihood distribution enables the latent code to only capture the abstract and high-level information of the data, while the remaining information is captured by the implicit conditional likelihood distribution.

For example, we show that implicit autoencoders can disentangle the global and local information, and perform deterministic or stochastic reconstructions of the images.

We further show that implicit autoencoders can disentangle discrete underlying factors of variation from the continuous factors in an unsupervised fashion, and perform clustering and semi-supervised learning.

We propose a generative autoencoder that can learn expressive posterior and conditional likelihood distributions using implicit distributions, and train the model using a new formulation of the ELBO.

Implicit Autoencoders


Strong inductive biases allow children to learn in fast and adaptable ways.

Children use the mutual exclusivity (ME) bias to help disambiguate how words map to referents, assuming that if an object has one label then it does not need another.

In this paper, we investigate whether or not standard neural architectures have a ME bias, demonstrating that they lack this learning assumption.

Moreover, we show that their inductive biases are poorly matched to lifelong learning formulations of classification and translation.

We demonstrate that there is a compelling case for designing neural networks that reason by mutual exclusivity, which remains an open challenge.

Children use the mutual exclusivity (ME) bias to learn new words, while standard neural nets show the opposite bias, hindering learning in naturalistic scenarios such as lifelong learning.

Mutual Exclusivity as a Challenge for Deep Neural Networks


Cortical neurons process and integrate information on multiple timescales.

In addition, these timescales or temporal receptive fields display functional and hierarchical organization.

For instance, areas important for working memory (WM), such as prefrontal cortex, utilize neurons with stable temporal receptive fields and long timescales to support reliable representations of stimuli.

Despite of the recent advances in experimental techniques, the underlying mechanisms for the emergence of neuronal timescales long enough to support WM are unclear and challenging to investigate experimentally.

Here, we demonstrate that spiking recurrent neural networks (RNNs) designed to perform a WM task reproduce previously observed experimental findings and that these models could be utilized in the future to study how neuronal timescales specific to WM emerge.

Spiking recurrent neural networks performing a working memory task utilize long heterogeneous timescales, strikingly similar to those observed in prefrontal cortex.

Spiking Recurrent Networks as a Model to Probe Neuronal Timescales Specific to Working Memory

work memory
spike recurrent
network
timescale


Conventional deep learning classifiers are static in the sense that they are trained on

a predefined set of classes and learning to classify a novel class typically requires

In this work, we address the problem of Low-shot network-expansion

We introduce a learning framework which enables expanding a pre-trained

(base) deep network to classify novel classes when the number of examples for the

We present a simple yet powerful distillation

method where the base network is augmented with additional weights to classify

the novel classes, while keeping the weights of the base network unchanged.

term this learning hard distillation, since we preserve the response of the network

on the old classes to be equal in both the base and the expanded network.

show that since only a small number of weights needs to be trained, the hard

distillation excels for low-shot training scenarios.

avoids detriment to classification performance on the base classes.

show that low-shot network expansion can be done with a very small memory

footprint by using a compact generative model of the base classes training data

with only a negligible degradation relative to learning with the full training set.

In this paper, we address the problem of Low-shot network-expansion learning

GENERATIVE LOW-SHOT NETWORK EXPANSION


People ask questions that are far richer, more informative, and more creative than current AI systems.

We propose a neural program generation framework for modeling human question asking, which represents questions as formal programs and generates programs with an encoder-decoder based deep neural network.

From extensive experiments using an information-search game, we show that our method can ask optimal questions in synthetic settings, and predict which questions humans are likely to ask in unconstrained settings.

We also propose a novel grammar-based question generation framework trained with reinforcement learning, which is able to generate creative questions without supervised data.

We introduce a model of human question asking that combines neural networks and symbolic programs, which can learn to generate good questions with or without supervised examples.

Modeling question asking using neural program generation


The classification of images taken in special imaging environments except air is the first challenge in extending the applications of deep learning.

We report on an UW-Net (Underwater Network), a new convolutional neural network (CNN) based network for underwater image classification.

In this model, we simulate the visual correlation of background attention with image understanding for special environments, such as fog and underwater by constructing an inception-attention (I-A) module.

The experimental results demonstrate that the proposed UW-Net achieves an accuracy of 99.3% on underwater image classification, which is significantly better than other image classification networks, such as AlexNet, InceptionV3, ResNet and Se-ResNet.

Moreover, we demonstrate the proposed IA module can be used to boost the performance of the existing object recognition networks.

By substituting the inception module with the I-A module, the Inception-ResnetV2 network achieves a 10.7% top1 error rate and a 0% top5 error rate on the subset of ILSVRC-2012, which further illustrates the function of the background attention in the image classifications.

A visual understanding mechanism for special environment

UW-NET: AN INCEPTION-ATTENTION NETWORK FOR UNDERWATER IMAGE CLASSIFICATION


Deep networks have achieved impressive results across a variety of important tasks.

However, a known weakness is a failure to perform well when evaluated on data which differ from the training distribution, even if these differences are very small, as is the case with adversarial examples.

We propose \emph{Fortified Networks}, a simple transformation of existing networks, which “fortifies” the hidden layers in a deep network by identifying when the hidden states are off of the data manifold, and maps these hidden states back to parts of the data manifold where the network performs well.

Our principal contribution is to show that fortifying these hidden states improves the robustness of deep networks and our experiments (i) demonstrate improved robustness to standard adversarial attacks in both black-box and white-box threat models; (ii) suggest that our improvements are not primarily due to the problem of deceptively good results due to degraded quality in the gradient signal (the gradient masking problem) and (iii) show the advantage of doing this fortification in the hidden layers instead of the input space.

We demonstrate improvements in adversarial robustness on three datasets (MNIST, Fashion MNIST, CIFAR10), across several attack parameters, both white-box and black-box settings, and the most widely studied attacks (FGSM, PGD, Carlini-Wagner).

We show that these improvements are achieved across a wide variety of hyperparameters.

Better adversarial training by learning to map back to the data manifold with autoencoders in the hidden states.

Fortified Networks: Improving the Robustness of Deep Networks by Modeling the Manifold of Hidden Representations


Neural networks could misclassify inputs that are slightly different from their training data, which indicates a small margin between their decision boundaries and the training dataset.

In this work, we study the binary classification of linearly separable datasets and show that linear classifiers could also have decision boundaries that lie close to their training dataset if cross-entropy loss is used for training.

In particular, we show that if the features of the training dataset lie in a low-dimensional affine subspace and the cross-entropy loss is minimized by using a gradient method, the margin between the training points and the decision boundary could be much smaller than the optimal value.

This result is contrary to the conclusions of recent related works such as (Soudry et al., 2018), and we identify the reason for this contradiction.

In order to improve the margin, we introduce differential training, which is a training paradigm that uses a loss function defined on pairs of points from each class.

We show that the decision boundary of a linear classifier trained with differential training indeed achieves the maximum margin.

The results reveal the use of cross-entropy loss as one of the hidden culprits of adversarial examples and introduces a new direction to make neural networks robust against them.

We show that minimizing the cross-entropy loss by using a gradient method could lead to a very poor margin if the features of the dataset lie on a low-dimensional subspace.

Cross-Entropy Loss Leads To Poor Margins


The concepts of unitary evolution matrices and associative memory have boosted the field of Recurrent Neural Networks (RNN) to state-of-the-art performance in a variety of sequential tasks.

However, RNN still has a limited capacity to manipulate long-term memory.

To bypass this weakness the most successful applications of RNN use external techniques such as attention mechanisms.

In this paper we propose a novel RNN model that unifies the state-of-the-art approaches: Rotational Unit of Memory (RUM).

The core of RUM is its rotational operation, which is,  naturally,  a unitary matrix, providing architectures with the power to learn long-term dependencies by overcoming the vanishing and exploding gradients problem.

Moreover,  the rotational unit also serves as associative memory.

We evaluate our model on synthetic memorization, question answering and language modeling tasks.

RUM learns the Copying Memory task completely and improves the state-of-the-art result in the Recall task.

RUM’s performance in the bAbI Question Answering task is comparable to that of models with attention mechanism.

We also improve the state-of-the-art result to 1.189 bits-per-character (BPC) loss in the Character Level Penn Treebank (PTB) task, which is to signify the applications of RUM to real-world sequential data.

The universality of our construction, at the core of RNN, establishes RUM as a promising approach to language modeling, speech recognition and machine translation.

A novel RNN model which outperforms significantly the current frontier of models in a variety of sequential tasks.

Rotational Unit of Memory


While many recent advances in deep reinforcement learning rely on model-free methods, model-based approaches remain an alluring prospect for their potential to exploit unsupervised data to learn environment dynamics.

One prospect is to pursue hybrid approaches, as in AlphaGo, which combines Monte-Carlo Tree Search (MCTS)—a model-based method—with deep-Q networks (DQNs)—a model-free method.

MCTS requires generating rollouts, which is computationally expensive.

In this paper, we propose to simulate roll-outs, exploiting the latest breakthroughs in image-to-image transduction, namely Pix2Pix GANs, to predict the dynamics of the environment.

Our proposed algorithm, generative adversarial tree search (GATS), simulates rollouts up to a specified depth using both a GAN- based dynamics model and a reward predictor.

GATS employs MCTS for planning over the simulated samples and uses DQN to estimate the Q-function at the leaf states.

Our theoretical analysis establishes some favorable properties of GATS vis-a-vis the bias-variance trade-off and empirical results show that on 5 popular Atari games, the dynamics and reward predictors converge quickly to accurate solutions.

However, GATS fails to outperform DQNs in 4 out of 5 games.

Notably, in these experiments, MCTS has only short rollouts (up to tree depth 4), while previous successes of MCTS have involved tree depth in the hundreds.

We present a hypothesis for why tree search with short rollouts can fail even given perfect modeling.

Surprising negative results on Model Based + Model deep RL

Surprising Negative Results for Generative  Adversarial Tree Search


We present a novel architecture of GAN for a disentangled representation learning.

The new model architecture is inspired by Information Bottleneck (IB) theory thereby named IB-GAN.

IB-GAN objective is similar to that of InfoGAN but has a crucial difference; a capacity regularization for mutual information is adopted, thanks to which the generator of IB-GAN can harness a latent representation in disentangled and interpretable manner.

To facilitate the optimization of IB-GAN in practice, a new variational upper-bound is derived.

With experiments on CelebA, 3DChairs, and dSprites datasets, we demonstrate that the visual quality of samples generated by IB-GAN is often better than those by β-VAEs.

Moreover, IB-GAN achieves much higher disentanglement metrics score than β-VAEs or InfoGAN on the dSprites dataset.

Inspired by Information Bottleneck theory,  we propose a new architecture of GAN for a disentangled representation learning

IB-GAN: Disentangled Representation Learning with Information Bottleneck GAN

disentangle representation
information bottleneck
gan


We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs.

As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters.

We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramér GAN critic.

Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs.

In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance.

We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.

Explain bias situation with MMD GANs; MMD GANs work with smaller critic networks than WGAN-GPs; new GAN evaluation metric.

Demystifying MMD GANs


We extend the Consensus Network framework to Transductive Consensus Network (TCN), a semi-supervised multi-modal classification framework, and identify its two mechanisms: consensus and classification.

By putting forward three variants as ablation studies, we show both mechanisms should be functioning together.

Overall, TCNs outperform or align with the best benchmark algorithms when only 20 to 200 labeled data points are available.

A semi-supervised multi-modal classification framework, TCN, that outperforms various benchmarks.

Semi-supervised classification by reaching consensus among modalities


Separating mixed distributions is a long standing challenge for machine learning and signal processing.

Applications include: single-channel multi-speaker separation (cocktail party problem), singing voice separation and separating reflections from images.

Most current methods either rely on making strong assumptions on the source distributions (e.g. sparsity, low rank, repetitiveness) or rely on having training samples of each source in the mixture.

In this work, we tackle the scenario of extracting an unobserved distribution additively mixed with a signal from an observed (arbitrary) distribution.

We introduce a new method: Neural Egg Separation - an iterative method that learns to separate the known distribution from progressively finer estimates of the unknown distribution.

In some settings, Neural Egg Separation is initialization sensitive, we therefore introduce GLO Masking which ensures a good initialization.

Extensive experiments show that our method outperforms current methods that use the same level of supervision and often achieves similar performance to full supervision.

An iterative neural method for extracting signals that are only observed mixed with other signals

Neural separation of observed and unobserved distributions


In health, machine learning is increasingly common, yet neural network embedding (representation) learning is arguably under-utilized for physiological signals.

This inadequacy stands out in stark contrast to more traditional computer science domains, such as computer vision (CV), and natural language processing (NLP).

For physiological signals, learning feature embeddings is a natural solution to data insufficiency caused by patient privacy concerns -- rather than share data, researchers may share informative embedding models (i.e., representation models), which map patient data to an output embedding.

Here, we present the PHASE (PHysiologicAl Signal Embeddings) framework, which consists of three components: i) learning neural network embeddings of physiological signals, ii) predicting outcomes based on the learned embedding, and iii) interpreting the prediction results by estimating feature attributions in the "stacked" models (i.e., feature embedding model followed by prediction model).

PHASE is novel in three ways: 1) To our knowledge, PHASE is the first instance of transferal of neural networks to create physiological signal embeddings.

2) We present a tractable method to obtain feature attributions through stacked models.

We prove that our stacked model attributions can approximate Shapley values -- attributions known to have desirable properties -- for arbitrary sets of models.

3) PHASE was extensively tested in a cross-hospital setting including publicly available data.

In our experiments, we show that PHASE significantly outperforms alternative embeddings -- such as raw, exponential moving average/variance, and autoencoder -- currently in use.

Furthermore, we provide evidence that transferring neural network embedding/representation learners between distinct hospitals still yields performant embeddings and offer recommendations when transference is ineffective.

Physiological signal embeddings for prediction performance and hospital transference with a general Shapley value interpretability method for stacked models.

Physiological Signal Embeddings (PHASE) via Interpretable Stacked Models


We consider the dictionary learning problem, where the aim is to model the given data as a linear combination of a few columns of a matrix known as a dictionary, where the sparse weights forming the linear combination are known as coefficients.

Since the dictionary and coefficients, parameterizing the linear model are unknown, the corresponding optimization is inherently non-convex.

This was a major challenge until recently, when provable algorithms for dictionary learning were proposed.

Yet, these provide guarantees only on the recovery of the dictionary, without explicit recovery guarantees on the coefficients.

Moreover, any estimation error in the dictionary adversely impacts the ability to successfully localize and estimate the coefficients.

This potentially limits the utility of existing provable dictionary learning methods in applications where coefficient recovery is of interest.

To this end, we develop NOODL: a simple Neurally plausible alternating Optimization-based Online Dictionary Learning algorithm, which recovers both the dictionary and coefficients exactly at a geometric rate, when initialized appropriately.

Our algorithm, NOODL, is also scalable and amenable for large scale distributed implementations in neural architectures, by which we mean that it only involves simple linear and non-linear operations.

Finally, we corroborate these theoretical results via experimental evaluation of the proposed algorithm with the current state-of-the-art techniques.

We present a provable algorithm for exactly recovering both factors of the dictionary learning model.

NOODL: Provable Online Dictionary Learning and Sparse Coding


Real-world Relation Extraction (RE) tasks are challenging to deal with, either due to limited training data or class imbalance issues.

In this work, we present Data Augmented Relation Extraction (DARE), a simple method to augment training data by properly finetuning GPT2 to generate examples for specific relation types.

The generated training data is then used in combination with the gold dataset to train a BERT-based RE classifier.

In a series of experiments we show the advantages of our method, which leads in improvements of up to 11 F1 score points compared to a strong baseline.

Also, DARE achieves new state-of-the-art in three widely used biomedical RE datasets surpassing the previous best results by 4.7 F1 points on average.

DARE: Data Augmented Relation Extraction with GPT-2


This paper addresses the problem of representing a system's belief using multi-variate normal distributions (MND) where the underlying model is based on a deep neural network (DNN).

The major challenge with DNNs is the computational complexity that is needed to obtain model uncertainty using MNDs.

To achieve a scalable method, we propose a novel approach that expresses the parameter posterior in sparse information form.

Our inference algorithm is based on a novel Laplace Approximation scheme, which involves a diagonal correction of the Kronecker-factored eigenbasis.

As this makes the inversion of the information matrix intractable - an operation that is required for full Bayesian analysis, we devise a low-rank   approximation of this eigenbasis and a memory-efficient sampling scheme.

We provide both a theoretical analysis and an empirical evaluation on various benchmark data sets, showing the superiority of our approach over existing methods.

An approximate inference algorithm for deep learning

Representing Model Uncertainty of Neural Networks in Sparse Information Form


The ever-increasing size of modern datasets combined with the difficulty of obtaining label information has made semi-supervised learning of significant practical importance in modern machine learning applications.

In comparison to supervised learning, the key difficulty in semi-supervised learning is how to make full use of the unlabeled data.

In order to utilize manifold information provided by unlabeled data, we propose a novel regularization called the tangent-normal adversarial regularization, which is composed by two parts.

The two parts complement with each other and jointly enforce the smoothness along two different directions that are crucial for semi-supervised learning.

One is applied along the tangent space of the data manifold, aiming to enforce local invariance of the classifier on the manifold, while the other is performed on the normal space orthogonal to the tangent space, intending to impose robustness on the classifier against the noise causing the observed data deviating from the underlying data manifold.

Both of the two regularizers are achieved by the strategy of virtual adversarial training.

Our method has achieved state-of-the-art performance on semi-supervised learning tasks on both artificial dataset and practical datasets.

We propose a novel manifold regularization strategy based on adversarial training, which can significantly improve the performance of semi-supervised learning.

Tangent-Normal Adversarial Regularization for Semi-supervised Learning

semi-supervised learning
adversarial
regularization


Universal approximation property of neural networks is one of the motivations to use these models in various real-world problems.

However, this property is not the only characteristic that makes neural networks unique as there is a wide range of other approaches with similar property.

Another characteristic which makes these models interesting is that they can be trained with the backpropagation algorithm which allows an efficient gradient computation and gives these universal approximators the ability to efficiently learn complex manifolds from a large amount of data in different domains.

Despite their abundant use in practice, neural networks are still not well understood and a broad range of ongoing research is to study the interpretability of neural networks.

On the other hand, topological data analysis (TDA) relies on strong theoretical framework of (algebraic) topology along with other mathematical tools for analyzing possibly complex datasets.

In this work, we leverage a universal approximation theorem originating from algebraic topology to build a connection between TDA and common neural network training framework.

We introduce the notion of automatic subdivisioning and devise a particular type of neural networks for regression tasks: Simplicial Complex Networks (SCNs).

SCN's architecture is defined with a set of bias functions along with a particular policy during the forward pass which alternates the common architecture search framework in neural networks.

We believe the view of SCNs can be used as a step towards building interpretable deep learning models.

Finally, we verify its performance on a set of regression problems.

A novel method for supervised learning through subdivisioning the input space along with function approximation.

Simplicial Complex Networks


The goal of network representation learning is to learn low-dimensional node embeddings that capture the graph structure and are useful for solving downstream tasks.

However, despite the proliferation of such methods there is currently no study of their robustness to adversarial attacks.

We provide the first adversarial vulnerability analysis on the widely used family of methods based on random walks.

We derive efficient adversarial perturbations that poison the network structure and have a negative effect on both the quality of the embeddings and the downstream tasks.

We further show that our attacks are transferable since they generalize to many models, and are successful even when the attacker is restricted.

Adversarial attacks on unsupervised node embeddings based on eigenvalue perturbation theory.

Adversarial Attacks on Node Embeddings


Hierarchical reinforcement learning methods offer a powerful means of planning flexible behavior in complicated domains.

However, learning an appropriate hierarchical decomposition of a domain into subtasks remains a substantial challenge.

We present a novel algorithm for subtask discovery, based on the recently introduced multitask linearly-solvable Markov decision process (MLMDP) framework.

The MLMDP can perform never-before-seen tasks by representing them as a linear combination of a previously learned basis set of tasks.

In this setting, the subtask discovery problem can naturally be posed as finding an optimal low-rank approximation of the set of tasks the agent will face in a domain.

We use non-negative matrix factorization to discover this minimal basis set of tasks, and show that the technique learns intuitive decompositions in a variety of domains.

Our method has several qualitatively desirable features: it is not limited to learning subtasks with single goal states, instead learning distributed patterns of preferred states; it learns qualitatively different hierarchical decompositions in the same domain depending on the ensemble of tasks the agent will face; and it may be straightforwardly iterated to obtain deeper hierarchical decompositions.

We present a novel algorithm for hierarchical subtask discovery which leverages the multitask linear Markov decision process framework.

Hierarchical Subtask Discovery with Non-Negative Matrix Factorization


This paper introduces a framework for solving combinatorial optimization problems by learning from input-output examples of optimization problems.

We introduce a new memory augmented neural model in which the memory is not resettable (i.e the information stored in the memory after processing an input example is kept for the next seen examples).

We used deep reinforcement learning to train a memory controller agent to store useful memories.

Our model was able to outperform hand-crafted solver on Binary Linear Programming (Binary LP).

The proposed model is tested on different Binary LP instances with large number of variables (up to 1000 variables) and constrains (up to 700 constrains).

We propose a memory network model to solve Binary LP instances where the memory information is perseved for long-term use.

Long Term Memory Network for Combinatorial Optimization Problems


Sequence-to-sequence (Seq2Seq) models with attention have excelled at tasks which involve generating natural language sentences such as machine translation, image captioning and speech recognition.

Performance has further been improved by leveraging unlabeled data, often in the form of a language model.

In this work, we present the Cold Fusion method, which leverages a pre-trained language model during training, and show its effectiveness on the speech recognition task.

We show that Seq2Seq models with Cold Fusion are able to better utilize language information enjoying

i) faster convergence and better generalization, and

ii) almost complete transfer to a new domain while using less than 10% of the labeled training data.

We introduce a novel method to train Seq2Seq models with language models that converge faster, generalize better and can almost completely transfer to a new domain using less than 10% of labeled data.

COLD FUSION: TRAINING SEQ2SEQ MODELS TOGETHER WITH LANGUAGE MODELS


A central capability of intelligent systems is the ability to continuously build upon previous experiences to speed up and enhance learning of new tasks.

Two distinct research paradigms have studied this question.

Meta-learning views this problem as learning a prior over model parameters that is amenable for fast adaptation on a new task, but typically assumes the set of tasks are available together as a batch.

In contrast, online (regret based) learning considers a sequential setting in which problems are revealed one after the other, but conventionally train only a single model without any task-specific adaptation.

This work introduces an online meta-learning setting, which merges ideas from both the aforementioned paradigms to better capture the spirit and practice of continual lifelong learning.

We propose the follow the meta leader (FTML) algorithm which extends the MAML algorithm to this setting.

Theoretically, this work provides an O(logT) regret guarantee for the FTML algorithm.

Our experimental evaluation on three different large-scale tasks suggest that the proposed algorithm significantly outperforms alternatives based on traditional online learning approaches.

We introduce the online meta learning problem setting to better capture the spirit and practice of continual lifelong learning.

Online Meta-Learning


We investigate the extent to which individual attention heads in pretrained transformer language models, such as BERT and RoBERTa, implicitly capture syntactic dependency relations.

We employ two methods—taking the maximum attention weight and computing the maximum spanning tree—to extract implicit dependency relations from the attention weights of each layer/head, and compare them to the ground-truth Universal Dependency (UD) trees.

We show that, for some UD relation types, there exist heads that can recover the dependency type significantly better than baselines on parsed English text, suggesting that some self-attention heads act as a proxy for syntactic structure.

We also analyze BERT fine-tuned on two datasets—the syntax-oriented CoLA and the semantics-oriented MNLI—to investigate whether fine-tuning affects the patterns of their self-attention, but we do not observe substantial differences in the overall dependency relations extracted using our methods.

Our results suggest that these models have some specialist attention heads that track individual dependency types, but no generalist head that performs holistic parsing significantly better than a trivial baseline, and that analyzing attention weights directly may not reveal much of the syntactic knowledge that BERT-style models are known to learn.

Attention weights don't fully expose what BERT knows about syntax.

Do Attention Heads in BERT Track Syntactic Dependencies?


State-of-the-art face super-resolution methods employ deep convolutional neural networks to learn a mapping between low- and high-resolution facial patterns by exploring local appearance knowledge.

However, most of these methods do not well exploit facial structures and identity information, and struggle to deal with facial images that exhibit large pose variation and misalignment.

In this paper, we propose a novel face super-resolution method that explicitly incorporates 3D facial priors which grasp the sharp facial structures.

Firstly, the 3D face rendering branch is set up to obtain 3D priors of salient facial structures and identity knowledge.

Secondly, the Spatial Attention Mechanism is used to better exploit this hierarchical information (i.e. intensity similarity, 3D facial structure, identity content) for the super-resolution problem.

Extensive experiments demonstrate that the proposed algorithm achieves superior face super-resolution results and outperforms the state-of-the-art.

We propose a novel face super resolution method that explicitly incorporates 3D facial priors which grasp the sharp facial structures.

FACE SUPER-RESOLUTION GUIDED BY 3D FACIAL PRIORS


We present Cross-View Training (CVT), a simple but effective method for deep semi-supervised learning.

On labeled examples, the model is trained with standard cross-entropy loss.

On an unlabeled example, the model first performs inference (acting as a "teacher") to produce soft targets.

The model then learns from these soft targets (acting as a ``"student").

We deviate from prior work by adding multiple auxiliary student prediction layers to the model.

The input to each student layer is a sub-network of the full model that has a restricted view of the input  (e.g., only seeing one region of an image).

The students can learn from the teacher (the full model) because the teacher sees more of each example.

Concurrently, the students improve the quality of the representations used by the teacher as they learn to make predictions with limited data.

When combined with Virtual Adversarial Training, CVT improves upon the current state-of-the-art on semi-supervised CIFAR-10 and semi-supervised SVHN.

We also apply CVT to train models on five natural language processing tasks using hundreds of millions of sentences of unlabeled data.

On all tasks CVT substantially outperforms supervised learning alone, resulting in models that improve upon or are competitive with the current state-of-the-art.

Self-training with different views of the input gives excellent results for semi-supervised image recognition, sequence tagging, and dependency parsing.

Cross-View Training for Semi-Supervised Learning


I show how it can be beneficial to express Metropolis accept/reject decisions in terms of comparison with a uniform [0,1] value, and to then update this uniform value non-reversibly, as part of the Markov chain state, rather than sampling it independently each iteration.

This provides a small improvement for random walk Metropolis and Langevin updates in high dimensions.

It produces a larger improvement when using Langevin updates with persistent momentum, giving performance comparable to that of Hamiltonian Monte Carlo (HMC) with long trajectories.

This is of significance when some variables are updated by other methods, since if HMC is used, these updates can be done only between trajectories, whereas they can be done more often with Langevin updates.

This is seen for a Bayesian neural network model, in which connection weights are updated by persistent Langevin or HMC, while hyperparameters are updated by Gibbs sampling.

A non-reversible way of making accept/reject decisions can be beneficial

Non-reversibly updating a uniform [0,1] value for accept/reject decisions


We introduce ES-MAML, a new framework for solving the model agnostic meta learning (MAML) problem based on Evolution Strategies (ES).

Existing algorithms for MAML are based on policy gradients, and incur significant difficulties when attempting to estimate second derivatives using backpropagation on stochastic policies.

We show how ES can be applied to MAML to obtain an algorithm which avoids the problem of estimating second derivatives, and is also conceptually simple and easy to implement.

Moreover, ES-MAML can handle new types of nonsmooth adaptation operators, and other techniques for improving performance and estimation of ES methods become applicable.

We show empirically that ES-MAML is competitive with existing methods and often yields better adaptation with fewer queries.

We provide a new framework for MAML in the ES/blackbox setting, and show that it allows deterministic and linear policies, better exploration, and non-differentiable adaptation operators.

ES-MAML: Simple Hessian-Free Meta Learning


Transforming one probability distribution to another is a powerful tool in Bayesian inference and machine learning.

Some prominent examples are constrained-to-unconstrained transformations of distributions for use in Hamiltonian Monte-Carlo and constructing flexible and learnable densities such as normalizing flows.

We present Bijectors.jl, a software package for transforming distributions implemented in Julia, available at github.com/TuringLang/Bijectors.jl.

The package provides a flexible and composable way of implementing transformations of distributions without being tied to a computational framework.

We demonstrate the use of Bijectors.jl on improving variational inference by encoding known statistical dependencies into the variational posterior using normalizing flows, providing a general approach to relaxing the mean-field assumption usually made in variational inference.

We present a software framework for transforming distributions and demonstrate its flexibility on relaxing mean-field assumptions in variational inference with the use of coupling flows to replicate structure from the target generative model.

Bijectors.jl: Flexible transformations for probability distributions


Multi-domain learning (MDL) aims at obtaining a model with minimal average risk across multiple domains.

Our empirical motivation is automated microscopy data, where cultured cells are imaged after being exposed to known and unknown chemical perturbations, and each dataset displays significant experimental bias.

This paper presents a multi-domain adversarial learning approach, MuLANN, to leverage multiple datasets with overlapping but distinct class sets, in a semi-supervised setting.

i) a bound on the average- and worst-domain risk in MDL, obtained using the H-divergence;

ii) a new loss to accommodate semi-supervised multi-domain learning and domain adaptation;

iii) the experimental validation of the approach, improving on the state of the art on two standard image benchmarks, and a novel bioimage dataset, Cell.

Adversarial Domain adaptation and Multi-domain learning: a new loss to handle multi- and single-domain classes in the semi-supervised setting.

Multi-Domain Adversarial Learning


We introduce our Distribution Regression Network (DRN) which performs regression from input probability distributions to output probability distributions.

Compared to existing methods, DRN learns with fewer model parameters and easily extends to multiple input and multiple output distributions.

On synthetic and real-world datasets, DRN performs similarly or better than the state-of-the-art.

Furthermore, DRN generalizes the conventional multilayer perceptron (MLP).

In the framework of MLP, each node encodes a real number, whereas in DRN, each node encodes a probability distribution.

A learning network which generalizes the MLP framework to perform distribution-to-distribution regression

Distribution Regression Network


Existing sequence prediction methods are mostly concerned with time-independent sequences, in which the actual time span between events is irrelevant and the distance between events is simply the difference between their order positions in the sequence.

While this time-independent view of sequences is applicable for data such as natural languages, e.g., dealing with words in a sentence, it is inappropriate and inefficient for many real world events that are observed and collected at unequally spaced points of time as they naturally arise, e.g., when a person goes to a grocery store or makes a phone call.

The time span between events can carry important information about the sequence dependence of human behaviors.

In this work, we propose a set of methods for using time in sequence prediction.

Because neural sequence models such as RNN are more amenable for handling token-like input, we propose two methods for time-dependent event representation, based on the intuition on how time is tokenized in everyday life and previous work on embedding contextualization.

We also introduce two methods for using next event duration as regularization for training a sequence prediction model.

We discuss these methods based on recurrent neural nets.

We evaluate these methods as well as baseline models on five datasets that resemble a variety of sequence prediction tasks.

The experiments revealed that the proposed methods offer accuracy gain over baseline models in a range of settings.

Proposed methods for time-dependent event representation and regularization for sequence prediction; Evaluated these methods on five datasets that involve a range of sequence prediction tasks.

Time-Dependent Representation for Neural Event Sequence Prediction

sequence prediction
representation
event
time-dependent


Off-policy reinforcement learning algorithms promise to be applicable in settings where only a fixed data-set (batch) of environment interactions is available and no new experience can be acquired.

This property makes these algorithms appealing for real world problems such as robot control.

In practice, however, standard off-policy algorithms fail in the batch setting for continuous control.

In this paper, we propose a simple solution to this problem.

It admits the use of data generated by arbitrary behavior policies and uses a learned prior -- the advantage-weighted behavior model (ABM) -- to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task.

Our method can be seen as an extension of recent work on batch-RL that enables stable learning from conflicting data-sources.

We find  improvements on competitive baselines in a variety of RL tasks -- including standard continuous control benchmarks and multi-task learning for simulated and real-world robots.

We develop a method for stable offline reinforcement learning from logged data. The key is to regularize the RL policy towards a learned "advantage weighted" model of the data.

Keep Doing What Worked: Behavior Modelling Priors for Offline Reinforcement Learning


One of the main challenges in applying graph convolutional neural networks on gene-interaction data is the lack of understanding of the vector space  to which they belong and also the inherent difficulties involved in representing those interactions on a significantly lower dimension, viz Euclidean spaces.

The challenge becomes more prevalent when dealing with various types of heterogeneous data.

We introduce a systematic, generalized method, called iSOM-GSN, used to transform ``multi-omic'' data with higher dimensions onto a two-dimensional grid.

Afterwards, we apply a convolutional neural network to predict disease states of various types.

Based on the idea of Kohonen's self-organizing map, we generate a two-dimensional grid for each sample for a given set of genes that represent a gene similarity network.

We have tested the model to predict breast and prostate cancer using gene expression, DNA methylation and copy number alteration, yielding prediction accuracies in the 94-98% range for tumor stages of breast cancer and calculated Gleason scores of prostate cancer with just 11 input genes for both cases.

The scheme not only outputs nearly perfect classification accuracy, but also provides an enhanced scheme for representation learning, visualization, dimensionality reduction, and interpretation of the results.

This paper presents a deep learning model that combines self-organizing maps and convolutional neural networks for representation learning of multi-omics data

iSOM-GSN: An Integrative Approach for Transforming Multi-omic Data into Gene Similarity Networks via Self-organizing Maps


State-of-the-art performances on language comprehension tasks are achieved by huge language models pre-trained on massive unlabeled text corpora, with very light subsequent fine-tuning in a task-specific supervised manner.

It seems the pre-training procedure learns a very good common initialization for further training on various natural language understanding tasks, such that only few steps need to be taken in the parameter space to learn each task.

In this work, using Bidirectional Encoder Representations from Transformers (BERT) as an example, we verify this hypothesis by showing that task-specific fine-tuned language models are highly close in parameter space to the pre-trained one.

Taking advantage of such observations, we further show that the fine-tuned versions of these huge models, having on the order of $10^8$ floating-point parameters, can be made very computationally efficient.

First, fine-tuning only a fraction of critical layers suffices.

Second, fine-tuning can be adequately performed by learning a binary multiplicative mask on pre-trained weights, \textit{i.e.} by parameter-sparsification.

As a result, with a single effort, we achieve three desired outcomes: (1) learning to perform specific tasks, (2) saving memory by storing only binary masks of certain layers for each task, and (3) saving compute on appropriate hardware by performing sparse operations with model parameters.

Sparsification as fine-tuning of language models

How fine can fine-tuning be?  Learning efficient language models


We present a simple and effective algorithm designed to address the covariate shift problem in imitation learning.

It operates by training an ensemble of policies on the expert demonstration data, and using the variance of their predictions as a cost which is minimized with RL together with a supervised behavioral cloning cost.

Unlike adversarial imitation methods, it uses a fixed reward function which is easy to optimize.

We prove a regret bound for the algorithm in the tabular setting which is linear in the time horizon multiplied by a coefficient which we show to be low for certain problems in which behavioral cloning fails.

We evaluate our algorithm empirically across multiple pixel-based Atari environments and continuous control tasks, and show that it matches or significantly outperforms behavioral cloning and generative adversarial imitation learning.

Method for addressing covariate shift in imitation learning using ensemble uncertainty

Disagreement-Regularized Imitation Learning


We present and discuss a simple image preprocessing method for learning disentangled latent factors.

In particular, we utilize the implicit inductive bias contained in features from networks pretrained on the ImageNet database.

We enhance this bias by explicitly fine-tuning such pretrained networks on tasks useful for the NeurIPS2019 disentanglement challenge, such as angle and position estimation or color classification.

Furthermore, we train a VAE on regionally aggregate feature maps, and discuss its disentanglement performance using metrics proposed in recent literature.

We use supervised finetuning of feature vectors to improve transfer from simulation to the real world

Improved Disentanglement through Learned Aggregation of Convolutional Feature Maps


A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy.

In addition to their potential to generalize desirable behavior to unseen goals, such policies may also enable higher-level planning based on subgoals.

In sparse-reward environments, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended appears crucial to enable sample efficient learning.

However, reinforcement learning agents have only recently been endowed with such capacity for hindsight.

In this paper, we demonstrate how hindsight can be introduced to policy gradient methods, generalizing this idea to a broad class of successful algorithms.

Our experiments on a diverse selection of sparse-reward environments show that hindsight leads to a remarkable increase in sample efficiency.

We introduce the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended to policy gradient methods.

Hindsight policy gradients


Computer vision has undergone a dramatic revolution in performance, driven in large part through deep features trained on large-scale supervised datasets.

However, much of these improvements have focused on static image analysis; video understanding has seen rather modest improvements.

Even though new datasets and spatiotemporal models have been proposed, simple frame-by-frame classification methods often still remain competitive.

We posit that current video datasets are plagued with implicit biases over scene and object structure that can dwarf variations in temporal structure.

In this work, we build a video dataset with fully observable and controllable object and scene bias, and which truly requires spatiotemporal understanding in order to be solved.

Our dataset, named CATER, is rendered synthetically using a library of standard 3D objects, and tests the ability to recognize compositions of object movements that require long-term reasoning.

In addition to being a challenging dataset, CATER also provides a plethora of diagnostic tools to analyze modern spatiotemporal video architectures by being completely observable and controllable.

Using CATER, we provide insights into some of the most recent state of the art deep video architectures.

We propose a new video understanding benchmark, with tasks that by-design require temporal reasoning to be solved, unlike most existing video datasets.

CATER: A diagnostic dataset for Compositional Actions & TEmporal Reasoning


We address the efficiency issues caused by the straggler effect in the recently emerged federated learning, which collaboratively trains a model on decentralized non-i.i.d. (non-independent and identically distributed) data across massive worker devices without exchanging training data in the unreliable and heterogeneous networks.

We propose a novel two-stage analysis on the error bounds of general federated learning, which provides practical insights into optimization.

As a result, we propose a novel easy-to-implement federated learning algorithm that uses asynchronous settings and strategies to control discrepancies between the global model and delayed models and adjust the number of local epochs with the estimation of staleness to accelerate convergence and resist performance deterioration caused by stragglers.

Experiment results show that our algorithm converges fast and robust on the existence of massive stragglers.

We propose an efficient and robust asynchronous federated learning algorithm on the existence of stragglers

Efficient and Robust Asynchronous Federated Learning with Stragglers

efficient robust asynchronous federate
straggler


Long Short-Term Memory (LSTM) units have the ability to memorise and use long-term dependencies between inputs to generate predictions on time series data.

We introduce the concept of modifying the cell state (memory) of LSTMs using rotation matrices parametrised by a new set of trainable weights.

This addition shows significant increases of performance on some of the tasks from the bAbI dataset.

Adding a new set of weights to the LSTM that rotate the cell memory improves performance on some bAbI tasks.

Modifying memories in a Recurrent Neural Network Unit


We address the problem of marginal inference for an exponential family defined over the set of permutation matrices.

This problem is known to quickly become intractable as the size of the permutation increases, since its involves the computation of the permanent of a matrix, a #P-hard problem.

We introduce Sinkhorn variational marginal inference as a scalable alternative, a method whose validity is ultimately justified by the so-called Sinkhorn approximation of the permanent.

We demonstrate the efectiveness of our method in the problem of probabilistic identification of neurons in the worm C.elegans

New methodology for variational marginal inference of permutations based on Sinkhorn algorithm, applied to probabilistic identification of neurons

Sinkhorn Permutation  Variational Marginal Inference

variational marginal inference
sinkhorn
permutation


The robustness of neural networks to adversarial examples has received great attention due to security implications.

Despite various attack approaches to crafting visually imperceptible adversarial examples, little has been developed towards a comprehensive measure of robustness.

In this paper, we provide theoretical justification for converting robustness analysis into a local Lipschitz constant estimation problem, and propose to use the Extreme Value Theory for efficient evaluation.

Our analysis yields a novel robustness metric called CLEVER, which is short for Cross Lipschitz Extreme Value for nEtwork Robustness.

The proposed CLEVER score is attack-agnostic and is computationally feasible for large neural networks.

Experimental results on various networks, including ResNet, Inception-v3 and MobileNet, show that

(i) CLEVER is aligned with the robustness indication measured by the $\ell_2$ and $\ell_\infty$ norms of adversarial examples from powerful attacks, and

(ii) defended networks using defensive distillation or bounded ReLU indeed give better CLEVER scores.

To the best of our knowledge, CLEVER is the first attack-independent robustness metric that can be applied to any neural network classifiers.

We propose the first attack-independent robustness metric, a.k.a CLEVER, that can be applied to any neural network classifier.

Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach


Multi-agent collaboration is required by numerous real-world problems.

Although distributed setting is usually adopted by practical systems, local range communication and information aggregation still matter in fulfilling complex tasks.

For multi-agent reinforcement learning, many previous studies have been dedicated to design an effective communication architecture.

However, existing models usually suffer from an ossified communication structure, e.g., most of them predefine a particular communication mode by specifying a fixed time frequency and spatial scope for agents to communicate regardless of necessity.

Such design is incapable of dealing with multi-agent scenarios that are capricious and complicated, especially when only partial information is available.

Motivated by this, we argue that the solution is to build a spontaneous and self-organizing communication (SSoC) learning scheme.

By treating the communication behaviour as an explicit action, SSoC learns to organize communication in an effective and efficient way.

Particularly, it enables each agent to spontaneously decide when and who to send messages based on its observed states.

In this way, a dynamic inter-agent communication channel is established in an online and self-organizing manner.

The agents also learn how to adaptively aggregate the received messages and its own hidden states to execute actions.

Various experiments have been conducted to demonstrate that SSoC really learns intelligent message passing among agents located far apart.

With such agile communications, we observe that effective collaboration tactics emerge which have not been mastered by the compared baselines.

This paper proposes a spontaneous and self-organizing communication (SSoC) learning scheme for multi-agent RL tasks.

SSoC: Learning Spontaneous and Self-Organizing Communication for Multi-Agent Collaboration


We study the BERT language representation model and the sequence generation model with BERT encoder for multi-label text classification task.

We experiment with both models and explore their special qualities for this setting.

We also introduce and examine experimentally a mixed model, which is an ensemble of multi-label BERT and sequence generating BERT models.

Our experiments demonstrated that BERT-based models and the mixed model, in particular, outperform current baselines in several metrics achieving state-of-the-art results on three well-studied multi-label classification datasets with English texts and two private Yandex Taxi datasets with Russian texts.

On using BERT as an encoder for sequential prediction of labels in multi-label text classification task

BERT for Sequence-to-Sequence Multi-Label Text Classification


Click Through Rate (CTR) prediction is a critical task in industrial applications, especially for online social and commerce applications.

It is challenging to find a proper way to automatically discover the effective cross features in CTR tasks.

We propose a novel model for CTR tasks, called Deep neural networks with Encoder enhanced Factorization Machine (DeepEnFM).

Instead of learning the cross features directly, DeepEnFM adopts the Transformer encoder as a backbone to align the feature embeddings with the clues of other fields.

The embeddings generated from encoder are beneficial for the further feature interactions.

Particularly, DeepEnFM utilizes a bilinear approach to generate different similarity functions with respect to different field pairs.

Furthermore, the max-pooling method makes DeepEnFM feasible to capture both the supplementary and suppressing information among different attention heads.

Our model is validated on the Criteo and Avazu datasets, and achieves state-of-art performance.

DNN and Encoder enhanced FM with bilinear attention and max-pooling for CTR

DeepEnFM: Deep neural networks with Encoder enhanced Factorization Machine


For autonomous agents to successfully operate in the real world, the ability to anticipate future scene states is a key competence.

In real-world scenarios, future states become increasingly uncertain and multi-modal, particularly on long time horizons.

Dropout based Bayesian inference provides a computationally tractable, theoretically well grounded approach to learn different hypotheses/models to deal with uncertain futures and make predictions that correspond well to observations -- are well calibrated.

However, it turns out that such approaches fall short to capture complex real-world scenes, even falling behind in accuracy when compared to the plain deterministic approaches.

This is because the used log-likelihood estimate discourages diversity.

In this work, we propose a novel Bayesian formulation for anticipating future scene states which leverages synthetic likelihoods that encourage the learning of diverse models to accurately capture the multi-modal nature of future scene states.

We show that our approach achieves accurate state-of-the-art predictions and calibrated probabilities through extensive experiments for scene anticipation on Cityscapes dataset.

Moreover, we show that our approach generalizes across diverse tasks such as digit generation and precipitation forecasting.

Dropout based Bayesian inference is extended to deal with multi-modality and is evaluated on scene anticipation tasks.

Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods


Conditional generative adversarial networks (cGAN) have led to large improvements in the task of conditional image generation, which lies at the heart of computer vision.

The major focus so far has been on performance improvement, while there has been little effort in making cGAN more robust to noise.

The regression (of the generator) might lead to arbitrarily large errors in the output, which makes cGAN unreliable for real-world applications.

In this work, we introduce a novel conditional GAN model, called RoCGAN, which leverages structure in the target space of the model to address the issue.

Our model augments the generator with an unsupervised pathway, which promotes the outputs of the generator to span the target manifold even in the presence of intense noise.

We prove that RoCGAN share similar theoretical properties as GAN and experimentally verify that our model outperforms existing state-of-the-art cGAN architectures by a large margin in a variety of domains including images from natural scenes and faces.

We introduce a new type of conditional GAN, which aims to leverage structure in the target space of the generator. We augment the generator with a new, unsupervised pathway to learn the target structure.

Robust Conditional Generative Adversarial Networks


Though deep neural networks have achieved the state of the art performance in visual classification, recent studies have shown that they are all vulnerable to the attack of adversarial examples.

To solve the problem, some regularization adversarial training methods, constraining the output label or logit, have been studied.

In this paper, we propose a novel regularized adversarial training framework ATLPA,namely Adversarial Tolerant Logit Pairing with Attention.

Instead of constraining a hard distribution (e.g., one-hot vectors or logit) in adversarial training, ATLPA uses Tolerant Logit which consists of confidence distribution on top-k classes and captures inter-class similarities at the image level.

Specifically, in addition to minimizing the empirical loss, ATLPA encourages attention map for pairs of examples to be similar.

When applied to clean examples and their adversarial counterparts, ATLPA improves accuracy on adversarial examples over adversarial training.

We evaluate ATLPA with the state of the art algorithms, the experiment results show that our method outperforms these baselines with higher accuracy.

Compared with previous work, our work is evaluated under highly challenging PGD attack: the maximum perturbation $\epsilon$ is 64 and 128 with 10 to 200 attack iterations.

ATLPA:ADVERSARIAL TOLERANT LOGIT PAIRING WITH ATTENTION FOR CONVOLUTIONAL NEURAL NETWORK

adversarial tolerant logit pair attention


A fundamental trait of intelligence is the ability to achieve goals in the face of novel circumstances.

In this work, we address one such setting which requires solving a task with a novel set of actions.

Empowering machines with this ability requires generalization in the way an agent perceives its available actions along with the way it uses these actions to solve tasks.

Hence, we propose a framework to enable generalization over both these aspects: understanding an action’s functionality, and using actions to solve tasks through reinforcement learning.

Specifically, an agent interprets an action’s behavior using unsupervised representation learning over a collection of data samples reflecting the diverse properties of that action.

We employ a reinforcement learning architecture which works over these action representations, and propose regularization metrics essential for enabling generalization in a policy.

We illustrate the generalizability of the representation learning method and policy, to enable zero-shot generalization to previously unseen actions on challenging sequential decision-making environments.

Our results and videos can be found at sites.google.com/view/action-generalization/

We address the problem of generalization of reinforcement learning to unseen action spaces.

Generalizing Reinforcement Learning to Unseen Actions


Temporal point processes are the dominant paradigm for modeling sequences of events happening at irregular intervals.

The standard way of learning in such models is by estimating the conditional intensity function.

However, parameterizing the intensity function usually incurs several trade-offs.

We show how to overcome the limitations of intensity-based approaches by directly modeling the conditional distribution of inter-event times.

We draw on the literature on normalizing flows to design models that are flexible and efficient.

We additionally propose a simple mixture model that matches the flexibility of flow-based models, but also permits sampling and computing moments in closed form.

The proposed models achieve state-of-the-art performance in standard prediction tasks and are suitable for novel applications, such as learning sequence embeddings and imputing missing data.

Learn in temporal point processes by modeling the conditional density, not the conditional intensity.

Intensity-Free Learning of Temporal Point Processes


We propose a novel yet simple neural network architecture for topic modelling.

The method is based on training an autoencoder structure where the bottleneck represents the space of the topics distribution and the decoder outputs represent the space of the words distributions over the topics.

We exploit an auxiliary decoder to prevent mode collapsing in our model.

A key feature for an effective topic modelling method is having sparse topics and words distributions, where there is a trade-off between the sparsity level of topics and words.

This feature is implemented in our model by L-2 regularization and the model hyperparameters take care of the trade-off.

We show in our experiments that our model achieves competitive results compared to the state-of-the-art deep models for topic modelling, despite its simple architecture and training procedure.

The “New York Times” and “20 Newsgroups” datasets are used in the experiments.

Autoencoding Documents for Topic Modeling with L-2 Sparsity Regularization


Voice Conversion (VC) is a task of converting perceived speaker identity from a source speaker to a particular target speaker.

Earlier approaches in the literature primarily find a mapping between the given source-target speaker-pairs.

Developing mapping techniques for many-to-many VC using non-parallel data, including zero-shot learning remains less explored areas in VC.

Most of the many-to-many VC architectures require training data from all the target speakers for whom we want to convert the voices.

In this paper, we propose a novel style transfer architecture, which can also be extended to generate voices even for target speakers whose data were not used in the training (i.e., case of zero-shot learning).

In particular, propose Adaptive Generative Adversarial Network (AdaGAN), new architectural training procedure help in learning normalized speaker-independent latent representation, which will be used to generate speech with different speaking styles in the context of VC.

We compare our results with the state-of-the-art StarGAN-VC architecture.

In particular, the AdaGAN achieves 31.73%, and 10.37% relative improvement compared to the StarGAN in MOS tests for speech quality and speaker similarity, respectively.

The key strength of the proposed architectures is that it yields these results with less computational complexity.

AdaGAN is 88.6% less complex than StarGAN-VC in terms of FLoating Operation Per Second (FLOPS), and 85.46% less complex in terms of trainable parameters.

Novel adaptive instance normalization based GAN framework for non parallel many-to-many and zero-shot VC.

AdaGAN: Adaptive GAN for Many-to-Many Non-Parallel Voice Conversion


Self-attention-based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks.

Self attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context.

To tackle the problem, we propose a novel model called Sparse Transformer.

Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments.

Extensive experimental results on a series of natural language processing tasks, including neural machine translation, image captioning, and language modeling, all demonstrate the advantages of Sparse Transformer in model performance.

Sparse Transformer reaches the state-of-the-art performances in the IWSLT 2015 English-to-Vietnamese translation and IWSLT 2014 German-to-English translation.

In addition, we conduct qualitative analysis to account for Sparse Transformer's superior performance.

This work propose Sparse Transformer to improve the concentration of attention on the global context through an explicit selection of the most relevant segments for sequence to sequence learning.

Sparse Transformer: Concentrated Attention Through Explicit Selection

explicit selection
sparse transformer
attention


Human observers can learn to recognize new categories of objects from a handful of examples, yet doing so with machine perception remains an open challenge.

We hypothesize that data-efficient recognition is enabled by representations which make the variability in natural signals more predictable, as suggested by recent perceptual evidence.

We therefore revisit and improve Contrastive Predictive Coding, a recently-proposed unsupervised learning framework, and arrive at a representation which enables generalization from small amounts of labeled data.

When provided with only 1% of ImageNet labels (i.e. 13 per class), this model retains a strong classification performance, 73% Top-5 accuracy, outperforming supervised networks by 28% (a 65% relative improvement) and state-of-the-art semi-supervised methods by 14%.

We also find this representation to serve as a useful substrate for object detection on the PASCAL-VOC 2007 dataset, approaching the performance of representations trained with a fully annotated ImageNet dataset.

Unsupervised representations learned with Contrastive Predictive Coding enable data-efficient image classification.

Data-Efficient Image Recognition with Contrastive Predictive Coding

contrastive predictive
data-efficient image


We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels.

We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to identify layers and weights which reduce the error efficiently.

Sparse momentum redistributes pruned weights across layers according to the mean momentum magnitude of each layer.

Within a layer, sparse momentum grows weights according to the momentum magnitude of zero-valued weights.

We demonstrate state-of-the-art sparse performance on MNIST, CIFAR-10, and ImageNet, decreasing the mean error by a relative 8%, 15%, and 6% compared to other sparse algorithms.

Furthermore, we show that sparse momentum reliably reproduces dense performance levels while providing up to 5.61x faster training.

In our analysis, ablations show that the benefits of momentum redistribution and growth increase with the depth and size of the network.

Redistributing and growing weights according to the momentum magnitude enables the training of sparse networks from random initializations that can reach dense performance levels with 5% to 50% weights while accelerating training by up to 5.6x.

Sparse Networks from Scratch: Faster Training without Losing Performance


To provide principled ways of designing proper Deep Neural Network (DNN) models, it is essential to understand the loss surface of DNNs under realistic assumptions.

We introduce interesting aspects for understanding the local minima and overall structure of the loss surface.

The parameter domain of the loss surface can be decomposed into regions in which activation values (zero or one for rectified linear units) are consistent.

We found that, in each region, the loss surface have properties similar to that of linear neural networks where every local minimum is a global minimum.

This means that every differentiable local minimum is the global minimum of the corresponding region.

We prove that for a neural network with one hidden layer using rectified linear units under realistic assumptions.

There are poor regions that lead to poor local minima, and we explain why such regions exist even in the overparameterized DNNs.

The loss surface of neural networks is a disjoint union of regions where every local minimum is a global minimum of the corresponding region.

Understanding Local Minima in Neural Networks by Loss Surface Decomposition


Contextualized word representations, such as ELMo and BERT, were shown to perform well on a various of semantic and structural (syntactic) task.

In this work, we tackle the task of unsupervised disentanglement between semantics and structure in neural language representations: we aim to learn a transformation of the contextualized vectors, that discards the lexical semantics, but keeps the structural information.

To this end, we automatically generate groups of sentences which are structurally similar but semantically different, and use metric-learning approach to learn a transformation that emphasizes the structural component that is encoded in the vectors.

We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.

Finally, we demonstrate the utility of our distilled representations by showing that they outperform the original contextualized representations in few-shot parsing setting.

We distill language models representations for syntax by unsupervised metric learning

Unsupervised Distillation of Syntactic Information from Contextualized Word Representations


We introduce a new memory architecture for navigation in previously unseen environments, inspired by landmark-based navigation in animals.

The proposed semi-parametric topological memory (SPTM) consists of a (non-parametric) graph with nodes corresponding to locations in the environment and a (parametric) deep network capable of retrieving nodes from the graph based on observations.

The graph stores no metric information, only connectivity of locations corresponding to the nodes.

We use SPTM as a planning module in a navigation system.

Given only 5 minutes of footage of a previously unseen maze, an SPTM-based navigation agent can build a topological map of the environment and use it to confidently navigate towards goals.

The average success rate of the SPTM agent in goal-directed navigation across test environments is higher than the best-performing baseline by a factor of three.

Semi-parametric topological memory for navigation


The available resolution in our visual world is extremely high, if not infinite.

Existing CNNs can be applied in a fully convolutional way to images of arbitrary resolution, but as the size of the input increases, they can not capture contextual information.

In addition, computational requirements scale linearly to the number of input pixels, and resources are allocated uniformly across the input, no matter how informative different image regions are.

We attempt to address these problems by proposing a novel architecture that traverses an image pyramid in a top-down fashion, while it uses a hard attention mechanism to selectively process only the most informative image parts.

We conduct experiments on MNIST and ImageNet datasets, and we show that our models can significantly outperform fully convolutional counterparts, when the resolution of the input is that big that the receptive field of the baselines can not adequately cover the objects of interest.

Gains in performance come for less FLOPs, because of the selective processing that we follow.

Furthermore, our attention mechanism makes our predictions more interpretable, and creates a trade-off between accuracy and complexity that can be tuned both during training and testing time.

We propose a novel architecture that traverses an image pyramid in a top-down fashion, while it visits only the most informative regions along the way.

Image Classification Through Top-Down Image Pyramid Traversal


Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters.

We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters to optimal weights and biases.

We show how to construct scalable best-response approximations for neural networks by modeling the best-response as a single network whose hidden units are gated conditionally on the regularizer.

We justify this approximation by showing the exact best-response for a shallow linear network with L2-regularized Jacobian can be represented by a similar gating mechanism.

We fit this model using a gradient-based hyperparameter optimization algorithm which alternates between approximating the best-response around the current hyperparameters and optimizing the hyperparameters using the approximate best-response function.

Unlike other gradient-based approaches, we do not require differentiating the training loss with respect to the hyperparameters, allowing us to tune discrete hyperparameters, data augmentation hyperparameters, and dropout probabilities.

Because the hyperparameters are adapted online, our approach discovers hyperparameter schedules that can outperform fixed hyperparameter values.

Empirically, our approach outperforms competing hyperparameter optimization methods on large-scale deep learning problems.

We call our networks, which update their own hyperparameters online during training, Self-Tuning Networks (STNs).

We use a hypernetwork to predict optimal weights given hyperparameters, and jointly train everything together.

Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions


Conditional Generative Adversarial Networks (cGANs) are finding increasingly widespread use in many application domains.

Despite outstanding progress, quantitative evaluation of such models often involves multiple distinct metrics to assess different desirable properties, such as image quality, conditional consistency, and intra-conditioning diversity.

In this setting, model benchmarking becomes a challenge, as each metric may indicate a different "best" model.

In this paper, we propose the Frechet Joint Distance (FJD), which is defined as the Frechet distance between joint distributions of images and conditioning, allowing it to implicitly capture the aforementioned properties in a single metric.

We conduct proof-of-concept experiments on a controllable synthetic dataset, which consistently highlight the benefits of FJD when compared to currently established metrics.

Moreover, we use the newly introduced metric to compare existing cGAN-based models for a variety of conditioning modalities (e.g. class labels, object masks, bounding boxes, images, and text captions).

We show that FJD can be used as a promising single metric for model benchmarking.

We propose a new metric for evaluating conditional GANs that captures image quality, conditional consistency, and intra-conditioning diversity in a single measure.

On the Evaluation of Conditional GANs


Combining deep model-free reinforcement learning with on-line planning is a promising approach to building on the successes of deep RL.

On-line planning with look-ahead trees has proven successful in environments where transition models are known a priori.

However, in complex environments where transition models need to be learned from data, the deficiencies of learned models have limited their utility for planning.

To address these challenges, we propose TreeQN, a differentiable, recursive, tree-structured model that serves as a drop-in replacement for any value function network in deep RL with discrete actions.

TreeQN dynamically constructs a tree by recursively applying a transition model in a learned abstract state space and then aggregating predicted rewards and state-values using a tree backup to estimate Q-values.

We also propose ATreeC, an actor-critic variant that augments TreeQN with a softmax layer to form a stochastic policy network.

Both approaches are trained end-to-end, such that the learned model is optimised for its actual use in the tree.

We show that TreeQN and ATreeC outperform n-step DQN and A2C on a box-pushing task, as well as n-step DQN and value prediction networks (Oh et al., 2017) on multiple Atari games.

Furthermore, we present ablation studies that demonstrate the effect of different auxiliary losses on learning transition models.

We present TreeQN and ATreeC, new architectures for deep reinforcement learning in discrete-action domains that integrate differentiable on-line tree planning into the action-value function or policy.

TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning

deep reinforcement learning
treeqn atreec
differentiable


Multi-label classification (MLC) is the task of assigning a set of target labels for a given sample.

Modeling the combinatorial label interactions in MLC has been a long-haul challenge.

Recurrent neural network (RNN) based encoder-decoder models have shown state-of-the-art performance for solving MLC.

However, the sequential nature of modeling label dependencies through an RNN limits its ability in parallel computation, predicting dense labels, and providing interpretable results.

In this paper, we propose Message Passing Encoder-Decoder (MPED) Networks,  aiming to provide fast, accurate, and interpretable MLC.

MPED networks model the joint prediction of labels by replacing all RNNs in the encoder-decoder architecture with message passing mechanisms and dispense with autoregressive inference entirely.

The proposed models are simple, fast, accurate, interpretable, and structure-agnostic (can be used on known or unknown structured data).

Experiments on seven real-world MLC datasets show the proposed models outperform autoregressive RNN models across five different metrics with a significant speedup during training and testing time.

We propose Message Passing Encoder-Decode networks for a fast and accurate way of modelling label dependencies for multi-label classification.

Neural Message Passing for Multi-Label Classification


Recent few-shot learning algorithms have enabled models to quickly adapt to new tasks based on only a few training samples.

Previous few-shot learning works have mainly focused on classification and reinforcement learning.

In this paper, we propose a few-shot meta-learning system that focuses exclusively on regression tasks.

Our model is based on the idea that the degree of freedom of the unknown function can be significantly reduced if it is represented as a linear combination of a set of sparsifying basis functions.

This enables a few labeled samples to approximate the function.

We design a Basis Function Learner network to encode basis functions for a task distribution, and a Weights Generator network to generate the weight vector for a novel task.

We show that our model outperforms the current state of the art meta-learning methods in various regression tasks.

We propose a method of doing few-shot regression by learning a set of basis functions to represent the function distribution.

Few-Shot Regression via Learning Sparsifying Basis Functions

basis function
few-shot regression
learn


Large-scale pre-trained language model, such as BERT, has recently achieved great success in a wide range of language understanding tasks.

However, it remains an open question how to utilize BERT for text generation tasks.

In this paper, we present a novel approach to addressing this challenge in a generic sequence-to-sequence (Seq2Seq) setting.

We first propose a new task, Conditional Masked Language Modeling (C-MLM), to enable fine-tuning of BERT on target text-generation dataset.

The fine-tuned BERT (i.e., teacher) is then exploited as extra supervision to improve conventional Seq2Seq models (i.e., student) for text generation.

By leveraging BERT's idiosyncratic bidirectional nature, distilling the knowledge learned from BERT can encourage auto-regressive Seq2Seq models to plan ahead, imposing global sequence-level supervision for coherent text generation.

Experiments show that the proposed approach significantly outperforms strong baselines of Transformer on multiple text generation tasks, including machine translation (MT) and text summarization.

Our proposed model also achieves new state-of-the-art results on the IWSLT German-English and English-Vietnamese MT datasets.

We propose a model-agnostic way to leverage BERT for text generation and achieve improvements over Transformer on 2 tasks over 4 datasets.

Distilling the Knowledge of BERT for Text Generation


Humans have the remarkable ability to correctly classify images despite possible degradation.

Many studies have suggested that this hallmark of human vision results from the interaction between feedforward signals from bottom-up pathways of the visual cortex and feedback signals provided by top-down pathways.

Motivated by such interaction, we propose a new neuro-inspired model, namely Convolutional Neural Networks with Feedback (CNN-F).

CNN-F extends CNN with a feedback generative network, combining bottom-up and top-down inference to perform approximate loopy belief propagation.

We show that CNN-F's iterative inference allows for disentanglement of latent variables across layers.

We validate the advantages of CNN-F over the baseline CNN.

Our experimental results suggest that the CNN-F is more robust to image degradation such as pixel noise, occlusion, and blur.

Furthermore, we show that the CNN-F is capable of restoring original images from the degraded ones with high reconstruction accuracy while introducing negligible artifacts.

CNN-F extends CNN with a feedback generative network for robust vision.

Brain-inspired Robust Vision using Convolutional Neural Networks with Feedback


We develop new approximation and statistical learning theories of convolutional neural networks (CNNs) via the ResNet-type structure where the channel size, filter size, and width are fixed.

It is shown that a ResNet-type CNN is a universal approximator and its expression ability is no worse than fully-connected neural networks (FNNs) with a \textit{block-sparse} structure even if the size of each layer in the CNN is fixed.

Our result is general in the sense that we can automatically translate any approximation rate achieved by block-sparse FNNs into that by CNNs.

Thanks to the general theory, it is shown that learning on CNNs satisfies optimality in approximation and estimation of several important function classes.

As applications, we consider two types of function classes to be estimated: the Barron class and H\"older class.

We prove the clipped empirical risk minimization (ERM) estimator can achieve the same rate as FNNs even the channel size, filter size, and width of CNNs are constant with respect to the sample size.

This is minimax optimal (up to logarithmic factors) for the H\"older class.

Our proof is based on sophisticated evaluations of the covering number of CNNs and the non-trivial parameter rescaling technique to control the Lipschitz constant of CNNs to be constructed.

It is shown that ResNet-type CNNs are a universal approximator and its expression ability is not worse than fully connected neural networks (FNNs) with a \textit{block-sparse} structure even if the size of each layer in the CNN is fixed.

Approximation and non-parametric estimation of ResNet-type convolutional neural networks via block-sparse fully-connected neural networks


Few shot image classification aims at learning a classifier from limited labeled data.

Generating the classification weights has been applied in many meta-learning approaches for few shot image classification due to its simplicity and effectiveness.

However, we argue that it is difficult to generate the exact and universal classification weights for all the diverse query samples from very few training samples.

In this work, we introduce Attentive Weights Generation for few shot learning via Information Maximization (AWGIM), which addresses current issues by two novel contributions.

i) AWGIM generates different classification weights for different query samples by letting each of query samples attends to the whole support set.

ii) To guarantee the generated weights adaptive to different query sample, we re-formulate the problem to maximize the lower bound of mutual information between generated weights and query as well as support data.

As far as we can see, this is the first attempt to unify information maximization into few shot learning.

Both two contributions are proved to be effective in the extensive experiments and we show that AWGIM is able to achieve state-of-the-art performance on benchmark datasets.

A novel few shot learning method to generate query-specific classification weights via information maximization.

Attentive Weights Generation for Few Shot Learning via Information Maximization

via information maximization
shot
weight


Conversational question answering (CQA) is a novel QA task that requires the understanding of dialogue context.

Different from traditional single-turn machine reading comprehension (MRC), CQA is a comprehensive task comprised of passage reading, coreference resolution, and contextual understanding.

In this paper, we propose an innovative contextualized attention-based deep neural network, SDNet, to fuse context into traditional MRC models.

Our model leverages both inter-attention and self-attention to comprehend the conversation and passage.

Furthermore, we demonstrate a novel method to integrate the BERT contextual model as a sub-module in our network.

Empirical results show the effectiveness of SDNet.

On the CoQA leaderboard, it outperforms the previous best model's F1 score by 1.6%.

Our ensemble model further improves the F1 score by 2.7%.

A neural method for conversational question answering with attention mechanism and a novel usage of BERT as contextual embedder

SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering


Generative adversarial networks (GANs) are one of the most popular approaches when it comes to training generative models, among which variants of Wasserstein GANs are considered superior to the standard GAN formulation in terms of learning stability and sample quality.

However, Wasserstein GANs require the critic to be 1-Lipschitz, which is often enforced implicitly by penalizing the norm of its gradient, or by globally restricting its Lipschitz constant via weight normalization techniques.

Training with a regularization term penalizing the violation of the Lipschitz constraint explicitly, instead of through the norm of the gradient, was found to be practically infeasible in most situations.

Inspired by Virtual Adversarial Training, we propose a method called Adversarial Lipschitz Regularization, and show that using an explicit Lipschitz penalty is indeed viable and leads to competitive performance when applied to Wasserstein GANs, highlighting an important connection between Lipschitz regularization and adversarial training.

Adversarial Lipschitz Regularization


Multi-task learning promises to use less data, parameters, and time than training separate single-task models.

But realizing these benefits in practice is challenging.

In particular, it is difficult to define a suitable architecture that has enough capacity to support many tasks while not requiring excessive compute for each individual task.

There are difficult trade-offs when deciding how to allocate parameters and layers across a large set of tasks.

To address this, we propose a method for automatically searching over multi-task architectures that accounts for resource constraints.

We define a parameterization of feature sharing strategies for effective coverage and sampling of architectures.

We also present a method for quick evaluation of such architectures with feature distillation.

Together these contributions allow us to quickly optimize for parameter-efficient multi-task models.

We benchmark on Visual Decathlon, demonstrating that we can automatically search for and identify architectures that effectively make trade-offs between task resource requirements while maintaining a high level of final performance.

automatic search for multi-task architectures that reduce per-task feature use

Feature Partitioning for Efficient Multi-Task Architectures


As distributed approaches to natural language semantics have developed and diversified, embedders for linguistic units larger than words (e.g., sentences) have come to play an increasingly important role.

To date, such embedders have been evaluated using benchmark tasks (e.g., GLUE) and linguistic probes.

We propose a comparative approach, nearest neighbor overlap (N2O), that quantifies similarity between embedders in a task-agnostic manner.

N2O requires only a collection of examples and is simple to understand: two embedders are more similar if, for the same set of inputs, there is greater overlap between the inputs' nearest neighbors.

We use N2O to compare 21 sentence embedders and show the effects of different design choices and architectures.

We propose nearest neighbor overlap, a procedure which quantifies similarity between embedders in a task-agnostic manner, and use it to compare 21 sentence embedders.

Situating Sentence Embedders with Nearest Neighbor Overlap


Generative Adversarial Networks (GANs) can achieve state-of-the-art sample quality in generative modelling tasks but suffer from the mode collapse problem.

Variational Autoencoders (VAE) on the other hand explicitly maximize a reconstruction-based data log-likelihood forcing it to cover all modes, but suffer from poorer sample quality.

Recent works have proposed hybrid VAE-GAN frameworks which integrate a GAN-based synthetic likelihood to the VAE objective to address both the mode collapse and sample quality issues, with limited success.

This is because the VAE objective forces a trade-off between the data log-likelihood and divergence to the latent prior.

The synthetic likelihood ratio term also shows instability during training.

We propose a novel objective with a ``"Best-of-Many-Samples" reconstruction cost and a stable direct estimate of the synthetic likelihood.

This enables our hybrid VAE-GAN framework to achieve high data log-likelihood and low divergence to the latent prior at the same time and shows significant improvement over both hybrid VAE-GANS and plain GANs in mode coverage and quality.

We propose a new objective for training hybrid VAE-GANs which lead to significant improvement in mode coverage and quality.

``"Best-of-Many-Samples" Distribution Matching