We study how, in generative adversarial networks, variance in the discriminator's output affects the generator's ability to learn the data distribution.

In particular, we contrast the results from various well-known techniques for training GANs when the discriminator is near-optimal and updated multiple times per update to the generator.

As an alternative, we propose an additional method to train GANs by explicitly modeling the discriminator's output as a bi-modal Gaussian distribution over the real/fake indicator variables.

In order to do this, we train the Gaussian classifier to match the target bi-modal distribution implicitly through meta-adversarial training.

We observe that our new method, when trained together with a strong discriminator, provides meaningful, non-vanishing gradients.

We introduce meta-adversarial learning, a new technique to regularize GANs, and propose a training method by explicitly controlling the discriminator's output distribution.

The paper proposes variance regularizing adversarial learning for training GANs to ensure that the gradient for the generator does not vanish

Variance Regularizing Adversarial Learning

variance regularize adversarial learning


We introduce NoisyNet, a deep reinforcement learning agent with parametric noise added to its weights, and show that the induced stochasticity of the agent’s policy can be used to aid efficient exploration.

The parameters of the noise are learned with gradient descent along with the remaining network weights.

NoisyNet is straightforward to implement and adds little computational overhead.

We find that replacing the conventional exploration heuristics for A3C, DQN and Dueling agents (entropy reward and epsilon-greedy respectively) with NoisyNet yields substantially higher scores for a wide range of Atari games, in some cases advancing the agent from sub to super-human performance.

A deep reinforcement learning agent with parametric noise added to its weights can be used to aid efficient exploration.

This paper introduces NoisyNets, neural networks whose parameters are perturbed by a parametric noise function, that obtain substantial performance improvement over baseline deep reinforcement learning algorithms.

New exploration method for deep RL by injecting noise into deep networks' weights, with the noise taking various forms

Noisy Networks For Exploration

weight
parametric noise
deep
exploration
network
reinforcement


Localization is the problem of estimating the location of an autonomous agent from an observation and a map of the environment.

Traditional methods of localization, which filter the belief based on the observations, are sub-optimal in the number of steps required, as they do not decide the actions taken by the agent.

We propose "Active Neural Localizer", a fully differentiable neural network that learns to localize efficiently.

The proposed model incorporates ideas of traditional filtering-based localization methods, by using a structured belief of the state with multiplicative interactions to propagate belief, and combines it with a policy model to minimize the number of steps required for localization.

Active Neural Localizer is trained end-to-end with reinforcement learning.

We use a variety of simulation environments for our experiments which include random 2D mazes, random mazes in the Doom game engine and a photo-realistic environment in the Unreal game engine.

The results on the 2D environments show the effectiveness of the learned policy in an idealistic setting while results on the 3D environments demonstrate the model's capability of learning the policy and perceptual model jointly from raw-pixel based RGB observations.

We also show that a model trained on random textures in the Doom environment generalizes well to a photo-realistic office space environment in the Unreal engine.

"Active Neural Localizer", a fully differentiable neural network that learns to localize efficiently using deep reinforcement learning.

This paper formulates the problem of localisation on a known map using a belief network as an RL problem where the agent's goal is to minimise the number of steps to localise itself.

This is a clear and interesting paper that builds a parameterized network to select actions for a robot in a simulated environment

Active Neural Localization


Machine translation is an important real-world application, and neural network-based AutoRegressive Translation (ART) models have achieved very promising accuracy.

Due to the unparallelizable nature of the autoregressive factorization, ART models have to generate tokens one by one during decoding and thus suffer from high inference latency.

Recently, Non-AutoRegressive Translation (NART) models were proposed to reduce the inference time.

However, they could only achieve inferior accuracy compared with ART models.

To improve the accuracy of NART models, in this paper, we propose to leverage the hints from a well-trained ART model to train the NART model.

We define two hints for the machine translation task: hints from hidden states and hints from word alignments, and use such hints to regularize the optimization of NART models.

Experimental results show that the NART model trained with hints could achieve significantly better translation performance than previous NART models on several tasks.

In particular, for the WMT14 En-De and De-En task, we obtain BLEU scores of 25.20 and 29.52 respectively, which largely outperforms the previous non-autoregressive baselines.

It is even comparable to a strong LSTM-based ART model (24.60 on WMT14 En-De), but one order of magnitude faster in inference.

We develop a training algorithm for non-autoregressive machine translation models, achieving comparable accuracy to strong autoregressive baselines, but one order of magnitude faster in inference.

Distills knowledge from intermediary hidden states and attention weights to improve non-autoregressive neural machine translation.

Proposes to leverage well trained autoregressive model to inform the hidden states and the word alignment of non-autoregressive Neural Machine Translation models.

Hint-based Training for Non-Autoregressive Translation


Artificial neural networks are built on the basic operation of linear combination and non-linear activation function.

Theoretically this structure can approximate any continuous function with three layer architecture.

But in practice learning  the parameters of such network can be hard.

Also the choice of activation function can greatly impact the performance of the network.

In this paper we are proposing to replace the basic linear combination operation with non-linear operations that do away with the need of additional non-linear activation function.

To this end we are proposing the use of elementary  morphological operations (dilation and erosion) as the basic operation in neurons.

We show that these networks (Denoted as Morph-Net) with morphological operations can approximate any smooth function requiring less number of parameters than what is necessary for normal neural networks.

The results show that our network perform favorably when compared with similar structured network.

We have carried out our experiments on  MNIST, Fashion-MNIST, CIFAR10 and CIFAR100.

Using mophological operation (dilation and erosion) we have defined a class of network which can approximate any continious function.

This paper proposes to replace standard RELU/tanh units with a combination of dilation and erosion operations, observing that the new operator creates more hyper-planes and has more expressive power.

The authors introduce Morph-Net, a single layer neural network where the mapping is performed using morphological dilation and erosion.

Dense Morphological Network: An Universal Function Approximator

dilation erosion
network
operation
function
morphological


With the rapidly scaling up of deep neural networks (DNNs), extensive research studies on network model compression such as weight pruning have been performed for efficient deployment.

This work aims to advance the compression beyond the weights to the activations of DNNs.

We propose the Integral Pruning (IP) technique which integrates the activation pruning with the weight pruning.

Through the learning on the different importance of neuron responses and connections, the generated network, namely IPnet, balances the sparsity between activations and weights and therefore further improves execution efficiency.

The feasibility and effectiveness of IPnet are thoroughly evaluated through various network models with different activation functions and on different datasets.

With <0.5% disturbance on the testing accuracy, IPnet saves 71.1% ~ 96.35% of computation cost, compared to the original dense models with up to 5.8x and 10x reductions in activation and weight numbers, respectively.

This work advances DNN compression beyond the weights to the activations by integrating the activation pruning with the weight pruning.

An integral model compression method that handles both weight and activation pruning, leading to more efficient network computation and effective reduction of the number of multiply-and-accumulate.

This article presents a novel approach to reduce the computation cost of deep neural networks by integrating activation pruning along with weight pruning and show that common techniques of exclusive weight pruning  increases the number of non-zero activations after ReLU.

Integral Pruning on Activations and Weights for Efficient Neural Networks

integrate activation prune
weight pruning
number
computation
efficient
neural network
compression
integral


The Variational Auto Encoder (VAE) is a popular generative
latent variable model that is often
applied for representation learning.

Standard VAEs assume continuous valued
latent variables and are trained by maximization
of the evidence lower bound (ELBO).

Conventional methods obtain a
differentiable estimate of the ELBO with reparametrized sampling and
optimize it with Stochastic Gradient Descend (SGD).

However, this is not possible if
we want to train VAEs with discrete valued latent variables,
since reparametrized sampling is not possible.

Till now, there
exist no simple solutions to circumvent this problem.

In this paper, we propose an easy method to train VAEs
with binary or categorically valued latent representations.

Therefore, we use a differentiable
estimator for the ELBO which is based on importance sampling.

In experiments, we verify the approach and
train two different VAEs architectures with Bernoulli and
Categorically distributed latent representations on two different benchmark
datasets.

We propose an easy method to train Variational Auto Encoders (VAE) with discrete latent representations, using importance sampling

Introducting an importance sampling distribution and using samples from distribution to compute importance-weighted estimate of the gradient

This paper proposes to use important sampling to optimize VAE with discrete latent variables.

Training Variational Auto Encoders with Discrete Latent Representations using Importance Sampling

discrete latent representation importance sampling


Distributed computing can significantly reduce the training time of neural networks.

Despite its potential, however, distributed training has not been widely adopted: scaling the training process is difficult, and existing SGD methods require substantial tuning of hyperparameters and learning schedules to achieve sufficient accuracy when increasing the number of workers.

In practice, such tuning can be prohibitively expensive given the huge number of potential hyperparameter configurations and the effort required to test each one.

We propose DANA, a novel approach that scales out-of-the-box to large clusters using the same hyperparameters and learning schedule optimized for training on a single worker, while maintaining similar final accuracy without additional overhead.

DANA estimates the future value of model parameters by adapting Nesterov Accelerated Gradient to a distributed setting, and so mitigates the effect of gradient staleness, one of the main difficulties in scaling SGD to more workers.

Evaluation on three state-of-the-art network architectures and three datasets shows that DANA scales as well as or better than existing work without having to tune any hyperparameters or tweak the learning schedule.

For example, DANA achieves 75.73% accuracy on ImageNet when training ResNet-50 with 16 workers, similar to the non-distributed baseline.

A new distributed asynchronous SGD algorithm that achieves state-of-the-art accuracy on existing architectures without any additional tuning or overhead.

Proposes an improvement to existing ASGD approaches at mid-size scaling using momentum with SGD for asynchronous training across a distributed worker pool.

This paper addresses the gradient staleness vs parallel performance problem in distributed deep learning training, and proposes an approach to estimate future model parameters at the slaves to reduce communication latency effects.

DANA: Scalable Out-of-the-box Distributed ASGD Without Retuning

exist approach distribute asynchronous sgd without asgd training


This paper proposes a novel approach to train deep neural networks by unlocking the layer-wise dependency of backpropagation training.

The approach employs additional modules called local critic networks besides the main network model to be trained, which are used to obtain error gradients without complete feedforward and backward propagation processes.

We propose a cascaded learning strategy for these local networks.

In addition, the approach is also useful from multi-model perspectives, including structural optimization of neural networks, computationally efficient progressive inference, and ensemble classification for performance improvement.

Experimental results show the effectiveness of the proposed approach and suggest guidelines for determining appropriate algorithm parameters.

We propose a new learning algorithm of deep neural networks, which unlocks the layer-wise dependency of backpropagation.

An alternative training paradigm for DNIs in which the auxiliary module is trained to approximate directly the final output of the original model, offering side benefits.

Describes a method of training neural networks without update locking.

Local Critic Training of Deep Neural Networks


\emph{Truncated Backpropagation Through Time} (truncated BPTT, \cite{jaeger2002tutorial}) is a widespread method for learning recurrent computational graphs.

Truncated BPTT keeps the computational benefits of \emph{Backpropagation Through Time} (BPTT \cite{werbos:bptt}) while relieving the need for a complete backtrack through the whole data sequence at every step.

However, truncation favors short-term dependencies: the gradient estimate of truncated BPTT is biased, so that it does not benefit from the convergence guarantees from stochastic gradient theory.

We introduce \emph{Anticipated Reweighted Truncated Backpropagation} (ARTBP), an algorithm that keeps the computational benefits of truncated BPTT, while providing unbiasedness.

ARTBP works by using variable truncation lengths together with carefully chosen compensation factors in the backpropagation equation.

We check the viability of ARTBP on two tasks.

First, a simple synthetic task where careful balancing of temporal dependencies at different scales is needed: truncated BPTT displays unreliable performance, and in worst case scenarios, divergence, while ARTBP converges reliably.

Second, on Penn Treebank character-level language modelling \cite{ptb_proc}, ARTBP slightly outperforms truncated BPTT.

Provides an unbiased version of truncated backpropagation by sampling truncation lengths and reweighting accordingly.

Proposes stochastic determination methods for truncation points in backpropagation through time.

A new approximation to backpropagation through time to overcome the computational and memory loads that arise when having to learn from long sequences.

Unbiasing Truncated Backpropagation Through Time

backpropagation time
truncated
truncation


Graph convolutional networks (GCNs) have been widely used for classifying graph nodes in the semi-supervised setting.

Previous works have shown that GCNs are vulnerable to the perturbation on adjacency and feature matrices of existing nodes.

However, it is unrealistic to change the connections of  existing nodes in many applications, such as existing users in social networks.

In this paper, we investigate methods attacking GCNs by adding fake nodes.

A greedy algorithm is proposed to generate adjacency and feature matrices of fake nodes, aiming to minimize the classification accuracy on the existing ones.

In additional, we introduce a discriminator to classify fake nodes from real nodes, and propose a Greedy-GAN algorithm to simultaneously update the discriminator and the attacker, to make fake nodes indistinguishable to the real ones.

Our non-targeted attack decreases the accuracy of GCN down to 0.10, and our targeted attack reaches a success rate of 0.99 for attacking the whole datasets, and 0.94 on average for attacking a single node.

non-targeted and targeted attack on GCN by adding fake nodes

The authors propose a new adversarial technique to add "fake" nodes to fool a GCN-based classifier

ATTACK GRAPH CONVOLUTIONAL NETWORKS BY ADDING FAKE NODES


Transfer learning aims to solve the data sparsity for a specific domain by applying information of another domain.

Given a sequence (e.g. a natural language sentence), the transfer learning, usually enabled by recurrent neural network (RNN), represent the sequential information transfer.

RNN uses a chain of repeating cells to model the sequence data.

However, previous studies of neural network based transfer learning simply transfer the information across the whole layers, which are unfeasible for seq2seq and sequence labeling.

Meanwhile, such layer-wise transfer learning mechanisms also lose the fine-grained cell-level information from the source domain.

In this paper, we proposed the aligned recurrent transfer, ART, to achieve cell-level information transfer.

ART is in a recurrent manner that different cells share the same parameters.

Besides transferring the corresponding information at the same position, ART transfers information from all collocated words in the source domain.

This strategy enables ART to capture the word collocation across domains in a more flexible way.

We conducted extensive experiments on both sequence labeling tasks (POS tagging, NER) and sentence classification (sentiment analysis).

ART outperforms the state-of-the-arts over all experiments.

Transfer learning for sequence via learning to align cell-level information across domains.

The paper proposed to use RNN/LSTM with collocation alignment as a representation learning method for transfer learning/domain adaptation in NLP.

Transfer Learning for Sequences via Learning to Collocate

transfer learn sequence via learn
domain


Addressing uncertainty is critical for autonomous systems to robustly adapt to the real world.

We formulate the problem of model uncertainty as a continuous Bayes-Adaptive Markov Decision Process (BAMDP), where an agent maintains a posterior distribution over latent model parameters given a history of observations and maximizes its expected long-term reward with respect to this belief distribution.

Our algorithm, Bayesian Policy Optimization, builds on recent policy optimization algorithms to learn a universal policy that navigates the exploration-exploitation trade-off to maximize the Bayesian value function.

To address challenges from discretizing the continuous latent parameter space, we propose a new policy network architecture that encodes the belief distribution independently from the observable state.

Our method significantly outperforms algorithms that address model uncertainty without explicitly reasoning about belief distributions and is competitive with state-of-the-art Partially Observable Markov Decision Process solvers.

We formulate model uncertainty in Reinforcement Learning as a continuous Bayes-Adaptive Markov Decision Process and present a method for practical and scalable Bayesian policy optimization.

Using a Bayesian approach, there is a better trade-off between exploration and exploitation in RL

Bayesian Policy Optimization for Model Uncertainty

bayesian policy optimization
model uncertainty


For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic.

We clarify a necessary condition for an evaluation metric not to behave this way: estimating the function must require a large sample from the model.

In search of such a metric, we turn to neural network divergences (NNDs), which are defined in terms of a neural network trained to distinguish between distributions.

The resulting benchmarks cannot be ``won'' by training set memorization, while still being perceptually correlated and computable only from samples.

We survey past work on using NNDs for evaluation, implement an example black-box metric based on these ideas, and validate experimentally that it can measure a notion of generalization.

We argue that GAN benchmarks must require a large sample from the model to penalize memorization and investigate whether neural network divergences have this property.

Authors propose criterion for evaluating the quality of samples produced by a Generative Adversarial Network.

Towards GAN Benchmarks Which Require Generalization


Conventional methods model open domain dialogue generation as a black box through end-to-end learning from large scale conversation data.

In this work, we make the first step to open the black box by introducing dialogue acts into open domain dialogue generation.

The dialogue acts are generally designed and reveal how people engage in social chat.

Inspired by analysis on real data, we propose jointly modeling dialogue act selection and response generation, and perform learning with human-human conversations tagged with a dialogue act classifier and a reinforcement approach to further optimizing the model for long-term conversation.

With the dialogue acts, we not only achieve significant improvement over state-of-the-art methods on response quality for given contexts and long-term conversation in both machine-machine simulation and human-machine conversation, but also are capable of explaining why such achievements can be made.

open domain dialogue generation with dialogue acts

The authors use a distant supervision technique to add dialogue act tags as a conditioning factor for generating responses in open-domain dialogues

The paper describes a technique to incorporate dialog acts into neural conversational agents

Towards Interpretable Chit-chat: Open Domain Dialogue Generation with Dialogue Acts

open domain dialogue generation dialogue act


We discuss the feasibility of the following learning problem: given unmatched samples from two domains and nothing else, learn a mapping between the two, which preserves semantics.

Due to the lack of paired samples and without any definition of the semantic information, the problem might seem ill-posed.

Specifically, in typical cases, it seems possible to build infinitely many alternative mappings  from every target mapping.

This apparent ambiguity stands in sharp contrast to the recent empirical success in solving this problem.

We identify the abstract notion of aligning two domains in a semantic way with concrete terms of minimal relative complexity.

A theoretical framework for measuring the complexity of compositions of functions is developed in order to show that it is reasonable to expect the minimal complexity mapping to be unique.

The measured complexity used is directly related to the depth of the neural networks being learned and a semantically aligned mapping could then be captured simply by learning using architectures that are not much bigger than the minimal architecture.

Various predictions are made based on the hypothesis that semantic alignment can be captured by the minimal mapping.

In addition, a new mapping algorithm is proposed and shown to lead to better mapping results.

Our hypothesis is that given two domains, the lowest complexity mapping that has a low discrepancy approximates the target mapping.

The paper addresses the problem of learning mappings between different domains without any supervision, stating three conjectures.

Demonstrates that in unsupervised learning on unaligned data it is possible to learn the between domains mapping using GAN only without a reconstruction loss.

The Role of Minimal Complexity Functions in Unsupervised Learning of Semantic Mappings

unsupervised learning
domain
mapping
learn
without
complexity


We present a novel approach for the certification of neural networks against adversarial perturbations which combines scalable overapproximation methods with precise (mixed integer) linear programming.

This results in significantly better precision than state-of-the-art verifiers on challenging feedforward and convolutional neural networks with piecewise linear activation functions.

We refine the over-approximation results from incomplete verifiers using MILP solvers to prove more robustness properties than state-of-the-art.

Introduces a verifier that obtains improvement on precision of incomplete verifiers and scalability of the complete verifiers using over-parameterization, mixed integer linear programming and linear programming relaxation.

A mixed strategy to obtain better precision on robustness verifications of feed-forward neural networks with piecewise linear activation functions, achieving better precision than incomplete verifiers and more scalability than complete verifiers.

Boosting Robustness Certification of Neural Networks

precision incomplete verifier scalability complete verifier


A distinct commonality between HMMs and RNNs is that they both learn hidden representations for sequential data.

In addition, it has been noted that the backward computation of the Baum-Welch algorithm for HMMs is a special case of the back-propagation algorithm used for neural networks (Eisner (2016)).

Do these observations suggest that, despite their many apparent differences, HMMs are a special case of RNNs?

In this paper, we show that that is indeed the case, and investigate a series of architectural transformations between HMMs and RNNs, both through theoretical derivations and empirical hybridization.

In particular, we investigate three key design factors—independence assumptions between the hidden states and the observation, the placement of softmaxes, and the use of non-linearities—in order to pin down their empirical effects.

We present a comprehensive empirical study to provide insights into the interplay between expressivity and interpretability in this model family with respect to language modeling and parts-of-speech induction.

Are HMMs a special case of RNNs? We investigate a series of architectural transformations between HMMs and RNNs, both through theoretical derivations and empirical hybridization and provide new insights.

This paper explores if HMMs are a special case of RNNs using language modeling and POS tagging

Bridging HMMs and RNNs through Architectural Transformations

hmms special case rnn
architectural transformation


Deep neural networks have been tremendously successful in a number of tasks.

One of the main reasons for this is their capability to automatically

learn representations of data in levels of abstraction,

increasingly disentangling the data as the internal transformations are applied.

In this paper we propose a novel regularization method that penalize covariance between dimensions of the hidden layers in a network, something that benefits the disentanglement.

This makes the network learn nonlinear representations that are linearly uncorrelated, yet allows the model to obtain good results on a number of tasks, as demonstrated by our experimental evaluation.

The proposed technique can be used to find the dimensionality of the underlying data, because it effectively disables dimensions that aren't needed.

Our approach is simple and computationally cheap, as it can be applied as a regularizer to any gradient-based learning model.

We propose a novel regularization method that penalize covariance between dimensions of the hidden layers in a network.

This paper presents a regularization mechanism which penalizes covariance between all dimensions in the latent representation of a neural network in order to disentangle the latent representation

Disentangled activations in deep networks

penalize covariance dimension
network
regularization
disentangle


This report introduces a training and recognition scheme, in which classification is realized via class-wise discerning.

Trained with datasets whose labels are randomly shuffled except for one class of interest, a neural network learns class-wise parameter values, and remolds itself from a feature sorter into feature filters, each of which discerns objects belonging to one of the classes only.

Classification of an input can be inferred from the maximum response of the filters.

A multiple check with multiple versions of filters can diminish fluctuation and yields better performance.

This scheme of discerning, maximum response and multiple check is a method of general viability to improve performance of feedforward networks, and the filter training itself is a promising feature abstraction procedure.

In contrast to the direct sorting, the scheme mimics the classification process mediated by a series of one component picking.

The proposed scheme mimics the classification process mediated by a series of one component picking.

A method to increase accuracy of deep-nets on multi-class classification tasks seemingly by a reduction of multi-class to binary classification.

A novel classification procedure of discerning, maxmum response, and multiple check to improve accuracy of mediocre networks and enhance feedforward networks.

Filter Training and Maximum Response: Classification via Discerning


A long-held conventional wisdom states that larger models train more slowly when using gradient descent.

This work challenges this widely-held belief, showing that larger models can potentially train faster despite the increasing computational requirements of each training step.

In particular, we study the effect of network structure (depth and width) on halting time and show that larger models---wider models in particular---take fewer training steps to converge.

We design simple experiments to quantitatively characterize the effect of overparametrization on weight space traversal.

Results show that halting time improves when growing model's width for three different applications, and the improvement comes from each factor: The distance from initialized weights to converged weights shrinks with a power-law-like relationship, the average step size grows with a power-law-like relationship, and gradient vectors become more aligned with each other during traversal.

Empirically shows that larger models train in fewer training steps, because all factors in weight space traversal improve.

This paper shows that wider RNNs improve convergence speed when applied to NLP problems, and by extension the effect of increasing the widths in deep neural networks on the convergence of optimization

This paper characterizes the impact of over-parametrization in the number of iterations it takes an algorithm to converge, and presents further empirical observations on the effects of over-parametrization in neural network training.

Empirically Characterizing Overparameterization Impact on Convergence

neural network
effect
show
training
convergence
impact
empirically
improve
characterize


Due to its potential to improve programmer productivity and software quality, automated program repair has been an active topic of research.

Newer techniques harness neural networks to learn directly from examples of buggy programs and their fixes.

In this work, we consider a recently identified class of bugs called variable-misuse bugs.

The state-of-the-art solution for variable misuse enumerates potential fixes for all possible bug locations in a program, before selecting the best prediction.

We show that it is beneficial to train a model that jointly and directly localizes and repairs variable-misuse bugs.

We present multi-headed pointer networks for this purpose, with one head each for localization and repair.

The experimental results show that the joint model significantly outperforms an enumerative solution that uses a pointer based model for repair alone.

Multi-headed Pointer Networks for jointly learning to localize and repair Variable Misuse bugs

Proposes a LSTM based model with pointers to break the problem of VarMisuse down into multiple steps.

This paper presents an LSTM-based model for bug detection and repair of the VarMisuse bug, and demonstrates significant improvements compared to prior approaches on several datasets.

Neural Program Repair by Jointly Learning to Localize and Repair

jointly learn localize repair
varmisuse
model
pointer
bug


Classification and clustering have been studied separately in machine learning and computer vision.

Inspired by the recent success of deep learning models in solving various vision problems (e.g., object recognition, semantic segmentation) and the fact that humans serve as the gold standard in assessing clustering algorithms, here, we advocate for a unified treatment of the two problems and suggest that hierarchical frameworks that progressively build complex patterns on top of the simpler ones (e.g., convolutional neural networks) offer a promising solution.

We do not dwell much on the learning mechanisms in these frameworks as they are still a matter of debate, with respect to biological constraints.

Instead, we emphasize on the compositionality of the real world structures and objects.

In particular, we show that CNNs, trained end to end using back propagation with noisy labels, are able to cluster data points belonging to several overlapping shapes, and do so much better than the state of the art algorithms.

The main takeaway lesson from our study is that mechanisms of human vision, particularly the hierarchal organization of the visual ventral stream should be taken into account in clustering algorithms (e.g., for learning representations in an unsupervised manner or with minimum supervision) to reach human level clustering performance.

This, by no means, suggests that other methods do not hold merits.

For example, methods relying on pairwise affinities (e.g., spectral clustering) have been very successful in many cases but still fail in some cases (e.g., overlapping clusters).

The paper validates the idea that deep convolutional neural networks could learn to cluster input data better than other clustering methods by noting their ability to interpret the context of every input point due to a large field of view.

This work combines deep learning for feature representation with the task of human-like unsupervised grouping.

Human-like Clustering with Deep Convolutional Neural Networks

deep convolutional neural network
human-like cluster


Instancewise feature scoring is a method for model interpretation, which yields, for each test instance, a vector of importance scores associated with features.

Methods based on the Shapley score have been proposed as a fair way of computing feature attributions, but incur an exponential complexity in the number of features.

This combinatorial explosion arises from the definition of Shapley value and prevents these methods from being scalable to large data sets and complex models.

We focus on settings in which the data have a graph structure, and the contribution of features to the target variable is well-approximated by a graph-structured factorization.

In such settings, we develop two algorithms with linear complexity for instancewise feature importance scoring on black-box models.

We establish the relationship of our methods to the Shapley value and a closely related concept known as the Myerson value from cooperative game theory.

We demonstrate on both language and image data that our algorithms compare favorably with other methods using both quantitative metrics and human evaluation.

We develop two linear-complexity algorithms for model-agnostic model interpretation based on the Shapley value, in the settings where the contribution of features to the target is well-approximated by a graph-structured factorization.

The paper proposes two approximations to the Shapley value used for generating feature scores for interpretability.

This paper proposes two methods for instance-wise feature importance scoring using Shapely values, and provides two efficient methods of computing approximate Shapely values when there is a known structure relating the features.

L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data

shapley value
two
feature
model interpretation
efficient


According to parallel distributed processing (PDP) theory in psychology, neural networks (NN) learn distributed rather than interpretable localist representations.

This view has been held so strongly that few researchers have analysed single units to determine if this assumption is correct.

However, recent results from psychology, neuroscience and computer science have shown the occasional existence of local codes emerging in artificial and biological neural networks.

In this paper, we undertake the first systematic survey of when local codes emerge in a feed-forward neural network, using generated input and output data with known qualities.

We find that the number of local codes that emerge from a NN follows a well-defined distribution across the number of hidden layer neurons, with a peak determined by the size of input data, number of examples presented and the sparsity of input data.

Using a 1-hot output code drastically decreases the number of local codes on the hidden layer.

The number of emergent local codes increases with the percentage of dropout applied to the hidden layer, suggesting that the localist encoding may offer a resilience to noisy networks.

This data suggests that localist coding can emerge from feed-forward PDP networks and suggests some of the conditions that may lead to interpretable localist representations in the cortex.

The findings highlight how local codes should not be dismissed out of hand.

Local codes have been found in feed-forward neural networks

A method for determining to what degree individual neurons in a hidden layer of an MLP encode a localist code, which is studied for different input representations.

Studies the development of localist representations in the hidden layers of feed-forward neural networks.

When and where do feed-forward neural networks learn localist representations?

hidden layer
feed-forward neural network
code
representation
localist
study


Representing entities and relations in an embedding space is a well-studied approach for machine learning on relational data.

Existing approaches however primarily focus on simple link structure between a finite set of entities, ignoring the variety of data types that are often used in relational databases, such as text, images, and numerical values.

In our approach, we propose a multimodal embedding using different neural encoders for this variety of data, and combine with existing models to learn embeddings of the entities.

We extend existing datasets to create two novel benchmarks, YAGO-10-plus and MovieLens-100k-plus, that contain additional relations such as textual descriptions and images of the original entities.

We demonstrate that our model utilizes the additional information effectively to provide further gains in accuracy.

Moreover, we test our learned multimodal embeddings by using them to predict missing multimodal attributes.

Extending relational modeling to support multimodal data using neural encoders.

This paper proposes to perform link prediction in Knowledge Bases by supplementing the original entities with multimodal information, and presents a model able to encode all sorts of information when scoring triples.

The paper is about incorporating information from different modalities into link prediction approaches

Embedding Multimodal Relational Data

link prediction
information
relational
multimodal
datum


An ensemble of neural networks is known to be more robust and accurate than an individual network, however usually with linearly-increased cost in both training and testing.

In this work, we propose a two-stage method to learn Sparse Structured Ensembles (SSEs) for neural networks.

In the first stage, we run SG-MCMC with group sparse priors to draw an ensemble of samples from the posterior distribution of network parameters.

In the second stage, we apply weight-pruning to each sampled network and then perform retraining over the remained connections.

In this way of learning SSEs with SG-MCMC and pruning, we not only achieve high prediction accuracy since SG-MCMC enhances exploration of the model-parameter space, but also reduce memory and computation cost significantly in both training and testing of NN ensembles.

This is thoroughly evaluated in the experiments of learning SSE ensembles of both FNNs and LSTMs.

For example, in LSTM based language modeling (LM), we obtain 21\% relative reduction in LM perplexity by learning a SSE of 4 large LSTM models, which has only 30\% of model parameters and 70\% of computations in total, as compared to the baseline large LSTM LM.

To the best of our knowledge, this work represents the first methodology and empirical study of integrating SG-MCMC, group sparse prior and network pruning together for learning NN ensembles.

Propose a novel method by integrating SG-MCMC sampling, group sparse prior and network pruning to learn Sparse Structured Ensemble (SSE) with improved performance and significantly reduced cost than traditional methods.

The authors propose a procedure to generate an ensemble of sparse structured models

A new framework for training ensemble neural networks that uses SG-MCMC methods within deep learning, and then increases computational efficiency by group sparsity+pruning.

This paper explores the use of FNN and LSTMs to make bayesian model averaging more computationally feasible and improve average model performance.

Learning Sparse Structured Ensembles with SG-MCMC and Network Pruning

learn sparse structure ensemble
sg-mcmc
performance
network
group
improve
method
model


This paper introduces a new framework for data efficient and versatile learning.

1) We develop ML-PIP, a general framework for Meta-Learning approximate Probabilistic Inference for Prediction.

ML-PIP extends existing probabilistic interpretations of meta-learning to cover a broad class of methods.

2) We introduce \Versa{}, an instance of the framework employing a flexible and versatile amortization network that takes few-shot learning datasets as inputs, with arbitrary numbers of shots, and outputs a distribution over task-specific parameters in a single forward pass.

\Versa{} substitutes optimization at test time with forward passes through inference networks, amortizing the cost of inference and relieving the need for second derivatives during training.

3) We evaluate \Versa{} on benchmark datasets where the method sets new state-of-the-art results, and can handle arbitrary number of shots, and for classification, arbitrary numbers of classes at train and test time.

The power of the approach is then demonstrated through a challenging few-shot ShapeNet view reconstruction task.

Novel framework for meta-learning that unifies and extends a broad class of existing few-shot learning methods. Achieves strong performance on few-shot learning benchmarks without requiring iterative test-time inference.

This work tackles few-shot learning from a probabilistic inference viewpoint, achieving state-of-the-art despite simpler setup than many competitors

Meta-Learning Probabilistic Inference for Prediction

few-shot learning
probabilistic inference
achieve
meta-learne


In recent years, softmax together with its fast approximations has become the de-facto loss function for deep neural networks with multiclass predictions.

However, softmax is used in many problems that do not fully fit the multiclass framework and where the softmax assumption of mutually exclusive outcomes can lead to biased results.

This is often the case for applications such as language modeling, next event prediction and matrix factorization, where many of the potential outcomes are not mutually exclusive, but are more likely to be independent conditionally on the state.

To this end, for the set of problems with positive and unlabeled data, we propose a relaxation of the original softmax formulation, where, given the observed state, each of the outcomes are conditionally independent but share a common set of negatives.

Since we operate in a regime where explicit negatives are missing, we create an adversarially-trained model of negatives and derive a new negative sampling and weighting scheme which we denote as Cooperative Importance Sampling (CIS).

We show empirically the advantages of our newly introduced negative sampling scheme by pluging it in the Word2Vec algorithm and benching it extensively against other negative sampling schemes on both language modeling and matrix factorization tasks and show large lifts in performance.

Defining a partially mutual exclusive softmax loss for postive data and implementing a cooperative based sampling scheme

This paper presents Cooperative Importance Sampling towards resolving the problem of the mutually exclusive assumption of traditional softmax being biased when negative samples are not explicitly defined

This paper proposes PMES methods to relax the exclusive outcome assumption in softmax loss, demonstrating empirical merit in improving word2vec type of embedding models.

Partially Mutual Exclusive Softmax for Positive and Unlabeled data

partially mutual exclusive softmax
loss
assumption
define
sample
cooperative
datum


Over the past few years, various tasks involving videos such as classification, description, summarization and question answering have received a lot of attention.

Current models for these tasks compute an encoding of the video by treating it as a sequence of images and going over every image in the sequence, which becomes computationally expensive for longer videos.

In this paper, we focus on the task of video classification and aim to reduce the computational cost by using the idea of distillation.

Specifically, we propose a Teacher-Student network wherein the teacher looks at all the frames in the video but the student looks at only a small fraction of the frames in the video.

The idea is to then train the student to minimize

(i)  the difference between the final representation computed by the student and the teacher and/or

(ii) the difference between the distributions predicted by the teacher and the student.

This smaller student network which involves fewer computations but still learns to mimic the teacher can then be employed at inference time for video classification.

We experiment with the YouTube-8M dataset and show  that the proposed student network can reduce the inference time by upto 30% with a negligent drop in the performance.

Teacher-Student framework for efficient video classification using fewer frames

The paper proposes an idea to distill from a full video classification model a small model that only receives smaller number of frames.

The authors present a teacher-student network to solve video classification problem, proposing serial and parallel training algorithms aimed at reducing computational costs.

A Teacher Student Network For Faster Video Classification

frame
video classification
network
teacher-student


Deep generative models have achieved impressive success in recent years.

Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), as powerful frameworks for deep generative model learning, have largely been considered as two distinct paradigms and received extensive independent studies respectively.

This paper aims to establish formal connections between GANs and VAEs through a new formulation of them.

We interpret sample generation in GANs as performing posterior inference, and show that GANs and VAEs involve minimizing KL divergences of respective posterior and inference distributions with opposite directions, extending the two learning phases of classic wake-sleep algorithm, respectively.

The unified view provides a powerful tool to analyze a diverse set of existing model variants, and enables to transfer techniques across research lines in a principled way.

For example, we apply the importance weighting method in VAE literatures for improved GAN learning, and enhance VAEs with an adversarial mechanism that leverages generated samples.

Experiments show generality and effectiveness of the transfered techniques.

A unified statistical view of the broad class of deep generative models

The paper develops a framework interpreting GAN algorithms as performing a form of variational inference on a generative model reconstructing an indicator variable of whether a sample is from the true of generative data distributions.

On Unifying Deep Generative Models


Deep neural networks have demonstrated promising prediction and classification performance on many healthcare applications.

However, the interpretability of those models are often lacking.

On the other hand, classical interpretable models such as rule lists or decision trees do not lead to the same level of accuracy as deep neural networks and can often be too complex to interpret (due to the potentially large depth of rule lists).

In this work, we present PEARL,  Prototype lEArning via Rule Lists, which iteratively uses rule lists to guide a neural network to learn representative data prototypes.

The resulting prototype neural network provides  accurate prediction, and the prediction can be easily explained by  prototype and its guiding rule lists.

Thanks to the prediction power of neural networks, the rule lists from				 prototypes are more concise and hence provide better interpretability.

On two real-world electronic healthcare records (EHR) datasets, PEARL consistently outperforms all baselines across both datasets, especially achieving performance improvement over conventional rule learning by up to 28% and over prototype learning by up to 3%.

Experimental results also show the resulting interpretation of PEARL is  simpler than the standard rule learning.

a method combining rule list learning and prototype learning

Presents a new interpretable prediction framework, which combines rule based learning, prototype learning, and NNs, that is particularly applicable to longitudinal data.

This paper aims at tackling the lack of interpretability of deep learning models, and propose Prototype lEArning via Rule Lists (PEARL), which combines rule learning and prototype learning to achieve more accurate classification and makes the task of interpretability simpler.

Pearl: Prototype lEArning via Rule Lists

prototype learning
combine rule
pearl
list
learn
via


Generative Adversarial Networks (GANs) are powerful tools for realistic image generation.

However, a major drawback of GANs is that they are especially hard to train, often requiring large amounts of data and long training time.

In this paper we propose the Deli-Fisher GAN, a GAN that generates photo-realistic images by enforcing structure on the latent generative space using similar approaches in \cite{deligan}.

The structure of the latent space we consider in this paper is modeled as a mixture of Gaussians, whose parameters are learned in the training process.

Furthermore, to improve stability and efficiency, we use the Fisher Integral Probability Metric as the divergence measure in our GAN model, instead of the Jensen-Shannon divergence.

We show by experiments that the Deli-Fisher GAN performs better than DCGAN, WGAN, and the Fisher GAN as measured by inception score.

This paper proposes a new Generative Adversarial Network that is more stable, more efficient, and produces better images than those of status-quo

This paper combines Fisher-GAN and Deli-GAN

This paper combines Deli-GAN, which has a mixture prior distribution in latent space, and Fisher GAN, which uses Fisher IPM instead of JSD as an objective.

Deli-Fisher GAN: Stable and Efficient Image Generation With Structured Latent Generative Space

stable efficient
combine
deli-gan
gan
generative
image
space
latent


Recent work on encoder-decoder models for sequence-to-sequence mapping has shown that integrating both temporal and spatial attentional mechanisms into neural networks increases the performance of the system substantially.

We report on a new modular network architecture that applies an attentional mechanism not on temporal and spatial regions of the input, but on sensor selection for multi-sensor setups.

This network called the sensor transformation attention network (STAN) is evaluated in scenarios which include the presence of natural noise or synthetic dynamic noise.

We demonstrate how the attentional signal responds dynamically to changing noise levels and sensor-specific noise, leading to reduced word error rates (WERs) on both audio and visual tasks using TIDIGITS and GRID; and also on CHiME-3, a multi-microphone real-world noisy dataset.

The improvement grows as more channels are corrupted as demonstrated on the CHiME-3 dataset.

Moreover, the proposed STAN architecture naturally introduces a number of advantages including ease of removing sensors from existing architectures, attentional interpretability, and increased robustness to a variety of noise environments.

We introduce a modular multi-sensor network architecture with an attentional mechanism that enables dynamic sensor selection on real-world noisy data from CHiME-3.

A generic neural architecture able to learn the attention that must be payed to different input channels depending on the relative quality of each sensor with respect to the others.

Considers the use of attention for sensor or channel selection with results on TIDIGITS and GRID showing a benefit of attention over concatentation of features.

Sensor Transformation Attention Networks

sensor attention selection network architecture channel


Massive data exist among user local platforms that usually cannot support deep neural network (DNN) training due to computation and storage resource constraints.

Cloud-based training schemes provide beneficial services but suffer from potential privacy risks due to excessive user data collection.

To enable cloud-based DNN training while protecting the data privacy simultaneously, we propose to leverage the intermediate representations of the data, which is achieved by splitting the DNNs and deploying them separately onto local platforms and the cloud.

The local neural network (NN) is used to generate the feature representations.

To avoid local training and protect data privacy, the local NN is derived from pre-trained NNs.

The cloud NN is then trained based on the extracted intermediate representations for the target learning task.

We validate the idea of DNN splitting by characterizing the dependency of privacy loss and classification accuracy on the local NN topology for a convolutional NN (CNN) based image classification task.

Based on the characterization, we further propose PrivyNet to determine the local NN topology, which optimizes the accuracy of the target learning task under the constraints on privacy loss, local computation, and storage.

The efficiency and effectiveness of PrivyNet are demonstrated with CIFAR-10 dataset.

To enable cloud-based DNN training while protecting the data privacy simultaneously, we propose to leverage the intermediate data representations, which is achieved by splitting the DNNs and deploying them separately onto local platforms and the cloud.

This paper proposes a technique to privatize data by learning a feature representation that is difficult to use for image reconstruction, but helpful for image classification.

PrivyNet: A Flexible Framework for Privacy-Preserving Deep Neural Network Training


Generative Adversarial Networks (GANs) have shown remarkable success as a framework for training models to produce realistic-looking data.

In this work, we propose a Recurrent GAN (RGAN) and Recurrent Conditional GAN (RCGAN) to produce realistic real-valued multi-dimensional time series, with an emphasis on their application to medical data.

RGANs make use of recurrent neural networks (RNNs) in the generator and the discriminator.

In the case of RCGANs, both of these RNNs are conditioned on auxiliary information.

We demonstrate our models in a set of toy datasets, where we show visually and quantitatively (using sample likelihood and maximum mean discrepancy) that they can successfully generate realistic time-series.

We also describe novel evaluation methods for GANs, where we generate a synthetic labelled training dataset, and evaluate on a real test set the performance of a model trained on the synthetic data, and vice-versa.

We illustrate with these metrics that RCGANs can generate time-series data useful for supervised training, with only minor degradation in performance on real test data.

This is demonstrated on digit classification from ‘serialised’ MNIST and by training an early warning system on a medical dataset of 17,000 patients from an intensive care unit.

We further discuss and analyse the privacy concerns that may arise when using RCGANs to generate realistic synthetic medical time series data, and demonstrate results from differentially private training of the RCGAN.

Conditional recurrent GANs for real-valued medical sequences generation, showing novel evaluation approaches and an empirical privacy analysis.

Proposes to use synthetic data generated by GANs as a replacement for personally identifiable data in training ML models for privacy-sensitive applications

The authors propose a novel recurrent GAN architecture that generates continuous domain sequences, and evaluate it on several synthetic tasks and an ICU timeseries data task.

Proposes to use RGANs and RCGANs to generate synthetic sequences of actual data.

Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs

recurrent gan
generation
generate
datum
sequence
medical
novel
conditional
synthetic


Emphasis effects – visual changes that make certain elements more

prominent – are commonly used in information visualization to draw

the user’s attention or to indicate importance.

frameworks of emphasis exist (that link visually diverse emphasis

effects through the idea of visual prominence compared to background

elements), most metrics for predicting how emphasis effects

will be perceived by users come from abstract models of human

vision which may not apply to visualization design.

it is difficult for designers to know, when designing a visualization,

how different emphasis effects will compare and what level of one

effect is equivalent to what level of another.

we carried out two studies that provide empirical evidence about

how users perceive different emphasis effects, using three visual

variables (colour, size, and blur/focus) and eight strength levels.

Results from gaze tracking, mouse clicks, and subjective responses

show that there are significant differences between visual variables

and between levels, and allow us to develop an initial understanding

our first study, and used it to predict the results in the second; the

model was accurate, with high correlations between predictions and

emphasis effects will be perceived by users.

Our studies and empirical models provide valuable new information for designers who want to understand and control how emphasis effects will be perceived by users

This paper considers which visual highlighting is perceived faster in data visualization and how different highlighting methods compare to each other

Two studies on the efficacy of emphasis effects, one assessing levels of useful differences, and one more applied using actual different visualizations for a more ecologically valid investigation.

A Baseline Study of Emphasis Effects in Information Visualization

emphasis effect
study
visualization
perceive
information
different


Memory Network based models have shown a remarkable progress on the task of relational reasoning.

Recently, a simpler yet powerful neural network module called Relation Network (RN) has been introduced.

Despite its architectural simplicity, the time complexity of relation network grows quadratically with data, hence limiting its application to tasks with a large-scaled memory.

We introduce Related Memory Network, an end-to-end neural network architecture exploiting both memory network and relation network structures.

We follow memory network's four components while each component operates similar to the relation network without taking a pair of objects.

As a result, our model is as simple as RN but the computational complexity is reduced to linear time.

It achieves the state-of-the-art results in jointly trained bAbI-10k story-based question answering and  bAbI dialog dataset.

A simple reasoning architecture based on the memory network (MemNN) and relation network (RN), reducing the time complexity compared to the RN and achieving state-of-the-are result on bAbI story based QA and bAbI dialog.

Introduces Related Memory Network (RMN), an improvement over Relationship Networks (RN).

Finding ReMO (Related Memory Object): A Simple neural architecture for Text based Reasoning

network rn
simple
relate memory
reasoning
architecture
base


We investigate in this paper the architecture of deep convolutional networks.

Building on existing state of the art models, we propose a reconfiguration of the model parameters into several parallel branches at the global network level, with each branch being a standalone CNN.

We show that this arrangement is an efficient way to significantly reduce the number of parameters while at the same time improving the performance.

The use of branches brings an additional form of regularization.

In addition to splitting the parameters into parallel branches, we propose a tighter coupling of these branches by averaging their log-probabilities.

The tighter coupling favours the learning of better representations, even at the level of the individual branches, as compared to when each branch is trained independently.

We refer to this branched architecture as "coupled ensembles".

The approach is very generic and can be applied with almost any neural network architecture.

With coupled ensembles of DenseNet-BC and parameter budget of 25M, we obtain error rates of 2.92%, 15.68% and 1.50% respectively on CIFAR-10, CIFAR-100 and SVHN tasks.

For the same parameter budget, DenseNet-BC has an error rate of 3.46%, 17.18%, and 1.8% respectively.

With ensembles of coupled ensembles, of DenseNet-BC networks, with 50M total parameters, we obtain error rates of 2.72%, 15.13% and 1.42% respectively on these tasks.

We show that splitting a neural network into parallel branches improves performance and that proper coupling of the branches improves performance even further.

The work proposes a reconfiguration of the existing state-of-the-art CNN model using a new branching architecture, with better performance.

This paper shows parameter-saving benefits of coupled ensembling.

Presents a deep network architecture which processes data using multiple parallel branches and combines the posterior from these branches to compute the final scores.

Coupled Ensembles of Neural Networks

parallel branch
neural network
show
performance
architecture
couple


Convolutional Neural Networks (CNN) are very popular in many fields including computer vision, speech recognition, natural language processing, to name a few.

Though deep learning leads to groundbreaking performance in these domains, the networks used are very demanding computationally and are far from real-time even on a GPU, which is not power efficient and therefore does not suit low power systems such as mobile devices.

To overcome this challenge, some solutions have been proposed for quantizing the weights and activations of these networks, which accelerate the runtime significantly.

Yet, this acceleration comes at the cost of a larger error.

The NICE method proposed in this work trains quantized neural networks by noise injection and a learned clamping, which improve the accuracy.

This leads to state-of-the-art results on various regression and classification tasks, e.g., ImageNet classification with architectures such as ResNet-18/34/50 with low as 3-bit weights and 3 -bit activations.

We implement the proposed solution on an FPGA to demonstrate its applicability for low power real-time applications.

Combine noise injection, gradual quantization and activation clamping learning to achieve state-of-the-art 3,4 and 5 bit quantization

Proposes to inject noise during training and clamp parameter values in a layer as well as activation output in neural network quantization.

A method for quantization of deep neural networks for classification and regression, using noise injection, clamping with learned maximum activations, and gradual block quantization to perform on-par or better than state-of-the-art methods.

NICE: noise injection and clamping estimation for neural network quantization

noise injection clamp
neural network quantization
well
learn
activation
gradual
state-of-the-art


In complex transfer learning scenarios new tasks might not be tightly linked to previous tasks.

Approaches that transfer information contained only in the final parameters of a source model will therefore struggle.

Instead, transfer learning at at higher level of abstraction is needed.

We propose Leap, a framework that achieves this by transferring knowledge across learning processes.

We associate each task with a manifold on which the training process travels from initialization to final parameters and construct a meta-learning objective that minimizes the expected length of this path.

Our framework leverages only information obtained during training and can be computed on the fly at negligible cost.

We demonstrate that our framework outperforms competing methods, both in meta-learning and transfer learning, on a set of computer vision tasks.

Finally, we demonstrate that Leap can transfer knowledge across learning processes in demanding reinforcement learning environments (Atari) that involve millions of gradient steps.

We propose Leap, a framework that transfers knowledge across learning processes by  minimizing the expected distance the training process travels on a task's loss surface.

The article proposes a novel meta-learning objective aimed at outperforming state-of-the-art approaches when dealing with collections of tasks that exhibit substantial between-task diversity

Transferring Knowledge across Learning Processes

transfer knowledge across learn process
task


Given an existing trained neural network, it is often desirable to learn new capabilities without hindering performance of those already learned.

Existing approaches either learn sub-optimal solutions, require joint training, or incur a substantial increment in the number of parameters for each added task, typically as many as the original network.

We propose a method called Deep Adaptation Networks (DAN) that constrains newly learned filters to be linear combinations of existing ones.

DANs preserve performance on the original task, require a fraction (typically 13%) of the number of parameters compared to standard fine-tuning procedures and converge in less cycles of training to a comparable or better level of performance.

When coupled with standard network quantization techniques, we further reduce the parameter cost to around 3% of the original with negligible or no loss in accuracy.

The learned architecture can be controlled to switch between various learned representations, enabling a single network to solve a task from multiple different domains.

We conduct extensive experiments showing the effectiveness of our method on a range of image classification tasks and explore different aspects of its behavior.

An alternative to transfer learning that learns faster, requires much less parameters (3-13 %), usually achieves better results and precisely preserves performance on old tasks.

Controller modules for increment learning on image classification datasets

Incremental Learning through Deep Adaptation


High throughput and low latency inference of deep neural networks are critical for the deployment of deep learning applications.

This paper presents a general technique toward 8-bit low precision inference of convolutional neural networks, including

1) channel-wise scale factors of weights, especially for depthwise convolution,

We experiment the techniques on top of a widely-used deep learning framework.

The 8-bit optimized model is automatically generated with a calibration process from FP32 model without the need of fine-tuning or retraining.

We perform a systematical and comprehensive study on 18 widely-used convolutional neural networks and demonstrate the effectiveness of 8-bit low precision inference across a wide range of applications and use cases, including image classification, object detection, image segmentation, and super resolution.

and latency are improved by 1.6X and 1.5X respectively with minimal within 0.6%1to no loss in accuracy from FP32 baseline.

We believe the methodology can provide the guidance and reference design of 8-bit low precision inference for other frameworks.

All the code and models will be publicly available soon.

We present a general technique toward 8-bit low precision inference of convolutional neural networks.

This paper designs a system to automatically quantize the CNN pretrained models

HIGHLY EFFICIENT 8-BIT LOW PRECISION INFERENCE OF CONVOLUTIONAL NEURAL NETWORKS

8-bit low precision inference convolutional neural network


Recent approaches have successfully demonstrated the benefits of learning the parameters of shallow networks in hyperbolic space.

We extend this line of work by imposing hyperbolic geometry on the embeddings used to compute the ubiquitous attention mechanisms for different neural networks architectures.

By only changing the geometry of embedding of object representations, we can use the embedding space more efficiently without increasing the number of parameters of the model.

Mainly as the number of objects grows exponentially for any semantic distance from the query, hyperbolic geometry  --as opposed to Euclidean geometry-- can encode those objects without having any interference.

Our method shows improvements in generalization on neural machine translation on WMT'14 (English to German), learning on graphs (both on synthetic and real-world graph tasks) and visual question answering (CLEVR) tasks while keeping the neural representations compact.

We propose to incorporate inductive biases and operations coming from hyperbolic geometry to improve the attention mechanism of the neural networks.

This paper replaces the dot-product similarity used in attention mechanisms with the negative hyperbolic distance, and applies it to the existing Transformer model, graph attention networks, and Relation Networks

The authors propose a novel approach to improve relational-attention by changing the matching and aggregation functions to use hyperbolic geometric.

Hyperbolic Attention Networks

attention mechanism
hyperbolic
network
improve


We present a method for evaluating the sensitivity of deep reinforcement learning (RL) policies.

We also formulate a zero-sum dynamic game for designing robust deep reinforcement learning policies.

Our approach mitigates the brittleness of policies when agents are trained in a simulated environment and are later exposed to the real world where it is hazardous to employ RL policies.

This framework for training deep RL policies involve a zero-sum  dynamic game against an adversarial agent, where the goal is to drive the system dynamics to a saddle region.

Using a variant of the guided policy search algorithm, our agent learns to adopt robust policies that require less samples for learning the dynamics and performs better than the GPS algorithm.

Without loss of generality, we demonstrate that deep RL policies trained in this fashion will be maximally robust to a ``worst" possible adversarial disturbances.

This paper demonstrates how H-infinity control theory can help better design robust deep policies for robot motor taks

Proposes to incorporate elements of robust control into guided policy research in order to devise a method that is resilient to perturbations and model mismatch.

The paper presents a method for evaluating the sensitivity and robustness of deep RL policies, and proposes a dynamic game approach for learning robust policies.

A dynamic game approach to training robust deep policies

dynamic game approach
robust deep policy
method
control


Deep networks have recently been shown to be vulnerable to universal perturbations: there exist very small image-agnostic perturbations that cause most natural images to be misclassified by such classifiers.

In this paper, we provide a quantitative analysis of the robustness of classifiers to universal perturbations, and draw a formal link between the robustness to universal perturbations, and the geometry of the decision boundary.

Specifically, we establish theoretical bounds on the robustness of classifiers under two decision boundary models (flat and curved models).

We show in particular that the robustness of deep networks to universal perturbations is driven by a key property of their curvature: there exist shared directions along which the decision boundary of deep networks is systematically positively curved.

Under such conditions, we prove the existence of small universal perturbations.

Our analysis further provides a novel geometric method for computing universal perturbations, in addition to explaining their properties.

Analysis of vulnerability of classifiers to universal perturbations and relation to the curvature of the decision boundary.

The paper provides an interesting analysis linking the geometry of classifier decision boundaries to small universal adversarial perturbations.

This paper discusses universal perturbations - perturbations that can mislead a trained classifier if added to most of input data points.

The paper develops models which attempt to explain the existence of universal perturbations which fool neural networks

Robustness of Classifiers to Universal Perturbations: A Geometric Perspective

classifier universal perturbation
decision boundary
analysis


Behavioral skills or policies for autonomous agents are conventionally learned from reward functions, via reinforcement learning, or from demonstrations, via imitation learning.

However, both modes of task specification have their disadvantages: reward functions require manual engineering, while demonstrations require a human expert to be able to actually perform the task in order to generate the demonstration.

Instruction following from natural language instructions provides an appealing alternative: in the same way that we can specify goals to other humans simply by speaking or writing, we would like to be able to specify tasks for our machines.

However, a single instruction may be insufficient to fully communicate our intent or, even if it is, may be insufficient for an autonomous agent to actually understand how to perform the desired task.

In this work, we propose an interactive formulation of the task specification problem, where iterative language corrections are provided to an autonomous agent, guiding it in acquiring the desired skill.

Our proposed language-guided policy learning algorithm can integrate an instruction and a sequence of corrections to acquire new skills very quickly.

In our experiments, we show that this method can enable a policy to follow instructions and corrections for simulated navigation and manipulation tasks, substantially outperforming direct, non-interactive instruction following.

We propose a meta-learning method for interactively correcting policies with natural language.

This paper provides a meta learning framework that shows how to learn new tasks in an interactive setup. Each task is learned through a reinforcement learning setup, and then the task is being updated by observing new instructions.

This paper teaches agents to complete tasks via natural language instructions in an iterative process.

Guiding Policies with Language via Meta-Learning

policy
natural language
instruction
task
via


Deep generative models such as Generative Adversarial Networks (GANs) and

Variational Auto-Encoders (VAEs) are important tools to capture and investigate

the properties of complex empirical data.

elements makes their functionment challenging to assess and modify.

respect, these architectures behave as black box models.

understand the function of such networks, we analyze their modularity based on

the counterfactual manipulation of their internal variables.

generation of human faces with VAEs and GANs support that modularity between

activation maps distributed over channels of generator architectures is achieved

to some degree, can be used to better understand how these systems operate and allow meaningful transformations of the generated images without further training.

erate and edit the content of generated images.

We investigate the modularity of deep generative models.

The paper provides a way to investigate the modular structure of the deep generative model, with the key concept to distribute over channels of generator architectures.

Tinkering with black boxes: counterfactuals uncover modularity in generative models

deep generative model
investigate
modularity


Relational databases store a significant amount of the worlds data.

However, accessing this data currently requires users to understand a query language such as SQL.

We propose Seq2SQL, a deep neural network for translating natural language questions to corresponding SQL queries.

Our model uses rewards from in the loop query execution over the database to learn a policy to generate the query, which contains unordered parts that are less suitable for optimization via cross entropy loss.

Moreover, Seq2SQL leverages the structure of SQL to prune the space of generated queries and significantly simplify the generation problem.

In addition to the model, we release WikiSQL, a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables fromWikipedia that is an order of magnitude larger than comparable datasets.

By applying policy based reinforcement learning with a query execution environment to WikiSQL, Seq2SQL outperforms a state-of-the-art semantic parser, improving execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.

We introduce Seq2SQL, which translates questions to SQL queries using rewards from online query execution, and WikiSQL, a SQL table/question/query dataset orders of magnitude larger than existing datasets.

A new semantic parsing dataset which focuses on generating SQL from natural language using a reinforcement-learning based model

Seq2SQL: Generating Structured Queries From Natural Language Using Reinforcement Learning

natural language
query
seq2sql
dataset
sql
generate


We introduce Explainable Adversarial Learning, ExL, an approach for training neural networks that are intrinsically robust to adversarial attacks.

We find that the implicit generative modeling of random noise with the same loss function used during posterior maximization, improves a model's understanding of the data manifold furthering adversarial robustness.

We prove our approach's efficacy and provide a simplistic visualization tool for understanding adversarial data, using Principal Component Analysis.

Our analysis reveals that adversarial robustness, in general, manifests in models with higher variance along the high-ranked principal components.

We show that models learnt with our approach perform remarkably well against a wide-range of attacks.

Furthermore, combining ExL with state-of-the-art adversarial training extends the robustness of a model, even beyond what it is adversarially trained for, in both white-box and black-box attack scenarios.

Noise modeling at the input during discriminative training improves adversarial robustness. Propose PCA based evaluation metric for adversarial robustness

This paper proposes, ExL, an adversarial training method using multiplicate noise that is shown to be helpful in defending against blackbox attacks on three datasets.

This paper includes multiplicative noise N in training data to achieve adversarial robustness, when training on both model parameters theta and on the noise itself.

Explainable Adversarial Learning: Implicit Generative Modeling of Random Noise during Training for Adversarial Robustness

adversarial robustness
model
noise
training


We propose a method which can visually explain the classification decision of deep neural networks (DNNs).

There are many proposed methods in machine learning and computer vision seeking to clarify the decision of machine learning black boxes, specifically DNNs.

All of these methods try to gain insight into why the network "chose class A" as an answer.

Humans, when searching for explanations, ask two types of questions.

The first question is, "Why did you choose this answer?

The second question asks, "Why did you not choose answer B over A?"

The previously proposed methods are either not able to provide the latter directly or efficiently.

We introduce a method capable of answering the second question both directly and efficiently.

In this work, we limit the inputs to be images.

In general, the proposed method generates explanations in the input space of any model capable of efficient evaluation and gradient evaluation.

We provide results, showing the superiority of this approach for gaining insight into the inner representation of machine learning models.

A method to answer "why not class B?" for explaining deep networks

The paper proposes an approach to provide contrastive visual explanations for deep neural networks.

CDeepEx: Contrastive Deep Explanations


We flip the usual approach to study invariance and robustness of neural networks by considering the non-uniqueness and instability of the inverse mapping.

We provide theoretical and numerical results on the inverse of ReLU-layers.

First, we derive a necessary and sufficient condition on the existence of invariance that provides a geometric interpretation.

Next, we move to robustness via analyzing local effects on the inverse.

To conclude, we show how this reverse point of view not only provides insights into key effects, but also enables to view adversarial examples from different perspectives.

We analyze the invertibility of deep neural networks by studying preimages of ReLU-layers and the stability of the inverse.

This paper studies the volume of preimage of a ReLU network’s activation at a certain layer, and it builds on the piecewise linearity of a ReLU network’s forward function.

This paper presents an analysis of the inverse invariance of ReLU networks and provides upper bounds on singular values of a train network.

Invariance and Inverse Stability under ReLU

relu network
inverse
study
preimage
stability
invariance


While deep learning has led to remarkable results on a number of challenging problems, researchers have discovered a vulnerability of neural networks in adversarial settings, where small but carefully chosen perturbations to the input can make the models produce extremely inaccurate outputs.

This makes these models particularly unsuitable for safety-critical application domains (e.g. self-driving cars) where robustness is extremely important.

Recent work has shown that augmenting training with adversarially generated data provides some degree of robustness against test-time attacks.

In this paper we investigate how this approach scales as we increase the computational budget given to the defender.

We show that increasing the number of parameters in adversarially-trained models increases their robustness, and in particular that ensembling smaller models while adversarially training the entire ensemble as a single model is a more efficient way of spending said budget than simply using a larger single model.

Crucially, we show that it is the adversarial training of the ensemble, rather than the ensembling of adversarially trained models, which provides robustness.

Adversarial training of ensembles provides robustness to adversarial examples beyond that observed in adversarially trained models and independently-trained ensembles thereof.

Proposes to train an ensemble of models jointly, where at each time step, a set of examples that are adversarial for the ensemble itself is incorporated in the learning.

Strength in Numbers: Trading-off Robustness and Computation via Adversarially-Trained Ensembles

adversarial robustness ensemble model example train


Multi-task learning (MTL) with neural networks leverages commonalities in tasks to improve performance, but often suffers from task interference which reduces the benefits of transfer.

To address this issue we introduce the routing network paradigm, a novel neural network and training algorithm.

A routing network is a kind of self-organizing neural network consisting of two components: a router and a set of one or more function blocks.

A function block may be any neural network – for example a fully-connected or a convolutional layer.

Given an input the router makes a routing decision, choosing a function block to apply and passing the output back to the router recursively, terminating when a fixed recursion depth is reached.

In this way the routing network dynamically composes different function blocks for each input.

We employ a collaborative multi-agent reinforcement learning (MARL) approach to jointly train the router and function blocks.

We evaluate our model against cross-stitch networks and shared-layer baselines on multi-task settings of the MNIST, mini-imagenet, and CIFAR-100 datasets.

Our experiments demonstrate a significant improvement in accuracy, with sharper convergence.

In addition, routing networks have nearly constant per-task training cost while cross-stitch networks scale linearly with the number of tasks.

On CIFAR100 (20 tasks) we obtain cross-stitch performance levels with an 85% average reduction in training time.

routing networks: a new kind of neural network which learns to adaptively route its input for multi-task learning

The paper suggests to use a modular network with a controller which makes decisions, at each time step, regarding the next nodule to apply.

The paper presents a novel formulation for learning the optimal architecture of a neural network in a multi-task learning framework by using multi-agent reinforcement learning to find a policy, and shows improvement over hard-coded architectures with shared layers.

Routing Networks: Adaptive Selection of Non-Linear Functions for Multi-Task Learning

neural network
learn
multi-task learning
route


We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero.

Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization.

AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization.

However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function.

We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero.

We show that, somewhat surprisingly, for certain distributions over the gates, the expected $L_0$ regularized objective is differentiable with respect to the distribution parameters.

We further propose the \emph{hard concrete} distribution for the gates, which is obtained by ``stretching'' a binary concrete distribution and then transforming its samples with a hard-sigmoid.

The parameters of the distribution over the gates can then be jointly optimized with the original network parameters.

As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way.

We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.

We show how to optimize the expected L_0 norm of parametric models with gradient descent and introduce a new distribution that facilitates hard gating.

The authors introduce a gradient-based approach to minimize an objective function with an L0 sparse penalty to help learn sparse neural networks

Learning Sparse Neural Networks through L_0 Regularization

learn sparse neural network
introduce
l_0


Recently popularized graph neural networks achieve the state-of-the-art accuracy on a number of standard benchmark datasets for graph-based semi-supervised learning, improving significantly over existing approaches.

These architectures alternate between a propagation layer that aggregates the hidden states of the local neighborhood and a fully-connected layer.

Perhaps surprisingly, we show that a linear model, that removes all the intermediate fully-connected layers, is still able to achieve a performance comparable to the state-of-the-art models.

This significantly reduces the number of parameters, which is critical for semi-supervised learning where number of labeled examples are small.

This in turn allows a room for designing more innovative propagation layers.

Based on this insight, we propose a novel graph neural network that removes all the intermediate fully-connected layers, and replaces the propagation layers with attention mechanisms that respect the structure of the graph.

The attention mechanism allows us to learn a dynamic and adaptive local summary of the neighborhood to achieve more accurate predictions.

In a number of experiments on benchmark citation networks datasets, we demonstrate that our approach outperforms competing methods.

By examining the attention weights among neighbors, we show that our model provides some interesting insights on how neighbors influence each other.

We propose a novel attention-based interpretable Graph Neural Network architecture which outperforms the current state-of-the-art Graph Neural Networks in standard benchmark datasets

The authors propose two extensions of GCNs, by removing intermediate non-linearities from the GCN computation and adding an attention mechanism in the aggregation layer.

The paper proposes a semi supervised learning algorithm for graph node classification with is inspired from Graph Neural Networks.

Attention-based Graph Neural Network for Semi-supervised Learning


Modern generative models are usually designed to match target distributions directly in the data space, where the intrinsic dimensionality of data can be much lower than the ambient dimensionality.

We argue that this discrepancy may contribute to the difficulties in training generative models.

We therefore propose to map both the generated and target distributions to the latent space using the encoder of a standard autoencoder, and train the generator (or decoder) to match the target distribution in the latent space.

The resulting method, perceptual generative autoencoder (PGA), is then incorporated with maximum likelihood or variational autoencoder (VAE) objective to train the generative model.

With maximum likelihood, PGA generalizes the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities.

When combined with VAE, PGA can generate sharper samples than vanilla VAE.

A framework for training autoencoder-based generative models, with non-adversarial losses and unrestricted neural network architectures.

This paper uses autoencoders to do distribution matching in high dimensional space.

Perceptual Generative Autoencoders


The quality of the representations achieved by embeddings is determined by how well the geometry of the embedding space matches the structure of the data.

Euclidean space has been the workhorse for embeddings; recently hyperbolic and spherical spaces have gained popularity due to their ability to better embed new types of structured data---such as hierarchical data---but most data is not structured so uniformly.

We address this problem by proposing learning embeddings in a product manifold combining multiple copies of these model spaces (spherical, hyperbolic, Euclidean), providing a space of heterogeneous curvature suitable for a wide variety of structures.

We introduce a heuristic to estimate the sectional curvature of graph data and directly determine an appropriate signature---the number of component spaces and their dimensions---of the product manifold.

Empirically, we jointly learn the curvature and the embedding in the product space via Riemannian optimization.

We discuss how to define and compute intrinsic quantities such as means---a challenging notion for product manifolds---and provably learnable optimization functions.

On a range of datasets and reconstruction tasks, our product space embeddings outperform single Euclidean or hyperbolic spaces used in previous works, reducing distortion by 32.55% on a Facebook social network dataset.

We learn word embeddings and find that a product of hyperbolic spaces in 50 dimensions consistently improves on baseline Euclidean and hyperbolic embeddings, by 2.6

points in Spearman rank correlation on similarity tasks

Product manifold embedding spaces with heterogenous curvature yield improved representations compared to traditional embedding spaces for a variety of structures.

Proposes a dimensionality reduction method that embeds data into a product manifold of spherical, Euclidean, and hyperbolic manifolds. The algorithm is based on matching the geodesic distances on the product manifold to graph distances.

Learning Mixed-Curvature Representations in Product Spaces

product manifold
space
representation
embed


Synthesizing user-intended programs from a small number of input-output exam-

ples is a challenging problem with several important applications like spreadsheet

manipulation, data wrangling and code refactoring.

either completely rely on deductive logic techniques that are extensively hand-

engineered or on purely statistical models that need massive amounts of data, and in

general fail to provide real-time synthesis on challenging benchmarks.

we propose Neural Guided Deductive Search (NGDS), a hybrid synthesis technique

that combines the best of both symbolic logic techniques and statistical models.

Thus, it produces programs that satisfy the provided specifications by construction

and generalize well on unseen examples, similar to data-driven systems.

technique effectively utilizes the deductive search framework to reduce the learning

problem of the neural component to a simple supervised learning setup.

this allows us to both train on sparingly available real-world data and still leverage

powerful recurrent neural network encoders.

of our method by evaluating on real-world customer scenarios by synthesizing

accurate programs with up to 12× speed-up compared to state-of-the-art systems.

We integrate symbolic (deductive) and statistical (neural-based) methods to enable real-time program synthesis with almost perfect generalization from 1 input-output example.

The paper presents a branch-and-bound approach to learn good programs where an LSTM is used to predict which branches in the search tree should lead to good programs

Proposes system that synthesizes programs from a single example that generalize better than prior state-of-the-art

Neural-Guided Deductive Search for Real-Time Program Synthesis from Examples

real-time program synthesis
example
search
deductive


Variational auto-encoders
 (VAEs) offer a tractable approach when performing approximate inference in otherwise intractable generative models.

However, standard VAEs often produce latent codes that are disperse and lack interpretability, thus making the resulting representations unsuitable for auxiliary tasks (e.g. classiﬁcation) and human interpretation.

We address these issues by merging ideas from variational auto-encoders and sparse coding, and propose to explicitly model sparsity in the latent space of a VAE with a Spike and Slab prior distribution.

We derive the evidence lower bound using a discrete mixture recognition function thereby making approximate posterior inference as computational efﬁcient as in the standard VAE case.

With the new approach, we are able to infer truly sparse representations with generally intractable non-linear probabilistic models.

We show that these sparse representations are advantageous over standard VAE representations on two benchmark classiﬁcation tasks (MNIST and Fashion-MNIST) by demonstrating improved classiﬁcation accuracy and signiﬁcantly increased robustness to the number of latent dimensions.

Furthermore, we demonstrate qualitatively that the sparse elements capture subjectively understandable sources of variation.

We explore the intersection of VAEs and sparse coding.

This paper proposes an extension of VAEs with sparse priors and posteriors to learn sparse interpretable representations.

Variational Sparse Coding