Online learning has attracted great attention due to the increasing demand for systems that have the ability of learning and evolving.

When the data to be processed is also high dimensional and dimension reduction is necessary for visualization or prediction enhancement, online dimension reduction will play an essential role.

The purpose of this paper is to propose new online learning approaches for supervised dimension reduction.

Our first algorithm is motivated by adapting the sliced inverse regression (SIR), a pioneer and effective algorithm for supervised dimension reduction, and making it implementable in an incremental manner.

The new algorithm, called incremental sliced inverse regression (ISIR), is able to update the subspace of significant factors with intrinsic lower dimensionality fast and efficiently when new observations come in.

We also refine the algorithm by using an overlapping technique  and develop an incremental overlapping sliced inverse regression (IOSIR) algorithm.

We verify the effectiveness and efficiency of both algorithms by simulations and real data applications.

We proposed two new approaches,  the incremental sliced inverse regression and incremental overlapping sliced inverse regression, to implement supervised dimension reduction in an online learning manner.

Studies sufficient dimension reduction problem and proposes an incremental sliced inverse regression algorithm.

This paper proposes an online learning algorithm for supervised dimension reduction, called incremental sliced inverse regression

Online Learning for Supervised Dimension Reduction

incremental slice inverse regression
supervised dimension reduction
online learning
algorithm
sliced


This paper presents a storage-efficient learning model titled Recursive Binary Neural Networks for embedded and mobile devices having a limited amount of on-chip data storage such as hundreds of kilo-Bytes.

The main idea of the proposed model is to recursively recycle data storage of weights (parameters) during training.

This enables a device with a given storage constraint to train and instantiate a neural network classifier with a larger number of weights on a chip, achieving better classification accuracy.

Such efficient use of on-chip storage reduces off-chip storage accesses, improving energy-efficiency and speed of training.

We verified the proposed training model with deep and convolutional neural network classifiers on the MNIST and voice activity detection benchmarks.

For the deep neural network, our model achieves data storage requirement of as low as 2 bits/weight, whereas the conventional binary neural network learning models require data storage of 8 to 32 bits/weight.

With the same amount of data storage, our model can train a bigger network having more weights, achieving 1% less test error than the conventional binary neural network learning model.

To achieve the similar classification error, the conventional binary neural network model requires 4× more data storage for weights than our proposed model.

For the convolution neural network classifier, the proposed model achieves 2.4% less test error for the same on-chip storage or 6× storage savings to achieve the similar accuracy.

We propose a learning model enabling DNN to learn with only 2 bit/weight, which is especially useful for on-device learning

Proposes a method to discretisize a NN incrementally to improve memory and performance.

Recursive Binary Neural Network Learning Model  with 2-bit/weight Storage Requirement


Within-class variation in a high-dimensional dataset can be modeled as being on a low-dimensional manifold due to the constraints of the physical processes producing that variation (e.g., translation, illumination, etc.).

We desire a method for learning a representation of the manifolds induced by identity-preserving transformations that can be used to increase robustness, reduce the training burden, and encourage interpretability in machine learning tasks.

In particular, what is needed is a representation of the transformation manifold that can robustly capture the shape of the manifold from the input data, generate new points on the manifold, and extend transformations outside of the training domain without significantly increasing the error.

Previous work has proposed algorithms to efficiently learn analytic operators (called transport operators) that define the process of transporting one data point on a manifold to another.

The main contribution of this paper is to define two transfer learning methods that use this generative manifold representation to learn natural transformations and incorporate them into new data.

The first method uses this representation in a novel randomized approach to transfer learning that employs the learned generative model to map out unseen regions of the data space.

These results are shown through demonstrations of transfer learning in a data augmentation task for few-shot image classification.

The second method use of transport operators for injecting specific transformations into new data examples which allows for realistic image animation and informed data augmentation.

These results are shown on stylized constructions using the classic swiss roll data structure and in demonstrations of transfer learning in a data augmentation task for few-shot image classification.

We also propose the use of transport operators for injecting transformations into new data examples which allows for realistic image animation.

Learning transport operators on manifolds forms a valuable representation for doing tasks like transfer learning.

Uses a dictionary learning framework to learn manifold transport operators on augmented USPS digits.

The paper considers the framework of manifold transport operator learning of Culpepper and Olshausen (2009), and interprets it as obtaining a MAP estimate under a probabilistic generative model.

Transfer Learning on Manifolds via Learned Transport Operators

manifold transport operator
learning
framework
learn
transfer


The seemingly infinite diversity of the natural world arises from a relatively small set of coherent rules, such as the laws of physics or chemistry.

We conjecture that these rules give rise to regularities that can be discovered through primarily unsupervised experiences and represented as abstract concepts.

If such representations are compositional and hierarchical, they can be recombined into an exponentially large set of new concepts.

This paper describes SCAN (Symbol-Concept Association Network), a new framework for learning such abstractions in the visual domain.

SCAN learns concepts through fast symbol association, grounding them in disentangled visual primitives that are discovered in an unsupervised manner.

Unlike state of the art multimodal generative model baselines, our approach requires very few pairings between symbols and images and makes no assumptions about the form of symbol representations.

Once trained, SCAN is capable of multimodal bi-directional inference, generating a diverse set of image samples from symbolic descriptions and vice versa.

It also allows for traversal and manipulation of the implicit hierarchy of visual concepts through symbolic instructions and learnt logical recombination operations.

Such manipulations enable SCAN to break away from its training data distribution and imagine novel visual concepts through symbolically instructed recombination of previously learnt concepts.

We present a neural variational model for learning language-guided compositional visual concepts.

Proposes a novel neural net architecture that learns object concepts by combining a beta-VAE and SCAN.

This paper introduces a VAE-based model for translating between images and text, with their latent representation well-suited to applying symbolic operations, giving them a more expressive language for sampling images from text.

This paper proposes a new model called SCAN (Symbol-Concept Association Network) for hierarchical concept learning and allows for generalization to new concepts composed from existing concepts using logical operators.

SCAN: Learning Hierarchical Compositional Visual Concepts

compositional visual concept
model
learn
scan
neural
hierarchical


Despite much success in many large-scale language tasks, sequence-to-sequence (seq2seq) models have not been an ideal choice for conversational modeling as they tend to generate generic and repetitive responses.

In this paper, we propose a Latent Topic Conversational Model (LTCM) that augments the seq2seq model with a neural topic component to better model human-human conversations.

The neural topic component encodes information from the source sentence to build a global “topic” distribution over words, which is then consulted by the seq2seq model to improve generation at each time step.

The experimental results show that the proposed LTCM can generate more diverse and interesting responses by sampling from its learnt latent representations.

In a subjective human evaluation, the judges also confirm that LTCM is the preferred option comparing to competitive baseline models.

Latent Topic Conversational Model, a hybrid of seq2seq and neural topic model to generate more diverse and interesting responses.

This paper proposed the combination of topic model and seq2seq conversational model

Proposes a conversational model with topical information by combining seq2seq model with neural topic models and shows the proposed model outperforms some the baseline model seq2seq and other latent variable model variant of seq2seq.

The paper addresses the issue of enduring topicality in conversation models and proposes a model which is a combination of a neural topic model and a seq2seq-based dialog system.

Latent Topic Conversational Models

combination
neural topic model
seq2seq
latent
conversational


Most of the existing Graph Neural Networks (GNNs) are the mere extension of the Convolutional Neural Networks (CNNs) to graphs.

Generally, they consist of several steps of message passing between the nodes followed by a global indiscriminate feature pooling function.

In many data-sets, however, the nodes are unlabeled or their labels provide no information about the similarity between the nodes and the locations of the nodes in the graph.

Accordingly, message passing may not propagate helpful information throughout the graph.

We show that this conventional approach can fail to learn to perform even simple graph classification tasks.

We alleviate this serious shortcoming of the GNNs by making them a two step method.

In the first of the proposed approach, a graph embedding algorithm is utilized to obtain a continuous feature vector for each node of the graph.

The embedding algorithm represents the graph as a point-cloud in the embedding space.

In the second step, the GNN is applied to the point-cloud representation of the graph provided by the embedding method.

The GNN learns to perform the given task by inferring the topological structure of the graph encoded in the spatial distribution of the embedded vectors.

In addition, we extend the proposed approach to the graph clustering problem and a new architecture for graph clustering is proposed.

Moreover, the spatial representation of the graph is utilized to design a graph pooling algorithm.

We turn the problem of graph down-sampling into a column sampling problem, i.e., the sampling algorithm selects a subset of the nodes whose feature vectors preserve the spatial distribution of all the feature vectors.

We apply the proposed approach to several popular benchmark data-sets and it is shown that the proposed geometrical approach strongly improves the state-of-the-art result for several data-sets.

For instance, for the PTC data-set, we improve the state-of-the-art result for more than 22 %.

The graph analysis problem is transformed into a point cloud analysis problem.

Proposes a deep GNN network for graph classification problems using their adaptive graph pooling layer.

The authors propose a method for learning representations for graphs

DEEP GEOMETRICAL GRAPH CLASSIFICATION


Deep neural networks (DNNs) have been found to be vulnerable to adversarial examples resulting from adding small-magnitude perturbations to inputs.

Such adversarial examples can mislead DNNs to produce adversary-selected results.

Different attack strategies have been proposed to generate adversarial examples, but how to produce them with high perceptual quality and more efficiently requires more research efforts.

In this paper, we propose AdvGAN to generate adversarial examples with generative adversarial networks (GANs), which can learn and approximate the distribution of original instances.

For AdvGAN, once the generator is trained, it can generate adversarial perturbations efficiently for any instance, so as to potentially accelerate adversarial training as defenses.

We apply AdvGAN in both semi-whitebox and black-box attack settings.

In semi-whitebox attacks, there is no need to access the original target model after the generator is trained, in contrast to traditional white-box attacks.

In black-box attacks, we dynamically train a distilled model for the black-box model and optimize the generator accordingly.

Adversarial examples generated by AdvGAN on different target models have high attack success rate under state-of-the-art defenses compared to other attacks.

Our attack  has placed the first with 92.76% accuracy on a public MNIST black-box attack challenge.

We propose to generate adversarial example based on generative adversarial networks in a semi-whitebox and black-box settings.

Describes AdvGAN, a conditional GAN plus adversarial loss, and evaluates AdvGAN on semi-white box and black box setting, reporting state-of-art results.

This paper proposes a way of generating adversarial examples that fool classification systems and wins MadryLab's mnist challenge.

Generating Adversarial Examples with Adversarial Networks

generate adversarial example
setting
network


This paper proposes a new model for the rating prediction task in recommender systems which significantly outperforms previous state-of-the art models on a time-split Netflix data set.

Our model is based on deep autoencoder with 6 layers and is trained end-to-end without any layer-wise pre-training.

a) deep autoencoder models generalize much better than the shallow ones,

b) non-linear activation functions with negative parts are crucial for training deep models, and

c) heavy use of regularization techniques such as dropout is necessary to prevent over-fitting.

We also propose a new training algorithm based on iterative output re-feeding to overcome natural sparseness of collaborate filtering.

The new algorithm significantly speeds up training and improves model performance.

This paper demonstrates how to train deep autoencoders end-to-end to achieve SoA results on time-split Netflix data set.

This paper presents a deep autoencoder model for rating prediction that outperforms other state-of-the-art approahces on the Netflix prize dataset.

Proposes to use a deep AE to do rating prediction tasks in recommender systems.

The authors present a model for more accurate Netflix recommendations demonstrating that a deep autoencoder can out-perform more complex RNN-based models that have temporal information.

Training Deep AutoEncoders for Recommender Systems

train deep autoencoder
rating prediction
demonstrate
recommender system
model
present
netflix


Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks.

However, their inherently sequential computation makes them slow to train.

Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times.

Despite these successes, however, popular feed-forward sequence models like the Transformer fail to generalize in many simple tasks that recurrent models handle with ease, e.g. copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time.

We propose the Universal Transformer (UT), a parallel-in-time self-attentive recurrent sequence model which can be cast as a generalization of the Transformer model and which addresses these issues.

UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs.

We also add a dynamic per-position halting mechanism and find that it improves accuracy on several tasks.

In contrast to the standard Transformer, under certain assumptions UTs can be shown to be Turing-complete.

Our experiments show that UTs outperform standard Transformers on a wide range of algorithmic and language understanding tasks, including the challenging LAMBADA language modeling task where UTs achieve a new state of the art, and machine translation where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De dataset.

We introduce the Universal Transformer, a self-attentive parallel-in-time recurrent sequence model that outperforms Transformers and LSTMs on a wide range of sequence-to-sequence tasks, including machine translation.

Proposes a new model UT, based on the Transformer model, with added recurrence and dynamic halting of the recurrence.

This paper extends Transformer by recursively applying a multi-head self-attention block, rather than stacking multiple blocks in the vanilla Transformer.

Universal Transformers


We present a framework for interpretable continual learning (ICL).

We show that explanations of previously performed tasks can be used to improve performance on future tasks.

ICL generates a good explanation of a finished task, then uses this to focus attention on what is important when facing a new task.

The ICL idea is general and may be applied to many continual learning approaches.

Here we focus on the variational continual learning framework to take advantage of its flexibility and efficacy in overcoming catastrophic forgetting.

We use saliency maps to provide explanations of performed tasks and propose a new metric to assess their quality.

Experiments show that ICL achieves state-of-the-art results in terms of overall continual learning performance as measured by average classification accuracy, and also in terms of its explanations, which are assessed qualitatively and quantitatively using the proposed metric.

The paper develops an interpretable continual learning framework where explanations of the finished tasks are used to enhance the attention of the learner during the future tasks, and where an explanation metric is proposed too.

The authors propose a framework for continual learning based on explanations for performed classifications of previously learned tasks

This paper proposes an extension to the continual learning framework using existing variational continual learning as the base method with weight of evidence.

Interpretable Continual Learning

continual learning base
explanation
framework
task
interpretable


The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikevicius et al. (2017).

On the other hand, while a lot of research has also happened in the domain of low and mixed-precision Integer training, these works either present results for non-SOTA networks (for instance only AlexNet for ImageNet-1K), or relatively small datasets (like CIFAR-10).

In this work, we train state-of-the-art visual understanding neural networks on the ImageNet-1K dataset, with Integer operations on General Purpose (GP) hardware.

In particular, we focus on Integer Fused-Multiply-and-Accumulate (FMA) operations which take two pairs of INT16 operands and accumulate results into an INT32 output.We propose a shared exponent representation of tensors and develop a Dynamic Fixed Point (DFP) scheme suitable for common neural network operations.

The nuances of developing an efficient integer convolution kernel is examined, including methods to handle overflow of the INT32 accumulator.

We implement CNN training for ResNet-50, GoogLeNet-v1, VGG-16 and AlexNet; and these networks achieve or exceed SOTA accuracy within the same number of iterations as their FP32 counterparts without any change in hyper-parameters and with a 1.8X improvement in end-to-end training throughput.

To the best of our knowledge these results represent the first INT16 training results on GP hardware for ImageNet-1K dataset using SOTA CNNs and achieve highest reported accuracy using half precision

Mixed precision training pipeline using 16-bit integers on general purpose HW;  SOTA accuracy for ImageNet-class CNNs; Best reported accuracy for ImageNet-1K classification task with any reduced precision training;

This paper shows that a careful implementation of mixed-precision dynamic fixed point computation can achieve state of the art accuracy using a reduced precision deep learning model with a 16 bit integer representation

Proposes a "dynamic fixed point" scheme that shares the exponent part for a tensor and develops procedures to do NN computing with this format and demonstrates this for limited precision training.

Mixed Precision Training of Convolutional Neural Networks using Integer Operations

reduce precision
dynamic fix point
accuracy
integer
training
mix


In this paper we introduce a new speech recognition system, leveraging a simple letter-based ConvNet acoustic model.

The acoustic model requires only audio transcription for training -- no alignment annotations, nor any forced alignment step is needed. At inference, our decoder takes only a word list and a language model, and is fed with letter scores from the acoustic model -- no phonetic word lexicon is needed.

Key ingredients for the acoustic model are Gated Linear Units and high dropout.

We show near state-of-the-art results in word error rate on the LibriSpeech corpus with MFSC features, both on the clean and other configurations.

A letter-based ConvNet acoustic model leads to a simple and competitive speech recognition pipeline.

This paper applies gated convolutional neural networks to speech recognition, using the training criterion ASG.

Gated ConvNets for Letter-Based ASR

speech recognition
letter-based
convnet
gate


Generative adversarial networks (GANs) are a powerful framework for generative tasks.

However, they are difficult to train and tend to miss modes of the true data generation process.

Although GANs can learn a rich representation of the covered modes of the data in their latent space, the framework misses an inverse mapping from data to this latent space.

We propose Invariant Encoding Generative Adversarial Networks (IVE-GANs), a novel GAN framework that introduces such a mapping for individual samples from the data by utilizing features in the data which are invariant to certain transformations.

Since the model maps individual samples to the latent space, it naturally encourages the generator to cover all modes.

We demonstrate the effectiveness of our approach in terms of generative performance and learning rich representations on several datasets including common benchmark image generation tasks.

A noval GAN framework that utilizes transformation-invariant features to learn rich representations and strong generators.

Proposes a modified GAN objective consisting of a classic GAN term and an invariant encoding term.

This paper presents the IVE-GAN, a model that introduces en encoder to the Generative Adversarial Network framework.

IVE-GAN: Invariant Encoding Generative Adversarial Networks

invariant encoding
generative adversarial network
framework
ive-gan
gan


We propose a method for learning the dependency structure between latent variables in deep latent variable models.

Our general modeling and inference framework combines the complementary strengths of deep generative models and probabilistic graphical models.

In particular, we express the latent variable space of a variational autoencoder (VAE) in terms of a Bayesian network with a learned, flexible dependency structure.

The network parameters, variational parameters as well as the latent topology are optimized simultaneously with a single objective.

Inference is formulated via a sampling procedure that produces expectations over latent variable structures and incorporates top-down and bottom-up reasoning over latent variable values.

We validate our framework in extensive experiments on MNIST, Omniglot, and CIFAR-10.

Comparisons to state-of-the-art structured variational autoencoder baselines show improvements in terms of the expressiveness of the learned model.

We propose a method for learning latent dependency structure in variational autoencoders.

Uses a matrix of binary random variables to capture dependencies between latent variables in a hierarchical deep generative model.

This paper presents a VAE approach in which a dependency structure on the latent variable is learned during training.

The authors propose to augment the latent space of a VAE with an auto-regressive structure, to improve the expressiveness of both the inference network and the latent prior

Variational Autoencoders with Jointly Optimized Latent Dependency Structure

latent variable
variational autoencoder
dependency structure
learn
vae


Many real-world time series, such as in activity recognition, finance, or climate science, have changepoints where the system's structure or parameters change.

Detecting changes is important as they may indicate critical events.

However, existing methods for changepoint detection face challenges when (1) the patterns of change cannot be modeled using simple and predefined metrics, and (2) changes can occur gradually, at multiple time-scales.

To address this, we show how changepoint detection can be treated as a supervised learning problem, and propose a new deep neural network architecture that can efficiently identify both abrupt and gradual changes at multiple scales.

Our proposed method, pyramid recurrent neural network (PRNN), is designed to be scale-invariant, by incorporating wavelets and pyramid analysis techniques from multi-scale signal processing.

Through experiments on synthetic and real-world datasets, we show that PRNN can detect abrupt and gradual changes with higher accuracy than the state of the art and can extrapolate to detect changepoints at novel timescales that have not been seen in training.

We introduce a scale-invariant neural network architecture for changepoint detection in multivariate time series.

The paper leverages the concept of wavelet transform within a deep architecture to solve change point detection.

This paper proposes a pyramid based neural net and applies it to 1D signals with underlying processes occurring at different time scales where the task is change point detection

Pyramid Recurrent Neural Networks for Multi-Scale Change-Point Detection

change point detection
architecture
neural network
time
pyramid


We demonstrate how to learn efficient heuristics for automated reasoning algorithms through deep reinforcement learning.

We focus on backtracking search algorithms for quantified Boolean logics, which already can solve formulas of impressive size - up to 100s of thousands of variables.

The main challenge is to find a representation of these formulas that lends itself to making predictions in a scalable way.

For challenging problems, the heuristic learned through our approach reduces execution time by a factor of 10 compared to the existing handwritten heuristics.

RL finds better heuristics for automated reasoning algorithms.

Aims to learn a heuristic for a backtracking search algorithm utilizing Reinforcement learning and proposes a model that makes use of Graphical Neural Networks to produce literal and clauses embedding, and use them to predict the quality of each literal to decide the probability of each action.

The paper proposes an approach to automatically learning variable selection heuristics for QBF using deep learning

Learning Heuristics for Automated Reasoning through Reinforcement Learning

reinforcement learning
learn heuristic
reasoning
algorithm


We consider the question of how to assess generative adversarial networks, in particular with respect to whether or not they generalise beyond memorising the training data.

We propose a simple procedure for assessing generative adversarial network performance based on a principled consideration of what the actual goal of generalisation is.

Our approach involves using a test set to estimate the Wasserstein distance between the generative distribution produced by our procedure, and the underlying data distribution.

We use this procedure to assess the performance of several modern generative adversarial network architectures.

We find that this procedure is sensitive to the choice of ground metric on the underlying data space, and suggest a choice of ground metric that substantially improves performance.

We finally suggest that attending to the ground metric used in Wasserstein generative adversarial network training may be fruitful, and outline a concrete pathway towards doing so.

Assess whether or not your GAN is actually doing something other than memorizing the training data.

Aims to provide a quality measure/test for GANs and proposes to evaluate the current approximation of a distribution learnt by a GAN by using Wasserstein distance between two distributions made of a sum of Diracs as a baseline performance.

This paper proposed a procedure for assessing the performance of GANs by re-considering the key of observation, using the procedure to test and improve the current GANs

Towards a Testable Notion of Generalization for Generative Adversarial Networks


The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance.

Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU).

Although various hand-designed alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains.

In this work, we propose to leverage automatic search techniques to discover new activation functions.

Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions.

We verify the effectiveness of the searches by conducting an empirical evaluation with the best discovered activation function.

Our experiments show that the best discovered activation function, f(x) = x * sigmoid(beta * x), which we name Swish, tends to work better than ReLU on deeper models across a number of challenging datasets.

For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2.

The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.

We use search techniques to discover novel activation functions, and our best discovered activation function, f(x) = x * sigmoid(beta * x), outperforms ReLU on a number of challenging tasks like ImageNet.

Proposes a reinforcement learning based approach for finding non-linearity by searching through combinations from a set of unary and binary operators.

This paper utilizes reinforcement learning to search the combination of a set of unary and binary functions resulting in a new activation function

The author uses reinforcement learning to find new potential activation functions from a rich set of possible candidates.

Searching for Activation Functions

search combination set unary
activation function
reinforcement learning
find
binary
new


Successful training of convolutional neural networks is often associated with suffi-

ciently deep architectures composed of high amounts of features.

typically rely on a variety of regularization and pruning techniques to converge

We introduce a novel bottom-up approach to expand

representations in fixed-depth architectures.

single feature per layer and greedily increase width of individual layers to attain

effective representational capacities needed for a specific task.

growth can rely on a family of metrics, we propose a computationally efficient

version based on feature time evolution and demonstrate its potency in determin-

ing feature importance and a networks’ effective capacity.

automatically expanded architectures converge to similar topologies that benefit

from lesser amount of parameters or improved accuracy and exhibit systematic

correspondence in representational complexity with the specified task.

to conventional design patterns with a typical monotonic increase in the amount of

features with increased depth, we observe that CNNs perform better when there is

more learnable parameters in intermediate, with falloffs to earlier and later layers.

A bottom-up algorithm that expands CNNs starting with one feature per layer to architectures with sufficient representational capacity.

Proposes to dynamically adjust the feature map depth of a fully convolutional neural network, formulating a measure of self-resemblance and boosting performance.

Introduces a simple correlation-based metric to measure whether filters in neural networks are being used effectively, as a proxy for effective capacity.

Aims to address the deep learning architecture search problem via incremental addition and removal of channels in intermediate layers of the network.

Building effective deep neural networks one feature at a time

network
one feature
layer
capacity
measure
architecture
neural
effective
deep


Deep neural networks are almost universally trained with reverse-mode automatic differentiation (a.k.a. backpropagation).

Biological networks, on the other hand, appear to lack any mechanism for sending gradients back to their input neurons, and thus cannot be learning in this way.

In response to this, Scellier & Bengio (2017) proposed Equilibrium Propagation - a method for gradient-based train- ing of neural networks which uses only local learning rules and, crucially, does not rely on neurons having a mechanism for back-propagating an error gradient.

Equilibrium propagation, however, has a major practical limitation: inference involves doing an iterative optimization of neural activations to find a fixed-point, and the number of steps required to closely approximate this fixed point scales poorly with the depth of the network.

In response to this problem, we propose Initialized Equilibrium Propagation, which trains a feedforward network to initialize the iterative inference procedure for Equilibrium propagation.

This feed-forward network learns to approximate the state of the fixed-point using a local learning rule.

After training, we can simply use this initializing network for inference, resulting in a learned feedforward network.

Our experiments show that this network appears to work as well or better than the original version of Equilibrium propagation.

This shows how we might go about training deep networks without using backpropagation.

We train a feedforward network without backprop by using an energy-based model to provide local targets

This paper aims at quickening the iterative inference procedure in energy-based models trained with Equilibrium Propagation (EP), by proposing to train a feedforward network to predict a fixed point of the "equilibrating network".

Training a separate network to initialize recurrent networks trained using equilibrium propagation

Initialized Equilibrium Propagation for Backprop-Free Training

train feedforward network
energy-based model
equilibrium propagation
initialize


We propose a novel generative model architecture designed to learn representations for images that factor out a single attribute from the rest of the representation.

A single object may have many attributes which when altered do not change the identity of the object itself.

Consider the human face; the identity of a particular person is independent of whether or not they happen to be wearing glasses.

The attribute of wearing glasses can be changed without changing the identity of the person.

However, the ability to manipulate and alter image attributes without altering the object identity is not a trivial task.

Here, we are interested in learning a representation of the image that separates the identity of an object (such as a human face) from an attribute (such as 'wearing glasses').

We demonstrate the success of our factorization approach by using the learned representation to synthesize the same face with and without a chosen attribute.

We refer to this specific synthesis process as image attribute manipulation.

We further demonstrate that our model achieves competitive scores, with state of the art, on a facial attribute classification task.

Learn representations for images that factor out a single attribute.

This paper builds on Conditional VAE GANs to allow attribute manipulation in the synthesis process.

This paper proposes a generative model to learn the representation which can separate the identity of an object from an attribute, and extends the autoencoder adversarial by adding an auxiliary network.

Adversarial Information Factorization

learn representation
attribute
adversarial


Stochastic video prediction models take in a sequence of image frames, and generate a sequence of consecutive future image frames.

These models typically generate future frames in an autoregressive fashion, which is slow and requires the input and output frames to be consecutive.

We introduce a model that overcomes these drawbacks by generating a latent representation from an arbitrary set of frames that can then be used to simultaneously and efficiently sample temporally consistent frames at arbitrary time-points.

For example, our model can "jump" and directly sample frames at the end of the video, without sampling intermediate frames.

Synthetic video evaluations confirm substantial gains in speed and functionality without loss in fidelity.

We also apply our framework to a 3D scene reconstruction dataset.

Here, our model is conditioned on camera location and can sample consistent sets of images for what an occluded region of a 3D scene might look like, even if there are multiple possibilities for what that region might contain.

Reconstructions and videos are available at https://bit.ly/2O4Pc4R.

We present a model for consistent 3D reconstruction and jumpy video prediction e.g. producing image frames multiple time-steps in the future without generating intermediate frames.

This paper proposes a general method for indexed data modeling by encoding index information together with observation into a neural network, and then decode the observation condition on the target index.

Proposes using a VAE that encodes input video in a permuation invariant way to predict future frames of a video.

Consistent Jumpy Predictions for Videos and Scenes

video frame model encode jumpy future prediction consistent


The ADAM optimizer is exceedingly popular in the deep learning community.

Often it works very well, sometimes it doesn’t.

We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of the stochastic gradient, whereas the update magnitude is solely determined by an estimate of its relative variance.

We  disentangle these two aspects and analyze them in isolation, shedding light on ADAM ’s inner workings.

Transferring the "variance adaptation” to momentum- SGD gives rise to a novel method, completing the practitioner’s toolbox for problems where ADAM fails.

The paper trys to improve Adam based on variance adaption with momentum by proposing two algorithms

This paper analyzes the scale-invariance and the particular shape of the learning rate used in Adam, arguing that Adam's update is a combination of a sign-update and a variance-based learning rate.

The paper splits ADAM algorithm into two components: stochastic direction in sign of gradient and adaptive stepwise with relative variance, and two algorithms are proposed to test each of them.

Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients

two algorithm
adam
sign
variance
analyze
stochastic
gradient


We propose a novel framework to adaptively adjust the dropout rates for the deep neural network based on a Rademacher complexity bound.

The state-of-the-art deep learning algorithms impose dropout strategy to prevent feature co-adaptation.

However, choosing the dropout rates remains an art of heuristics or relies on empirical grid-search over some hyperparameter space.

In this work, we show the network Rademacher complexity is bounded by a function related to the dropout rate vectors and the weight coefficient matrices.

Subsequently, we impose this bound as a regularizer and provide a theoretical justified way to trade-off between model complexity and representation power.

Therefore, the dropout rates and the empirical loss are unified into the same objective function, which is then optimized using the block coordinate descent algorithm.

We discover that the adaptively adjusted dropout rates converge to some interesting distributions that reveal meaningful patterns.Experiments on the task of image and document classification also show our method achieves better performance compared to the state-of the-art dropout algorithms.

The authors connect dropout parameters to a bound of the Rademacher complexity of the network

Relates complexity of networks' learnability to dropout rates in backpropagation.

Adaptive Dropout with Rademacher Complexity Regularization

dropout rate
rademacher complexity
network
bind


Sensor fusion is a key technology that integrates various sensory inputs to allow for robust decision making in many applications such as autonomous driving and robot control.

Deep neural networks have been adopted for sensor fusion in a body of recent studies.

Among these, the so-called netgated architecture was proposed, which has demonstrated improved performances over the conventional convolu- tional neural networks (CNN).

In this paper, we address several limitations of the baseline negated architecture by proposing two further optimized architectures: a coarser-grained gated architecture employing (feature) group-level fusion weights and a two-stage gated architectures leveraging both the group-level and feature- level fusion weights.

Using driving mode prediction and human activity recogni- tion datasets, we demonstrate the significant performance improvements brought by the proposed gated architectures and also their robustness in the presence of sensor noise and failures.

Optimized gated deep learning architectures for sensor fusion is proposed.

The authors improve upon several limitations of the baseline negated architecture by proposing a coarser-grained gated fusion architecture and a two-stage gated fusion architecture

Proposes two gated deep learning architectures for sensor fusion and by having the grouped features, demonstrates improved performance, especially in the presence of random sensor noise and failures.

Optimized Gated Deep Learning Architectures for Sensor Fusion

gate deep learning architecture sensor fusion


We develop a mean field theory for batch normalization in fully-connected feedforward neural networks.

In so doing, we provide a precise characterization of signal propagation and gradient backpropagation in wide batch-normalized networks at initialization.

Our theory shows that gradient signals grow exponentially in depth and that these exploding gradients cannot be eliminated by tuning the initial weight variances or by adjusting the nonlinear activation function.

Indeed, batch normalization itself is the cause of gradient explosion.

As a result, vanilla batch-normalized networks without skip connections are not trainable at large depths for common initialization schemes, a prediction that we verify with a variety of empirical simulations.

While gradient explosion cannot be eliminated, it can be reduced by tuning the network close to the linear regime, which improves the trainability of deep batch-normalized networks without residual connections.

Finally, we investigate the learning dynamics of batch-normalized networks and observe that after a single step of optimization the networks achieve a relatively stable equilibrium in which gradients have dramatically smaller dynamic range.

Our theory leverages Laplace, Fourier, and Gegenbauer transforms and we derive new identities that may be of independent interest.

Batch normalization causes exploding gradients in vanilla feedforward networks.

Develops a mean field theory for batch normalization (BN) in fully-connected networks with randomly initialized weights.

Provides a dynamic perspective on deep neural network using the evolution of the covariance matrix along with the layers.

A Mean Field Theory of Batch Normalization

mean field theory batch normalization
network


We present NeuroSAT, a message passing neural network that learns to solve SAT problems after only being trained as a classifier to predict satisfiability.

Although it is not competitive with state-of-the-art SAT solvers, NeuroSAT can solve problems that are substantially larger and more difficult than it ever saw during training by simply running for more iterations.

Moreover, NeuroSAT generalizes to novel distributions; after training only on random SAT problems, at test time it can solve SAT problems encoding graph coloring, clique detection, dominating set, and vertex cover problems, all on a range of distributions over small random graphs.

We train a graph network to predict boolean satisfiability and show that it learns to search for solutions, and that the solutions it finds can be decoded from its activations.

The paper describes a general neural network architecture for predicting satisfiability

This paper presents the NeuroSAT architecture which uses a deep message passing neural net for predicting the satisfiability of CNF instances

Learning a SAT Solver from Single-Bit Supervision

predict satisfiability
learn
architecture
network
neural


Spatiotemporal forecasting has various applications in neuroscience, climate and transportation domain.

Traffic forecasting is one canonical example of such learning task.

The task is challenging due to (1) complex spatial dependency on road networks, (2) non-linear temporal dynamics with changing road conditions and (3) inherent difficulty of long-term forecasting.

To address these challenges, we propose to model the traffic flow as a diffusion process on a directed graph and introduce Diffusion Convolutional Recurrent Neural Network (DCRNN), a deep learning framework for traffic forecasting that incorporates both spatial and temporal dependency in the traffic flow.

Specifically, DCRNN captures the spatial dependency using bidirectional random walks on the graph, and the temporal dependency using the encoder-decoder architecture with scheduled sampling.

We evaluate the framework on two real-world large-scale road network traffic datasets and observe consistent improvement of 12% - 15% over state-of-the-art baselines

A neural sequence model that learns to forecast on a directed graph.

The paper proposes the Diffusion Convolutional Recurrent Neural Network architecture for the spatiotemporal traffic forecasting problem

Proposes to build a traffic forecasting model using a diffusion process for convolutional recurrent neural networks to address saptio-temporal autocorrelation.

Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting

diffusion convolutional recurrent neural network


Obtaining reliable uncertainty estimates of neural network predictions is a long standing challenge.

Bayesian neural networks have been proposed as a solution, but it remains open how to specify their prior.

In particular, the common practice of a standard normal prior in weight space imposes only weak regularities, causing the function posterior to possibly generalize in unforeseen ways on inputs outside of the training distribution.

We propose noise contrastive priors (NCPs) to obtain reliable uncertainty estimates.

The key idea is to train the model to output high uncertainty for data points outside of the training distribution.

NCPs do so using an input prior, which adds noise to the inputs of the current mini batch, and an output prior, which is a wide distribution given these inputs.

NCPs are compatible with any model that can output uncertainty estimates, are easy to scale, and yield reliable uncertainty estimates throughout training.

Empirically, we show that NCPs prevent overfitting outside of the training distribution and result in uncertainty estimates that are useful for active learning.

We demonstrate the scalability of our method on the flight delays data set, where we significantly improve upon previously published results.

We train neural networks to be uncertain on noisy inputs to avoid overconfident predictions outside of the training distribution.

Presents an approach to obtain uncertainty estimates for neural network predictions that has good performance when quantifying predictive uncertainty at points that are outside of the training distribution.

The paper considers the problem of uncertainty estimation of neural networks and proposes to use Bayesian approach with noice contrastive prior

Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors

outside training distribution
neural network
uncertainty estimate
approach
contrastive prior
prediction


Convolutional neural networks (CNNs) were inspired by human vision and, in some settings, achieve a performance comparable to human object recognition.

This has lead to the speculation that both systems use similar mechanisms to perform recognition.

In this study, we conducted a series of simulations that indicate that there is a fundamental difference between human vision and CNNs: while object recognition in humans relies on analysing shape, CNNs do not have such a shape-bias.

We teased apart the type of features selected by the model by modifying the CIFAR-10 dataset so that, in addition to containing objects with shape, the images concurrently contained non-shape features, such as a noise-like mask.

When trained on these modified set of images, the model did not show any bias towards selecting shapes as features.

Instead it relied on whichever feature allowed it to perform the best prediction -- even when this feature was a noise-like mask or a single predictive pixel amongst 50176 pixels.

We also found that regularisation methods, such as batch normalisation or Dropout, did not change this behaviour and neither did past or concurrent experience with images from other datasets.

This study highlights a key difference between human vision and CNNs: while object recognition in humans relies on analysing shape, CNNs do not have such a shape-bias.

Seeks to establish via a series of well-designed experiments that CNNs trained for image classification don’t encode shape-bias like human vision.

This paper highlights the fact that CNNs will not necessarily learn to recognize objects based on their shape and shows they will overfeat to noise based features.

What a difference a pixel makes: An empirical examination of features used by CNNs for categorisation

cnn
human vision
difference
shape
highlight
feature
object
shape-bias


The development of high-dimensional generative models has recently gained a great surge of interest with the introduction of variational auto-encoders and generative adversarial neural networks.

Different variants have been proposed where the underlying latent space is structured, for example, based on attributes describing the data to generate.

We focus on a particular problem where one aims at generating samples corresponding to a number of objects under various views.

We assume that the distribution of the data is driven by two independent latent factors: the content, which represents the intrinsic features of an object, and the view, which stands for the settings of a particular observation of that object.

Therefore, we propose a generative model and a conditional variant built on such a disentangled latent space.

This approach allows us to generate realistic samples corresponding to various objects in a high variety of views.

Unlike many multi-view approaches, our model doesn't need any supervision on the views but only on the content.

Compared to other conditional generation approaches that are mostly based on binary or categorical attributes, we make no such assumption about the factors of variations.

Our model can be used on problems with a huge, potentially infinite, number of categories.

We experiment it on four images datasets on which we demonstrate the effectiveness of the model and its ability to generalize.

We describe a novel multi-view generative model that can generate multiple views of the same object, or multiple objects in the same view with no need of label on views.

This paper presents a GAN-based method for image generation that attempts to separate latent variables describing image content from those describing properties of view.

This paper proposes a GAN architecture that aims at decomposing the underlying distribution of a particular class into "content" and "view".

Proposes a new generative model based on the Generative Adversarial Network (GAN) that disentangles the content and the view of objects without view supervision and extends GMV into a conditional generative model that takes an input image and generates different views of the object in the input image.

Multi-View Data Generation Without View Supervision

view object
gan
describe
generative model
content
generate
image
multi-view
generation
without
supervision


The huge size of deep networks hinders their use in small computing devices.

In this paper, we consider compressing the network by weight quantization.

We extend a recently proposed loss-aware weight binarization scheme to ternarization, with possibly different scaling parameters for the positive and negative weights, and m-bit (where m > 2) quantization.

Experiments on feedforward and recurrent neural networks show that the proposed scheme outperforms state-of-the-art weight quantization algorithms, and is as accurate (or even more accurate) than the full-precision network.

A loss-aware weight quantization algorithm that directly considers its effect on the loss is proposed.

Proposes a method of compressing network by means of weight ternarization.

The paper proposes a new method to train DNNs with quantized weights, by including the quantization as a constraint in a proximal quasi-Newton algorithm, which simultaneously learns a scaling for the quantized values.

The paper extends the loss-aware weight binarization scheme to terarization and arbitrary m-bit quantization and demonstrate its promising performance.

Loss-aware Weight Quantization of Deep Networks

loss-aware weight quantization
algorithm
method
network


In the pursuit of increasingly intelligent learning systems, abstraction plays a vital role in enabling sophisticated decisions to be made in complex environments.

The options framework provides formalism for such abstraction over sequences of decisions.

However most models require that options be given a priori, presumably specified by hand, which is neither efficient, nor scalable.

Indeed, it is preferable to learn options directly from interaction with the environment.

Despite several efforts, this remains a difficult problem: many approaches require access to a model of the environmental dynamics, and inferred options are often not interpretable, which limits our ability to explain the system behavior for verification or debugging purposes.

In this work we develop a novel policy gradient method for the automatic learning of policies with options.

This algorithm uses inference methods to simultaneously improve all of the options available to an agent, and thus can be employed in an off-policy manner, without observing option labels.

Experimental results show that the options learned can be interpreted.

Further, we find that the method presented here is more sample efficient than existing methods, leading to faster and more stable learning of policies with options.

We develop a novel policy gradient method for the automatic learning of policies with options using a differentiable inference step.

The paper presents a new policy gradient technique for learning options, where a single sample can be used to update all options.

Proposes an off-policy method for learning options in complex continuous problems.

An inference-based policy gradient method for learning options


The paper, interested in unsupervised feature selection, aims to retain the features best accounting for the local patterns in the data.

The proposed approach, called Locally Linear Unsupervised Feature Selection, relies on a dimensionality reduction method to characterize such patterns; each feature is thereafter assessed according to its compliance w.r.t. the local patterns, taking inspiration from Locally Linear Embedding (Roweis and Saul, 2000).

The experimental validation of the approach on the scikit-feature benchmark suite demonstrates its effectiveness compared to the state of the art.

Unsupervised feature selection through capturing the local linear structure of the data

Proposes locally linear unsupervised feature selection.

The paper proposes the LLUFS method for feature selection.


Humans can understand and produce new utterances effortlessly, thanks to their systematic compositional skills.

Once a person learns the meaning of a new verb "dax," he or she can immediately understand the meaning of "dax twice" or "sing and dax."

In this paper, we introduce the SCAN domain, consisting of a set of simple compositional navigation commands paired with the corresponding action sequences.

We then test the zero-shot generalization capabilities of a variety of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence methods.

We find that RNNs can generalize well when the differences between training and test commands are small, so that they can apply "mix-and-match" strategies to solve the task.

However, when generalization requires systematic compositional skills (as in the "dax" example above), RNNs fail spectacularly.

We conclude with a proof-of-concept experiment in neural machine translation, supporting the conjecture that lack of systematicity is an important factor explaining why neural networks need very large training sets.

Using a simple language-driven navigation task, we study the compositional capabilities of modern seq2seq recurrent networks.

This paper focuses on the zero-shot learning compositional capabilities of modern sequence-to-sequence RNNs and  exposes the short-comings of current seq2seq RNN architectures.

The paper analyzes the composition abilities of RNNs, specifically, the generalization ability of RNNs on random subset of SCAN commands, on longer SCAN commands, and of composition over primitive commands.

The authors introduce a new dataset that facilitates the analysis of a Seq2Seq learning case

Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks

compositional capability modern
recurrent network
rnn
seq2seq
sequence-to-sequence


This paper addresses the challenging problem of retrieval and matching of graph structured objects, and makes two key contributions.

First, we demonstrate how  Graph Neural Networks (GNN), which have emerged as an effective model for various supervised prediction problems defined on structured data, can be trained to produce embedding of graphs in vector spaces that enables efficient similarity reasoning.

Second, we propose a novel Graph Matching Network model that, given a pair of graphs as input, computes a similarity score between them by jointly reasoning on the pair through a new cross-graph attention-based matching mechanism.

We demonstrate the effectiveness of our models on different domains including the challenging problem of control-flow-graph based function similarity search that plays an important role in the detection of vulnerabilities in software systems.

The experimental analysis demonstrates that our models are not only able to exploit structure in the context of similarity learning but they can also outperform domain-specific baseline systems that have been carefully hand-engineered for these problems.

We tackle the problem of similarity learning for structured objects with applications in particular in computer security, and propose a new model graph matching networks that excels on this task.

Authors introduce a Graph Matching Network for retrival and matching of graph structured objects.

The authors attack the problem of graph matching by proposing an extension of graph embedding networks

The authors present two methods for learning a similarity score between pairs of graphs and show the beneficiality of introducing idesa from graph matching to graph neural networks.

Graph Matching Networks for Learning the Similarity of Graph Structured Objects

graph matching network
learn similarity
structured object
problem
introduce
author
match


Context information plays an important role in human language understanding, and it is also useful for machines to learn vector representations of language.

In this paper, we explore an asymmetric encoder-decoder structure for unsupervised context-based sentence representation learning.

As a result, we build an encoder-decoder architecture with an RNN encoder and a CNN decoder, and we show that neither an autoregressive decoder nor an RNN decoder is required.

We further combine a suite of effective designs to significantly improve model efficiency while also achieving better performance.

Our model is trained on two different large unlabeled corpora, and in both cases transferability is evaluated on a set of downstream language understanding tasks.

We empirically show that our model is simple and fast while producing rich sentence representations that excel in downstream tasks.

We proposed an RNN-CNN encoder-decoder model for fast unsupervised sentence representation learning.

Modifications to the skip-thought framework for learning sentence embeddings.

This paper presents a new RNN encoder–CNN decoder hybrid design for use in pretraining, which does not require an autoregressive decoder when pretraining encoders.

The authors extend Skip-thought by decoding only one target sentence using a CNN decoder.

Exploring Asymmetric Encoder-Decoder Structure for Context-based Sentence Representation Learning

sentence representation learning
skip-thought
decoder
encoder-decoder


Building on the success of deep learning, two modern approaches to learn a probability model of the observed data are Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs).

VAEs consider an explicit probability model for the data and compute a generative distribution by maximizing a variational lower-bound on the log-likelihood function.

GANs, however, compute a generative model by minimizing a distance between observed and generated probability distributions without considering an explicit model for the observed data.

The lack of having explicit probability models in GANs prohibits computation of sample likelihoods in their frameworks and limits their use in statistical inference problems.

In this work, we show that an optimal transport GAN with the entropy regularization can be viewed as a generative model that maximizes a lower-bound on average sample likelihoods, an approach that VAEs are based on.

In particular, our proof constructs an explicit probability model for GANs that can be used to compute likelihood statistics within GAN's framework.

Our numerical results on several datasets demonstrate consistent trends with the proposed theory.

A statistical approach to compute sample likelihoods in Generative Adversarial Networks

Show that WGAN with entropic regularization maximizes a lower bound on the likelihood of the observed data distribution.

Authors claim it is possible to leverage the upper bound from an entropy regularized optimal transport to come up with a measure of 'sample likelihood'.

Entropic GANs meet VAEs: A Statistical Approach to Compute Sample Likelihoods in GANs

statistical approach compute sample likelihood


We introduce geomstats, a Python package for Riemannian modelization and optimization over manifolds such as hyperspheres, hyperbolic spaces, SPD matrices or Lie groups of transformations.

First, geomstats allows the flexible modeling of many a machine learning problem through an efficient and extensively unit-tested implementations of these manifolds, as well as the set of useful Riemannian metrics, exponential and logarithm maps that we provide.

Moreover, the wide choice of loss functions and our implementation of the corresponding gradients allow fast and easy optimization over manifolds.

Finally, geomstats is the only package to provide a unified framework for Riemannian geometry, as the operations implemented in geomstats are available with different computing backends (numpy,tensorflow and keras), as well as with a GPU-enabled mode–-thus considerably facilitating the application of Riemannian geometry in machine learning.

In this paper, we present geomstats through a review of the utility and advantages of manifolds in machine learning, using the concrete examples that they span to show the efficiency and practicality of their implementation using our package

We introduce geomstats, an efficient Python package for Riemannian modelization and optimization over manifolds compatible with both numpy and tensorflow .

The paper introduces the software package geomstats, which provides simple use of Riemannian manifolds and metrics within machine learning models

Proposes a Python package for optimization and applications on Reimannian manifolds and highlights the differences between Geomstats package and other packages.

Introduces a geometric toolbox, Geomstats, for machine learning on Reimannian manifolds.

Geomstats: a Python Package for Riemannian Geometry in Machine Learning

python package
reimannian manifold
geomstat
optimization
machine learning
introduce
riemannian


We propose to execute deep neural networks (DNNs) with dynamic and sparse graph (DSG) structure for compressive memory and accelerative execution during both training and inference.

The great success of DNNs motivates the pursuing of lightweight models for the deployment onto embedded devices.

However, most of the previous studies optimize for inference while neglect training or even complicate it.

(i) the neurons dominate the memory cost rather than the weights in inference;

(ii) the dynamic activation makes previous sparse acceleration via one-off optimization on fixed weight invalid;

(iii) batch normalization (BN) is critical for maintaining accuracy while its activation reorganization damages the sparsity.

To address these issues, DSG activates only a small amount of neurons with high selectivity at each iteration via a dimensionreduction search and obtains the BN compatibility via a double-mask selection.

Experiments show significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x) with little accuracy loss on various benchmarks.

We construct dynamic sparse graph via dimension-reduction search to reduce compute and memory cost in both DNN training and inference.

The authors propose to use dynamic sparse computation graph for reducing the computation memory and time cost in deep neural network (DNN).

This paper proposes a method to speed up training and inference of deep neural networks using dynamic pruning of the compute graph.

Dynamic Sparse Graph for Efficient Deep Learning

deep neural network
training inference
dnn
dynamic sparse graph
cost
memory
compute
reduce


Efficient exploration remains a major challenge for reinforcement learning.

One reason is that the variability of the returns often depends on the current state and action, and is therefore heteroscedastic.

Classical exploration strategies such as upper confidence bound algorithms and Thompson sampling fail to appropriately account for heteroscedasticity, even in the bandit setting.

Motivated by recent findings that address this issue in bandits, we propose to use Information-Directed Sampling (IDS) for exploration in reinforcement learning.

As our main contribution, we build on recent advances in distributional reinforcement learning and propose a novel, tractable approximation of IDS for deep Q-learning.

The resulting exploration strategy explicitly accounts for both parametric uncertainty and heteroscedastic observation noise.

We evaluate our method on Atari games and demonstrate a significant improvement over alternative approaches.

We develop a practical extension of Information-Directed Sampling for Reinforcement Learning, which accounts for parametric uncertainty and heteroscedasticity in the return distribution for exploration.

The authors propose a way of extending Information-Directed Sampling to reinforcement learning by combining two types of uncertainty to obtain a simple exploration strategy based on IDS.

This paper investigates sophistical exploration approaches for reinforcement learning built on Information Direct Sampling and on Distributional Reinforcement Learning

Information-Directed Exploration for Deep Reinforcement Learning

information-directed sampling reinforcement learning


We address the problem of learning structured policies for continuous control.

In traditional reinforcement learning, policies of agents are learned by MLPs which take the concatenation of all observations from the environment as input for predicting actions.

In this work, we propose NerveNet to explicitly model the structure of an agent, which naturally takes the form of a graph.

Specifically, serving as the agent's policy network, NerveNet first propagates information over the structure of the agent and then predict actions for different parts of the agent.

In the experiments, we first show that our NerveNet is comparable to state-of-the-art methods on standard MuJoCo environments.

We further propose our customized reinforcement learning environments for benchmarking two types of structure transfer learning tasks, i.e., size and disability transfer.

We demonstrate that policies learned by NerveNet are significantly better than policies learned by other models and are able to transfer even in a zero-shot setting.

using graph neural network to model structural information of the agents to improve policy and transferability

A method for representing and learning structured policy for continuous control tasks using Graph Neural Networks

The submission proposes incorporation of additional structure into reinforcement learning problems, particularly the structure of the agent's morphology

Propose an application of Graph Neural Networks to learning policies for controlling "centipede" robots of different lengths.

NerveNet: Learning Structured Policy with Graph Neural Networks

graph neural network
learn structured policy
agent
control


Real-world tasks are often highly structured.

Hierarchical reinforcement learning (HRL) has attracted research interest as an approach for leveraging the hierarchical structure of a given task in reinforcement learning (RL).

However, identifying the hierarchical policy structure that enhances the performance of RL is not a trivial task.

In this paper, we propose an HRL method that learns a latent variable of a hierarchical policy using mutual information maximization.

Our approach can be interpreted as a way to learn a discrete and latent representation of the state-action space.

To learn option policies that correspond to modes of the advantage function, we introduce advantage-weighted importance sampling.

In our HRL method, the gating policy learns to select option policies based on an option-value function, and these option policies are optimized based on the deterministic policy gradient method.

This framework is derived by leveraging the analogy between a monolithic policy in standard RL and a hierarchical policy in HRL by using a deterministic option policy.

Experimental results indicate that our HRL approach can learn a diversity of options and that it can enhance the performance of RL in continuous control tasks.

This paper presents a hierarchical reinforcement learning framework based on deterministic option policies and mutual information maximization.

Proposes an HRL algorithm that attempts to learn options that maximize their mutual information with the state-action density under the optimal policy.

This paper proposes an HRL system in which the mutal information of the latent variable and the state-action pairs is approximately maximized.

Proposes a criterion that aims to maximize the mutual information between options and state-action pairs and show empirically that the learned options decompose the state-action space but not the state space.

Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization

maximize mutual information
learn option
state-action pair
hierarchical reinforcement learning
hrl
policy
maximization


Deep neural networks (DNNs) have achieved impressive predictive performance due to their ability to learn complex, non-linear relationships between variables.

However, the inability to effectively visualize these relationships has led to DNNs being characterized as black boxes and consequently limited their applications.

To ameliorate this problem, we introduce the use of hierarchical interpretations to explain DNN predictions through our proposed method: agglomerative contextual decomposition (ACD).

Given a prediction from a trained DNN, ACD produces a hierarchical clustering of the input features, along with the contribution of each cluster to the final prediction.

This hierarchy is optimized to identify clusters of features that the DNN learned are predictive.

We introduce ACD using examples from Stanford Sentiment Treebank and ImageNet, in order to diagnose incorrect predictions, identify dataset bias, and extract polarizing phrases of varying lengths.

Through human experiments, we demonstrate that ACD enables users both to identify the more accurate of two DNNs and to better trust a DNN's outputs.

We also find that ACD's hierarchy is largely robust to adversarial perturbations, implying that it captures fundamental aspects of the input and ignores spurious noise.

We introduce and validate hierarchical local interpretations, the first technique to automatically search for and display important interactions for individual predictions made by LSTMs and CNNs.

A novel approach to explain neural network predictions by learning hierarchical representations of groups of input features and their contribution to the final prediction

Extends an existing feature interpretation method for LSTMs to more generic DNNs and introduces a hierarchical clustering of the input features and the contributions of each cluster to the final prediction.

This paper proposes a hierarchical extension of contextual decomposition.

Hierarchical interpretations for neural network predictions

input feature contribution
final prediction
lstms
introduce
interpretation
neural network
hierarchical


Principal Filter Analysis (PFA) is an easy to implement, yet effective method for neural network compression.

PFA exploits the intrinsic correlation between filter responses within network layers to recommend a smaller network footprint.

We propose two compression algorithms: the first allows a user to specify the proportion of the original spectral energy that should be preserved in each layer after compression, while the second is a heuristic that leads to a parameter-free approach that automatically selects the compression used at each layer.

Both algorithms are evaluated against several architectures and datasets, and we show considerable compression rates without compromising accuracy, e.g., for VGG-16 on CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x, and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively.

In our tests we also demonstrate that networks compressed with PFA achieve an accuracy that is very close to the empirical upper bound for a given compression ratio.

Finally, we show how PFA is an effective tool for simultaneous compression and domain adaptation.

We propose an easy to implement, yet effective method for neural network compression. PFA exploits the intrinsic correlation between filter responses within network layers to recommend a smaller network footprints.

Proposes to prune convolutional networks by analyzing the observed correlation between the filters of a same layer as expressed by the eigenvalue spectrum of their covariance matrix.

This paper introduces an approach to compressing neural networks by looking at the correlation of filter responses in each layer via two strategies.

This paper proposes a compression method based on spectral analysis

NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES

correlation filter
neural network
layer
compression
response
method
analysis


We propose a method to efficiently learn diverse strategies in reinforcement learning for query reformulation in the tasks of document retrieval and question answering.

In the proposed framework an agent consists of multiple specialized sub-agents and a meta-agent that learns to aggregate the answers from sub-agents to produce a final answer.

Sub-agents are trained on disjoint partitions of the training data, while the meta-agent is trained on the full training set.

Our method makes learning faster, because it is highly parallelizable, and has better generalization performance than strong baselines, such as an ensemble of agents trained on the full data.

We show that the improved performance is due to the increased diversity of reformulation strategies.

Multiple diverse query reformulation agents trained with reinforcement learning to improve search engines.

Parellelization of ensemble method in reinforement learning for query reformulation, speeding up training and improving the diversity of learnt freformulations

The authors propose to train multiple distinct agents, each over a different subset of the training set.

The authors propose an ensemble approach for query reformulation

Learning to Coordinate Multiple Reinforcement Learning Agents for Diverse Query Reformulation

author
diverse query reformulation
training
train
learning
improve
ensemble
agent
reinforcement
multiple


Network Embeddings (NEs) map the nodes of a given network into $d$-dimensional Euclidean space $\mathbb{R}^d$.

Ideally, this mapping is such that 'similar' nodes are mapped onto nearby points, such that the NE can be used for purposes such as link prediction (if 'similar' means being 'more likely to be connected') or classification (if 'similar' means 'being more likely to have the same label').

In recent years various methods for NE have been introduced, all following a similar strategy: defining a notion of similarity between nodes (typically some distance measure within the network), a distance measure in the embedding space, and a loss function that penalizes large distances for similar nodes and small distances for dissimilar nodes.

A difficulty faced by existing methods is that certain networks are fundamentally hard to embed due to their structural properties: (approximate) multipartiteness, certain degree distributions, assortativity, etc.

To overcome this, we introduce a conceptual innovation to the NE literature and propose to create \emph{Conditional Network Embeddings} (CNEs); embeddings that maximally add information with respect to given structural properties (e.g. node degrees, block densities, etc.).

We use a simple Bayesian approach to achieve this, and propose a block stochastic gradient descent algorithm for fitting it efficiently.

We demonstrate that CNEs are superior for link prediction and multi-label classification when compared to state-of-the-art methods, and this without adding significant mathematical or computational complexity.

Finally, we illustrate the potential of CNE for network visualization.

We introduce a network embedding method that accounts for prior information about the network, yielding superior empirical performance.

The paper proposed to use a prior distribution to constraint the network embedding, for the formulation this paper used very restricted Gaussian distributions.

Proposes learning unsupervised node embeddings by considering the structural properties of networks.

Conditional Network Embeddings


This paper studies a class of adaptive gradient based momentum algorithms that update the  search directions and learning rates simultaneously using past gradients.

This class, which we refer to as the ''``Adam-type'', includes the popular algorithms such as Adam, AMSGrad, AdaGrad.

Despite their popularity in training deep neural networks (DNNs), the convergence of these algorithms for solving  non-convex problems remains an open question.

In this paper, we develop an analysis framework and a set of mild sufficient conditions that guarantee the convergence of the Adam-type methods, with a convergence rate of order   $O(\log{T}/\sqrt{T})$ for non-convex stochastic optimization.

Our convergence analysis applies to a new algorithm called AdaFom (AdaGrad with First Order Momentum).

We show that the conditions are essential, by identifying concrete examples in which violating the conditions makes an algorithm diverge.

Besides providing one of the first comprehensive analysis for Adam-type methods in the non-convex setting, our results can also help the practitioners to easily  monitor the progress of algorithms and determine their convergence behavior.

We analyze convergence of Adam-type algorithms and provide mild sufficient conditions to guarantee their convergence, we also show  violating the conditions can makes an algorithm diverge.

Presents a convergence analysis in the non-convex setting for a family of optimization algorithms.

This paper investigates the convergence condition of Adam-type optimizers in the unconstrained non-convex optimization problems.

On the Convergence of A Class of Adam-Type Algorithms  for Non-Convex Optimization

convergence
adam-type algorithm
condition
non-convex
optimization


This research paper describes a simplistic architecture named as AANN: Absolute Artificial Neural Network, which can be used to create highly interpretable representations of the input data.

These representations are generated by penalizing the learning of the network in such a way that those learned representations correspond to the respective labels present in the labelled dataset used for supervised training; thereby, simultaneously giving the network the ability to classify the input data.

The network can be used in the reverse direction to generate data that closely resembles the input by feeding in representation vectors as required.

This research paper also explores the use of mathematical abs (absolute valued) functions as activation functions which constitutes the core part of this neural network architecture.

Finally the results obtained on the MNIST dataset by using this technique are presented and discussed in brief.

Tied weights auto-encoder with abs function as activation function, learns to do classification in the forward direction and regression in the backward direction due to specially defined cost function.

The paper proposes using the absolute value activation function in an autoencoder architecture with an additional supervised learning term in the objective function

This paper introduces a reversible network with absolute value used as the activation function

absolute value activation function
network


Current state-of-the-art relation extraction methods typically rely on a set of lexical, syntactic, and semantic features, explicitly computed in a pre-processing step.

Training feature extraction models requires additional annotated language resources, which severely restricts the applicability and portability of relation extraction to novel languages.

Similarly, pre-processing introduces an additional source of error.

To address these limitations, we introduce TRE, a Transformer for Relation Extraction, extending the OpenAI Generative Pre-trained Transformer [Radford et al., 2018].

Unlike previous relation extraction models, TRE uses pre-trained deep language representations instead of explicit linguistic features to inform the relation classification and combines it with the self-attentive Transformer architecture to effectively model long-range dependencies between entity mentions.

TRE allows us to learn implicit linguistic features solely from plain text corpora by unsupervised pre-training, before fine-tuning the learned language representations on the relation extraction task.

TRE obtains a new state-of-the-art result on the TACRED and SemEval 2010 Task 8 datasets, achieving a test F1 of 67.4 and 87.1, respectively.

Furthermore, we observe a significant increase in sample efficiency.

With only 20% of the training examples, TRE matches the performance of our baselines and our model trained from scratch on 100% of the TACRED dataset.

We open-source our trained models, experiments, and source code.

We propose a Transformer based relation extraction model that uses pre-trained language representations instead of explicit linguistic features.

Presents a transformer-based relation extraction model that leverages pre-training on unlabeled text with a language modeling objective.

This article describes a novel application of Transformer networks for relation extraction.

The paper presents a Transformer based architecture for relaxation extraction, evaluating on two datasets.

Improving Relation Extraction by Pre-trained Language Representations

pre-trained language representation
relation extraction model
transformer base
present


Neural networks have recently had a lot of success for many tasks.

network architectures that perform well are still typically designed manually

by experts in a cumbersome trial-and-error process.

to automatically search for well-performing CNN architectures based on a simple

hill climbing procedure whose operators apply network morphisms, followed

by short optimization runs by cosine annealing.

yields competitive results, despite only requiring resources in the same order of

and trains networks with an error rate below 6% in only 12 hours on a single GPU;

training for one day reduces this error further, to almost 5%.

We propose a simple and efficent method for architecture search for convolutional neural networks.

Proposes a neural architecture search method that achieves close to state-of-the-art accuracy on CIFAR10 and takes much less computational resources.

Presents a method to search neural network architectures at the same time of training which dramatically saves training time and architecture searching time.

Proposes variant of neural architecture search using network morphisms to define a search space using CNN architectures completing the CIFAR image classification task

Simple and efficient architecture search for Convolutional Neural Networks

architecture search convolutional neural network


We propose GraphGAN - the first implicit generative model for graphs that enables to mimic real-world networks.

We pose the problem of graph generation as learning the distribution of biased random walks over a single input graph.

Our model is based on a stochastic neural network that generates discrete output samples, and is trained using the Wasserstein GAN objective.

GraphGAN enables us to generate sibling graphs, which have similar properties yet are not exact replicas of the original graph.

Moreover, GraphGAN learns a semantic mapping from the latent input space to the generated graph's properties.

We discover that sampling from certain regions of the latent space leads to varying properties of the output graphs, with smooth transitions between them.

Strong generalization properties of GraphGAN are highlighted by its competitive performance in link prediction as well as promising results on node classification, even though not specifically trained for these tasks.

Using GANs to generate graphs via random walks.

The authors proposed a generative model of random walks on graphs that allows for model-agnostic learning, controllable fitting, ensemble graph generation

Proposes a WGAN formulation for generating graphs based on random walks using node embeddings and an LSTM architecture for modeling.

GraphGAN: Generating Graphs via Random Walks


The ability of a classifier to recognize unknown inputs is important for many classification-based systems.

We discuss the problem of simultaneous classification and novelty detection, i.e. determining whether an input is from the known set of classes and from which specific class, or from an unknown domain and does not belong to any of the known classes.

We propose a method based on the Generative Adversarial Networks (GAN) framework.

We show that a multi-class discriminator trained with a generator that generates samples from a mixture of nominal and novel data distributions is the optimal novelty detector.

We approximate that generator with a mixture generator trained with the Feature Matching loss and empirically show that the proposed method outperforms conventional methods for novelty detection.

Our findings demonstrate a simple, yet powerful new application of the GAN framework for the task of novelty detection.

We propose to solve a problem of simultaneous classification and novelty detection within the GAN framework.

Proposes a GAN to unify classification and novelty detection.

The paper presents a method for novelty detection based on a multi-class GAN which is trained to output images generated from a mixture of the nominal and novel distributions.

The paper proposes a GAN for novelty detection using a mixture generator with feature matching loss

Novelty Detection with GAN

classification novelty detection
gan
mixture


Verifying a person's identity based on their voice is a challenging, real-world problem in biometric security.

A crucial requirement of such speaker verification systems is to be domain robust.

Performance should not degrade even if speakers are talking in languages not seen during training.

To this end, we present a flexible and interpretable framework for learning domain invariant speaker embeddings using Generative Adversarial Networks.

We combine adversarial training with an angular margin loss function, which encourages the speaker embedding model to be discriminative by directly optimizing for cosine similarity between classes.

We are able to beat a strong baseline system using a cosine distance classifier and a simple score-averaging strategy.

Our results also show that models with adversarial adaptation perform significantly better than unadapted models.

In an attempt to better understand this behavior, we quantitatively measure the degree of invariance induced by our proposed methods using Maximum Mean Discrepancy and Frechet distances.

Our analysis shows that our proposed adversarial speaker embedding models significantly reduce the distance between source and target data distributions, while performing similarly on the former and better on the latter.

Speaker verificaiton performance can be significantly improved by adapting the model to in-domain data using Generative Adversarial Networks. Furthermore, the adaptation can be performed in an unsupervised way.

Propose a number of GAN variants on the task of speaker recognition in the domain mismatched condition.

SpeakerGAN: Recognizing Speakers in New Languages with Generative Adversarial Networks


Learning disentangling representations of the independent factors of variations that explain the data in an unsupervised setting is still a major challenge.

In the following paper we address the task of disentanglement and introduce a new state-of-the-art approach called Non-synergistic variational Autoencoder (Non-Syn VAE).

Our model draws inspiration from population coding, where the notion of synergy arises when we describe the encoded information by neurons in the form of responses from the stimuli.

If those responses convey more information together than separate as independent sources of encoding information, they are acting synergetically.

By penalizing the synergistic mutual information within the latents we encourage information independence and by doing that disentangle the latent factors.

Notably, our approach could be added to the VAE framework easily, where the new ELBO function is still a lower bound on the log likelihood.

In addition, we qualitatively compare our model with Factor VAE and show that this one implicitly minimises the synergy of the latents.

Minimising the synergistic mutual information within the latents and the data for the task of disentanglement using the VAE framework.

Proposes a new objective function for learning dientangled representations in a variational framework by minimizing the synergy of the information provided.

The authors aim at training a VAE that has disentangled latent representations in a "synergistically" maximal way.

This paper proposes a new approach to enforcing disentanglement in VAEs using a term that penalizes the synergistic mutual information between the latent variables.

Non-Synergistic Variational Autoencoders

synergistic mutual information
latent
vae
representation
new
disentanglement
framework
variational


Metric embeddings are   immensely useful representations of associations between entities   (images, users, search queries, words, and more).

Embeddings are learned by  optimizing a loss objective of the general form of a sum over example associations.

Typically, the optimization uses stochastic gradient updates over minibatches of examples that are arranged  independently at random.

In this work, we propose the use of {\em structured arrangements} through randomized {\em microbatches} of examples that are more likely to include similar ones.

We make a principled argument for the properties of our arrangements  that accelerate the training and present efficient algorithms to generate microbatches that respect the marginal  distribution of training examples.

Finally, we observe experimentally that our structured arrangements accelerate training by 3-20\%.

Structured arrangements emerge as a powerful and novel performance knob for SGD that is independent and complementary to other SGD  hyperparameters and thus is a candidate for wide deployment.

Accelerating SGD by arranging examples differently

The paper presents a method for improving the convergence rate of Stochastic Gradient Descent for learning embeddings by grouping similar training samples together.

Proposes a non-uniform sampling strategy to construct minibatches in SGD for the task of learning embeddings for object associations.

LSH Microbatches for Stochastic Gradients:  Value in Rearrangement

learn embedding
stochastic gradient
sgd
sample


Ubuntu dialogue corpus is the largest public available dialogue corpus to make it feasible to build end-to-end
deep neural network models directly from the conversation data.

One challenge of Ubuntu dialogue corpus is
the large number of out-of-vocabulary words.

In this paper we proposed an algorithm which combines the general pre-trained word embedding vectors with those  generated on the task-specific training set to address this issue.

We integrated character embedding into Chen et al's Enhanced LSTM method (ESIM) and used it to evaluate the effectiveness of our proposed method.

For the task of next utterance selection, the proposed method has demonstrated a significant performance improvement against original ESIM and the new model has achieved state-of-the-art results on both Ubuntu dialogue corpus and Douban conversation corpus.

In addition, we investigated the performance impact of end-of-utterance and end-of-turn token tags.

Combine information between pre-built word embedding and task-specific word representation to address out-of-vocabulary issue

This paper proposes an approach to improve the out-of-vocabulary embedding prediction for the task of modeling dialogue conversations with sizable gains over the baselines.

Proposes combining external pretrained word embeddings and pretrained word embeddings on training data by keeping them as two views.

Proposes method to extend the coverage of pre-trained word embeddings to deal with the OOV problem that arises when applying them to conversational datasets and applies new variants of LSTM-based model to the task of response-selection in dialogue modeling.

Enhance Word Representation for Out-of-Vocabulary on Ubuntu Dialogue Corpus

task
word representation
model
out-of-vocabulary
embedding
combine
embed
dialogue