Stochastic gradient Markov chain Monte Carlo (SG-MCMC) has become increasingly popular for simulating posterior samples in large-scale Bayesian modeling.

However, existing SG-MCMC schemes are not tailored to any specific probabilistic model, even a simple modification of the underlying dynamical system requires significant physical intuition.

This paper presents the first meta-learning algorithm that allows automated design for the underlying continuous dynamics of an SG-MCMC sampler.

The learned sampler generalizes Hamiltonian dynamics with state-dependent drift and diffusion, enabling fast traversal and efficient exploration of energy landscapes.

Experiments validate the proposed approach on Bayesian fully connected neural network, Bayesian convolutional neural network and Bayesian recurrent neural network tasks, showing that the learned sampler outperforms generic, hand-designed SG-MCMC algorithms, and generalizes to different datasets and larger architectures.

This paper proposes a method to automate the design of stochastic gradient MCMC proposal using meta learning approach.

Prsents a meta-learning approach to automatically design MCMC sampler based on Hamiltonian dynamics to mix faster on problems similar to training problems

Parameterizes diffusion and curl matrices by neural networks and meta-learn and optimize an sg-mcmc algorithm.

Meta-Learning For Stochastic Gradient MCMC

stochastic gradient mcmc
approach
design


We propose a new, multi-component energy function for energy-based Generative Adversarial Networks (GANs) based on methods from the image quality assessment literature.

Our approach expands on the Boundary Equilibrium Generative Adversarial Network (BEGAN) by outlining some of the short-comings of the original energy and loss functions.

We address these short-comings by incorporating an l1 score, the Gradient Magnitude Similarity score, and a chrominance score into the new energy function.

We then provide a set of systematic experiments that explore its hyper-parameters.

We show that each of the energy function's components is able to represent a slightly different set of features, which require their own evaluation criteria to assess whether they have been adequately learned.

We show that models using the new energy function are able to produce better image representations than the BEGAN model in predicted ways.

Image Quality Assessment Techniques Improve Training and Evaluation of Energy-Based Generative Adversarial Networks

Proposes an energy-based formulation to the BEGAN modeal and modifies it to include an image quality assessment based term

Proposes some new energy function in the BEGAN (boundary equilibrium GAN framework), including l_1 score, Gradient magnitude similarity score, and chrominance score.

image quality assessment technique improve training evaluation energy-based


Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions.

Its performance depends crucially on a damping coefficient.

Largecamping  coefficients can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typically resorts to small values such as 0.5 or 0.9.

We propose Aggregated Momentum (AggMo), a variant of momentum which combines multiple velocity vectors with different damping coefficients.

AggMo is trivial to implement, but significantly dampens oscillations, enabling it to remain stable even for aggressive damping coefficients such as 0.999.

We reinterpret Nesterov's accelerated gradient descent as a special case of AggMo and analyze rates of convergence for quadratic objectives.

Empirically, we find that AggMo is a suitable drop-in replacement for other momentum methods, and frequently delivers faster convergence with little to no tuning.

We introduce a simple variant of momentum optimization which is able to outperform classical momentum, Nesterov, and Adam on deep learning tasks with minimal hyperparameter tuning.

Introduces a variant of momentum that aggregates several velocities with different dampening coefficients that significantly decreases oscillation

Proposed an aggregated momentum methods for gradient based optimization by using multiple velocity vectors with different damping factors instead of a single velocity vector to improve stability

The authors combine several update steps together to achieve aggregated momentum also demonstrating it is more stable than the other momentum methods

Aggregated Momentum: Stability Through Passive Damping

optimization
variant momentum
introduce
different
aggregate
several
velocity
method
damp
aggregated
stability


Recurrent Neural Networks architectures excel at processing sequences by

modelling dependencies over different timescales.

Recurrent Weighted Average (RWA) unit captures long term dependencies

far better than an LSTM on several challenging tasks.

this by applying attention to each input and computing a weighted average

over the full history of its computations.

change the attention it has assigned to previous timesteps, and so struggles

with carrying out consecutive tasks or tasks with changing requirements.

We present the Recurrent Discounted Attention (RDA) unit that builds on

the RWA by additionally allowing the discounting of the past.

We empirically compare our model to RWA, LSTM and GRU units on

On tasks with a single output the RWA, RDA and

GRU units learn much quicker than the LSTM and with better performance.

On the multiple sequence copy task our RDA unit learns the task three

times as quickly as the LSTM or GRU units while the RWA fails to learn at

On the Wikipedia character prediction task the LSTM performs best

but it followed closely by our RDA unit.

well and is sample efficient on a large variety of sequence tasks.

We introduce the Recurrent Discounted Unit which applies attention to any length sequence in linear time

This paper proposes the Recurrent Discounted Attention (RDA), an extension to Recurrent Weighted Average (RWA) by adding a discount factor.

Extends the recurrent weight average to overcome the limitation of the original method while maintaining its advantage and proposes the method of using Elman nets as the base RNN

Efficiently applying attention to sequential data with the Recurrent Discounted Attention unit

recurrent weight average
apply attention
discount
unit


Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging.

In this paper, we introduce variance layers, a different kind of stochastic layers.

Each weight of a variance layer follows a zero-mean distribution and is only parameterized by its variance.

It means that each object is represented by a zero-mean distribution in the space of the activations.

We show that such layers can learn surprisingly well, can serve as an efficient exploration tool in reinforcement learning tasks and provide a decent defense against adversarial attacks.

We also show that a number of conventional Bayesian neural networks naturally converge to such zero-mean posteriors.

We observe that in these cases such zero-mean parameterization leads to a much better training objective than more flexible conventional parameterizations where the mean is being learned.

It is possible to learn a zero-centered Gaussian distribution over the weights of a neural network by learning only variances, and it works surprisingly well.

This paper investigates the effects of mean of variational posterior and proposes variance layer, which only uses variance to store information

Studies variance neural networks which approximate the posterior of Bayesian neural networks with zero-mean Gaussian distributions

Variance Networks: When Expectation Does Not Meet Your Expectations

variance
neural network
posterior
gaussian distribution


Graph Convolutional Networks (GCNs) are a recently proposed architecture which has had success in semi-supervised learning on graph-structured data.

At the same time, unsupervised learning of graph embeddings has benefited from the information contained in random walks.

In this paper we propose a model, Network of GCNs (N-GCN), which marries these two lines of work.

At its core, N-GCN trains multiple instances of GCNs over node pairs discovered at different distances in random walks, and learns a combination of the instance outputs which optimizes the classification objective.

Our experiments show that our proposed N-GCN model achieves state-of-the-art performance on all of the challenging node classification tasks we consider: Cora, Citeseer, Pubmed, and PPI.

In addition, our proposed method has other desirable properties, including generalization to recently proposed semi-supervised learning methods such as GraphSAGE, allowing us to propose N-SAGE, and resilience to adversarial input perturbations.

We make a network of Graph Convolution Networks, feeding each a different power of the adjacency matrix, combining all their representation into a classification sub-network, achieving state-of-the-art on semi-supervised node classification.

Proposes a new network of GCNs with two approaches: a fully connected layer on top of stacked features and attention mechanism that uses scalar weight per GCN.

Presents a Network of Graph Convolutional Networks that uses random walk statistics to extract information from near and distant neighbors in the graph

Network of Graph Convolutional Networks Trained on Random Walks

network graph convolutional network
random walk


Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers often with little or no drop in classification accuracy.

However most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain classification accuracy.

In this paper we propose a cheap pruning algorithm based on difference of convex (DC) optimisation.

We also provide theoretical analysis for the growth in the Generalisation Error (GE) of the new pruned network.

Our method can be used with any convex regulariser and allows for a controlled degradation in classification accuracy while being orders of magnitude faster than competing approaches.

Experiments on common feedforward neural networks show that for sparsity levels above 90% our method achieves 10% higher classification accuracy compared to Hard Thresholding.

A fast pruning algorithm for fully connected DNN layers with theoretical analysis of degradation in Generalisation Error.

Presents a cheap pruning algorithm for dense layers of DNNs.

Proposes a solution to the problem of pruning DNNs by posing the Net-trim objective function as a Difference of convex(DC) function.

Cheap DNN Pruning with Performance Guarantees


Action segmentation as a milestone towards building automatic systems to understand untrimmed videos has received considerable attention in the recent years.

It is typically being modeled as a sequence labeling problem but contains intrinsic and sufficient differences than text parsing or speech processing.

In this paper, we introduce a novel hybrid temporal convolutional and recurrent network (TricorNet), which has an encoder-decoder architecture: the encoder consists of a hierarchy of temporal convolutional kernels that capture the local motion changes of different actions; the decoder is a hierarchy of recurrent neural networks that are able to learn and memorize long-term action dependencies after the encoding stage.

Our model is simple but extremely effective in terms of video sequence labeling.

The experimental results on three public action segmentation datasets have shown that the proposed model achieves superior performance over the state of the art.

We propose a new hybrid temporal network that achieves state-of-the-art performance on video action segmentation on three public datasets.

Discusses the problem of action segmentation in long videos, up to 10 minutes long by using a temporal convolutional encoder-decoder architecture

Proposes a combination of temporal convolutional and recurrent network for video action segmentation

Video Action Segmentation with Hybrid Temporal Networks

hybrid temporal network
video action segmentation
convolutional


Convolutional Neural Networks (CNNs) become deeper and deeper in recent years, making the study of model acceleration imperative.

It is a common practice to employ a shallow network, called student, to learn from a deep one, which is termed as teacher.

Prior work made many attempts to transfer different types of knowledge from teacher to student, however, there are two problems remaining unsolved.

Firstly, the knowledge used by existing methods is highly dependent on task and dataset, limiting their applications.

Secondly, there lacks an effective training scheme for the transfer process, leading to degradation of performance.

In this work, we argue that feature is the most important knowledge from teacher.

It is sufficient for student to just learn good features regardless of the target task.

From this discovery, we further present an efficient learning strategy to mimic features stage by stage.

Extensive experiments demonstrate the importance of features and show that the proposed approach significantly narrows down the gap between student and teacher, outperforming the state-of-the-art methods.

This paper proposes to transfer knowledge from deep model to shallow one by mimicking features stage by stage.

Explains a stage by stage knowledge transer method by using different structures of resnets

This paper proposes dividing a network into multiple parts and distilling each part sequentially to improve distillation performance in deep teacher networks

Feature Matters: A Stage-by-Stage Approach for Task Independent Knowledge Transfer

stage stage
deep
feature
knowledge
transfer


We augment adversarial training (AT) with worst case adversarial training

(WCAT) which improves adversarial robustness by 11% over the current state-

of-the-art result in the `2-norm on CIFAR-10.

Total Variation Regularization, which is a fundamental tool in mathematical im-

age processing, and WCAT as Lipschitz regularization, which appears in Image

We obtain verifiable worst and average case robustness guarantees,

based on the expected and maximum values of the norm of the gradient of the

Improvements to adversarial robustness, as well as provable robustness guarantees, are obtained by augmenting adversarial training with a tractable Lipschitz regularization

Explores augmenting the training loss with an additional gradient regularization term to improve robustness of models against adversarial examples

Uses a trick to simplify the adversarial loss by one in which the adversarial perturbation appears in closed form.

Improved robustness to adversarial examples using Lipschitz regularization of the loss

loss
adversarial example
improve robustness
lipschitz regularization
augment
training


The task of Reading Comprehension with Multiple Choice Questions, requires a human (or machine) to read a given \{\textit{passage, question}\} pair and select one of the $n$ given options.

The current state of the art model for this task first computes a query-aware representation for the passage and then \textit{selects} the option which has the maximum similarity with this representation.

However, when humans perform this task they do not just focus on option selection but use a combination of \textit{elimination} and \textit{selection}. Specifically, a human would first try to eliminate the most irrelevant option and then read the document again in the light of this new information (and perhaps ignore portions corresponding to the eliminated option).

This process could be repeated multiple times till the reader is finally ready to select the correct option.

We propose \textit{ElimiNet}, a neural network based model which tries to mimic this process.

Specifically, it has gates which decide whether an option can be eliminated given the \{\textit{document, question}\} pair and if so it tries to make the document representation orthogonal to this eliminatedd option (akin to ignoring portions of the document corresponding to the eliminated option).

The model makes multiple rounds of partial elimination to refine the document representation and finally uses a selection module to pick the best option.

We evaluate our model on the recently released large scale RACE dataset and show that it outperforms the current state of the art model on 7 out of the 13 question types in this dataset.

Further we show that taking an ensemble of our \textit{elimination-selection} based method with a \textit{selection} based method gives us an improvement of 7\% (relative) over the best reported performance on this dataset.

A model combining elimination and selection for answering multiple choice questions

Gives an elaboration on the Gated Attention Reader adding gates based on answer elimination in multiple choice reading comprehension

This paper proposes the use of an elimination gate in model architectures for reading comprehension tasks but does not achieve state-of-the-art results

This paper propses a new multi-choice reading comprehension model based on the idea that some options should be eliminated to infer better passage/question representations.

ElimiNet: A Model for Eliminating Options for Reading Comprehension with Multiple Choice Questions

multiple choice question
read comprehension
model
elimination
answer
base
gate
eliminate
option
reading


Humans are capable of attributing latent mental contents such as beliefs, or intentions to others.

The social skill is critical in everyday life to reason about the potential consequences of their behaviors so as to plan ahead.

It is known that humans use this reasoning ability recursively, i.e. considering what others believe about their own beliefs.

In this paper, we start  from level-$1$ recursion and introduce a probabilistic recursive reasoning (PR2) framework for multi-agent reinforcement learning.

Our hypothesis is that it is beneficial for each agent to account for how the opponents would react to its future behaviors.

Under the PR2 framework, we adopt variational Bayes methods to approximate the opponents' conditional policy, to which each agent finds the  best response and then improve their own policy.

We develop  decentralized-training-decentralized-execution  algorithms, PR2-Q and PR2-Actor-Critic, that are proved to converge in the self-play scenario when there is one Nash equilibrium.

Our methods are tested on both the matrix game and the differential game, which have a non-trivial equilibrium where common gradient-based methods fail to converge.

Our experiments show that it is critical to reason about how the opponents believe about what the agent believes.

We expect our work to contribute a new idea of modeling the opponents to the multi-agent reinforcement learning community.

We proposed a novel probabilisitic recursive reasoning (PR2) framework for multi-agent deep reinforcement learning tasks.

Proposes a new approach for fully decentralized training in multi-agent reinforcement learning

Tackles the problem of endowing RL agents with recursive reasoning capabilities in a multi-agent setting based on the hypothesis that recursive reasoning is beneficial for them to converge to non-trival equilibria

The paper introduces a decentralized training method for multi-agent reinforcement learning, where the agents infer the policies of other agents and use the inferred models for decision making.

Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning

multi-agent reinforcement learning
recursive reasoning
decentralized training
agent


Due to the substantial computational cost, training state-of-the-art deep neural networks for large-scale datasets often requires distributed training using multiple computation workers.

However, by nature, workers need to frequently communicate gradients, causing severe bottlenecks, especially on lower bandwidth connections.

A few methods have been proposed to compress gradient for efficient communication, but they either suffer a low compression ratio or significantly harm the resulting model accuracy, particularly when applied to convolutional neural networks.

To address these issues, we propose a method to reduce the communication overhead of distributed deep learning.

Our key observation is that gradient updates can be delayed until an unambiguous (high amplitude, low variance) gradient has been calculated.

We also present an efficient algorithm to compute the variance and prove that it can be obtained with negligible additional cost.

We experimentally show that our method can achieve very high compression ratio while maintaining the result model accuracy.

We also analyze the efficiency using computation and communication cost models and provide the evidence that this method enables distributed deep learning for many scenarios with commodity environments.

A new algorithm to reduce the communication overhead of distributed deep learning by distinguishing ‘unambiguous’ gradients.

Proposes a variance-based gradient compression method to reducee the communication overhead of distributed deep learning

Proposes a novel way of compressing gradient updates for distributed SGD in order to speed up overall execution

Introduces variance-based gradient compression method for efficient distributed training of neural networks and measuring ambuiguity.

Variance-based Gradient Compression for Efficient Distributed Deep Learning

communication overhead distribute deep learning


In this work, we face the problem of unsupervised domain adaptation with a novel deep learning approach which leverages our finding that entropy minimization is induced by the optimal alignment of second order statistics between source and target domains.

We formally demonstrate this hypothesis and, aiming at achieving an optimal alignment in practical cases, we adopt a more principled strategy which, differently from the current Euclidean approaches, deploys alignment along geodesics.

Our pipeline can be implemented by adding to the standard classification loss (on the labeled source domain), a source-to-target regularizer that is weighted in an unsupervised and data-driven fashion.

We provide extensive experiments to assess the superiority of our framework on standard domain and modality adaptation benchmarks.

A new unsupervised deep domain adaptation technique which efficiently unifies correlation alignment and entropy minimization

Improves the correlation alignment approach to domain adaptation by replacing the Euclidean distance with the geodesic Log-Euclidean distance between two covariance matices, and automatically selecting the balancing cost by the entropy on the target domain.

Proposal for minimal-entropy correlation alignment, an unsupervised domain adaptation algorithm which links together entropy minimization and correlation alignment methods.

Minimal-Entropy Correlation Alignment for Unsupervised Deep Domain Adaptation

unsupervised deep domain adaptation
minimal-entropy correlation alignment
entropy minimization


Catastrophic interference has been a major roadblock in the research of continual learning.

Here we propose a variant of the back-propagation algorithm, "Conceptor-Aided Backprop" (CAB), in which gradients are shielded by conceptors against degradation of previously learned tasks.

Conceptors have their origin in reservoir computing, where they have been previously shown to overcome catastrophic forgetting.

CAB extends these results to deep feedforward networks.

On the disjoint and permuted MNIST tasks, CAB outperforms two other methods for coping with catastrophic interference that have recently been proposed.

We propose a variant of the backpropagation algorithm, in which gradients are shielded by conceptors against degradation of previously learned tasks.

This paper applies the notion of conceptors, a form a regulariser, to prevent forgetting in continual learning in the training of neural networks on sequential tasks.

Introduces a method for learning new tasks, without interfering previous tasks, using conceptors.

Overcoming Catastrophic Interference using Conceptor-Aided Backpropagation


Recent advances in neural Sequence-to-Sequence (Seq2Seq) models reveal a purely data-driven approach to the response generation task.

Despite its diverse variants and applications, the existing Seq2Seq models are prone to producing short and generic replies, which blocks such neural network architectures from being utilized in practical open-domain response generation tasks.

In this research, we analyze this critical issue from the perspective of the optimization goal of models and the specific characteristics of human-to-human conversational corpora.

Our analysis is conducted by decomposing the goal of Neural Response Generation (NRG) into the optimizations of word selection and ordering.

It can be derived from the decomposing that Seq2Seq based NRG models naturally tend to select common words to compose responses, and ignore the semantic of queries in word ordering.

On the basis of the analysis, we propose a max-marginal ranking regularization term to avoid Seq2Seq models from producing the generic and uninformative responses.

The empirical experiments on benchmarks with several metrics have validated our analysis and proposed methodology.

Analyze the reason for neural response generative models preferring universal replies; Propose a method to avoid it.

Investigates the problem of universal replies plaguing the Seq2Seq neural generation models

The paper looks into improving the neural response generation task by deemphasizing the common responses using modification of the loss function and presentation the common/universal responses during the training phase.

Why Do Neural Response Generation Models Prefer Universal Replies?

model prefer universal reply
neural response generation


The ability to generate natural language sequences from source code snippets has a variety of applications such as code summarization, documentation, and retrieval.

Sequence-to-sequence (seq2seq) models, adopted from neural machine translation (NMT), have achieved state-of-the-art performance on these tasks by treating source code as a sequence of tokens.

We present code2seq: an alternative approach that leverages the syntactic structure of programming languages to better encode source code.

Our model represents a code snippet as the set of compositional paths in its abstract syntax tree (AST) and uses attention to select the relevant paths while decoding.

We demonstrate the effectiveness of our approach for two tasks, two programming languages, and four datasets of up to 16M examples.

Our model significantly outperforms previous models that were specifically designed for programming languages, as well as general state-of-the-art NMT models.

An interactive online demo of our model is available at http://code2seq.org.

Our code, data and trained models are available at http://github.com/tech-srl/code2seq.

We leverage the syntactic structure of source code to generate natural language sequences.

Presents a method for generating sequences from code by parsing and producing a syntax tree

This paper introduces an AST-based encoding for programming code and shows its effectiveness in the tasks of extreme code summarization and code captioning.

This paper presents a new code-to-sequence model that leverages the syntactic structure of programming languages to encode source code snippets and then decode them to natural language

code2seq: Generating Sequences from Structured Representations of Code

leverage syntactic structure
generate sequence
code
natural language
present
programming
source


We propose a novel attention mechanism to enhance Convolutional Neural Networks for fine-grained recognition.

The proposed mechanism reuses CNN feature activations to find the most informative parts of the image at different depths with the help of gating mechanisms and without part annotations.

Thus, it can be used to augment any layer of a CNN to extract low- and high-level local information to be more discriminative.

Differently, from other approaches, the mechanism we propose just needs a single pass through the input and it can be trained end-to-end through SGD.

As a consequence, the proposed mechanism is modular, architecture-independent, easy to implement, and faster than iterative approaches.

Experiments show that, when augmented with our approach, Wide Residual Networks systematically achieve superior performance on each of five different fine-grained recognition datasets: the Adience age and gender recognition benchmark, Caltech-UCSD Birds-200-2011, Stanford Dogs, Stanford Cars, and UEC Food-100, obtaining competitive and state-of-the-art scores.

We enhance CNNs with a novel attention mechanism for fine-grained recognition. Superior performance is obtained on 5 datasets.

Describes a novel attentional mechanism applid to fine-grained recognition that consistently improves the recognition accuracy of the baseline

This paper proposes a feed-forward attention mechanism for fine-grained image classification

This paper presents an interesting attention mechanism for fine-grained image classification.

A Painless Attention Mechanism for Convolutional Neural Networks

attention mechanism fine-grained image classification


Most existing GANs architectures that generate images use transposed convolution or resize-convolution as their upsampling algorithm from lower to higher resolution feature maps in the generator.

We argue that this kind of fixed operation is problematic for GANs to model objects that have very different visual appearances.

We propose a novel adaptive convolution method that learns the upsampling algorithm based on the local context at each location to address this problem.

We modify a baseline GANs architecture by replacing normal convolutions with adaptive convolutions in the generator.

Experiments on CIFAR-10 dataset show that our modified models improve the baseline model by a large margin.

Furthermore, our models achieve state-of-the-art performance on CIFAR-10 and STL-10 datasets in the unsupervised setting.

We replace normal convolutions with adaptive convolutions to improve GANs generator.

Proposes to replace convolutions in the generator with an Adaptive Convolution Block that learns to generate convolution weigths adn biases of upsampling operations adaptively per pixel location

Uses Adaptive Convolution in the context of GANs with a block called AdaConvBlock that replaces regular Convolution, this gives more local context per kernel weight so that it can generate locally flexible objects.

Generative Adversarial Networks using Adaptive Convolution

adaptive convolution
replace
generator
generate
block
gan
per


Techniques such as ensembling and distillation promise model quality improvements when paired with almost any base model.

However, due to increased test-time cost (for ensembles) and increased complexity of the training pipeline (for distillation), these techniques are challenging to use in industrial settings.

In this paper we explore a variant of distillation which is relatively straightforward to use as it does not require a complicated multi-stage setup or many new hyperparameters.

Our first claim is that online distillation enables us to use extra parallelism to fit very large datasets about twice as fast.

Crucially, we can still speed up training even after we have already reached the point at which additional parallelism provides no benefit for synchronous or asynchronous stochastic gradient descent.

Two neural networks trained on disjoint subsets of the data can share knowledge by encouraging each model to agree with the predictions the other model would have made.

These predictions can come from a stale version of the other model so they can be safely computed using weights that only rarely get transmitted.

Our second claim is that online distillation is a cost-effective way to make the exact predictions of a model dramatically more reproducible.

We support our claims using experiments on the Criteo Display Ad Challenge dataset, ImageNet, and the largest to-date dataset used for neural language modeling, containing $6\times 10^{11}$ tokens and based on the Common Crawl repository of web data.

We perform large scale experiments to show that a simple online variant of distillation can help us scale distributed neural network training to more machines.

Proposes a method to scale distributed training beyond the current limits of mini-batch stochastic gradient descent

Proposal for an online distillation method called co-distillation, applied at scale, where two different models are trained to match predictions of the other model in addition to minimizing its own loss.

Online distillation technique is introduced to accelerate traditional algorithms for large-scaled distributed neural network training

Large scale distributed neural network training through online distillation

scale distribute neural network training


Support Vector Machines (SVMs) are one of the most popular algorithms for classification and regression analysis.

Despite their popularity, even efficient implementations have proven to be computationally expensive to train at a large-scale, especially in streaming settings.

In this paper, we propose a novel coreset construction algorithm for efficiently generating compact representations of massive data sets to speed up SVM training.

A coreset is a weighted subset of the original data points such that SVMs trained on the coreset are provably competitive with those trained on the original (massive) data set.

We provide both lower and upper bounds on the number of samples required to obtain accurate approximations to the SVM problem as a function of the complexity of the input data.

Our analysis also establishes sufficient conditions on the existence of sufficiently compact and representative coresets for the SVM problem.

We empirically evaluate the practical effectiveness of our algorithm against synthetic and real-world data sets.

We present an algorithm for speeding up SVM training on massive data sets by constructing compact representations that provide efficient and provably approximate inference.

Studies the approach of coreset for SVM and aims at sampling a small set of weighted points such that the loss function over the points provably approximates that over the whole dataset

The paper suggests an importance sampling based Coreset construction to represent large training data for SVMs

Small Coresets to Represent Large Training Data for Support Vector Machines

represent large training datum
provably approximate
set
svm
coreset
small
sample


The sign stochastic gradient descent method (signSGD) utilizes only the sign of the stochastic gradient in its updates.

Since signSGD carries out one-bit quantization of the gradients, it is extremely practical for distributed optimization where gradients need to be aggregated from different processors.

For the first time, we establish convergence rates for signSGD on general non-convex functions under transparent conditions.

We show that the rate of signSGD to reach first-order critical points matches that of SGD in terms of number of stochastic gradient calls, up to roughly a linear factor in the dimension.

We carry out simple experiments to explore the behaviour of sign gradient descent (without the stochasticity) close to saddle points and show that it often helps completely avoid them without using either stochasticity or curvature information.

We prove a non-convex convergence rate for the sign stochastic gradient method. The algorithm has links to algorithms like Adam and Rprop, as well as gradient quantisation schemes used in distributed machine learning.

Provided a convergence analysis of Sign SGD algorithm for non-covex cases

The paper explores an algorithm that uses the sign of the gradients instead of actual gradients for training deep models

Convergence rate of sign stochastic gradient descent for non-convex functions

sign stochastic gradient
algorithm
convergence rate
non-convex


Deep learning has found numerous applications thanks to its versatility and accuracy on pattern recognition problems such as visual object detection.

Learning and inference in deep neural networks, however, are memory and compute intensive and so improving efficiency is one of the major challenges for frameworks such as PyTorch, Tensorflow, and Caffe.

While the efficiency problem can be partially addressed with specialized hardware and its corresponding proprietary libraries, we believe that neural network acceleration should be transparent to the user and should support all hardware platforms and deep learning libraries.

To this end, we introduce a transparent middleware layer for neural network acceleration.

The system is built around a compiler for deep learning, allowing one to combine device-specific libraries and custom optimizations while supporting numerous hardware devices.

In contrast to other projects, we explicitly target the optimization of both prediction and training of neural networks.

We present the current development status and some preliminary but encouraging results: on a standard x86 server, using CPUs our system achieves a 11.8x speed-up for inference and a 8.0x for batched-prediction (128); on GPUs we achieve a 1.7x and 2.3x speed-up respectively.

We introduce a transparent middleware for neural network acceleration, with own compiler engine, achieving up to 11.8x speed up on CPUs and 2.3x on GPUs.

This paper proposes a transparent middleware layer for neural network acceleration and obtains some acceleration results on basic CPU and GPU architectures

Towards Transparent Neural Network Acceleration

neural network acceleration
transparent middleware
gpu


Performance of neural networks can be significantly improved by encoding known invariance for particular tasks.

Many image classification tasks, such as those related to cellular imaging, exhibit invariance to rotation.

In particular, to aid convolutional neural networks in learning rotation invariance, we consider a simple, efficient conic convolutional scheme that encodes rotational equivariance, along with a method for integrating the magnitude response of the 2D-discrete-Fourier transform (2D-DFT) to encode global rotational invariance.

We call our new method the Conic Convolution and DFT Network (CFNet).

We evaluated the efficacy of CFNet as compared to a standard CNN and group-equivariant CNN (G-CNN) for several different image classification tasks and demonstrated improved performance, including classification accuracy, computational efficiency, and its robustness to hyperparameter selection.

Taken together, we believe CFNet represents a new scheme that has the potential to improve many imaging analysis applications.

We propose conic convolution and the 2D-DFT to encode rotation equivariance into an neural network.

In the context of image classification, the paper proposes a convolutional neural network architecture with rotation-equivariant feature maps that are eventually made rotation-invariant by using the magnitude of the 2D discrete Fourier transform (DFT).

Authors provide a rotation invariant neural network via combining conic convolution and 2D-DFT

Rotation Equivariant Networks via Conic Convolution and the DFT

conic convolution
dft
2d-dft
neural network
rotation
via


The problem of visual metamerism is defined as finding a family of perceptually

indistinguishable, yet physically different images.

NeuroFovea metamer model, a foveated generative model that is based on a mixture

of peripheral representations and style transfer forward-pass algorithms.

gradient-descent free model is parametrized by a foveated VGG19 encoder-decoder

which allows us to encode images in high dimensional space and interpolate

between the content and texture information with adaptive instance normalization

1) A framework for
computing metamers that resembles a noisy communication system via a foveated
feed-forward encoder-decoder network – We observe that metamerism arises as a
byproduct of noisy perturbations that partially lie in the perceptual null space;

2)
A perceptual optimization scheme as a solution to the hyperparametric nature of
our metamer model that requires tuning of the image-texture tradeoff coefficients
everywhere in the visual field which are a consequence of internal noise;

ABX psychophysical evaluation of our metamers where we also find that the rate

of growth of the receptive fields in our model match V1 for reference metamers

Our model also renders metamers at roughly

a second, presenting a ×1000 speed-up compared to the previous work, which now

allows for tractable data-driven metamer experiments.

We introduce a novel feed-forward framework to generate visual metamers

Proposes a NeuroFovea model for generation of point-of-fixation metamers by using a style transfer approach via and Encoder-Decoder style architecture

An analysis of metamerism and a model capable of rapidly producing metamers of value for experimental psychophysics and other domains.

The paper proposes a fast method for generating visual metamers – physically different images that cannot be told apart from an original – via foveated, fast, arbitrary style transfer

Towards Metamerism via Foveated Style Transfer

generate visual metamer
model
style transfer
via foveate
metamerism


Past works have shown that, somewhat surprisingly, over-parametrization can help generalization in neural networks.

Towards explaining this phenomenon, we adopt a margin-based perspective.

1) for multi-layer feedforward relu networks, the global minimizer of a weakly-regularized cross-entropy loss has the maximum normalized margin among all networks,

2) as a result, increasing the over-parametrization improves the normalized margin and generalization error bounds for deep networks.

In the case of two-layer networks, an infinite-width neural network enjoys the best generalization guarantees.

The typical infinite feature methods are kernel methods; we compare the neural net margin with that of kernel methods and construct natural instances where kernel methods have much weaker generalization guarantees.

We validate this gap between the two approaches empirically.

Finally, this infinite-neuron viewpoint is also fruitful for analyzing optimization.

We show that a perturbed gradient flow on infinite-size networks finds a global optimizer in polynomial time.

We show that training feedforward relu networks with a weak regularizer results in a maximum margin and analyze the implications of this result.

Studies margin theory for neural sets  and shows that max margin is monotonically increasing in size of the network

This paper studies the implicit bias of minimizers of a regularized cross entropy loss of a two-layer network with ReLU activations, obtaining a generalization upper bound which does not increase with the network size.

On the Margin Theory of Feedforward Neural Networks

show
network
margin theory
size
increase
feedforward
relu
neural
study


We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible.

The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network.

The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors.

Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.

A distributed architecture for deep reinforcement learning at scale, using parallel data-generation to improve the state of the art on the Arcade Learning Environment benchmark in a fraction of the wall-clock training time of previous approaches.

Examines a distirbuted Deep RL system in which experiences, rather than gradients, are shared between the parallel works and the cetralized learner

A parallel approach to DQN training, based on the idea of having multiple actors collecting data in parallel while a single learner trains the model from experiences sampled from central replay memory.

This paper proposes a distributed architecture for deep reinforcement learning at scale, focusing on adding parallelization in actor algorithm in Prioritized Experience Replay framework

Distributed Prioritized Experience Replay

distribute architecture deep reinforcement learning scale


Designing neural networks for continuous-time stochastic processes is challenging, especially when observations are made irregularly.

In this article, we analyze neural networks from a frame theoretic perspective to identify the sufficient conditions that enable smoothly recoverable representations of signals in L^2(R).

Moreover, we show that, under certain assumptions, these properties hold even when signals are irregularly observed.

As we converge to the family of (convolutional) neural networks that satisfy these conditions, we show that we can optimize our convolution filters while constraining them so that they effectively compute a Discrete Wavelet Transform.

Such a neural network can efficiently divide the time-axis of a signal into orthogonal sub-spaces of different temporal scale and localization.

We evaluate the resulting neural network on an assortment of synthetic and real-world tasks: parsimonious auto-encoding, video classification, and financial forecasting.

Neural architectures providing representations of irregularly observed signals that provably enable signal reconstruction.

Proves that convolutional neural networks with Leaky ReLU activation function are nonlinear frames, with similar results for non-uniformly sampled time-series

This article considers neural networks over time-series and show that the first convolutional filters can be chosen to represent a discrete wavelet transform.

Neural Networks for irregularly observed continuous-time Stochastic Processes

irregularly observe
neural network
convolutional
time-serie


Most state-of-the-art neural machine translation systems, despite being different

in architectural skeletons (e.g., recurrence, convolutional), share an indispensable

However, most existing attention methods are token-based

and ignore the importance of phrasal alignments, the key ingredient for the success

of phrase-based statistical machine translation.

novel phrase-based attention methods to model n-grams of tokens as attention

We incorporate our phrase-based attentions into the recently proposed

Transformer network, and demonstrate that our approach yields improvements of

1.3 BLEU for English-to-German and 0.5 BLEU for German-to-English translation

tasks, and 1.75 and 1.35 BLEU points in English-to-Russian and Russian-to-English translation tasks

on WMT newstest2014 using WMT’16 training data.

Phrase-based attention mechanisms to assign attention on phrases, achieving token-to-phrase, phrase-to-token, phrase-to-phrase attention alignments, in addition to existing token-to-token attentions.

Paper presents an attention mechanism that computes a weighted sum over not only single tokens but ngrams(phrases).

Phrase-Based Attentions


Intuitively, unfamiliarity should lead to lack of confidence.

In reality, current algorithms often make highly confident yet wrong predictions when faced with unexpected test samples from an unknown distribution different from training.

Unlike domain adaptation methods, we cannot gather an "unexpected dataset" prior to test, and unlike novelty detection methods, a best-effort original task prediction is still expected.

We compare a number of methods from related fields such as calibration and epistemic uncertainty modeling, as well as two proposed methods that reduce overconfident errors of samples from an unknown novel distribution without drastically increasing evaluation time: (1) G-distillation, training an ensemble of classifiers and then distill into a single model using both labeled and unlabeled examples, or (2) NCR, reducing prediction confidence based on its novelty detection score.

Experimentally, we investigate the overconfidence problem and evaluate our solution by creating "familiar" and "novel" test splits, where "familiar" are identically distributed with training and "novel" are not.

We discover that calibrating using temperature scaling on familiar data is the best single-model method for improving novel confidence, followed by our proposed methods.

In addition, some methods' NLL performance are roughly equivalent to a regularly trained model with certain degree of smoothing.

Calibrating can also reduce confident errors, for example, in gender recognition by 95% on demographic groups different from the training data.

Deep networks are more likely to be confidently wrong when testing on unexpected data. We propose an experimental methodology to study the problem, and two methods to reduce confident errors on unknown input distributions.

Proposes two ideas for reducing overconfident wrong predictions: "G-distillation" of am ensemble with extra unsupervised data and Novelty Confidence Reduction using novelty detector

The authors propose two methods for estimating classification confidence on novel unseen data distributions. The first idea is to use ensemble methods as the base approach to help identify uncertain cases and then use distillation methods to reduce the ensemble into a single model mimicking behavior of the ensemble. The second idea is to use a novelty detector classifier and weight the network output by the novelty score.

Reducing Overconfident Errors outside the Known Distribution

two method
ensemble
novelty detector
distribution
network
datum
reduce
idea
wrong
error
overconfident
confidence


Progress in deep learning is slowed by the days or weeks it takes to train large models.

The natural solution of using more hardware is limited by diminishing returns, and leads to inefficient use of additional resources.

In this paper, we present a large batch, stochastic optimization algorithm that is both faster than widely used algorithms for fixed amounts of computation, and also scales up substantially better as more computational resources become available.

Our algorithm implicitly computes the inverse Hessian of each mini-batch to produce descent directions; we do so without either an explicit approximation to the Hessian or Hessian-vector products.

We demonstrate the effectiveness of our algorithm by successfully training large ImageNet models (InceptionV3, ResnetV1-50, ResnetV1-101 and InceptionResnetV2) with mini-batch sizes of up to 32000 with no loss in validation error relative to current baselines, and no increase in the total number of steps.

At smaller mini-batch sizes, our optimizer improves the validation error in these models by 0.8-0.9\%.

Alternatively, we can trade off this accuracy to reduce the number of training steps needed by roughly 10-30\%.

Our work is practical and easily usable by others -- only one hyperparameter (learning rate) needs tuning, and furthermore, the algorithm is as computationally cheap as the commonly used Adam optimizer.

We describe a practical optimization algorithm for deep neural networks that works faster and generates better models compared to widely used algorithms.

Proposes a new algorithm where they claim to use Hessian implicitly and are using a motivation from power-series

Presents a new 2nd-order algorithm that implicitly uses curvature information and shows the intuition behind the approximation schemes in the algorithms and validates the heuristics in various experiments.

Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks

practical optimization algorithm deep neural network


Recent work has shown that performing inference with fast, very-low-bitwidth

(e.g., 1 to 2 bits) representations of values in models can yield surprisingly accurate

However, although 2-bit approximated networks have been shown to

be quite accurate, 1 bit approximations, which are twice as fast, have restrictively

We propose a method to train models whose weights are a mixture

of bitwidths, that allows us to more finely tune the accuracy/speed trade-off.

present the “middle-out” criterion for determining the bitwidth for each value, and

show how to integrate it into training models with a desired mixture of bitwidths.

We evaluate several architectures and binarization techniques on the ImageNet

We show that our heterogeneous bitwidth approximation achieves superlinear

Using an average of only 1.4 bits, we are

able to outperform state-of-the-art 2-bit architectures.

We introduce fractional bitwidth approximation and show it has significant advantages.

Suggests a method for varying the degree of quantization in a neural network during the forward propagation phase

Maintaining the accuracy of 2bits netword while using less than 2bits weights

Heterogeneous Bitwidth Binarization in Convolutional Neural Networks


Pruning units in a deep network can help speed up inference and training as well as reduce the size of the model.

We show that bias propagation is a pruning technique which consistently outperforms the common approach of merely removing units,  regardless of the architecture and the dataset.

We also show how a simple adaptation to an existing scoring function allows us to select the best units to prune.

Finally,  we show that the units selected by the best performing scoring functions are somewhat consistent over the course of training, implying the dead parts of the network appear during the stages of training.

Mean Replacement is an efficient method to improve the loss after pruning and Taylor approximation based scoring functions works better with absolute values.

Proposes a simple improvement to methods for unit pruning using "mean replacement"

This paper presents a mean-replacement pruning strategy and utilizes the absolute-valued Taylor expansion as the scoring function for the pruning

Mean Replacement Pruning

mean replacement
scoring function
method
pruning
taylor


Due to the phenomenon of “posterior collapse,” current latent variable generative models pose a challenging design choice that either weakens the capacity of the decoder or requires altering the training objective.

We develop an alternative that utilizes the most powerful generative models as decoders, optimize the variational lower bound, and ensures that the latent variables preserve and encode useful information.

Our proposed δ-VAEs achieve this by constraining the variational family for the posterior to have a minimum distance to the prior.

For sequential latent variable models, our approach resembles the classic representation learning approach of slow feature analysis.

We demonstrate our method’s efficacy at modeling text on LM1B and modeling images: learning representations, improving sample quality, and achieving state of the art log-likelihood on CIFAR-10 and ImageNet 32 × 32.

Avoid posterior collapse by lower bounding the rate.

Presents an approach to preventing posterior collapse in VAEs by limiting the family of the variational approximation to the posterior

This paper introduces a constraint on the family of variational posteriors such that the KL term can be controlled to combat posterior collapse in deep generative models such as VAEs

Preventing Posterior Collapse with delta-VAEs

family variational
prevent posterior collapse
vaes


Mini-batch gradient descent and its variants are commonly used in deep learning.

The principle of mini-batch gradient descent is to use noisy gradient calculated on a batch to estimate the real gradient, thus balancing the computation cost per iteration and the uncertainty of noisy gradient.

However, its batch size is a fixed hyper-parameter requiring manual setting before training the neural network.

Yin et al. (2017) proposed a batch adaptive stochastic gradient descent (BA-SGD) that can dynamically choose a proper batch size as learning proceeds.

We extend the BA-SGD to momentum algorithm and evaluate both the BA-SGD and the batch adaptive momentum (BA-Momentum) on two deep learning tasks from natural language processing to image classification.

Experiments confirm that batch adaptive methods can achieve a lower loss compared with mini-batch methods after scanning the same epochs of data.

Furthermore, our BA-Momentum is more robust against larger step sizes, in that it can dynamically enlarge the batch size to reduce the larger uncertainty brought by larger step sizes.

We also identified an interesting phenomenon, batch size boom.

The code implementing batch adaptive framework is now open source, applicable to any gradient-based optimization problems.

We developed a batch adaptive momentum that can achieve lower loss compared with mini-batch methods after scanning same epochs of data, and it is more robust against large step size.

This paper addresses the problem of automatically tuning batch size during deep learning training, and claims to extend batch adaptive SGD to adaptive momentum and adopt the algorithms to complex neural networks problems.

The paper proposes generalizing an algorithm which performs SGD with adaptive batch sizes by adding momentum to the utility function

On Batch Adaptive Training for Deep Learning: Lower Loss and Larger Step Size

large step size
adaptive momentum
deep learning
algorithm
batch
low loss
training
sgd


Deep learning models for graphs have advanced the state of the art on many tasks.

Despite their recent success, little is known about their robustness.

We investigate training time attacks on graph neural networks for node classification that perturb the discrete graph structure.

Our core principle is to use meta-gradients to solve the bilevel problem underlying training-time attacks, essentially treating the graph as a hyperparameter to optimize.

Our experiments show that small graph perturbations consistently lead to a strong decrease in performance for graph convolutional networks, and even transfer to unsupervised embeddings.

Remarkably, the perturbations created by our algorithm can misguide the graph neural networks such that they perform worse than a simple baseline that ignores all relational information.

Our attacks do not assume any knowledge about or access to the target classifiers.

We use meta-gradients to attack the training procedure of deep neural networks for graphs.

Studies the problem of learning a better poisoned graph parameters that can maximize the loss of a graph neural network.

An algorithm to alter graph structure by adding/deleting edges so as to degrade the global performance of node classification, and the idea to use meta-learning to solve the bilevel optimization problem.

Adversarial Attacks on Graph Neural Networks via Meta Learning


Numerous models for grounded language understanding have been recently proposed, including

(i) generic models that can be easily adapted to any given task and

(ii) intuitively appealing modular models that require background knowledge to be instantiated.

We compare both types of models in how much they lend themselves to a particular form of systematic generalization.

Using a synthetic VQA test, we evaluate which models are capable of reasoning about all possible object pairs after training on only a small subset of them.

Our findings show that the generalization of modular models is much more systematic and that it is highly sensitive to the module layout, i.e. to how exactly the modules are connected.

We furthermore investigate if modular models that generalize well could be made more end-to-end by learning their layout and parametrization.

We find that end-to-end methods from prior work often learn inappropriate layouts or parametrizations that do not facilitate systematic generalization.

Our results suggest that, in addition to modularity, systematic generalization in language understanding may require explicit regularizers or priors.

We show that modular structured models are the best in terms of systematic generalization and that their end-to-end versions don't generalize as well.

This paper evaluates systemic generalization between modular neural networks and otherwise generic models via introduction of a new, spatial reasoning dataset

A targeted empirical evaluation of generalization in models for visual reasoning, focused on the problem of recognizing (object, relation, object) triples in synthetic scenes featuring letters and numbers.

Systematic Generalization: What Is Required and Can It Be Learned?

systematic generalization
model
modular
reasoning


The behavioral dynamics of multi-agent systems have a rich and orderly structure, which can be leveraged to understand these systems, and to improve how artificial agents learn to operate in them.

Here we introduce Relational Forward Models (RFM) for multi-agent learning, networks that can learn to make accurate predictions of agents' future behavior in multi-agent environments.

Because these models operate on the discrete entities and relations present in the environment, they produce interpretable intermediate representations which offer insights into what drives agents' behavior, and what events mediate the intensity and valence of social interactions.

Furthermore, we show that embedding RFM modules inside agents results in faster learning systems compared to non-augmented baselines.

As more and more of the autonomous systems we develop and interact with become multi-agent in nature, developing richer analysis tools for characterizing how and why agents make decisions is increasingly necessary.

Moreover, developing artificial agents that quickly and safely learn to coordinate with one another, and with humans in shared environments, is crucial.

Relational Forward Models for multi-agent learning make accurate predictions of agents' future behavior, they produce intepretable representations and can be used inside agents.

A way of reducing variance in model free learning by having an explicit model, that uses a graph conv net-like architecture, of actions that other agents will take.

Predicting multi-agent behavior using a relational forward model with a recurrent component, outperforming two baselines and two ablations

Relational Forward Models for Multi-Agent Learning

relational forward model multi-agent learning


We show that gradient descent on an unregularized logistic regression

problem, for almost all separable datasets, converges to the same direction as the max-margin solution.

The result generalizes also to other monotone decreasing loss functions with an infimum at infinity, and we also discuss a multi-class generalizations to the cross entropy loss.

we show this convergence is very slow, and only logarithmic in the

of continuing to optimize the logistic or cross-entropy loss even

after the training error is zero and the training loss is extremely

small, and, as we show, even if the validation loss increases.

methodology can also aid in understanding implicit regularization

in more complex models and with other optimization methods.

The normalized solution of gradient descent on logistic regression (or a similarly decaying loss) slowly converges to the L2 max margin solution on separable data.

The paper offers a formal proof that gradient descent on the logistic loss converges very slowly to the hard SVM solution in the case where the data are linearly separable.

This paper focuses on characterising the behaviour of log loss minimisation on linearly separable data, and shows that log-loss, minimised with gradient descent, leads to convergence to the max-margin solution.

The Implicit Bias of Gradient Descent on Separable Data

gradient descent logistic
separable datum
converge
solution
slowly
linearly
loss


Despite impressive performance as evaluated on i.i.d. holdout data, deep neural networks depend heavily on superficial statistics of the training data and are liable to break under distribution shift.

For example, subtle changes to the background or texture of an image can break a seemingly powerful classifier.

Building on previous work on domain generalization, we hope to produce a classifier that will generalize to previously unseen domains, even when domain identifiers are not available during training.

This setting is challenging because the model may extract many distribution-specific (superficial) signals together with distribution-agnostic (semantic) signals.

To overcome this challenge, we incorporate the gray-level co-occurrence matrix (GLCM) to extract patterns that our prior knowledge suggests are superficial: they are sensitive to the texture but unable to capture the gestalt of an image.

Then we introduce two techniques for improving our networks' out-of-sample performance.

The first method is built on the reverse gradient method that pushes our model to learn representations from which the GLCM representation is not predictable.

The second method is built on the independence introduced by projecting the model's representation onto the subspace orthogonal to GLCM representation's.

We test our method on the battery of standard domain generalization data sets and, interestingly, achieve comparable or better performance as compared to other domain generalization methods that explicitly require samples from the target distribution for training.

A domain generalization approach to reveal semantic information based on a linear projection scheme from CNN and NGLCM output layers.

The paper proposes an unsupervised approach to identify image features that are not meaningful for image classification tasks

Learning Robust Representations by Projecting Superficial Statistics Out


In this paper, we conduct an intriguing experimental study about the physical adversarial attack on object detectors in the wild.

In particular, we learn a camouflage pattern to hide vehicles from being detected by state-of-the-art convolutional neural network based detectors.

Our approach alternates between two threads.

In the first, we train a neural approximation function to imitate how a simulator applies a camouflage to vehicles and how a vehicle detector performs given images of the camouflaged vehicles.

In the second, we minimize the approximated detection score by searching for the optimal camouflage.

Experiments show that the learned camouflage can not only hide a vehicle from the image-based detectors under many test cases but also generalizes to different environments, vehicles, and object detectors.

We propose a method to learn physical vehicle camouflage to adversarially attack object detectors in the wild. We find our camouflage effective and transferable.

The authors investigate the problem of learning a camouflage pattern which, when applied to a simulated vehicle, will prevent an object detector from detecting it.

This paper targets adversarial learning for interfering car detection by learning camouflage patterns

CAMOU: Learning Physical Vehicle Camouflages to Adversarially Attack Detectors in the Wild

learn physical vehicle camouflage adversarially attack


As deep learning-based classifiers are increasingly adopted in real-world applications, the importance of understanding how a particular label is chosen grows.

Single decision trees are an example of a simple, interpretable classifier, but are unsuitable for use with complex, high-dimensional data.

On the other hand, the variational autoencoder (VAE) is designed to learn a factored, low-dimensional representation of data, but typically encodes high-likelihood data in an intrinsically non-separable way.

We introduce the differentiable decision tree (DDT) as a modular component of deep networks and a simple, differentiable loss function that allows for end-to-end optimization of a deep network to compress high-dimensional data for classification by a single decision tree.

We also explore the power of labeled data in a  supervised VAE (SVAE) with a Gaussian mixture prior, which leverages label information to produce a high-quality generative model with improved bounds on log-likelihood.

We combine the SVAE with the DDT to get our classifier+VAE (C+VAE), which is competitive in both classification error and log-likelihood, despite optimizing both simultaneously and using a very simple encoder/decoder architecture.

We combine differentiable decision trees with supervised variational autoencoders to enhance interpretability of classification.

This paper proposes a hybrid model of a variational autoencoder composed with a differentiable decision tree, and an accompanying training scheme, with experiments demonstrating tree classification performance, neg. log likelihood performance, and latent space interpretability.

The paper tries to build an interpretable and accurate classifier via stacking a supervised VAE and a differentiable decision tree

Interpretable Classification via Supervised Variational Autoencoders and Differentiable Decision Trees

differentiable decision tree
supervised variational autoencoder
interpretability
classification
via
interpretable


We propose Regularized Learning under Label shifts (RLLS), a principled and a practical domain-adaptation algorithm to correct for shifts in the label distribution between a source and a target domain.

We first estimate importance weights using labeled source data and unlabeled target data, and then train a classifier on the weighted source samples.

We derive a generalization bound for the classifier on the target domain which is independent of the (ambient) data dimensions, and instead only depends on the complexity of the function class.

To the best of our knowledge, this is the first generalization bound for the label-shift problem where the labels in the target domain are not available.

Based on this bound, we propose a regularized estimator for the small-sample regime which accounts for the uncertainty in the estimated weights.

Experiments on the CIFAR-10 and MNIST datasets show that RLLS improves classification accuracy, especially in the low sample and large-shift regimes, compared to previous methods.

A practical and provably guaranteed approach for training efficiently classifiers in the presence of label shifts between Source and Target data sets

The authors propose a new algorithm for improving the stability of class importance weighting estimation procedure with a two-step procedure.

The authors consider the problem of learning under label shifts, where label proportions differ while conditionals are equal, and propose an improved estimator with regularization.

Regularized Learning for  Domain Adaptation under Label Shifts


The statistics of the real visual world presents a long-tailed distribution: a few classes have significantly more training instances than the remaining classes in a dataset.

This is because the real visual world has a few classes that are common while others are rare.

Unfortunately, the performance of a convolutional neural network is typically unsatisfactory when trained using a long-tailed dataset.

To alleviate this issue, we propose a method that discriminatively learns an embedding in which a simple Bayesian classifier can balance the class-priors to generalize well for rare classes.

To this end, the proposed approach uses a Gaussian mixture model to factor out class-likelihoods and class-priors in a long-tailed dataset.

The proposed method is simple and easy-to-implement in existing deep learning frameworks.

Experiments on publicly available datasets show that the proposed approach improves the performance on classes with few training instances, while maintaining a comparable performance to the state-of-the-art on classes with abundant training examples.

Approach to improve classification accuracy on classes in the tail.

The main goal of this paper is to learn a ConvNet classifier which performs better for classes in the tail of the class occurrence distribution.

Proposal for a Bayesian framework with a Gaussian mixture model to address an issue in classification applications, that the number of training data from different classes is unbalanced.

Bayesian Embeddings for Long-Tailed Datasets


As deep reinforcement learning is being applied to more and more tasks, there is a growing need to better understand and probe the learned agents.

Visualizing and understanding the decision making process can be very valuable to comprehend and identify problems in the learned behavior.

However, this topic has been relatively under-explored in the reinforcement learning community.

In this work we present a method for synthesizing states of interest for a trained agent.

Such states could be situations (e.g. crashing or damaging a car) in which specific actions are necessary.

Further, critical states in which a very high or a very low reward can be achieved (e.g. risky states) are often interesting to understand the situational awareness of the system.

To this end, we learn a generative model over the state space of the environment and use its latent space to optimize a target function for the state of interest.

In our experiments we show that this method can generate insightful visualizations for a variety of environments and reinforcement learning methods.

We explore these issues in the standard Atari benchmark games as well as in an autonomous driving simulator.

Based on the efficiency with which we have been able to identify significant decision scenarios with this technique, we believe this general approach could serve as an important tool for AI safety applications.

We present a method to synthesize states of interest for reinforcement learning agents in order to analyze their behavior.

This paper proposes a generative model of visual observations in RL that is capable of generating observations of interests.

An approach for visualizing states of interest that involves a variational autoencoder that learns to reconstruct state space and an optimization step that finds conditioning parameters to generate synthetic images.

Visualizing and Discovering Behavioural Weaknesses in Deep Reinforcement Learning

state interest
reinforcement learning
generate
visualize


We introduce the deep abstaining classifier -- a deep neural network trained with a novel loss function that provides an abstention option during training.

This allows the  DNN to abstain on confusing or difficult-to-learn examples while improving performance on the non-abstained samples.

We show that such deep abstaining classifiers can:

(i) learn representations for structured noise -- where noisy training labels or confusing examples are correlated with underlying features -- and then learn to abstain based on such features;

(ii) enable robust learning in the presence of arbitrary or unstructured noise by identifying noisy samples; and

(iii) be used as an effective out-of-category detector that learns to reliably abstain when presented with samples from  unknown classes.

We provide analytical results on loss function behavior that enable automatic tuning of accuracy and coverage, and demonstrate the utility of the deep abstaining classifier using multiple image benchmarks, Results indicate significant improvement in learning in the presence of label noise.

A deep abstaining neural network trained with a novel loss function that learns representations for when to abstain enabling robust learning in the presence of different types of noise.

A new loss function for training a deep neural network which can abstain, with performance looked at from angles in existence of structured noise, in existence of unstructured noise, and open world detection.

This manuscript introduces deep abstaining classifiers which modifies the multiclass cross-entropy loss with an abstention loss, which is then applied to perturbed image classification tasks

Knows When it Doesn’t Know: Deep Abstaining Classifiers

abstain
noise
train
loss function
neural network
deep
classifier


Temporal Difference learning with function approximation has been widely used recently and has led to several successful results.

However, compared with the original tabular-based methods, one major drawback of temporal difference learning with neural networks and other function approximators is that they tend to over-generalize across temporally successive states, resulting in slow convergence and even instability.

In this work, we propose a novel TD learning method, Hadamard product Regularized TD (HR-TD), that reduces over-generalization and thus leads to faster convergence.

This approach can be easily applied to both linear and nonlinear function approximators.

HR-TD is evaluated on several linear and nonlinear benchmark domains, where we show improvement in learning behavior and performance.

A regularization technique for TD learning that avoids temporal over-generalization, especially in Deep Networks

A variation on temporal difference learning for the function approximation case that attempts to resolve the issue of over-generalization across temporally-successive states.

The paper introduces HR-TD, a variation of the TD(0) algorithm, meant to improve the over-generalization problem in conventional TD

HR-TD: A Regularized TD Method to Avoid Over-Generalization

variation hr-td avoid learn over-generalization temporal td


We present an efficient convolution kernel for Convolutional Neural Networks (CNNs) on unstructured grids using parameterized differential operators while focusing on spherical signals such as panorama images or planetary signals.

To this end, we replace conventional convolution kernels with linear combinations of differential operators that are weighted by learnable parameters.

Differential operators can be efficiently estimated on unstructured grids using one-ring neighbors, and learnable parameters can be optimized through standard back-propagation.

As a result, we obtain extremely efficient neural networks that match or outperform state-of-the-art network architectures in terms of performance but with a significantly lower number of network parameters.

We evaluate our algorithm in an extensive series of experiments on a variety of computer vision and climate science tasks, including shape classification, climate pattern segmentation, and omnidirectional image semantic segmentation.

Overall, we present (1) a novel CNN approach on unstructured grids using parameterized differential operators for spherical signals, and (2) we show that our unique kernel parameterization allows our model to achieve the same or higher accuracy with significantly fewer network parameters.

We present a new CNN kernel for unstructured grids for spherical signals, and show significant accuracy and parameter efficiency gain on tasks such as 3D classfication and omnidirectional image segmentation.

An efficient method enabling deep learning on spherical data that reaches competitive/state-of-the-art numbers with much less parameters than popular approaches.

The paper proposes a novel convolutional kernel for CNN on the unstructured grids and formulates the convolution by a linear combination of differential operators.

Spherical CNNs on Unstructured Grids

unstructured grid
spherical
kernel
parameter
cnn


Prediction is arguably one of the most basic functions of an intelligent system.

In general, the problem of predicting events in the future or between two waypoints is exceedingly difficult.

However, most phenomena naturally pass through relatively predictable bottlenecks---while we cannot predict the precise trajectory of a robot arm between being at rest and holding an object up, we can be certain that it must have picked the object up.

To exploit this, we decouple visual prediction from a rigid notion of time.

While conventional approaches predict frames at regularly spaced temporal intervals, our time-agnostic predictors (TAP) are not tied to specific times so that they may instead discover predictable "bottleneck" frames no matter when they occur.

We evaluate our approach for future and intermediate frame prediction across three robotic manipulation tasks.

Our predictions are not only of higher visual quality, but also correspond to coherent semantic subgoals in temporally extended tasks.

In visual prediction tasks, letting your predictive model choose which times to predict does two things: (i) improves prediction quality, and (ii) leads to semantically coherent "bottleneck state" predictions, which are useful for planning.

A method on prediction of frames in a video, the approach including that target prediction is floating, resolved by a minimum on the error of prediction.

Reformulates the task of video prediction/interpolation so that a predictor is not forced to generate frames at fixed time intervals, but instead is trained to generate frames that happen at any point in the future.

Time-Agnostic Prediction: Predicting Predictable Video Frames

prediction task predict frame video time


In cities with tall buildings, emergency responders need an accurate floor level location to find 911 callers quickly.

We introduce a system to estimate a victim's floor level via their mobile device's sensor data in a two-step process.

First, we train a neural network to determine when a smartphone enters or exits a building via GPS signal changes.

Second, we use a barometer equipped smartphone to measure the change in barometric pressure from the entrance of the building to the victim's indoor location.

Unlike impractical previous approaches, our system is the first that does not require the use of beacons, prior knowledge of the building infrastructure, or knowledge of user behavior.

We demonstrate real-world feasibility through 63 experiments across five different tall buildings throughout New York City where our system predicted the correct floor level with 100% accuracy.

We used an LSTM to detect when a smartphone walks into a building. Then we predict the device's floor level using data from sensors aboard the smartphone.

The paper introduces a system to estimate a floor-level via their mobile device's sensor data using an LSTM and changes in barometric pressure

Proposal for a two-step method to determine which floor a mobile phone is on inside a tall building.

Predicting Floor-Level for 911 Calls with Neural Networks and Smartphone Sensor Data

lstm
datum
smartphone
building
device 's
mobile
floor-level
floor
sensor
predict


Sparse reward is one of the most challenging problems in reinforcement learning (RL).

Hindsight Experience Replay (HER) attempts to address this issue by converting a failure experience to a successful one by relabeling the goals.

Despite its effectiveness, HER has limited applicability because it lacks a compact and universal goal representation.

We present Augmenting experienCe via TeacheR's adviCE (ACTRCE), an efficient reinforcement learning technique that extends the HER framework using natural language as the goal representation.

We first analyze the differences among goal representation, and show that ACTRCE can efficiently solve difficult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn.

We also show that with language goal representations, the agent can generalize to unseen instructions, and even generalize to instructions with unseen lexicons.

We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, but we also found that little amount of hindsight advice is sufficient for the learning to take off, showing the practical aspect of the method.

Combine language goal representation with hindsight experience replays.

This paper considers the assumption implicit in hindsight experience replay, that there is access to a mapping from states to goals, and proposes a natural language goal representation.

This submission uses Hindsight Experience Replay framework with natural language goals to improve the sample-efficiency of instruction-following models.

ACTRCE: Augmenting Experience via Teacher’s Advice

hindsight experience replay
natural language goal
representation


Learning rich and compact representations is an open topic in many fields such as word embedding, visual question-answering, object recognition or image retrieval.

Although deep neural networks (convolutional or not) have made a major breakthrough during the last few years by providing hierarchical, semantic and abstract representations for all of these tasks, these representations are not necessary as rich as needed nor as compact as expected.

Models using higher order statistics, such as bilinear pooling, provide richer representations at the cost of higher dimensional features.

Factorization schemes have been proposed but without being able to reach the original compactness of first order models, or at a heavy loss in performances.

This paper addresses these two points by extending factorization schemes to codebook strategies, allowing compact representations with the same dimensionality as first order representations, but with second order performances.

Moreover, we extend this framework with a joint codebook and factorization scheme, granting a reduction both in terms of parameters and computation cost.

This formulation leads to state-of-the-art results and compact second-order models with few additional parameters and intermediate representations with a dimension similar to that of first-order statistics.

We propose a joint codebook and factorization scheme to improve second order pooling.

This paper presents a way to combine existing factorized second order representations with a codebook style hard assignment.

Proposal for a novel bilinear representation based on a codebook model, and an efficient formulation in which codebook-based projections are factorized via shared projection to further reduce parameter size.

Efficient Codebook and Factorization for Second Order Representation Learning

second order representation
codebook factorization
factorize
efficient


Natural language understanding research has recently shifted towards complex Machine Learning and Deep Learning algorithms.

Such models often outperform their simpler counterparts significantly.

However, their performance relies on the availability of large amounts of labeled data, which are rarely available.

To tackle this problem, we propose a methodology for extending training datasets to arbitrarily big sizes and training complex, data-hungry models using weak supervision.

We apply this methodology on biomedical relation extraction, a task where training datasets are excessively time-consuming and expensive to create, yet has a major impact on downstream applications such as drug discovery.

We demonstrate in two small-scale controlled experiments that our method consistently enhances the performance of an LSTM network, with performance improvements comparable to hand-labeled training data.

Finally, we discuss the optimal setting for applying weak supervision using this methodology.

We propose and apply a meta-learning methodology based on Weak Supervision, for combining Semi-Supervised and Ensemble Learning on the task of Biomedical Relationship Extraction.

A semi-supervised method for relation classification, which trains multiple base learners using a small labeled dataset and applies some of them to annotate unlabeled examples for semi-supervised learning.

This paper addresses the problem of generating training data for biological relation extraction, and uses predictions from data labeled by weak classifiers as additional training data for a meta learning algorithm.

This paper proposes a combination of semi-supervised learning and ensemble learning for information extraction, with experiments conducted on a biomedical relation extraction task

Semi-supervised Ensemble Learning with Weak Supervision for Biomedical Relationship Extraction

apply
biomedical relationship extraction
ensemble learning
weak supervision
semi-supervised
base
task
relation
label


We introduce contextual explanation networks (CENs)---a class of models that learn to predict by generating and leveraging intermediate explanations.

CENs are deep networks that generate parameters for context-specific probabilistic graphical models which are further used for prediction and play the role of explanations.

Contrary to the existing post-hoc model-explanation tools, CENs learn to predict and to explain jointly.

Our approach offers two major advantages:

(i) for each prediction, valid instance-specific explanations are generated with no computational overhead and

(ii) prediction via explanation acts as a regularization and boosts performance in low-resource settings.

We prove that local approximations to the decision boundary of our networks are consistent with the generated explanations.

Our results on image and text classification and survival analysis tasks demonstrate that CENs are competitive with the state-of-the-art while offering additional insights behind each prediction, valuable for decision support.

A class of networks that generate simple models on the fly (called explanations) that act as a regularizer and enable consistent model diagnostics and interpretability.

The authors claim that the previous art directly integrate neural networks into the graphical models as components, which renders the models uninterpretable.

Proposal for a combination of neural nets and graphical models by using a deep neural net to predict the parameters of a graphical model.

Contextual Explanation Networks

graphical model
network
explanation
neural


The goal of imitation learning (IL) is to enable a learner to imitate an expert’s behavior given the expert’s demonstrations.

Recently, generative adversarial imitation learning (GAIL) has successfully achieved it even on complex continuous control tasks.

However, GAIL requires a huge number of interactions with environment during training.

We believe that IL algorithm could be more applicable to the real-world environments if the number of interactions could be reduced.

To this end, we propose a model free, off-policy IL algorithm for continuous control.

The keys of our algorithm are two folds:

1) adopting deterministic policy that allows us to derive a novel type of policy gradient which we call deterministic policy imitation gradient (DPIG),

2) introducing a function which we call state screening function (SSF) to avoid noisy policy updates with states that are not typical of those appeared on the expert’s demonstrations.

Experimental results show that our algorithm can achieve the goal of IL with at least tens of times less interactions than GAIL on a variety of continuous control tasks.

We propose a model free imitation learning algorithm that is able to reduce number of interactions with environment in comparison with state-of-the-art imitation learning algorithm namely GAIL.

Proposes to extend the determinist policy gradient algorithm to learn from demonstrations, while combined with a type of density estimation of the expert.

This paper considers the problem of model-free imitation learning and proposes an extension of the generative adversarial imitation learning algorithm by replacing the stochastic policy of the learner with a deterministic one.

The paper combines IRL, adversarial training, and ideas from deterministic policy gradients with the goal of decreasng sample complexity

Deterministic Policy Imitation Gradient Algorithm

imitation learn algorithm
combine
policy gradient
deterministic
adversarial


Convolution acts as a local feature extractor in convolutional neural networks (CNNs).

However, the convolution operation is not applicable when the input data is supported on an irregular graph such as with social networks, citation networks, or knowledge graphs.

This paper proposes the topology adaptive graph convolutional network (TAGCN), a novel graph convolutional network that generalizes CNN architectures to graph-structured data and provides a systematic way to design a set of fixed-size learnable filters to perform convolutions on graphs.

The topologies of these filters are adaptive to the topology of the graph when they scan the graph to perform convolution, replacing the square filter for the grid-structured data in traditional CNNs.

The outputs are the weighted sum of these filters’ outputs, extraction of both vertex features and strength of correlation between vertices.

can be used with both directed and undirected graphs.

The proposed TAGCN not only inherits the properties of convolutions in CNN for grid-structured data, but it is also consistent with convolution as defined in graph signal processing.

Further, as no approximation to the convolution is needed, TAGCN exhibits better performance than existing graph-convolution-approximation methods on a number

As only the polynomials of degree two of the adjacency matrix are used, TAGCN is also computationally simpler than other recent methods.

Low computational complexity graph CNN (without approximation) with better classification accuracy

Proposes a new CNN approach to graph classification using a filter based on outgoing walks of increasing length to incorporate information from more distant vertices in one propagation step.

Proposal for a new neural network architecture for semi-supervised graph classification, building upon graph polynomial filters and utilizing them on successive neural network layers with ReLU activation functions.

The paper introduces Topology Adaptive GCN to generalize convolutional networks to graph-structured data

Topology Adaptive Graph Convolutional  Networks

new
graph classification
filter
topology adaptive
cnn
network
convolutional


Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks.

Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift.

We define a ``forgetting event'' to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning.

Across several benchmark data sets, we find that:

(i) certain examples are forgotten with high frequency, and some not at all;

(ii) a data set's (un)forgettable examples generalize across neural architectures; and

(iii) based on forgetting dynamics, a significant fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.

We show that catastrophic forgetting occurs within what is considered to be a single task and find that examples that are not prone to forgetting can be removed from the training set without loss of generalization.

Studies the forgetting behavior of training examples during SGD, and shows there exist "support examples" in neural network training across different network architectures.

This paper analyzes the extent to which networks learn to correctly classify specific examples and then forget these examples over the course of training.

The paper studies whether some examples in training neural networks are harder to learn than others. Such examples are forgotten and relearned multiple times through learning.

An Empirical Study of Example Forgetting during Deep Neural Network Learning

example
forget
training
show
neural network
train
study
learn


Discovering objects and their attributes is of great importance for autonomous agents to effectively operate in human environments.

This task is particularly challenging due to the ubiquitousness of objects and all their nuances in perceptual and semantic detail.

In this paper we present an unsupervised approach for learning disentangled representations of objects entirely from unlabeled monocular videos.

These continuous representations are not biased by or limited by a discrete set of labels determined by human labelers.

The proposed representation is trained with a metric learning loss, where objects with homogeneous features are pushed together, while those with heterogeneous features are pulled apart.

We show these unsupervised embeddings allow to discover object attributes and can enable robots to self-supervise in previously unseen environments.

We quantitatively evaluate performance on a large-scale synthetic dataset with 12k object models, as well as on a real dataset collected by a robot and show that our unsupervised object understanding generalizes to previously unseen objects.

Specifically, we demonstrate the effectiveness of our approach on robotic manipulation tasks, such as pointing at and grasping of objects.

An interesting and perhaps surprising finding in this approach is that given a limited set of objects, object correspondences will naturally emerge when using metric learning without requiring explicit positive pairs.

An unsupervised approach for learning disentangled representations of objects entirely from unlabeled monocular videos.

Designs a feature representation from video sequences captured from a scene from different view points.

Proposal for an unsupervised representation learning method for visual inputs that incorporates a metric learning approach pulling nearest neighbor pairs of image patches close in embedding space while pushing apart other pairs.

This paper explores self-supervised learning of object representations, with the main idea to encourage objects with similar features to get further ‘attracted’ to each other.

Object-Contrastive Networks: Unsupervised Object Representations

object video representation unsupervised feature approach learning