A widely observed phenomenon in deep learning is the degradation problem: increasing

the depth of a network leads to a decrease in performance on both test and training data.

Novel architectures such as ResNets and Highway networks have addressed this issue by introducing various flavors of skip-connections or gating mechanisms.

However, the degradation problem persists in the context of plain feed-forward networks.

In this work we propose a simple method to address this issue.

The proposed method poses the learning of weights in deep networks as a constrained optimization problem where the presence of skip-connections is penalized by Lagrange multipliers.

This allows for skip-connections to be introduced during the early stages of training and subsequently phased out in a principled manner.

We demonstrate the benefits of such an approach with experiments on MNIST, fashion-MNIST, CIFAR-10 and CIFAR-100 where the proposed method is shown to greatly decrease the degradation effect (compared to plain networks) and is often competitive with ResNets.

Phasing out skip-connections in a principled manner avoids degradation in deep feed-forward networks.

The authors present a new training strategy, VAN, for training very deep feed-forward networks without skip connections

The paper introduces an architecture that linearly interpolates between ResNets and vanilla deep nets without skip connections.

Avoiding degradation in deep feed-forward networks by phasing out skip-connections

avoid degradation deep feed-forward network


Deep learning is becoming more widespread in its application due to its power in solving complex classification problems.

However, deep learning models often require large memory and energy consumption, which may prevent them from being deployed effectively on embedded platforms, limiting their applications.

This work addresses the problem by proposing methods {\em Weight Reduction Quantisation} for compressing the memory footprint of the models, including reducing the number of weights and the number of bits to store each weight.

Beside, applying with sparsity-inducing regularization, our work focuses on speeding up stochastic variance reduced gradients (SVRG) optimization on non-convex problem.

Our method that mini-batch SVRG with $\ell$1 regularization on non-convex problem has faster and smoother convergence rates than SGD by using adaptive learning rates.

Experimental evaluation of our approach uses MNIST and CIFAR-10 datasets on LeNet-300-100 and LeNet-5 models, showing our approach can reduce the memory requirements both in the convolutional and fully connected layers by up to 60$\times$ without affecting their test accuracy.

Compression of Deep neural networks deployed on embedded device.

The authors present an l-1 regularized SVRG based training algorithm that is able to force many weights of the network to be 0.

Sparse Regularized Deep Neural Networks For Efficient Embedded Learning


It has been argued that the brain is a prediction machine that continuously learns how to make better predictions about the stimuli received from the external environment.

For this purpose, it builds a model of the world around us and uses this model to infer the external stimulus.

Predictive coding has been proposed as a mechanism through which the brain might be able to build such a model of the external environment.

However, it is not clear how predictive coding can be used to build deep neural network models of the brain while complying with the architectural constraints imposed by the brain.

In this paper, we describe an algorithm to build a deep generative model using predictive coding that can be used to infer latent representations about the stimuli received from external environment.

Specifically, we used predictive coding to train a deep neural network on real-world images in a unsupervised learning paradigm.

To understand the capacity of the network with regards to modeling the external environment, we studied the latent representations generated by the model on images of objects that are never presented to the model during training.

Despite the novel features of these objects the model is able to infer the latent representations for them.

Furthermore, the reconstructions of the original images obtained from these latent representations preserve the important details of these objects.

A predictive coding based learning algorithm for building deep neural network models of the brain

The paper considers learning of a generative neural network using a predictive coding setup

A Deep Predictive Coding Network for Learning Latent Representations

predictive code
neural network
learn
deep


In this paper, we propose deep convolutional generative adversarial networks (DCGAN) that learn to produce a 'mental image' of the input image as internal representation of a certain category of input data distribution.

This mental image is what the DCGAN 'imagines' that the input image might look like under ideal conditions.

The mental image contains a version of the input that is iconic, without any peculiarities that do not contribute to the ideal representation of the input data distribution within a category.

A DCGAN learns this association by training an encoder to capture salient features from the original image and a decoder to convert salient features into its associated mental image representation.

Our new approach, which we refer to as a Mental Image DCGAN (MIDCGAN), learns features that are useful for recognizing entire classes of objects, and that this in turn has the benefit of helping single and zero shot recognition.

We demonstrate our approach on object instance recognition and handwritten digit recognition tasks.

Object instance recognition with adversarial autoencoders was performed with a novel 'mental image' target that is canonical representation of the input image.

The paper proposes a method to learn features for object recognition that is invariant to various transformations of the object, most notably object pose.

This paper investigated the task of few shot recognition via a generated “mental image” as intermediate representation given the input image.

Learning with Mental Imagery

input image
mental
recognition
learn
object
representation


An obstacle that prevents the wide adoption of (deep) reinforcement learning (RL) in control systems is its need for a large number of interactions with the environment in order to master a skill.

The learned skill usually generalizes poorly across domains and re-training is often necessary when presented with a new task.

We present a framework that combines techniques in \textit{formal methods} with \textit{hierarchical reinforcement learning} (HRL).

The set of techniques we provide allows for the convenient specification of tasks with logical expressions, learns hierarchical policies (meta-controller and low-level controllers) with well-defined intrinsic rewards using any RL methods and is able to construct new skills from existing ones without additional learning.

We evaluate the proposed methods in a simple grid world simulation as well as simulation on a Baxter robot.

Combine temporal logic with hierarchical reinforcement learning for skill composition

The paper offers a strategy for constructing a product MDP out of an original MDP and the automaton associated with an LTL formula.

Proposes to join temporal logic with hierarchical reinforcement learning to simplify skill composition.

AUTOMATA GUIDED HIERARCHICAL REINFORCEMENT LEARNING FOR ZERO-SHOT SKILL COMPOSITION

temporal logic hierarchical reinforcement learning


The tremendous memory and computational complexity of Convolutional Neural Networks (CNNs) prevents the inference deployment on resource-constrained systems.

As a result, recent research focused on CNN optimization techniques, in particular quantization, which allows weights and activations of layers to be represented with just a few bits while achieving impressive prediction performance.

However, aggressive quantization techniques still fail to achieve full-precision prediction performance on state-of-the-art CNN architectures on large-scale classification tasks.

In this work we propose a method for weight and activation quantization that is scalable in terms of quantization levels (n-ary representations) and easy to compute while maintaining the performance close to full-precision CNNs.

Our weight quantization scheme is based on trainable scaling factors and a nested-means clustering strategy which is robust to weight updates and therefore exhibits good convergence properties.

The flexibility of nested-means clustering enables exploration of various n-ary weight representations with the potential of high parameter compression.

For activations, we propose a linear quantization strategy that takes the statistical properties of batch normalization into account.

We demonstrate the effectiveness of our approach using state-of-the-art models on ImageNet.

We propose a quantization scheme for weights and activations of deep neural networks. This reduces the memory footprint substantially and accelerates inference.

CNN model compression aand inference acceleration using quantization.

N-Ary Quantization for CNN Model Compression and Inference Acceleration

quantization
inference acceleration
cnn model compression


Reinforcement learning (RL) agents optimize only the features specified in a reward function and are indifferent to anything left out inadvertently.

This means that we must not only specify what to do, but also the much larger space of what not to do.

It is easy to forget these preferences, since these preferences are already satisfied in our environment.

This motivates our key insight: when a robot is deployed in an environment that humans act in, the state of the environment is already optimized for what humans want.

We can therefore use this implicit preference information from the state to fill in the blanks.

We develop an algorithm based on Maximum Causal Entropy IRL and use it to evaluate the idea in a suite of proof-of-concept environments designed to show its properties.

We find that information from the initial state can be used to infer both side effects that should be avoided as well as preferences for how the environment should be organized.

Our code can be found at https://github.com/HumanCompatibleAI/rlsp.

When a robot is deployed in an environment that humans have been acting in, the state of the environment is already optimized for what humans want, and we can use this to infer human preferences.

The authors propose to augment the explicitly stated reward function of an RL agent with auxiliary rewards/costs inferred from the initial state and a model of the state dynamics

This work proposes a way to infer the implicit information in the initial state using IRL and combine the inferred reward with a specified reward.

Preferences Implicit in the State of the World

state infer initial implicit preference reward


Regularization is one of the crucial ingredients of deep learning, yet the term regularization has various definitions, and regularization methods are often studied separately from each other.

In our work we present a novel, systematic, unifying taxonomy to categorize existing methods.

We distinguish methods that affect data, network architectures, error terms, regularization terms, and optimization procedures.

We identify the atomic building blocks of existing methods, and decouple the assumptions they enforce from the mathematical tools they rely on.

We do not provide all details about the listed methods; instead, we present an overview of how the methods can be sorted into meaningful categories and sub-categories.

This helps revealing links and fundamental similarities between them.

Finally, we include practical recommendations both for users and for developers of new regularization methods.

Systematic categorization of regularization methods for deep learning, revealing their similarities.

Attempts to build a taxonomy for regularization techniques employed in deep learning.

Regularization for Deep Learning: A Taxonomy


Deep neural networks are surprisingly efficient at solving practical tasks,

but the theory behind this phenomenon is only starting to catch up with

Numerous works show that depth is the key to this efficiency.

A certain class of deep convolutional networks – namely those that correspond

to the Hierarchical Tucker (HT) tensor decomposition – has been

proven to have exponentially higher expressive power than shallow networks.

I.e. a shallow network of exponential width is required to realize

the same score function as computed by the deep architecture.

we prove the expressive power theorem (an exponential lower bound on

the width of the equivalent shallow network) for a class of recurrent neural

networks – ones that correspond to the Tensor Train (TT) decomposition.

This means that even processing an image patch by patch with an RNN

can be exponentially more efficient than a (shallow) convolutional network

Using theoretical results on the relation between

the tensor decompositions we compare expressive powers of the HT- and

We also implement the recurrent TT-Networks and provide

numerical evidence of their expressivity.

We prove the exponential efficiency of recurrent-type neural networks over shallow networks.

The authors compare the complexity of tensor train networks with networks structured by CP decomposition

Expressive power of recurrent neural networks


Probabilistic modelling is a principled framework to perform model aggregation, which has been a primary mechanism to combat mode collapse in the context of Generative Adversarial Networks (GAN).

In this paper, we propose a novel probabilistic framework for GANs, ProbGAN, which iteratively learns a distribution over generators with a carefully crafted prior.

Learning is efficiently triggered by a tailored stochastic gradient Hamiltonian Monte Carlo with a novel gradient approximation to perform Bayesian inference.

Our theoretical analysis further reveals that our treatment is the first probabilistic framework that yields an equilibrium where generator distributions are faithful to the data distribution.

Empirical evidence on synthetic high-dimensional multi-modal data and image databases (CIFAR-10, STL-10, and ImageNet) demonstrates the superiority of our method over both start-of-the-art multi-generator GANs and other probabilistic treatment for GANs.

A novel probabilistic treatment for GAN with theoretical guarantee.

This paper proposes a Bayesian GAN that has theoretical guarantees of convergence to the real distribution and put likelihoods over the generator and discriminator with logarithms proportional to the traditional GAN objective functions.

ProbGAN: Towards Probabilistic GAN with Theoretical Guarantees


In the adversarial-perturbation problem of neural networks, an adversary starts with a neural network model $F$ and a point $\bfx$ that $F$ classifies correctly, and applies a \emph{small perturbation} to  $\bfx$ to produce another point $\bfx'$ that $F$ classifies \emph{incorrectly}.

In this paper, we propose taking into account \emph{the inherent confidence information} produced by models when studying adversarial perturbations, where a natural measure of ``confidence'' is \|F(\bfx)\|_\infty$ (i.e. how confident $F$ is about its prediction?)

. Motivated by a thought experiment based on the manifold assumption, we propose a ``goodness property'' of models which states that \emph{confident regions of a good model should be well separated}.

We give formalizations of this property and examine existing robust training objectives in view of them.

Interestingly, we find that a recent objective by Madry et al. encourages training a model that satisfies well our formal version of the goodness property, but has a weak control of points that are wrong but with low confidence.

However, if Madry et al.'s model is indeed a good solution to their objective, then good and bad points are now distinguishable and we can try to embed uncertain points back to the closest confident region to get (hopefully) correct predictions.

We thus propose embedding objectives and algorithms, and perform an empirical study using this method.

Our experimental results are encouraging: Madry et al.'s model wrapped with our embedding procedure achieves almost perfect success rate in defending against attacks that the base model fails on, while retaining good generalization behavior.

Defending against adversarial perturbations of neural networks from manifold assumption

The manuscript proposes two objective functions based on the manifold assumption as defense mechanisms against adversarial examples.

Defending against adversarial attacks based on the manifold assumption of natural data

The Manifold Assumption and Defenses Against Adversarial Perturbations

base manifold assumption
adversarial perturbation
defend
defense


Recently Neural Architecture Search (NAS) has aroused great interest in both academia and industry, however it remains challenging because of its huge and non-continuous search space.

Instead of applying evolutionary algorithm or reinforcement learning as previous works, this paper proposes a Direct Sparse Optimization NAS (DSO-NAS) method.

In DSO-NAS, we provide a novel model pruning view to NAS problem.

In specific, we start from a completely connected block, and then introduce scaling factors to scale the information flow between operations.

Next, we impose sparse regularizations to prune useless connections in the architecture.

Lastly, we derive an efficient and theoretically sound optimization method to solve it.

Our method enjoys both advantages of differentiability and efficiency, therefore can be directly applied to large datasets like ImageNet.

Particularly, On CIFAR-10 dataset, DSO-NAS achieves an average test error 2.84%, while on the ImageNet dataset DSO-NAS achieves 25.4% test error under 600M FLOPs with 8 GPUs in 18 hours.

single shot neural architecture search via direct sparse optimization

Presents an architecture search method where connections are removed with sparse regularization.

This paper proposes Direct Sparse Optimization, which is a method to obtain neural architectures on specific problems, at a reasonable computational cost.

This paper proposes a neural architecture search method based on a direct sparse optimization

Single Shot Neural Architecture Search Via Direct Sparse Optimization


Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning.

One aspect of the field receiving considerable attention is efficiently executing deep models in resource-constrained environments, such as mobile or embedded devices.

This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger teacher networks into smaller student networks.

The first method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher, into the training of a student network whose weights are quantized to a limited set of levels.

The second method,  differentiable quantization, optimizes the location of quantization points through stochastic gradient descent, to better fit the behavior of the teacher model.

We validate both methods through experiments on convolutional and recurrent architectures.

We show that quantized shallow students can reach similar accuracy levels to full-precision teacher models, while providing order of magnitude compression, and inference speedup that is linear in the depth reduction.

In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.

Obtains state-of-the-art accuracy for quantized, shallow nets by leveraging distillation.

Proposes small and low-cost models by combining distillation and quantization for vision and neural machine translation experiments

This paper presents a framework of using the teacher model to help the compression for the deep learning model in the context of model compression.

Model compression via distillation and quantization

distillation quantization
model
compression


Previous work has demonstrated the benefits of incorporating additional linguistic annotations such as syntactic trees into neural machine translation.

However the cost of obtaining those syntactic annotations is expensive for many languages and the quality of unsupervised learning linguistic structures is too poor to be helpful.

In this work, we aim to improve neural machine translation via source side dependency syntax but without explicit annotation.

We propose a set of models that learn to induce dependency trees on the source side and learn to use that information on the target side.

Importantly, we also show that our dependency trees capture important syntactic features of language and improve translation quality on two language pairs En-De and En-Ru.

This paper describes a method to induce source-side dependency structures in service to neural machine translation.

Inducing Grammars with and for Neural Machine Translation


Model-free reinforcement learning (RL) requires a large number of trials to learn a good policy, especially in environments with sparse rewards.

We explore a method to improve the sample efficiency when we have access to demonstrations.

Our approach, Backplay, uses a single demonstration to construct a curriculum for a given task.

Rather than starting each training episode in the environment's fixed initial state, we start the agent near the end of the demonstration and move the starting point backwards during the course of training until we reach the initial state.

Our contributions are that we analytically characterize the types of environments where Backplay can improve training speed, demonstrate the effectiveness of Backplay both in large grid worlds and a complex four player zero-sum game (Pommerman), and show that Backplay compares favorably to other competitive methods known to improve sample efficiency.

This includes reward shaping, behavioral cloning, and reverse curriculum generation.

Learn by working backwards from a single demonstration, even an inefficient one, and progressively have the agent do more of the solving itself.

This paper presents a method for increasing the efficiency of sparse reward RL methods through a backward curriculum on expert demonstrations.

The paper presents a strategy for solving sparse reward tasks with RL by sampling initial states from demonstrations.

Backplay: 'Man muss immer umkehren'

present
demonstration
solve
sparse reward
rl


Episodic memory is a psychology term which refers to the ability to recall specific events from the past.

We suggest one advantage of this particular type of memory is the ability to easily assign credit to a specific state when remembered information is found to be useful.

Inspired by this idea, and the increasing popularity of external memory mechanisms to handle long-term dependencies in deep learning systems, we propose a novel algorithm which uses a reservoir sampling procedure to maintain an external memory consisting of a fixed number of past states.

The algorithm allows a deep reinforcement learning agent to learn online to preferentially remember those states which are found to be useful to recall later on.

Critically this method allows for efficient online computation of gradient estimates with respect to the write process of the external memory.

Thus unlike most prior mechanisms for external memory it is feasible to use in an online reinforcement learning setting.

External memory for online reinforcement learning based on estimating gradients over a novel reservoir sampling technique.

The paper proposes a modified approach to RL, where an additional "episodic memory" is kept by the agent and use a "query network" that based on the current state.

Integrating Episodic Memory into a Reinforcement Learning Agent Using Reservoir Sampling

base memory agent reinforcement reservoir episodic


We achieve bias-variance decomposition for Boltzmann machines using an information geometric formulation.

Our decomposition leads to an interesting phenomenon that the variance does not necessarily increase when more parameters are included in Boltzmann machines, while the bias always decreases.

Our result gives a theoretical evidence of the generalization ability of deep learning architectures because it provides the possibility of increasing the representation power with avoiding the variance inflation.

The goal of this paper is to analyze the effectiveness and generalizability of deep learning by presenting a theoretical analysis of bias-variance decomposition for hierarchical models, specifically Boltzmann Machines

The paper arrives at the main conclusion that it is possible to reduce both the bias and the variance in a hierarchical model.

Bias-Variance Decomposition for Boltzmann Machines

bias-variance decomposition boltzmann machine


Recurrent Neural Networks (RNNs) are powerful tools for solving sequence-based problems, but their efficacy and execution time are dependent on the size of the network.

Following recent work in simplifying these networks with model pruning and a novel mapping of work onto GPUs, we design an efficient implementation for sparse RNNs.

We investigate several optimizations and tradeoffs: Lamport timestamps, wide memory loads, and a bank-aware weight layout.

With these optimizations, we achieve speedups of over 6x over the next best algorithm for a hidden layer of size 2304, batch size of 4, and a density of 30%.

Further, our technique allows for models of over 5x the size to fit on a GPU for a speedup of 2x, enabling larger networks to help advance the state-of-the-art.

We perform case studies on NMT and speech recognition tasks in the appendix, accelerating their recurrent layers by up to 3x.

Combining network pruning and persistent kernels into a practical, fast, and accurate network implementation.

This paper introduces sparse persistent RNNs, a mechanism to add pruning to the existing work of stashing RNN weights on a chip.

Sparse Persistent RNNs: Squeezing Large Recurrent Networks On-Chip


Weight pruning has proven to be an effective method in reducing the model size and computation cost while not sacrificing the model accuracy.

Conventional sparse matrix formats, however, involve irregular index structures with large storage requirement and sequential reconstruction process, resulting in inefficient use of highly parallel computing resources.

Hence, pruning is usually restricted to inference with a batch size of one, for which an efficient parallel matrix-vector multiplication method exists.

In this paper, a new class of sparse matrix representation utilizing Viterbi algorithm that has a high, and more importantly, fixed index compression ratio regardless of the pruning rate, is proposed.

In this approach, numerous sparse matrix candidates are first generated by the Viterbi encoder, and then the one that aims to minimize the model accuracy degradation is selected by the Viterbi algorithm.

The model pruning process based on the proposed Viterbi encoder and Viterbi algorithm is highly parallelizable, and can be implemented efficiently in hardware to achieve low-energy, high-performance index decoding process.

Compared with the existing magnitude-based pruning methods, index data storage requirement can be further compressed by 85.2% in MNIST and 83.9% in AlexNet while achieving similar pruning rate.

Even compared with the relative index compression technique, our method can still reduce the index storage requirement by 52.7% in MNIST and 35.5% in AlexNet.

We present a new pruning method and sparse matrix format to enable high index compression ratio and parallel index decoding process.

The authors use Viterbi encoding to dramatically compress the sparse matrix index of a pruned network, reducing one of the main memory overheads and speeding up inference in the parallel setting.

Viterbi-based Pruning for Sparse Matrix with Fixed and High Index Compression Ratio

high index compression ratio
sparse matrix
parallel
pruning


Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL).

It is also a requirement for its deployment in real-world scenarios.

This paper proposes a novel framework for efficient multi-task reinforcement learning.

Our framework trains agents to employ hierarchical policies that decide when to use a previously learned policy and when to learn a new skill.

This enables agents to continually acquire new skills during different stages of training.

Each learned task corresponds to a human language description.

Because agents can only access previously learned skills through these descriptions, the agent can always provide a human-interpretable description of its choices.

In order to help the agent learn the complex temporal dependencies necessary for the hierarchical policy, we provide it with a stochastic temporal grammar that modulates when to rely on previously learned skills and when to execute new skills.

We validate our approach on Minecraft games designed to explicitly test the ability to reuse previously learned skills while simultaneously learning new skills.

A novel hierarchical policy network which can reuse previously learned skills alongside and as subcomponents of new skills by discovering the underlying relations between skills.

This paper aims to learn hierarchical policies by using a recursive policy structure regulated by a stochastic temporal grammar

This paper proposes an approach to learning hierarchical policies in a lifelong learning context by stacking policies and then using an explicit "switch" policy.

Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning


Embeddings are a fundamental component of many modern machine learning and natural language processing models.

Understanding them and visualizing them is essential for gathering insights about the information they capture and the behavior of the models.

State of the art in analyzing embeddings consists in projecting them in two-dimensional planes without any interpretable semantics associated to the axes of the projection, which makes detailed analyses and comparison among multiple sets of embeddings challenging.

In this work, we propose to use explicit axes defined as algebraic formulae over embeddings to project them into a lower dimensional, but semantically meaningful subspace, as a simple yet effective analysis and visualization methodology.

This methodology assigns an interpretable semantics to the measures of variability and the axes of visualizations, allowing for both comparisons among different sets of embeddings and fine-grained inspection of the embedding spaces.

We demonstrate the power of the proposed methodology through a series of case studies that make use of visualizations constructed around the underlying methodology and through a user study.

The results show how the methodology is effective at providing more profound insights than classical projection methods and how it is widely applicable to many other use cases.

We propose to use explicit vector algebraic formulae projection as an alternative way to visualize embedding spaces specifically tailored for goal-oriented analysis tasks and it outperforms t-SNE in our user study.

Analysis of embedding psaces in a non-parametric (example-based_ way

Visualizing and Understanding the Semantics of Embedding Spaces via Algebraic Formulae

visualize
embed space
algebraic formulae
analysis
way


Learning deep networks which can resist large variations between training andtesting data is essential to build accurate and robust image classifiers.

Towardsthis end, a typical strategy is to apply data augmentation to enlarge the trainingset.

However,  standard  data  augmentation  is  essentially  a  brute-force  strategywhich is inefficient,  as it performs all the pre-defined transformations  to everytraining sample.

In this paper, we propose a principled approach to train networkswith  significantly  improved  resistance  to  large  variations  between  training  andtesting data.

This is achieved by embedding a learnable transformation moduleinto the introspective networks (Jin et al., 2017; Lazarow et al., 2017; Lee et al.,2018), which is a convolutional neural network (CNN) classifier empowered withgenerative capabilities.

Our approach alternatively synthesizes pseudo-negativesamples with learned transformations and enhances the classifier by retraining itwith synthesized samples.

Experimental results verify that our approach signif-icantly improves the ability of deep networks to resist large variations betweentraining and testing data and achieves classification accuracy improvements onseveral benchmark datasets, including MNIST, affNIST, SVHN and CIFAR-10.

We propose a principled approach that endows classifiers with the ability to resist larger variations between training and testing data in an intelligent and efficient manner.

Using introspective learning to handle data variations at test time

This paper suggests the use of learned transformation networks, embedded within introspective networks to improve classification performance with synthesized examples.

Towards Resisting Large Data Variations via Introspective Learning

resist large
variation
introspective learning
datum


It is well known that it is possible to construct "adversarial examples"

for neural networks: inputs which are misclassified by the network

modification to standard neural network architectures, thermometer

encoding, which significantly increases the robustness of the network to

We demonstrate this robustness with experiments

on the MNIST, CIFAR-10, CIFAR-100, and SVHN datasets, and show that

models with thermometer-encoded inputs consistently have higher accuracy

on adversarial examples, without decreasing generalization.

State-of-the-art accuracy under the strongest known white-box attack was

increased from 93.20% to 94.30% on MNIST and 50.00% to 79.16% on CIFAR-10.

We explore the properties of these networks, providing evidence

that thermometer encodings help neural networks to

find more-non-linear decision boundaries.

Input discretization leads to robustness against adversarial examples

The authors present an in-depth study of discretizing / quantizing the input as a defense against adversarial examples

Thermometer Encoding: One Hot Way To Resist Adversarial Examples


Low-precision training is a promising way of decreasing the time and energy cost of training machine learning models.

Previous work has analyzed low-precision training algorithms, such as low-precision stochastic gradient descent, and derived theoretical bounds on their convergence rates.

These bounds tend to depend on the dimension of the model $d$ in that the number of bits needed to achieve a particular error bound increases as $d$ increases.

This is undesirable because a motivating application for low-precision training is large-scale models, such as deep learning, where $d$ can be huge.

In this paper, we prove dimension-independent bounds for low-precision training algorithms that use fixed-point arithmetic, which lets us better understand what affects the convergence of these algorithms as parameters scale.

Our methods also generalize naturally to let us prove new convergence bounds on low-precision training with other quantization schemes, such as low-precision floating-point computation and logarithmic quantization.

we proved dimension-independent bounds for low-precision training algorithms

This paper discusses conditions under which the convergence of training models with low-precision weights do not rely on model dimension.

Dimension-Free Bounds for Low-Precision Training


We consider the problem of exploration in meta reinforcement learning.

Two new meta reinforcement learning algorithms are suggested: E-MAML and ERL2.

Results are presented on a novel environment we call 'Krazy World'  and a set of maze environments.

We show E-MAML and ERL2 deliver better performance on tasks where exploration is important.

Modifications to MAML and RL2 that should allow for better exploration.

The paper proposes a trick of extending objective functions to drive exploration in meta-RL on top of two recent meta-RL algorithms

Some Considerations on Learning to Explore via Meta-Reinforcement Learning


We propose a new class of probabilistic neural-symbolic models for visual question answering (VQA) that provide interpretable explanations of their decision making in the form of programs, given a small annotated set of human programs.

The key idea of our approach is to learn a rich latent space which effectively propagates program annotations from known questions to novel questions.

We do this by formalizing prior work on VQA, called module networks (Andreas, 2016) as discrete, structured, latent variable models on the joint distribution over questions and answers given images, and devise a procedure to train the model effectively.

Our results on a dataset of compositional questions about SHAPES (Andreas, 2016) show that our model generates more interpretable programs and obtains better accuracy on VQA in the low-data regime than prior work.

A probabilistic neural symbolic model with a latent program space, for more interpretable question answering

This paper proposes a discrete, structured latent variable model for visual question answering that involves compositional generalization and reasoning with significant gain in performance and capability.

Probabilistic Neural-Symbolic Models for Interpretable Visual Question Answering

visual question answer
model
interpretable
probabilistic
latent


The ability to deploy neural networks in real-world, safety-critical systems is severely limited by the presence of adversarial examples: slightly perturbed inputs that are misclassified by the network.

In recent years, several techniques have been proposed for training networks that are robust to such examples; and each time stronger attacks have been devised, demonstrating the shortcomings of existing defenses.

This highlights a key difficulty in designing an effective defense: the inability to assess a network's robustness against future attacks.

We propose to address this difficulty through formal verification techniques.

We construct ground truths: adversarial examples with a provably-minimal distance from a given input point.

We demonstrate how ground truths can serve to assess the effectiveness of attack techniques, by comparing the adversarial examples produced by those attacks to the ground truths; and also of defense techniques, by computing the distance to the ground truths before and after the defense is applied, and measuring the improvement.

We use this technique to assess recently suggested attack and defense techniques.

We use formal verification to assess the effectiveness of techniques for finding adversarial examples or for defending against adversarial examples.

This paper proposes a method for computing adversarial examples with minimum distance to the original inputs.

The authors propose to employ provably minimal-distance examples as a tool to evaluate the robustness of a trained network.

The paper describes a method for generating adversarial examples that have minimal distance to the training example used to generate them

Ground-Truth Adversarial Examples


This paper introduces a new framework for open-domain question answering in which the retriever and the reader \emph{iteratively interact} with each other.

The framework is agnostic to the architecture of the machine reading model provided it has \emph{access} to the token-level hidden representations of the reader.

The retriever uses fast nearest neighbor search that allows it to scale to corpora containing millions of paragraphs.

A gated recurrent unit updates the query at each step conditioned on the \emph{state} of the reader and the \emph{reformulated} query is used to re-rank the paragraphs by the retriever.

We conduct analysis and show that iterative interaction helps in retrieving informative paragraphs from the corpus.

Finally, we show that our multi-step-reasoning framework brings consistent improvement when applied to two widely used reader architectures (\drqa and \bidaf) on various large open-domain datasets ---\tqau, \quasart, \searchqa, and \squado\footnote{Code and pretrained models are available at \url{https://github.com/rajarshd/Multi-Step-Reasoning}}.

Paragraph retriever and machine reader interacts with each other via reinforcement learning to yield large improvements on open domain datasets

The paper introduces a new framework of bi-directional interaction between document retriever and reader for open-domain question answering with idea of 'reader state' from reader to retriever.

The paper proposes a multi-document extractive machine reading model composed of 3 distinct parts and an algorithm.

Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering

open-domain question answer
retriever
interaction
machine
reader


Many imaging tasks require global information about all pixels in an image.

Conventional bottom-up classification networks globalize information by decreasing resolution; features are pooled and down-sampled into a single output.

But for semantic segmentation and object detection tasks, a network must provide higher-resolution pixel-level outputs.

To globalize information while preserving resolution, many researchers propose the inclusion of sophisticated auxiliary blocks, but these come at the cost of a considerable increase in network size and computational cost.

This paper proposes stacked u-nets (SUNets), which iteratively combine features from different resolution scales while maintaining resolution.

SUNets leverage the information globalization power of u-nets in a deeper net- work architectures that is capable of handling the complexity of natural images.

SUNets perform extremely well on semantic segmentation tasks using a small number of parameters.

Presents new architecture which leverages information globalization power of u-nets in a deeper networks and performs well across tasks without any bells and whistles.

A network architecture for semantic image segmentation, based on composing a stack of basic U-Net architectures, that reduces the number of parameters and improves results.

This proposes a stacked U-Net architecture for image segmentation.

Stacked U-Nets: A No-Frills Approach to Natural Image Segmentation

stack u-net
image segmentation
architecture
network


Asking questions is an important ability for a chatbot.

This paper focuses on question generation.

Although there are existing works on question generation based on a piece of descriptive text, it remains to be a very challenging problem.

In the paper, we propose a new question generation problem, which also requires the input of a target topic in addition to a piece of descriptive text.

The key reason for proposing the new problem is that in practical applications, we found that useful questions need to be targeted toward some relevant topics.

One almost never asks a random question in a conversation.

Due to the fact that given a descriptive text, it is often possible to ask many types of questions, generating a question without knowing what it is about is of limited use.

To solve the problem, we propose a novel neural network that is able to generate topic-specific questions.

One major advantage of this model is that it can be trained directly using a question-answering corpus without requiring any additional annotations like annotating topics in the questions or answers.

Experimental results show that our model outperforms the state-of-the-art baseline.

We propose a neural network that is able to generate topic-specific questions.

Presents a neural network-based approach to generate topic-specific questions with the motivation that topical questions are more meaningful in practical applications.

Proposes a topic-based generation method using an LSTM to extract topics using a two-stage encoding technique

Topic-Based Question Generation

generate topic-specific question
neural
topic-based
generation


Brain-Machine Interfaces (BMIs) have recently emerged as a clinically viable option

to restore voluntary movements after paralysis.

ability to extract information about movement intent from neural signals recorded

using multi-electrode arrays chronically implanted in the motor cortices of the

However, the inherent loss and turnover of recorded neurons requires repeated

recalibrations of the interface, which can potentially alter the day-to-day

The resulting need for continued user adaptation interferes with

the natural, subconscious use of the BMI.

approach that decodes movement intent from a low-dimensional latent representation

We implement various domain adaptation methods

to stabilize the interface over significantly long times.

Correlation Analysis used to align the latent variables across days; this method

requires prior point-to-point correspondence of the time series across domains.

Alternatively, we match the empirical probability distributions of the latent variables

across days through the minimization of their Kullback-Leibler divergence.

These two methods provide a significant and comparable improvement in the performance

However, implementation of an Adversarial Domain

Adaptation Network trained to match the empirical probability distribution of the

residuals of the reconstructed neural signals outperforms the two methods based

on latent variables, while requiring remarkably few data points to solve the domain

We implement an adversarial domain adaptation network to stabilize a fixed Brain-Machine Interface against gradual changes in the recorded neural signals.

Describes a novel approach for implanted brain-machine interface in order to address calibration problem and covariate shift.

The authors define a BMI that uses an autoencoder and then address the problem of data drift in BMI.

Adversarial Domain Adaptation for Stable Brain-Machine Interfaces

adversarial domain adaptation
brain-machine interface
address
problem


Neural network-based systems can now learn to locate the referents of words and phrases in images, answer questions about visual scenes, and even execute symbolic instructions as first-person actors in partially-observable worlds.

To achieve this so-called grounded language learning, models must overcome certain well-studied learning challenges that are also fundamental to infants learning their first words.

While it is notable that models with no meaningful prior knowledge overcome these learning obstacles, AI researchers and practitioners currently lack a clear understanding of exactly how they do so.

Here we address this question as a way of achieving a clearer general understanding of grounded language learning, both to inform future research and to improve confidence in model predictions.

For maximum control and generality, we focus on a simple neural network-based language learning agent trained via policy-gradient methods to interpret synthetic linguistic instructions in a simulated 3D world.

We apply experimental paradigms from developmental psychology to this agent, exploring the conditions under which established human biases and learning effects emerge.

We further propose a novel way to visualise and analyse semantic representation in grounded language learning agents that yields a plausible computational account of the observed effects.

Analysing and understanding how neural network agents learn to understand simple grounded language

The authors connect psychological experimental methods to understanding how the black box of deep learning methods solves problems.

This paper presents an analysis of the agents who learn grounded language through reinforcement learning in a simple environment that combines verbal instruction with visual information

Understanding Grounded Language Learning Agents

understand
agent learn
ground language
simple
learning


Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future.

In this paper, we propose a novel formulation that simultaneously learns a hierarchical, disentangled object representation and a dynamics model for object parts from unlabeled videos.

Our Parts, Structure, and Dynamics (PSD) model learns to, first, recognize the object parts via a layered image representation; second, predict hierarchy via a structural descriptor that composes low-level concepts into a hierarchical structure; and third, model the system dynamics by predicting the future.

Experiments on multiple real and synthetic datasets demonstrate that our PSD model works well on all three tasks: segmenting object parts, building their hierarchical structure, and capturing their motion distributions.

Learning object parts, hierarchical structure, and dynamics by watching how they move

Proposes an unsupervised learning model that learns to disentangle objects into parts, predict hierarchical structure for the parts, and based on the disentangled parts and the hierarchy, predict motion.

Unsupervised Discovery of Parts, Structure, and Dynamics

structure dynamic
object part
learn
hierarchical


A successful application of convolutional architectures is to increase the resolution of single low-resolution images -- a image restoration task called super-resolution (SR).

Naturally, SR is of value to resource constrained devices like mobile phones, electronic photograph frames and televisions to enhance image quality.

However, SR demands perhaps the most extreme amounts of memory and compute operations of any mainstream vision task known today, preventing SR from being deployed to devices that require them.

In this paper, we perform a early systematic study of system resource efficiency for SR, within the context of a variety of architectural and low-precision approaches originally developed for discriminative neural networks.

We present a rich set of insights, representative SR architectures, and efficiency trade-offs; for example, the prioritization of ways to compress models to reach a specific memory and computation target and techniques to compact SR models so that they are suitable for DSPs and FPGAs.

As a result of doing so, we manage to achieve better and comparable performance with previous models in the existing literature, highlighting the practicality of using existing efficiency techniques in SR tasks.

Collectively, we believe these results provides the foundation for further research into the little explored area of resource efficiency for SR.

We build an understanding of resource-efficient techniques on Super-Resolution

The paper proposes a detailed empirical evaluation of the trade-offs achieved by various convolutional neural networks on the super resolution problem.

This paper proposed to improve the system resource efficiency for super resolution networks.

Understanding Opportunities for Efficiency in Single-image Super Resolution Networks


Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with 100% accuracy.

In this work we present properties of neural networks that complement this aspect of expressivity.

By using tools from Fourier analysis, we show that deep ReLU networks are biased towards low frequency functions, meaning that they cannot have local fluctuations without affecting their global behavior.

Intuitively, this property is in line with the observation that over-parameterized networks find simple patterns that generalize across data samples.

We also investigate how the shape of the data manifold affects expressivity by showing evidence that learning high frequencies gets easier with increasing manifold complexity, and present a theoretical understanding of this behavior.

Finally, we study the robustness of the frequency components with respect to parameter perturbation, to develop the intuition that the parameters must be finely tuned to express high frequency functions.

We investigate ReLU networks in the Fourier domain and demonstrate peculiar behaviour.

Fourier analysis of ReLU network, finding that they are biased towards learning low frequency

This paper has theoretical and empirical contributions on topic of Fourier coefficients of neural networks

On the Spectral Bias of Neural Networks


Instance embeddings are an efficient and versatile image representation that facilitates applications like recognition, verification, retrieval, and clustering.

Many metric learning methods represent the input as a single point in the embedding space.

Often the distance between points is used as a proxy for match confidence.

However, this can fail to represent uncertainty which can arise when the input is ambiguous, e.g., due to occlusion or blurriness.

This work addresses this issue and explicitly models the uncertainty by “hedging” the location of each input in the embedding space.

We introduce the hedged instance embedding (HIB) in which embeddings are modeled as random variables and the model is trained under the variational information bottleneck principle (Alemi et al., 2016; Achille & Soatto, 2018).

Empirical results on our new N-digit MNIST dataset show that our method leads to the desired behavior of “hedging its bets” across the embedding space upon encountering ambiguous inputs.

This results in improved performance for image matching and classification tasks, more structure in the learned embedding space, and an ability to compute a per-exemplar uncertainty measure which is correlated with downstream performance.

The paper proposes using probability distributions instead of points for instance embeddings tasks such as recognition and verification.

Paper proposes an alternative to current point embedding and a technique to train them.

The paper proposes a model using uncertain-embeddings to extend deep learning to Bayesian applications

Modeling Uncertainty with Hedged Instance Embeddings


Convolution neural networks typically consist of many convolutional layers followed by several fully-connected layers.

While convolutional layers map between high-order activation tensors, the fully-connected layers operate on flattened activation vectors.

Despite its success, this approach has notable drawbacks.

Flattening discards the multi-dimensional structure of the activations, and the fully-connected layers require a large number of parameters.

We present two new techniques to address these problems.

First, we introduce tensor contraction layers which can replace the ordinary fully-connected layers in a neural network.

Second, we introduce tensor regression layers, which express the output of a neural network as a low-rank multi-linear mapping from a high-order activation tensor to the softmax layer.

Both the contraction and regression weights are learned end-to-end by backpropagation.

By imposing low rank on both, we use significantly fewer parameters.

Experiments on the ImageNet dataset show that applied to the popular VGG and ResNet architectures, our methods significantly reduce the number of parameters in the fully connected layers (about 65% space savings) while negligibly impacting accuracy.

We propose tensor contraction and low-rank tensor regression layers to preserve and leverage the multi-linear structure throughout the network, resulting in huge space savings with little to no impact on performance.

This paper proposes new layer architectures of neural networks using a low-rank representation of tensors

This paper incorporates tensor decomposition and tensor regression into CNN by using a new tensor regression layer.

Tensor Contraction & Regression Networks

tensor regression layer
new
network
low-rank
contraction


We explore ways of incorporating bilingual dictionaries to enable semi-supervised

Conventional back-translation methods have shown

success in leveraging target side monolingual data.

back-translation models is tied to the size of the available parallel corpora, this

could adversely impact the synthetically generated sentences in a low resource

We propose a simple data augmentation technique to address both this

We incorporate widely available bilingual dictionaries that yield

word-by-word translations to generate synthetic sentences.

expands the vocabulary of the model while maintaining high quality content.

method shows an appreciable improvement in performance over strong baselines.

We use bilingual dictionaries for data augmentation for neural machine translation

This paper investigates using bilingual dictionaries to create synthetic sources for target-side monolingual data to improve over NMT models trained with small amounts of parallel data.

INCORPORATING BILINGUAL DICTIONARIES FOR LOW RESOURCE SEMI-SUPERVISED NEURAL MACHINE TRANSLATION

bilingual dictionary
neural machine translation
datum


Rewards are sparse in the real world and most of today's reinforcement learning algorithms struggle with such sparsity.

One solution to this problem is to allow the agent to create rewards for itself - thus making rewards dense and more suitable for learning.

In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus.

Such bonus is summed up with the real task reward - making it possible for RL algorithms to learn from the combined reward.

We propose a new curiosity method which uses episodic memory to form the novelty bonus.

To determine the bonus, the current observation is compared with the observations in memory.

Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory - which incorporates rich information about environment dynamics.

This allows us to overcome the known "couch-potato" issues of prior work - when the agent finds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences.

We test our approach in visually rich 3D environments in ViZDoom, DMLab and MuJoCo.

In navigational tasks from ViZDoom and DMLab, our agent outperforms the state-of-the-art curiosity method ICM.

In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only.

The code is available at https://github.com/google-research/episodic-curiosity/.

We propose a novel model of curiosity based on episodic memory and the ideas of reachability which allows us to overcome the known "couch-potato" issues of prior work.

Proposes to give exploration bonuses in RL algorithms by giving larger bonuses to observations that are father away in environment steps.

The authors propose an exploration bonus that is aimed to aid in sparse reward RL problems and considers many experiments on complex 3D environments

Episodic Curiosity through Reachability

exploration bonus
curiosity
reachability
episodic
environment
rl


We introduce a new dataset of logical entailments for the purpose of measuring models' ability to capture and exploit the structure of logical expressions against an entailment prediction task.

We use this task to compare a series of architectures which are ubiquitous in the sequence-processing literature, in addition to a new model class---PossibleWorldNets---which computes entailment as a ``convolution over possible worlds''.

Results show that convolutional networks present the wrong inductive bias for this class of problems relative to LSTM RNNs, tree-structured neural networks outperform LSTM RNNs due to their enhanced ability to exploit the syntax of logic, and PossibleWorldNets outperform all benchmarks.

The paper proposes a new model to use deep models for detecting logical entailment as a product of continuous functions over possible worlds.

Proposes a new model designed for machine learning with predicting logical entailment.

Can Neural Networks Understand Logical Entailment?


Deep convolutional neural network (DCNN) based supervised learning is a widely practiced approach for large-scale image classification.

However, retraining these large networks to accommodate new, previously unseen data demands high computational time and energy requirements.

Also, previously seen training samples may not be available at the time of retraining.

We propose an efficient training methodology and incrementally growing a DCNN to allow new classes to be learned while sharing part of the base network.

Our proposed methodology is inspired by transfer learning techniques, although it does not forget previously learned classes.

An updated network for learning new set of classes is formed using previously learned convolutional layers (shared from initial part of base network) with addition of few newly added convolutional kernels included in the later layers of the network.

We evaluated the proposed scheme on several recognition applications.

The classification accuracy achieved by our approach is comparable to the regular incremental learning approach (where networks are updated with new training samples only, without any network sharing).

The paper is about a new energy-efficient methodology for Incremental learning

Proposes procedure for incremental learning as transfer learning.

The paper presents a method to train deep convolutional neural networks in an incremental fashion, in which data are available in small batches over a period of time.

Presents an approach to class-incremental learning using deep networks by proposing three different learning strategies in the final/best approach.

Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing

deep convolutional neural network
incremental learning
present


Recurrent neural networks (RNNs) are widely used to model sequential data but

their non-linear dependencies between sequence elements prevent parallelizing

We show the training of RNNs with only linear

sequential dependencies can be parallelized over the sequence length using the

parallel scan algorithm, leading to rapid training on long sequences even with

We develop a parallel linear recurrence CUDA kernel and

show that it can be applied to immediately speed up training and inference of

several state of the art RNN architectures by up to 9x.

on linear RNNs into a new framework of linear surrogate RNNs and develop a

linear surrogate model for the long short-term memory unit, the GILR-LSTM, that

extremely long sequence regimes that were previously out of reach by

successfully training a GILR-LSTM on a synthetic sequence classification task

use parallel scan to parallelize linear recurrent neural nets. train model on length 1 million dependency

Proposes accelerating RNN by applying the method from Blelloch.

The authors propose a parallel algorithm for Linear Surrogate RNNs, which produces speedups over the existing implements of Quasi-RNN, SRU, and LSTM.

Parallelizing Linear Recurrent Neural Nets Over Sequence Length

parallelize linear recurrent neural net
parallel
rnn
length


Neural text generation models are often autoregressive language models or seq2seq models.

Neural autoregressive and seq2seq models that generate text by sampling words sequentially, with each word conditioned on the previous model, are state-of-the-art for several machine translation and summarization benchmarks.

These benchmarks are often defined by validation perplexity even though this is not a direct measure of sample quality.

Language models are typically trained via maximum likelihood and most often with teacher forcing.

Teacher forcing is well-suited to optimizing perplexity but can result in poor sample quality because generating text requires conditioning on sequences of words that were never observed at training time.

We propose to improve sample quality using Generative Adversarial Network (GANs), which explicitly train the generator to produce high quality samples and have shown a lot of success in image generation.

GANs were originally to designed to output differentiable values, so discrete language generation is challenging for them.

We introduce an actor-critic conditional GAN that fills in missing text conditioned on the surrounding context.

We show qualitatively and quantitatively, evidence that this produces more realistic text samples compared to a maximum likelihood trained model.

Natural language GAN for filling in the blank

This paper proposes to generate text using GANs.

Generating text samples using GAN and a mechanism to fill in missing words conditional on the surrounding text

MaskGAN: Better Text Generation via Filling in the _______


Parametric texture models have been applied successfully to synthesize artificial images.

Psychophysical studies show that under defined conditions observers are unable to differentiate between model-generated and original natural textures.

In industrial applications the reverse case is of interest: a texture analysis system should decide if human observers are able to discriminate between a reference and a novel texture.

For example, in case of inspecting decorative surfaces the de- tection of visible texture anomalies without any prior knowledge is required.

Here, we implemented a human-vision-inspired novelty detection approach.

Assuming that the features used for texture synthesis are important for human texture percep- tion, we compare psychophysical as well as learnt texture representations based on activations of a pretrained CNN in a novelty detection scenario.

Additionally, we introduce a novel objective function to train one-class neural networks for novelty detection and compare the results to standard one-class SVM approaches.

Our experiments clearly show the differences between human-vision-inspired texture representations and learnt features in detecting visual anomalies.

Based on a dig- ital print inspection scenario we show that psychophysical texture representations are able to outperform CNN-encoded features.

Comparison of psychophysical and CNN-encoded  texture representations in a one-class neural network novelty detection application.

This paper focuses on novelty detection and shows that psychophysical representations can outperform VGG-encoder features in some part of this task

This paper considers detecting anomalies in textures and proposes original loss function.

Proposes training two anomaly detectors from three different models to detect perceptual anomalies in visual textures.

Psychophysical vs. learnt texture representations in novelty detection

texture representation
novelty detection
psychophysical
anomaly
detect


In representation learning (RL), how to make the learned representations easy to interpret and less overfitted to training data are two important but challenging issues.

To address these problems, we study a new type of regularization approach that encourages the supports of weight vectors in RL models to have small overlap, by simultaneously promoting near-orthogonality among vectors and sparsity of each vector.

We apply the proposed regularizer to two models: neural networks (NNs) and sparse coding (SC), and develop an efficient ADMM-based algorithm for regularized SC.

Experiments on various datasets demonstrate that weight vectors learned under our regularizer are more interpretable and have better generalization performance.

We propose a new type of regularization approach that encourages non-overlapness in representation learning, for the sake of improving interpretability and reducing overfitting.

The paper introduces a matrix regularizer to simultaneously induce both sparsity and approximate orthogonality.

The paper studies a regularization method to promote sparsity and reduce the overlap among the supports of the weight vectors in the learned representations to enhance interpretability and avoid overfitting

The paper proposed a new regularization approach that simultaneously encourages the weight vectors (W) to be sparse and orthogonal to each other.

Learning Less-Overlapping Representations

new
weight vector
regularization approach
encourage
representation
interpretability
reduce
simultaneously
sparsity
learn


Successful recurrent models such as long short-term memories (LSTMs) and gated recurrent units (GRUs) use \emph{ad hoc} gating mechanisms.

Empirically these models have been found to improve the learning of medium to long term temporal dependencies and to help with vanishing gradient issues.

We prove that learnable gates in a recurrent model formally provide \emph{quasi-invariance to general time transformations} in the input data.

We recover part of the LSTM architecture from a simple axiomatic approach.

This result leads to a new way of initializing gate biases in LSTMs and GRUs.

Experimentally, this new \emph{chrono initialization} is shown to greatly improve learning of long term dependencies, with minimal implementation effort.

Proves that gating mechanisms provide invariance to time transformations. Introduces and tests a new initialization for LSTMs from this insight.

Paper links recurrent network deisgn and its effect on how the network reacts to time transformations, and uses this to develop a simple bias initialization scheme.

Can recurrent neural networks warp time?

time transformation
initialization
recurrent
network


Teaching plays a very important role in our society, by spreading human knowledge and educating our next generations.

A good teacher will select appropriate teaching materials, impact suitable methodologies, and set up targeted examinations, according to the learning behaviors of the students.

In the field of artificial intelligence, however, one has not fully explored the role of teaching, and pays most attention to machine \emph{learning}.

In this paper, we argue that equal attention, if not more, should be paid to teaching, and furthermore, an optimization framework (instead of heuristics) should be used to obtain good teaching strategies.

We call this approach ``learning to teach''.

In the approach, two intelligent agents interact with each other: a student model (which corresponds to the learner in traditional machine learning algorithms), and a teacher model (which determines the appropriate data, loss function, and hypothesis space to facilitate the training of the student model).

The teacher model leverages the feedback from the student model to optimize its own teaching strategies by means of reinforcement learning, so as to achieve teacher-student co-evolution.

To demonstrate the practical value of our proposed approach, we take the training of deep neural networks (DNN) as an example, and show that by using the learning to teach techniques, we are able to use much less training data and fewer iterations to achieve almost the same accuracy for different kinds of DNN models (e.g., multi-layer perceptron, convolutional neural networks and recurrent neural networks) under various machine learning tasks (e.g., image classification and text understanding).

We propose and verify the effectiveness of learning to teach, a new framework to automatically guide machine learning process.

This paper focuses on "machine teaching" and proposes leveraging reinforcement learning by defining the reward as how fast the learner learns and using policy gradient to update the teacher parameters

The authors define a deep learning model composed of four components: a student model, a teacher model, a loss function, and a data set.

Suggests a "learning to teach" framework, corresponding to choices over the data presented to the learner.

Learning to Teach

learn framework teach teacher learner define " machine learning


We present DL2, a system for training and querying neural networks with logical constraints.

The key idea is to translate these constraints into a differentiable loss with desirable mathematical properties and to then either train with this loss in an iterative manner or to use the loss for querying the network for inputs subject to the constraints.

We empirically demonstrate that DL2 is effective in both training and querying scenarios, across a range of constraints and data sets.

A differentiable loss for logic constraints for training and querying neural networks.

A framework for turning queries over parameters and input, ouput pairs to neural networks into differentiable loss functions and an associated declarative language for specifying these queries

This paper tackles the problem of combining logical approaches with neural networks by translating a logical formula into a non-negative loss function for a neural network.

DL2: Training and Querying Neural Networks with Logic

training query neural network
loss function
logic
differentiable


Genetic algorithms have been widely used in many practical optimization problems.

Inspired by natural selection, operators, including mutation, crossover

and selection, provide effective heuristics for search and black-box optimization.

However, they have not been shown useful for deep reinforcement learning, possibly

due to the catastrophic consequence of parameter crossovers of neural networks.

Here, we present Genetic Policy Optimization (GPO), a new genetic algorithm

for sample-efficient deep policy optimization.

for policy crossover in the state space and applies policy gradient methods for mutation.

Our experiments on MuJoCo tasks show that GPO as a genetic algorithm

is able to provide superior performance over the state-of-the-art policy gradient

methods and achieves comparable or higher sample efficiency.

Genetic algorithms based approach for optimizing deep neural network policies

The authors present an algorithm for training ensembles of policy networks that regularly mixes different policies in the ensemble together.

This paper proposes a genetic algorithm inspired policy optimization method, which mimics the mutation and the crossover operators over policy networks.

Policy Optimization by Genetic Distillation

policy network
genetic algorithm
optimization


To make deep neural networks feasible in resource-constrained environments (such as mobile devices), it is beneficial to quantize models by using low-precision weights.

One common technique for quantizing neural networks is the straight-through gradient method, which enables back-propagation through the quantization mapping.

Despite its empirical success, little is understood about why the straight-through gradient method works.

Building upon a novel observation that the straight-through gradient method is in fact identical to the well-known Nesterov’s dual-averaging algorithm on a quantization constrained optimization problem, we propose a more principled alternative approach, called ProxQuant , that formulates quantized network training as a regularized learning problem instead and optimizes it via the prox-gradient method.

ProxQuant does back-propagation on the underlying full-precision vector and applies an efficient prox-operator in between stochastic gradient steps to encourage quantizedness.

For quantizing ResNets and LSTMs, ProxQuant outperforms state-of-the-art results on binary quantization and is on par with state-of-the-art on multi-bit quantization.

We further perform theoretical analyses showing that ProxQuant converges to stationary points under mild smoothness assumptions, whereas variants such as lazy prox-gradient method can fail to converge in the same setting.

A principled framework for model quantization using the proximal gradient method, with empirical evaluation and theoretical convergence analyses.

Proposes ProxQuant method to train neural networks with quantized weights.

Proposes solving binary nets and its variants using proximal gradient descent.

ProxQuant: Quantized Neural Networks via Proximal Operators

proximal gradient
method
quantize
neural network
proxquant


Background: Statistical mechanics results (Dauphin et al. (2014); Choromanska et al. (2015)) suggest that local minima with high error are exponentially rare in high dimensions.

However, to prove low error guarantees for Multilayer Neural Networks (MNNs), previous works so far required either a heavily modified MNN model or training method, strong assumptions on the labels (e.g., “near” linear separability), or an unrealistically wide hidden layer with \Omega\(N) units.

Results: We examine a MNN with one hidden layer of piecewise linear units, a single output, and a quadratic loss.

We prove that, with high probability in the limit of N\rightarrow\infty datapoints, the volume of differentiable regions of the empiric loss containing sub-optimal differentiable local minima is exponentially vanishing in comparison with the same volume of global minima, given standard normal input of dimension d_0=\tilde{\Omega}(\sqrt{N}), and a more realistic number of d_1=\tilde{\Omega}(N/d_0) hidden units.

We demonstrate our results numerically: for example, 0% binary classification training error on CIFAR with only N/d_0 = 16 hidden neurons.

"Bad" local minima are vanishing in a multilayer neural net: a proof with more reasonable assumptions than before

In networks with a single hidden layer, the volume of suboptimal local minima exponentially decreases in comparison to global minima.

This paper aims to answer why standard SGD based algorithms on neural network converge to 'good' solutions.

Exponentially vanishing sub-optimal local minima in multilayer neural networks

multilayer neural
local minima
vanish
network
exponentially


Deep neural networks are vulnerable to adversarial examples, which can mislead classifiers by adding imperceptible perturbations.

An intriguing property of adversarial examples is their good transferability, making black-box attacks feasible in real-world applications.

Due to the threat of adversarial attacks, many methods have been proposed to improve the robustness, and several state-of-the-art defenses are shown to be robust against transferable adversarial examples.

In this paper, we identify the attention shift phenomenon, which may hinder the transferability of adversarial examples to the defense models.

It indicates that the defenses rely on different discriminative regions to make predictions compared with normally trained models.

Therefore, we propose an attention-invariant attack method to generate more transferable adversarial examples.

Extensive experiments on the ImageNet dataset validate the effectiveness of the proposed method.

Our best attack fools eight state-of-the-art defenses at an 82% success rate on average based only on the transferability, demonstrating the insecurity of the defense techniques.

We propose an attention-invariant attack method to generate more transferable adversarial examples for black-box attacks, which can fool state-of-the-art defenses with a high success rate.

The paper proposes a new way of overcoming state of the art defences against adversarial attacks on CNN.

This paper suggests that "attention shift" is a key property behind failure of adversarial attacks to transfer and propose an attention-invariant attack method

Evading Defenses to Transferable Adversarial Examples by Mitigating Attention Shift

attention-invariant attack method
transferable adversarial example
defense
attention shift


We present Merged-Averaged Classifiers via Hashing (MACH) for $K$-classification with large $K$. Compared to traditional one-vs-all classifiers that require $O(Kd)$ memory and inference cost, MACH only need $O(d\log{K})$ memory while only requiring $O(K\log{K} + d\log{K})$ operation for inference.

MACH is the first generic $K$-classification algorithm, with provably theoretical guarantees, which requires $O(\log{K})$ memory without any assumption on the relationship between classes.

MACH uses universal hashing to reduce classification with a large number of classes to few independent classification task with very small (constant) number of classes.

We provide theoretical quantification of accuracy-memory tradeoff by showing the first connection between extreme classification and heavy hitters.

With MACH we can train ODP dataset with 100,000 classes and 400,000 features on a single Titan X GPU (12GB), with the classification accuracy of 19.28\%, which is the best-reported accuracy on this dataset.

Before this work, the best performing baseline is a one-vs-all classifier that requires 40 billion parameters (320 GB model size) and achieves 9\% accuracy.

In contrast, MACH can achieve 9\% accuracy with 480x reduction in the model size (of mere 0.6GB).

With MACH, we also demonstrate complete training of fine-grained imagenet dataset (compressed size 104GB), with 21,000 classes, on a single GPU.

How to Training 100,000 classes on a single GPU

Proposes an efficient hashing method MACH for softmax approximation in the context of large output space, which saves both memory and computation.

A method for classification scheme for problems involving a large number of classes in a multi-class setting demonstrated on ODP and Imagenet-21K datasets

The paper presents a hashing based scheme for reducing memory and computation time for K-way classification when K is large

MACH: Embarrassingly parallel $K$-class classification in $O(d\log{K})$ memory and $O(K\log{K} + d\log{K})$ time, instead of $O(Kd)$

class
memory computation
mach
large
classification
scheme
time
method


Gradient-based optimization is the foundation of deep learning and reinforcement learning.

Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy.

We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables, based on gradients of a learned function.

These estimators can be jointly trained with model parameters or policies, and are applicable in both discrete and continuous settings.

We give unbiased, adaptive analogs of state-of-the-art reinforcement learning methods such as advantage actor-critic.

We also demonstrate this framework for training discrete latent-variable models.

We present a general method for unbiased estimation of gradients of black-box functions of random variables. We apply this method to discrete variational inference and reinforcement learning.

Suggests a new approach to performing gradient descent for blackbox optimization or training discrete latent variable models.

Backpropagation through the Void: Optimizing control variates for black-box gradient estimation

black-box gradient estimation variable discrete


Do GANS (Generative Adversarial Nets) actually learn the target distribution?

The foundational paper of Goodfellow et al. (2014) suggested they do, if they were given sufficiently large deep nets, sample size, and computation time.

A recent theoretical analysis in Arora et al. (2017) raised doubts whether the same holds when discriminator has bounded size.

It showed that the training objective can approach its optimum value even if the generated distribution has very low support.

In other words, the training objective is unable to prevent mode collapse.

The current paper makes two contributions.

(1) It proposes a novel test for estimating support size using the birthday paradox of discrete probability.

Using this  evidence is presented that well-known GANs approaches do learn distributions of fairly low support.

(2) It theoretically studies encoder-decoder GANs architectures (e.g., BiGAN/ALI), which were proposed to learn more meaningful features via GANs, and consequently to also solve the mode-collapse issue.

Our result shows that such encoder-decoder training objectives also cannot guarantee learning of the full distribution because they cannot prevent serious mode collapse.

More seriously, they cannot prevent learning meaningless codes for data, contrary to usual intuition.

We propose a support size estimator of GANs's learned distribution to show they indeed suffer from mode collapse, and we prove that encoder-decoder GANs do not avoid the issue as well.

The paper attempts to estimate the size of the support for solutions produced by typical GANs experimentally.

This paper proposes a clever new test based on the birthday paradox for measuring diversity in generated sample, with experiment results interpreted to mean that mode collapse is strong in a number of state-of-the-art generative models.

The paper uses birthday paradox to show that some GAN architectures generate distributions with fairly low support.

Do GANs learn the distribution? Some Theory and Empirics

birthday paradox
mode collapse
support
learn distribution
gan
show
size
generate


Due to their complex nature, it is hard to characterize the ways in which machine learning models can misbehave or be exploited when deployed.

Recent work on adversarial examples, i.e. inputs with minor perturbations that result in substantially different model predictions, is helpful in evaluating the robustness of these models by exposing the adversarial scenarios where they fail.

However, these malicious perturbations are often unnatural, not semantically meaningful, and not applicable to complicated domains such as language.

In this paper, we propose a framework to generate natural and legible adversarial examples that lie on the data manifold, by searching in semantic space of dense and continuous data representation, utilizing the recent advances in generative adversarial networks.

We present generated adversaries to demonstrate the potential of the proposed approach for black-box classifiers for a wide range of applications such as image classification, textual entailment, and machine translation.

We include experiments to show that the generated adversaries are natural, legible to humans, and useful in evaluating and analyzing black-box classifiers.

We propose a framework to generate “natural” adversaries against black-box classifiers for both visual and textual domains, by doing the search for adversaries in the latent semantic space.

Suggests a method for creation of semantical adversary examples.

Proposes a framework to generate natural adversarial examples by searching adversaries in a latent space of dense and continuous data representation

Generating Natural Adversarial Examples

search adversary latent
framework generate
natural adversarial example
space


Kronecker-factor Approximate Curvature (Martens & Grosse, 2015) (K-FAC) is a 2nd-order optimization method which has been shown to give state-of-the-art performance on large-scale neural network optimization tasks (Ba et al., 2017).

It is based on an approximation to the Fisher information matrix (FIM) that makes assumptions about the particular structure of the network and the way it is parameterized.

The original K-FAC method was applicable only to fully-connected networks, although it has been recently extended by Grosse & Martens (2016) to handle convolutional networks as well.

In this work we extend the method to handle RNNs by introducing a novel approximation to the FIM for RNNs.

This approximation works by modelling the covariance structure between the gradient contributions at different time-steps using a chain-structured linear Gaussian graphical model, summing the various cross-covariances, and computing the inverse in closed form.

We demonstrate in experiments that our method significantly outperforms general purpose state-of-the-art optimizers like SGD with momentum and Adam on several challenging RNN training tasks.

We extend the K-FAC method to RNNs by developing a new family of Fisher approximations.

The authors extends the K-FAC method to RNNs and presents 3 ways of approximating F, showing optimization results on 3 datasets, which outperforms ADAM in both number of updates and computation time.

Proposes to extend the Kronecker-factor Appropriate Curvature optimization method to the setting of recurrent neural networks.

The authors present a second-order method that is specifically designed for RNNs

Kronecker-factored Curvature Approximations for Recurrent Neural Networks

extend k-fac method rnns
recurrent neural network
approximation
author
present
optimization
curvature


Bayesian inference is known to provide a general framework for incorporating prior knowledge or specific properties into machine learning models via carefully choosing a prior distribution.

In this work, we propose a new type of prior distributions for convolutional neural networks, deep weight prior (DWP), that exploit generative models to encourage a specific structure of trained convolutional filters e.g., spatial correlations of weights.

We define DWP in the form of an implicit distribution and propose a method for variational inference with such type of implicit priors.

In experiments, we show that DWP improves the performance of Bayesian neural networks when training data are limited, and initialization of weights with samples from DWP accelerates training of conventional convolutional neural networks.

The generative model for kernels of convolutional neural networks, that acts as a prior distribution while training on new datasets.

A method for modeling convolutional neural networks using a Bayes method.

Proposes the 'deep weight prior': the idea is to elicit a prior on an auxilary dataset and then use that prior over the CNN filters to jump start inference for a data set of interest.

This paper explores learning informative priors for convolutional neural network models with similar problem domains by using autoencoders to obtain an expressive prior on the filtered weights of the trained networks.

The Deep Weight Prior

convolutional neural network
deep weight prior
model
dataset
filter


The high dimensionality of hyperspectral imaging forces unique challenges in scope, size and processing requirements.

Motivated by the potential for an in-the-field cell sorting detector, we examine a Synechocystis sp.

PCC 6803 dataset wherein cells are grown alternatively in nitrogen rich or deplete cultures.

We use deep learning techniques to both successfully classify cells and generate a mask segmenting the cells/condition from the background.

Further, we use the classification accuracy to guide a data-driven, iterative feature selection method, allowing the design neural networks requiring 90% fewer input features with little accuracy degradation.

We applied deep learning techniques to hyperspectral image segmentation and iterative feature sampling.

Proposes a greedy scheme to select a subset of highly correlated spectral features in a classification task.

The paper explores the use of neural networks for classification and segmentation of hypersepctral imaging (HSI) of cells.

Classifying cells and implementing cell segmentation based on deep learning techniques with reduction of input features

Data-driven Feature Sampling for Deep Hyperspectral Classification and Segmentation

deep learn technique
classification
segmentation
cell
feature
hyperspectral


Data Interpretation is an important part of Quantitative Aptitude exams and requires an individual to answer questions grounded in plots such as bar charts, line graphs, scatter plots, \textit{etc}.

Recently, there has been an increasing interest in building models which can perform this task by learning from datasets containing triplets of the form \{plot, question, answer\}.

Two such datasets have been proposed in the recent past which contain plots generated from synthetic data with limited

(iii) answer vocabulary and hence do not adequately capture the challenges posed by this task.

To overcome these limitations of existing datasets, we introduce a new dataset containing $9.7$ million question-answer pairs grounded over $270,000$ plots with three main differentiators.

First, the plots in our dataset contain a wide variety of realistic $x$-$y$ variables such as CO2 emission, fertility rate, \textit{etc.

} extracted from  real word data sources such as World Bank, government sites, \textit{etc}.

Second, the questions in our dataset are more complex as they are based on templates extracted from interesting questions asked by a crowd of workers using a fraction of these plots.

Lastly, the answers in our dataset are not restricted to a small vocabulary and a large fraction of the answers seen at test time are not present in the training vocabulary.

As a result, existing models for Visual Question Answering which largely use end-to-end models in a multi-class classification framework cannot be used for this task.

We establish initial results on this dataset and emphasize the complexity of the task using a multi-staged modular pipeline with various sub-components to

(i) extract relevant data from the plot and convert it to a semi-structured table

(ii) combine the question with this table and use compositional semantic parsing to arrive at a logical form from which the answer can be derived.

We believe that such a modular framework is the best way to go forward as it would enable the research community to independently make progress on all the sub-tasks involved in plot question answering.

We created a new dataset for data interpretation over plots and also propose a baseline for the same.

The authors propose a pipeline to solve the DIP problem involving learning from datasets containing triplets of the form {plot, question, answer}

Proposes an algorithm that can interpret data shown in scientific plots.

Data Interpretation and Reasoning Over Scientific Plots

datum interpretation
scientific plot
dataset


Learning to predict complex time-series data is a fundamental challenge in a range of disciplines including Machine Learning, Robotics, and Natural Language Processing.

Predictive State Recurrent Neural Networks (PSRNNs) (Downey et al.) are a state-of-the-art approach for modeling time-series data which combine the benefits of probabilistic filters and Recurrent Neural Networks into a single model.

PSRNNs leverage the concept of Hilbert Space Embeddings of distributions (Smola et al.) to embed predictive states into a Reproducing Kernel Hilbert Space, then estimate, predict, and update these embedded states using Kernel Bayes Rule.

Practical implementations of PSRNNs are made possible by the machinery of Random Features, where input features are mapped into a new space where dot products approximate the kernel well.

Unfortunately PSRNNs often require a large number of RFs to obtain good results, resulting in large models which are slow to execute and slow to train.

Orthogonal Random Features (ORFs) (Choromanski et al.) is an improvement on RFs which has been shown to decrease the number of RFs required for pointwise kernel approximation.

Unfortunately, it is not clear that ORFs can be applied to PSRNNs, as PSRNNs rely on Kernel Ridge Regression as a core component of their learning algorithm, and the theoretical guarantees of ORF do not apply in this setting.

In this paper, we extend the theory of ORFs to Kernel Ridge Regression and show that ORFs can be used to obtain Orthogonal PSRNNs (OPSRNNs), which are smaller and faster than PSRNNs.

In particular, we show that OPSRNN models clearly outperform LSTMs and furthermore, can achieve accuracy similar to PSRNNs with an order of magnitude smaller number of features needed.

Improving Predictive State Recurrent Neural Networks via Orthogonal Random Features

Proposes improving the performances of Predicitve State Recurrent Neural Networks by considering Orthogonal Random Features.

The paper tackles the problem of training predictive state recurrent neural networks and makes two contributions.

Initialization matters: Orthogonal Predictive State Recurrent Neural Networks


Training deep neural networks requires many training samples, but in practice training labels are expensive to obtain and may be of varying quality, as some may be from trusted expert labelers while others might be from heuristics or other sources of weak supervision such as crowd-sourcing.

This creates a fundamental quality- versus-quantity trade-off in the learning process.

Do we learn from the small amount of high-quality data or the potentially large amount of weakly-labeled data?

We argue that if the learner could somehow know and take the label-quality into account when learning the data representation, we could get the best of both worlds.

To this end, we propose “fidelity-weighted learning” (FWL), a semi-supervised student- teacher approach for training deep neural networks using weakly-labeled data.

FWL modulates the parameter updates to a student network (trained on the task we care about) on a per-sample basis according to the posterior confidence of its label-quality estimated by a teacher (who has access to the high-quality labels).

Both student and teacher are learned from the data.

We evaluate FWL on two tasks in information retrieval and natural language processing where we outperform state-of-the-art alternative semi-supervised methods, indicating that our approach makes better use of strong and weak labels, and leads to better task-dependent data representations.

We propose Fidelity-weighted Learning, a semi-supervised teacher-student approach for training neural networks using weakly-labeled data.

This paper suggests an approach for learning with weak supervision by using a clean and a noisy dataset and assuming a teacher and student networks

The paper attemps to train deep neural network models with few labelled training samples.

The authors propose an approach for training deep learning models for situations where there is not enough reliable annotated data.

Fidelity-Weighted Learning

approach train
datum
fidelity-weighted learning
neural network
model
deep