The backpropagation algorithm is the most popular algorithm training neural networks nowadays.

However, it suffers from the forward locking, backward locking and update locking problems, especially when a neural network is so large that its layers are distributed across multiple devices.

Existing solutions either can only handle one locking problem or lead to severe accuracy loss or memory inefficiency.

Moreover, none of them consider the straggler problem among devices.

In this paper, we propose \textbf{Layer-wise Staleness} and a novel efficient training algorithm, \textbf{Diversely Stale Parameters} (DSP), which can address all these challenges without loss of accuracy nor memory issue.

We also analyze the convergence of DSP with two popular gradient-based methods and prove that both of them are guaranteed to converge to critical points for non-convex problems.

Finally, extensive experimental results on training deep convolutional neural networks demonstrate that our proposed DSP algorithm can achieve significant training speedup with stronger robustness and better generalization than compared methods.

We propose Diversely Stale Parameters to break lockings of the backpropoagation algorithm and train a CNN in parallel.

Diversely Stale Parameters for Efficient Training of Deep Convolutional Networks


The emergence of language in multi-agent settings is a promising research direction to ground natural language in simulated agents.

If AI would be able to understand the meaning of language through its using it, it could also transfer it to other situations flexibly.

That is seen as an important step towards achieving general AI.

The scope of emergent communication is so far, however, still limited.

It is necessary to enhance the learning possibilities for skills associated with communication to increase the emergable complexity.

We took an example from human language acquisition and the importance of the empathic connection in this process.

We propose an approach to introduce the notion of empathy to multi-agent deep reinforcement learning.

We extend existing approaches on referential games with an auxiliary task for the speaker to predict the listener's mind change improving the learning time.

Our experiments show the high potential of this architectural element by doubling the learning speed of the test setup.

An auxiliary prediction task can speed up learning in language emergence setups.

Enhancing Language Emergence through Empathy


Image paragraph captioning is the task of automatically generating multiple sentences for describing images in grain-fined and coherent text.

Existing typical deep learning-based models for image captioning consist of an image encoder to extract visual features and a language model decoder, which has shown promising results in single high-level sentence generation.

However, only the word-level scalar guiding signal is available when the image encoder is optimized to extract visual features.

The inconsistency between the parallel extraction of visual features and sequential text supervision limits its success when the length of the generated text is long (more than 50 words).

In this paper, we propose a new module, called the Text Embedding Bank (TEB) module, to address the problem for image paragraph captioning.

This module uses the paragraph vector model to learn fixed-length feature representations from a variable-length paragraph.

We refer to the fixed-length feature as the TEB.

This TEB module plays two roles to benefit paragraph captioning performance.

First, it acts as a form of global and coherent deep supervision to regularize visual feature extraction in the image encoder.

Second, it acts as a distributed memory to provide features of the whole paragraph to the language model, which alleviating the long-term dependency problem.

Adding this module to two existing state-of-the-art methods achieves a new state-of-the-art result by a large margin on the paragraph captioning Visual Genome dataset.

Text Embedding Bank Module for Detailed Image Paragraph Caption


Learning Mahalanobis metric spaces is an important problem that has found numerous applications.

Several algorithms have been designed for this problem, including Information Theoretic Metric Learning (ITML) [Davis et al. 2007] and Large Margin Nearest Neighbor (LMNN) classification [Weinberger and Saul 2009].

We consider a formulation of Mahalanobis metric learning as an optimization problem,where the objective is to minimize the number of violated similarity/dissimilarity constraints.

We show that for any fixed ambient dimension, there exists a fully polynomial time approximation scheme (FPTAS) with nearly-linear running time.

This result is obtained using tools from the theory of linear programming in low dimensions.

We also discuss improvements of the algorithm in practice, and present experimental results on synthetic and real-world data sets.

Our algorithm is fully parallelizable and performs favorably in the presence of adversarial noise.

Fully parallelizable and adversarial-noise resistant metric learning algorithm with theoretical guarantees.

Learning Mahalanobis Metric Spaces via Geometric Approximation Algorithms


Standard image captioning tasks such as COCO and Flickr30k are factual, neutral in tone and (to a human) state the obvious (e.g., “a man playing a guitar”).

While such tasks are useful to verify that a machine understands the content of an image,  they are not engaging to humans as captions.

With this in mind we define a new task, Personality-Captions, where the goal is to be as engaging to humans as possible by incorporating controllable style and personality traits.

We collect and release a large dataset of 201,858 of such captions conditioned over 215 possible traits.

We build models that combine existing work from (i) sentence representations (Mazaré et al., 2018) with Transformers trained on 1.7 billion dialogue examples; and (ii) image representations (Mahajan et al., 2018) with ResNets trained on 3.5 billion social media images.

We obtain state-of-the-art performance on Flickr30k and COCO, and strong performance on our new task.

Finally, online evaluations validate that our task and models are engaging to humans, with our best model close to human performance.

We develop engaging image captioning models conditioned on personality that are also state of the art on regular captioning tasks.

Engaging Image Captioning Via Personality


Machine learning (ML) models trained by differentially private stochastic gradient descent (DP-SGD) have much lower utility than the non-private ones.

To mitigate this degradation, we propose a DP Laplacian smoothing SGD (DP-LSSGD) to train ML models with differential privacy (DP) guarantees.

At the core of DP-LSSGD is the Laplacian smoothing, which smooths out the Gaussian noise used in the Gaussian mechanism.

Under the same amount of noise used in the Gaussian mechanism, DP-LSSGD attains the same DP guarantee, but a better utility especially for the scenarios with strong DP guarantees.

In practice, DP-LSSGD makes training both convex and nonconvex ML models more stable and enables the trained models to generalize better.

The proposed algorithm is simple to implement and the extra computational complexity and memory overhead compared with DP-SGD are negligible.

DP-LSSGD is applicable to train a large variety of ML models, including DNNs.

We propose a differentially private Laplacian smoothing stochastic gradient descent to train machine learning models with better utility and maintain differential privacy guarantees.

DP-LSSGD: An Optimization Method to Lift the Utility in Privacy-Preserving ERM


We study the robust one-bit compressed sensing problem whose goal is to design an algorithm that faithfully recovers any sparse target vector

$\theta_0\in\mathbb{R}^d$ \emph{uniformly} from $m$ quantized noisy measurements.

Under the assumption that the measurements are sub-Gaussian,  to recover any $k$-sparse $\theta_0$ ($k\ll d$) \emph{uniformly} up to an error $\varepsilon$ with high probability, the best known computationally tractable algorithm requires\footnote{Here, an algorithm is ``computationally tractable'' if it has provable convergence guarantees.

The notation $\tilde{\mathcal{O}}(\cdot)$ omits a logarithm factor of $\varepsilon^{-1}$.

In this paper, we consider a new framework for the one-bit sensing problem where the sparsity is implicitly enforced via mapping a low dimensional representation $x_0$ through a known $n$-layer ReLU generative network $G:\mathbb{R}^k\rightarrow\mathbb{R}^d$.

Such a framework poses low-dimensional priors on $\theta_0$ without a known basis.

We propose to recover the target $G(x_0)$ via an unconstrained empirical risk minimization (ERM) problem under a much weaker \emph{sub-exponential  measurement assumption}.

For such a problem, we establish a joint statistical and computational analysis

. In particular, we prove that the ERM estimator in this new framework achieves an improved statistical rate of $m=\tilde{\mathcal{O}} (kn\log d /\epsilon^2)$ recovering any $G(x_0)$ uniformly up to an error $\varepsilon$.

Moreover, from the lens of computation, we prove that under proper conditions on the ReLU weights, our proposed empirical risk, despite non-convexity, has no stationary point outside of small neighborhoods around the true representation $x_0$ and its negative multiple.

Furthermore, we show that the global minimizer of the empirical risk stays within the neighborhood around $x_0$ rather than its negative multiple.

Our analysis sheds some light on the possibility of inverting a deep generative model under partial and quantized measurements, complementing the recent success of using deep generative models for inverse problems.

We provide statistical and computational analysis of one-bit compressed sensing problem with a generative prior.

Robust One-Bit Recovery via ReLU Generative Networks: Improved Statistical Rate and Global Landscape Analysis


We introduce an unsupervised structure learning algorithm for deep, feed-forward, neural networks.

We propose a new interpretation for depth and inter-layer connectivity where a hierarchy of independencies in the input distribution is encoded in the network structure.

This results in structures allowing neurons to connect to neurons in any deeper layer skipping intermediate layers.

Moreover, neurons in deeper layers encode low-order (small condition sets) independencies and have a wide scope of the input, whereas neurons in the first layers encode higher-order (larger condition sets) independencies and have a narrower scope.

Thus, the depth of the network is automatically determined---equal to the maximal order of independence in the input distribution, which is the recursion-depth of the algorithm.

The proposed algorithm constructs two main graphical models:

1) a generative latent graph (a deep belief network) learned from data and

2) a deep discriminative graph constructed from the generative latent graph.

We prove that conditional dependencies between the nodes in the learned generative latent graph are preserved in the class-conditional discriminative graph.

Finally, a deep neural network structure is constructed based on the discriminative graph.

We demonstrate on image classification benchmarks that the algorithm replaces the deepest layers (convolutional and dense layers) of common convolutional networks, achieving high classification accuracy, while constructing significantly smaller structures.

The proposed structure learning algorithm requires a small computational cost and runs efficiently on a standard desktop CPU.

A principled approach for structure learning of deep neural networks with a new interpretation for depth and inter-layer connectivity.

Unsupervised Deep Structure Learning by Recursive Dependency Analysis


L1 and L2 regularizers are critical tools in machine learning due to their ability to simplify solutions.

However, imposing strong L1 or L2 regularization with gradient descent method easily fails, and this limits the generalization ability of the underlying neural networks.

To understand this phenomenon, we investigate how and why training fails for strong regularization.

Specifically, we examine how gradients change over time for different regularization strengths and provide an analysis why the gradients diminish so fast.

We find that there exists a tolerance level of regularization strength, where the learning completely fails if the regularization strength goes beyond it.

We propose a simple but novel method, Delayed Strong Regularization, in order to moderate the tolerance level.

Experiment results show that our proposed approach indeed achieves strong regularization for both L1 and L2 regularizers and improves both accuracy and sparsity on public data sets.

We investigate how and why strong L1/L2 regularization fails and propose a method than can achieve strong regularization.

Achieving Strong Regularization for Deep Neural Networks


Despite neural network’s high performance, the lack of interpretability has been the main bottleneck for its safe usage in practice.

In domains with high stakes (e.g., medical diagnosis), gaining insights into the network is critical for gaining trust and being adopted.

One of the ways to improve interpretability of a NN is to explain the importance of a particular concept (e.g., gender) in prediction.

This is useful for explaining reasoning behind the networks’ predictions, and for revealing any biases the network may have.

This work aims to provide quantitative answers to \textit{the relative importance of concepts of interest} via concept activation vectors (CAV).

In particular, this framework enables non-machine learning experts to express concepts of interests and  test hypotheses using examples (e.g., a set of pictures that illustrate  the concept).

Testing with CAV, for example, can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other set of concepts.

Interpreting with CAV does not require any retraining or modification of the network.

We show that many levels of meaningful concepts are learned (e.g., color, texture, objects, a person’s occupation), and we present CAV’s \textit{empirical deepdream} — where we maximize an activation using a set of example pictures.

We show how various insights can be gained from the relative importance testing with CAV.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.

TCAV: Relative concept importance testing with Linear Concept Activation Vectors

concept activation vector
relative
testing
importance


We present a new family of objective functions, which we term the Conditional Entropy Bottleneck (CEB).

These objectives are motivated by the Minimum Necessary Information (MNI) criterion.

We demonstrate the application of CEB to classification tasks.

We show that CEB gives: well-calibrated predictions; strong detection of challenging out-of-distribution examples and powerful whitebox adversarial examples; and substantial robustness to those adversaries.

Finally, we report that CEB fails to learn from information-free datasets, providing a possible resolution to the problem of generalization observed in Zhang et al. (2016).

The Conditional Entropy Bottleneck is an information-theoretic objective function for learning optimal representations.

The Conditional Entropy Bottleneck


Deep representation learning has become one of the most widely adopted approaches for visual search, recommendation, and identification.

Retrieval of such  representations from a large database is however computationally challenging.

Approximate methods based on learning compact representations, have been widely explored for this problem, such as locality sensitive hashing, product quantization, and PCA.

In this work, in contrast to learning compact representations, we propose to learn high dimensional and sparse representations that have similar representational capacity as dense embeddings while being more efficient due to sparse matrix multiplication operations which can be much faster than dense multiplication.

Following the key insight that the number of operations decreases quadratically with the sparsity of embeddings provided the non-zero entries are distributed uniformly across dimensions, we propose a novel approach to learn such distributed sparse embeddings via the use of a carefully constructed regularization function that directly minimizes a continuous relaxation of the number of floating-point operations (FLOPs) incurred during retrieval.

Our experiments show that our approach is competitive to the other baselines and yields a similar or better speed-vs-accuracy tradeoff on practical datasets.

We propose an approach to learn sparse high dimensional representations that are fast to search, by incorporating a surrogate of the number of operations directly into the loss function.

Minimizing FLOPs to Learn Efficient Sparse Representations


Model interpretability and systematic, targeted model adaptation present central challenges in deep learning.

In the domain of intuitive physics, we study the task of visually predicting stability of block towers with the goal of understanding and influencing the model's reasoning.

Firstly, we introduce neural stethoscopes as a framework for quantifying the degree of importance of specific factors of influence in deep networks as well as for actively promoting and suppressing information as appropriate.

In doing so, we unify concepts from multitask learning as well as training with auxiliary and adversarial losses.

Secondly, we deploy the stethoscope framework to provide an in-depth analysis of a state-of-the-art deep neural network for stability prediction, specifically examining its physical reasoning.

We show that the baseline model is susceptible to being misled by incorrect visual cues.

This leads to a performance breakdown to the level of random guessing when training on scenarios where visual cues are inversely correlated with stability.

Using stethoscopes to promote meaningful feature extraction increases performance from 51% to 90% prediction accuracy.

Conversely, training on an easy dataset where visual cues are positively correlated with stability, the baseline model learns a bias leading to poor performance on a harder dataset.

Using an adversarial stethoscope, the network is successfully de-biased, leading to a performance increase from 66% to 88%.

Combining auxiliary and adversarial training to interrogate and help physical understanding.

Guiding Physical Intuition with Neural Stethoscopes


Flow based models such as Real NVP are an extremely powerful approach to density estimation.

However, existing flow based models are restricted to transforming continuous densities over a continuous input space into similarly continuous distributions over continuous latent variables.

This makes them poorly suited for modeling and representing discrete structures in data distributions, for example class membership or discrete symmetries.

To address this difficulty, we present a normalizing flow architecture which relies on domain partitioning using locally invertible functions, and possesses both real and discrete valued latent variables.

This Real and Discrete (RAD) approach retains the desirable normalizing flow properties of exact sampling, exact inference, and analytically computable probabilities, while at the same time allowing simultaneous modeling of both continuous and discrete structure in a data distribution.

Flow based models, but non-invertible, to also learn discrete variables

A RAD approach to deep mixture models


We investigate the loss surface of neural networks.

We prove that even for one-hidden-layer networks with "slightest" nonlinearity, the empirical risks have spurious local minima in most cases.

Our results thus indicate that in general "no spurious local minim" is a property limited to deep linear networks, and insights obtained from linear networks may not be robust.

Specifically, for ReLU(-like) networks we constructively prove that for almost all practical datasets there exist infinitely many local minima.

We also present a counterexample for more general activations (sigmoid, tanh, arctan, ReLU, etc.), for which there exists a bad local minimum.

Our results make the least restrictive assumptions relative to existing results on spurious local optima in neural networks.

We complete our discussion by presenting a comprehensive characterization of global optimality for deep linear networks, which unifies other results on this topic.

We constructively prove that even the slightest nonlinear activation functions introduce spurious local minima, for general datasets and activation functions.

Small nonlinearities in activation functions create bad local minima in neural networks


The tasks that an agent will need to solve often aren’t known during training.

However, if the agent knows which properties of the environment we consider im- portant, then after learning how its actions affect those properties the agent may be able to use this knowledge to solve complex tasks without training specifi- cally for them.

Towards this end, we consider a setup in which an environment is augmented with a set of user defined attributes that parameterize the features of interest.

We propose a model that learns a policy for transitioning between “nearby” sets of attributes, and maintains a graph of possible transitions.

Given a task at test time that can be expressed in terms of a target set of attributes, and a current state, our model infers the attributes of the current state and searches over paths through attribute space to get a high level plan, and then uses its low level policy to execute the plan.

We show in grid-world games and 3D block stacking that our model is able to generalize to longer, more complex tasks at test time even when it only sees short, simple tasks at train time.

Compositional attribute-based planning that generalizes to long test tasks, despite being trained on short & simple tasks.

Composable Planning with Attributes


The ability to learn new concepts with small amounts of data is a critical aspect of intelligence that has proven challenging for deep learning methods.

Meta-learning has emerged as a promising technique for leveraging data from previous tasks to enable efficient learning of new tasks.

However, most meta-learning algorithms implicitly require that the meta-training tasks be mutually-exclusive, such that no single model can solve all of the tasks at once.

For example, when creating tasks for few-shot image classification, prior work uses a per-task random assignment of image classes to N-way classification labels.

If this is not done, the meta-learner can ignore the task training data and learn a single model that performs all of the meta-training tasks zero-shot, but does not adapt effectively to new image classes.

This requirement means that the user must take great care in designing the tasks, for example by shuffling labels or removing task identifying information from the inputs.

In some domains, this makes meta-learning entirely inapplicable.

In this paper, we address this challenge by designing a meta-regularization objective using information theory that places precedence on data-driven adaptation.

This causes the meta-learner to decide what must be learned from the task training data and what should be inferred from the task testing input.

By doing so, our algorithm can successfully use data from non-mutually-exclusive tasks to efficiently adapt to novel tasks.

We demonstrate its applicability to both contextual and gradient-based meta-learning algorithms, and apply it in practical settings where applying standard meta-learning has been difficult.

Our approach substantially outperforms standard meta-learning algorithms in these settings.

We identify and formalize the memorization problem in meta-learning and solve this problem with novel meta-regularization method, which greatly expand the domain that meta-learning can be  applicable to and effective on.

Meta-Learning without Memorization


Learning an efficient update rule from data that promotes rapid learning of new tasks from the same distribution remains an open problem in meta-learning.

Typically, previous works have approached this issue either by attempting to train a neural network that directly produces updates or by attempting to learn better initialisations or scaling factors for a gradient-based update rule.

On one hand, directly producing an update forgoes a useful inductive bias and can easily lead to non-converging behaviour.

On the other hand, approaches that try to control a gradient-based update rule typically resort to computing gradients through the learning process to obtain their meta-gradients, leading to methods that can not scale beyond few-shot task adaptation.

In this work we propose Warped Gradient Descent (WarpGrad), a method that intersects these approaches to mitigate their limitations.

WarpGrad meta-learns an efficiently parameterised preconditioning matrix that facilitates gradient descent across the task distribution.

Preconditioning arises by interleaving non-linear layers, referred to as warp-layers, between the layers of a task-learner.

Warp-layers are meta-learned without backpropagating through the task training process in a manner similar to methods that learn to directly produce updates.

WarpGrad is computationally efficient, easy to implement, and can scale to arbitrarily large meta-learning problems.

We provide a geometrical interpretation of the approach and evaluate its effectiveness in a variety of settings, including few-shot, standard supervised, continual and reinforcement learning.

We propose a novel framework for meta-learning a gradient-based update rule that scales to beyond few-shot learning and is applicable to any form of learning, including continual learning.

Meta-Learning with Warped Gradient Descent


In recent years, deep neural network approaches have been widely adopted for machine learning tasks, including classification.

However, they were shown to be vulnerable to adversarial perturbations: carefully crafted small perturbations can cause misclassification of legitimate images.

We propose Defense-GAN, a new framework leveraging the expressive capability of generative models to defend deep neural networks against such attacks.

Defense-GAN is trained to model the distribution of unperturbed images.

At inference time, it finds a close output to a given image which does not contain the adversarial changes.

This output is then fed to the classifier.

Our proposed method can be used with any classification model and does not modify the classifier structure or training procedure.

It can also be used as a defense against any attack as it does not assume knowledge of the process for generating the adversarial examples.

We empirically show that Defense-GAN is consistently effective against different attack methods and improves on existing defense strategies.

Defense-GAN uses a Generative Adversarial Network to defend against white-box and black-box attacks in classification models.

Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models

generative attack defense-gan model adversarial


We study the problem of learning similarity functions over very large corpora using neural network embedding models.

These models are typically trained using SGD with random sampling of unobserved pairs, with a sample size that grows quadratically with the corpus size, making it expensive to scale.

We propose new efficient methods to train these models without having to sample unobserved pairs.

Inspired by matrix factorization, our approach relies on adding a global quadratic penalty and expressing this term as the inner-product of two generalized Gramians.

We show that the gradient of this term can be efficiently computed by maintaining estimates of the Gramians, and develop variance reduction schemes to improve the quality of the estimates.

We conduct large-scale experiments that show a significant improvement both in training time and generalization performance compared to sampling methods.

We develop efficient methods to train neural embedding models with a dot-product structure, by reformulating the objective function in terms of generalized Gram matrices, and maintaining estimates of those matrices.

Efficient Training on Very Large Corpora via Gramian Estimation


For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusive to a single task, genre, or dataset.

In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks.

By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks.

GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models.

We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all tasks performs better than training a separate model per task.

However, the low absolute performance of our best model indicates the need for improved general NLU systems.

We present a multi-task benchmark and analysis platform for evaluating generalization in natural language understanding systems.

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

multi-task benchmark analysis platform
natural language understanding


A variety of cooperative multi-agent control problems require agents to achieve individual goals while contributing to collective success.

This multi-goal multi-agent setting poses difficulties for recent algorithms, which primarily target settings with a single global reward, due to two new challenges: efficient exploration for learning both individual goal attainment and cooperation for others' success, and credit-assignment for interactions between actions and goals of different agents.

To address both challenges, we restructure the problem into a novel two-stage curriculum, in which single-agent goal attainment is learned prior to learning multi-agent cooperation, and we derive a new multi-goal multi-agent policy gradient with a credit function for localized credit assignment.

We use a function augmentation scheme to bridge value and policy functions across the curriculum.

The complete architecture, called CM3, learns significantly faster than direct adaptations of existing algorithms on three challenging multi-goal multi-agent problems: cooperative navigation in difficult formations, negotiating multi-vehicle lane changes in the SUMO traffic simulator, and strategic cooperation in a Checkers environment.

A modular method for fully cooperative multi-goal multi-agent reinforcement learning, based on curriculum learning for efficient exploration and credit assignment for action-goal interactions.

CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning

multi-agent reinforcement learning
cooperative multi-goal


We identify two issues with the family of algorithms based on the Adversarial Imitation Learning framework.

The first problem is implicit bias present in the reward functions used in these algorithms.

While these biases might work well for some environments, they can also lead to sub-optimal behavior in others.

Secondly, even though these algorithms can learn from few expert demonstrations, they require a prohibitively large number of interactions with the environment in order to imitate the expert for many real-world applications.

In order to address these issues, we propose a new algorithm called Discriminator-Actor-Critic that uses off-policy Reinforcement Learning to reduce policy-environment interaction sample complexity by an average factor of 10.

Furthermore, since our reward function is designed to be unbiased, we can apply our algorithm to many problems without making any task-specific adjustments.

We address sample inefficiency and reward bias in adversarial imitation learning algorithms such as GAIL and AIRL.

Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning

address sample inefficiency reward bias adversarial imitation learn


Capsule Networks have shown encouraging results on defacto benchmark computer vision datasets such as MNIST, CIFAR and smallNORB.

Although, they are yet to be tested on tasks where (1) the entities detected inherently have more complex internal representations and (2) there are very few instances per class to learn from and (3) where point-wise classification is not suitable.

Hence, this paper carries out experiments on face verification in both controlled and uncontrolled settings that together address these points.

In doing so we introduce Siamese Capsule Networks, a new variant that can be used for pairwise learning tasks.

The model is trained using contrastive loss with l2-normalized capsule encoded pose features.

We find that Siamese Capsule Networks perform well against strong baselines on both pairwise learning datasets, yielding best results in the few-shot learning setting where image pairs in the test set contain unseen subjects.

A variant of capsule networks that can be used for pairwise learning tasks. Results shows that Siamese Capsule Networks work well in the few shot learning setting.

Siamese Capsule Networks


Animals excel at adapting their intentions, attention, and actions to the environment, making them remarkably efficient at interacting with a rich, unpredictable and ever-changing external world, a property that intelligent machines currently lack.

Such adaptation property strongly relies on cellular neuromodulation, the biological mechanism that dynamically controls neuron intrinsic properties and response to external stimuli in a context dependent manner.

In this paper, we take inspiration from cellular neuromodulation to construct a new deep neural network architecture that is specifically designed to learn adaptive behaviours.

The network adaptation capabilities are tested on navigation benchmarks in a meta-learning context and compared with state-of-the-art approaches.

Results show that neuromodulation is capable of adapting an agent to different tasks and that neuromodulation-based approaches provide a promising way of improving adaptation of artificial systems.

This paper introduces neuromodulation in artificial neural networks.

Cellular neuromodulation in artificial networks


Convolution operator is the core of convolutional neural networks (CNNs) and occupies the most computation cost.

To make CNNs more efficient, many methods have been proposed to either design lightweight networks or compress models.

Although some efficient network structures have been proposed, such as MobileNet or ShuffleNet, we find that there still exists redundant information between convolution kernels.

To address this issue, we propose a novel dynamic convolution method named \textbf{DyNet} in this paper, which can adaptively generate convolution kernels based on image contents.

To demonstrate the effectiveness, we apply DyNet on multiple state-of-the-art CNNs.

The experiment results show that DyNet can reduce the computation cost remarkably, while maintaining the performance nearly unchanged.

Specifically, for ShuffleNetV2 (1.0), MobileNetV2 (1.0), ResNet18 and ResNet50, DyNet reduces 40.0%, 56.7%, 68.2% and 72.4% FLOPs respectively while the Top-1 accuracy on ImageNet only changes by +1.0%, -0.27%, -0.6% and -0.08%.

Meanwhile, DyNet further accelerates the inference speed of MobileNetV2 (1.0), ResNet18 and ResNet50 by 1.87x,1.32x and 1.48x on CPU platform respectively.

To verify the scalability, we also apply DyNet on segmentation task, the results show that DyNet can reduces 69.3% FLOPs while maintaining the Mean IoU on segmentation task.

We propose a dynamic convolution method to significantly accelerate inference time of CNNs while maintaining the accuracy.

DyNet: Dynamic Convolution for Accelerating Convolution Neural Networks


Operating deep neural networks on devices with limited resources requires the reduction of their memory footprints and computational requirements.

In this paper we introduce a training method, called look-up table quantization (LUT-Q), which learns a dictionary and assigns each weight to one of the dictionary's values.

We show that this method is very flexible and that many other techniques can be seen as special cases of LUT-Q.

For example, we can constrain the dictionary trained with LUT-Q to generate networks with pruned weight matrices or restrict the dictionary to powers-of-two to avoid the need for multiplications.

In order to obtain fully multiplier-less networks, we also introduce a multiplier-less version of batch normalization.

Extensive experiments on image recognition and object detection tasks show that LUT-Q consistently achieves better performance than other methods with the same quantization bitwidth.

Iteratively Training Look-Up Tables for Network Quantization


An unintended consequence of feature sharing is the model fitting to correlated tasks within the dataset, termed negative transfer.

In this paper, we revisit the problem of negative transfer in multitask setting and find that its corrosive effects are applicable to a wide range of linear and non-linear models, including neural networks.

We first study the effects of negative transfer in a principled way and show that previously proposed counter-measures are insufficient, particularly for trainable features.

We propose an adversarial training approach to mitigate the effects of negative transfer by viewing the problem in a domain adaptation setting.

Finally, empirical results on attribute prediction multi-task on AWA and CUB datasets further validate the need for correcting negative sharing in an end-to-end manner.

We look at negative transfer from a domain adaptation point of view to derive an adversarial learning algorithm.

REVISTING NEGATIVE TRANSFER USING ADVERSARIAL LEARNING


Recent theoretical and experimental results suggest the possibility of using current and near-future quantum hardware in challenging sampling tasks.

In this paper, we introduce free-energy-based reinforcement learning (FERL) as an application of quantum hardware.

We propose a method for processing a quantum annealer’s measured qubit spin configurations in approximating the free energy of a quantum Boltzmann machine (QBM).

We then apply this method to perform reinforcement learning on the grid-world problem using the D-Wave 2000Q quantum annealer.

The experimental results show that our technique is a promising method for harnessing the power of quantum sampling in reinforcement learning tasks.

We train Quantum Boltzmann Machines using a replica stacking method and a quantum annealer to perform a reinforcement learning task.

Reinforcement Learning via Replica Stacking of Quantum Measurements for the Training of Quantum Boltzmann Machines

quantum boltzmann machine
replica stack
reinforcement learning


Deep learning models are vulnerable to adversarial examples crafted by applying human-imperceptible perturbations on benign inputs.

However, under the black-box setting, most existing adversaries often have a poor transferability to attack other defense models.

In this work, from the perspective of regarding the adversarial example generation as an optimization process, we propose two new methods to improve the transferability of adversarial examples, namely Nesterov Iterative Fast Gradient Sign Method (NI-FGSM) and Scale-Invariant attack Method (SIM).

NI-FGSM aims to adapt Nesterov accelerated gradient into the iterative attacks so as to effectively look ahead and improve the transferability of adversarial examples.

While SIM is based on our discovery on the scale-invariant property of deep learning models, for which we leverage to optimize the adversarial perturbations over the scale copies of the input images so as to avoid "overfitting” on the white-box model being attacked and generate more transferable adversarial examples.

NI-FGSM and SIM can be naturally integrated to build a robust gradient-based attack to generate more transferable adversarial examples against the defense models.

Empirical results on ImageNet dataset demonstrate that our attack methods exhibit higher transferability and achieve higher attack success rates than state-of-the-art gradient-based attacks.

We proposed a Nesterov Iterative Fast Gradient Sign Method (NI-FGSM) and a Scale-Invariant attack Method (SIM) that can boost the transferability of adversarial examples for image classification.

Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks


Low bit-width weights and activations are an effective way of combating the increasing need for both memory and compute power of Deep Neural Networks.

In this work, we present a probabilistic training method for Neural Network with both binary weights and activations, called PBNet.

By embracing stochasticity during training, we circumvent the need to approximate the gradient of functions for which the derivative is zero almost always, such as $\textrm{sign}(\cdot)$, while still obtaining a fully Binary Neural Network at test time.

Moreover, it allows for anytime ensemble predictions for improved performance and uncertainty estimates by sampling from the weight distribution.

Since all operations in a layer of the PBNet operate on random variables, we introduce stochastic versions of Batch Normalization and max pooling, which transfer well to a deterministic network at test time.

We evaluate two related training methods for the PBNet: one in which activation distributions are propagated throughout the network, and one in which binary activations are sampled in each layer.

Our experiments indicate that sampling the binary activations is an important element for stochastic training of binary Neural Networks.

We introduce a stochastic training method for training Binary Neural Network with both binary weights and activations.

Probabilistic Binary Neural Networks


The goal of generative models is to model the underlying data distribution of a

Our intuition is that an accurate model should in principle

also include the sample based dataset as part of its induced probability distribution.

To investigate this, we look at fully trained generative models using the Generative

Adversarial Networks (GAN) framework and analyze the resulting generator

Further, we show that the size of the initial

latent space is paramount to allow for an accurate reconstruction of the training

This gives us a link to compression theory, where Autoencoders (AE) are

used to lower bound the reconstruction capabilities of our generative model.

we observe similar results to the perception-distortion tradeoff (Blau & Michaeli

Given a small latent space, the AE produces low quality and the GAN

produces high quality outputs from a perceptual viewpoint.

By increasing the dimensionality of the latent

space the distortion decreases for both models, but the perceptual quality only

We analyze the impact of the latent space of fully trained generators by pseudo inverting them.

Impact of the latent space on the ability of GANs to fit the distribution


We address the problem of open-set authorship verification, a classification task that consists of attributing texts of unknown authorship to a given author when the unknown documents in the test set are excluded from the training set.

We present an end-to-end model-building process that is universally applicable to a wide variety of corpora with little to no modification or fine-tuning.

It relies on transfer learning of a deep language model and uses a generative adversarial network and a number of text augmentation techniques to improve the model's generalization ability.

The language model encodes documents of known and unknown authorship into a domain-invariant space, aligning document pairs as input to the classifier, while keeping them separate.

The resulting embeddings are used to train to an ensemble of recurrent and quasi-recurrent neural networks.

The entire pipeline is bidirectional; forward and backward pass results are averaged.

We perform experiments on four traditional authorship verification datasets, a collection of machine learning papers mined from the web, and a large Amazon-Reviews dataset.

Experimental results surpass baseline and current state-of-the-art techniques, validating the proposed approach.

We propose and end-to-end model-building process that is universally applicable to a wide variety of authorship verification corpora and outperforms state-of-the-art with little to no modification or fine-tuning.

Robust Authorship Verification with Transfer Learning


We consider tackling a single-agent RL problem by distributing it to $n$ learners.

These learners, called advisors, endeavour to solve the problem from a different focus.

Their advice, taking the form of action values, is then communicated to an aggregator, which is in control of the system.

We show that the local planning method for the advisors is critical and that none of the ones found in the literature is flawless: the \textit{egocentric} planning overestimates values of states where the other advisors disagree, and the \textit{agnostic} planning is inefficient around danger zones.

We introduce a novel approach called \textit{empathic} and discuss its theoretical aspects.

We empirically examine and validate our theoretical findings on a fruit collection task.

Multi-Advisor Reinforcement Learning


We present a hybrid framework that leverages the trade-off between temporal and frequency precision in audio representations to improve the performance of speech enhancement task.

We first show that conventional approaches using specific representations such as raw-audio and spectrograms are each effective at targeting different types of noise.

By integrating both approaches, our model can learn multi-scale and multi-domain features, effectively removing noise existing on different regions on the time-frequency space in a complementary way.

Experimental results show that the proposed hybrid model yields better performance and robustness than using each model individually.

A hybrid model utilizing both raw-audio and spectrogram information for speech enhancement tasks.

Multi-Domain Processing via Hybrid Denoising Networks for Speech Enhancement


Consistently checking the statistical significance of experimental results is the first mandatory step towards reproducible science.

This paper presents a hitchhiker's guide to rigorous comparisons of reinforcement learning algorithms.

After introducing the concepts of statistical testing, we review the relevant statistical tests and compare them empirically in terms of false positive rate and statistical power as a function of the sample size (number of seeds) and effect size.

We further investigate the robustness of these tests to violations of the most common hypotheses (normal distributions, same distributions, equal variances).

Beside simulations, we compare empirical distributions obtained by running Soft-Actor Critic and Twin-Delayed Deep Deterministic Policy Gradient on Half-Cheetah.

We conclude by providing guidelines and code to perform rigorous comparisons of RL algorithm performances.

This paper compares statistical tests for RL comparisons (false positive, statistical power), checks robustness to assumptions using simulated distributions and empirical distributions (SAC, TD3), provides guidelines for RL students and researchers.

A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms


In the Information Bottleneck (IB), when tuning the relative strength between compression and prediction terms, how do the two terms behave, and what's their relationship with the dataset and the learned representation?

In this paper, we set out to answer these questions by studying multiple phase transitions in the IB objective: IB_β[p(z|x)] = I(X; Z) − βI(Y; Z) defined on the encoding distribution p(z|x) for input X, target Y and representation Z, where sudden jumps of dI(Y; Z)/dβ and prediction accuracy are observed with increasing β.

We introduce a definition for IB phase transitions as a qualitative change of the IB loss landscape, and show that the transitions correspond to the onset of learning new classes.

Using second-order calculus of variations, we derive a formula that provides a practical condition for IB phase transitions, and draw its connection with the Fisher information matrix for parameterized models.

We provide two perspectives to understand the formula, revealing that each IB phase transition is finding a component of maximum (nonlinear) correlation between X and Y orthogonal to the learned representation, in close analogy with canonical-correlation analysis (CCA) in linear settings.

Based on the theory, we present an algorithm for discovering phase transition points.

Finally, we verify that our theory and algorithm accurately predict phase transitions in categorical datasets, predict the onset of learning new classes and class difficulty in MNIST, and predict prominent phase transitions in CIFAR10.

We give a theoretical analysis of the Information Bottleneck objective to understand and predict observed phase transitions in the prediction vs. compression tradeoff.

Phase Transitions for the Information Bottleneck in Representation Learning


We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks.

The local adaptation of activation function is achieved by introducing scalable hyper-parameters in each layer (layer-wise) and for every neuron separately (neuron-wise), and then optimizing it using the stochastic gradient descent algorithm.

Introduction of neuron-wise activation function acts like a vector activation function as opposed to the traditional scalar activation function given by fixed, global and layer-wise activations.

In order to further increase the training speed, an activation slope based slope recovery term is added in the loss function, which further accelerate convergence, thereby reducing the training cost.

For numerical experiments, a nonlinear discontinuous function is approximated using a deep neural network with layer-wise and neuron-wise locally adaptive activation functions with and without the slope recovery term and compared with its global counterpart.

Moreover, solution of the nonlinear Burgers equation, which exhibits steep gradients, is also obtained using the proposed methods.

On the theoretical side, we prove that in the proposed method the gradient descent algorithms are not attracted to sub-optimal critical points or local minima under practical conditions on the initialization and learning rate.

Furthermore, the proposed adaptive activation functions with the slope recovery are shown to accelerate the training process in standard deep learning benchmarks using CIFAR-10, CIFAR-100, SVHN, MNIST, KMNIST, Fashion-MNIST, and Semeion data sets with and without data augmentation.

Proposing locally adaptive activation functions in deep and physics-informed neural networks for faster convergence

Locally adaptive activation functions with slope recovery term for deep and physics-informed neural networks

deep physics-informed neural network
locally adaptive activation function


Learning in Gaussian Process models occurs through the adaptation of hyperparameters of the mean and the covariance function.

The classical approach entails maximizing the marginal likelihood yielding fixed point estimates (an approach called Type II maximum likelihood or ML-II).

An alternative learning procedure is to infer the posterior over hyperparameters in a hierarchical specification of GPs we call Fully Bayesian Gaussian Process Regression (GPR)

. This work considers two approximations to the intractable hyperparameter posterior

, 1) Hamiltonian Monte Carlo (HMC) yielding a sampling based approximation and

2) Variational Inference (VI) where the posterior over hyperparameters is approximated by a factorized Gaussian (mean-field) or a full rank Gaussian accounting for correlations between hyperparameters.

We analyse the predictive performance for fully Bayesian GPR on a range of benchmark data sets.

Analysis of Bayesian Hyperparameter Inference in Gaussian Process Regression

Approximate Inference for Fully Bayesian Gaussian Process Regression

gaussian process regression
inference
bayesian


We consider the problem of using variational latent-variable models for data compression.

For such models to produce a compressed binary sequence, which is the universal data representation in a digital world, the latent representation needs to be subjected to entropy coding.

Range coding as an entropy coding technique is optimal, but it can fail catastrophically if the computation of the prior differs even slightly between the sending and the receiving side.

Unfortunately, this is a common scenario when floating point math is used and the sender and receiver operate on different hardware or software platforms, as numerical round-off is often platform dependent.

We propose using integer networks as a universal solution to this problem, and demonstrate that they enable reliable cross-platform encoding and decoding of images using variational models.

We train variational models with quantized networks for computational determinism. This enables using them for cross-platform data compression.

Integer Networks for Data Compression with Latent-Variable Models


Neural networks powered with external memory simulate computer behaviors.

These models, which use the memory to store data for a neural controller, can learn algorithms and other complex tasks.

In this paper, we introduce a new memory to store weights for the controller, analogous to the stored-program memory in modern computer architectures.

The proposed model, dubbed Neural Stored-program Memory, augments current memory-augmented neural networks, creating differentiable machines that can switch programs through time, adapt to variable contexts and thus fully resemble the Universal Turing Machine.

A wide range of experiments demonstrate that the resulting machines not only excel in classical algorithmic problems, but also have potential for compositional, continual, few-shot learning and question-answering tasks.

A neural simulation of Universal Turing Machine

Neural Stored-program Memory


It is common practice to decay the learning rate.

Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training.

This procedure is successful for stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum, and Adam.

It reaches equivalent test accuracies after the same number of training epochs, but with fewer parameter updates, leading to greater parallelism and shorter training times.

We can further reduce the number of parameter updates by increasing the learning rate $\epsilon$ and scaling the batch size $B \propto \epsilon$.

Finally, one can increase the momentum coefficient $m$ and scale $B \propto 1/(1-m)$, although this tends to slightly reduce the test accuracy.

Crucially, our techniques allow us to repurpose existing training schedules for large batch training with no hyper-parameter tuning.

We train ResNet-50 on ImageNet to 76.1% validation accuracy in under 30 minutes.

Decaying the learning rate and increasing the batch size during training are equivalent.

Don't Decay the Learning Rate, Increase the Batch Size


Traditional models for question answering optimize using cross entropy loss, which encourages exact answers at the cost of penalizing nearby or overlapping answers that are sometimes equally accurate.

We propose a mixed objective that combines cross entropy loss with self-critical policy learning, using rewards derived from word overlap to solve the misalignment between evaluation metric and optimization objective.

In addition to the mixed objective, we introduce a deep residual coattention encoder that is inspired by recent work in deep self-attention and residual networks.

Our proposals improve model performance across question types and input lengths, especially for long questions that requires the ability to capture long-term dependencies.

On the Stanford Question Answering Dataset, our model achieves state of the art results with 75.1% exact match accuracy and 83.1% F1, while the ensemble obtains 78.9% exact match accuracy and 86.0% F1.

We introduce the DCN+ with deep residual coattention and mixed-objective RL, which achieves state of the art performance on the Stanford Question Answering Dataset.

DCN+: Mixed Objective And Deep Residual Coattention for Question Answering

deep residual coattention
question answer
dcn+


Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years.

It could be time to take a step back and analyze the good and bad aspects in the field of NAS.

A variety of algorithms search architectures under different search space.

These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization.

This raises a comparability problem when comparing the performance of various NAS algorithms.

NAS-Bench-101 has shown success to alleviate this problem.

In this work, we propose an extension to NAS-Bench-101: NAS-Bench-201 with a different search space, results on multiple datasets, and more diagnostic information.

NAS-Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms.

The design of our search space is inspired by the one used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph.

Each edge here is associated with an operation selected from a predefined operation set.

For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-201 includes all possible architectures generated by 4 nodes and 5 associated operation options, which results in 15,625 neural cell candidates in total.

The training log using the same setup and the performance for each architecture candidate are provided for three datasets.

This allows researchers to avoid unnecessary repetitive training for selected architecture and focus solely on the search algorithm itself.

The training time saved for every architecture also largely improves the efficiency of most NAS algorithms and presents a more computational cost friendly NAS community for a broader range of researchers.

We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms.

In further support of the proposed NAS-Bench-102, we have analyzed it from many aspects and benchmarked 10 recent NAS algorithms, which verify its applicability.

A NAS benchmark applicable to almost any NAS algorithms.

NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search


Generative Adversarial Networks (GANs) are one of the most popular tools for learning complex high dimensional distributions.

However, generalization properties of GANs have not been well understood.

In this paper, we analyze the generalization of GANs in practical settings.

We show that discriminators trained on discrete datasets with the original GAN loss have poor generalization capability and do not approximate the theoretically optimal discriminator.

We propose a zero-centered gradient penalty for improving the generalization of the discriminator by pushing it toward the optimal discriminator.

The penalty guarantees the generalization and convergence of GANs.

Experiments on synthetic and large scale datasets verify our theoretical analysis.

We propose a zero-centered gradient penalty for improving generalization and stability of GANs

Improving Generalization and Stability of Generative Adversarial Networks


Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion.

While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant amount of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of ``tricks".

The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, and neural architectures.

In this work we take a sober view of the current state of GANs from a practical perspective.

We reproduce the current state of the art and go beyond fairly exploring the GAN landscape.

We discuss common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub.

A sober view on the current state of GANs from a practical perspective

The GAN Landscape: Losses, Architectures, Regularization, and Normalization


As deep reinforcement learning (RL) is applied to more tasks, there is a need to visualize and understand the behavior of learned agents.

Saliency maps explain agent behavior by highlighting the features of the input state that are most relevant for the agent in taking an action.

Existing perturbation-based approaches to compute saliency often highlight regions of the input that are not relevant to the action taken by the agent.

Our approach generates more focused saliency maps by balancing two aspects (specificity and relevance) that capture different desiderata of saliency.

The first captures the impact of perturbation on the relative expected reward of the action to be explained.

The second downweights irrelevant features that alter the relative expected rewards of actions other than the action to be explained.

We compare our approach with existing approaches on agents trained to play board games (Chess and Go) and Atari games (Breakout, Pong and Space Invaders).

We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that our approach generates saliency maps that are more interpretable for humans than existing approaches.

We propose a model-agnostic approach to explain the behaviour of black-box deep RL agents, trained to play Atari and board games, by highlighting relevant features of an input state.

Explain Your Move: Understanding Agent Actions Using Focused Feature Saliency


To understand the inner work of deep neural networks and provide possible theoretical explanations, we study the deep representations through the untrained, random weight CNN-DCN architecture.

As a convolutional AutoEncoder, CNN indicates the portion of a convolutional neural network from the input to an intermediate convolutional layer, and DCN indicates the corresponding deconvolutional portion.

As compared with DCN training for pre-trained CNN, training the DCN for random-weight CNN converges more quickly and yields higher quality image reconstruction.

Then, what happens for the overall random CNN-DCN?

We gain intriguing results that the image can be reconstructed with good quality.

To gain more insight on the intermediate random representation, we investigate the impact of network width versus depth, number of random channels, and size of random kernels on the reconstruction quality, and provide theoretical justifications on empirical observations.

We further provide a fast style transfer application using the random weight CNN-DCN architecture to show the potential of our observation.

We investigate the deep representation of untrained, random weight CNN-DCN architectures, and show their image reconstruction quality and possible applications.

Randomness in Deconvolutional Networks for Visual Representation


The current trade-off between depth and computational cost makes it difficult to adopt deep neural networks for many industrial applications, especially when computing power is limited.

Here, we are inspired by the idea that, while deeper embeddings are needed to discriminate difficult samples, a large number of samples can be well discriminated via much shallower embeddings.

In this study, we introduce the concept of decision gates (d-gate), modules trained to decide whether a sample needs to be projected into a deeper embedding or if an early prediction can be made at the d-gate, thus enabling the computation of dynamic representations at different depths.

The proposed d-gate modules can be integrated with any deep neural network and reduces the average computational cost of the deep neural networks while maintaining modeling accuracy.

Experimental results show that leveraging the proposed d-gate modules led to a ~38% speed-up and ~39% FLOPS reduction on ResNet-101 and ~46% speed-up and $\sim$36\% FLOPS reduction on DenseNet-201 trained on the CIFAR10 dataset with only ~2% drop in accuracy.

This paper introduces a new dynamic feature representation approach to provide a more efficient way to do inference on deep neural networks.

Efficient Inference on Deep Neural Networks by Dynamic Representations and Decision Gates

inference deep neural network
efficient
representation
dynamic


This work provides an additional step in the theoretical understanding of neural networks.

We consider neural networks with one hidden layer and show that when learning symmetric functions, one can choose initial conditions so that standard SGD training efficiently produces generalization guarantees.

We empirically verify this and show that this does not hold when the initial conditions are chosen at random.

The proof of convergence investigates the interaction between the two layers of the network.

Our results highlight the importance of using symmetry in the design of neural networks.

When initialized properly, neural networks can learn the simple class of symmetric functions; when initialized randomly, they fail.

On Symmetry and Initialization for Neural Networks


Architecture search aims at automatically finding neural architectures that are competitive with architectures designed by human experts.

While recent approaches have achieved state-of-the-art predictive performance for image recognition, they are problematic under resource constraints for two reasons: (1) the neural architectures found are solely optimized for high predictive performance, without penalizing excessive resource consumption; (2)most architecture search methods require vast computational resources.

We address the first shortcoming by proposing LEMONADE, an evolutionary algorithm for multi-objective architecture search that allows approximating the Pareto-front of architectures under multiple objectives, such as predictive performance and number of parameters, in a single run of the method.

We address the second shortcoming by proposing a Lamarckian inheritance mechanism for LEMONADE which generates children networks that are warmstarted with the predictive performance of their trained parents.

This is accomplished by using (approximate) network morphism operators for generating children.

The combination of these two contributions allows finding models that are on par or even outperform different-sized NASNets, MobileNets, MobileNets V2 and Wide Residual Networks on CIFAR-10 and ImageNet64x64 within only one week on eight GPUs, which is about 20-40x less compute power than previous architecture search methods that yield state-of-the-art performance.

We propose a method for efficient Multi-Objective Neural Architecture Search based on Lamarckian inheritance and evolutionary algorithms.

Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution

efficient multi-objective neural architecture search


We address two challenges of probabilistic topic modelling in order to better estimate

the probability of a word in a given context, i.e., P(wordjcontext) : (1) No

Language Structure in Context: Probabilistic topic models ignore word order by

summarizing a given context as a “bag-of-word” and consequently the semantics

In this work, we incorporate language structure

by combining a neural autoregressive topic model (TM) with a LSTM based language

model (LSTM-LM) in a single probabilistic framework.

learns a vector-space representation of each word by accounting for word order

in local collocation patterns, while the TM simultaneously learns a latent representation

characteristics of language (e.g., syntax and semantics), while the TM discovers

the underlying thematic structure in a collection of documents.

paradigms of learning the meaning of word occurrences by combining

a topic model and a language model in a unified probabilistic framework, named

(2) Limited Context and/or Smaller training corpus of documents:

In settings with a small number of word occurrences (i.e., lack of context)

in short text or data sparsity in a corpus of few documents, the application of TMs

We address this challenge by incorporating external knowledge

into neural autoregressive topic models via a language modelling approach: we

use word embeddings as input of a LSTM-LM with the aim to improve the wordtopic

mapping on a smaller and/or short-text corpus.

We present novel neural autoregressive topic model variants coupled with neural

language models and embeddings priors that consistently outperform state-of-theart

generative topic models in terms of generalization (perplexity), interpretability

(topic coherence) and applicability (retrieval and classification) over 6 long-text

and 8 short-text datasets from diverse domains.

Unified neural model of topic and language modeling to introduce language structure  in topic models for contextualized topic vectors

textTOvec: DEEP CONTEXTUALIZED NEURAL AUTOREGRESSIVE TOPIC MODELS OF LANGUAGE WITH DISTRIBUTED COMPOSITIONAL PRIOR

topic model
language
neural
contextualize


The large memory requirements of deep neural networks strain the capabilities of many devices, limiting their deployment and adoption.

Model compression methods effectively reduce the memory requirements of these models, usually through applying transformations such as weight pruning or quantization.

In this paper, we present a novel scheme for lossy weight encoding which complements conventional compression techniques.

The encoding is based on the Bloomier filter, a probabilistic data structure that can save space at the cost of introducing random errors.

Leveraging the ability of neural networks to tolerate these imperfections and by re-training around the errors, the proposed technique, Weightless, can compress DNN weights by up to 496x; with the same model accuracy, this results in up to a 1.51x improvement over the state-of-the-art.

We propose a new way to compress neural networks using probabilistic data structures.

Weightless: Lossy Weight Encoding For Deep Neural Network Compression


Recent work has shown that contextualized word representations derived from neural machine translation (NMT) are a viable alternative to such from simple word predictions tasks.

This is because the internal understanding that needs to be built in order to be able to translate from one language to another is much more comprehensive.

Unfortunately, computational and memory limitations as of present prevent NMT models from using large word vocabularies, and thus alternatives such as subword units (BPE and morphological segmentations) and characters have been used.

Here we study the impact of using different kinds of units on the quality of the resulting representations when used to model syntax, semantics, and morphology.

We found that while representations derived from subwords are slightly better for modeling syntax, character-based representations are superior for modeling morphology and are also more robust to noisy input.

We study the impact of using different kinds of subword units on the quality of the resulting representations when used to model syntax, semantics, and morphology.

What Is in a Translation Unit?  Comparing Character and Subword Representations Beyond Translation


Few-shot learning is the process of learning novel classes using only a few examples and it remains a challenging task in machine learning.

Many sophisticated few-shot learning algorithms have been proposed based on the notion that networks can easily overfit to novel examples if they are simply fine-tuned using only a few examples.

In this study, we show that in the commonly used low-resolution mini-ImageNet dataset, the fine-tuning method achieves higher accuracy than common few-shot learning algorithms in the 1-shot task and nearly the same accuracy as that of the state-of-the-art algorithm in the 5-shot task.

We then evaluate our method with more practical tasks, namely the high-resolution single-domain and cross-domain tasks.

With both tasks, we show that our method achieves higher accuracy than common few-shot learning algorithms.

We further analyze the experimental results and show that:

1) the retraining process can be stabilized by employing a low learning rate,

2) using adaptive gradient optimizers during fine-tuning can increase test accuracy, and

3) test accuracy can be improved by updating the entire network when a large domain-shift exists between base and novel classes.

An empirical study that provides a novel perspective on few-shot learning, in which a fine-tuning method shows comparable accuracy to more complex state-of-the-art methods in several classification tasks.

Revisiting Fine-tuning for Few-shot Learning


We propose an approach to generate realistic and high-fidelity stock market data based on generative adversarial networks.

We model the order stream as a stochastic process with finite history dependence, and employ a conditional Wasserstein GAN to capture history dependence of orders in a stock market.

We test our approach with actual market and synthetic data on a number of different statistics, and find the generated data to be close to real data.

Generating Realistic Stock Market Order Streams


We present a novel black-box adversarial attack algorithm with state-of-the-art model evasion rates for query efficiency under $\ell_\infty$ and $\ell_2$ metrics.

It exploits a \textit{sign-based}, rather than magnitude-based, gradient estimation approach that shifts the gradient estimation from continuous to binary black-box optimization.

It adaptively constructs queries to estimate the gradient, one query relying upon the previous, rather than re-estimating the gradient each step with random query construction.

Its reliance on sign bits yields  a smaller memory footprint and it requires neither hyperparameter tuning or dimensionality reduction.

Further, its theoretical performance is guaranteed and it can characterize  adversarial subspaces better than white-box gradient-aligned subspaces.

On two public black-box attack challenges and a model robustly trained against transfer attacks, the algorithm's evasion rates surpass all submitted attacks.

For a suite of published models,  the algorithm is $3.8\times$ less failure-prone while spending $2.5\times$  fewer queries versus the best combination of state of art algorithms.

For example, it evades a standard MNIST model using just $12$ queries on average.

Similar performance is observed on a standard IMAGENET model with an average of $579$ queries.

We present a sign-based, rather than magnitude-based, gradient estimation approach that shifts gradient estimation from continuous to binary black-box optimization.

Sign Bits Are All You Need for Black-Box Attacks


Recurrent Neural Networks (RNNs) are widely used models for sequence data.

Just like for feedforward networks, it has become common to build "deep" RNNs, i.e., stack multiple recurrent layers to obtain higher-level abstractions of the data.

However, this works only for a handful of layers.

Unlike feedforward networks, stacking more than a few recurrent units (e.g., LSTM cells) usually hurts model performance, the reason being vanishing or exploding gradients during training.

We investigate the training of multi-layer RNNs and examine the magnitude of the gradients as they propagate through the network.

We show that, depending on the structure of the basic recurrent unit, the gradients are systematically attenuated or amplified, so that with an increasing depth they tend to vanish, respectively explode.

Based on our analysis we design a new type of gated cell that better preserves gradient magnitude, and therefore makes it possible to train deeper RNNs.

We experimentally validate our design with five different sequence modelling tasks on three different datasets.

The proposed stackable recurrent (STAR) cell allows for substantially deeper recurrent architectures, with improved performance.

We analyze the gradient propagation in deep RNNs and from our analysis, we propose a new multi-layer deep RNN.

Gating Revisited: Deep Multi-layer RNNs That Can Be Trained


Despite its empirical success, the theoretical underpinnings of the stability, convergence and acceleration properties of batch normalization (BN) remain elusive.

In this paper, we attack this problem from a modelling approach, where we perform thorough theoretical analysis on BN applied to simplified model: ordinary least squares (OLS).

We discover that gradient descent on OLS with BN has interesting properties, including a scaling law, convergence for arbitrary learning rates for the weights, asymptotic acceleration effects, as well as insensitivity to choice of learning rates.

We then demonstrate numerically that these findings are not specific to the OLS problem and hold qualitatively for more complex supervised learning problems.

This points to a new direction towards uncovering the mathematical principles that underlies batch normalization.

We mathematically analyze the effect of batch normalization on a simple model and obtain key new insights that applies to general supervised learning.

On the Convergence and Robustness of Batch Normalization


Solving tasks in Reinforcement Learning is no easy feat.

As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior.

While constraints may solve this issue, there is no closed form solution for general constraints.

In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one.

We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.

For complex constraints in which it is not easy to estimate the gradient, we use the discounted penalty as a guiding signal. We prove that under certain assumptions it converges to a feasible solution.

Reward Constrained Policy Optimization


The Handheld Virtual Panel (HVP) is the virtual panel attached to the non-dominant hand’s controller in virtual reality (VR).

The HVP is the go-to technique for enabling menus and toolboxes in VR devices.

In this paper, we investigate target acquisition performance for the HVP as a function of four factors: target width, target distance, the direction of approach with respect to gravity, and the angle of approach.

Our results show that all four factors have significant effects on user performance.

Based on the results, we propose guidelines towards the ergonomic and performant design of the HVP interfaces.

The paper investigates target acquisition for handheld virtual panels in VR and shows that target width, distance, direction of approach with respect to gravity, and angle of approach, all impact user performance.

Target Acquisition for Handheld Virtual Panels in VR

target acquisition handheld virtual panel vr


Deep neural networks have demonstrated unprecedented success in various knowledge management applications.

However, the networks created are often very complex, with large numbers of trainable edges which require extensive computational resources.

We note that many successful networks nevertheless often contain large numbers of redundant edges.

Moreover, many of these edges may have negligible contributions towards the overall network performance.

In this paper, we propose a novel iSparse framework and experimentally show, that we can sparsify the network, by 30-50%, without impacting the network performance.

iSparse leverages a novel edge significance score, E, to determine the importance of an edge with respect to the final network output.

Furthermore, iSparse can be applied both while training a model or on top of a pre-trained model, making it a  retraining-free approach - leading to a minimal computational overhead.

Comparisons of iSparse against PFEC, NISP, DropConnect, and Retraining-Free on benchmark datasets show that iSparse leads to effective network sparsifications.

iSparse eliminates irrelevant or insignificant network edges with minimal impact on network performance by determining edge importance w.r.t. the final network output.

iSparse: Output Informed Sparsification of Neural Networks


Language modeling tasks, in which words are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases.

Motivated by the observation that efforts to code

world knowledge into machine readable knowledge bases tend to be entity-centric,

we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the contexts in which those entities were mentioned.

We show that large scale training of neural models allows us to learn extremely high fidelity entity typing information, which we demonstrate with few-shot reconstruction of Wikipedia categories.

Our learning approach is powerful enough

to encode specialized topics such as Giro d’Italia cyclists.

We learn entity representations that can reconstruct Wikipedia categories with just a few exemplars.

Learning Entity Representations for Few-Shot Reconstruction of Wikipedia Categories

learn entity representation
wikipedia category


Variational autoencoders (VAEs) have been successful at learning a low-dimensional manifold from high-dimensional data with complex dependencies.

At their core, they consist of a powerful Bayesian probabilistic inference model, to capture the salient features of the data.

In training, they exploit the power of variational inference, by optimizing a lower bound on the model evidence.

The latent representation and the performance of VAEs are heavily influenced by the type of bound used as a cost function.

Significant research work has been carried out into the development of tighter bounds than the original ELBO, to more accurately approximate the true log-likelihood.

By leveraging the q-deformed logarithm in the traditional lower bounds, ELBO and IWAE, and the upper bound CUBO, we bring contributions to this direction of research.

In this proof-of-concept study, we explore different ways of creating these q-deformed bounds that are tighter than the classical ones and we show improvements in the performance of such VAEs on the binarized MNIST dataset.

Using the q-deformed logarithm, we derive tighter bounds than IWAE, to train variational autoencoders.

Variational autoencoders trained with q-deformed lower bounds

train
variational autoencoder
q-deformed
bound


A belief persists long in machine learning that enlargement of margins over training data accounts for the resistance of models to overfitting by increasing the robustness.

Yet Breiman shows a dilemma (Breiman, 1999) that a uniform improvement on margin distribution \emph{does not} necessarily reduces generalization error.

In this paper, we revisit Breiman's dilemma in deep neural networks with recently proposed normalized margins using Lipschitz constant bound by spectral norm products.

With both simplified theory and extensive experiments, Breiman's dilemma is shown to rely on dynamics of normalized margin distributions, that reflects the trade-off between model expression power and data complexity.

When the complexity of data is comparable to the model expression power in the sense that training and test data share similar phase transitions in normalized margin dynamics, two efficient ways are derived via classic margin-based generalization bounds to successfully predict the trend of generalization error.

On the other hand, over-expressed models that exhibit uniform improvements on training normalized margins may lose such a prediction power and fail to prevent the overfitting.

Bregman's dilemma is shown in deep learning that improvement of margins of over-parameterized models may result in overfitting, and dynamics of normalized margin distributions are proposed to predict generalization error and identify such a dilemma.

ON BREIMAN’S DILEMMA IN NEURAL NETWORKS: SUCCESS AND FAILURE OF NORMALIZED MARGINS


It has been an open research challenge for developing an end-to-end multi-domain task-oriented dialogue system, in which a human can converse with the dialogue agent to complete tasks in more than one domain.

First, tracking belief states of multi-domain dialogues is difficult as the dialogue agent must obtain the complete belief states from all relevant domains, each of which can have shared slots common among domains as well as unique slots specifically for the domain only.

Second, the dialogue agent must also process various types of information, including contextual information from dialogue context, decoded dialogue states of current dialogue turn, and queried results from a knowledge base, to semantically shape context-aware and task-specific responses to human.

To address these challenges, we propose an end-to-end neural architecture for task-oriented dialogues in multiple domains.

We propose a novel Multi-level Neural Belief Tracker which tracks the dialogue belief states by learning signals at both slot and domain level independently.

The representations are combined in a Late Fusion approach to form joint feature vectors of (domain, slot) pairs.

Following recent work in end-to-end dialogue systems, we incorporate the belief tracker with generation components to address end-to-end dialogue tasks.

We achieve state-of-the-art performance on the MultiWOZ2.1 benchmark with 50.91% joint goal accuracy and competitive measures in task-completion and response generation.

We proposed an end-to-end dialogue system with a novel multi-level dialogue state tracker and achieved consistent performance on MultiWOZ2.1 in state tracking, task completion, and response generation performance.

End-to-End Multi-Domain Task-Oriented Dialogue Systems with Multi-level Neural Belief Tracker

dialogue system
tracker
multi-level
end-to-end


Score matching provides an effective approach to learning flexible unnormalized models, but its scalability is limited by the need to evaluate a second-order derivative.

In this paper,we connect a general family of learning objectives including score matching to Wassersteingradient flows.

This connection enables us to design a scalable approximation to theseobjectives, with a form similar to single-step contrastive divergence.

We present applications in training implicit variational and Wasserstein auto-encoders with manifold-valued priors.

We present a scalable approximation to a wide range of EBM objectives, and applications in implicit VAEs and WAEs

A Wasserstein Minimum Velocity Approach to Learning Unnormalized Models


This paper develops variational continual learning (VCL), a simple but general framework for continual learning that fuses online variational inference (VI) and recent advances in Monte Carlo VI for neural networks.

The framework can successfully train both deep discriminative models and deep generative models in complex continual learning settings where existing tasks evolve over time and entirely new tasks emerge.

Experimental results show that VCL outperforms state-of-the-art continual learning methods on a variety of tasks, avoiding catastrophic forgetting in a fully automatic way.

This paper develops a principled method for continual learning in deep models.

Variational Continual Learning


In partially observable (PO) environments, deep reinforcement learning (RL) agents often suffer from unsatisfactory performance, since two problems need to be tackled together: how to extract information from the raw observations to solve the task, and how to improve the policy.

In this study, we propose an RL algorithm for solving PO tasks.

Our method comprises two parts: a variational recurrent model (VRM) for modeling the environment, and an RL controller that has access to both the environment and the VRM.

The proposed algorithm was tested in two types of PO robotic control tasks, those in which either coordinates or velocities were not observable and those that require long-term memorization.

Our experiments show that the proposed algorithm achieved better data efficiency and/or learned more optimal policy than other alternative approaches in tasks in which unobserved states cannot be inferred from raw observations in a simple manner.

A deep RL algorithm for solving POMDPs by auto-encoding the underlying states using a variational recurrent model

Variational Recurrent Models for Solving Partially Observable Control Tasks


This paper formalises the problem of online algorithm selection in the context of Reinforcement Learning (RL).

The setup is as follows: given an episodic task and a finite number of off-policy RL algorithms, a meta-algorithm has to decide which RL algorithm is in control during the next episode so as to maximize the expected return.

The article presents a novel meta-algorithm, called Epochal Stochastic Bandit Algorithm Selection (ESBAS).

Its principle is to freeze the policy updates at each epoch, and to leave a rebooted stochastic bandit in charge of the algorithm selection.

Under some assumptions, a thorough theoretical analysis demonstrates its near-optimality considering the structural sampling budget limitations.

ESBAS is first empirically evaluated on a dialogue task where it is shown to outperform each individual algorithm in most configurations.

ESBAS is then adapted to a true online setting where algorithms update their policies after each transition, which we call SSBAS.

SSBAS is evaluated on a fruit collection task where it is shown to adapt the stepsize parameter more efficiently than the classical hyperbolic decay, and on an Atari game, where it improves the performance by a wide margin.

This paper formalises the problem of online algorithm selection in the context of Reinforcement Learning.

Reinforcement Learning Algorithm Selection

reinforcement learning
algorithm selection


In this paper, we present a new generative model for learning latent embeddings.

Compared to the classical generative process, where each observed data point is generated from an individual latent variable, our approach assumes a global latent variable to generate the whole set of observed data points.

We then propose a learning objective that is derived as an approximation to a lower bound to the data log likelihood, leading to our algorithm, WiSE-ALE.

Compared to the standard ELBO objective, where the variational posterior for each data point is encouraged to match the prior distribution, the WiSE-ALE objective matches the averaged posterior, over all samples, with the prior, allowing the sample-wise posterior distributions to have a wider range of acceptable embedding mean and variance and leading to better reconstruction quality in the auto-encoding process.

Through various examples and comparison to other state-of-the-art VAE models, we demonstrate that WiSE-ALE has excellent information embedding properties, whilst still retaining the ability to learn a smooth, compact representation.

We propose a new latent variable model to learn latent embeddings for some high-dimensional data.

WiSE-ALE: Wide Sample Estimator for Aggregate Latent Embedding


We improve the robustness of deep neural nets  to adversarial attacks by using an interpolating function as the output activation.

This data-dependent activation function remarkably improves both classification accuracy and stability to adversarial perturbations.

Together with the total variation minimization of adversarial images and augmented training, under the strongest attack, we achieve up to 20.6%, 50.7%, and 68.7% accuracy improvement w.r.t.

the fast gradient sign method, iterative fast gradient sign method, and Carlini-WagnerL2attacks, respectively.

Our defense strategy is additive to many of the existing methods.

We give an intuitive explanation of our defense strategy via analyzing the geometry of the feature space.

For reproducibility, the code will be available on GitHub.

We proposal strategies for adversarial defense based on data dependent activation function, total variation minimization, and training data augmentation

Adversarial Defense Via Data Dependent Activation Function and  Total Variation Minimization

datum dependent activation function
total variation minimization
adversarial defense


This work presents a scalable solution to continuous visual speech recognition.

To achieve this, we constructed the largest existing visual speech recognition dataset, consisting of pairs of text and video clips of faces speaking (3,886 hours of video).

In tandem, we designed and trained an integrated lipreading system, consisting of a video processing pipeline that maps raw video to stable videos of lips and sequences of phonemes, a scalable deep neural network that maps the lip videos to sequences of phoneme distributions, and a production-level speech decoder that outputs sequences of words.

The proposed system achieves a word error rate (WER) of 40.9% as measured on a held-out set.

In comparison, professional lipreaders achieve either 86.4% or 92.9% WER on the same dataset when having access to additional types of contextual information.

Our approach significantly improves on previous lipreading approaches, including variants of LipNet and of Watch, Attend, and Spell (WAS), which are only capable of 89.8% and 76.8% WER respectively.

Large-Scale Visual Speech Recognition


Previous work (Bowman et al., 2015; Yang et al., 2017) has found difficulty developing generative models based on variational autoencoders (VAEs) for text.

To address the problem of the decoder ignoring information from the encoder (posterior collapse), these previous models weaken the capacity of the decoder to force the model to use information from latent variables.

However, this strategy is not ideal as it degrades the quality of generated text and increases hyper-parameters.

In this paper, we propose a new VAE for text utilizing a multimodal prior distribution, a modified encoder, and multi-task learning.

We show our model can generate well-conditioned sentences without weakening the capacity of the decoder.

Also, the multimodal prior distribution improves the interpretability of acquired representations.

We propose a model of variational autoencoders for text modeling without weakening the decoder, which improves the quality of text generation and interpretability of acquired representations.

Variational Autoencoders for Text Modeling without Weakening the Decoder

variational autoencoder text model without weaken decoder


Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model.

Recent reports (Han et al., 2015; Narang et al., 2017) prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction in model size.

This hints at the possibility that the baseline models in these experiments are perhaps severely over-parameterized at the outset and a viable alternative for model compression might be to simply reduce the number of hidden units while maintaining the model's dense connection structure, exposing a similar trade-off in model size and accuracy.

We investigate these two distinct paths for model compression within the context of energy-efficient inference in resource-constrained environments and propose a new gradual pruning technique that is simple and straightforward to apply across a variety of models/datasets with minimal tuning and can be seamlessly incorporated within the training process.

We compare the accuracy of large, but pruned models (large-sparse) and their smaller, but dense (small-dense) counterparts with identical memory footprint.

Across a broad range of neural network architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find large-sparse models to consistently outperform small-dense models and achieve up to 10x reduction in number of non-zero parameters with minimal loss in accuracy.

We demonstrate that large, but pruned models (large-sparse) outperform their smaller, but dense (small-dense) counterparts with identical memory footprint.

To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression