Recent deep multi-task learning (MTL) has been witnessed its success in alleviating data scarcity of some task by utilizing domain-specific knowledge from related tasks.

Nonetheless, several major issues of deep MTL, including the effectiveness of sharing mechanisms, the efficiency of model complexity and the flexibility of network architectures, still remain largely unaddressed.

To this end, we propose a novel generalized latent-subspace based knowledge sharing mechanism for linking task-specific models, namely tensor ring multi-task learning (TRMTL).

TRMTL has a highly compact representation, and it is very effective in transferring task-invariant knowledge while being super flexible in learning task-specific features, successfully mitigating the dilemma of both negative-transfer in lower layers and under-transfer in higher layers.

Under our TRMTL, it is feasible for each task to have heterogenous input data dimensionality or distinct feature sizes at different hidden layers.

Experiments on a variety of datasets demonstrate our model is capable of significantly improving each single task’s performance, particularly favourable in scenarios where some of the tasks have insufficient data.

a deep multi-task learning model adapting tensor ring representation

A variant of tensor ring formulation for multi-task learning by sharing some of the TT cores for learning "common task" while learning individual TT cores for each separate task

TENSOR RING NETS ADAPTED DEEP MULTI-TASK LEARNING

deep multi-task learning
tensor ring
adapt


Neural Processes (NPs) (Garnelo et al., 2018) approach regression by learning to map a context set of observed input-output pairs to a distribution over regression functions.

Each function models the distribution of the output given an input, conditioned on the context.

NPs have the benefit of fitting observed data efficiently with linear complexity in the number of context input-output pairs, and can learn a wide family of conditional distributions; they learn predictive distributions conditioned on context sets of arbitrary size.

Nonetheless, we show that NPs suffer a fundamental drawback of underfitting, giving inaccurate predictions at the inputs of the observed data they condition on.

We address this issue by incorporating attention into NPs, allowing each input location to attend to the relevant context points for the prediction.

We show that this greatly improves the accuracy of predictions, results in noticeably faster training, and expands the range of functions that can be modelled.

A model for regression that learns conditional distributions of a stochastic process, by incorporating attention into Neural Processes.

Proposes to resolve the issue of underfitting in the neural process method by adding an attention mechanism to the deterministic path.

An extension to the framework of Neural Processes that adds an attention-based conditioning mechanism, allowing the model to better capture dependencies in the conditioning set.

The authors extend neural processes by incorporating self-attention for enriching the features of the context points and cross-attention for producing a query-specific representation. They resolve the underfitting problem of NPs and show ANPs to converge better and faster than NPs.

Attentive Neural Processes

neural process
resolve
model
add
incorporate
attention
mechanism
underfitte
well


Deconvolutional layers have been widely used in a variety of deep

models for up-sampling, including encoder-decoder networks for

semantic segmentation and deep generative models for unsupervised

One of the key limitations of deconvolutional operations

is that they result in the so-called checkerboard problem.

caused by the fact that no direct relationship exists among adjacent

propose the pixel deconvolutional layer (PixelDCL) to establish

direct relationships among adjacent pixels on the up-sampled feature

Our method is based on a fresh interpretation of the regular

replace any deconvolutional layer in a plug-and-play manner without

compromising the fully trainable capabilities of original models.

The proposed PixelDCL may result in slight decrease in efficiency,

but this can be overcome by an implementation trick.

results on semantic segmentation demonstrate that PixelDCL can

consider spatial features such as edges and shapes and yields more

accurate segmentation outputs than deconvolutional layers.

in image generation tasks, our PixelDCL can largely overcome the

checkerboard problem suffered by regular deconvolution operations.

Solve checkerboard problem in Deconvolutional layer by building dependencies between pixels

This work proposes pixel deconvolutional layers for convolutional neural networks as a way to alleviate the checkerboard effect.

A novel technique to generalize deconvolution operations used in standard CNN architectures, which proposes doing sequential prediction of adjacent pixel features, resulting in more spatially smooth outputs for deconvolution layers.

Pixel Deconvolutional Networks

deconvolutional layer
pixel
checkerboard
network


In this paper, the preparation of a neural network for pruning and few-bit quantization is formulated as a variational inference problem.

To this end, a quantizing prior that leads to a multi-modal, sparse posterior distribution over weights, is introduced and a differentiable Kullback-Leibler divergence approximation for this prior is derived.

After training with Variational Network Quantization, weights can be replaced by deterministic quantization values with small to negligible loss of task accuracy (including pruning by setting weights to 0).

The method does not require fine-tuning after quantization.

Results are shown for ternary quantization on LeNet-5 (MNIST) and DenseNet (CIFAR-10).

We quantize and prune neural network weights using variational Bayesian inference with a multi-modal, sparsity inducing prior.

Proposes to use a mixture of continuous spike propto 1/abs as prior for a Bayesian neural network and demonstrates the good performance with relatively sparsified convnets for minist and cifar-10.

This paper presents a variational Bayesian approach for quantising neural network weights to ternary values post-training in a principled way.

Variational Network Quantization

neural network weight
variational bayesian
prior


Deep neural networks (DNNs) although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices.

Extensive research work have been conducted on DNN model compression or pruning.

However, most of the previous work took heuristic approaches.

This work proposes a progressive weight pruning approach based on ADMM (Alternating Direction Method of Multipliers), a powerful technique to deal with non-convex optimization problems with potentially combinatorial constraints.

Motivated by dynamic programming, the proposed method reaches extremely high pruning rate by using partial prunings with moderate pruning rates.

Therefore, it resolves the accuracy degradation and long convergence time problems when pursuing extremely high pruning ratios.

It achieves up to 34× pruning rate for ImageNet dataset and 167× pruning rate for MNIST dataset, significantly higher than those reached by the literature work.

Under the same number of epochs, the proposed method also achieves faster convergence and higher compression rates.

The codes and pruned DNN models are released in the anonymous link bit.ly/2zxdlss.

We implement a DNN weight pruning approach that achieves the highest pruning rates.

This paper focuses on weight pruning for neural network compression, achiving 30x compression rate for AlexNet and VGG for ImageNet.

A progressive pruning technique which imposes structural sparsity constraint on the weight parameter and rewrites the optimization as an ADMM framework, achieving higher accurancy than projected gradient descent.

Progressive Weight Pruning Of Deep Neural Networks Using ADMM

achieve high
weight pruning
neural network
rate
admm
progressive


In this paper, we present a new deep learning architecture for addressing the problem of supervised learning with sparse and irregularly sampled multivariate time series.

The architecture is based on the use of a semi-parametric interpolation network followed by the application of a prediction network.

The interpolation network allows for information to be shared across multiple dimensions of a multivariate time series during the interpolation stage, while any standard deep learning model can be used for the prediction network.

This work is motivated by the analysis of physiological time series data in electronic health records, which are sparse, irregularly sampled, and multivariate.

We investigate the performance of this architecture on both classification and regression tasks, showing that our approach outperforms a range of baseline and recently proposed models.

This paper presents a new deep learning architecture for addressing the problem of supervised learning with sparse and irregularly sampled multivariate time series.

Proposes a framework for making predictions on sparse, irregularly sampled time-series data using an interpolation module that models the missing values in using smooth interpolation, non-smooth interpolation, and intensity.

Solves the problem of supervised learning with sparse and irregularly sampled multivariate time series using a semi-parametric interpolation network followed by a prediction network.

Interpolation-Prediction Networks for Irregularly Sampled Time Series

problem supervised learning sparse irregularly sample


We introduce an analytic distance function for moderately sized point sets of known cardinality that is shown to have very desirable properties, both as a loss function as well as a regularizer for machine learning applications.

We compare our novel construction to other point set distance functions and show proof of concept experiments for training neural networks end-to-end on point set prediction tasks such as object detection.

Permutation-invariant loss function for point set prediction.

Proposes a new loss for points registration (aligning two point sets) with preferable permutation invariant property.

This paper introduces a novel distance function between point sets, applies two other permutation distances in an end-to-end object detection task, and shows that in two dimensions all local minima of the holographic loss are global minima.

Proposes permutation invariant loss functions which depend on the distance of sets.

Holographic and other Point Set Distances for Machine Learning

function point set
holographic
distance
permutation invariant
loss
two


We introduce a hierarchical model for efficient placement of computational graphs onto hardware devices, especially in heterogeneous environments with a mixture of CPUs, GPUs, and other computational devices.

Our method learns to assign graph operations to groups and to allocate those groups to available devices.

The grouping and device allocations are learned jointly.

The proposed method is trained with policy gradient and requires no human intervention.

computer vision and natural language models show that our algorithm can find optimized, non-trivial placements for TensorFlow computational graphs with over 80,000 operations.

In addition, our approach outperforms placements by human

experts as well as a previous state-of-the-art placement method based on deep reinforcement learning.

Our method achieves runtime reductions of up to 60.6% per training step when applied to models such as Neural Machine Translation.

We introduce a hierarchical model for efficient, end-to-end placement of computational graphs onto hardware devices.

Proposes to jointly learn groups of operators to colocate and to place learned groups on devices to distribute operations for deep learning via reinforcement learning.

The authors purpose a fully connect network to replace the co-location step in an auto-placement method proposed to accelerate a TensorFlow model's runtime.

Proposes a device placement algorithm to place operations of tensorflow on devices.

A Hierarchical Model for Device Placement

hierarchical model
device
place
operation
tensorflow
placement


Motion is an important signal for agents in dynamic environments, but learning to represent motion from unlabeled video is a difficult and underconstrained problem.

We propose a model of motion based on elementary group properties of transformations and use it to train a representation of image motion.

While most methods of estimating motion are based on pixel-level constraints, we use these group properties to constrain the abstract representation of motion itself.

We demonstrate that a deep neural network trained using this method captures motion in both synthetic 2D sequences and real-world sequences of vehicle motion, without requiring any labels.

Networks trained to respect these constraints implicitly identify the image characteristic of motion in different sequence types.

In the context of vehicle motion, this method extracts information useful for localization, tracking, and odometry.

Our results demonstrate that this representation is useful for learning motion in the general setting where explicit labels are difficult to obtain.

We propose of method of using group properties to learn a representation of motion without labels and demonstrate the use of this method for representing 2D and 3D motion.

Proposes to learn the rigid motion group from a latent representation of image sequences without the need for explicit labels and experimentally demonstrates method on sequences of MINST digits and the KITTI dataset.

This paper proposes an approach for learning video motion features in an unsupervised manner, using constraints to optimize the neural network to produce features that can be used to regress odometry.

Understanding image motion with group representations

learn group method motion label representation demonstrate without image


This paper introduces the concept of continuous convolution to neural networks and deep learning applications in general.

Rather than directly using discretized information, input data is first projected into a high-dimensional Reproducing Kernel Hilbert Space (RKHS), where it can be modeled as a continuous function using a series of kernel bases.

We then proceed to derive a closed-form solution to the continuous convolution operation between two arbitrary functions operating in different RKHS.

Within this framework, convolutional filters also take the form of continuous functions, and the training procedure involves learning the RKHS to which each of these filters is projected, alongside their weight parameters.

This results in much more expressive filters, that do not require spatial discretization and benefit from properties such as adaptive support and non-stationarity.

Experiments on image classification are performed, using classical datasets, with results indicating that the proposed continuous convolutional neural network is able to achieve competitive accuracy rates with far fewer parameters and a faster convergence rate.

This paper proposes a novel convolutional layer that operates in a continuous Reproducing Kernel Hilbert Space.

Projecting examples into an RK Hilbert space and performing convolution and filtering into that space.

This paper formulates a variant of convolutional neural networks which models both activations and filters as continuous functions composed from kernel bases

Continuous Convolutional Neural Networks for Image Classification

convolutional neural network
hilbert space
continuous
kernel


Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes.

Some recent studies suggest a more important role of image textures.

We here put these conflicting hypotheses to a quantitative test by evaluating CNNs and human observers on images with a texture-shape cue conflict.

We show that ImageNet-trained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence and reveals fundamentally different classification strategies.

We then demonstrate that the same standard architecture (ResNet-50) that learns a texture-based representation on ImageNet is able to learn a shape-based representation instead when trained on 'Stylized-ImageNet', a stylized version of ImageNet.

This provides a much better fit for human behavioural performance in our well-controlled psychophysical lab setting (nine experiments totalling 48,560 psychophysical trials across 97 observers) and comes with a number of unexpected emergent benefits such as improved object detection performance and previously unseen robustness towards a wide range of image distortions, highlighting advantages of a shape-based representation.

ImageNet-trained CNNs are biased towards object texture (instead of shape like humans). Overcoming this major difference between human and machine vision yields improved detection performance and previously unseen robustness to image distortions.

Using image stylizaton to augment training data for ImageNet-trained CNNs to make resulting networks appear more aligned with human judgements

This paper studies CNNs like AlexNet, VGG, GoogleNet, and ResNet50, shows these models are biased towards texture when trained on ImageNet, and proposes a new ImageNet dataset.

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

imagenet-traine cnn bias towards
robustness
texture
image
human
shape
like
improve


In this work, we exploited different strategies to provide prior knowledge to commonly used generative modeling approaches aiming to obtain speaker-dependent low dimensional representations from short-duration segments of speech data, making use of available information of speaker identities.

Namely, convolutional variational autoencoders are employed, and statistics of its learned posterior distribution are used as low dimensional representations of fixed length short-duration utterances.

In order to enforce speaker dependency in the latent layer, we introduced a variation of the commonly used prior within the variational autoencoders framework, i.e. the model is simultaneously trained for reconstruction of inputs along with a discriminative task performed on top of latent layers outputs.

The effectiveness of both triplet loss minimization and speaker recognition are evaluated as implicit priors on the challenging cross-language NIST SRE 2016 setting and compared against fully supervised and unsupervised baselines.

We evaluate the effectiveness of having auxiliary discriminative tasks performed on top of statistics of the posterior distribution learned by variational autoencoders to enforce speaker dependency.

Propose an autoencoder model to learn a representation for speaker verification using short-duration analysis windows.

A modified version of the variational autoencoder model that tackles the speaker recognition problem in the context of short-duration segments

Variational Autoencoders with implicit priors for short-duration text-independent speaker verification

variational autoencoder
learn
speaker
model
short-duration
verification


The importance-weighted autoencoder (IWAE) approach of Burda et al. defines a sequence of increasingly tighter bounds on the marginal likelihood of latent variable models.

Recently, Cremer et al. reinterpreted the IWAE bounds as ordinary variational evidence lower bounds (ELBO) applied to increasingly accurate variational distributions.

In this work, we provide yet another perspective on the IWAE bounds.

We interpret each IWAE bound as a biased estimator of the true marginal likelihood where for the bound defined on $K$ samples we show the bias to be of order O(1/K).

In our theoretical analysis of the IWAE objective we derive asymptotic bias and variance expressions.

Based on this analysis we develop jackknife variational inference (JVI),

a family of bias-reduced estimators reducing the bias to $O(K^{-(m+1)})$ for any given m < K while retaining computational efficiency.

Finally, we demonstrate that JVI leads to improved evidence estimates in variational autoencoders.

We also report first results on applying JVI to learning variational autoencoders.

Our implementation is available at https://github.com/Microsoft/jackknife-variational-inference

Variational inference is biased, let's debias it.

Introduces jackknife variational inference, a method for debiasing Monte Carlo objectives such as the importance weighted auto-encoder.

The authors analyze the bias and variance of the IWAE bound and derive a jacknife approach to estimate moments as a way to debias IWAE for finite importance weighted samples.

Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference

jackknife variational inference
bias
debias
importance weight
debiase


In this paper, we consider the problem of autonomous lane changing for self driving vehicles in a multi-lane, multi-agent setting.

We present a framework that demonstrates a more structured and data efficient alternative to end-to-end complete policy learning on problems where the high-level policy is hard to formulate using traditional optimization or rule based methods but well designed low-level controllers are available.

Our framework uses deep reinforcement learning solely to obtain a high-level policy for tactical decision making, while still maintaining a tight integration with the low-level controller, thus getting the best of both worlds.

We accomplish this with Q-masking, a technique with which we are able to incorporate prior knowledge, constraints, and information from a low-level controller, directly in to the learning process thereby simplifying the reward function and making learning faster and data efficient.

We provide preliminary results in a simulator and show our approach to be more efficient than a greedy baseline, and more successful and safer than human driving.

A framework that provides a policy for autonomous lane changing by learning to make high-level tactical decisions with deep reinforcement learning, and maintaining a tight integration with a low-level controller to take low-level actions.

Considers the problem of autonomous lane changing for self-driving cars in multi-lane multi-agent slot car setting, proposes a new learning strategy Q-masking - coupling a defined low level controller with a high level tactical decision making policy.

This paper proposes a deep Q-learning approach to the problem of lane change using "Q-masking," which reduces the action space according to contraints or prior knowledge.

Authors propose a method which uses a Q-learning-based high-level policy that is combined with a contextual mask derived from safety-contraints and low-level controllers, which disable certain actions from being selectable at certain states.

Tactical Decision Making for Lane Changing with Deep Reinforcement Learning

low-level controller
autonomous lane change
deep reinforcement learning
policy
problem
tactical decision make
action
high-level
q-masking


Despite the recent successes in robotic locomotion control, the design of robot relies heavily on human engineering.

Automatic robot design has been a long studied subject, but the recent progress has been slowed due to the large combinatorial search space and the difficulty in evaluating the found candidates.

To address the two challenges, we formulate automatic robot design as a graph search problem and perform evolution search in graph space.

We propose Neural Graph Evolution (NGE), which performs selection on current candidates and evolves new ones iteratively.

Different from previous approaches, NGE uses graph neural networks to parameterize the control policies, which reduces evaluation cost on new candidates with the help of skill transfer from previously evaluated designs.

In addition, NGE applies Graph Mutation with Uncertainty (GM-UC) by incorporating model uncertainty, which reduces the search space by balancing exploration and exploitation.

We show that NGE significantly outperforms previous methods by an order of magnitude.

As shown in experiments, NGE is the first algorithm that can automatically discover kinematically preferred robotic graph structures, such as a fish with two symmetrical flat side-fins and a tail, or a cheetah with athletic front and back legs.

Instead of using thousands of cores for weeks, NGE efficiently solves searching problem within a day on a single 64 CPU-core Amazon EC2

Automatic robotic design search with graph neural networks

Proposes an approach for automatic robot design based on Neural graph evolution. The experiments demonstrate that optimizing both controller and hardware is better than optimizing just the controller.

The authors propose a scheme based on a graph representation of the robot structure, and a graph-neural-network as controllers to optimize robot structures, combined with their controllers.

Neural Graph Evolution: Towards Efficient Automatic Robot Design

controller
neural graph evolution
automatic robot design
base
optimize


Deep learning on graphs has become a popular research topic with many applications.

However, past work has concentrated on learning graph embedding tasks only, which is in contrast with advances in generative models for images and text.

Is it possible to transfer this progress to the domain of graphs?

We propose to sidestep hurdles associated with linearization of such discrete structures by having a decoder output a probabilistic fully-connected graph of a predefined maximum size directly at once.

Our method is formulated as a variational autoencoder.

We evaluate on the challenging task of conditional molecule generation.

We demonstate an autoencoder for graphs.

Learning to generate graphs using deep learning methods in "one shot", directly outputting node and edge existence probabilities, and node attribute vectors.

A variational auto encoder to generate graphs

GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders


Long Short-Term Memory (LSTM) is one of the most widely used recurrent structures in sequence modeling.

Its goal is to use gates to control the information flow (e.g., whether to skip some information/transformation or not) in the recurrent computations, although its practical implementation based on soft gates only partially achieves this goal and is easy to overfit.

In this paper, we propose a new way for LSTM training, which pushes the values of the gates towards 0 or 1.

By doing so, we can (1) better control the information flow: the gates are mostly open or closed, instead of in a middle state; and (2) avoid overfitting to certain extent: the gates operate at their flat regions, which is shown to correspond to better generalization ability.

However, learning towards discrete values of the gates is generally difficult.

To tackle this challenge, we leverage the recently developed Gumbel-Softmax trick from the field of variational methods, and make the model trainable with standard backpropagation.

Experimental results on language modeling and machine translation show that (1) the values of the gates generated by our method are more reasonable and intuitively interpretable, and (2) our proposed method generalizes better and achieves better accuracy on test sets in all tasks.

Moreover, the learnt models are not sensitive to low-precision approximation and low-rank approximation of the gate parameters due to the flat loss surface.

We propose a new algorithm for LSTM training by learning towards binary-valued gates which we shown has many nice properties.

Propose a new "gate" function for LSTM to enable the values of the gates towards 0 or 1.

The paper aims to push LSTM gates to be binary by employing the recent Gumbel-Softmax trick to obtain end-to-end trainable categorical distribution.

Towards Binary-Valued Gates for Robust LSTM Training

towards binary-valued gate
new
lstm
training


We present a personalized recommender system using neural network for recommending

products, such as eBooks, audio-books, Mobile Apps, Video and Music.

It produces recommendations based on customer’s implicit feedback history such

Our key contribution is to formulate recommendation

problem as a model that encodes historical behavior to predict the future

behavior using soft data split, combining predictor and auto-encoder models.

introduce convolutional layer for learning the importance (time decay) of the purchases

depending on their purchase date and demonstrate that the shape of the time

decay function can be well approximated by a parametrical function.

offline experimental results showing that neural networks with two hidden layers

can capture seasonality changes, and at the same time outperform other modeling

techniques, including our recommender in production.

demonstrate that our model can be scaled to all digital categories, and we observe

significant improvements in an online A/B test.

to the neural network model and describe our production pipeline.

we open-sourced our deep learning library which supports multi-gpu model parallel

This is an important feature in building neural network based recommenders

with large dimensionality of input and output data.

Improving recommendations using time sensitive modeling with neural networks in multiple product categories on a retail website

The paper proposes a new neural network based method for recommendation.

The authors describe a procedure of building their production recommender system from scratch and integrate time decay of purchases into the learning framework.

THE EFFECTIVENESS OF A TWO-LAYER NEURAL NETWORK FOR RECOMMENDATIONS


Deep Learning (DL) algorithms based on Generative Adversarial Network (GAN) have demonstrated great potentials in computer vision tasks such as image restoration.

Despite the rapid development of image restoration algorithms using DL and GANs, image restoration for specific scenarios, such as medical image enhancement and super-resolved identity recognition, are still facing challenges.

How to ensure visually realistic restoration while avoiding hallucination or mode- collapse?

How to make sure the visually plausible results do not contain hallucinated features jeopardizing downstream tasks such as pathology identification and subject identification?

Here we propose to resolve these challenges by coupling the GAN based image restoration framework with another task-specific network.

With medical imaging restoration as an example, the proposed model conducts additional pathology recognition/classification task to ensure the preservation of detailed structures that are important to this task.

Validated on multiple medical datasets, we demonstrate the proposed method leads to improved deep learning based image restoration while preserving the detailed structure and diagnostic features.

Additionally, the trained task network show potentials to achieve super-human level performance in identifying pathology and diagnosis.

Further validation on super-resolved identity recognition tasks also show that the proposed method can be generalized for diverse image restoration tasks.

Couple the GAN based image restoration framework with another task-specific network to generate realistic image while preserving task-specific features.

A novel method of Task-GAN of image coupling that couples GAN and a task-specific network, which alleviates to avoid hallucination or mode collapse.

The authors propose to augment GAN-based image restoration with another task-specific branch, such as classification tasks, for further improvement.

Task-GAN for Improved GAN based Image Restoration

gan base image restoration
another task-specific
couple
network
task-gan


Unsupervised anomaly detection on multi- or high-dimensional data is of great importance in both fundamental machine learning research and industrial applications, for which density estimation lies at the core.

Although previous approaches based on dimensionality reduction followed by density estimation have made fruitful progress, they mainly suffer from decoupled model learning with inconsistent optimization goals and incapability of preserving essential information in the low-dimensional space.

In this paper, we present a Deep Autoencoding Gaussian Mixture Model (DAGMM) for unsupervised anomaly detection.

Our model utilizes a deep autoencoder to generate a low-dimensional representation and reconstruction error for each input data point, which is further fed into a Gaussian Mixture Model (GMM).

Instead of using decoupled two-stage training and the standard Expectation-Maximization (EM) algorithm, DAGMM jointly optimizes the parameters of the deep autoencoder and the mixture model simultaneously in an end-to-end fashion, leveraging a separate estimation network to facilitate the parameter learning of the mixture model.

The joint optimization, which well balances autoencoding reconstruction, density estimation of latent representation, and regularization, helps the autoencoder escape from less attractive local optima and further reduce reconstruction errors, avoiding the need of pre-training.

Experimental results on several public benchmark datasets show that, DAGMM significantly outperforms state-of-the-art anomaly detection techniques, and achieves up to 14% improvement based on the standard F1 score.

An end-to-end trained deep neural network that leverages Gaussian Mixture Modeling to perform density estimation and unsupervised anomaly detection in a low-dimensional space learned by deep autoencoder.

The paper presents a joint deep learning framework for dimension reduction-clustering, leads to competitive anomaly detection.

A new technique for anomaly detection where the dimension reduction and density estimation steps are jointly optimized.

Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection

gaussian mixture model
anomaly detection
density estimation
dimension
deep


Generalization from limited examples, usually studied under the umbrella of meta-learning, equips learning techniques with the ability to adapt quickly in dynamical environments and proves to be an essential aspect of lifelong learning.

In this paper, we introduce the Projective Subspace Networks (PSN), a deep learning paradigm that learns non-linear embeddings from limited supervision.

In contrast to previous studies, the embedding in PSN deems samples of a given class to form an affine subspace.

We will show that such modeling leads to robust solutions, yielding competitive results on supervised and semi-supervised few-shot classification.

Moreover, our PSN approach has the ability of end-to-end learning.

In contrast to previous works, our projective subspace can be thought of as a richer representation capturing higher-order information datapoints for modeling new concepts.

We proposed Projective Subspace Networks for few-shot and semi-supervised few-shot learning

This paper proposes a new embedding-based approach for the problem of few-shot learning and an extension to this model to the semi-supervised few-shot learning setting.

New method for fully and semi-supervised few-shot classification based on learning a general embedding and then learning a subspace of it for each class

Projective Subspace Networks For Few-Shot Learning

projective subspace network few-shot
semi-supervised
learning
new


This paper investigates whether learning contingency-awareness and controllable aspects of an environment can lead to better exploration in reinforcement learning.

To investigate this question, we consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE).

In this study, we develop an attentive dynamics model (ADM) that discovers controllable elements of the observations, which are often associated with the location of the character in Atari games.

The ADM is trained in a self-supervised fashion to predict the actions taken by the agent.

The learned contingency information is used as a part of the state representation for exploration purposes.

We demonstrate that combining actor-critic algorithm with count-based exploration using our representation achieves impressive results on a set of notoriously challenging Atari games due to sparse rewards.

For example, we report a state-of-the-art score of >11,000 points on Montezuma's Revenge without using expert demonstrations, explicit high-level information (e.g., RAM states), or supervisory data.

Our experiments confirm that contingency-awareness is indeed an extremely powerful concept for tackling exploration problems in reinforcement learning and opens up interesting research questions for further investigations.

We investigate contingency-awareness and controllable aspects in exploration and achieve state-of-the-art performance on Montezuma's Revenge without expert demonstrations.

This paper investigates the problem of extracting a meaningful state representation to help with exploration when confronted with a sparse reward task by identifying controllable (learned) features of the state

This paper proposes the novel idea of using contingency awareness to aid exploration in sparse-reward reinforcement learning tasks, obtaining state of the art results.

Contingency-Aware Exploration in Reinforcement Learning

exploration
state
investigate
controllable
task
reinforcement learning


Disentangling factors of variation has always been a challenging problem in representation learning.

Existing algorithms suffer from many limitations, such as unpredictable disentangling factors, bad quality of generated images from encodings, lack of identity information, etc.

In this paper, we proposed a supervised algorithm called DNA-GAN trying to disentangle different attributes of images.

The latent representations of images are DNA-like, in which each individual piece represents an independent factor of variation.

By annihilating the recessive piece and swapping a certain piece of two latent representations, we obtain another two different representations which could be decoded into images.

In order to obtain realistic images and also disentangled representations, we introduced the discriminator for adversarial training.

Experiments on Multi-PIE and CelebA datasets demonstrate the effectiveness of our method and the advantage of overcoming limitations existing in other methods.

We proposed a supervised algorithm, DNA-GAN, to disentangle multiple attributes of images.

This paper investigates the problem of attribute-conditioned image generation using generative adversarial networks, and proposes to generate images from attribute and latent code as high-level representation.

This paper proposed a new method to disentangle different attributes of images using a novel DNA structure GAN

DNA-GAN: Learning Disentangled Representations from Multi-Attribute Images

attribute image
disentangle
dna-gan
representation


Representations learnt through deep neural networks tend to be highly informative, but opaque in terms of what information they learn to encode.

We introduce an approach to probabilistic modelling that learns to represent data with two separate deep representations: an invariant representation that encodes the information of the class from which the data belongs, and an equivariant representation that encodes the symmetry transformation defining the particular data point within the class manifold (equivariant in the sense that the representation varies naturally with symmetry transformations).

This approach to representation learning is conceptually transparent, easy to implement, and in-principle generally applicable to any data comprised of discrete classes of continuous distributions (e.g. objects in images, topics in language, individuals in behavioural data).

We demonstrate qualitatively compelling representation learning and competitive quantitative performance, in both supervised and semi-supervised settings, versus comparable modelling approaches in the literature with little fine tuning.

This paper presents a novel latent-variable generative modelling technique that enables the representation of global information into one latent variable and local information into another latent variable.

The paper presents a VAE that uses labels to separate the learned representation into an invariant and a covariant part.

Invariant-equivariant representation learning for multi-class data


Convolutional neural networks (CNNs) have been successfully applied to many recognition and learning tasks using a universal recipe;  training a deep model on a very large dataset of supervised examples.

However, this approach is rather restrictive in practice since collecting a large set of labeled images is very expensive.

One way to ease this problem is coming up with smart ways for choosing images to be labelled from a  very large collection (i.e. active learning).

Our empirical study suggests that many of the active learning heuristics in the literature are not effective when applied to CNNs when applied in batch setting.

Inspired by these limitations, we define the problem of active learning as core-set selection, i.e. choosing set of points such that a model learned over the selected subset is competitive for the remaining data points.

We further present a theoretical result characterizing the performance of any selected subset using the geometry of the datapoints.

As an active learning algorithm, we choose the subset which is expected to yield best result according to our characterization.

Our experiments show that the proposed method significantly outperforms existing approaches in image classification experiments by a large margin.

We approach to the problem of active learning as a core-set selection problem and show that this approach is especially useful in the batch active learning setting which is crucial when training CNNs.

The authors provide an algorithm-agnostic active learning algorithm for multi-class classification

The paper proposes a batch mode active learning algorithm for CNN as a core-set problem which outperforms random sampling and uncertainty sampling.

Studies active learning for convolutional neural networks and formulates the active learning problem as core-set selection and presents a novel strategy

Active Learning for Convolutional Neural Networks: A Core-Set Approach

active learning convolutional neural network


Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP).

This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because EVGP prevents important gradient components from being back-propagated adequately over a large number of steps.

We introduce a simple stochastic algorithm (\textit{h}-detach) that is specific to LSTM optimization and targeted towards addressing this problem.

Specifically, we show that when the LSTM weights are large, the gradient components through the linear path (cell state) in the LSTM computational graph get suppressed.

Based on the hypothesis that these components carry information about long term dependencies (which we show empirically), their suppression can prevent LSTMs from capturing them.

Our algorithm\footnote{Our code is available at https://github.com/bhargav104/h-detach.

} prevents gradients flowing through this path from getting suppressed, thus allowing the LSTM to capture such dependencies better.

We show significant improvements over vanilla LSTM gradient based training in terms of convergence speed, robustness to seed and learning rate, and generalization using our modification of LSTM gradient on various benchmark datasets.

A simple algorithm to improve optimization and handling of long term dependencies in LSTM

The paper introduces a simple stochastic algorithm called h-detach that is specific to LSTM optimization and targeted towards addressing this problem.

Proposes a simple modification to the training process of the LSTM to facilitate gradient propogation along cell states, or the "linear temporal path"

h-detach: Modifying the LSTM Gradient Towards Better Optimization

optimization simple lstm h-detach algorithm towards gradient


Convolutional Neural Networks (CNNs) significantly improve the state-of-the-art for many applications, especially in computer vision.

However, CNNs still suffer from a tendency to confidently classify out-distribution samples from unknown classes into pre-defined known classes.

Further, they are also vulnerable to adversarial examples.

We are relating these two issues through the tendency of CNNs to over-generalize for areas of the input space not covered well by the training set.

We show that a CNN augmented with an extra output class can act as a simple yet effective end-to-end model for controlling over-generalization.

As an appropriate training set for the extra class, we introduce two resources that are computationally efficient to obtain: a representative natural out-distribution set and interpolated in-distribution samples.

To help select a representative natural out-distribution set among available ones, we propose a simple measurement to assess an out-distribution set's fitness.

We also demonstrate that training such an augmented CNN with representative out-distribution natural datasets and some interpolated samples allows it to better handle a wide range of unseen out-distribution samples and black-box adversarial examples without training it on any adversaries.

Finally, we show that generation of white-box adversarial attacks using our proposed augmented CNN can become harder, as the attack algorithms have to get around the rejection regions when generating actual adversaries.

Properly training CNNs with dustbin class increase their robustness to adversarial attacks and their capacity to deal with out-distribution samples.

This paper proposes adding an additional label for detecting OOD samples and adversarial examples in CNN models.

The paper proposes an additional class that incorporates natural out-distribution images and interpolated images for adversarial and out-distribution samples in CNNs

Controlling Over-generalization and its Effect on Adversarial Examples Detection and Generation

out-distribution sample
adversarial example
cnn
additional
class


Modern deep artificial neural networks have achieved impressive results through models with very large capacity---compared to the number of training examples---that control overfitting with the help of different forms of regularization.

Regularization can be implicit, as is the case of stochastic gradient descent or parameter sharing in convolutional layers, or explicit.

Most common explicit regularization techniques, such as dropout and weight decay, reduce the effective capacity of the model and typically require the use of deeper and wider architectures to compensate for the reduced capacity.

Although these techniques have been proven successful in terms of results, they seem to waste capacity.

In contrast, data augmentation techniques reduce the generalization error by increasing the number of training examples and without reducing the effective capacity.

In this paper we systematically analyze the effect of data augmentation on some popular architectures and conclude that data augmentation alone---without any other explicit regularization techniques---can achieve the same performance or higher as regularized models, especially when training with fewer examples.

In a deep convolutional neural network trained with sufficient level of data augmentation, optimized by SGD, explicit regularizers (weight decay and dropout) might not provide any additional generalization improvement.

This paper proposes data augmentation as an alternative to commonly used regularisation techniques, and shows that for a few reference models/tasks that the same generalization performance can be achived using only data augmentation.

This paper presents a systematic study of data augmentation in image classification with deep neural networks, suggesting that data augmentation can replicit some common regularizers like weight decay and dropout.

Data augmentation instead of explicit regularization

weight decay dropout
datum augmentation
neural network
explicit
deep
generalization
regularizer


Text editing on mobile devices can be a tedious process.

To perform various editing operations, a user must repeatedly move his or her fingers between the text input area and the keyboard, making multiple round trips and breaking the flow of typing.

In this work, we present Gedit, a system of on-keyboard gestures for convenient mobile text editing.

Our design includes a ring gesture and flicks for cursor control, bezel gestures for mode switching, and four gesture shortcuts for copy, paste, cut, and undo.

Variations of our gestures exist for one and two hands.

We conducted an experiment to compare Gedit with the de facto touch+widget based editing interactions.

Our results showed that Gedit’s gestures were easy to learn, 24% and 17% faster than the de facto interactions for one- and two-handed use, respectively, and preferred by participants.

Reports the design and evaluation of the Gedit interaction techniques.

Presents a new set of touch gestures to perform seamless transition between text entry and text editing in mobile devices

Gedit: Keyboard Gestures for Mobile Text Editing

gedit
text editing
gesture
present
mobile


Deep learning achieves remarkable generalization capability with overwhelming number of model parameters.

Theoretical understanding of deep learning generalization receives recent attention yet remains not fully explored.

This paper attempts to provide an alternative understanding from the perspective of maximum entropy.

We first derive two feature conditions that softmax regression strictly apply maximum entropy principle.

DNN is then regarded as approximating the feature conditions with multilayer feature learning, and proved to be a recursive solution towards maximum entropy principle.

The connection between DNN and maximum entropy well explains why typical designs such as shortcut and regularization improves model generalization, and provides instructions for future model development.

We prove that DNN is a recursively approximated solution to the maximum entropy principle.

Presents a derivation which links a DNN to recursive application of maximum entropy model fitting.

The paper aims to provide a view of deep learning from the perspective of maximum entropy principle.

Understanding Deep Learning Generalization by Maximum Entropy


As people learn to navigate the world, autonomic nervous system (e.g., ``fight or flight) responses provide intrinsic feedback about the potential consequence of action choices (e.g., becoming nervous when close to a cliff edge or driving fast around a bend.) Physiological changes are correlated with these biological preparations to protect one-self from danger.

We present a novel approach to reinforcement learning that leverages a task-independent intrinsic reward function trained on peripheral pulse measurements that are correlated with human autonomic nervous system responses.

Our hypothesis is that such reward functions can circumvent the challenges associated with sparse and skewed rewards in reinforcement learning settings and can help improve sample efficiency.

We test this in a simulated driving environment and show that it can increase the speed of learning and reduce the number of collisions during the learning stage.

Proposes a reinforcement learning framework based on human emotional reaction in the context of autonomous driving.

The authors propose to use signals, such as basic autonomic visceral responses that influence decision-making, within the RL framework by augmenting RL reward functions with a model learned directly from human nervous system responses.

Proposes to use physiological signals to improve performance of reinforcement learning algorithms and build an intrinsic reward function that is less sparse by measuring heart pulse amplitude

Visceral Machines: Risk-Aversion in  Reinforcement Learning with Intrinsic Physiological Rewards

nervous system response
reinforcement learning
reward function
human
signal
intrinsic
pulse
autonomic
framework
visceral
physiological


Deep convolutional neural networks (CNNs) are known to be robust against label noise on extensive datasets.

However, at the same time, CNNs are capable of memorizing all labels even if they are random, which means they can memorize corrupted labels.

Much of researches focusing on such memorization uses class-independent label noise to simulate label corruption, but this setting is simple and unrealistic.

In this paper, we investigate the behavior of CNNs under class-dependently simulated label noise, which is generated based on the conceptual distance between classes of a large dataset (i.e., ImageNet-1k).

Contrary to previous knowledge, we reveal CNNs are more robust to such class-dependent label noise than class-independent label noise.

We also demonstrate the networks under class-dependent noise situations learn similar representation to the no noise situation, compared to class-independent noise situations.

Are CNNs robust or fragile to label noise? Practically, robust.

The authors challenge the CNNs robustness to label noise using ImageNet 1k tree of WordNet.

An analysis of convolutional neural network model performance when class dependent and class independent noise is introduced

Demonstrates that CNNs are more robust to class-relevant label noise and argues that real-world noise should be class-relevant

Investigating CNNs' Learning Representation under label noise


Efficient audio synthesis is an inherently difficult machine learning task, as human perception is sensitive to both global structure and fine-scale waveform coherence.

Autoregressive models, such as WaveNet, model local structure at the expense of global latent structure and slow iterative sampling, while Generative Adversarial Networks (GANs), have global latent conditioning and efficient parallel sampling, but struggle to generate locally-coherent audio waveforms.

Herein, we demonstrate that GANs can in fact generate high-fidelity and locally-coherent audio by modeling log magnitudes and instantaneous frequencies  with sufficient frequency resolution in the spectral domain.

Through extensive empirical investigations on the NSynth dataset, we demonstrate that GANs are able to outperform strong WaveNet baselines on automated and human evaluation metrics, and efficiently generate audio several orders of magnitude faster than their autoregressive counterparts.

Proposes an approach that uses GAN framework to generate audio through modeling log magnitudes and instantaneous frequencies with sufficient frequency resolution in the spectral domain.

A strategy to generate audio samples from noise with GANs, with changes to the architecture and representation necessary to generate convincing audio that contains an interpretable latent code.

Presents a simple idea for better representing audio data so that convolutional models such as generative adversarial networks can be applied

GANSynth: Adversarial Neural Audio Synthesis

generate audio
gan
model
synthesis
adversarial


In this work we propose a novel approach for learning graph representation of the data using gradients obtained via backpropagation.

Next we build a neural network architecture compatible with our optimization approach and motivated by graph filtering in the vertex domain.

We demonstrate that the learned graph has richer structure than often used nearest neighbors graphs constructed based on features similarity.

Our experiments demonstrate that we can improve prediction quality for several convolution on graphs architectures, while others appeared to be insensitive to the input graph.

Graph Optimization with signal filtering in the vertex domain.

The paper investigates learning adjacency matrix of a graph with sparsely connected undirected graph with nonnegative edge weights uses a projected sub-gradient descent algorithm.

Develops a novel scheme for backpropogating on the adjacency matrix of a neural network graph

UPS: optimizing Undirected Positive Sparse graph for neural graph filtering

adjacency matrix
graph
neural
filtering
undirected


The use of AR in an industrial context could help for the training of new operators.

To be able to use an AR guidance system, we need a tool to quickly create a 3D representation of the assembly line and of its AR annotations.

This tool should be very easy to use by an operator who is not an AR or VR specialist: typically the manager of the assembly line.

This is why we proposed WAAT, a 3D authoring tool allowing user to quickly create 3D models of the workstations, and also test the AR guidance placement.

WAAT makes on-site authoring possible, which should really help to have an accurate 3D representation of the assembly line.

The verification of AR guidance should also be very useful to make sure everything is visible and doesn't interfere with technical tasks.

In addition to these features, our future work will be directed in the deployment of WAAT into a real boiler assembly line to assess the usability of this solution.

This paper describe a 3D authoring tool for providing AR in assembly lines of industry 4.0

The paper addresses how AR authoring tools support training of assembly line systems and proposes an approach

An AR guidance system for industrial assembly lines that allows for on-site authoring of AR content.

Presents a system that allows factory workers to be trained more efficiently using augmented reality system.

WAAT: a Workstation AR Authoring Tool for Industry 4.0

ar author tool
assembly line
industry 4.0
system
allow


Generative adversarial network (GAN) is one of the best known unsupervised learning techniques these days due to its superior ability to learn data distributions.

In spite of its great success in applications, GAN is known to be notoriously hard to train.

The tremendous amount of time it takes to run the training algorithm and its sensitivity to hyper-parameter tuning have been haunting researchers in this area.

To resolve these issues, we need to first understand how GANs work.

Herein, we take a step toward this direction by examining the dynamics of GANs.

We relate a large class of GANs including the Wasserstein GANs to max-min optimization problems with the coupling term being linear over the discriminator.

By developing new primal-dual optimization tools, we show that, with a proper stepsize choice, the widely used first-order iterative algorithm in training GANs would in fact converge to a stationary solution with a sublinear rate.

The same framework also applies to multi-task learning and distributional robust learning problems.

We verify our analysis on numerical examples with both synthetic and real data sets.

We hope our analysis shed light on future studies on the theoretical properties of relevant machine learning problems.

We show that, with a proper stepsize choice, the widely used first-order iterative algorithm in training GANs would in fact converge to a stationary solution with a sublinear rate.

This paper uses GANs and multi-task learning to provide a convergence guarantee for primal-dual algorithms on certain min-max problems.

Analyses the learning dynamics of GANs by formulating the problem as a primal-dual optimisation problem by assuming a limited class of models

Understand the dynamics of GANs via Primal-Dual Optimization

dynamic gan
algorithm
learning
primal-dual
problem


Social dilemmas, where mutual cooperation can lead to high payoffs but participants face incentives to cheat, are ubiquitous in multi-agent interaction.

We wish to construct agents that cooperate with pure cooperators, avoid exploitation by pure defectors, and incentivize cooperation from the rest.

However, often the actions taken by a partner are (partially) unobserved or the consequences of individual actions are hard to predict.

We show that in a large class of games good strategies can be constructed by conditioning one's behavior solely on outcomes (ie. one's past rewards).

We call this consequentialist conditional cooperation.

We show how to construct such strategies using deep reinforcement learning techniques and demonstrate, both analytically and experimentally, that they are effective in social dilemmas beyond simple matrix games.

We also show the limitations of relying purely on consequences and discuss the need for understanding both the consequences of and the intentions behind an action.

We show how to use deep RL to construct agents that can solve social dilemmas beyond matrix games.

Learning to play two-player general-sum games with state with imperfect information

Specifies a trigger strategy (CCC) and corresponding algorithm, demonstrating convergence to efficient outcomes in social dilemmas without need for agents to observe each other's actions.

Consequentialist conditional cooperation in social dilemmas with imperfect information

agent
imperfect information
social dilemma
game


In distributed training, the communication cost due to the transmission of gradients

or the parameters of the deep model is a major bottleneck in scaling up the number

To address this issue, we propose dithered quantization for

the transmission of the stochastic gradients and show that training with Dithered

Quantized Stochastic Gradients (DQSG) is similar to the training with unquantized

SGs perturbed by an independent bounded uniform noise, in contrast to the other

quantization methods where the perturbation depends on the gradients and hence,

algorithms using DQSG and the trade off between the number of quantization

Next, we observe that there is a correlation among the

SGs computed by workers that can be utilized to further reduce the communication

Hence, we develop a simple yet effective

quantization scheme, nested dithered quantized SG (NDQSG), that can reduce the

communication significantly without requiring the workers communicating extra

We prove that although NDQSG requires significantly

less bits, it can achieve the same quantization variance bound as DQSG.

simulation results confirm the effectiveness of training using DQSG and NDQSG

in reducing the communication bits or the convergence time compared to the

existing methods without sacrificing the accuracy of the trained model.

The paper proposes and analyzes two quantization schemes for communicating Stochastic Gradients in distributed learning which would reduce communication costs compare to the state of the art while maintaining the same accuracy.

The authors propose applying dithered quantization to the stochastic gradients computed through the training process, which improves quantization error and achieves superior results compared to baselines, and propose a nested scheme to reduce communication cost.

Authors establish a connection between communication reduction in distributed optimization and dithered quantization and develops two new distributed training algorithms where communication overhead is significantly reduced.

Nested Dithered Quantization for Communication Reduction in Distributed Training

communication reduction distribute
dithered quantization
stochastic gradient
compare
scheme
reduce
two
cost
author
training
nest


Deep neural networks have been shown to perform well in many classical machine learning problems, especially in image classification tasks.

However, researchers have found that neural networks can be easily fooled, and they are surprisingly sensitive to small perturbations imperceptible to humans.

Carefully crafted input images (adversarial examples) can force a well-trained neural network to provide arbitrary outputs.

Including adversarial examples during training is a popular defense mechanism against adversarial attacks.

In this paper we propose a new defensive mechanism under the generative adversarial network~(GAN) framework.

We model the adversarial noise using a generative network, trained jointly with a classification discriminative network as a minimax game.

We show empirically that our adversarial network approach works well against black box attacks, with performance on par with state-of-art methods such as ensemble adversarial training and adversarial training with projected gradient descent.

Jointly train an adversarial noise generating network with a classification network to provide better robustness to adversarial attacks.

A GAN solution for deep models of classification, faced to white and black box attacks, that produces robust models.

The paper proposes a defensive mechanism against adversarial attacks using GANs with generated perturbations used as adversarial examples and a discriminator used to distinguish between them

A Direct Approach to Robust Deep Learning Using Adversarial Networks

adversarial attack
classification
gan
generate
network
deep
robust


Deep learning has become the state of the art approach in many machine learning problems such as classification.

It has recently been shown that deep learning is highly vulnerable to adversarial perturbations.

Taking the camera systems of self-driving cars as an example, small adversarial perturbations can cause the system to  make errors in important tasks, such as classifying traffic signs or detecting pedestrians.

Hence, in order to use deep learning without safety concerns a proper defense strategy is required.

We propose to use ensemble methods as a defense strategy against adversarial perturbations.

We find that an attack leading one model to misclassify does not imply the same for other networks performing the same task.

This makes ensemble methods an attractive defense strategy against adversarial attacks.

We empirically show for the MNIST and the CIFAR-10 data sets that ensemble methods not only improve the accuracy of neural networks on test data but also increase their robustness against adversarial perturbations.

Using ensemble methods as a defense to adversarial perturbations against deep neural networks.

This paper proposes to use ensembling as an adversarial defense mechanism.

Empirally investigated the robustness of different deep neural entworks ensembles to the two types of attacks, FGSM and BIM, on two popular datasets, MNIST and CIFAR10

Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural Networks

method defense adversarial perturbation deep neural


In this paper, we propose the Associative Conversation Model that generates visual information from textual information and uses it for generating sentences in order to utilize visual information in a dialogue system without image input.

In research on Neural Machine Translation, there are studies that generate translated sentences using both images and sentences, and these studies show that visual information improves translation performance.

However, it is not possible to use sentence generation algorithms using images for the dialogue systems since many text-based dialogue systems only accept text input.

Our approach generates (associates) visual information from input text and generates response text using context vector  fusing associative visual information and sentence textual information.

A comparative experiment between our proposed model and a model without association showed that our proposed model is generating useful sentences by associating visual information related to sentences.

Furthermore, analysis experiment of visual association showed that our proposed model generates (associates) visual information effective for sentence generation.

Proposal of the sentence generation method based on fusion between textual information and visual information associated with the textual information

This work describes a deep learning model for dialogue systems that takes advantage of visual information.

This paper proposes a novel dataset for grounded dialog and makes a computational observation that it could help to reason about vision even when performing text-based dialog.

Proposes to augment traditional text-based sentence generation/dialogue approaches by incorporating visual information by collecting a bunch of data consisting of both text and associated images or video

Associative Conversation Model: Generating Visual Information from Textual Information

textual information
associate
visual
sentence generation
model
dialogue
text-based


Feedforward convolutional neural network has achieved a great success in many computer vision tasks.

While it validly imitates the hierarchical structure of biological visual system, it still lacks one essential architectural feature: contextual recurrent connections with feedback, which widely exists in biological visual system.

In this work, we designed a Contextual Recurrent Convolutional Network with this feature embedded in a standard CNN structure.

We found that such feedback connections could enable lower layers to ``rethink" about their representations given the top-down contextual information.

We carefully studied the components of this network, and showed its robustness and superiority over feedforward baselines in such tasks as noise image classification, partially occluded object recognition and fine-grained image classification.

We believed this work could be an important step to help bridge the gap between computer vision models and real biological visual system.

we proposed a novel contextual recurrent convolutional network with robust property of visual learning

This paper introduces feedback connection to enhance feature learning through incorporating context information.

The paper proposes to add "recurrent" connections inside a convolution network with gating mechanism.

Contextual Recurrent Convolutional Model for Robust Visual Learning

contextual recurrent convolutional
network
robust
visual learning
connection


Deep neural networks have led to a series of breakthroughs, dramatically improving the state-of-the-art in many domains.

The techniques driving these advances, however, lack a formal method to account for model uncertainty.

While the Bayesian approach to learning provides a solid theoretical framework to handle uncertainty, inference in Bayesian-inspired deep neural networks is difficult.

In this paper, we provide a practical approach to Bayesian learning that relies on a regularization technique found in nearly every modern network, batch normalization.

We show that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models, and we demonstrate how this finding allows us to make useful estimates of the model uncertainty.

Using our approach, it is possible to make meaningful uncertainty estimates using conventional architectures without modifying the network or the training procedure.

Our approach is thoroughly validated in a series of empirical experiments on different tasks and using various measures, showing it to outperform baselines on a majority of datasets with strong statistical significance.

We show that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models, and we demonstrate how this finding allows us to make useful estimates of the model uncertainty in conventional networks.

This paper proposes using batch normalisation at test time to get the predictive uncertainty, and shows Monte Carlo prediction at test time using batch norm is better than dropout.

Proposes that the regularization procedure called batch normalization can be understood as performing approximate Bayesian inference, which performs similarly to MC dropout in terms of the estimates of uncertainty that it produces.

Bayesian Uncertainty Estimation for Batch Normalized Deep Networks

batch normalization
show
estimate
uncertainty
deep network
inference
dropout
approximate
bayesian


Data-parallel neural network training is network-intensive, so gradient dropping was designed to exchange only large gradients.

However, gradient dropping has been shown to slow convergence.

We propose to improve convergence by having each node combine its locally computed gradient with the sparse global gradient exchanged over the network.

We empirically confirm with machine translation tasks that gradient dropping with local gradients approaches convergence 48% faster than non-compressed multi-node training and 28% faster compared to vanilla gradient dropping.

We also show that gradient dropping with a local gradient update does not reduce the model's final quality.

We improve gradient dropping (a technique of only exchanging large gradients on distributed training) by incorporating local gradients while doing a parameter update to reduce quality loss and further improve the training time.

This paper proposes a 3 modes for combining local and global gradients to better use more computing nodes

Looks at the problem of reducing the communication requirement for implementing the distributed optimiztion techniques, particularly SGD

Combining Global Sparse Gradients with Local Gradients

local gradient
distribute
reduce
technique
combine
global


We establish the relation between Distributional RL and the Upper Confidence Bound (UCB) approach to exploration.

In this paper we show that the density of the Q function estimated by Distributional RL can be successfully used for the estimation of UCB.

This approach does not require counting and, therefore, generalizes well to the Deep RL.

We also point to the asymmetry of the empirical densities estimated by the Distributional RL algorithms like QR-DQN.

This observation leads to the reexamination of the variance's performance in the UCB type approach to exploration.

We introduce truncated variance as an alternative estimator of the UCB and a novel algorithm based on it.

We empirically show that newly introduced algorithm achieves better performance in multi-armed bandits setting.

Finally, we extend this approach to high-dimensional setting and test it on the Atari 2600 games.

New approach achieves better performance compared to QR-DQN in 26 of games, 13 ties out of 49 games.

Exploration using Distributional RL and truncagted variance.

Presents an RL method to manage exploration-explotation trade-offs via UCB techniques.

A method to use the distribution learned by Quantile Regression DQN for exploration, in place of the usual epsilon-greedy strategy.

Proposes new algorithsms (QUCB and QUCB+) to handle the exploration tradeoff in Multi-Armed Bendits and more generally in Reinforcement Learning

Exploration using Distributional RL and UCB

exploration distributional rl
method
ucb


Good representations facilitate transfer learning and few-shot learning.

Motivated by theories of language and communication that explain why communities with large number of speakers have, on average, simpler languages with more regularity, we cast the representation learning problem in terms of learning to communicate.

Our starting  point sees traditional autoencoders as  a single encoder with a fixed decoder partner that must learn to communicate.

Generalizing from there, we introduce community-based autoencoders in which multiple encoders and decoders collectively learn representations by being randomly paired up on successive training iterations.

Our experiments show that increasing community sizes reduce idiosyncrasies in the learned codes, resulting in more invariant representations with increased reusability and structure.

Motivated by theories of language and communication, we introduce community-based autoencoders, in which multiple encoders and decoders collectively learn structured and reusable representations.

The authors tackle the problem of representation learning, aim to build reusable and structured represenation, argue co-adaptation between encoder and decoder in traditional AE yields poor representation, and introduce community based auto-encoders.

The paper presents a community based autoencoder framework to address co-adaptation of encoders and decoders and aims at constructing better representations.

Shaping representations through communication

co-adaptation encoder decoder
introduce
aim
representation
reusable
structured
communication
community base


Humans are experts at high-fidelity imitation -- closely mimicking a demonstration, often in one attempt.

Humans use this ability to quickly solve a  task instance, and to bootstrap learning of new tasks.

Achieving these abilities in autonomous agents is an open problem.

In this paper, we introduce an off-policy RL algorithm (MetaMimic) to narrow this gap.

(i) policies for high-fidelity one-shot imitation of diverse novel skills, and

(ii) policies that enable the agent to solve tasks more efficiently than the demonstrators.

MetaMimic relies on the principle of storing all experiences in a memory and replaying these to learn massive deep neural network policies by off-policy RL.

This paper introduces, to the best of our knowledge, the largest existing neural networks for deep RL and shows that larger networks with normalization are needed to achieve one-shot high-fidelity imitation on a challenging manipulation task.

The results also show that both types of policy can be learned from vision, in spite of the task rewards being sparse, and without access to demonstrator actions.

We present MetaMimic, an algorithm that takes as input a demonstration dataset and outputs (i) a one-shot high-fidelity imitation policy (ii) an unconditional task policy.

The paper looks at the problem of one-shot imitation with high accuracy of imitation, extending DDPGfD to use only state trajectories.

This paper proposes an approach for one-shot imitation with high accuracy, and addresses the common problem of exploration in imitation learning.

Presents an RL method for learning from video demonstration without access to expert actions

One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL

one-shot imitation high accuracy
present
problem
rl
demonstration
high-fidelity


Normalization methods are a central building block in the deep learning toolbox.

They accelerate and stabilize training, while decreasing the dependence on manually tuned learning rate schedules.

When learning from multi-modal distributions, the effectiveness of batch normalization (BN), arguably the most prominent normalization method, is reduced.

As a remedy, we propose a more flexible approach: by extending the normalization to more than a single mean and variance, we detect modes of data on-the-fly, jointly normalizing samples that share common features.

We demonstrate that our method outperforms BN and other widely used normalization techniques in several experiments, including single and multi-task datasets.

We present a novel normalization method for deep neural networks that is robust to multi-modalities in intermediate feature distributions.

Normalization method that learns multi-modal distribution in the feature space

Proposes a generalization of Batch Normalization under the assumption that the statistics of the unit activations over the batches and over the spatial dimensions is not unimodal

Mode Normalization

normalization method
distribution
feature


Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving.

However, traditional multilingual translation usually yields inferior accuracy compared with the counterpart using individual models for each language pair, due to language diversity and model capacity limitations.

In this paper, we propose a distillation-based approach to boost the accuracy of multilingual machine translation.

Specifically, individual models are first trained and regarded as teachers, and then the multilingual model is trained to fit the training data and match the outputs of individual models simultaneously through knowledge distillation.

Experiments on IWSLT, WMT and Ted talk translation datasets demonstrate the effectiveness of our method.

Particularly, we show that one model is enough to handle multiple languages (up to 44 languages in our experiment), with comparable or even better accuracy than individual models.

We proposed a knowledge distillation based method to boost the accuracy of multilingual neural machine translation.

A many-to-one multilingual neural machine translation model that first training separate models for each language pair then performs distillation.

The paper aims at training a machine translation model by augmenting the standard cross-entropy loss with a distillation component based on individual (single-language-pair) teacher models.

Multilingual Neural Machine Translation with Knowledge Distillation

multilingual neural machine translation
knowledge distillation
model
base
train


What makes humans so good at solving seemingly complex video games?

Unlike computers, humans bring in a great deal of prior knowledge about the world, enabling efficient decision making.

This paper investigates the role of human priors for solving video games.

Given a sample game, we conduct a series of ablation studies to quantify the importance of various priors.

We do this by modifying the video game environment to systematically mask different types of visual information that could be used by humans as priors.

We find that removal of some prior knowledge causes a drastic degradation in the speed with which human players solve the game, e.g. from 2 minutes to over 20 minutes.

Furthermore, our results indicate that general priors, such as the importance of objects and visual consistency, are critical for efficient game-play.

We investigate the various kinds of prior knowledge that help human learning and find that general priors about objects play the most critical role in guiding human gameplay.

The authors study by experiment, what aspects of human priors are the important for reinforcement learning in video games.

The authors present a study of priors employed by humans in playing video games and demonstrates the existence of a taxonomy of features that affect the ability to complete tasks in the game to varying degrees.

Investigating Human Priors for Playing Video Games

play video game
human prior
learning
author
study
investigate


Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated.

Examples include grid search, uniform random search, low discrepancy sequences, and other sampling distributions.

In particular, we propose the use of $k$-determinantal point processes in  hyperparameter optimization via random search.

Compared to conventional uniform random search where hyperparameter settings are sampled independently, a $k$-DPP promotes diversity.

We describe an approach that transforms hyperparameter search spaces for efficient use with a $k$-DPP.

In addition, we introduce a novel Metropolis-Hastings algorithm which can sample from $k$-DPPs defined over spaces with a mixture of discrete and continuous dimensions.

Our experiments show significant benefits over uniform random search  in realistic scenarios with a limited budget for training supervised learners, whether in serial or parallel.

Driven by the need for parallelizable, open-loop hyperparameter optimization methods, we propose the use of $k$-determinantal point processes in  hyperparameter optimization via random search.

Proposes using the k-DPP to select candidate points in hyperparameter searches.

The authors propose k-DPP as an open loop method for hyperparameter optimization and provide its empirical study and comparison with other methods.

Considers non-sequential and uninformed hyperparameter search using determinantal point processes, which are probability distributions over subsets of a ground set with the property that subsets with more 'diverse' elements haev higher probability

Open Loop Hyperparameter Optimization and Determinantal Point Processes

method
k-dpp
determinantal point process
hyperparameter search
optimization
open loop


In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch.

When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task.

However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task.

In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model.

We eventually recommend a simple $L^2$ penalty using the pre-trained model as a reference, and we show that this approach behaves much better than the standard scheme using weight decay on a partially frozen network.

Addresses the problem of transfer learning in deep networks and proposes to have a regularization term that penalizes divergence from initialization.

Proposes an analysis on different adaptive regularization techniques for deep transfer learning, specifically focusing on the use of an L@-SP condition

Explicit Induction Bias for Transfer Learning with Convolutional Networks

transfer learn
learning
convolutional network
deep
regularization


Artificial neural networks have opened up a world of possibilities in data science and artificial intelligence, but neural networks are cumbersome tools that grow with the complexity of the learning problem.

We make contributions to this issue by considering a modified version of the fully connected layer we call a block diagonal inner product layer.

These modified layers have weight matrices that are block diagonal, turning a single fully connected layer into a set of densely connected neuron groups.

This idea is a natural extension of group, or depthwise separable, convolutional layers applied to the fully connected layers.

Block diagonal inner product layers can be achieved by either initializing a purely block diagonal weight matrix or by iteratively pruning off diagonal block entries.

This method condenses network storage and speeds up the run time without significant adverse effect on the testing accuracy, thus offering a new approach to improve network computation efficiency.

We look at neural networks with block diagonal inner product layers for efficiency.

This paper proposes making the inner layers in a neural network be block diagonal, and discusses that block diagonal matrices are more efficient than pruning and block diagonal layers lead to more efficient networks.

Replacing fully connected layers with block-diagonal fully connected layers

Neural Networks with Block Diagonal Inner Product Layers

neural network block diagonal inner product layer


One of the challenges in the study of generative adversarial networks is the instability of its training.

In this paper, we propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator.

Our new normalization technique is computationally light and easy to incorporate into existing implementations.

We tested the efficacy of spectral normalization on CIFAR10, STL-10, and ILSVRC2012 dataset, and we experimentally confirmed that spectrally normalized GANs (SN-GANs) is capable of generating images of better or equal quality relative to the previous training stabilization techniques.

We propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator of GANs.

This paper uses spectral regularization to normalize GAN objectives, and the ensuing GAN, called SN-GAN, essentially ensures the Lipschitz property of the discriminator.

This paper proposes"spectral normalization", moving a nice step forward in improving the training of GANs.

Spectral Normalization for Generative Adversarial Networks

discriminator
training
gan
spectral normalization
call


Humans acquire complex skills by exploiting previously learned skills and making transitions between them.

To empower machines with this ability, we propose a method that can learn transition policies which effectively connect primitive skills to perform sequential tasks without handcrafted rewards.

To efficiently train our transition policies, we introduce proximity predictors which induce rewards gauging proximity to suitable initial states for the next skill.

The proposed method is evaluated on a set of complex continuous control tasks in bipedal locomotion and robotic arm manipulation which traditional policy gradient methods struggle at.

We demonstrate that transition policies enable us to effectively compose complex skills with existing primitive skills.

The proposed induced rewards computed using the proximity predictor further improve training efficiency by providing more dense information than the sparse rewards from the environments.

We make our environments, primitive skills, and code public for further research at https://youngwoon.github.io/transition .

Transition policies enable agents to compose complex skills by smoothly connecting previously acquired primitive skills.

Proposes a scheme for transitioning to favorable strating states for executing given options in continuous domains. This uses two learning processes carried out simultaneously.

Presents a method for learning policies for transitioning from one task to another with the goal of completing complex tasks using state proximity estimator to reward for transition policy.

Proposes a new training scheme with a learned auxiliary reward function to optimise transition policies that connect the ending state of a previous macro action/option with good initiation states of the following macro action/option

Composing Complex Skills by Learning Transition Policies

compose complex skill
transition
scheme
option
state
learn
policy
connect
reward


Gated recurrent units (GRUs) were inspired by the common gated recurrent unit, long short-term memory (LSTM), as a means of capturing temporal structure with less complex memory unit architecture.

Despite their incredible success in tasks such as natural and artificial language processing, speech, video, and polyphonic music, very little is understood about the specific dynamic features representable in a GRU network.

As a result, it is difficult to know a priori how successful a GRU-RNN will perform on a given data set.

In this paper, we develop a new theoretical framework to analyze one and two dimensional GRUs as a continuous dynamical system, and classify the dynamical features obtainable with such system.

We found rich repertoire that includes stable limit cycles over time (nonlinear oscillations), multi-stable state transitions with various topologies, and homoclinic orbits.

In addition, we show that any finite dimensional GRU cannot precisely replicate the dynamics of a ring attractor, or more generally, any continuous attractor, and is limited to finitely many isolated fixed points in theory.

These findings were then experimentally verified in two dimensions by means of time series prediction.

We classify the the dynamical features one and two GRU cells can and cannot capture in continuous time, and verify our findings experimentally with k-step time series prediction.

The authors analyse GRUs with hidden sizes of one and two as continuous-time dynamical systems, claiming that the expressive power of the hidden state representation can provide prior knowledge on how well a GRU will perform on a given dataset

This paper analyzes GRUs from a dynamical systems perspective, and shows that 2d GRUs can be trained to adopt a variety of fixed points and can approximate line attractors, but cannot mimic a ring attractor.

Converts GRU equations into continuous time and uses theory and experiemnts to study 1- and 2-dimensional GRU networks and showcase every variety of dynamical topology available in these systems

The Expressive Power of Gated Recurrent Units as a Continuous Dynamical System

continuous time
expressive power
one two
dynamical
system
grus
variety
gru


Stacked hourglass network has become an important model for Human pose estimation.

The estimation of human body posture depends on the global information of the keypoints type and the local information of the keypoints location.

The consistent processing of inputs and constraints makes it difficult to form differentiated and determined collaboration mechanisms for each stacked hourglass network.

In this paper, we propose a Multi-Scale Stacked Hourglass (MSSH) network to high-light the differentiation capabilities of each Hourglass network for human pose estimation.

The pre-processing network forms feature maps of different scales,and dispatch them to various locations of the stack hourglass network, where the small-scale features reach the front of stacked hourglass network, and large-scale features reach the rear of stacked hourglass network.

And a new loss function is proposed for multi-scale stacked hourglass network.

Different keypoints have different weight coefficients of loss function at different scales, and the keypoints weight coefficients are dynamically adjusted from the top-level hourglass network to the bottom-level hourglass network.

Experimental results show that the pro-posed method is competitive with respect to the comparison algorithm on MPII and LSP datasets.

Differentiated inputs cause functional differentiation of the network, and the interaction of loss functions between networks can affect the optimization process.

A modification to the original hourglass network for single pose estimation that yields improvements over the original baseline.

Authors extend a stacked hourglass network with inception-resnet-A modules and propose a multi-scale approach for human pose estimation in still RGB images.

Multi-Scale Stacked Hourglass Network for Human Pose Estimation

stack hourglass network
human pose estimation
multi-scale


We present a new unsupervised method for learning general-purpose sentence embeddings.

Unlike existing methods which rely on local contexts, such as words

inside the sentence or immediately neighboring sentences, our method selects, for

each target sentence, influential sentences in the entire document based on a document

We identify a dependency structure of sentences using metadata

Furthermore, we propose a novel out-of-vocabulary word handling

technique to model many domain-specific terms, which were mostly discarded by

showing 30% precision improvement in coreference resolution in a technical domain,

and 7.5% accuracy increase in paraphrase detection compared to baselines.

To train a sentence embedding using technical documents, our approach considers document structure to find broader context and handle out-of-vocabulary words.

Presents ideas for improving sentence embedding by drawing from more context.

Learning sentence representations with sentences dependencies information

Extends the idea of forming an unsupervised representation of sentences used in the SkipThough approach by using a broader set of evidence for forming the representation of a sentence

UNSUPERVISED SENTENCE EMBEDDING USING DOCUMENT STRUCTURE-BASED CONTEXT

sentence embed
approach
document
context
idea
representation
broad


Neural network training relies on our ability to find ````````"good" minimizers of highly non-convex loss functions.

It is well known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better.

However, the reasons for these differences, and their effect on the underlying loss landscape, is not well understood.

In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods.

First, we introduce a simple ``"filter normalization" method that helps us visualize loss function curvature, and make meaningful side-by-side comparisons between loss functions.

Then, using a variety of visualizations, we explore how network architecture effects the loss landscape, and how training parameters affect the shape of minimizers.

We explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods.

This paper proposes a method to visualize the loss function of a NN and provides insights on the trainability and generalization of NNs.

Investigates the non-convexity of the loss surface and optimization paths.

Visualizing the Loss Landscape of Neural Nets

loss function
generalization
method
neural
landscape
visualize


Deep models are state-of-the-art for many computer vision tasks including image classification and object detection.

However, it has been shown that deep models are vulnerable to adversarial examples.

We highlight how one-hot encoding directly contributes to this vulnerability and propose breaking away from this widely-used, but highly-vulnerable mapping.

We demonstrate that by leveraging a different output encoding, multi-way encoding, we can make models more robust.

Our approach makes it more difficult for adversaries to find useful gradients for generating adversarial attacks.

We present state-of-the-art robustness results for black-box, white-box attacks, and achieve higher clean accuracy on four benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN when combined with adversarial training.

The strength of our approach is also presented in the form of an attack for model watermarking, raising challenges in detecting stolen models.

We demonstrate that by leveraging a multi-way output encoding, rather than the widely used one-hot encoding, we can make deep models more robust to adversarial attacks.

This paper proposes replacing the final cross-entropy layer trained on one-hot labels in classifiers by encoding each label as a high-dimensional vector and training the classifier to minimize L2 distance from the encoding of the correct class.

Authors propose new method against adversarial attacks that shows significant amount of gains compared to baselines

Multi-way Encoding for Robustness to Adversarial Attacks

adversarial attack
encoding
one-hot
multi-way