Generative adversarial networks (GANs) are a family of generative models that do not minimize a single training criterion.

Unlike other generative models, the data distribution is learned via a game between a generator (the generative model) and a discriminator (a teacher providing training signal) that each minimize their own cost.

GANs are designed to reach a Nash equilibrium at which each player cannot reduce their cost without changing the other players’ parameters.

One useful approach for the theory of GANs is to show that a divergence between the training distribution and the model distribution obtains its minimum value at equilibrium.

Several recent research directions have been motivated by the idea that this divergence is the primary guide for the learning process and that every step of learning should decrease the divergence.

We show that this view is overly restrictive.

During GAN training, the discriminator provides learning signal in situations where the gradients of the divergences between distributions would not be useful.

We provide empirical counterexamples to the view of GAN training as divergence minimization.

Specifically, we demonstrate that GANs are able to learn distributions in situations where the divergence minimization point of view predicts they would fail.

We also show that gradient penalties motivated from the divergence minimization perspective are equally helpful when applied in other contexts in which the divergence minimization perspective does not predict they would be helpful.

This contributes to a growing body of evidence that GAN training may be more usefully viewed as approaching Nash equilibria via trajectories that do not necessarily minimize a specific divergence at each step.

We find evidence that divergence minimization may not be an accurate characterization of GAN training.

The submission aims to present empirical evidence that the theory of divergence minimization is more a tool to understand the outcome of training GANs than a necessary condition to be enforce during training itself

This paper studies non-saturating GANs and the effect of two penalized gradient approaches, considering several thought experiments to demonstrate observations and validate them on real data experiments.

Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step


Measuring Mutual Information (MI) between high-dimensional, continuous, random variables from observed samples has wide theoretical and practical applications.

Recent works have developed accurate MI estimators through provably low-bias approximations and tight variational lower bounds assuming abundant supply of samples, but require an unrealistic number of samples to guarantee statistical significance of the estimation.

In this work, we focus on improving data efficiency and propose a Data-Efficient MINE Estimator (DEMINE) that can provide a tight lower confident interval of MI under limited data, through adding cross-validation to the MINE lower bound (Belghazi et al., 2018).

Hyperparameter search is employed and a novel meta-learning approach with task augmentation is developed to increase robustness to hyperparamters, reduce overfitting and improve accuracy.

With improved data-efficiency, our DEMINE estimator enables statistical testing of dependency at practical dataset sizes.

We demonstrate the effectiveness of DEMINE on synthetic benchmarks and a real world fMRI dataset, with application of inter-subject correlation analysis.

A new & practical statistical test of dependency using neural networks, benchmarked on synthetic and a real fMRI datasets.

Proposes a neural-network-based estimation of mutal information which can reliably work with small datasets, reducing the sample complexity by decoupling the network learning problem and the estimation problem.

A Data-Efficient Mutual Information Neural Estimator for Statistical Dependency Testing

network dataset dependency neural statistical information


Language and vision are processed as two different modal in current work for image captioning.

However, recent work on Super Characters method shows the effectiveness of two-dimensional word embedding, which converts text classification problem into image classification problem.

In this paper, we propose the SuperCaptioning method, which borrows the idea of two-dimensional word embedding from Super Characters method, and processes the information of language and vision together in one single CNN model.

The experimental results on Flickr30k data shows the proposed method gives high quality image captions.

An interactive demo is ready to show at the workshop.

Image captioning using two-dimensional word embedding.

SuperCaptioning: Image Captioning Using Two-dimensional Word Embedding

image captioning two-dimensional word embed


Determining the optimal order in which data examples are presented to Deep Neural Networks during training is a non-trivial problem.

However, choosing a non-trivial scheduling method may drastically improve convergence.

In this paper, we propose a Self-Paced Learning (SPL)-fused Deep Metric Learning (DML) framework, which we call Learning Embeddings for Adaptive Pace (LEAP).

Our method parameterizes mini-batches dynamically based on the \textit{easiness} and \textit{true diverseness} of the sample within a salient feature representation space.

In LEAP, we train an \textit{embedding} Convolutional Neural Network (CNN) to learn an expressive representation space by adaptive density discrimination using the Magnet Loss.

The \textit{student} CNN classifier dynamically selects samples to form a mini-batch based on the \textit{easiness} from cross-entropy losses and \textit{true diverseness} of examples from the representation space sculpted by the \textit{embedding} CNN.

We evaluate LEAP using deep CNN architectures for the task of supervised image classification on MNIST, FashionMNIST, CIFAR-10, CIFAR-100, and SVHN.

We show that the LEAP framework converges faster with respect to the number of mini-batch updates required to achieve a comparable or better test performance on each of the datasets.

LEAP combines the strength of adaptive sampling with that of mini-batch online learning and adaptive representation learning to formulate a representative self-paced strategy in an end-to-end DNN training protocol.

Introduces a method for creating mini batches for a student network by using a second learned representation space to dynamically select examples by their 'easiness and true diverseness'.

Experiments the classification accuracy on MNIST, FashionMNIST, and CIFAR-10 datasets to learn a representation with curriculum learning style minibatch selection in an end-to-end framework.

LEAP: Learning Embeddings for Adaptive Pace

end-to-end learn adaptive representation leap


Conventional deep reinforcement learning typically determines an appropriate primitive action at each timestep, which requires enormous amount of time and effort for learning an effective policy, especially in large and complex environments.

To deal with the issue fundamentally, we incorporate macro actions, defined as sequences of primitive actions, into the primitive action space to form an augmented action space.

The problem lies in how to find an appropriate macro action to augment the primitive action space.

The agent using a proper augmented action space is able to jump to a farther state and thus speed up the exploration process as well as facilitate the learning procedure.

In previous researches, macro actions are developed by mining the most frequently used action sequences or repeating previous actions.

However, the most frequently used action sequences are extracted from a past policy, which may only reinforce the original behavior of that policy.

On the other hand, repeating actions may limit the diversity of behaviors of the agent.

Instead, we propose to construct macro actions by a genetic algorithm, which eliminates the dependency of the macro action derivation procedure from the past policies of the agent.

Our approach appends a macro action to the primitive action space once at a time and evaluates whether the augmented action space leads to promising performance or not.

We perform extensive experiments and show that the constructed macro actions are able to speed up the learning process for a variety of deep reinforcement learning methods.

Our experimental results also demonstrate that the macro actions suggested by our approach are transferable among deep reinforcement learning methods and similar environments.

We further provide a comprehensive set of ablation analysis to validate our methodology.

We propose to construct macro actions by a genetic algorithm, which eliminates the dependency of the macro action derivation procedure from the past policies of the agent.

This paper proposes a generic algorithm for constructing macro actions for deep reinforcement learning by appending a macro action to the primitive action space.

Construction of Macro Actions for Deep Reinforcement Learning

macro action deep reinforcement learning


A key problem in neuroscience and life sciences more generally is that the data generation process is often best thought of as a hierarchy of dynamic systems.

One example of this is in-vivo calcium imaging data, where observed calcium transients are driven by a combination of electro-chemical kinetics where hypothesized trajectories around manifolds determining the frequency of these transients.

A recent approach using sequential variational auto-encoders demonstrated it was possible to learn the latent dynamic structure of reaching behaviour from spiking data modelled as a Poisson process.

Here we extend this approach using a ladder method to infer the spiking events driving calcium transients along with the deeper latent dynamic system.

We show strong performance of this approach on a benchmark synthetic dataset against a number of alternatives.

We propose an extension to LFADS capable of inferring spike trains to reconstruct calcium fluorescence traces using hierarchical VAEs.

Inferring hierarchies of latent features in calcium imaging data


In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs.

There have been several proposals to alleviate this issue with, for instance, triangulation and semi-supervised learning techniques, but they still require a strong cross-lingual signal.

In this work, we completely remove the need of parallel data and propose a novel method to train an NMT system in a completely unsupervised manner, relying on nothing but monolingual corpora.

Our model builds upon the recent work on unsupervised embedding mappings, and consists of a slightly modified attentional encoder-decoder model that can be trained on monolingual corpora alone using a combination of denoising and backtranslation.

Despite the simplicity of the approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014 French-to-English and German-to-English translation.

The model can also profit from small parallel corpora, and attains 21.81 and 15.24 points when combined with 100,000 parallel sentences, respectively.

Our implementation is released as an open source project.

We introduce the first successful method to train neural machine translation in an unsupervised manner, using nothing but monolingual corpora

The authors present a model for unsupervised NMT which requires no parallel corpora between the two languages of interest.

This is a paper on unsupervised MT which trains a standard architecture using word embeddings in a shared embedding space only with bilingual word papers and an encoder-decoder trained using monolingual data.

Unsupervised Neural Machine Translation

neural machine translation
monolingual
unsupervised
train
corpora
unsupervise


We describe a new training methodology for generative adversarial networks.

The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses.

This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2.

We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10.

Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator.

Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation.

As an additional contribution, we construct a higher-quality version of the CelebA dataset.

We train generative adversarial networks in a progressive fashion, enabling us to generate high-resolution images with high quality.

Introduces progressive growing and a simple parameter-free minibatch summary statistic feature for use in GAN training to enable synthesis of high-resolution images.

Progressive Growing of GANs for Improved Quality, Stability, and Variation

high-resolution image
gan
enable
quality
progressive


Designing a convolution for a spherical neural network requires a delicate tradeoff between efficiency and rotation equivariance.

DeepSphere, a method based on a graph representation of the discretized sphere, strikes a controllable balance between these two desiderata.

First, we study both theoretically and empirically how equivariance is affected by the underlying graph with respect to the number of pixels and neighbors.

Second, we evaluate DeepSphere on relevant problems.

Experiments show state-of-the-art performance and demonstrates the efficiency and flexibility of this formulation.

Perhaps surprisingly, comparison with previous work suggests that anisotropic filters might be an unnecessary price to pay.

A graph-based spherical CNN that strikes an interesting balance of trade-offs for a wide variety of applications.

Combines existing CNN frameworks based on the discretization of a sphere as a graph to show a convergence result which is related to the rotation equivalence on a sphere.

The authors use the existing graph CNN formulation and a pooling strategy that exploits hierarchical pixelations of the sphere to learn from the discretized sphere.

DeepSphere: a graph-based spherical CNN

graph-based spherical cnn
sphere
exist
graph


The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics.

In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity.

Here, we derive stationary fluctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm.

These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule.

We can further use the relations to efficiently extract information pertaining to a loss-function landscape such as the magnitudes of its Hessian and anharmonicity.

We prove fluctuation-dissipation relations for SGD, which can be used to (i) adaptively set learning rates and (ii) probe loss surfaces.

Paper's concepts work in the discrete-time formalism, use the master equation, and  remove reliance on a locally quadratic approximation of the loss function or on any Gaussian asumptions of the SGD noise.

The authors derive the stationary fluctuation-dissipation relations that link measurable quantities and hyperparameters in SGD and use the relations to set training schedule adaptively and analyze the loss-function landscape.

Fluctuation-dissipation relations for stochastic gradient descent

sgd
fluctuation-dissipation relation
adaptively
set
loss


Recurrent neural networks (RNNs) are difficult to train on sequence processing tasks, not only because input noise may be amplified through feedback, but also because any inaccuracy in the weights has similar consequences as input noise.

We describe a method for denoising the hidden state during training to achieve more robust representations thereby improving generalization performance.

Attractor dynamics are incorporated into the hidden state to `clean up' representations at each step of a sequence.

The attractor dynamics are trained through an auxillary denoising loss to recover previously experienced hidden states from noisy versions of those states.

This state-denoised recurrent neural network (SDRNN) performs multiple steps of internal processing for each external sequence step.

On a range of tasks, we show that the SDRNN outperforms a generic RNN as well as a variant of the SDRNN with attractor dynamics on the hidden state but without the auxillary loss.

We argue that attractor dynamics---and corresponding connectivity constraints---are an essential component of the deep learning arsenal and should be invoked not only for recurrent networks but also for improving deep feedforward nets and intertask transfer.

We propose a mechanism for denoising the internal state of an RNN to improve generalization performance.

State-Denoised Recurrent Neural Networks


We consider reinforcement learning in input-driven environments, where an exogenous, stochastic input process affects the dynamics of the system.

Input processes arise in many applications, including queuing systems, robotics control with disturbances, and object tracking.

Since the state dynamics and rewards depend on the input process, the state alone provides limited information for the expected future returns.

Therefore, policy gradient methods with standard state-dependent baselines suffer high variance during training.

We derive a bias-free, input-dependent baseline to reduce this variance, and analytically show its benefits over state-dependent baselines.

We then propose a meta-learning approach to overcome the complexity of learning a baseline that depends on a long sequence of inputs.

Our experimental results show that across environments from queuing systems, computer networks, and MuJoCo robotic locomotion, input-dependent baselines consistently improve training stability and result in better eventual policies.

For environments dictated partially by external input processes, we derive an input-dependent baseline that provably reduces the variance for policy gradient methods and improves the policy performance in a wide range of RL tasks.

The authors consider the problem of learning in input-driven environments, show how the PG theorem still applies for an input-aware critic, and show that input-dependent baselines are the best to use in conjecture with that critic.

This paper introduces the notion of input-dependent baselines in Policy Gradient Methods in RL, and proposes different methods to train the input dependent baseline function to help clear variance from external factor perturbation.

Variance Reduction for Reinforcement Learning in Input-Driven Environments

policy gradient method
input-dependent baseline
input-driven environment
external
variance
rl
input


Deep networks have shown great performance in classification tasks.

However, the parameters learned by the classifier networks usually discard stylistic information of the input, in favour of information strictly relevant to classification.

We introduce a network that has the capacity to do both classification and reconstruction by adding a "style memory" to the output layer of the network.

We also show how to train such a neural network as a deep multi-layer autoencoder, jointly minimizing both classification and reconstruction losses.

The generative capacity of our network demonstrates that the combination of style-memory neurons with the classifier neurons yield good reconstructions of the inputs when the classification is correct.

We further investigate the nature of the style memory, and how it relates to composing digits and letters.

Augmenting the top layer of a classifier network with a style memory enables it to be generative.

This paper proposes to train a classifier neural network not just to classifiy, but also to reconstruct a representation of its input, in order to factorize the class information from the appearance .

The paper proposes training an autoencoder such that the middle layer representation consists of the class label of the input and a hidden vector representation

Style Memory: Making a Classifier Network Generative

train
input
style memory
network
classifier
class
layer
generative
representation


Routing models, a form of conditional computation where examples are routed through a subset of components in a larger network, have shown promising results in recent works.

Surprisingly, routing models to date have lacked important properties, such as architectural diversity and large numbers of routing decisions.

Both architectural diversity and routing depth can increase the representational power of a routing network.

In this work, we address both of these deficiencies.

We discuss the significance of architectural diversity in routing models, and explain the tradeoffs between capacity and optimization when increasing routing depth.

In our experiments, we find that adding architectural diversity to routing models significantly improves performance, cutting the error rates of a strong baseline by 35% on an Omniglot setup.

However, when scaling up routing depth, we find that modern routing techniques struggle with optimization.

We conclude by discussing both the positive and negative results, and suggest directions for future research.

Per-example routing models benefit from architectural diversity, but still struggle to scale to a large number of routing decisions.

Adds diversity to the type of architectural unit available for the router at each decision and scaling to deeper networks, achieving state of the art performance on Omniglot.

This work extends routing networks to use diverse architectures across routed modules

Diversity and Depth in Per-Example Routing Models

scale
diversity
per-example routing model
architectural
decision
network
route


Across numerous applications, forecasting relies on numerical solvers for partial differential equations (PDEs).

Although the use of deep-learning techniques has been proposed, the uses have been restricted by the fact the training data are obtained using PDE solvers.

Thereby, the uses were limited to domains, where the PDE solver was applicable, but no further.

We present methods for training on small domains, while applying the trained models on larger domains, with consistency constraints ensuring the solutions are physically meaningful even at the boundary of the small domains.

We demonstrate the results on an air-pollution forecasting model for Dublin, Ireland.

We present RNNs for training surrogate models of PDEs, wherein consistency constraints ensure the solutions are physically meaningful, even when the training uses much smaller domains than the trained model is applied to.

Scaling up Deep Learning for PDE-based Models


We address the issue of limit cycling behavior in training Generative Adversarial Networks and propose the use of Optimistic Mirror Decent (OMD) for training Wasserstein GANs.

Recent theoretical results have shown that optimistic mirror decent (OMD) can enjoy faster regret rates in the context of zero-sum games.

WGANs is exactly a context of solving a zero-sum game with simultaneous no-regret dynamics.

Moreover, we show that optimistic mirror decent addresses the limit cycling problem in training WGANs.

We formally show that in the case of bi-linear zero-sum games the last iterate of OMD dynamics converges to an equilibrium, in contrast to GD dynamics which are bound to cycle.

We also portray the huge qualitative difference between GD and OMD dynamics with toy examples, even when GD is modified with many adaptations proposed in the recent literature, such as gradient penalty or momentum.

We apply OMD WGAN training to a bioinformatics problem of generating DNA sequences.

We observe that models trained with OMD achieve consistently smaller KL divergence with respect to the true underlying distribution, than models trained with GD variants.

Finally, we introduce a new algorithm, Optimistic Adam, which is an optimistic variant of Adam.

We apply it to WGAN training on CIFAR10 and observe improved performance in terms of inception score as compared to Adam.

We propose the use of optimistic mirror decent to address cycling problems in the training of GANs. We also introduce the Optimistic Adam algorithm

This paper proposes the use of optimistic mirror descent to train WGANs

The paper proposes to use optimistic gradient descent for GAN training that avoids the cycling behavior observed with SGD and its variants and provides promising results in GAN training.

This paper proposes a simple modification of standard gradient descent, claiming to improve the convergence of GANs and other minimax optimization problems.

Training GANs with Optimism

optimistic mirror
gradient descent
training
gan
problem
cycling
train


Learning good representations of users and items is crucially important to recommendation with implicit feedback.

Matrix factorization is the basic idea to derive the representations of users and items by decomposing the given interaction matrix.

However, existing matrix factorization based approaches share the limitation in that the interaction between user embedding and item embedding is only weakly enforced by fitting the given individual rating value, which may lose potentially useful information.

In this paper, we propose a novel Augmented Generalized Matrix Factorization (AGMF) approach that is able to incorporate the historical interaction information of users and items for learning effective representations of users and items.

Despite the simplicity of our proposed approach, extensive experiments on four public implicit feedback datasets demonstrate that our approach outperforms state-of-the-art counterparts.

Furthermore, the ablation study demonstrates that by using multi-hot encoding to enrich user embedding and item embedding for Generalized Matrix Factorization, better performance, faster convergence, and lower training loss can be achieved.

A simple extension of generalized matrix factorization can outperform state-of-the-art approaches for recommendation.

The work presents a matrix factorization framework for enforcing the effect of historical data when learning user preferences in collaborative filtering settings.

Rethinking Generalized Matrix Factorization for Recommendation: The Importance of Multi-hot Encoding

generalized matrix factorization
recommendation


We propose an unsupervised method for building dynamic representations of sequential data, particularly of observed interactions.

The method simultaneously acquires representations of input data and its dynamics.

It is based on a hierarchical generative model composed of two levels.

In the first level, a model learns representations to generate observed data.

In the second level, representational states encode the dynamics of the lower one.

The model is designed as a Bayesian network with switching variables represented in the higher level, and which generates transition models.

The method actively explores the latent space guided by its knowledge and the uncertainty about it.

That is achieved by updating the latent variables from prediction error signals backpropagated to the latent space.

So, no encoder or inference models are used since the generators also serve as their inverse transformations.

The method is evaluated in two scenarios, with static images and with videos.

The results show that the adaptation over time leads to better performance than with similar architectures without temporal dependencies, e.g., variational autoencoders.

With videos, it is shown that the system extracts the dynamics of the data in states that highly correlate with the ground truth of the actions observed.

A method that build representations of sequential data and its dynamics through generative models with an active process

Combines neural networks and Gaussian distributions to create an architecture and generative model for images and video which minimizes the error between generated and supplied images.

The paper proposes a Bayesian network model, realized as a neural network, that learns different data in the form of a linear dynamical system

Representing dynamically: An active process for describing sequential data

active process
generative model
datum
neural network
sequential


Activation is a nonlinearity function that plays a predominant role in the convergence and performance of deep neural networks.

While Rectified Linear Unit (ReLU) is the most successful activation function, its derivatives have shown superior performance on benchmark datasets.

In this work, we explore the polynomials as activation functions (order ≥ 2) that can approximate continuous real valued function within a given interval.

Leveraging this property, the main idea is to learn the nonlinearity, accepting that the ensuing function may not be monotonic.

While having the ability to learn more suitable nonlinearity, we cannot ignore the fact that it is a challenge to achieve stable performance due to exploding gradients - which is prominent with the increase in order.

To handle this issue, we introduce dynamic input scaling, output scaling, and lower learning rate for the polynomial weights.

Moreover, lower learning rate will control the abrupt fluctuations of the polynomials between weight updates.

In experiments on three public datasets, our proposed method matches the performance of prior activation functions, thus providing insight into a network’s nonlinearity preference.

We propose polynomial as activation functions.

The authors introduce learnable activation functions that are parameterized by polynomial functions and show results slightly better than ReLU.

POLYNOMIAL ACTIVATION FUNCTIONS


We introduce CBF, an exploration method that works in the absence of rewards or end of episode signal.

CBF is based on intrinsic reward derived from the error of a dynamics model operating in feature space.

It was inspired by (Pathak et al., 2017), is easy to implement, and can achieve results such as passing four levels of Super Mario Bros, navigating VizDoom mazes and passing two levels of SpaceInvaders.

We investigated the effect of combining the method with several auxiliary tasks, but find inconsistent improvements over the CBF baseline.

A simple intrinsic motivation method using forward dynamics model error in feature space of the policy.

Curiosity-driven Exploration by Bootstrapping Features


This paper is concerned with the robustness of VAEs to adversarial attacks.

We highlight that conventional VAEs are brittle under attack but that methods recently introduced for disentanglement such as β-TCVAE (Chen et al., 2018) improve robustness, as demonstrated through a variety of previously proposed adversarial attacks (Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al.(2018)).

This motivated us to develop Seatbelt-VAE, a new hierarchical disentangled VAE that is designed to be significantly more robust to adversarial attacks than existing approaches, while retaining high quality reconstructions.

We show that disentangled VAEs are more robust than vanilla VAEs to adversarial attacks that aim to trick them into decoding the adversarial input to a chosen target. We then develop an even more robust hierarchical disentangled VAE, Seatbelt-VAE.

The authors propose a new VAE model called seatbelt-VAE, showing to be more robust for latent attack than benchmarks.

Disentangling Improves VAEs' Robustness to Adversarial Attacks

robust
adversarial attack
show
seatbelt-vae
vae


The backpropagation algorithm is the de-facto standard for credit assignment in artificial neural networks due to its empirical results.

Since its conception, variants of the backpropagation algorithm have emerged.

More specifically, variants that leverage function changes in the backpropagation equations to satisfy their specific requirements.

Feedback Alignment is one such example, which replaces the weight transpose matrix in the backpropagation equations with a random matrix in search of a more biologically plausible credit assignment algorithm.

In this work, we show that function changes in the  backpropagation procedure is equivalent to adding an implicit learning rate to an artificial neural network.

Furthermore, we learn activation function derivatives in the backpropagation equations to demonstrate early convergence in these artificial neural networks.

Our work reports competitive performances with early convergence on MNIST and CIFAR10 on sufficiently large deep neural network architectures.

We demonstrate that function changes in the backpropagation is equivalent to an implicit learning rate

Function changes in the Backpropagation equation is equivalent to an implicit learning rate

equivalent implicit learning rate
function change backpropagation


Unsupervised text style transfer is the task of re-writing text of a given style into a target style without using a parallel corpus of source style and target style sentences for training.

Style transfer systems are evaluated on their ability to generate sentences that

3) preserve the non-stylistic parts (content) of the source sentence.

We train a reinforcement learning (RL) based unsupervised style transfer system that incorporates rewards for the above measures, and describe novel rewards shaping methods for the same.

Our approach does not attempt to disentangle style and content, and leverages the power of massively pre-trained language models as well as the Transformer.

Our system significantly outperforms existing state-of-art systems based on human as well as automatic evaluations on target style, fluency and content preservation as well as on overall success of style transfer, on a variety of datasets.

A reinforcement learning approach to text style transfer

Introduces an RL-based method which leverages a pre-trained language model to transfer text style, without a disentanglement objective, while using style-transfer generations from another model.

The authors propose a combination reward composed of fluency, content, and style for text style transfer.

RL-ST: Reinforcing Style, Fluency and Content Preservation for Unsupervised Text Style Transfer


Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what networks have learned inside the deep generative representations and how photo-realistic images are able to be composed from random noises.

In this work, we show that highly-structured semantic hierarchy emerges from the generative representations as the variation factors for synthesizing scenes.

By probing the layer-wise representations with a broad set of visual concepts at different abstraction levels, we are able to quantify the causality between the activations and the semantics occurring in the output image.

Such a quantification identifies the human-understandable variation factors learned by GANs to compose scenes.

The qualitative and quantitative results suggest that the generative representations learned by GAN are specialized to synthesize different hierarchical semantics: the early layers tend to determine the spatial layout and configuration, the middle layers control the categorical objects, and the later layers finally render the scene attributes as well as color scheme.

Identifying such a set of manipulatable latent semantics facilitates semantic scene manipulation.

We show that highly-structured semantic hierarchy emerges in the deep generative representations as a result for synthesizing scenes.

Paper investigates the aspects encoded by the latent variables input into different layers in StyleGAN.

The paper presents a visually-guided interpretation of activations of the convolution layers in the generator of StyleGAN on layout, scene category, scene attributes, and color.

Semantic Hierarchy Emerges in the Deep Generative Representations for Scene Synthesis

semantic hierarchy emerge deep generative representation


Variational autoencoders (VAEs) defined over SMILES string and graph-based representations of molecules promise to improve the optimization of molecular properties, thereby revolutionizing the pharmaceuticals and materials industries.

However, these VAEs are hindered by the non-unique nature of SMILES strings and the computational cost of graph convolutions.

To efficiently pass messages along all paths through the molecular graph, we encode multiple SMILES strings of a single molecule using a set of stacked recurrent neural networks, harmonizing hidden representations of each atom between SMILES representations, and use attentional pooling to build a final fixed-length latent representation.

By then decoding to a disjoint set of SMILES strings of the molecule, our All SMILES VAE learns an almost bijective mapping between molecules and latent representations near the high-probability-mass subspace of the prior.

Our SMILES-derived but molecule-based latent representations significantly surpass the state-of-the-art in a variety of fully- and semi-supervised property regression and molecular property optimization tasks.

We pool messages amongst multiple SMILES strings of the same molecule to pass information along all paths through the molecular graph, producing latent representations that significantly surpass the state-of-the-art in a variety of tasks.

Method uses multiple inputs of SMILES strings, character-wise feature fusion across those strings, and network training through multiple output targets of SMILES strings, creating a robust fixed-length latent representation independent of SMILES variation.

The authors describe a novel variational autoencoder like method for molecules which encode molecules as strings to reduce the operations needed to share information across atoms in the molecule.

All SMILES Variational Autoencoder for Molecular Property Prediction and Optimization

smile string
molecule
molecular
latent representation
method
variational autoencoder
multiple
information
across


We propose a simple yet highly effective method that addresses the mode-collapse problem in the Conditional Generative  Adversarial  Network (cGAN).

Although conditional distributions are multi-modal (i.e., having many modes) in practice, most cGAN approaches tend to learn an overly simplified distribution where an input is always mapped to a single output regardless of variations in latent code.

To address such issue, we propose to explicitly regularize the generator to produce diverse outputs depending on latent codes.

The proposed regularization is simple, general, and can be easily integrated into most conditional GAN objectives.

Additionally, explicit regularization on generator allows our method to control a balance between visual quality and diversity.

We demonstrate the effectiveness of our method on three conditional generation tasks: image-to-image translation, image inpainting, and future video prediction.

We show that simple addition of our regularization to existing models leads to surprisingly diverse generations, substantially outperforming the previous approaches for multi-modal conditional generation specifically designed in each individual task.

We propose a simple and general approach that avoids a mode collapse problem in various conditional GANs.

The paper proposes a regularization term for the conditional GAN objective in order to promote diverse multimodal generation and prevent mode collapse.

The paper proposes a method for generating diverse outputs for various conditional GAN frameworks including image-to-image translation, image-inpainting, and video prediction, which can be applied to various conditional synthesis frameworks for various tasks.

Diversity-Sensitive Conditional Generative Adversarial Networks

various conditional gan
mode collapse
diverse


The transformer is a state-of-the-art neural translation model that uses attention to iteratively refine lexical representations with information drawn from the surrounding context.

Lexical features are fed into the first layer and propagated through a deep network of hidden layers.

We argue that the need to represent and propagate lexical features in each layer limits the model’s capacity for learning and representing other information relevant to the task.

To alleviate this bottleneck, we introduce gated shortcut connections between the embedding layer and each subsequent layer within the encoder and decoder.

This enables the model to access relevant lexical content dynamically, without expending limited resources on storing it within intermediate states.

We show that the proposed modification yields consistent improvements on standard WMT translation tasks and reduces the amount of lexical information passed along the hidden layers.

We furthermore evaluate different ways to integrate lexical connections into the transformer architecture and present ablation experiments exploring the effect of proposed shortcuts on model behavior.

Equipping the transformer model with shortcuts to the embedding layer frees up model capacity for learning novel information.

Addressing the Representation Bottleneck in Neural Machine Translation with Lexical Shortcuts


Probability density estimation is a classical and well studied problem, but standard density estimation methods have historically lacked the power to model complex and high-dimensional image distributions.

More recent generative models leverage the power of neural networks to implicitly learn and represent probability models over complex images.

We describe methods to extract explicit probability density estimates from GANs, and explore the properties of these image density functions.

We perform sanity check experiments to provide evidence that these probabilities are reasonable.

However, we also show that density functions of natural images are difficult to interpret and thus limited in use.

We study reasons for this lack of interpretability, and suggest that we can get better interpretability by doing density estimation on latent representations of images.

We examine the relationship between probability density values and image content in non-invertible GANs.

The authors try to estimate the probability distribution of the image with the help of GAN and develop a proper approximation to the PDFs in the latent space.

Understanding the (Un)interpretability of Natural Image Distributions Using Generative Models


Convolutional Neural Networks (CNNs) are composed of multiple convolution layers and show elegant performance in vision tasks.

The design of the regular convolution is based on the Receptive Field (RF) where the information within a specific region is processed.

In the view of the regular convolution's RF, the outputs of neurons in lower layers with smaller RF are bundled to create neurons in higher layers with larger RF.

As a result, the neurons in high layers are able to capture the global context even though the neurons in low layers only see the local information.

However, in lower layers of the biological brain, the information outside of the RF changes the properties of neurons.

In this work, we extend the regular convolution and propose spatially shuffled convolution (ss convolution).

In ss convolution, the regular convolution is able to use the information outside of its RF by spatial shuffling which is a simple and lightweight operation.

We perform experiments on CIFAR-10 and ImageNet-1k dataset, and show that ss convolution improves the classification performance across various CNNs.

We propose spatially shuffled convolution that the regular convolution incorporates the information from outside of its receptive field.

Proposes SS convulation which uses information outside of its RF, showing improved results when tested on multiple CNN models.

The authors proposed a shuffle strategy for convolution layers in convolution layers in convolutional neural networks.

Incorporating Horizontal Connections in Convolution by Spatial Shuffling

information outside
convolution
shuffle
incorporate


We propose a framework to model the distribution of sequential data coming from

a set of entities connected in a graph with a known topology.

based on a mixture of shared hidden Markov models (HMMs), which are trained

in order to exploit the knowledge of the graph structure and in such a way that the

Experiments in different application domains

demonstrate the effectiveness and versatility of the method.

A method to model the generative distribution of sequences coming from graph connected entities.

The authors propose a method to model sequential data from multiple interconnected sources using a mixture of common pool of HMM's.

SpaMHMM: Sparse Mixture of Hidden Markov Models for Graph Connected Entities

method model
graph connect entity
mixture


To gain high rewards in muti-agent scenes, it is sometimes necessary to understand other agents and make corresponding optimal decisions.

We can solve these tasks by first building models for other agents and then finding the optimal policy with these models.

To get an accurate model, many observations are needed and this can be sample-inefficient.

What's more, the learned model and policy can overfit to current agents and cannot generalize if the other agents are replaced by new agents.

In many practical situations, each agent we face can be considered as a sample from a population with a fixed but unknown distribution.

Thus we can treat the task against some specific agents as a task sampled from a task distribution.

We apply meta-learning method to build models and learn policies.

Therefore when new agents come, we can adapt to them efficiently.

Experiments on grid games show that our method can quickly get high rewards.

Our work applies meta-learning to multi-agent Reinforcement Learning to help our agent efficiently adapted to new coming opponents.

This paper focuses on fast adaptation to new behaviour of the other agents of the environment using a method based on MAML

The paper presents an approach to multi-agent learning based on the framework of model-agnostic meta learning for the task of opponent modeling for multi-agent RL.

Sample-efficient policy learning in multi-agent Reinforcement Learning via meta-learning

multi-agent reinforcement learning
opponent
agent
new
base
meta-learne


We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation.

This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball.

We show that this is an effective regularizer;  for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2% to 5.3%.

The paper is dedicated to computation of singular values of convolutional layers

Derives exact formulas for computing singular values of convolutional layers of deep neural networks and show that computing the singular values can be done much faster than computing the full SVD of the convolution matrix by appealing to fast FFT transformations.

The Singular Values of Convolutional Layers

singular value convolutional layer
computation
transformation


Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning.

A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment.

Computing a Bayes-optimal policy is however intractable for all but the smallest tasks.

In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection.

In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty.

We also evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher return during training than existing methods.

VariBAD opens a path to tractable approximate Bayes-optimal exploration for deep RL using ideas from meta-learning, Bayesian RL, and approximate variational inference.

This paper presents a new deep reinforcement learning method that can efficiently trade-off exploration and exploitation that combines meta-learning, variational inference, and bayesian RL.

VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning

bayesian rl
variational inference
exploration
meta-learning
method
deep
varibad


In a continual learning setting, new categories may be introduced over time, and an ideal learning system should perform well on both the original categories and the new categories.

While deep neural nets have achieved resounding success in the classical setting, they are known to forget about knowledge acquired in prior episodes of learning if the examples encountered in the current episode of learning are drastically different from those encountered in prior episodes.

This makes deep neural nets ill-suited to continual learning.

In this paper, we propose a new model that can both leverage the expressive power of deep neural nets and is resilient to forgetting when new categories are introduced.

We demonstrate an improvement in terms of accuracy on original classes compared to a vanilla deep neural net.

We show metric learning can help reduce catastrophic forgetting

This paper applies metric learning to reduce catastrophic forgetting on neural networks by improving the expressiveness of the final layer, leading to better results in continual learning.

Better Knowledge Retention through Metric Learning

reduce catastrophic forgetting
metric learning
well


Biomedical knowledge bases are crucial in modern data-driven biomedical sciences, but auto-mated biomedical knowledge base construction remains challenging.

In this paper, we consider the problem of disease entity normalization, an essential task in constructing a biomedical knowledge base.

We present NormCo, a deep coherence model which considers the semantics of an entity mention, as well as the topical coherence of the mentions within a single document.

NormCo mod-els entity mentions using a simple semantic model which composes phrase representations from word embeddings, and treats coherence as a disease concept co-mention sequence using an RNN rather than modeling the joint probability of all concepts in a document, which requires NP-hard inference.

To overcome the issue of data sparsity, we used distantly supervised data and synthetic data generated from priors derived from the BioASQ dataset.

Our experimental results show thatNormCo outperforms state-of-the-art baseline methods on two disease normalization corpora in terms of (1) prediction quality and (2) efficiency, and is at least as performant in terms of accuracy and F1 score on tagged documents.

We present NormCo, a deep coherence model which considers the semantics of an entity mention, as well as the topical coherence of the mentions within a single document to perform disease entity normalization.

Uses a GRU autoencoder to represent the "context" (related enitities of a given disease within the span of a sentence), solving the BioNLP task with significant improvements over the best-known methods.

NormCo: Deep Disease Normalization for Biomedical Knowledge Base Construction

within normco deep normalization disease


We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others.

Multiplicative interaction layers as primitive operations have a long-established presence in the literature, though this often not emphasized and thus under-appreciated.

We begin by showing that such layers strictly enrich the representable function classes of neural networks.

We conjecture that multiplicative interactions offer a particularly powerful inductive bias when fusing multiple streams of information or when conditional computation is required.

We therefore argue that they should be considered in many situation where multiple compute or information paths need to be combined, in place of the simple and oft-used concatenation operation.

Finally, we back up our claims and demonstrate the potential of multiplicative interactions by applying them in large-scale complex RL and sequence modelling tasks, where their use allows us to deliver state-of-the-art results, and thereby provides new evidence in support of multiplicative interactions playing a more prominent role when designing new neural network architectures.

Presents multiplicative interaction as a unified characterization for representing commonly used model architecture design components, showing empirical proof of superior performance on tasks like RL and sequence modeling.

The paper explores different types of multiplicative interactions and finds MI models able to achieve a state-of-the-art performance on language modeling and reinforcement learning problems.

Multiplicative Interactions and Where to Find Them

multiplicative interaction
explore
performance
find
model


Developing conditional generative models for text-to-video synthesis is an extremely challenging yet an important topic of research in machine learning.

In this work, we address this problem by introducing Text-Filter conditioning Generative Adversarial Network (TFGAN), a GAN model with novel conditioning scheme that aids improving the text-video associations.

With a combination of this conditioning scheme and a deep GAN architecture, TFGAN generates photo-realistic videos from text on very challenging real-world video datasets.

In addition, we construct a benchmark synthetic dataset of moving shapes to systematically evaluate our conditioning scheme.

Extensive experiments demonstrate that TFGAN significantly outperforms the existing approaches, and can also generate videos of novel categories not seen during training.

An effective text-conditioning GAN framework for generating videos from text

This paper presents a GAN-based method for video generation conditioned on text description, with a new conditioning method that generates convolution filters from the encoded text, and uses them for a convolution in the discriminator.

This paper proposes conditional GAN models for text-to-video synthesis: developing text-feature-conditioned CNN filters and constructing moving-shape dataset with improved performance on video/image generation.

TFGAN: Improving Conditioning for Text-to-Video Synthesis

text-to-video synthesis
generate
text
video
filter
gan
conditioning
generation


Over-parameterization is ubiquitous nowadays in training neural networks to benefit both optimization in seeking global optima and generalization in reducing prediction error.

However, compressive networks are desired in many real world applications and direct training of small networks may be trapped in local optima.

In this paper, instead of pruning or distilling over-parameterized models to compressive ones, we propose a new approach based on \emph{differential inclusions of inverse scale spaces}, that generates a family of models from simple to complex ones by coupling gradient descent and mirror descent to explore model structural sparsity.

It has a simple discretization, called the Split Linearized Bregman Iteration (SplitLBI), whose global convergence analysis in deep learning is established that from any initializations, algorithmic iterations converge to a critical point of empirical risks.

Experimental evidence shows that\ SplitLBI may achieve state-of-the-art performance in large scale training on ImageNet-2012 dataset etc., while with \emph{early stopping} it unveils effective subnet architecture with comparable test accuracies to dense models after retraining instead of pruning well-trained ones.

SplitLBI is applied to deep learning to explore model structural sparsity, achieving state-of-the-art performance in ImageNet-2012 and unveiling effective subnet architecture.

Proposes an optimization based algorithm for finding important sparse structures of large-scale neural networks by coupling the learning of weight matrix and sparsity constraints, offering guaranteed convergence on nonconvex optimization problems.

Split LBI for Deep Learning: Structural Sparsity via Differential Inclusion Paths


In this paper, we study the learned iterative shrinkage thresholding algorithm (LISTA) for solving sparse coding problems.

Following assumptions made by prior works, we first discover that the code components in its estimations may be lower than expected, i.e., require gains, and to address this problem, a gated mechanism amenable to theoretical analysis is then introduced.

Specific design of the gates is inspired by convergence analyses of the mechanism and hence its effectiveness can be formally guaranteed.

In addition to the gain gates, we further introduce overshoot gates for compensating insufficient step size in LISTA.

Extensive empirical results confirm our theoretical findings and verify the effectiveness of our method.

We propose gated mechanisms to enhance learned ISTA for sparse coding, with theoretical guarantees on the superiority of the method.

Proposes extensions to LISTA which address underestimation by introducing "gain gates" and including momentum with "overshoot gates", showing improved convergence rates.

This paper is focused on solving sparse coding problems using LISTA-type networks by proposing a "gain gating function" to mitigate the weakness of the "no false positive" assumption.

Sparse Coding with Gated Learned ISTA


The learning of hierarchical representations for image classification has experienced an impressive series of successes due in part to the availability of large-scale labeled data for training.

On the other hand, the trained classifiers have traditionally been evaluated on a handful of test images, which are deemed to be extremely sparsely distributed in the space of all natural images.

It is thus questionable whether recent performance improvements on the excessively re-used test sets generalize to real-world natural images with much richer content variations.

In addition, studies on adversarial learning show that it is effortless to construct adversarial examples that fool nearly all image classifiers, adding more complications to relative performance comparison of existing models.

This work presents an efficient framework for comparing image classifiers, which we name the MAximum Discrepancy (MAD) competition.

Rather than comparing image classifiers on fixed test sets, we adaptively sample a test set from an arbitrarily large corpus of unlabeled images so as to maximize the discrepancies between the classifiers, measured by the distance over WordNet hierarchy.

Human labeling on the resulting small and model-dependent image sets reveals the relative performance of the competing classifiers and provides useful insights on potential ways to improve them.

We report the MAD competition results of eleven ImageNet classifiers while noting that the framework is readily extensible and cost-effective to add future classifiers into the competition.

We present an efficient and adaptive framework for comparing image classifiers to maximize the discrepancies between the classifiers, in place of comparing on fixed test sets.

Error spotting mechanism which compares image classifiers by sampling their "most disagreed" test set, measuring disagreement through a semantics-aware distance derived form WordNet ontology.

I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively

compare image classifier
test set
discrepancy


Robustness of neural networks has recently been highlighted by the adversarial examples, i.e., inputs added with well-designed  perturbations which are imperceptible to humans but can cause the network to give incorrect outputs.

In this paper, we design a new CNN architecture that by itself has good robustness.

We introduce a simple but powerful technique, Random Mask, to modify existing CNN structures.

We show that CNN with Random Mask achieves state-of-the-art performance against black-box adversarial attacks without applying any adversarial training.

We next investigate the adversarial examples which “fool” a CNN with Random Mask.

Surprisingly, we find that these adversarial examples often “fool” humans as well.

This raises fundamental questions on how to define adversarial examples and robustness properly.

We propose a technique that modifies CNN structures to enhance robustness while keeping high test accuracy, and raise doubt on whether current definition of adversarial examples is appropriate by generating adversarial examples able to fool humans.

This paper proposes a simple technique for improving the robustness of neural networks against black-box attacks.

The authors propose a simple method for increasing the robustness of convolutional neural networks against adversarial examples, with surprisingly good results.

RANDOM MASK: Towards Robust Convolutional Neural Networks

adversarial example
simple
robustness
neural network
technique
convolutional


Supervised deep learning methods require cleanly labeled large-scale datasets, but collecting such data is difficult and sometimes impossible.

There exist two popular frameworks to alleviate this problem: semi-supervised learning and robust learning to label noise.

Although these frameworks relax the restriction of supervised learning, they are studied independently.

Hence, the training scheme that is suitable when only small cleanly-labeled data are available remains unknown.

In this study, we consider learning from bi-quality data as a generalization of these studies, in which a small portion of data is cleanly labeled, and the rest is corrupt.

Under this framework, we compare recent algorithms for semi-supervised and robust learning.

The results suggest that semi-supervised learning outperforms robust learning with noisy labels.

We also propose a training strategy for mixing mixup techniques to learn from such bi-quality data effectively.

We propose to compare semi-supervised and robust learning to noisy label under a shared setting

The authors propose a strategy based on mixup for training a model in a formal setting that includes the semi-supervised and the robust learning tasks as special cases.

Unifying semi-supervised and robust learning by mixup

semi-supervised robust learning
mixup
setting


Hierarchical Sparse Coding (HSC) is a powerful model to efficiently represent multi-dimensional, structured data such as images.

The simplest solution to solve this computationally hard problem is to decompose it into independent layerwise subproblems.

However, neuroscientific evidence would suggest inter-connecting these subproblems as in the Predictive Coding (PC) theory, which adds top-down connections between consecutive layers.

In this study, a new model called Sparse Deep Predictive Coding (SDPC) is introduced to assess the impact of this inter-layer feedback connection.

In particular, the SDPC is compared with a Hierarchical Lasso (Hi-La) network made out of a sequence of Lasso layers.

A 2-layered SDPC and a Hi-La networks are trained on 3 different databases and with different sparsity parameters on each layer.

First, we show that the overall prediction error generated by SDPC is lower thanks to the feedback mechanism as it transfers prediction error between layers.

Second, we demonstrate that the inference stage of the SDPC is faster to converge than for the Hi-La model.

Third, we show that the SDPC also accelerates the learning process.

Finally, the qualitative analysis of both models dictionaries, supported by their activation probability, show that the SDPC features are more generic and informative.

This paper experimentally demonstrates the beneficial effect of top-down connections in Hierarchical Sparse Coding algorithm.

This paper presents a study that compares techniques for Hierarchical Sparse Coding, showing that the top-down term is beneficial in reducing predictive error and can learn faster.

Effect of top-down connections in Hierarchical Sparse Coding

effect top-down connection hierarchical sparse code


Explaining a deep learning model can help users understand its behavior and allow researchers to discern its shortcomings.

Recent work has primarily focused on explaining models for tasks like image classification or visual question answering.

In this paper, we introduce an explanation approach for image similarity models, where a model's output is a score measuring the similarity of two inputs rather than a classification.

In this task, an explanation depends on both of the input images, so standard methods do not apply.

We propose an explanation method that pairs a saliency map identifying important image regions with an attribute that best explains the match.

We find that our explanations provide additional information not typically captured by saliency maps alone, and can also improve performance on the classic task of attribute recognition.

Our approach's ability to generalize is demonstrated on two datasets from diverse domains, Polyvore Outfits and Animals with Attributes 2.

A black box approach for explaining the predictions of an image similarity model.

Introduces method for image similarity model explanation which identifies attributes that contribute positively to the similarity score and pairs them with a generated saliency map.

The paper proposes an explanation mechanism that pairs the typical saliency map regions together with attributes for similarity matching deep neural networks.

Why do These Match? Explaining the Behavior of Image Similarity Models

image similarity model
pair
explain
saliency map
attribute
explanation
match


Adversarial examples have been shown to be an effective way of assessing the robustness of neural sequence-to-sequence (seq2seq) models, by applying perturbations to the input of a model leading to large degradation in performance.

However, these perturbations are only indicative of a weakness in the model if they do not change the semantics of the input in a way that would change the expected output.

Using the example of machine translation (MT), we propose a new evaluation framework for adversarial attacks on seq2seq models taking meaning preservation into account and demonstrate that existing methods may not preserve meaning in general.

Based on these findings, we propose new constraints for attacks on word-based MT systems and show, via human and automatic evaluation, that they produce more semantically similar adversarial inputs.

Furthermore, we show that performing adversarial training with meaning-preserving attacks is beneficial to the model in terms of adversarial robustness without hurting test performance.

How you should evaluate adversarial attacks on seq2seq

The authors investigate ways of generating adversarial examples, showing that adversarial training with the attack most consistent with the introduced meaning-preservation criteria results in improved robustness to this type of attack without degradation in the non-adversarial setting.

The paper is about meaning-preserving adversarial perturbations in the context of Seq2Seq models

On Meaning-Preserving Adversarial Perturbations for Sequence-to-Sequence Models


We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs.

The proposed technique keeps the contribution of positive and negative weights to the layer output in equilibrium.

We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.

An alternative normalization technique to batch normalization

Introduces a normalization technique, which normalizes the weights of convolutional layers.

This manuscript introduces a new layer-wise transform, EquiNorm, to improve upon batch normalization that does not modify the inputs to the layers but rather the layer weights.

CONTROLLING COVARIATE SHIFT USING EQUILIBRIUM NORMALIZATION OF WEIGHTS

normalization technique
layer
weight
introduce
batch


We present a framework for building unsupervised representations of entities and their compositions, where each entity is viewed as a probability distribution rather than a fixed length vector.

In particular, this distribution is supported over the contexts which co-occur with the entity and are embedded in a suitable low-dimensional space.

This enables us to consider the problem of representation learning with a perspective from Optimal Transport and take advantage of its numerous tools such as Wasserstein distance and Wasserstein barycenters.

We elaborate how the method can be applied for obtaining unsupervised representations of text and illustrate the performance quantitatively as well as qualitatively on tasks such as measuring sentence similarity and word entailment, where we empirically observe significant gains (e.g., 4.1% relative improvement over Sent2vec and GenSen).

The key benefits of the proposed approach include:

(a) capturing uncertainty and polysemy via modeling the entities as distributions,

(b) utilizing the underlying geometry of the particular task (with the ground cost),

(c) simultaneously providing interpretability with the notion of optimal transport between contexts and

(d) easy applicability on top of existing point embedding methods.

In essence, the framework can be useful for any unsupervised or supervised problem (on text or other modalities); and only requires a co-occurrence structure inherent to many problems.

The code, as well as pre-built histograms, are available under https://github.com/context-mover.

Represent each entity as a probability distribution over contexts embedded in a ground space.

Proposes to construct word embeddings from a histogram over context words, instead of as point vectors, which allows for measuring distances between two words in terms of optimal transport between the histograms through a method that augments representation of an entity from standard "point in a vector space" to a histogram with bins located at some points in that vector space.

Context Mover's Distance & Barycenters: Optimal transport of contexts for building representations

entity
optimal transport
space
context
representation
distance


Over the last few years, the phenomenon of adversarial examples --- maliciously constructed inputs that fool trained machine learning models --- has captured the attention of the research community, especially when the adversary is restricted to making small modifications of a correctly handled input.

At the same time, less surprisingly, image classifiers lack human-level performance on randomly corrupted images, such as images with additive Gaussian noise.

In this work, we show that these are two manifestations of the same underlying phenomenon.

We establish this connection in several ways.

First, we find that adversarial examples exist at the same distance scales we would expect from a linear model with the same performance on corrupted images.

Next, we show that Gaussian data augmentation during training improves robustness to small adversarial perturbations and that adversarial training improves robustness to several types of image corruptions.

Finally, we present a model-independent upper bound on the distance from a corrupted image to its nearest error given test performance and show that in practice we already come close to achieving the bound, so that improving robustness further for the corrupted image distribution requires significantly reducing test error.

All of this suggests that improving adversarial robustness should go hand in hand with improving performance in the presence of more general and realistic image corruptions.

This yields a computationally tractable evaluation metric for defenses to consider: test error in noisy image distributions.

Small adversarial perturbations should be expected given observed error rates of models outside the natural data distribution.

This paper proposes an alternative view for adversarial examples in high dimension spaces by considering the "error rate" in a Gaussian distribution centered at each test point.

Adversarial Examples Are a Natural Consequence of Test Error in Noise

adversarial example
error rate
natural
test
distribution


Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training.

Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019).

However, surprisingly,  the simple baseline of just pre-training and fine-tuning compact models has been overlooked.

In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work.

Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation.

The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements.

Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data.

One surprising observation is that they have a compound effect even when sequentially applied on the same data.

To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.

Studies how self-supervised learning and knowledge distillation interact in the context of building compact models.

Investigates training compact pre-trained language models via distillation and shows that using a teacher for distilling a compact student model performs better than directly pre-training the model.

This submission shows that pre-training a student directly on masked language modeling is better than distillation, and the best is to combine both and distill from that pre-trained student model.

Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

well model directly pre-traine distillation show compact pre-trained student language


In this paper, we investigate lossy compression of deep neural networks (DNNs) by weight quantization and lossless source coding for memory-efficient deployment.

Whereas the previous work addressed non-universal scalar quantization and entropy coding of DNN weights, we for the first time introduce universal DNN compression by universal vector quantization and universal source coding.

In particular, we examine universal randomized lattice quantization of DNNs, which randomizes DNN weights by uniform random dithering before lattice quantization and can perform near-optimally on any source without relying on knowledge of its probability distribution.

Moreover, we present a method of fine-tuning vector quantized DNNs to recover the performance loss after quantization.

Our experimental results show that the proposed universal DNN compression scheme compresses the 32-layer ResNet (trained on CIFAR-10) and the AlexNet (trained on ImageNet) with compression ratios of $47.1$ and $42.5$, respectively.

We introduce the universal deep neural network compression scheme, which is applicable universally for compression of any models and can perform near-optimally regardless of their weight distribution.

Introduces a pipeline for network compression that is similar to deep compression and uses randomized lattice quantization instead of the classical vector quantization, and uses universal source coding (bzip2) instead of Huffman coding.

Universal Deep Neural Network Compression


What would be learned by variational autoencoder(VAE) and what influence the disentanglement of VAE?

This paper tries to preliminarily address VAE's intrinsic dimension, real factor, disentanglement and indicator issues theoretically in the idealistic situation and implementation issue practically through noise modeling perspective in the realistic case.

On intrinsic dimension issue, due to information conservation, the idealistic VAE learns and only learns intrinsic factor dimension.

Besides, suggested by mutual information separation property, the constraint induced by Gaussian prior to the VAE objective encourages the information sparsity in dimension.

On disentanglement issue,   subsequently, inspired by information conservation theorem the clarification on disentanglement in this paper is made.

On real factor issue, due to factor equivalence, the idealistic VAE possibly learns any factor set in the equivalence class.

On indicator issue, the behavior of current disentanglement metric is discussed, and several performance indicators regarding the disentanglement and generating influence are subsequently raised to evaluate the performance of VAE model and to supervise the used factors.

On implementation issue, the experiments under noise modeling and constraints empirically testify the theoretical analysis and also show their own characteristic in pursuing disentanglement.

This paper tries to preliminarily address the disentanglement theoretically in the idealistic situation and practically through noise modelling perspective in the realistic case.

Studies the importance of the noise modelling in Gaussian VAE and proposes to train the noise using Empirical-Bayes like fashion.

Modifying how noise factors are treated when developing VAE models

Preliminary theoretical troubleshooting in Variational Autoencoder


Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization.

Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ.

We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures.

We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization.

Our results provide insight into how to improve the regularization of neural networks.

We investigate weight decay regularization for different optimizers and identify three distinct mechanisms by which weight decay improves generalization.

Discusses the effect of weight decay on the training of deep network models with and without batch normalization and when using first/second order optimization methods and hypothesizes that a larger learning rate has a regularization effect.

Three Mechanisms of Weight Decay Regularization

weight decay
mechanism
regularization
three


In this paper we present the first freely available dataset for the development and evaluation of domain adaptation methods, for the sound event detection task.

The dataset contains 40 log mel-band energies extracted from $100$ different synthetic sound event tracks, with additive noise from nine different acoustic scenes (from indoor, outdoor, and vehicle environments), mixed at six different sound-to-noise ratios, SNRs, (from -12 to -27 dB with a step of -3 dB), and totaling to 5400 (9 * 100 * 6) sound files and a total length of 30 564 minutes.

We provide the dataset as is, the code to re-create the dataset and remix the sound event tracks and the acoustic scenes with different SNRs, and a baseline method that tests the adaptation performance with the proposed dataset and establishes some first results.

The very first freely available domain adaptation dataset for sound event detection.

DASED: A Multi-Domain Dataset for Sound Event Detection Domain Adaptation

dataset sound event detection
domain adaptation


This paper aims to address the limitations of mutual information estimators based on variational optimization.

By redefining the cost using generalized functions from nonextensive statistical mechanics we raise the upper bound of previous estimators and enable the control of the bias variance trade off.

Variational based estimators outperform previous methods especially in high dependence high dimensional scenarios found in machine learning setups.

Despite their performance, these estimators either exhibit a high variance or are upper bounded by log(batch size).

Our approach inspired by nonextensive statistical mechanics uses different generalizations for the logarithm and the exponential in the partition function.

This enables the estimator to capture changes in mutual information over a wider range of dimensions and correlations of the input variables whereas previous estimators saturate them.

Mutual information estimator based nonextensive statistical mechanics

This paper tries to establish novel variational lower bounds for mutual information by introducing parameter q and defining q-algebra, showing that the lower bounds have smaller variance and achieves high values.

Variational lower bounds on mutual information based on nonextensive statistical mechanics

variational low bound mutual information


Generative adversarial networks (GANs) are a widely used framework for learning generative models.

Wasserstein GANs (WGANs), one of the most successful variants of GANs, require solving a minmax problem to global optimality, but in practice, are successfully trained with stochastic gradient descent-ascent.

In this paper, we show that, when the generator is a one-layer network, stochastic gradient descent-ascent converges to a global solution in polynomial time and sample complexity.

We show that stochastic gradient descent ascent converges to a global optimum for WGAN with one-layer generator network.

Attempts to prove that the Stochastic Gradient Decent-Ascent could converge to a global solution for the min-max problem of WGAN.

SGD Learns One-Layer Networks in WGANs

converge global
wgan
stochastic gradient
network
one-layer


Classifiers such as deep neural networks have been shown to be vulnerable against adversarial perturbations on problems with high-dimensional input space.

While adversarial training improves the robustness of classifiers against such adversarial perturbations, it leaves classifiers sensitive to them on a non-negligible fraction of the inputs.

We argue that there are two different kinds of adversarial perturbations: shared perturbations which fool a classifier on many inputs and singular perturbations which only fool the classifier on a small fraction of the data.

We find that adversarial training increases the robustness of classifiers against shared perturbations.

Moreover, it is particularly effective in removing universal perturbations, which can be seen as an extreme form of shared perturbations.

Unfortunately, adversarial training does not consistently increase the robustness against singular perturbations on unseen inputs.

However, we find that adversarial training decreases robustness of the remaining perturbations against image transformations such as changes to contrast and brightness or  Gaussian blurring.

It thus makes successful attacks on the classifier in the physical world less likely.

Finally, we show that even singular perturbations can be easily detected and must thus exhibit generalizable patterns even though the perturbations are specific for certain inputs.

We empirically show that adversarial training is effective for removing universal perturbations, makes adversarial examples less robust to image transformations, and leaves them detectable for a detection approach.

Analyses adversarial training and its effect on universal adversarial examples as well as standard (basic iteration) adversarial examples and how adversarial training affects detection.

The authors show that adversarial training is effective in protecting against "shared" adversarial perturbation, in particular against universal perturbation, but less effective to protect against singular perturbations.

Universality, Robustness, and Detectability of Adversarial Perturbations under Adversarial Training

show adversarial training effective
universal perturbation
detection
example
less


We address the challenging problem of efficient deep learning model deployment, where the goal is to design neural network architectures that can fit different hardware platform constraints.

Most of the traditional approaches either manually design or use Neural Architecture Search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally expensive and unscalable.

Our key idea is to decouple model training from architecture search to save the cost.

To this end, we propose to train a once-for-all network (OFA) that supports diverse architectural settings (depth, width, kernel size, and resolution).

Given a deployment scenario, we can then quickly get a specialized sub-network by selecting from the OFA network without additional training.

To prevent interference between many sub-networks during training, we also propose a novel progressive shrinking algorithm, which can train a surprisingly large number of sub-networks ($> 10^{19}$) simultaneously.

Extensive experiments on various hardware platforms (CPU, GPU, mCPU, mGPU, FPGA accelerator) show that OFA consistently outperforms SOTA NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3) while reducing orders of magnitude GPU hours and $CO_2$ emission.

In particular, OFA achieves a new SOTA 80.0% ImageNet top1 accuracy under the mobile setting ($<$600M FLOPs).

Code and pre-trained models are released at https://github.com/mit-han-lab/once-for-all.

We introduce techniques to train a single once-for-all network that fits many hardware platforms.

Method results in a network from which one can extract sub-networks for various resouce constraints (latency, memory) which perform well without a need for retraining.

This paper tries to tackle the problem of searching best architectures for specialized resource constraint deployment scenarios with a prediction based NAS method.

Once for All: Train One Network and Specialize it for Efficient Deployment

train network method constraint one deployment


A deep generative model is a powerful method of learning a data distribution, which has achieved tremendous success in numerous scenarios.

However, it is nontrivial for a single generative model to faithfully capture the distributions of the complex data such as images with complicate structures.

In this paper, we propose a novel approach of cascaded boosting for boosting generative models, where meta-models (i.e., weak learners) are cascaded together to produce a stronger model.

Any hidden variable meta-model can be leveraged as long as it can support the likelihood evaluation.

We derive a decomposable variational lower bound of the boosted model, which allows each meta-model to be trained separately and greedily.

We can further improve the learning power of the generative models by combing our cascaded boosting framework with the multiplicative boosting framework.

Propose an approach for boosting generative models by cascading hidden variable models

This paper proposed a novel approach of cascaded boosting for boosting generative models which allows each each meta-model to be trained separately and greedily.

Boosting Generative Models by Leveraging Cascaded Meta-Models

boost generative model
approach
cascade
meta-model


Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks.

Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline.

We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena.

We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

We probe for sentence structure in ELMo and related contextual embedding models. We find existing models efficiently encode syntax and show evidence of long-range dependencies, but only offer small improvements on semantic tasks.

Proposes the "edge probing" method and focuses on the relationship between spans rather than individual words, enabling the authors to look at syntactic constituency, dependencies, entity labels, and semantic role labeling.

Provides new insights on what is captured contextualized word embeddings by compiling a set of “edge probing” tasks.

What do you learn from context? Probing for sentence structure in contextualized word representations

probe sentence structure
dependency
" edge
task
semantic
word


Deep reinforcement learning has succeeded in sophisticated games such as Atari, Go, etc.

Real-world decision making, however, often requires reasoning with partial information extracted from complex visual observations.

This paper presents  Discriminative Particle Filter Reinforcement Learning (DPFRL), a new reinforcement learning framework for partial and complex observations.

DPFRL encodes a differentiable particle filter with learned transition and observation models in a neural network, which allows for reasoning with partial observations over multiple time steps.

While a standard particle filter relies on a generative observation model, DPFRL learns a discriminatively parameterized model that is training directly for decision making.

We show that the discriminative parameterization results in significantly improved performance, especially for tasks with complex visual observations, because it circumvents the difficulty of modelling observations explicitly.

In most cases, DPFRL outperforms state-of-the-art POMDP RL models in Flickering Atari Games, an existing POMDP RL benchmark, and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark that we introduce.

We further show that DPFRL performs well for visual navigation with real-world data.

We introduce DPFRL, a framework for reinforcement learning under partial and complex observations with a fully differentiable discriminative particle filter

Introduces ideas for training DLR agents with latent state variables, modeled as a belief distribution, so they can handle partially observed environments.

This paper introduces a principled method for POMDP RL: Discriminative Particle Filter Reinforcement Learning that allows for reasoning with partial observations over multiple time steps, achieving state-of-the-art on benchmarks.

Discriminative Particle Filter Reinforcement Learning for Complex Partial observations

discriminative particle filter reinforcement learning


Extending models with auxiliary latent variables is a well-known technique to in-crease model expressivity.

Bachman & Precup (2015); Naesseth et al. (2018); Cremer et al. (2017); Domke & Sheldon (2018) show that Importance Weighted Autoencoders (IWAE) (Burda et al., 2015) can be viewed as extending the variational family with auxiliary latent variables.

Similarly, we show that this view encompasses many of the recent developments in variational bounds (Maddisonet al., 2017; Naesseth et al., 2018; Le et al., 2017; Yin & Zhou, 2018; Molchanovet al., 2018; Sobolev & Vetrov, 2018).

The success of enriching the variational family with auxiliary latent variables motivates applying the same techniques to the generative model.

We develop a generative model analogous to the IWAE bound and empirically show that it outperforms the recently proposed Learned Accept/Reject Sampling algorithm (Bauer & Mnih, 2018), while being substantially easier to implement.

Furthermore, we show that this generative process provides new insights on ranking Noise Contrastive Estimation (Jozefowicz et al.,2016; Ma & Collins, 2018) and Contrastive Predictive Coding (Oord et al., 2018).

Monte Carlo Objectives are analyzed using auxiliary variable variational inference, yielding a new analysis of CPC and NCE as well as a new generative model.

Proposes a different view on improving variational bounds with auxiliary latent variable models and explores the use of those models in the generative model.

Revisiting Auxiliary Latent Variables in Generative Models

auxiliary latent variable
generative model
variational


Stochastic Gradient Descent or SGD is the most popular optimization algorithm for large-scale problems.

SGD estimates the gradient by uniform sampling with sample size one.

There have been several other works that suggest faster epoch wise convergence by using weighted non-uniform sampling for better gradient estimates.

Unfortunately, the per-iteration cost of maintaining this adaptive distribution for gradient estimation is more than calculating the full gradient.

As a result, the false impression of faster convergence in iterations leads to slower convergence in time, which we call a chicken-and-egg loop.

In this paper, we break this barrier by providing the first demonstration of a sampling scheme, which leads to superior gradient estimation, while keeping the sampling cost per iteration similar to that of the uniform sampling.

Such an algorithm is possible due to the sampling view of Locality Sensitive Hashing (LSH), which came to light recently.

As a consequence of superior and fast estimation, we reduce the running time of all existing gradient descent algorithms.

We demonstrate the benefits of our proposal on both SGD and AdaGrad.

We improve the running of all existing gradient descent algorithms.

Authors propose sampling stochastic gradients from a monotonic function proportional to gradient magnitudes by using LSH.

Considers SGD over an objective of the form of a sum over examples of a quadratic loss.

LSH-SAMPLING BREAKS THE COMPUTATIONAL CHICKEN-AND-EGG LOOP IN ADAPTIVE STOCHASTIC GRADIENT ESTIMATION


In recent years we have made significant progress identifying computational principles that underlie neural function.

While not yet complete, we have sufficient evidence that a synthesis of these ideas could result in an understanding of how neural computation emerges from a combination of innate dynamics and plasticity, and which could potentially be used to construct new AI technologies with unique capabilities.

I discuss the relevant principles, the advantages they have for computation, and how they can benefit AI.

Limitations of current AI are generally recognized, but fewer people are aware that we understand enough about the brain to immediately offer novel AI formulations.

Convolutionary, Evolutionary, Revolutionary: What’s next for Bodies, Brains and AI?


Recent work has demonstrated how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments.

We propose question-answering as a general paradigm to decode and understand the representations that such agents develop, applying our method to two recent approaches to predictive modeling – action-conditional CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019).

After training agents with these predictive objectives in a visually-rich, 3D environment with an assortment of objects, colors, shapes, and spatial configurations, we probe their internal state representations with a host of synthetic (English) questions, without backpropagating gradients from the question-answering decoder into the agent.

The performance of different agents when probed in this way reveals that they learn to encode detailed, and seemingly compositional, information about objects, properties and spatial relations from their physical environment.

Our approach is intuitive, i.e. humans can easily interpret the responses of the model as opposed to inspecting continuous vectors, and model-agnostic, i.e. applicable to any modeling approach.

By revealing the implicit knowledge of objects, quantities, properties and relations acquired by agents as they learn, question-conditional agent probing can stimulate the design and development of stronger predictive learning objectives.

We use question-answering to evaluate how much knowledge about the environment can agents learn by self-supervised prediction.

Proposes QA as a tool to investigate what agents learn about in the world, arguing this as an intuitive method for humans which allows for arbitrary complexity.

The authors propose a framework to assess representations built by predictive models that contain sufficient information to answer questions about the environment they are trained on, showing those by SimCore contained sufficient information for the LSTM to answer questions accurately.

Probing Emergent Semantics in Predictive Agents via Question Answering

environment
agent learn
predictive
question
answer