Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood.

In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations.

First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable.

Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients.

Finally, we present anchored simGD, a new method, and show convergence with stochastic subgradients.

Convergence proof of stochastic sub-gradients method and variations on convex-concave minimax problems

An anaysis of simultaneous stochastic subgradient, simultaneous gradient with optimism, and simultaneous gradient with anchoring in the context of minmax convex concave games.

This paper analyzes the dynamics of stochastic gradient descent when applied to convex-concave games, as well as GD with optimism and a new anchored GD algorithm that converges under weaker assumptions than SGD or SGD with optimism.

ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring  for Minimax Problems and GANs

optimism
stochastic gradient
method
convex-concave
minimax problem
anchor
game


Small spacecraft now have precise attitude control systems available commercially, allowing them to slew in 3 degrees of freedom, and capture images within short notice.

When combined with appropriate software, this agility can significantly increase response rate, revisit time and coverage.

In prior work, we have demonstrated an algorithmic framework that combines orbital mechanics, attitude control and scheduling optimization to plan the time-varying, full-body orientation of agile, small spacecraft in a constellation.

The proposed schedule optimization would run at the ground station autonomously, and the resultant schedules uplinked to the spacecraft for execution.

The algorithm is generalizable over small steerable spacecraft, control capability, sensor specs, imaging requirements, and regions of interest.

In this article, we modify the algorithm to run onboard small spacecraft, such that the constellation can make time-sensitive decisions to slew and capture images autonomously, without ground control.

We have developed a communication module based on Delay/Disruption Tolerant Networking (DTN) for onboard data management and routing among the satellites, which will work in conjunction with the other modules to optimize the schedule of agile communication and steering.

We then apply this preliminary framework on representative constellations to simulate targeted measurements of episodic precipitation events and subsequent urban floods.

The command and control efficiency of our agile algorithm is compared to non-agile (11.3x improvement) and non-DTN (21% improvement) constellations.

We propose an algorithmic framework to schedule constellations of small spacecraft with 3-DOF re-orientation capabilities, networked with inter-sat links.

This paper proposes a communication module to optimize the schedule of communication for the problem of spacecraft constellations, and compares the algorithm in distributed and centralized settings.

Autonomous Scheduling of Agile Spacecraft Constellations with Delay Tolerant Networking for Reactive Imaging


Importance sampling (IS) is a standard Monte Carlo (MC) tool to compute information about random variables such as moments or quantiles with unknown distributions.

IS is
asymptotically consistent as the number of MC samples, and hence deltas (particles) that parameterize the density estimate, go to infinity.

However, retaining infinitely many particles is intractable.

We propose a scheme for only keeping a \emph{finite representative subset} of particles and their augmented importance weights that is \emph{nearly consistent}. To do so in {an online manner}, we approximate importance sampling in two ways.

First, we replace the deltas by kernels, yielding kernel density estimates (KDEs).

Second, we sequentially project KDEs onto nearby lower-dimensional subspaces.

We characterize the asymptotic bias of this scheme as determined by a compression parameter and kernel bandwidth, which yields a tunable tradeoff between consistency and memory.

In experiments,  we observe a favorable tradeoff between memory and accuracy, providing for the first time near-consistent compressions of arbitrary posterior distributions.

We proposed a novel compressed kernelized importance sampling algorithm.

A Projection Operator to Balance Consistency and Complexity in Importance Sampling


We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator?

(2) how to correctly use cross-validation to choose the regularization parameter?

and (3) how to accelerate computation without losing too much accuracy?

We consider the three problems in a unified large-data linear model.

We give a precise representation of ridge regression as a covariance matrix-dependent linear combination of the true parameter and the noise.

We study the bias of $K$-fold cross-validation for choosing the regularization parameter, and propose a simple bias-correction.

We analyze the accuracy of primal and dual sketching for ridge regression, showing they are surprisingly accurate.

Our results are illustrated by simulations and by analyzing empirical data.

We study the structure of ridge regression in a high-dimensional asymptotic framework, and get insights about cross-validation and sketching.

A theoretical study of ridge regression by exploiting a new asymptotic characterisation of the ridge regression estimator.

Ridge Regression: Structure, Cross-Validation, and Sketching

ridge regression
structure
cross-validation
study
asymptotic


Attention mechanisms have advanced the state of the art in several machine learning tasks.

Despite significant empirical gains, there is a lack of theoretical analyses on understanding their effectiveness.

In this paper, we address this problem by studying the landscape of population and empirical loss functions of attention-based neural networks.

Our results show that, under mild assumptions, every local minimum of a two-layer global attention model has low prediction error, and attention models require lower sample complexity than models not employing attention.

We then extend our analyses to the popular self-attention model, proving that they deliver consistent predictions with a more expressive class of functions.

Additionally, our theoretical results provide several guidelines for designing attention mechanisms.

Our findings are validated with satisfactory experimental results on MNIST and IMDB reviews dataset.

We analyze the loss landscape of neural networks with attention and explain why attention is helpful in training neural networks to achieve good performance.

This paper proves from the theoretical perspective that attention networks can generalize better than non-attention baselines for fixed-attention (single-layer and multi-layer) and self-attention in the single layer setting.

Understanding Attention Mechanisms


Recent advances in deep learning techniques has shown the usefulness of the deep neural networks in extracting features required to perform the task at hand.

However, these features learnt are in particular helpful only for the initial task.

This is due to the fact that the features learnt are very task specific and does not capture the most general and task agnostic features of the input.

In fact the way humans are seen to learn is by disentangling features which task agnostic.

This indicates that leaning task agnostic features by disentangling only the most informative features from the input data.

Recently Variational Auto-Encoders (VAEs) have shown to be the de-facto models to capture the latent variables in a generative sense.

As these latent features can be represented as continuous and/or discrete variables, this indicates us to use VAE with a mixture of continuous and discrete variables for the latent space.

We achieve this by performing our experiments using a modified version of joint-vae to learn the disentangled features.

Mixture Model for Neural Disentanglement

Neural Disentanglement using Mixture Latent Space with Continuous and Discrete Variables


To improve how neural networks function it is crucial to understand their learning process.

The information bottleneck theory of deep learning proposes that neural networks achieve good generalization by compressing their representations to disregard information that is not relevant to the task.

However, empirical evidence for this theory is conflicting, as compression was only observed when networks used saturating activation functions.

In contrast, networks with non-saturating activation functions achieved comparable levels of task performance but did not show compression.

In this paper we developed more robust mutual information estimation techniques, that adapt to hidden activity of neural networks and produce more sensitive measurements of activations from all functions, especially unbounded functions.

Using these adaptive estimation techniques, we explored compression in networks with a range of different activation functions.

With two improved methods of estimation, firstly, we show that saturation of the activation function is not required for compression, and the amount of compression varies between different activation functions.

We also find that there is a large amount of variation in compression between different network initializations.

Secondary, we see that L2 regularization leads to significantly increased compression, while preventing overfitting.

Finally, we show that only compression of the last layer is positively correlated with generalization.

We developed robust mutual information estimates for DNNs and used them to observe compression in networks with non-saturating activation functions

This paper studied the popular belief that deep neural networks do information compression for supervised tasks

This paper proposes a method for the estimation of mutual information for networks with unbounded activation functions and the use of L2 regularization to induce more compression.

Adaptive Estimators Show Information Compression in Deep Neural Networks

deep neural network
information compression
activation function
mutual


In this work, we address the problem of musical timbre transfer, where the goal is to manipulate the timbre of a sound sample from one instrument to match another instrument while preserving other musical content, such as pitch, rhythm, and loudness.

In principle, one could apply image-based style transfer techniques to a time-frequency representation of an audio signal, but this depends on having a representation that allows independent manipulation of timbre as well as high-quality waveform generation.

We introduce TimbreTron, a method for musical timbre transfer which applies “image” domain style transfer to a time-frequency representation of the audio signal, and then produces a high-quality waveform using a conditional WaveNet synthesizer.

We show that the Constant Q Transform (CQT) representation is particularly well-suited to convolutional architectures due to its approximate pitch equivariance.

Based on human perceptual evaluations, we confirmed that TimbreTron recognizably transferred the timbre while otherwise preserving the musical content, for both monophonic and polyphonic samples.

We made an accompanying demo video here: https://www.cs.toronto.edu/~huang/TimbreTron/index.html which we strongly encourage you to watch before reading the paper.

We present the TimbreTron, a pipeline for perfoming high-quality timbre transfer on musical waveforms using CQT-domain style transfer.

A method for converting recordings of a specific musical instrument to another by applying CycleGAN, developed for image style transfer, to transfer spectrograms.

The authors use multiple techniques/tools to enable neural timbre transfer (converting music from one instrument to another) without paired training examples.

Describes a model for musical timbre transfer with the results indicating that the proposed system is effective for pitch and tempo transfer, as well as timbre adaptation.

TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer

pipeline
instrument another
musical timbre transfer
timbretron
convert
style


Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them.

But also generic hardware and software implementations of deep learning run more efficiently for sparse networks.

Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints.

We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network.

DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded.

We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance.

DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.

The paper presents Deep Rewiring, an algorithm that can be used to train deep neural networks when the network connectivity is severely constrained during training.

An approach to implement deep learning directly on sparsely connected graphs, allowing networks to be trained efficiently online and for fast and flexible learning.

The authors provide a simple algorithm capable of training with limited memory

Deep Rewiring: Training very sparse deep networks

network
training
deep rewiring
train
algorithm


Deep learning's success has led to larger and larger models to handle more and more complex tasks; trained models can contain millions of parameters.

These large models are compute- and memory-intensive, which makes it a challenge to deploy them with minimized latency, throughput, and storage requirements.

Some model compression methods have been successfully applied on image classification and detection or language models, but there has been very little work compressing generative adversarial networks (GANs) performing complex tasks.

In this paper, we show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods.

We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator.

We show that this framework has a compelling performance to high degrees of sparsity, generalizes well to new tasks and models, and enables meaningful comparisons between different pruning granularities.

Existing pruning methods fail when applied to GANs tackling complex tasks, so we present a simple and robust method to prune generators that works well for a wide variety of networks and tasks.

The authors propose a modification to the classic distillation method for the task of compressing a network to address the failure of previous solutions when applied to generative adversarial networks.

Self-Supervised GAN Compression


Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure.

The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections.

In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth.

To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training.

We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus.

On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270x to 600x without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB.

Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile.

we find 99.9% of the gradient exchange in distributed SGD is redundant; we reduce the communication bandwidth by two orders of magnitude without losing accuracy.

This paper proposes additional improvement over gradient dropping to improve communication efficiency

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

reduce communication bandwidth
distribute
gradient


Image-to-image translation has recently received significant attention due to advances in deep learning.

Most works focus on learning either a one-to-one mapping in an unsupervised way or a many-to-many mapping in a supervised way.

However, a more practical setting is many-to-many mapping in an unsupervised way, which is harder due to the lack of supervision and the complex inner- and cross-domain variations.

To alleviate these issues, we propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain.

We assume that an image comprises of a content component which is shared across domains, and a style component specific to each domain.

Under the guidance of an exemplar from the target domain we apply Adaptive Instance Normalization to the shared content component, which allows us to transfer the style information of the target domain to the source domain.

To avoid semantic inconsistencies during translation that naturally appear due to the large inner- and cross-domain variations, we introduce the concept of feature masks that provide coarse semantic guidance without requiring the use of any semantic labels.

Experimental results on various datasets show that EGSC-IT does not only translate the source image to diverse instances in the target domain, but also preserves the semantic consistency during the process.

We propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain.

Discusses a core failing and need for I2I translation models.

The paper explores the idea that an image has two components and applies an attention model where the feature masks that steer the translation process do not require semantic labels

Exemplar Guided Unsupervised Image-to-Image Translation with Semantic Consistency

translation process
image
exemplar guide
model
image-to-image
semantic


Deep neural networks can learn meaningful representations of data.

However, these representations are hard to interpret.

For example, visualizing a latent layer is generally only possible for at most three dimensions.

Neural networks are able to learn and benefit from much higher dimensional representations but these are not visually interpretable because nodes have arbitrary ordering within a layer.

Here, we utilize the ability of the human observer to identify patterns in structured representations to visualize higher dimensions.

To do so, we propose a class of regularizations we call \textit{Graph Spectral Regularizations} that impose graph-structure on latent layers.

This is achieved by treating activations as signals on a predefined graph and constraining those activations using graph filters, such as low pass and wavelet-like filters.

This framework allows for any kind of graph as well as filter to achieve a wide range of structured regularizations depending on the inference needs of the data.

First, we show a synthetic example that the graph-structured layer can reveal topological features of the data.

Next, we show that a smoothing regularization can impose semantically consistent ordering of nodes when applied to capsule nets.

Further, we show that the graph-structured layer, using wavelet-like spatially localized filters, can form localized receptive fields for improved image and biomedical data interpretation.

In other words, the mapping between latent layer, neurons and the output space becomes clear due to the localization of the activations.

Finally, we show that when structured as a grid, the representations create coherent images that allow for image-processing techniques such as convolutions.

Imposing graph structure on neural network layers for improved visual interpretability.

A novel regularizer to impose graph structure upon hidden layers of a Neural Network to improve the interpretability of hidden representations.

Highlights the contribution of graph spectral regularizer to the interpretability of neural networks.

Graph Spectral Regularization For Neural Network Interpretability

impose graph structure
neural network
interpretability
layer
improve
regularizer
spectral


Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation.

The dominant parametric approach is based on locally normalized models which predict one word at a time.

While these work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process.

In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level.

In order to make training tractable, we first work in the residual of a pretrained locally normalized language model and second we train using noise contrastive estimation.

Furthermore, since the EBM works at the sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa.

Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity compared to locally normalized baselines.

Moreover, generation via importance sampling is very efficient and of higher quality than the baseline models according to human evaluation.

We show that Energy-Based models when trained on the residual of an auto-regressive language model can be used effectively and efficiently to generate text.

A proposed Residual Energy-based Model (EBM) for text generation which operates at the sentence level, and can therefore leverage BERT, and achieves lower perplexity and is preferred by human evaluation.

Residual Energy-Based Models for Text Generation

residual energy-based model
text generation


We investigate the robustness properties of image recognition models equipped with two features inspired by human vision, an explicit episodic memory and a shape bias, at the ImageNet scale.

As reported in previous work, we show that an explicit episodic memory improves the robustness of image recognition models against small-norm adversarial perturbations under some threat models.

It does not, however, improve the robustness against more natural, and typically larger, perturbations.

Learning more robust features during training appears to be necessary for robustness in this second sense.

We show that features derived from a model that was encouraged to learn global, shape-based representations (Geirhos et al., 2019) do not only improve the robustness against natural perturbations, but when used in conjunction with an episodic memory, they also provide additional robustness against adversarial perturbations.

Finally, we address three important design choices for the episodic memory: memory size, dimensionality of the memories and the retrieval method.

We show that to make the episodic memory more compact, it is preferable to reduce the number of memories by clustering them, instead of reducing their dimensionality.

systematic study of large-scale cache-based image recognition models, focusing particularly on their robustness properties

This paper proposed to use memory cache to improve robustness against adversarial image examples, and concluded that using a large continous cache is not superior to hard attention.

Improving the robustness of ImageNet classifiers using elements of human visual cognition


Group convolutional neural networks (G-CNNs) can be used to improve classical CNNs by equipping them with the geometric structure of groups.

Central in the success of G-CNNs is the lifting of feature maps to higher dimensional disentangled representations, in which data characteristics are effectively learned, geometric data-augmentations are made obsolete, and predictable behavior under geometric transformations (equivariance) is guaranteed via group theory.

Currently, however, the practical implementations of G-CNNs are limited to either discrete groups (that leave the grid intact) or continuous compact groups such as rotations (that enable the use of Fourier theory).

In this paper we lift these limitations and propose a modular framework for the design and implementation of G-CNNs for arbitrary Lie groups.

In our approach the differential structure of Lie groups is used to expand convolution kernels in a generic basis of B-splines that is defined on the Lie algebra.

This leads to a flexible framework that enables localized, atrous, and deformable convolutions in G-CNNs by means of respectively localized, sparse and non-uniform B-spline expansions.

The impact and potential of our approach is studied on two benchmark datasets: cancer detection in histopathology slides (PCam dataset) in which rotation equivariance plays a key role and facial landmark localization (CelebA dataset) in which scale equivariance is important.

In both cases, G-CNN architectures outperform their classical 2D counterparts and the added value of atrous and localized group convolutions is studied in detail.

The paper describes a flexible framework for building CNNs that are equivariant to a large class of transformations groups.

A framework for building group CNN with an arbitrary Lie group G, which shows superiority over a CNN in tumor classification and landmark localization.

B-Spline CNNs on Lie groups


Global feature pooling is a modern variant of feature pooling providing better interpretatability and regularization.

Although alternative pooling methods exist (eg. max, lp norm, stochastic), the averaging operation is still the dominating global pooling scheme in popular models.

As fine-grained recognition requires learning subtle, discriminative features, we consider the question: is average pooling the optimal strategy?

We first ask: ``is there a difference between features learned by global average and max pooling?'' Visualization and quantitative analysis show that max pooling encourages learning features of different spatial scales.

We then ask ``is there a single global feature pooling variant that's most suitable for fine-grained recognition?'' A thorough evaluation of nine representative pooling algorithms finds that: max pooling outperforms average pooling consistently across models, datasets, and image resolutions; it does so by reducing the generalization gap; and generalized pooling's performance increases almost monotonically as it changes from average to max.

We finally ask: ``what's the best way to combine two heterogeneous pooling schemes?'' Common strategies struggle because of potential gradient conflict but the ``freeze-and-train'' trick works best.

We also find that post-global batch normalization helps with faster convergence and improves model performance consistently.

A benchmark of nine representative global pooling schemes reveals some interesting findings.

For fine-grained classification tasks, this paper validated that maxpooling would encourage sparser feature maps than and outperform avgpooling.

On Global Feature Pooling for Fine-grained Visual Categorization


We present a technique to improve the generalization of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions.

Although recent research has shown benefits of self-supervised learning (SSL) on large unlabeled datasets, its utility on small datasets is unknown.

We find that SSL reduces the relative error rate of few-shot meta-learners by 4%-27%, even when the datasets are small and only utilizing images within the datasets.

The improvements are greater when the training set is smaller or the task is more challenging.

Though the benefits of SSL may increase with larger training sets, we observe that SSL can have a negative impact on performance when there is a domain shift between distribution of images used for meta-learning and SSL.

Based on this analysis we present a technique that automatically select images for SSL from a large, generic pool of unlabeled images for a given dataset using a domain classifier that provides further improvements.

We present results using several meta-learners and self-supervised tasks across datasets with varying degrees of domain shifts and label sizes to characterize the effectiveness of SSL for few-shot learning.

Self-supervision improves few-shot recognition on small and challenging datasets without relying on extra data; Extra data helps only when it is from the same or similar domain.

An empirical study of different self-supervised learning (SSL) methods, showing SSL helps more when the dataset is harder, that domain matters for training, and a method to choose samples from an unlabeled dataset.

When Does Self-supervision Improve Few-shot Learning?

help
self-supervision improve few-shot
domain
dataset
learning


Abstraction of Markov Decision Processes is a useful tool for solving complex problems, as it can ignore unimportant aspects of an environment, simplifying the process of learning an optimal policy.

In this paper, we propose a new algorithm for finding abstract MDPs in environments with continuous state spaces.

It is based on MDP homomorphisms, a structure-preserving mapping between MDPs.

We demonstrate our algorithm's ability to learns abstractions from collected experience and show how to reuse the abstractions to guide exploration in new tasks the agent encounters.

Our novel task transfer method beats a baseline based on a deep Q-network.

We create abstract models of environments from experience and use them to learn new tasks faster.

A methodology that uses the idea of MDP homomorphisms to transform a complex MDP with a continuous state space to a simpler one.

Online abstraction with MDP homomorphisms for Deep Learning


A number of recent methods to understand neural networks have focused on quantifying the role of individual features.

One such method, NetDissect identifies interpretable features of a model using the Broden dataset of visual semantic labels (colors, materials, textures, objects and scenes).

Given the recent rise of a number of action recognition datasets, we propose extending the Broden dataset to include actions to better analyze learned action models.

We describe the annotation process, results from interpreting action recognition models on the extended Broden dataset and examine interpretable feature paths to help us understand the conceptual hierarchy used to classify an action.

We expand Network Dissection to include action interpretation and examine interpretable feature paths to understand the conceptual hierarchy used to classify an action.

Examining Interpretable Feature Relationships in Deep Networks for Action recognition

examine interpretable feature
action
network


Automatic melody generation for pop music has been a long-time aspiration for

However, learning to generate euphonious

melody has turned out to be highly challenging due to a number of factors.

of multivariate property of notes has been one of the primary challenges.

It is also difficult to remain in the permissible spectrum of musical variety, outside

of which would be perceived as a plain random play without auditory pleasantness.

Observing the conventional structure of pop music poses further challenges.

In this paper, we propose to represent each note and its properties as a unique

‘word,’ thus lessening the prospect of misalignments between the properties, as

well as reducing the complexity of learning.

on the range of notes, thus encouraging the generated melody to stay close

to what humans would find easy to follow.

conditioned on song part information, thus replicating the overall structure of a

Experimental results demonstrate that our model can generate auditorily

pleasant songs that are more indistinguishable from human-written ones than

We propose a novel model to represent notes and their properties, which can enhance the automatic melody generation.

This paper proposes a generative model of symbolic (MIDI) melody in western popular music which jointly encodes note symbols with timing and duration information to form musical "words".

The paper proposes to facilitate generation of melody by representing notes as "words", representing all of the note's properties and thus allowing the generation of musical "sentences".

Melody Generation for Pop Music via Word Representation of Musical Properties

represent note
" word "
melody generation
model
property
musical
music


Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts.

We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth.

We propose robust growing and stopping policies to generalize to different network architectures and datasets.

Our experiments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet.

For example, in terms of accuracy-computation trade-off, AutoGrow discovers a better depth combination in ResNets than human experts.

It discovers depth within similar time of training a single DNN.

A method that automatically grows layers in neural networks to discover optimal depth.

A framework to interleave training a shallower network and adding new layers which provides insights into the paradigm of 'growing networks'.

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks


Given the importance of remote sensing, surprisingly little attention has been paid to it by the representation learning community.

To address it and to speed up innovation in this domain, we provide simplified access to 5 diverse remote sensing datasets in a standardized form.

We specifically explore in-domain representation learning and address the question of "what characteristics should a dataset have to be a good source for remote sensing representation learning".

The established baselines achieve state-of-the-art performance on these datasets.

Exploration of in-domain representation learning for remote sensing datasets.

This paper provided several standardized remote sensing data sets and showed that in-domain representation could produce better baseline results for remote sensing compared to fine-tuning on ImageNet or learning from scratch.

In-Domain Representation Learning For Remote Sensing

in-domain representation learning remote


Generative seq2seq dialogue systems are trained to predict the next word in dialogues that have already occurred.

They can learn from large unlabeled conversation datasets, build a deep understanding of conversational context, and generate a wide variety of responses.

This flexibility comes at the cost of control.

Undesirable responses in the training data will be reproduced by the model at inference time, and longer generations often don’t make sense.

Instead of generating responses one word at a time, we train a classifier to choose from a predefined list of full responses.

The classifier is trained on (conversation context, response class) pairs, where each response class is a noisily labeled group of interchangeable responses.

At inference, we generate the exemplar response associated with the predicted response class.

Experts can edit and improve these exemplar responses over time without retraining the classifier or invalidating old training data.

Human evaluation of 775 unseen doctor/patient conversations shows that this tradeoff improves responses.

Only 12% of our discriminative approach’s responses are worse than the doctor’s response in the same conversational context, compared to 18% for the generative model.

A discriminative model trained without any manual labeling of response classes achieves equal performance to the generative model.

Avoid generating responses one word at a time by using weak supervision to training a classifier  to pick a full response.

A way to generate responses for medical dialog using a classifier to select from expert-curated responses based on the conversation context.

Classification as Decoder: Trading Flexibility for Control in Multi Domain Dialogue


There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs).

This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP.

In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels.

We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible.

Surprisingly, in the absence of pooling layers, the GPs corresponding to CNNs with and without weight sharing are identical.

As a consequence, translation equivariance, beneficial in finite channel CNNs trained with stochastic gradient descent (SGD), is guaranteed to play no role in the Bayesian treatment of the infinite channel limit - a qualitative difference between the two regimes that is not present in the FCN case.

We confirm experimentally, that while in some scenarios the performance of SGD-trained finite CNNs approaches that of the corresponding GPs as the channel count increases, with careful tuning SGD-trained CNNs can significantly outperform their corresponding GPs, suggesting advantages from SGD training compared to fully Bayesian parameter estimation.

Finite-width SGD trained CNNs vs. infinitely wide fully Bayesian CNNs. Who wins?

The paper establishes a connection between infinite channel Bayesian convolutional neural network and Gaussian processes.

Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes

gaussian process
network
bayesian
convolutional
channel


Bayesian inference promises to ground and improve the performance of deep neural networks.

It promises to be robust to overfitting, to simplify the training procedure and the space of hyperparameters, and to provide a calibrated measure of uncertainty that can enhance decision making, agent exploration and prediction fairness.

Markov Chain Monte Carlo (MCMC) methods enable Bayesian inference by generating samples from the posterior distribution over model parameters.

Despite the theoretical advantages of Bayesian inference and the similarity between MCMC and optimization methods, the performance of sampling methods has so far lagged behind  optimization methods for large scale deep learning tasks.

We aim to fill this gap and introduce ATMC, an adaptive noise MCMC algorithm that estimates and is able to sample from the posterior of a neural network.

ATMC dynamically adjusts the amount of momentum and noise applied to each parameter update in order to compensate for the use of stochastic gradients.

We use a ResNet architecture without batch normalization to test ATMC on the Cifar10 benchmark and the large scale ImageNet benchmark and show that, despite the  absence of batch normalization, ATMC outperforms a strong optimization baseline in terms of both classification accuracy and test log-likelihood.

We show that ATMC is intrinsically robust to overfitting on the training data and that ATMC provides a better calibrated measure of uncertainty compared to the optimization baseline.

We scale Bayesian Inference to ImageNet classification and achieve competitive results accuracy and uncertainty calibration.

An adaptive noise MCMC algorithm for image classification that dynamically adjusts the momentum and noise applied to each parameter update, and is robust to overfitting and provides an uncertainty measure with predictions.

Bayesian Inference for Large Scale Image Classification

bayesian inference
classification
uncertainty
scale
image


Now GANs can generate more and more realistic face images that can easily fool human beings.

In contrast, a common convolutional neural network(CNN), e.g. ResNet-18, can achieve more than 99.9% accuracy in discerning fake/real faces if training and testing faces are from the same source.

In this paper, we performed both human studies and CNN experiments, which led us to two important findings.

One finding is that the textures of fake faces are substantially different from real ones.

CNNs can capture local image texture information for recognizing fake/real face, while such cues are easily overlooked by humans.

The other finding is that global image texture information is more robust to image editing and generalizable to fake faces from different GANs and datasets.

Based on the above findings, we propose  a  novel  architecture  coined  as  Gram-Net,  which  incorporates  “Gram Block” in multiple semantic levels to extract global image texture representations.

Experimental results demonstrate that our Gram-Net performs better than existing approaches for fake face detection.

Especially, our Gram-Net is more robust to image editing, e.g.  downsampling, JPEG compression, blur, and noise.

More importantly, our Gram-Net generalizes significantly better in detecting fake faces from GAN models not seen in the training phase.

An empirical study on fake images reveals that texture is an important cue that current fake images differ from real images. Our improved model capturing global texture statistics shows better cross-GAN fake image detection performance.

The paper proposes a way to improve model performance for fake face detection in images generated by a GAN to be more generalizable based on texture information.

Real or Fake: An Empirical Study and Improved Model for Fake Face Detection

empirical study
fake face detection
improve model
texture
real
performance
image


The Wasserstein probability metric has received much attention from the machine learning community.

Unlike the Kullback-Leibler divergence, which strictly measures change in probability, the Wasserstein metric reflects the underlying geometry between outcomes.

The value of being sensitive to this geometry has been demonstrated, among others, in ordinal regression and generative modelling, and most recently in reinforcement learning.

In this paper we describe three natural properties of probability divergences that we believe reflect requirements from machine learning: sum invariance, scale sensitivity, and unbiased sample gradients.

The Wasserstein metric possesses the first two properties but, unlike the Kullback-Leibler divergence, does not possess the third.

We provide empirical evidence suggesting this is a serious issue in practice.

Leveraging insights from probabilistic forecasting we propose an alternative to the Wasserstein metric, the Cramér distance.

We show that the Cramér distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences.

We give empirical results on a number of domains comparing these three divergences.

To illustrate the practical relevance of the Cramér distance we design a new algorithm, the Cramér Generative Adversarial Network (GAN), and show that it has a number of desirable properties over the related Wasserstein GAN.

The Wasserstein distance is hard to minimize with stochastic gradient descent, while the Cramer distance can be optimized easily and works just as well.

The manuscript proposes to use the Cramer distance to act as a loss when optimizing an objective function using stochastic gradient descent because it has unbiased sample gradients.

The contribution of the article is related to performance criteria, in particular to the Wasserstein/Mallows metric

The Cramer Distance as a Solution to Biased Wasserstein Gradients

cramer distance
stochastic gradient descent
optimize
wasserstein


We humans have an innate understanding of the asymmetric progression of time, which we use to efficiently and safely perceive and manipulate our environment.

Drawing inspiration from that, we approach the problem of learning an arrow of time in a Markov (Decision) Process.

We illustrate how a learned arrow of time can capture salient information about the environment, which in turn can be used to measure reachability, detect side-effects and to obtain an intrinsic reward signal.

Finally, we propose a simple yet effective algorithm to parameterize the problem at hand and learn an arrow of time with a function approximator (here, a deep neural network).

Our empirical results span a selection of discrete and continuous environments, and demonstrate for a class of stochastic processes that the learned arrow of time agrees reasonably well with a well known notion of an arrow of time due to Jordan, Kinderlehrer and Otto (1998).

We learn the arrow of time for MDPs and use it to measure reachability, detect side-effects and obtain a curiosity reward signal.

This work proposes the h-potential as a solution to an objective that measures state-transition asymmetry in an MDP.

Learning the Arrow of Time for Problems in Reinforcement Learning


We formulate stochastic gradient descent (SGD) as a novel factorised Bayesian filtering problem, in which each parameter is inferred separately, conditioned on the corresopnding backpropagated gradient.

Inference in this setting naturally gives rise to BRMSprop and BAdam: Bayesian variants of RMSprop and Adam.

Remarkably, the Bayesian approach recovers many features of state-of-the-art adaptive SGD methods, including amongst others root-mean-square normalization, Nesterov acceleration and AdamW.  As such, the Bayesian approach provides one explanation for the empirical effectiveness of state-of-the-art adaptive SGD algorithms.

Empirically comparing BRMSprop and BAdam with naive RMSprop and Adam on MNIST, we find that Bayesian methods have the potential to considerably reduce test loss and classification error.

We formulated SGD as a Bayesian filtering problem, and show that this gives rise to RMSprop, Adam, AdamW, NAG and other features of state-of-the-art adaptive methods

The paper analyzes stochastic gradient descent through Bayesian filtering as a framework for analyzing adaptive methods.

The authors attempt to unify existing adaptive gradient methods under the Bayesian filtering framework with the dynamical prior

A unified theory of adaptive stochastic gradient descent as Bayesian filtering

bayesian filtering framework
stochastic gradient descent
adaptive method


Data augmentation (DA) has been widely utilized to improve generalization in training deep neural networks.

Recently, human-designed data augmentation has been gradually replaced by automatically learned augmentation policy.

Through finding the best policy in well-designed search space of data augmentation, AutoAugment (Cubuk et al., 2019) can significantly improve validation accuracy on image classification tasks.

However, this approach is not computationally practical for large-scale problems.

In this paper, we develop an adversarial method to arrive at a computationally-affordable solution called Adversarial AutoAugment, which can simultaneously optimize target related object and augmentation policy search loss.

The augmentation policy network attempts to increase the training loss of a target network through generating adversarial augmentation policies, while the target network can learn more robust features from harder examples to improve the generalization.

In contrast to prior work, we reuse the computation in target network training for policy evaluation, and dispense with the retraining of the target network.

Compared to AutoAugment, this leads to about 12x reduction in computing cost and 11x shortening in time overhead on ImageNet.

We show experimental results of our approach on CIFAR-10/CIFAR-100, ImageNet, and demonstrate significant performance improvements over state-of-the-art.

On CIFAR-10, we achieve a top-1 test error of 1.36%, which is the currently best performing single model.

On ImageNet, we achieve a leading performance of top-1 accuracy 79.40% on ResNet-50 and 80.00% on ResNet-50-D without extra data.

We introduce the idea of adversarial learning into automatic data augmentation to improve the generalization  of a targe network.

A technique called Adversarial AutoAugment which dynamically learns good data augmentation policies during training using an adversarial approach.

Adversarial AutoAugment

datum augmentation
adversarial
autoaugment


In this study we focus on first-order meta-learning algorithms that aim to learn a parameter initialization of a network which can quickly adapt to new concepts, given a few examples.

We investigate two approaches to enhance generalization and speed of learning of such algorithms, particularly expanding on the Reptile (Nichol et al., 2018) algorithm.

We introduce a novel regularization technique called meta-step gradient pruning and also investigate the effects of increasing the depth of network architectures in first-order meta-learning.

We present an empirical evaluation of both approaches, where we match benchmark few-shot image classification results with 10 times fewer iterations using Mini-ImageNet dataset and with the use of deeper networks, we attain accuracies that surpass the current benchmarks of few-shot image classification using Omniglot dataset.

The study introduces two approaches to enhance generalization of first-order meta-learning and presents empirical evaluation on few-shot image classification.

The paper presents an empirical study of the first-order meta-learning Reptile algorithm, investigating a proposed regularization technique and deeper networks

Enhancing Generalization of First-Order Meta-Learning

first-order meta-learning
enhance generalization
present empirical
study


In this paper, we propose the use of in-training matrix factorization to reduce the model size for neural machine translation.

Using in-training matrix factorization, parameter matrices may be decomposed into the products of smaller matrices, which can compress large machine translation architectures by vastly reducing the number of learnable parameters.

We apply in-training matrix factorization to different layers of standard neural architectures and show that in-training factorization is capable of reducing nearly 50% of learnable parameters without any associated loss in BLEU score.

Further, we find that in-training matrix factorization is especially powerful on embedding layers, providing a simple and effective method to curtail the number of parameters with minimal impact on model performance, and, at times, an increase in performance.

This paper proposes using matrix factorization at training time for neural machine translation, which can reduce model size and decrease training time without harming performance.

This paper proposes to compress models using matrix factorization during training for deep neural networks of machine translation.

In-training Matrix Factorization for Parameter-frugal Neural Machine Translation

matrix factorization training
machine translation
neural
model


Though state-of-the-art sentence representation models can perform tasks requiring significant knowledge of grammar, it is an open question how best to evaluate their grammatical knowledge.

We explore five experimental methods inspired by prior work evaluating pretrained sentence representation models.

We use a single linguistic phenomenon, negative polarity item (NPI) licensing, as a case study for our experiments.

NPIs like 'any' are grammatical only if they appear in a licensing environment like negation ('Sue doesn't have any cats' vs. '*Sue has any cats').

This phenomenon is challenging because of the variety of NPI licensing environments that exist.

We introduce an artificially generated dataset that manipulates key features of NPI licensing for the experiments.

We find that BERT has significant knowledge of these features, but its success varies widely across different experimental methods.

We conclude that a variety of methods is necessary to reveal all relevant aspects of a model's grammatical knowledge in a given domain.

Different methods for analyzing BERT suggest different (but compatible) conclusions in a case study on NPIs.

Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs


The primate visual system builds robust, multi-purpose representations of the external world in order to support several diverse downstream cortical processes.

Such representations are required to be invariant to the sensory inconsistencies caused by dynamically varying lighting, local texture distortion, etc.

A key architectural feature combating such environmental irregularities is ‘long-range horizontal connections’ that aid the perception of the global form of objects.

In this work, we explore the introduction of such horizontal connections into standard deep convolutional networks; we present V1Net -- a novel convolutional-recurrent unit that models linear and nonlinear horizontal inhibitory and excitatory connections inspired by primate visual cortical connectivity.

We introduce the Texturized Challenge -- a new benchmark to evaluate object recognition performance under perceptual noise -- which we use to evaluate V1Net against an array of carefully selected control models with/without recurrent processing.

Additionally, we present results from an ablation study of V1Net demonstrating the utility of diverse neurally inspired horizontal connections for state-of-the-art AI systems on the task of object boundary detection from natural images.

We also present the emergence of several biologically plausible horizontal connectivity patterns, namely center-on surround-off, association fields and border-ownership connectivity patterns in a V1Net model trained to perform boundary detection on natural images from the Berkeley Segmentation Dataset 500 (BSDS500).

Our findings suggest an increased representational similarity between V1Net and biological visual systems, and highlight the importance of neurally inspired recurrent contextual processing principles for learning visual representations that are robust to perceptual noise and furthering the state-of-the-art in computer vision.

In this work, we present V1Net -- a novel recurrent neural network modeling cortical horizontal connections that give rise to robust visual representations through perceptual grouping.

The authors propose to modify a convolutional variant of LSTM to include horizontal connections inspired by known interactions in visual cortex.

V1Net: A computational model of cortical horizontal connections

model cortical horizontal connection
visual
v1net


Humans understand novel sentences by composing meanings and roles of core language components.

In contrast, neural network models for natural language modeling fail when such compositional generalization is required.

The main contribution of this paper is to hypothesize that language compositionality is a form of group-equivariance.

Based on this hypothesis, we propose a set of tools for constructing equivariant sequence-to-sequence models.

Throughout a variety of experiments on the SCAN tasks, we analyze the behavior of existing models under the lens of equivariance, and demonstrate that our equivariant architecture is able to achieve the type compositional generalization required in human language understanding.

We propose a link between permutation equivariance and compositional generalization, and provide equivariant language models

This work focuses on learning locally equivariant representations and functions over input/output words for the purposes of SCAN task.

Permutation Equivariant Models for Compositional Generalization in Language

compositional generalization
model
equivariant
language
permutation


Variational inference (VI) is a popular approach for approximate Bayesian inference that is particularly promising for highly parameterized models such as deep neural networks.

A key challenge of variational inference is to approximate the posterior over model parameters with a distribution that is simpler and tractable yet sufficiently expressive.

In this work, we propose a method for training highly flexible variational distributions by starting with a coarse approximation and iteratively refining it.

Each refinement step makes cheap, local adjustments and only requires optimization of simple variational families.

We demonstrate theoretically that our method always improves a bound on the approximation (the Evidence Lower BOund) and observe this empirically across a variety of benchmark tasks.

In experiments, our method consistently outperforms recent variational inference methods for deep learning in terms of log-likelihood and the ELBO.

We see that the gains are further amplified on larger scale models, significantly outperforming standard VI and deep ensembles on residual networks on CIFAR10.

The paper proposes an algorithm to increase the flexibility of the variational posterior in Bayesian neural networks through iterative optimization.

A method for training flexible variational posterior distributions, applied to Bayesian neural nets to perform variation inference (VI) over the weights.

Refining the variational posterior through iterative optimization

variational posterior
iterative optimization
bayesian neural


In this paper, we propose a residual non-local attention network for high-quality image restoration.

Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial- and channel-wise features.

To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts.

Specifically, we design trunk branch and (non-)local mask branch in each (non-)local attention block.

The trunk branch is used to extract hierarchical features.

Local and non-local mask branches aim to adaptively rescale these hierarchical features with mixed attentions.

The local mask branch concentrates on more local structures with convolutional operations, while non-local attention considers more about long-range dependencies in the whole feature map.

Furthermore, we propose residual local and non-local attention learning to train the very deep network, which further enhance the representation ability of the network.

Our proposed method can be generalized for various image restoration applications, such as image denoising, demosaicing, compression artifacts reduction, and super-resolution.

Experiments demonstrate that our method obtains comparable or better results compared with recently leading methods quantitatively and visually.

New state-of-the-art framework for image restoration

The paper proposes a convolutional neural network architecture that includes blocks for local and non-local attention mechanisms, which are claimed to be responsible for achieving excellent results in four image restoration applications.

This paper proposes a residual non-local attention network for image restoration

Residual Non-local Attention Networks for Image Restoration

residual non-local attention network image restoration


Most approaches to learning action planning models heavily rely on a significantly large volume of training samples or plan observations.

In this paper, we adopt a different approach based on deductive learning from domain-specific knowledge, specifically from logic formulae that specify constraints about the possible states of a given domain.

The minimal input observability required by our approach is a single example composed of a full initial state and a partial goal state.

We will show that exploiting specific domain knowledge enables to constrain the space of possible action models as well as to complete partial observations, both of which turn out helpful to learn good-quality action models.

Hybrid approach to model acquisition that compensates a lack of available data with domain specific knowledge provided by experts

A domain acquisition approach that considers using a different representation for the partial domain model by using schematic mutex relations in place of pre/post conditions.

One-shot learning: From domain knowledge to action models

domain model approach knowledge acquisition


We release the largest public ECG dataset of continuous raw signals for representation learning containing over 11k patients and 2 billion labelled beats.

Our goal is to enable semi-supervised ECG models to be made as well as to discover unknown subtypes of arrhythmia and anomalous ECG signal events.

To this end, we propose an unsupervised representation learning task, evaluated in a semi-supervised fashion.

We provide a set of baselines for different feature extractors that can be built upon.

Additionally, we perform qualitative evaluations on results from PCA embeddings, where we identify some clustering of known subtypes indicating the potential for representation learning in arrhythmia sub-type discovery.

We release a dataset constructed from single-lead ECG data from 11,000 patients who were prescribed to use the {DEVICENAME}(TM) device.

This paper describes a large-scale ECG dataset the authors intend to publish and provides unsupervised analysis and visualization of the dataset.

{COMPANYNAME}11K: An Unsupervised Representation Learning Dataset for Arrhythmia Subtype Discovery


As the basic building block of Convolutional Neural Networks (CNNs), the convolutional layer is designed to extract local patterns and lacks the ability to model global context in its nature.

Many efforts have been recently made to complement CNNs with the global modeling ability, especially by a family of works on global feature interaction.

In these works, the global context information is incorporated into local features before they are fed into convolutional layers.

However, research on neuroscience reveals that, besides influences changing the inputs to our neurons, the neurons' ability of modifying their functions dynamically according to context is essential for perceptual tasks, which has been overlooked in most of CNNs.

Motivated by this, we propose one novel Context-Gated Convolution (CGC) to explicitly modify the weights of convolutional layers adaptively under the guidance of global context.

As such, being aware of the global context, the modulated convolution kernel of our proposed CGC can better extract representative local patterns and compose discriminative features.

Moreover, our proposed CGC is lightweight, amenable to modern CNN architectures, and consistently improves the performance of CNNs according to extensive experiments on image classification, action recognition, and machine translation.

A novel Context-Gated Convolution which incorporates global context information into CNNs by explicitly modulating convolution kernels, and thus captures more representative local patterns and extract discriminative features.

This paper uses global context to modulate the weights of convolutional layers and help CNNs capture more discriminative features with high performance and fewer parameters than feature map modulating.

Context-Gated Convolution

discriminative feature
global context
capture
context-gated convolution
cnn
modulate


We analyze the trade-off between quantization noise and clipping distortion in low precision networks.

We identify the statistics of various tensors, and derive exact expressions for the mean-square-error degradation due to clipping.

By optimizing these expressions, we show marked improvements over standard quantization schemes that normally avoid clipping.

For example, just by choosing the accurate clipping values, more than 40\% accuracy improvement is obtained for the quantization of VGG-16 to 4-bits of precision.

Our results have many applications for the quantization of neural networks at both training and inference time.

We analyze the trade-off between quantization noise and clipping distortion in low precision networks, and show marked improvements over standard quantization schemes that normally avoid clipping

Derives a formula for finding the minimum and maximum clipping values for uniform quantization which minimize the square error resulting from quantization, for either a Laplace or Gaussian distribution over pre-quantized value.

ACIQ: Analytical Clipping for Integer Quantization of neural networks


Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field.

But its performance can awfully degrade with insufficient batch size.

This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of memory consumption.

Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and increase huge consumption.

In this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before.

The extra batch statistics associated with gradients also can severely affect the training of deep neural network.

Based on our analysis, we propose a novel normalization method, named Moving Average Batch Normalization (MABN).

MABN can completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations in inference procedure.

We prove the benefits of MABN by both theoretical analysis and experiments.

Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO.

The code has been released in https://github.com/megvii-model/MABN.

We propose a novel normalization method to handle small batch size cases.

A method to deal with the small batch size problem of BN which applies moving average operation without too much overhead and reduces the number of statistics of BN for better stability.

Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization

small batch size
method
statistic
normalization


We present a simple proof for the benefit of depth in multi-layer feedforward network with rectifed activation (``"depth separation").

Specifically we present a sequence of classification problems f_i such that

(a) for any fixed depth rectified network we can find an index m such that problems with index > m require exponential network width to fully represent the function f_m; and

(b) for any problem f_m in the family, we present a concrete neural network with linear depth and bounded width that fully represents it.

While there are several previous work showing similar results, our proof uses substantially simpler tools and techniques, and should be accessible to undergraduate students in computer science and people with similar backgrounds.

ReLU MLP depth seperation proof with gemoteric arguments

A proof that deeper networks need less units than shallower ones for a family of problems.

A Simple Geometric Proof for the Benefit of Depth in ReLU Networks


The rich and accessible labeled data fuel the revolutionary success of deep learning.

Nonetheless, massive supervision remains a luxury for many real applications, boosting great interest in label-scarce techniques such as few-shot learning (FSL).

An intuitively feasible approach to FSL is to conduct data augmentation via synthesizing additional training samples.

The key to this approach is how to guarantee both discriminability and diversity of the synthesized samples.

In this paper, we propose a novel FSL model, called $\textrm{D}^2$GAN, which synthesizes Diverse and Discriminative features based on Generative Adversarial Networks (GAN).

$\textrm{D}^2$GAN secures discriminability of the synthesized features by constraining them to have high correlation with real features of the same classes while low correlation with those of different classes.

Based on the observation that noise vectors that are closer in the latent code space are more likely to be collapsed into the same mode when mapped to feature space, $\textrm{D}^2$GAN incorporates a novel anti-collapse regularization term, which encourages feature diversity by penalizing the ratio of the logarithmic similarity of two synthesized features and the logarithmic similarity of the latent codes generating them.

Experiments on three common benchmark datasets verify the effectiveness of $\textrm{D}^2$GAN by comparing with the state-of-the-art.

A new GAN based few-shot learning algorithm by synthesizing  diverse and discriminative Features

A meta-learning method that learns a generative model that can augment the support set of a few-shot learner which optimizes a combination of losses.

$\textrm{D}^2$GAN: A Few-Shot Learning Approach with Diverse and Discriminative Feature Synthesis

diverse discriminative feature
few-shot
learning


The lack of crisp mathematical models that capture the structure of real-world

data sets is a major obstacle to the detailed theoretical understanding of deep

Here, we first demonstrate the effect of structured data sets

by experimentally comparing the dynamics and the performance of two-layer

networks trained on two different data sets:

(i) an unstructured synthetic data
set containing random i.i.d. inputs, and

Our analysis reveals two phenomena related to the dynamics of

the networks and their ability to generalise that only appear when training on

Second, we introduce a generative model for data sets,

where high-dimensional inputs lie on a lower-dimensional manifold and have

labels that depend only on their position within this manifold.

*hidden manifold model* and we experimentally demonstrate that training

networks on data sets drawn from this model reproduces both the phenomena seen

We demonstrate how structure in data sets impacts neural networks and introduce a generative model for synthetic data sets that reproduces this impact.

The paper studies how different settings of data structure affect learning of neural networks and how to mimic behavior on real datasets when learning on a synthetic one.

Modelling the influence of data structure on learning in neural networks

learn neural network
datum structure
model
synthetic


In this paper, we study deep diagonal circulant neural networks, that is deep neural networks in which weight matrices are the product of diagonal and circulant ones.

Besides making a theoretical analysis of their expressivity, we introduced principled techniques for training these models: we devise an initialization scheme and proposed a smart use of non-linearity functions in order to train deep diagonal circulant networks.

Furthermore, we show that these networks outperform recently introduced deep networks with other types of structured layers.

We conduct a thorough experimental study to compare the performance of deep diagonal circulant networks with state of the art models based on structured matrices and with dense models.

We show that our models achieve better accuracy than other structured approaches while required 2x fewer weights as the next best approach.

Finally we train deep diagonal circulant networks to build a compact and accurate models on a real world video classification dataset with over 3.8 million training examples.

We train deep neural networks based on diagonal and circulant matrices, and show that this type of networks are both compact and accurate on real world applications.

The authors provide a theoretical analysis of the expressive power of diagonal circulant neural networks (DCNN) and propose an initialization scheme for deep DCNNs.

Understanding and Training Deep Diagonal Circulant Neural Networks

diagonal circulant neural network
train deep


Interpretability has largely focused on local explanations, i.e. explaining why a model made a particular prediction for a sample.

These explanations are appealing due to their simplicity and local fidelity.

However, they do not provide information about the general behavior of the model.

We propose to leverage model distillation to learn global additive explanations that describe the relationship between input features and model predictions.

These global explanations take the form of feature shapes, which are more expressive than feature attributions.

Through careful experimentation, we show qualitatively and quantitatively that global additive explanations are able to describe model behavior and yield insights about models such as neural nets.

A visualization of our approach applied to a neural net as it is trained is available at https://youtu.be/ErQYwNqzEdc

We propose to leverage model distillation to learn global additive explanations in the form of feature shapes (that are more expressive than feature attributions) for models such as neural nets trained on tabular data.

This paper incorporates Generalized Additive Models (GAMs) with model distillation to provide global explanations of neural nets.

Learning Global Additive Explanations for Neural Nets Using Model Distillation

learn global additive explanation
neural net
model distillation


A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner.

These representations are typically used as general purpose features for words across a range of NLP problems.

However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem.

Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations.

In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model.

We train this model on several data sources with multiple training objectives on over 100 million sentences.

Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods.

We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

A large-scale multi-task learning framework with diverse training objectives to learn fixed-length sentence representations

This paper is about learning sentence embeddings by combining several training signals: skip-thought, predicting translation, classifying entailment relationships, and predicting the constituent parse.

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

sentence representation
learn
multi-task learning
training


In a time where neural networks are increasingly adopted in sensitive applications, algorithmic bias has emerged as an issue with moral implications.

While there are myriad ways that a system may be compromised by bias, systematically isolating and evaluating existing systems on such scenarios is non-trivial, i.e., bias may be subtle, natural and inherently difficult to quantify.

To this end, this paper proposes the first systematic study of benchmarking state-of-the-art neural models against biased scenarios.

More concretely, we postulate that the bias annotator problem can be approximated with neural models, i.e., we propose generative models of latent bias to deliberately and unfairly associate latent features to a specific class.

All in all, our framework provides a new way for principled quantification and evaluation of models against biased datasets.

Consequently, we find that state-of-the-art NLP models (e.g., BERT, RoBERTa, XLNET) are readily compromised by biased data.

We propose a neural bias annotator to benchmark models on their robustness to biased text datasets.

A method to generate biased datasets for NLP, relying on a conditional adversarially regularized autoencoder (CARA).

Generating Biased Datasets for Neural Natural Language Processing


We consider the problem of topic modeling in a weakly semi-supervised setting.

In this scenario, we assume that the user knows a priori a subset of the topics she wants the model to learn and is able to provide a few exemplar documents for those topics.

In addition, while each document may typically consist of multiple topics, we do not assume that the user will identify all its topics exhaustively.

Recent state-of-the-art topic models such as NVDM, referred to herein as Neural Topic Models (NTMs), fall under the variational autoencoder framework.

We extend NTMs to the weakly semi-supervised setting by using informative priors in the training objective.

After analyzing the effect of informative priors, we propose a simple modification of the NVDM model using a logit-normal posterior that we show achieves better alignment to user-desired topics versus other NTM models.

We propose supervising VAE-style topic models by intelligently adjusting the prior on a per document basis. We find a logit-normal posterior provides the best performance.

A flexible method of weakly supervising a topic model to achieve better alignment with user intuition.

WEAKLY SEMI-SUPERVISED NEURAL TOPIC MODELS


Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently as a tool to gain insight into, among others, their generalization ability.

However, it is by no means obvious how to estimate mutual information (MI) between each hidden layer and the input/desired output, to construct the IP.

For instance, hidden layers with many neurons require MI estimators with robustness towards the high dimensionality associated with such layers.

MI estimators should also be able to naturally handle convolutional layers, while at the same time being computationally tractable to scale to large networks.

None of the existing IP methods to date have been able to study truly deep Convolutional Neural Networks (CNNs), such as the e.g.\ VGG-16.

In this paper, we propose an IP analysis using the new matrix--based R\'enyi's entropy coupled with tensor kernels over convolutional layers, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data.

The obtained results shed new light on the previous literature concerning small-scale DNNs, however using a completely new approach.

Importantly, the new framework enables us to provide the first comprehensive IP analysis of contemporary large-scale DNNs and CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.

First comprehensive information plane analysis of large scale deep neural networks using matrix based entropy and tensor kernels.

The authors propose a tensor-kernel based estimator for mutual information estimation between high-dimensional layers in a neural network.

Information Plane Analysis of Deep Neural Networks via Matrix--Based Renyi's Entropy and Tensor Kernels

information plane analysis
deep neural network
entropy tensor kernel
base


Developing agents that can learn to follow natural language instructions has been an emerging research direction.

While being accessible and flexible, natural language instructions can sometimes be ambiguous even to humans.

To address this, we propose to utilize programs, structured in a formal language, as a precise and expressive way to specify tasks.

We then devise a modular framework that learns to perform a task specified by a program – as different circumstances give rise to diverse ways to accomplish the task, our framework can perceive which circumstance it is currently under, and instruct a multitask policy accordingly to fulfill each subtask of the overall task.

Experimental results on a 2D Minecraft environment not only demonstrate that the proposed framework learns to reliably accomplish program instructions and achieves zero-shot generalization to more complex instructions but also verify the efficiency of the proposed modulation mechanism for learning the multitask policy.

We also conduct an analysis comparing various models which learn from programs and natural language instructions in an end-to-end fashion.

We propose a modular framework that can accomplish tasks specified by programs and achieve zero-shot generalization to more complex tasks.

This paper investigates training RL agents with instructions and task decompositions formalized as programs, proposing a model for a program guided agent that interprets a program and proposes subgoals to an action module.

Program Guided Agent


We analyze the convergence of (stochastic) gradient descent algorithm for learning a convolutional filter with Rectified Linear Unit (ReLU) activation function.

Our analysis does not rely on any specific form of the input distribution and our proofs only use the definition of ReLU, in contrast with previous works that are restricted to standard Gaussian input.

We show that (stochastic) gradient descent with random initialization can learn the convolutional filter in polynomial time and the convergence rate depends on the smoothness of the input distribution and the closeness of patches.

To the best of our knowledge, this is the first recovery guarantee of gradient-based algorithms for convolutional filter on non-Gaussian input distributions.

Our theory also justifies the two-stage learning rate strategy in deep neural networks.

While our focus is theoretical, we also present experiments that justify our theoretical findings.

We prove randomly initialized (stochastic) gradient descent learns a convolutional filter in polynomial time.

Studies the problem of learning a single convolutional filter using SGD and shows that under certain conditions, SGD learns a single convolutional filter.

This paper extends the Gaussian distribution assumption to a more general angular smoothness assumption, which covers a wider family of input distributions

When is a Convolutional Filter Easy to Learn?


Deep neural networks (DNNs) are widely adopted in real-world cognitive applications because of their high accuracy.

The robustness of DNN models, however, has been recently challenged by adversarial attacks where small disturbance on input samples may result in misclassification.

State-of-the-art defending algorithms, such as adversarial training or robust optimization, improve DNNs' resilience to adversarial attacks by paying high computational costs.

Moreover, these approaches are usually designed to defend one or a few known attacking techniques only.

The effectiveness to defend other types of attacking methods, especially those that have not yet been discovered or explored, cannot be guaranteed.

This work aims for a general approach of enhancing the robustness of DNN models under adversarial attacks.

In particular, we propose Bamboo -- the first data augmentation method designed for improving the general robustness of DNN without any hypothesis on the attacking algorithms.

Bamboo augments the training data set with a small amount of data uniformly sampled on a fixed radius ball around each training data and hence, effectively increase the distance between natural data points and decision boundary.

Our experiments show that Bamboo substantially improve the general robustness against arbitrary types of attacks and noises, achieving better results comparing to previous adversarial training methods, robust optimization methods and other data augmentation methods with the same amount of data points.

The first data augmentation method specially designed for improving the general robustness of DNN without any hypothesis on the attacking algorithms.

Proposes a data augmentation training method to gain model robustness against adversarial perturbations, by augmenting uniformly random samples from a fixed-radius sphere centered at training data.

Bamboo: Ball-Shape Data Augmentation Against Adversarial Attacks from All Directions

robustness
datum augmentation
adversarial
method


The ability to synthesize realistic patterns of neural activity is crucial for studying neural information processing.

Here we used the Generative Adversarial Networks (GANs) framework to simulate the concerted activity of a population of neurons.

We adapted the Wasserstein-GAN variant to facilitate the generation of unconstrained neural population activity patterns while still benefiting from parameter sharing in the temporal domain.

We demonstrate that our proposed GAN, which we termed Spike-GAN, generates spike trains that match accurately the first- and second-order statistics of datasets of tens of neurons and also approximates well their higher-order statistics.

We applied Spike-GAN to a real dataset recorded from salamander retina and showed that it performs as well as state-of-the-art approaches based on the maximum entropy and the dichotomized Gaussian frameworks.

Importantly, Spike-GAN does not require to specify a priori the statistics to be matched by the model, and so constitutes a more flexible method than these alternative approaches.

Finally, we show how to exploit a trained Spike-GAN  to construct 'importance maps' to detect the most relevant statistical structures present in a spike train.

Spike-GAN provides a powerful, easy-to-use technique for generating realistic spiking neural activity and for describing the most relevant features of the large-scale neural population recordings studied in modern systems neuroscience.

Using Wasserstein-GANs to generate realistic neural activity and to detect the most relevant features present in neural population patterns.

A method for simulating spike trains from populations of neurons which match empirical data using a semi-convolutional GAN.

The paper proposes to use GANs for synthesizing realistic neural activity patterns

Synthesizing realistic neural population activity patterns using Generative Adversarial Networks

realistic neural activity
pattern
gan
population
synthesize


Deep latent variable models have become a popular model choice due to the scalable learning algorithms introduced by (Kingma & Welling 2013, Rezende et al. 2014).

These approaches maximize a variational lower bound on the intractable log likelihood of the observed data.

Burda et al. (2015) introduced a multi-sample variational bound, IWAE, that is at least as tight as the standard variational lower bound and becomes increasingly tight as the number of samples increases.

Counterintuitively, the typical inference network gradient estimator for the IWAE bound performs poorly as the number of samples increases (Rainforth et al. 2018, Le et al. 2018).

(2017) propose an improved gradient estimator, however, are unable to show it is unbiased.

We show that it is in fact biased and that the bias can be estimated efficiently with a second application of the reparameterization trick.

The doubly reparameterized gradient (DReG) estimator does not suffer as the number of samples increases, resolving the previously raised issues.

The same idea can be used to improve many recently introduced training techniques for latent variable models.

In particular, we show that this estimator reduces the variance of the IWAE gradient, the reweighted wake-sleep update (RWS) (Bornschein & Bengio 2014), and the jackknife variational inference (JVI) gradient (Nowozin 2018).

Finally, we show that this computationally efficient, drop-in estimator translates to improved performance for all three objectives on several modeling tasks.

Doubly reparameterized gradient estimators provide unbiased variance reduction which leads to improved performance.

Author experimentally found that the estimator of the existing work(STL) is biased and proposes to reduce the bias to improve the gradient estimator of the ELBO.

Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives

doubly reparameterize gradient estimator


Zeroth-order optimization is the process of minimizing an objective $f(x)$, given oracle access to evaluations at adaptively chosen inputs $x$.

In this paper, we present two simple yet powerful GradientLess Descent (GLD) algorithms that do not rely on an underlying gradient estimate and are numerically stable.

We analyze our algorithm from a novel geometric perspective and we show that for {\it any monotone transform} of a smooth and strongly convex objective with latent dimension $k \ge n$, we present a novel analysis that shows convergence within an $\epsilon$-ball of the optimum in $O(kQ\log(n)\log(R/\epsilon))$ evaluations, where the input dimension is $n$, $R$ is the diameter of the input space and $Q$ is the condition number.

Our rates are the first of its kind to be both

1) poly-logarithmically dependent on dimensionality and

2) invariant under monotone transformations.

We further leverage our geometric perspective to show that our analysis is optimal.

Both monotone invariance and its ability to utilize a low latent dimensionality are key to the empirical success of our algorithms, as demonstrated on synthetic and MuJoCo benchmarks.

Gradientless Descent is a provably efficient gradient-free algorithm that is monotone-invariant and fast for high-dimensional zero-th order optimization.

This paper proposes stable GradientLess Descent (GLD) algorithms that do not rely on gradient estimate.

Gradientless Descent: High-Dimensional Zeroth-Order Optimization

algorithm
gradientless descent
high-dimensional
optimization


Many processes can be concisely represented as a sequence of events leading from a starting state to an end state.

Given raw ingredients, and a finished cake, an experienced chef can surmise the recipe.

Building upon this intuition,  we propose a new class of visual generative models: goal-conditioned predictors (GCP).

Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video.

GCP instead treats videos as start-goal transformations, making video generation easier by conditioning on the more informative context provided by the first and final frames.

Not only do existing forward prediction approaches synthesize better and longer videos when modified to become goal-conditioned,  but GCP models can also utilize structures that are not linear in time, to accomplish hierarchical prediction

. To this end, we study both auto-regressive GCP models and novel tree-structured GCP models that generate frames recursively, splitting the video iteratively into finer and finer segments delineated by subgoals

. In experiments across simulated and real datasets, our GCP methods generate high-quality sequences over long horizons

.  Tree-structured GCPs are also substantially easier to parallelize than auto-regressive GCPs, making training  and  inference  very  efficient, and allowing the model to train on sequences that are thousands of frames in length.Finally, we demonstrate the utility of GCP approaches for imitation learning in the setting without access to expert actions

. Videos are on the supplementary website: https://sites.google.com/view/video-gcp

We propose a new class of visual generative models: goal-conditioned predictors. We show experimentally that conditioning on the goal allows to reduce uncertainty and produce predictions over much longer horizons.

This paper reformulates video prediction problem as interpolation instead of extrapolation by conditioning the prediction on the start and end (goal) frame, resulting in higher quality predictions.

Goal-Conditioned Video Prediction


Recent advances in computing technology and sensor design have made it easier to collect longitudinal or time series data from patients, resulting in a gigantic amount of available medical data.

Most of the medical time series lack annotations or even when the annotations are available they could be subjective and prone to human errors.

Earlier works have developed natural language processing techniques to extract concept annotations and/or clinical narratives from doctor notes.

However, these approaches are slow and do not use the accompanying medical time series data.

To address this issue, we introduce the problem of concept annotation for the medical time series data, i.e., the task of predicting and localizing medical concepts by using the time series data as input.

We propose Relational Multi-Instance Learning (RMIL) - a deep Multi Instance Learning framework based on recurrent neural networks, which uses pooling functions and attention mechanisms for the concept annotation tasks.

Empirical results on medical datasets show that our proposed models outperform various multi-instance learning models.

We propose a deep Multi Instance Learning framework based on recurrent neural networks which uses pooling functions and attention mechanisms for the concept annotation tasks.

The paper addresses the classification of medical time-series data and proposes to model the temporal relationship between the instances in each series using a recurrent neural network architecture.

Proposes a novel Multiple Instance Learning (MIL) formulation called Relation MIL (RMIL), and discussed a number of its variants with LSTM, Bi-LSTM, S2S, etc. and explores integrating RMIL with various attention mechanisms, and demonstrates its usage on medical concept prediction from time series data.

Relational Multi-Instance Learning for Concept Annotation from Medical Time Series

recurrent neural network
attention mechanism
concept annotation
medical
datum
time series
learn
instance


The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing.

However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting.

We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance.

We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.

Embedding layers are factorized with Tensor Train decomposition to reduce their memory footprint.

This paper proposes a low-rank tensor decomposition model to parameterize the embedding matrix in Natural Language Processing (NLP), which compresses the network and sometimes increases test accuracy.

Tensorized Embedding Layers for Efficient Model Compression


We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor.

We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function.

We provide empirical evidence that our proposed modification

(i)
decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and

(ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter).

We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence.

Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32.

Our source code will become available after the review process.

Fixing weight decay regularization in adaptive gradient methods such as Adam

Proposes idea to decouple the weight decay from the number of steps taken by the optimization process.

The paper presents an alternative way to implement weight decay in Adam with empirical results shown

Investigates weight decay issues lied in the SGD variants and proposes the decoupling method between weight decay and the gradient-based update.

Fixing Weight Decay Regularization in Adam

fix weight decay regularization
adam
decouple
method