Learning from a scalar reward in continuous action space environments is difficult and often requires millions if not billions of interactions.

We introduce state aligned vector rewards, which are easily defined in metric state spaces and allow our deep reinforcement learning agent to tackle the curse of dimensionality.

Our agent learns to map from action distributions to state change distributions implicitly defined in a quantile function neural network.

We further introduce a new reinforcement learning technique inspired by quantile regression which does not limit agents to explicitly parameterized action distributions.

Our results in high dimensional state spaces show that training with vector rewards allows our agent to learn multiple times faster than an agent training with scalar rewards.

We train with state aligned vector rewards an agent predicting state changes from action distributions, using a new reinforcement learning technique inspired by quantile regression.

Presents algorithm that aims to speed up reinforcement learning in situations where the reward is aligned with the state space.

This paper addresses RL in the continuous action space, using a re-parametrised policy and a novel vector-based training objective.

This work proposes to mix distributional RL with a net in charge of modeling the evolution of the world in terms of quantiles, claiming improvements in sample efficiency.

Quantile Regression Reinforcement Learning with State Aligned Vector Rewards

state align vector reward
reinforcement learning
rl
quantile
space
regression
action


We propose Episodic Backward Update - a new algorithm to boost the performance of a deep reinforcement learning agent by fast reward propagation.

In contrast to the conventional use of the replay memory with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state into its previous states.

Our computationally efficient recursive algorithm allows sparse and delayed rewards to propagate effectively throughout the sampled episode.

We evaluate our algorithm on 2D MNIST Maze Environment and 49 games of the Atari 2600 Environment and show that our agent improves sample efficiency with a competitive computational cost.

We propose Episodic Backward Update, a novel deep reinforcement learning algorithm which samples transitions episode by episode and updates values recursively in a backward manner to achieve fast and stable learning.

Proposes a new DQN where the targets are computed on a full episode by a backward update (end to start) for faster propagation of rewards by the episode end.

The authors propose to modify the DQN algorithm by applying the max Bellman operator recursively on a trajectory with some decay to prevent accumulating errors with the nested max.

In deep-Q networks, update Q values starting from the end of the episode in order to facilitate quick propagation of rewards back along the episode.

Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update

backward update
recursively
episode
deep reinforcement learning
propagation reward
start
end
fast
algorithm
dqn
episodic
value


Survival Analysis (time-to-event analysis) in the presence of multiple possible adverse events, i.e., competing risks, is a challenging, yet very important problem in medicine, finance, manufacturing, etc.

Extending classical survival analysis to competing risks is not trivial since only one event (e.g. one cause of death) is observed and hence, the incidence of an event of interest is often obscured by other related competing events.

This leads to the nonidentifiability of the event times’ distribution parameters, which makes the problem significantly more challenging.

In this work we introduce Siamese Survival Prognosis Network, a novel Siamese Deep Neural Network architecture that is able to effectively learn from data in the presence of multiple adverse events.

The Siamese Survival Network is especially crafted to issue pairwise concordant time-dependent risks, in which longer event times are assigned lower risks.

Furthermore, our architecture is able to directly optimize an approximation to the C-discrimination index, rather than relying on well-known metrics of cross-entropy etc., and which are not able to capture the unique requirements of survival analysis with competing risks.

Our results show consistent performance improvements on a number of publicly available medical datasets over both statistical and deep learning state-of-the-art methods.

In this work we introduce a novel Siamese Deep Neural Network architecture that is able to effectively learn from data in the presence of multiple adverse events.

This paper introduces siamese neural networks to the competing risks framework by optimizing for the c-index directly

The authors address issues of estimating risk in a survival analysis setting with competing risks and propose directly optimizing the time-dependent discrimination index using a siamese survival network

Siamese Survival Analysis with Competing Risks

compete risk
introduce
survival analysis
neural network
optimize
siamese
directly


The digitization of data has resulted in making datasets available to millions of users in the form of relational databases and spreadsheet tables.

However, a majority of these users come from diverse backgrounds and lack the programming expertise to query and analyze such tables.

We present a system that allows for querying data tables using natural language questions, where the system translates the question into an executable SQL query.

We use a deep sequence to sequence model in wich the decoder uses a simple type system of SQL expressions to structure the output prediction.

Based on the type, the decoder either copies an output token from the input question using an attention-based copying mechanism or generates it from a fixed vocabulary.

We also introduce a value-based loss function that transforms a distribution over locations to copy from into a distribution over the set of input tokens to improve training of our model.

We evaluate our model on the recently released WikiSQL dataset and show that our model trained using only supervised learning significantly outperforms the current state-of-the-art Seq2SQL model that uses reinforcement learning.

We present a type-based pointer network model together with a value-based loss method to effectively train a neural model to translate natural language to SQL.

The paper claims to develop a novel method to map natural language queries to SQL by using a grammar to guide decoding and using a new loss function for pointer / copy mechanism

Pointing Out SQL Queries From Text

sql
method
natural language
query
pointer
loss


To backpropagate the gradients through stochastic binary layers, we propose the augment-REINFORCE-merge (ARM) estimator that is unbiased, exhibits low variance, and has low computational complexity.

Exploiting variable augmentation, REINFORCE, and reparameterization, the ARM estimator achieves adaptive variance reduction for Monte Carlo integration by merging two expectations via common random numbers.

The variance-reduction mechanism of the ARM estimator can also be attributed to either antithetic sampling in an augmented space, or the use of an optimal anti-symmetric "self-control" baseline function together with the REINFORCE estimator in that augmented space.

Experimental results show the ARM estimator provides state-of-the-art performance in auto-encoding variational inference and maximum likelihood estimation, for discrete latent variable models with one or multiple stochastic binary layers.

Python code for reproducible research is publicly available.

An unbiased and low-variance gradient estimator for discrete latent variable models

Proposes a new variance-reduction technique to use when computing an expected loss gradient where the expectation is with respect to independent binary random variables.

An algorithm combining Rao-Blackwellization and common random numbers for lowering the variance of the score-function gradient estimator in the special case of stochastic binary networks

An unbiased and low variance augment-REINFORCE-merge (ARM) estimator for calculating and backpropagating gradients in binary neural networks

ARM: Augment-REINFORCE-Merge Gradient for Stochastic Binary Networks

stochastic binary network
unbiased
gradient estimator
augment-reinforce-merge
arm
variable
random
variance


Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training.

The scheme can reach a linear speed-up with respect to the number of workers, but this is rarely seen in practice as the scheme often suffers from large network delays and bandwidth limits.

To overcome this communication bottleneck recent works propose to reduce the communication frequency.

An algorithm of this type is local SGD that runs SGD independently in parallel on different workers and averages the sequences only once in a while.

This scheme shows promising results in practice, but eluded thorough theoretical analysis.

We prove concise convergence rates for local SGD on convex problems and show that it converges at the same rate as mini-batch SGD in terms of number of evaluated gradients, that is, the scheme achieves linear speed-up in the number of workers and mini-batch size.

The number of  communication rounds can be reduced up to a factor of T^{1/2}---where T denotes the number of total steps---compared to mini-batch SGD.

This also holds for asynchronous implementations.

Local SGD can also be used for large scale training of deep learning models.

The results shown here aim serving as a guideline to further explore the theoretical and practical aspects of local SGD in these applications.

We prove that parallel local SGD achieves linear speedup with much lesser communication than parallel mini-batch SGD.

Provides a convergence proof for local SGD, and proves that local SGD can provide the same speedup gains as minibatch, but may be able to communicate significantly less.

This paper presents an analysis of local SGD and bounds on how frequent the estimators obtained by running SGD required to be averaged in order to yield linear parallelization speedups.

The authors analyze the local SGD algorithm, where $K$ parallel chains of SGD are run, and the iterates are occasionally synchronized across machines by averaging

Local SGD Converges Fast and Communicates Little

local sgd
prove
speedup
communicate
run
average
parallel
less
linear


Extracting relevant information, causally inferring and predicting the future states with high accuracy is a crucial task for modeling complex systems.

The endeavor to address these tasks is made even more challenging when we have to deal with high-dimensional heterogeneous data streams.

Such data streams often have higher-order inter-dependencies across spatial and temporal dimensions.

We propose to perform a soft-clustering of the data and learn its dynamics to produce a compact dynamical model while still ensuring the original objectives of causal inference and accurate predictions.

To efficiently and rigorously process the dynamics of soft-clustering, we advocate for an information theory inspired approach that incorporates stochastic calculus and seeks to determine a trade-off between the predictive accuracy and compactness of the mathematical representation.

We cast the model construction as a maximization of the compression of the state variables such that the predictive ability and causal interdependence (relatedness) constraints between the original data streams and the compact model are closely bounded.

We provide theoretical guarantees concerning the convergence of the proposed learning algorithm.

To further test the proposed framework, we consider a high-dimensional Gaussian case study and describe an iterative scheme for updating the new model parameters.

Using numerical experiments, we demonstrate the benefits on compression and prediction accuracy for a class of dynamical systems.

Finally, we apply the proposed algorithm to the real-world dataset of multimodal sentiment intensity and show improvements in prediction with reduced dimensions.

Studies the problem of compactly representing the model of a complex dynamic system while preserving information by using an information bottleneck method.

This paper studied the Gaussian linear dynamic and proposed an algorithm for computing the Information Bottleneck Hierarchy (IBH).

Learning Information Propagation in the Dynamical Systems via Information Bottleneck Hierarchy

information bottleneck
dynamical
study
dynamic
system
hierarchy


We propose the dense RNN, which has the fully connections from each hidden state to multiple preceding hidden states of all layers directly.

As the density of the connection increases, the number of paths through which the gradient flows can be increased.

It increases the magnitude of gradients, which help to prevent the vanishing gradient problem in time.

Larger gradients, however, can also cause exploding gradient problem.

To complement the trade-off between two problems, we propose an attention gate, which controls the amounts of gradient flows.

We describe the relation between the attention gate and the gradient flows by approximation.

The experiment on the language modeling using Penn Treebank corpus shows dense connections with the attention gate improve the model’s performance.

Dense RNN that has fully connections from each hidden state to multiple preceding hidden states of all layers directly.

Proposes a new RNN architecture that models long-term dependencies better, can learn multiscale representation of sequential data, and sidestep the gradients problem by using parametrized gating units.

This paper proposes a fully connected dense RNN architecture with gated connections to every layer and preceding layer connections, and it's results on PTB charcter-level modelling task.

Dense Recurrent Neural Network with Attention Gate

fully
connection
rnn architecture
layer
dense
precede
gate


We propose a new algorithm for training generative adversarial networks to jointly learn latent codes for both identities (e.g. individual humans) and observations (e.g. specific photographs).

In practice, this means that by fixing the identity portion of latent codes, we can generate diverse images of the same subject, and by fixing the observation portion we can traverse the manifold of subjects while maintaining contingent aspects such as lighting and pose.

Our algorithm features a pairwise training scheme in which each sample from the generator consists of two images with a common identity code.

Corresponding samples from the real dataset consist of two distinct photographs of the same subject.

In order to fool the discriminator, the generator must produce images that are both photorealistic, distinct, and appear to depict the same person.

We augment both the DCGAN and BEGAN approaches with Siamese discriminators to accommodate pairwise training.

Experiments with human judges and an off-the-shelf face verification system demonstrate our algorithm’s ability to generate convincing, identity-matched photographs.

SD-GANs disentangle latent codes according to known commonalities in a dataset (e.g. photographs depicting the same person).

This paper investigates the problem of controlled image generation and proposes an algorithm that produces a pair of images with the same identity.

This paper proposes, SD-GAN, a method of training GANs to disentangle the identity and non-identity information in the latent vector input Z.

Semantically Decomposing the Latent Spaces of Generative Adversarial Networks


The goal of unpaired cross-domain translation is to learn useful mappings between two domains, given unpaired sets of datapoints from these domains.

While this formulation is highly underconstrained, recent work has shown that it is possible to learn mappings useful for downstream tasks by encouraging approximate cycle consistency in the mappings between the two domains [Zhu et al., 2017].

In this work, we propose AlignFlow, a framework for unpaired cross-domain translation that ensures exact cycle consistency in the learned mappings.

Our framework uses a normalizing flow model to specify a single invertible mapping between the two domains.

In contrast to prior works in cycle-consistent translations, we can learn AlignFlow via adversarial training, maximum likelihood estimation, or a hybrid of the two methods.

Theoretically, we derive consistency results for AlignFlow which guarantee recovery of desirable mappings under suitable assumptions.

Empirically, AlignFlow demonstrates significant improvements over relevant baselines on image-to-image translation and unsupervised domain adaptation tasks on benchmark datasets.

We propose a learning framework for cross-domain translations which is exactly cycle-consistent and can be learned via adversarial training, maximum likelihood estimation, or a hybrid.

Proposes AlignFlow, an efficient way of implementing cycle consistency principle using invertible flows.

Flow models for unpaired image to image translation

AlignFlow: Cycle Consistent Learning from Multiple Domains via Normalizing Flows

learning alignflow via translation flow cycle


Program synthesis is a class of regression problems where one seeks a solution, in the form of a source-code program, that maps the inputs to their corresponding outputs exactly.

Due to its precise and combinatorial nature, it is commonly formulated as a constraint satisfaction problem, where input-output examples are expressed constraints, and solved with a constraint solver.

A key challenge of this formulation is that of scalability: While constraint solvers work well with few well-chosen examples, constraining the entire set of example constitutes a significant overhead in both time and memory.

In this paper we address this challenge by constructing a representative subset of examples that is both small and is able to constrain the solver sufficiently.

We build the subset one example at a time, using a trained discriminator to predict the probability of unchosen input-output examples conditioned on the chosen input-output examples, adding the least probable example to the subset.

Experiment on a diagram drawing domain shows our approach produces subset of examples that are small and representative for the constraint solver.

In a program synthesis context where the input is a set of examples, we reduce the cost by computing a subset of representative examples

Proposes a method for identifying representative examples for program synthesis to increase the scalability of existing constraint programming solutions.

A method for choosing a subset of examples on which to run a constraint solver in order to solve program synthesis problems.

This paper proposes a method for speeding up the general-purpose program synthesizers.

Learning to select examples for program synthesis

example
program synthesis
method
subset
representative
constraint


Humans possess an ability to abstractly reason about objects and their interactions, an ability not shared with state-of-the-art deep learning models.

Relational networks, introduced by Santoro et al. (2017), add the capacity for relational reasoning to deep neural networks, but are limited in the complexity of the reasoning tasks they can address.

We introduce recurrent relational networks which increase the suite of solvable tasks to those that require an order of magnitude more steps of relational reasoning.

We use recurrent relational networks to solve Sudoku puzzles and achieve state-of-the-art results by solving 96.6% of the hardest Sudoku puzzles, where relational networks fail to solve any.

We also apply our model to the BaBi textual QA dataset solving 19/20 tasks which is competitive with state-of-the-art sparse differentiable neural computers.

The recurrent relational network is a general purpose module that can augment any neural network model with the capacity to do many-step relational reasoning.

We introduce Recurrent Relational Networks, a powerful and general neural network module for relational reasoning, and use it to solve 96.6% of the hardest Sudokus and 19/20 BaBi tasks.

Introduced recurrent relational network (RRNs) that can be added to any neural networks to add relational reasoning capacity.

Introduction of a deep neural network for structured prediction that achieves state-of-the-art performance on Soduku puzzles and the BaBi task.

This paper describes a method called relational network to add relational reasoning capacity to deep neural networks.

Recurrent Relational Networks for complex relational reasoning

network add relational reasoning capacity


Empirical risk minimization (ERM), with proper loss function and regularization, is the common practice of supervised classification.

In this paper, we study training arbitrary (from linear to deep) binary classifier from only unlabeled (U) data by ERM.

We prove that it is impossible to estimate the risk of an arbitrary binary classifier in an unbiased manner given a single set of U data, but it becomes possible given two sets of U data with different class priors.

These two facts answer a fundamental question---what the minimal supervision is for training any binary classifier from only U data.

Following these findings, we propose an ERM-based learning method from two sets of U data, and then prove it is consistent.

Experiments demonstrate the proposed method could train deep models and outperform state-of-the-art methods for learning from two sets of U data.

Three class priors are all you need to train deep models from only U data, while any two should not be enough.

Proposes an unbiased estimator that allows for training models with weak supervision on two unlabeled datasets with known class priors and discusses theoretical properties of the estimators.

A methodology for training any binary classifier from only unlabeled data, and an empirical risk minimization method for two sets of unlabeled data where class priors are given.

On the Minimal Supervision for Training Any Binary Classifier from Only Unlabeled Data

train binary classifier unlabeled datum
class prior
model
two
supervision


Deep Neural Networks (DNNs) excel on many complex perceptual tasks but it has proven notoriously difficult to understand how they reach their decisions.

We here introduce a high-performance DNN architecture on ImageNet whose decisions are considerably easier to explain.

Our model, a simple variant of the ResNet-50 architecture called BagNet, classifies an image based on the occurrences of small local image features without taking into account their spatial ordering.

This strategy is closely related to the bag-of-feature (BoF) models popular before the onset of deep learning and reaches a surprisingly high accuracy on ImageNet (87.6% top-5 for 32 x 32 px features and Alexnet performance for 16 x16 px features).

The constraint on local features makes it straight-forward to analyse how exactly each part of the image influences the classification.

Furthermore, the BagNets behave similar to state-of-the art deep neural networks such as VGG-16, ResNet-152 or DenseNet-169 in terms of feature sensitivity, error distribution and interactions between image parts.

This suggests that the improvements of DNNs over previous bag-of-feature classifiers in the last few years is mostly achieved by better fine-tuning rather than by qualitatively different decision strategies.

Aggregating class evidence from many small image patches suffices to solve ImageNet, yields more interpretable models and can explain aspects of the decision-making of popular DNNs.

This paper suggests a novel and compact neural network architecture which uses the information within bag-of-words features. The proposed algorithm only uses the patch information independently and performs majority voting using independently classified patches.

Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet


Somatic cancer mutation detection at ultra-low variant allele frequencies (VAFs) is an unmet challenge that is intractable with current state-of-the-art mutation calling methods.

Specifically, the limit of VAF detection is closely related to the depth of coverage, due to the requirement of multiple supporting reads in extant methods, precluding the detection of mutations at VAFs that are orders of magnitude lower than the depth of coverage.

Nevertheless, the ability to detect cancer-associated mutations in ultra low VAFs is a fundamental requirement for low-tumor burden cancer diagnostics applications such as early detection, monitoring, and therapy nomination using liquid biopsy methods (cell-free DNA).

Here we defined a spatial representation of sequencing information adapted for convolutional architecture that enables variant detection at VAFs, in a manner independent of the depth of sequencing.

This method enables the detection of cancer mutations even in VAFs as low as 10x-4^, >2 orders of magnitude below the current state-of-the-art.

We validated our method on both simulated plasma and on clinical cfDNA plasma samples from cancer patients and non-cancer controls.

This method introduces a new domain within bioinformatics and personalized medicine – somatic whole genome mutation calling for liquid biopsy.

Current somatic mutation methods do not work with liquid biopsies (ie low coverage sequencing), we apply a CNN architecture to a unique representation of a read and its ailgnment, we show significant improvement over previous methods in the low frequency setting.

Proposes a CNN based solution called Kittyhawk for somatic mutation calling at ultra low allele frequencies.

A new algorithm to detect cancer mutations from sequencing cell free DNA that will identify the sequence context that characterize sequencing errors from true mutations.

This paper proposes a deep learning framework to predict somatic mutations at extremely low frequencies which occurs in detecting tumor from cell-free DNA

Deep learning mutation prediction enables early stage lung cancer detection in liquid biopsy

low frequency
liquid biopsy
somatic mutation
cnn
sequence
detect
cancer
dna
deep


This paper presents the formal release of {\em MedMentions}, a new manually annotated resource for the recognition of biomedical concepts.

What distinguishes MedMentions from other annotated biomedical corpora is its size (over 4,000 abstracts and over 350,000 linked mentions), as well as the size of the concept ontology (over 3 million concepts from UMLS 2017) and its broad coverage of biomedical disciplines.

In addition to the full corpus, a sub-corpus of MedMentions is also presented, comprising annotations for a subset of UMLS 2017 targeted towards document retrieval.

To encourage research in Biomedical Named Entity Recognition and Linking, data splits for training and testing are included in the release, and a baseline model and its metrics for entity linking are also described.

The paper introduces a new gold-standard corpus corpus of biomedical scientific literature manually annotated with UMLS concept mentions.

Details the construction of a manually annotated dataset covering biomedical concepts that is larger and covered by a larger ontology than previous datasets.

This paper uses MedMentions, a TaggerOne semi-Markov model for end-to-end concept recognition and linking on a set of Pubmed abstracts to label papers with biomedical concepts/entities

MedMentions: A Large Biomedical Corpus Annotated with UMLS Concepts

annotate umls concept
large
medmention
biomedical
manually
corpus


In this paper we propose a Deep Autoencoder Mixture Clustering (DAMIC) algorithm.

It is based on a mixture of deep autoencoders where each cluster is represented by an autoencoder.

A clustering network transforms the data into another space and then selects one of the clusters.

Next, the autoencoder associated with this cluster is used to reconstruct the data-point.

The clustering algorithm jointly learns the nonlinear data representation and the set of autoencoders.

The optimal clustering is found by minimizing the reconstruction loss of the mixture of autoencoder network.

Unlike other deep clustering algorithms, no regularization term is needed to avoid data collapsing to a single point.

Our experimental evaluations on image and text corpora show significant improvement over state-of-the-art methods.

We propose a deep clustering method where instead of a centroid each cluster is represented by an autoencoder

Presents deep clustering based on a mixture of autoencoders, where data points are allocated to a cluster based the representation error if the autoencoder network were used to represent it.

A deep clustering approach that uses an autoencoder framework to learn a low-dimensional embedding of the data simultaneously while clustering data using a deep neural network.

A deep clustering method which represents each cluster with different auto-encoders, works in an end-to-end manner, and also can be used to cluster new incoming data without redoing the whole clustering procedure.

Deep clustering based on a mixture of autoencoders

deep clustering base mixture autoencoder


We propose a new Integral Probability Metric (IPM) between distributions: the Sobolev IPM.

The Sobolev IPM compares the mean discrepancy of two distributions for functions (critic) restricted to a Sobolev ball defined with respect to a dominant measure mu.

We show that the Sobolev IPM compares two distributions in high dimensions based on weighted conditional Cumulative Distribution Functions (CDF) of each coordinate on a leave one out basis.

The Dominant measure mu plays a crucial role as it defines the support on which conditional CDFs are compared.

Sobolev IPM can be seen as an extension of the one dimensional Von-Mises Cramer statistics to high dimensional distributions.

We show how Sobolev IPM can be used to train Generative Adversarial Networks (GANs).

We then exploit the intrinsic conditioning implied by Sobolev IPM in text generation.

Finally we show that a variant of Sobolev GAN achieves competitive results in semi-supervised learning on CIFAR-10, thanks to the smoothness enforced on the critic by Sobolev GAN which relates to Laplacian regularization.

We define a new Integral Probability Metric (Sobolev IPM) and show how it can be used for training GANs for text generation and semi-supervised learning.

Suggests a novel regularization scheme for GANs based on a Sobolev norm, measuring deviations between L2 norms of derivatives.

The authors provide another type of GAN using the typical setup of a GAN but with a different function class, and produce a recipe for training GANs with that sort of function class.

The paper proposes a different gradient penalty for GAN critics that forces the expected squared norm of the gradient to be equal to 1

Sobolev GAN


We propose in this paper a new approach to train the Generative Adversarial Nets (GANs) with a mixture of generators to overcome the mode collapsing problem.

The main intuition is to employ multiple generators, instead of using a single one as in the original GAN.

The idea is simple, yet proven to be extremely effective at covering diverse data modes, easily overcoming the mode collapsing problem and delivering state-of-the-art results.

A minimax formulation was able to establish among a classifier, a discriminator, and a set of generators in a similar spirit with GAN.

Generators create samples that are intended to come from the same distribution as the training data, whilst the discriminator determines whether samples are true data or generated by generators, and the classifier specifies which generator a sample comes from.

The distinguishing feature is that internal samples are created from multiple generators, and then one of them will be randomly selected as final output similar to the mechanism of a probabilistic mixture model.

We term our method Mixture Generative Adversarial Nets (MGAN).

We develop theoretical analysis to prove that, at the equilibrium, the Jensen-Shannon divergence (JSD) between the mixture of generators’ distributions and the empirical data distribution is minimal, whilst the JSD among generators’ distributions is maximal, hence effectively avoiding the mode collapsing problem.

By utilizing parameter sharing, our proposed model adds minimal computational cost to the standard GAN, and thus can also efficiently scale to large-scale datasets.

We conduct extensive experiments on synthetic 2D data and natural image databases (CIFAR-10, STL-10 and ImageNet) to demonstrate the superior performance of our MGAN in achieving state-of-the-art Inception scores over latest baselines, generating diverse and appealing recognizable objects at different resolutions, and specializing in capturing different types of objects by the generators.

We propose a new approach to train GANs with a mixture of generators to overcome the mode collapsing problem.

Address the problem of mode collapse in GANs using a constrained mixture distribution for the generator and an auxiliary classifier which predicts the source mixture component.

The paper proposes a mixture of generators to train GANs without extra computational cost

The authors present that using MGAN, which aims to overcome model collapsing problem by mixture generators, achieves state-of-art results

MGAN: Training Generative Adversarial Nets with Multiple Generators

mixture generator
train gan
collapse problem
mode
overcome
mgan


Allowing humans to interactively train artificial agents to understand language instructions is desirable for both practical and scientific reasons.

Though, given  the lack of sample efficiency in current learning methods, reaching this goal may require substantial research efforts.

We introduce the BabyAI research platform, with the goal of supporting investigations towards including humans in the loop for grounded language learning.

The BabyAI platform comprises an extensible suite of 19 levels of increasing difficulty.

Each level gradually leads the agent towards acquiring a combinatorially rich synthetic language, which is a proper subset of English.

The platform also provides a hand-crafted bot agent, which simulates a human teacher.

We report estimated amount of supervision required for training neural reinforcement and behavioral-cloning agents on some BabyAI levels.

We put forward strong evidence that current deep learning methods are not yet sufficiently sample-efficient in the context of learning a language with compositional properties.

We present the BabyAI platform for studying data efficiency of language learning with a human in the loop

Presents a research platform with a bot in the loop for learning to execute language instructions in which language has compositional structures

Introduces a platform for grounded language learning that replaces any human in the loop with a heuristic teacher and uses a synthetic language mapped to a 2D grid world

BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning

human loop
babyai platform
ground language learning
learn
efficiency
study
present


Recently, there has been growing interest in methods that perform neural network compression, namely techniques that attempt to substantially reduce the size of a neural network without significant reduction in performance.

However, most existing methods are post-processing approaches in that they take a learned neural network as input and output a compressed network by either forcing several parameters to take the same value (parameter tying via quantization) or pruning irrelevant edges (pruning) or both.

In this paper, we propose a novel algorithm that jointly learns and compresses a neural network.

The key idea in our approach is to change the optimization criteria by adding $k$ independent Gaussian priors over the parameters and a sparsity penalty.

We show that our approach is easy to implement using existing neural network libraries, generalizes L1 and L2 regularization and elegantly enforces parameter tying as well as pruning constraints.

Experimentally, we demonstrate that our new algorithm yields state-of-the-art compression on several standard benchmarks with minimal loss in accuracy while requiring little to no hyperparameter tuning as compared with related, competing approaches.

A k-means prior combined with L1 regularization yields state-of-the-art compression results.

This paper explores soft parameter tying and compression of DNNs/CNNs

Automatic Parameter Tying in Neural Networks


The application of stochastic variance reduction to optimization has shown remarkable recent theoretical and practical success.

The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem.

We show that naive application of the SVRG technique and related approaches fail, and explore why.

The SVRG method fails on modern deep learning problems

This paper presents an analysis of SVRG style methods, showing that dropout, batch norm, data augmentation (random crop/rotation/translations) tend to increase bias and/or variance of the updates.

This paper investigates the applicability of SVGD to modern neural networks and shows the naive application of SVGD typically fails.

On the Ineffectiveness of Variance Reduced Optimization for Deep Learning

show
svrg
deep learning
fail
modern
variance
method


The ground-breaking performance obtained by deep convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks.

One of the main challenge in applying CNNs to 3D shape analysis is how to define a natural convolution operator on non-euclidean surfaces.

In this paper, we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere.

A cascade set of geodesic disk filters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks.

We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance, and its effectiveness is evaluated in applications of non-rigid/ rigid shape classification and shape retrieval.

A method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere.

Presents a polar anisotropic convolution scheme on a unit sphere by replacing filter translation with filter rotation.

This paper explores deep learning of 3D shapes using alt-az anisotropic 2-sphere convolution

Deep Learning 3D Shapes Using Alt-az Anisotropic 2-Sphere Convolution

3d shape alt-az anisotropic 2-sphere convolution


Recent breakthroughs in computer vision make use of large deep neural networks, utilizing the substantial speedup offered by GPUs.

For applications running on limited hardware, however, high precision real-time processing can still be a challenge.

One approach to solving this problem is training networks with binary or ternary weights, thus removing the need to calculate multiplications and significantly reducing memory size.

In this work, we introduce LR-nets (Local reparameterization networks), a new method for training neural networks with discrete weights using stochastic parameters.

We show how a simple modification to the local reparameterization trick, previously used to train Gaussian distributed weights, enables the training of discrete weights.

Using the proposed training we test both binary and ternary models on MNIST, CIFAR-10 and ImageNet benchmarks and reach state-of-the-art results on most experiments.

Training binary/ternary networks using local reparameterization with the CLT approximation

Trains binary and ternary weight distribution networks using backpropagation to sample neuron pre-activations with reparameterization trick

This paper suggests using stochastic parameters in combination with the local reparametrisation trick to train neural networks with binary or ternary weights, which leads to state of the art results.

Learning Discrete Weights Using the Local Reparameterization Trick

local reparameterization
binary ternary weight
network
train
trick


We present Optimal Completion Distillation (OCD), a training procedure for optimizing sequence to sequence models based on edit distance.

OCD is efficient, has no hyper-parameters of its own, and does not require pre-training or joint optimization with conditional log-likelihood.

Given a partial sequence generated by the model, we first identify the set of optimal suffixes that minimize the total edit distance, using an efficient dynamic programming algorithm.

Then, for each position of the generated sequence, we use a target distribution which puts equal probability on the first token of all the optimal suffixes.

OCD achieves the state-of-the-art performance on end-to-end speech recognition, on both Wall Street Journal and Librispeech datasets, achieving $9.3\%$ WER and $4.5\%$ WER, respectively.

Optimal Completion Distillation (OCD) is a training procedure for optimizing sequence to sequence models based on edit distance which achieves state-of-the-art on end-to-end Speech Recognition tasks.

Alternative approach to training seq2seq models using a dynamic program to compute optimal continuations of predicted prefixes

A training algorithm for auto-regressive models that does not require any MLE pre-training and can directly optimize from the sampling.

The paper considers a shortcoming of sequence to sequence models trained using maximum likelihood estimation and propose an approach based on edit distances and the implicit use of given label sequences during training

Optimal Completion Distillation for Sequence Learning

base edit distance
optimal completion distillation
sequence sequence model
training
optimize
approach
train


As an emerging field, federated learning has recently attracted considerable attention.

Compared to distributed learning in the datacenter setting, federated learning

has more strict constraints on computate efficiency of the learned model and communication

federated learning framework based on variational dropout.

to jointly learn a sparse model while reducing the amount of gradients exchanged

of our approach on achieving significant model compression and communication

a joint model and gradient sparsification method for federated learning

Applies variational dropout to reduce the communication cost of distributed training of neural networks, and does experiments on mnist, cifar10 and svhn datasets.

The authors propose an algorithm that reduces communication costs in federated learning by sending sparse gradients from device to server and back.

Combines distributed optimization algorithm with variational dropout to sparsify the gradients sent to master server from local learners.

Efficient Federated Learning via Variational Dropout

reduce communication cost
variational dropout
gradient
server
algorithm
federate learning
send
distribute


We prove a multiclass boosting theory for the ResNet architectures which simultaneously creates a new technique for multiclass boosting and provides a new algorithm for ResNet-style architectures.

Our proposed training algorithm, BoostResNet, is particularly suitable in non-differentiable architectures.

Our method only requires the relatively inexpensive sequential training of T "shallow ResNets".

We prove that the training error decays exponentially with the depth T if the weak module classifiers that we train perform slightly better than some weak baseline.

In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition.

A generalization error bound based on margin theory is proved and suggests that ResNet could be resistant to overfitting using a network with l_1 norm bounded weights.

Presents a boosting-style algorithm for training deep residual networks, a convergence analysis for training error, and a analysis of generalization ability.

A learning method for ResNet using the boosting framework that decomposes the learning of complex networks and uses less computational costs.

Authors propose the deep ResNet as a boosting algorithm, and they claim this is more efficient than standard end-to-end backropagation.

Learning Deep ResNet Blocks Sequentially using Boosting Theory

resnet network boosting algorithm boost theory deep


We consider the problem of learning a one-hidden-layer neural network: we assume the input x is from Gaussian distribution and the label $y = a \sigma(Bx) + \xi$, where a is a nonnegative vector and  $B$ is a full-rank weight matrix, and $\xi$ is a noise vector.

We first give an analytic formula for the population risk of the standard squared loss and demonstrate that it implicitly attempts to decompose a sequence of low-rank tensors simultaneously.

Inspired by the formula, we design a non-convex objective function $G$ whose landscape is guaranteed to have the following properties:

1.

All local minima of $G$ are also global minima.

All global minima of $G$ correspond to the ground truth parameters.

The value and gradient of $G$ can be estimated using samples.

With these properties, stochastic gradient descent on $G$ provably converges to the global minimum and learn the ground-truth parameters.

We also prove finite sample complexity results and validate the results by simulations.

The paper analyzes the optimization landscape of one-hidden-layer neural nets and designs a new objective that provably has no spurious local minimum.

This paper studies the problem of learning one-hidden layer neural networks, establishes a connection between least squares population loss and Hermite polynomials, and proposes a new loss function.

A tensor factorization-type method for leaning one hidden-layer neural netowrk

Learning One-hidden-layer Neural Networks with Landscape Design

neural network
new
landscape
design
one-hidden-layer
learn


Open information extraction (OIE) systems extract relations and their
  arguments from natural language text in an unsupervised manner.

The resulting
  extractions are a valuable resource for downstream tasks such as knowledge
  base construction, open question answering, or event schema induction.

In this
  paper, we release, describe, and analyze an OIE corpus called OPIEC, which was
  extracted from the text of English Wikipedia.

OPIEC complements the available
  OIE resources: It is the largest OIE corpus publicly available to date (over
  340M triples) and contains valuable metadata such as provenance information,
  confidence scores, linguistic annotations, and semantic annotations including
  spatial and temporal information.

We analyze the OPIEC corpus by comparing its
  content with knowledge bases such as DBpedia or YAGO, which are also based on
  Wikipedia.

We found that most of the facts between entities present in OPIEC
  cannot be found in DBpedia and/or YAGO, that OIE facts
  often differ in the level of specificity compared to knowledge base facts, and
  that OIE open relations are generally highly polysemous.

We believe that the
  OPIEC corpus is a valuable resource for future research on automated knowledge
  base construction.

An Open Information Extraction Corpus and its in-depth analysis

Builds a new corpus for information extraction which is larger than the prior public corpora and contains information not existing in current corpora.

Presents a dataset of open-IE triples that were collected from Wikipedia with the help of a recent extraction system.

The paper describes the creation of an Open IE corpus over English Wikipedia through an automatic manner

OPIEC: An Open Information Extraction Corpus

open information extraction corpus
wikipedia


The process of designing neural architectures requires expert knowledge and extensive trial and error.

While automated architecture search may simplify these requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in both flexibility and components.

We propose a domain-specific language (DSL) for use in automated architecture search which can produce novel RNNs of arbitrary depth and width.

The DSL is flexible enough to define standard architectures such as the Gated Recurrent Unit and Long Short Term Memory and allows the introduction of non-standard RNN components such as trigonometric curves and layer normalization.

Using two different candidate generation techniques, random search with a ranking function and reinforcement learning,
we explore the novel architectures produced by the RNN DSL for language modeling and machine translation domains.

The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed.

We define a flexible DSL for RNN architecture generation that allows RNNs of varying size and complexity and propose a ranking function that represents RNNs as recursive neural networks, simulating their performance to decide on the most promising architectures.

Introduces a new method to generate RNNs architectures using a domain-specific language for two types of generators (random and RL-based) together with a ranking function and evaluator.

This paper casts the search of good RNN Cell architectures as a black-box optimization problem where examples are represented as an operator tree and scored based on learnt functions or generated by a RL agent.

This paper investigates meta-learning strategy for automated architecture search in the context of RNN by using a DSL that specifies RNN recurrent operations.

A Flexible Approach to Automated RNN Architecture Generation

ranking function
rnn architecture generation
search
flexible
represent
dsl
generate
automate
rnns


Researches on deep neural networks with discrete parameters and their deployment in embedded systems have been active and promising topics.

Although previous works have successfully reduced precision in inference, transferring both training and inference processes to low-bitwidth integers has not been demonstrated simultaneously.

In this work, we develop a new method termed as ``"WAGE" to discretize both training and inference, where weights (W), activations (A), gradients (G) and errors (E) among layers are shifted and linearly constrained to low-bitwidth integers.

To perform pure discrete dataflow for fixed-point devices, we further replace batch normalization by a constant scaling layer and simplify other components that are arduous for integer implementation.

Improved accuracies can be obtained on multiple datasets, which indicates that WAGE somehow acts as a type of regularization.

Empirically, we demonstrate the potential to deploy training in hardware systems such as integer-based deep learning accelerators and neuromorphic chips with comparable accuracy and higher energy efficiency, which is crucial to future AI applications in variable scenarios with transfer and continual learning demands.

We apply training and inference with only low-bitwidth integers in DNNs

A method called WAGE which quantizes all operands and operators in a neural network to reduce the number of bits for representation in a network.

The authors propose discretized weights, activations, gradients, and errors at both training and testing time on neural networks

Training and Inference with Integers in Deep Neural Networks

training inference
neural network
integer


Modern Convolutional Neural Networks (CNNs) are complex, encompassing millions of parameters.

Their deployment exerts computational, storage and energy demands, particularly on embedded platforms.

Existing approaches to prune or sparsify CNNs require retraining to maintain inference accuracy.

Such retraining is not feasible in some contexts.

In this paper, we explore the sparsification of CNNs by proposing three model-independent methods.

Our methods are applied on-the-fly and require no retraining.

We show that the state-of-the-art models' weights can be reduced by up to 73% (compression factor of 3.7x) without incurring more than 5% loss in Top-5 accuracy.

Additional fine-tuning gains only 8% in sparsity, which indicates that our fast on-the-fly methods are effective.

In this paper, we develop fast retraining-free  sparsification methods that can be deployed for on-the-fly sparsification of CNNs in many industrial contexts.

This paper proposes approaches for pruning CNNs without retraining by introducing three schemes to determine the thresholds of pruning weights.

This paper describes a method for sparsification of CNNs without retraining.

Fast On-the-fly Retraining-free Sparsification of Convolutional Neural Networks

sparsification cnn
method
without retrain
retraining-free
on-the-fly
fast


Curriculum learning and Self paced learning are popular topics in the machine learning that suggest to put the training samples in order by considering their difficulty levels.

Studies in these topics show that starting with a small training set and adding new samples according to difficulty levels improves the learning performance.

In this paper we experimented that we can also obtain good results by adding the samples randomly without a meaningful order.

We compared our method with classical training, Curriculum learning, Self paced learning and their reverse ordered versions.

Results of the statistical tests show that the proposed method is better than classical method and similar with the others.

These results point a new training regime that removes the process of difficulty level determination in Curriculum and Self paced learning and as successful as these methods.

We propose that training with growing sets stage-by-stage provides an optimization for neural networks.

The authors compare curriculum learning to learning in a random order with stages that add a new sample of examples to the previously, randomly constructed set

This paper studies the influence of ordering in the Curriculum and Self paced learning, and shows that to some extent the ordering of training instances is not important.

Training with Growing Sets: A Simple Alternative to Curriculum Learning and Self Paced Learning

training grow set
self pace
order
curriculum
learning
learn


We study the problem of learning to map, in an unsupervised way, between domains $A$ and $B$, such that the samples $\vb \in B$ contain all the information that exists in samples $\va\in A$ and some additional information.

For example, ignoring occlusions, $B$ can be people with glasses, $A$ people without, and the glasses, would be the added information.

When mapping a sample $\va$ from the first domain to the other domain, the missing information is replicated from an independent reference sample $\vb\in B$.

Thus, in the above example, we can create, for every person without glasses a version with the glasses observed in any face image.

Our solution employs a single two-pathway encoder and a single decoder for both domains.

The common part of the two domains and the separate part are encoded as two vectors, and the separate part is fixed at zero for domain $A$.

The loss terms are minimal and involve reconstruction losses for the two domains and a domain confusion term.

Our analysis shows that under mild assumptions, this architecture, which is much simpler than the literature guided-translation methods, is enough to ensure disentanglement between the two domains.

We present convincing results in a few visual domains, such as no-glasses to glasses, adding facial hair based on a reference image, etc.

An image to image translation method which adds to one image the content of another thereby creating a new image.

This paper tackles the task of content transfer, with the novalty being on the loss.

Emerging Disentanglement in Auto-Encoder Based Unsupervised Image Content Transfer


Mathematical reasoning---a core ability within human intelligence---presents some unique challenges as a domain: we do not come to understand and solve mathematical problems primarily on the back of experience and evidence, but on the basis of inferring, learning, and exploiting laws, axioms, and symbol manipulation rules.

In this paper, we present a new challenge for the evaluation (and eventually the design) of neural architectures and similar system, developing a task suite of mathematics problems involving sequential questions and answers in a free-form textual input/output format.

The structured nature of the mathematics domain, covering arithmetic, algebra, probability and calculus, enables the construction of training and test spits designed to clearly illuminate the capabilities and failure-modes of different architectures, as well as evaluate their ability to compose and relate knowledge and learned processes.

Having described the data generation process and its potential future expansions, we conduct a comprehensive analysis of models from two broad classes of the most powerful sequence-to-sequence architectures and find notable differences in their ability to resolve mathematical problems and generalize their knowledge.

A dataset for testing mathematical reasoning (and algebraic generalization), and results on current sequence-to-sequence models.

Presents a new synthetic dataset to evaluate the mathematical reasoning ability of sequence-to-sequence models, and uses it to evaluate several models.

Analysing Mathematical Reasoning Abilities of Neural Models

mathematical reasoning ability
sequence-to-sequence model
dataset


Convolutional Neural Networks (CNNs) filter the input data using a series of spatial convolution operators with compactly supported stencils and point-wise nonlinearities.

Commonly, the convolution operators couple features from all channels.

For wide networks, this leads to immense computational cost in the training of and prediction with CNNs.

In this paper, we present novel ways to parameterize the convolution more efficiently, aiming to decrease the number of parameters in CNNs and their computational complexity.

We propose new architectures that use a sparser coupling between the channels and thereby reduce both the number of trainable weights and the computational cost of the CNN.

Our architectures arise as new types of residual neural network (ResNet) that can be seen as discretizations of a Partial Differential Equations (PDEs) and thus have predictable theoretical properties.

Our first architecture involves a convolution operator with a special sparsity structure, and is applicable to a large class of CNNs.

Next, we present an architecture that can be seen as a discretization of a diffusion reaction PDE, and use it with three different convolution operators.

We outline in our experiments that the proposed architectures,  although considerably reducing the number of trainable weights, yield comparable accuracy to existing CNNs that are fully coupled in the channel dimension.

This paper introduces efficient and economic parametrizations of convolutional neural networks motivated by partial differential equations

Introduces four "low cost" alternatives to the standard convolution operation that can be used in place of the standard convolution operation to reduce their computational complexity.

This paper introduces methods for reducing the computational cost of CNN implementations, and introduces new parameterizations of CNN like architectures that limit parameter coupling.

The paper proposes a PDE-based perspective to understand and parameterize CNNs

Low-Cost Parameterizations of Deep Convolutional Neural Networks

reduce computational
convolutional neural network
introduce
parameterization
cost
cnn


In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model?

One way to address this question is to find an upper bound on the probability (equivalently a lower bound on the negative log likelihood) that the model can assign to some data as one varies the prior and/or the likelihood function in a latent variable model.

The core of our contribution is to formally show that the problem of optimizing priors in latent variable models is exactly an instance of the variational optimization problem that information theorists solve when computing rate-distortion functions, and then to use this to derive a lower bound on negative log likelihood.

Moreover, we will show that if changing the prior can improve the log likelihood, then there is a way to change the likelihood function instead and attain the same log likelihood, and thus rate-distortion theory is of relevance to both optimizing priors as well as optimizing likelihood functions.

We will experimentally argue for the usefulness of quantities derived from rate-distortion theory in latent variable modeling by applying them to a problem in image modeling.

Use rate-distortion theory to bound how much a latent variable model can be improved

Addresses problems of optimization of the prior in the latent variable model and the selection of the likelihood function by proposing criteria based on a lower-bound on the negative log-likelihood.

Presents a theorem which gives a lower bound on negative log likelihood of rate-distortion for latent-variable modeling

The authors argue that the rate-distortion theory for lossy compression provides a natural toolkit for studying latent variable models proposes a lower bound.

Information Theoretic lower bounds on negative log likelihood

latent variable model
rate-distortion theory
negative log likelihood
bound
low


Backprop is the primary learning algorithm used in many machine learning algorithms.

In practice, however, Backprop in deep neural networks is a highly sensitive learning algorithm and successful learning depends on numerous conditions and constraints.

One set of constraints is to avoid weights that lead to saturated units.

The motivation for avoiding unit saturation is that gradients vanish and as a result learning comes to a halt.

Careful weight initialization and re-scaling schemes such as batch normalization ensure that input activity to the neuron is within the linear regime where gradients are not vanished and can flow.

Here we investigate backpropagating error terms only linearly.

That is, we ignore the saturation that arise by ensuring gradients always flow.

We refer to this learning rule as Linear Backprop since in the backward pass the network appears to be linear.

In addition to ensuring persistent gradient flow, Linear Backprop is also favorable when computation is expensive since gradients are never computed.

Our early results suggest that learning with Linear Backprop is competitive with Backprop and saves expensive gradient computations.

We ignore non-linearities and do not compute gradients in the backward pass to save computation and to ensure gradients always flow.

The author proposed linear backprop algorithms to ensure gradients flow for all parts during backpropagation.

Linear Backprop in non-linear networks


Deep neural networks with discrete latent variables offer the promise of better symbolic reasoning, and learning  abstractions that are more useful to new tasks.

There has been a surge in interest in discrete latent variable models,  however, despite several recent improvements, the training of discrete latent variable models has remained  challenging and their performance has mostly failed to match their continuous counterparts.

Recent work on vector quantized autoencoders (VQ-VAE) has made substantial progress in this direction, with its perplexity almost matching that of a VAE on datasets such as CIFAR-10.

In this work, we investigate an alternate training technique for VQ-VAE, inspired by its connection to the Expectation Maximization (EM) algorithm.

Training the discrete autoencoder with EM and combining it with sequence  level knowledge distillation alows us to develop a non-autoregressive machine translation model whose accuracy almost matches a strong greedy autoregressive baseline Transformer, while being 3.3 times faster at inference.

Understand the VQ-VAE discrete autoencoder systematically using EM and use it to design non-autogressive translation model matching a strong autoregressive baseline.

This paper introduces a new way of interpreting the VQ-VAE and proposes a new training algorithm based on the soft EM clustering.

The paper presents an alternative view on the training procedure for the VQ-VAE using the soft EM algorithm

Towards a better understanding of Vector Quantized Autoencoders

vq-vae soft autoencoder training algorithm


Recent research about margin theory has proved that maximizing the minimum margin like support vector machines does not necessarily lead to better performance, and instead, it is crucial to optimize the margin distribution.

In the meantime, margin theory has been used to explain the empirical success of deep network in recent studies.

In this paper, we present ODN (the Optimal margin Distribution Network), a network which embeds a loss function in regard to the optimal margin distribution.

We give a theoretical analysis for our method using the PAC-Bayesian framework, which confirms the significance of the margin distribution for classification within the framework of deep networks.

In addition, empirical results show that the ODN model always outperforms the baseline cross-entropy loss model consistently across different regularization situations.

model also outperforms the cross-entropy loss (Xent), hinge loss and soft hinge loss model in generalization task through limited training data.

This paper presents a deep neural network embedding a loss function in regard to the optimal margin distribution, which alleviates the overfitting problem theoretically and empirically.

Presents a PAC-Bayesian bound for a margin loss

Optimal margin Distribution Network

optimal margin distribution
present
network
loss


Deep network compression seeks to reduce the number of parameters in the network while maintaining a certain level of performance.

Deep network distillation seeks to train a smaller network that matches soft-max performance of a larger network.

While both regimes have led to impressive performance for their respective goals, neither provide insight into the importance of a given layer in the original model, which is useful if we are to improve our understanding of these highly parameterized models.

In this paper, we present the concept of deep net triage, which individually assesses small blocks of convolution layers to understand their collective contribution to the overall performance, which we call \emph{criticality}.  We call it triage because we assess this criticality by answering the question: what is the impact to the health of the overall network if we compress a block of layers into a single layer.

We propose a suite of triage methods and compare them on problem spaces of varying complexity.

We ultimately show that, across these problem spaces, deep net triage is able to indicate the of relative importance of different layers.

Surprisingly, our local structural compression technique also leads to an improvement in overall accuracy when the final model is fine-tuned globally.

We seek to understand learned representations in compressed networks via an experimental regime we call deep net triage

Compares various initialization and training methods of transferring knowledge from VGG network to a smaller student network by replacing blocks of layers with single layers.

This paper presents five methods for doing triaging or block layer compression for deep networks.

The paper proposes a method to compress a block of layers in a NN that evaluates several different sub-approaches

Deep Net Triage: Assessing The Criticality of Network Layers by Structural Compression

block layer
deep net triage
method
compress
network
compression


In this paper, we show a phenomenon, which we named ``super-convergence'', where residual networks can be trained using an order of magnitude fewer iterations than is used with standard training methods.

The existence of super-convergence is relevant to understanding why deep networks generalize well.

One of the key elements of super-convergence is training with cyclical learning rates and a large maximum learning rate.

Furthermore, we present evidence that training with large learning rates improves performance by regularizing the network.

In addition, we show that super-convergence provides a  greater boost in performance relative to standard training when the amount of labeled training data is limited.

We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate.

The architectures to replicate this work will be made available upon publication.

Empirical proof of a new phenomenon requires new theoretical insights and is relevent to the active discussions in the literature on SGD and understanding generalization.

The paper discusses a phenomenon where neural network training in very specific settings can profit much from a schedule including large learning rates

The authors analyze training of residual networks using large cyclic learning rates, and demonstrate fast convergence with cyclic learning rates and evidence of large learning rates acting as regularization.

Super-Convergence: Very Fast Training of Residual Networks Using Large Learning Rates

training residual network large
learning rate
phenomenon
fast


Infinite-width neural networks have been extensively used to study the theoretical properties underlying the extraordinary empirical success of standard, finite-width neural networks.

Nevertheless, until now, infinite-width networks have been limited to at most two hidden layers.

To address this shortcoming, we study the initialisation requirements of these networks and show that the main challenge for constructing them is defining the appropriate sampling distributions for the weights.

Based on these observations, we propose a principled approach to weight initialisation that correctly accounts for the functional nature of the hidden layer activations and facilitates the construction of arbitrarily many infinite-width layers, thus enabling the construction of arbitrarily deep infinite-width networks.

The main idea of our approach is to iteratively reparametrise the hidden-layer activations into appropriately defined reproducing kernel Hilbert spaces and use the canonical way of constructing probability distributions over these spaces for specifying the required weight distributions in a principled way.

Furthermore, we examine the practical implications of this construction for standard, finite-width networks.

In particular, we derive a novel weight initialisation scheme for standard, finite-width networks that takes into account the structure of the data and information about the task at hand.

We demonstrate the effectiveness of this weight initialisation approach on the MNIST, CIFAR-10 and Year Prediction MSD datasets.

We propose a method for the construction of arbitrarily deep infinite-width networks, based on which we derive a novel weight initialisation scheme for finite-width networks and demonstrate its competitive performance.

Proposes a weight initialization approach to enable infinitely deep and infinite-width networks with experimental results on small datasets.

Proposes deep neural networks of infinite width.

Infinitely Deep Infinite-Width Networks

deep infinite-width network
weight
infinitely


Working memory requires information about external stimuli to be represented in the brain even after those stimuli go away.

This information is encoded in the activities of neurons, and neural activities change over timescales of tens of milliseconds.

Information in working memory, however, is retained for tens of seconds, suggesting the question of how time-varying neural activities maintain stable representations.

Prior work shows that, if the neural dynamics are in the `  null space' of the representation - so that changes to neural activity do not affect the downstream read-out of stimulus information - then information can be retained for periods much longer than the time-scale of individual-neuronal activities.

The prior work, however, requires precisely constructed synaptic connectivity matrices, without explaining how this would arise in a biological neural network.

To identify mechanisms through which biological networks can self-organize to learn  memory function, we derived biologically plausible synaptic plasticity rules that dynamically modify the connectivity matrix to enable information storing.

Networks implementing this plasticity rule can successfully learn to form memory representations even if only 10% of the synapses are plastic, they are robust to synaptic noise, and they can represent information about multiple stimuli.

We derived biologically plausible synaptic plasticity learning rules for a recurrent neural network to store stimulus representations.

A neural network model consisting of recurrently connected neurons and one or more redouts which aims to retain some output over time.

This paper presents a self-organizing memory mechanism in a neural model, and introduces an objective function that minimizes changes in the signal to be memorized.

A Self-Organizing Memory Network

neural network
self-organizing memory
model


Generative Adversarial Networks (GANs) have been proposed as an approach to learning generative models.

While GANs have demonstrated promising performance on multiple vision tasks, their learning dynamics are not yet well understood, neither in theory nor in practice.

In particular, the work in this domain has been focused so far only on understanding the properties of the stationary solutions that this dynamics might converge to, and of the behavior of that dynamics in this solutions’ immediate neighborhood.

To address this issue, in this work we take a first step towards a principled study of the GAN dynamics itself.

To this end, we propose a model that, on one hand, exhibits several of the common problematic convergence behaviors (e.g., vanishing gradient, mode collapse, diverging or oscillatory behavior), but on the other hand, is sufficiently simple to enable rigorous convergence analysis.

This methodology enables us to exhibit an interesting phenomena: a GAN with an optimal discriminator provably converges, while guiding the GAN training using only a first order approximation of the discriminator leads to unstable GAN dynamics and mode collapse.

This suggests that such usage of the first order approximation of the discriminator, which is a de-facto standard in all the existing GAN dynamics, might be one of the factors that makes GAN training so challenging in practice.

Additionally, our convergence result constitutes the first rigorous analysis of a dynamics of a concrete parametric GAN.

To understand GAN training, we define simple GAN dynamics, and show quantitative differences between optimal and first order updates in this model.

The authors study the impact of GANs in settings where at each iteration, the discriminator trains to convergence and the generator updates with gradient steps, or where a few gradient steps are done for the disciminator and generator.

This paper studies the dynamics of adversarial training of GANs on a Gaussian mixture model

On the limitations of first order approximation in GAN dynamics

dynamic
first order
gan
training
update
study
model


The machine learning and computer vision community is witnessing an unprecedented rate of new tasks being proposed and addressed, thanks to the power of deep convolutional networks to find complex mappings from X to Y. The advent of each task often accompanies the release of a large-scale human-labeled dataset, for supervised training of the deep network.

However, it is expensive and time-consuming to manually label sufficient amount of training data.

Therefore, it is important to develop algorithms that can leverage off-the-shelf labeled dataset to learn useful knowledge for the target task.

While previous works mostly focus on transfer learning from a single source, we study multi-source transfer across domains and tasks (MS-DTT), in a semi-supervised setting.

We propose GradMix, a model-agnostic method applicable to any model trained with gradient-based learning rule.

GradMix transfers knowledge via gradient descent, by weighting and mixing the gradients from all sources during training.

Our method follows a meta-learning objective, by assigning layer-wise weights to the source gradients, such that the combined gradient follows the direction that can minimize the loss for a small set of samples from the target dataset.

In addition, we propose to adaptively adjust the learning rate for each mini-batch based on its importance to the target task, and a pseudo-labeling method to leverage the unlabeled samples in the target domain.

We perform experiments on two MS-DTT tasks: digit recognition and action recognition, and demonstrate the advantageous performance of the proposed method against multiple baselines.

We propose a gradient-based method to transfer knowledge from multiple sources across different domains and tasks.

This paper proposes to combine the gradients of source domains to help the learning in the target domain.

GradMix: Multi-source Transfer across Domains and Tasks


Bayesian phylogenetic inference is currently done via Markov chain Monte Carlo with simple mechanisms for proposing new states, which hinders exploration efficiency and often requires long runs to deliver accurate posterior estimates.

In this paper we present an alternative approach: a variational framework for Bayesian phylogenetic analysis.

We approximate the true posterior using an expressive graphical model for tree distributions, called a subsplit Bayesian network, together with appropriate branch length distributions.

We train the variational approximation via stochastic gradient ascent and adopt multi-sample based gradient estimators for different latent variables separately to handle the composite latent space of phylogenetic models.

We show that our structured variational approximations are flexible enough to provide comparable posterior estimation to MCMC, while requiring less computation due to a more efficient tree exploration mechanism enabled by variational inference.

Moreover, the variational approximations can be readily used for further statistical analysis such as marginal likelihood estimation for model comparison via importance sampling.

Experiments on both synthetic data and real data Bayesian phylogenetic inference problems demonstrate the effectiveness and efficiency of our methods.

The first variational Bayes formulation of phylogenetic inference, a challenging inference problem over structures with intertwined discrete and continuous components

Explores an approximate inference solution to the problem of Bayesian inference of phylogenetic trees by leveraging recently proposed subsplit Bayesian networks and modern gradient estimators for VI.

Proposes a variational approach to Bayesian posterior inference in phylogenetic trees.

Variational Bayesian Phylogenetic Inference

inference phylogenetic tree
problem
bayesian
variational


This paper introduces HybridNet, a hybrid neural network to speed-up autoregressive

models for raw audio waveform generation.

a hybrid model that combines an autoregressive network named WaveNet and a

conventional LSTM model to address speech synthesis.

one sample per time-step, the proposed HybridNet generates multiple samples per

time-step by exploiting the long-term memory utilization property of LSTMs.

the evaluation, when applied to text-to-speech, HybridNet yields state-of-art performance.

HybridNet achieves a 3.83 subjective 5-scale mean opinion score on

US English, largely outperforming the same size WaveNet in terms of naturalness

It is a hybrid neural architecture to speed-up autoregressive model.

Concludes that in order to scale up the model size without increasing inference time for sequential prediction, use a model that predicts multiple timesteps at once.

This paper presents HybridNet, a neural speech and other audio synthesis system that combines the WaveNet model with an LSTM with the goal of offering a model with faster inference-time audio generation.

HybridNet: A Hybrid Neural Architecture to Speed-up Autoregressive  Models

hybrid neural architecture speed-up autoregressive


Visual Interpretation and explanation of deep models is critical towards wide adoption of systems that rely on them.

In this paper, we propose a novel scheme for both interpretation as well as explanation in which, given a pretrained model, we automatically identify internal features relevant for the set of classes considered by the model, without relying on additional annotations.

We interpret the model through average visualizations of this reduced set of features.

Then, at test time, we explain the network prediction by accompanying the predicted class label with supporting visualizations derived from the identified features.

In addition, we propose a method to address the artifacts introduced by strided operations in deconvNet-based visualizations.

Moreover, we introduce an8Flower , a dataset specifically designed for objective quantitative evaluation of methods for visual explanation.

Experiments on the MNIST , ILSVRC 12, Fashion 144k and an8Flower datasets show that our method produces detailed explanations with good coverage of relevant features of the classes of interest.

Interpretation by Identifying model-learned features that serve as indicators for the task of interest. Explain model decisions by highlighting the response of these features in test data. Evaluate explanations objectively with a controlled dataset.

This paper proposes a method for producing visual explanations for deep neural network outputs and releases a new synthetic dataset.

A method for Deep Neural Networks that identifies automatically relevant features of the set of the classes, supporting interpretation and explanation without relying on additional annotations.

Visual Explanation by Interpretation: Improving Visual Feedback Capabilities of Deep Neural Networks

deep neural network
feature
method
visual explanation
interpretation
identify
dataset


In this work we propose a simple and efficient framework for learning sentence representations from unlabelled data.

Drawing inspiration from the distributional hypothesis and recent work on learning sentence representations, we reformulate the problem of predicting the context in which a sentence appears as a classification problem.

Given a sentence and the context in which it appears, a classifier distinguishes context sentences from other contrastive sentences based on their vector representations.

This allows us to efficiently learn different types of encoding functions, and we show that the model learns high-quality sentence representations.

We demonstrate that our sentence representations outperform state-of-the-art unsupervised and supervised representation learning methods on several downstream NLP tasks that involve understanding sentence semantics while achieving an order of magnitude speedup in training time.

A framework for learning high-quality sentence representations efficiently.

Proposes a faster algorithm for learning SkipThought-style sentence representations from corpora of ordered sentences that swaps the word-level decoder for a contrastive classification loss.

This paper proposes a framework for unsupervised learning of sentence representations by maximizing a model of the probability of true context sentences relative to random candidate sentences

An efficient framework for learning sentence representations


Many regularization methods have been proposed to prevent overfitting in neural networks.

Recently, a regularization method has been proposed to optimize the variational lower bound of the Information Bottleneck Lagrangian.

However, this method cannot be generalized to regular neural network architectures.

We present the activation norm penalty that is derived from the information bottleneck principle and is theoretically grounded in a variation dropout framework.

Unlike in previous literature, it can be applied to any general neural network.

We demonstrate that this penalty can give consistent improvements to different state of the art architectures both in language modeling and image classification.

We present analyses on the properties of this penalty and compare it to other methods that also reduce mutual information.

We derive a norm penalty on the output of the neural network from the information bottleneck perspective

Puts forward Activation Norm Penalty, an L_2 type regularization on the activations, deriving it from the Information Bottleneck principle

This paper creates a mapping between activation norm penalties and information bottleneck framework using variational dropout framework.

From Information Bottleneck To Activation Norm Penalty

activation norm penalty
information bottleneck
derive


Unsupervised learning of timeseries data is a challenging problem in machine learning.

Here,
we propose a novel algorithm, Deep Temporal Clustering (DTC), a fully unsupervised method, to naturally integrate dimensionality reduction and temporal clustering into a single end to end learning framework.

The algorithm starts with an initial cluster estimates using an autoencoder for dimensionality reduction and a novel temporal clustering layer for cluster assignment.

Then it jointly optimizes the clustering objective and the dimensionality reduction objective.

Based on requirement and application, the temporal clustering layer can be customized with any temporal similarity metric.

Several similarity metrics are considered and compared.

To gain insight into features that the network has learned for its clustering, we apply a visualization method that generates a heat map of regions of interest in the timeseries.

The viability of the algorithm is demonstrated using timeseries data from diverse domains, ranging from earthquakes to sensor data from spacecraft.

In each case, we show that our algorithm outperforms traditional methods.

This performance is attributed to fully integrated temporal dimensionality reduction and clustering criterion.

A fully unsupervised method, to naturally integrate dimensionality reduction and temporal clustering into a single end to end learning framework.

Proposes an algorithm that integrates autoencoder with time-series data clustering using a network structure that suits time-series data.

An algorithm for jointly performing dimensionality reduction and temporal clustering in a deep learning context, utilizing an autoencoder and clustering objective.

The authors proposed an unsupervised time series clustering methods built with deep neural networks and equipped with an encoder-decoder and a clustering mode to shorten the time series, extract local temporal features, and to get the encoded representations.

Deep Temporal Clustering: Fully unsupervised learning of time-domain features

dimensionality reduction temporal clustering


We study many-class few-shot (MCFS) problem in both supervised learning and meta-learning scenarios.

Compared to the well-studied many-class many-shot and few-class few-shot problems, MCFS problem commonly occurs in practical applications but is rarely studied.

MCFS brings new challenges because it needs to distinguish between many classes, but only a few samples per class are available for training.

In this paper, we propose ``memory-augmented hierarchical-classification network (MahiNet)'' for MCFS learning.

It addresses the ``many-class'' problem by exploring the class hierarchy, e.g., the coarse-class label that covers a subset of fine classes, which helps to narrow down the candidates for the fine class and is cheaper to obtain.

MahiNet uses a convolutional neural network (CNN) to extract features, and integrates a memory-augmented attention module with a multi-layer perceptron (MLP) to produce the probabilities over coarse and fine classes.

While the MLP extends the linear classifier, the attention module extends a KNN classifier, both together targeting the ''`few-shot'' problem.

We design different training strategies of MahiNet for supervised learning and meta-learning.

Moreover, we propose two novel benchmark datasets ''mcfsImageNet'' (as a subset of ImageNet) and ''mcfsOmniglot'' (re-splitted Omniglot) specifically for MCFS problem.

In experiments, we show that MahiNet outperforms several state-of-the-art models on MCFS classification tasks in both supervised learning and meta-learning scenarios.

A memory-augmented neural network that addresses many-class few-shot problem by leveraging class hierarchy in both supervised learning and meta-learning.

This paper presents methods for adding inductive bias to a classifier through coarse-to-fine prediction along a class hierarchy and learning a memory-based KNN classifier that keeps track of mislabeled instances during learning.

This paper formulates the many-class-few-shot classification problem from a supervised learning perspective and a meta-learning perspective.

MahiNet: A Neural Network for Many-Class Few-Shot Learning with Class Hierarchy

class hierarchy
supervised learning
meta-learning
neural network
many-class few-shot
problem


Learning a better representation with neural networks is a challenging problem, which has been tackled from different perspectives in the past few years.

In this work, we focus on learning a representation that would be useful in a clustering task.

We introduce two novel loss components that substantially improve the quality of produced clusters, are simple to apply to arbitrary models and cost functions, and do not require a complicated training procedure.

We perform an extensive set of experiments, supervised and unsupervised, and evaluate the proposed loss components on two most common types of models, Recurrent Neural Networks and Convolutional Neural Networks, showing that the approach we propose consistently improves the quality of KMeans clustering in terms of mutual information scores and outperforms previously proposed methods.

A novel loss component that forces the network to learn a representation that is well-suited for clustering during training for a classification task.

This paper proposes two regularization terms based on a compound hinge loss over the KL divergence between two softmax-normalized input arguments to encourage learning disentangled representations

Proposal for two regularizers intended to make the representations learned in the penultimate layer of a classifier more conforming to inherent structure in the data.

Forced Apart: Discovering Disentangled Representations Without Exhaustive Labels

learn representation two loss force disentangle


In high dimensions, the performance of nearest neighbor algorithms depends crucially on structure in the data.

While traditional nearest neighbor datasets consisted mostly of hand-crafted feature vectors, an increasing number of datasets comes from representations learned with neural networks.

We study the interaction between nearest neighbor algorithms and neural networks in more detail.

We find that the network architecture can significantly influence the efficacy of nearest neighbor algorithms even when the classification accuracy is unchanged.

Based on our experiments, we propose a number of training modifications that lead to significantly better datasets for nearest neighbor algorithms.

Our modifications lead to learned representations that can accelerate nearest neighbor queries by 5x.

We show how to get good representations from the point of view of Simiarity Search.

Studies the impact of changing the image classification part on top of the DNN on the ability to index the descriptors with a LSH or a kd-tree algorithm.

Proposes to use softmax cross-entropy loss to learn a network that tries to reduce the angles between inputs and the corresponding class vectors in a supervised framework using.

Learning Representations for Faster Similarity Search


Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices.

In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure.

Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid.

These are subsequently relaxed to continuous surrogates that can allow for efficient gradient-based optimization.

We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent.

We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classification.

We introduce a technique that allows for gradient based training of quantized neural networks.

Proposes a unified and general way of training neural networks with reduced precision quantized synaptic weights and activations.

A new approach to quantizing activations which is state of the art or competitive on several real image problems.

A method for learning neural networks with quantized weights and activations by stochastically quantizing values and replacing the resulting categotical distribution with a continuous relaxation

Relaxed Quantization for Discretized Neural Networks

weight activation
neural network
quantize


In most current formulations of adversarial training, the discriminators can be expressed as single-input operators, that is, the mapping they define is separable over observations.

In this work, we argue that this property might help explain the infamous mode collapse phenomenon in adversarially-trained generative models.

Inspired by discrepancy measures and two-sample tests between probability distributions, we propose distributional adversaries that operate on samples, i.e., on sets of multiple points drawn from a distribution, rather than on single observations.

We show how they can be easily implemented on top of existing models.

Various experimental results show that generators trained in combination with our distributional adversaries are much more stable and are remarkably less prone to mode collapse than traditional models trained with observation-wise prediction discriminators.

In addition, the application of our framework to domain adaptation results in strong improvement over recent state-of-the-art.

We show that the mode collapse problem in GANs may be explained by a lack of information sharing between observations in a training batch, and propose a distribution-based framework for globally sharing information between gradients that leads to more stable and effective adversarial training.

Proposes to replace single-sample discriminators in adversarial training with discriminators that explicitly operate on distributions of examples.

Theory on two-sample tests and MMD and how can be beneficially incorporated into GAN framework.

Distributional Adversarial Networks


Chemical information extraction is to convert chemical knowledge in text into true chemical database, which is a text processing task heavily relying on chemical compound name identification and standardization.

Once a systematic name for a chemical compound is given, it will naturally and much simply convert the name into the eventually required molecular formula.

However, for many chemical substances, they have been shown in many other names besides their systematic names which poses a great challenge for this task.

In this paper, we propose a framework to do the auto standardization from the non-systematic names to the corresponding systematic names by using the spelling error correction, byte pair encoding tokenization and neural sequence to sequence model.

Our framework is trained end to end and is fully data-driven.

Our standardization accuracy on the test dataset achieves 54.04% which has a great improvement compared to previous state-of-the-art result.

We designed an end-to-end framework using sequence to sequence model to do the  chemical names standardization.

Standardizes non systematic names in chemical information extraction by creating a parallel corpus of non-systematic and systematic names and building a seq2seq model.

This work presents a method to translate non-systematic names of chemical compounds into their systematic equivalents using a combination of mechanisms

CHEMICAL NAMES STANDARDIZATION USING NEURAL SEQUENCE TO SEQUENCE MODEL

sequence sequence model
chemical name standardization
systematic
non-systematic