Existing approaches to neural machine translation condition each output word on previously generated outputs.

We introduce a model that avoids this autoregressive property and produces its outputs in parallel, allowing an order of magnitude lower latency during inference.

Through knowledge distillation, the use of input token fertilities as a latent variable, and policy gradient fine-tuning, we achieve this at a cost of as little as 2.0 BLEU points relative to the autoregressive Transformer network used as a teacher.

We demonstrate substantial cumulative improvements associated with each of the three aspects of our training strategy, and validate our approach on IWSLT 2016 English–German and two WMT language pairs.

By sampling fertilities in parallel at inference time, our non-autoregressive model achieves near-state-of-the-art performance of 29.8 BLEU on WMT 2016 English–Romanian.

We introduce the first NMT model with fully parallel decoding, reducing inference latency by 10x.

This work proposes non-autoregressive decoder for the encoder-decoder framework in which the decision of generating a word does not depends on the prior decision of generated words

This paper describes an approach to decode non-autoregressively for neural machine translation with the possibility of more parallel decoding which can result in a significant speed-up.

Proposes the introduction of a set of latent variables to represent the fertility of each source word to make the target sentence generation non-autoregressive

Non-Autoregressive Neural Machine Translation

neural machine translation
decode
word
parallel
non-autoregressive


While neural networks have achieved high accuracy on standard image classification benchmarks, their accuracy drops to nearly zero in the presence of small adversarial perturbations to test inputs.

Defenses based on regularization and adversarial training have been proposed, but often followed by new, stronger attacks that defeat these defenses.

In this work, we study this problem for neural networks with one hidden layer.

We first propose a method based on a semidefinite relaxation that outputs a certificate that for a given network and test input, no attack can force the error to exceed a certain value.

Second, as this certificate is differentiable, we jointly optimize it with the network parameters, providing an adaptive regularizer that encourages robustness against all attacks.

On MNIST, our approach produces a network and a certificate that no that perturbs each pixel by at most $\epsilon = 0.1$ can cause more than $35\%$ test error.

We demonstrate a certifiable, trainable, and scalable method for defending against adversarial examples.

Proposes a new defense against security attacks on neural networks with the atack model that outputs a security certificate on the algorithm.

Derives an upper bound on adversarial perturbation for neural networks with one hidden layer

Certified Defenses against Adversarial Examples

neural network
adversarial example
defense


We formulate an information-based optimization problem for supervised classification.

For invertible neural networks, the control of these information terms is passed down to the latent features and parameter matrix in the last fully connected layer, given that mutual information is invariant under invertible map.

We propose an objective function and prove that it solves the optimization problem.

Our framework allows us to learn latent features in an more interpretable form while improving the classification performance.

We perform extensive quantitative and qualitative experiments in comparison with the existing state-of-the-art classification models.

we propose a regularizer that improves the classification performance of neural networks

the authors propose to train a model from a point of maximizing mutual information between the predictions and the true outputs, with a regularization term that minimizes irrelevant information while learning.

Proposes to decompose the parameters into an invertible feature map F and a linear transformation w in the last layer to maximize mutual information I(Y, \hat{T}) while constraining irrelevant information

Information Regularized Neural Networks

maximize mutual information
neural network
irrelevant


Powerful generative models, particularly in Natural Language Modelling, are commonly trained by maximizing a variational lower bound on the data log likelihood.

These models often suffer from poor use of their latent variable, with ad-hoc annealing factors used to encourage retention of information in the latent variable.

We discuss an alternative and general approach to latent variable modelling, based on an objective that encourages a perfect reconstruction by tying a stochastic autoencoder with a variational autoencoder (VAE).

This ensures by design that the latent variable captures information about the observations, whilst retaining the ability to generate well.

Interestingly, although our model is fundamentally different to a VAE, the lower bound attained is identical to the standard VAE bound but with the addition of a simple pre-factor; thus, providing a formal interpretation of the commonly used, ad-hoc pre-factors in training VAEs.

This paper introduces a novel generative modelling framework that avoids latent-variable collapse and clarifies the use of certain ad-hoc factors in training Variational Autoencoders.

The paper proposes to resolve the issue about a variational auto-encoder ignoring the latent variables.

This paper proposes adding a stochastic autoencoder to the original VAE model to address the problem that the LSTM decoder of a language model might be too strong to ignore the latent variable's information.

This paper presents AutoGen, which combines a generative variational autoencoder with a high-fidelity reconstruction model based on autoencoder to better utiliza latent representation

Improving latent variable descriptiveness by modelling rather than ad-hoc factors

ignore latent variable
ad-hoc factor
variational autoencoder
model
generative


This paper studies the problem of domain division which aims to segment instances drawn from different probabilistic distributions.

This problem exists in many previous recognition tasks, such as Open Set Learning (OSL) and Generalized Zero-Shot Learning (G-ZSL), where the testing instances come from either seen or unseen/novel classes with different probabilistic distributions.

Previous works only calibrate the conﬁdent prediction of classiﬁers of seen classes (WSVM Scheirer et al. (2014)) or taking unseen classes as outliers Socher et al. (2013).

In contrast, this paper proposes a probabilistic way of directly estimating and ﬁne-tuning the decision boundary between seen and unseen classes.

In particular, we propose a domain division algorithm to split the testing instances into known, unknown and uncertain domains, and then conduct recognition tasks in each domain.

Two statistical tools, namely, bootstrapping and KolmogorovSmirnov (K-S) Test, for the ﬁrst time, are introduced to uncover and ﬁne-tune the decision boundary of each domain.

Critically, the uncertain domain is newly introduced in our framework to adopt those instances whose domain labels cannot be predicted conﬁdently.

Extensive experiments demonstrate that our approach achieved the state-of-the-art performance on OSL and G-ZSL benchmarks.

This paper studies the problem of domain division by segmenting instances drawn from different probabilistic distributions.

This paper deals with the problem of novelty recognition in open set learning and generalized zero-shot learning and proposes a possible solution

An approach to domain separation based on bootstrapping to identify similarity cutoff thresholds for known classes, followed by a Kolmogorov-Smirnoff test to refine the bootstrapped in-distribution zones.

Proposes to introduce a new domain, the uncertain domain, to better handle the division between seen/unseen domains in open-set and generalized zero-shot learning

Learning to Separate Domains in Generalized Zero-Shot and Open Set Learning: a probabilistic perspective

open set learning
problem
generalized zero-shot
division
domain
probabilistic


Stochastic gradient descent (SGD) is widely believed to perform implicit regularization when used to train deep neural networks, but the precise manner in which this occurs has thus far been elusive.

We prove that SGD minimizes an average potential over the posterior distribution of weights along with an entropic regularization term.

This potential is however not the original loss function in general.

So SGD does perform variational inference, but for a different loss than the one used to compute the gradients.

Even more surprisingly, SGD does not even converge in the classical sense: we show that the most likely trajectories of SGD for deep networks do not behave like Brownian motion around critical points.

Instead, they resemble closed loops with deterministic components.

We prove that such out-of-equilibrium behavior is a consequence of highly non-isotropic gradient noise in SGD; the covariance matrix of mini-batch gradients for deep networks has a rank as small as 1% of its dimension.

We provide extensive empirical validation of these claims, proven in the appendix.

SGD implicitly performs variational inference; gradient noise is highly non-isotropic, so SGD does not even converge to critical points of the original loss

This paper provides a variational analysis of SGD as a non-equilibrium process.

This paper discusses the regularized objective function minimized by standard SGD in the context of neural nets, and provide a variational inference perspective using the Fokker-Planck equation.

Develops a theory to study the impact of stocastic gradient noise for SGD, especially for deep neural network models

Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks

provide variational
sgd
gradient noise
converge
deep
perform
inference
neural
network


The current dominant paradigm for imitation learning relies on strong supervision of expert actions to learn both 'what' and 'how' to imitate.

We pursue an alternative paradigm wherein an agent first explores the world without any expert supervision and then distills its experience into a goal-conditioned skill policy with a novel forward consistency loss.

In our framework, the role of the expert is only to communicate the goals (i.e., what to imitate) during inference.

The learned policy is then employed to mimic the expert (i.e., how to imitate) after seeing just a sequence of images demonstrating the desired task.

Our method is 'zero-shot' in the sense that the agent never has access to expert actions during training or for the task demonstration at inference.

We evaluate our zero-shot imitator in two real-world settings: complex rope manipulation with a Baxter robot and navigation in previously unseen office environments with a TurtleBot.

Through further experiments in VizDoom simulation, we provide evidence that better mechanisms for exploration lead to learning a more capable policy which in turn improves end task performance.

Videos, models, and more details are available at https://pathak22.github.io/zeroshot-imitation/.

Agents can learn to imitate solely visual demonstrations (without actions) at test time after learning from their own experience without any form of supervision at training time.

This paper proposes and approach for zero-shot visual learning by learning parametric skill functions.

A paper about imitation of a task presented just during inference, where learning is performed in a self-supervised manner and during training the agent explores related but different tasks.

Proposes a method for sidestepping the issue of expensive expert demonstration by using the random exploration of an agent to learn generalizable skills which can be applied without specific pretraining

Zero-Shot Visual Imitation

agent learn
demonstration
zero-shot visual
learning
without
skill
imitation


Distributional Semantics Models(DSM) derive word space from linguistic items

Meaning is obtained by defining a distance measure between vectors

work concentrates on quality of word embeddings, improvement of word embedding

vectors, applicability of a novel similarity metric used ‘on top’ of the

In this paper we provide comparison between two methods

for post process improvements to the baseline DSM vectors.

method which enforces antonymy and synonymy constraints into the Paragram

vector space representations recently showed improvement in the vectors’ capability

method applied to GloVe baseline vectors.

method, implementing relational knowledge into the model by retrofitting synonyms

and providing a new ranking similarity definition RESM that gives maximum

weight to the top vector component values we equal the results for the ESL

and TOEFL sets in comparison with our calculations using the Paragram and Paragram

For SIMLEX-999 gold standard since we cannot

use the RESM the results using GloVe and PPDB are significantly worse compared

Apparently, counter-fitting corrects hubness.

or our cosine retrofitting method are state-of-the-art results for the SIMLEX-999

They are 0.2 better for SIMLEX-999 than word2vec with sense

de-conflation (that was announced to be state-of the-art method for less reliable

Apparently relational knowledge and counter-fitting is more important

for judging semantic similarity than sense determination for words.

be mentioned, though that Paragram hyperparameters are fitted to SIMLEX-999

The lesson is that many corrections to word embeddings are necessary

and methods with more parameters and hyperparameters perform better.

Paper provides a description of a procedure to enhance word vector space model with an evaluation of Paragram and GloVe models for Similarity Benchmarks.

This paper suggests a new algorithm that adjusts GloVe word vectors and then uses a non-Euclidean similarity function between them.

The authors present observations on the weaknesses of the existing vector space models and list a 6-step approach for refining existing word vectors

Comparison of Paragram and GloVe Results for Similarity Benchmarks

vector space model
paragram glove
similarity benchmark
word


Recurrent neural networks have achieved excellent performance in many applications.

However, on portable devices with limited resources, the models are often too large to deploy.

For applications on the server with large scale concurrent requests, the latency during inference can also be very critical for costly computing resources.

In this work, we address these problems by quantizing the network, both weights and activations, into multiple binary codes {-1,+1}. We formulate the quantization as an optimization problem.

Under the key observation that once the quantization coefficients are fixed the binary codes can be derived efficiently by binary search tree, alternating minimization is then applied.

We test the quantization for two well-known RNNs, i.e., long short term memory (LSTM) and gated recurrent unit (GRU), on the language models.

Compared with the full-precision counter part, by 2-bit quantization we can achieve ~16x memory saving and  ~6x real inference acceleration on CPUs, with only a reasonable loss in the accuracy.

By 3-bit quantization, we can achieve almost no loss in the accuracy or even surpass the original model, with ~10.5x memory saving and ~3x real inference acceleration.

Both results beat the exiting quantization works with large margins.

We extend our alternating quantization to image classification tasks.

In both RNNs and feedforward neural networks, the method also achieves  excellent performance.

We propose a  new  quantization method and apply it to quantize RNNs for both compression and acceleration

This paper proposes a multi-bit quantization method for recurrent neural networks.

A technique for quantizing neural network weight matrices, and an alternating optimization procedure to estimate the set of k binary vectors and coefficients that best represent the original vector.

Alternating Multi-bit Quantization for Recurrent Neural Networks

recurrent neural network
quantization method
quantize
multi-bit
alternate


The goal of this paper is to demonstrate a method for tensorizing neural networks based upon an efficient way of approximating scale invariant quantum states, the Multi-scale Entanglement Renormalization Ansatz (MERA).

We employ MERA as a replacement for linear layers in a neural network and test this implementation on the CIFAR-10 dataset.

The proposed method outperforms factorization using tensor trains, providing greater compression for the same level of accuracy and greater accuracy for the same level of compression.

We demonstrate MERA-layers with 3900 times fewer parameters and a reduction in accuracy of less than 1% compared to the equivalent fully connected layers.

We replace the fully connected layers of a neural network with the multi-scale entanglement renormalization ansatz, a type of quantum operation which describes long range correlations.

In the paper the authors suggest to use MERA tensorization technique for compressing neural networks.

A new parameterization of linear maps for neural network use, using a hierarchical factorization of the linear map that reduces the number of parameters while still allowing for relatively complex interactions to be modelled.

Studies compressing feed forward layers using low rank tensor decompositions and explore a tree like decomposition

Compact Neural Networks based on the Multiscale Entanglement Renormalization Ansatz

neural network
entanglement renormalization ansatz
layer
compress


Deep learning models have outperformed traditional methods in many fields such

as natural language processing and computer vision.

tremendous success, the methods of designing optimal Convolutional Neural Networks

(CNNs) are still based on heuristics or grid search.

obtained using these techniques are often overparametrized with huge computational

This paper focuses on a structured, explainable

approach towards optimal model design that maximizes accuracy while keeping

We propose a single-shot analysis of a trained CNN

that uses Principal Component Analysis (PCA) to determine the number of filters

that are doing significant transformations per layer, without the need for retraining.

It can be interpreted as identifying the dimensionality of the hypothesis space

The proposed technique also helps estimate an optimal number

of layers by looking at the expansion of dimensions as the model gets deeper.

This analysis can be used to design an optimal structure of a given network on

a dataset, or help to adapt a predesigned network on a new dataset.

these techniques by optimizing VGG and AlexNet networks on CIFAR-10,

We present a single shot analysis of a trained neural network to remove redundancy and identify optimal network structure

This paper proposes a set of heuristics for identifying a good neural network architecture, based on PCA of unit activations over the dataset

This paper presents a framework for optimising neural networks architectures through the identification of redundant filters across layers

A SINGLE SHOT PCA-DRIVEN ANALYSIS OF NETWORK STRUCTURE TO REMOVE REDUNDANCY

neural network architecture
single shot
remove redundancy
present
identify
analysis
structure


Recent work has introduced attacks that extract the architecture information of deep neural networks (DNN), as this knowledge enhances an adversary’s capability to conduct attacks on black-box networks.

This paper presents the first in-depth security analysis of DNN fingerprinting attacks that exploit cache side-channels.

First, we define the threat model for these attacks:  our adversary does not need the ability to query the victim model; instead, she runs a co-located process on the host machine victim ’s deep learning  (DL) system is running and passively monitors the accesses of the target functions in the shared framework.

Second, we introduce DeepRecon, an attack that reconstructs the architecture of the victim network by using the internal information extracted via Flush+Reload, a cache side-channel technique.

Once the attacker observes function invocations that map directly to architecture attributes of the victim network, the attacker can reconstruct the victim’s entire network architecture.

In our evaluation, we demonstrate that an attacker can accurately reconstruct two complex networks (VGG19 and ResNet50) having only observed one forward propagation.

Based on the extracted architecture attributes, we also demonstrate that an attacker can build a meta-model that accurately fingerprints the architecture and family of the pre-trained model in a transfer learning setting.

From this meta-model,  we evaluate the importance of the observed attributes in the fingerprinting process.

Third, we propose and evaluate new framework-level defense techniques that obfuscate our attacker’s observations.

Our empirical security analysis represents a step toward understanding the DNNs’ vulnerability to cache side-channel attacks.

We conduct the first in-depth security analysis of DNN fingerprinting attacks that exploit cache side-channels, which represents a step toward understanding the DNN’s vulnerability to side-channel attacks.

This paper considers the problem of fingerprinting neural network architectures using cache side channels, and discusses security-through-obscurity defenses.

This paper performs cache side-channel attacks to extract attributes of a victim model and infer its architecture, as well as show they can achieve a nearly perfect classification accuracy.

Security Analysis of Deep Neural Networks Operating in the Presence of Cache Side-Channel Attacks

cache side-channel attack
security analysis
architecture
neural network


Learning with a primary objective, such as softmax cross entropy for classification and sequence generation, has been the norm for training deep neural networks for years.

Although being a widely-adopted approach, using cross entropy as the primary objective exploits mostly the information from the ground-truth class for maximizing data likelihood, and largely ignores information from the complement (incorrect) classes.

We argue that, in addition to the primary objective, training also using a complement objective that leverages information from the complement classes can be effective in improving model performance.

This motivates us to study a new training paradigm that maximizes the likelihood of the ground-truth class while neutralizing the probabilities of the complement classes.

We conduct extensive experiments on multiple tasks ranging from computer vision to natural language understanding.

The experimental results confirm that, compared to the conventional training with just one primary objective, training also with the complement objective further improves the performance of the state-of-the-art models across all tasks.

In addition to the accuracy improvement, we also show that models trained with both primary and complement objectives are more robust to single-step adversarial attacks.

We propose Complement Objective Training (COT), a new training paradigm that optimizes both the primary and complement objectives for effectively learning the parameters of neural networks.

Considers augmenting the cross-entropy objective with "complement" objective maximization, which aims at neutralizing the predicted probabilities of classes other than the ground truth labels.

The authors propose a secondary objective for softmax minimization based on evaluating the information gathered from the incorrect classes, leading to a new training approach.

Deals with the training of neural networks for classification or sequence generation tasks using across-entropy loss

Complement Objective Training

neural network
new training
complement objective
class


We present a new method for uncertainty estimation and out-of-distribution detection in neural networks with softmax output.

We extend softmax layer with an additional constant input.

The corresponding additional output is able to represent the uncertainty of the network.

The proposed method requires neither additional parameters nor multiple forward passes nor input preprocessing nor out-of-distribution datasets.

We show that our method performs comparably to more computationally expensive methods and outperforms baselines on our experiments from image recognition and sentiment analysis domains.

Uncertainty estimation in a single forward pass without additional learnable parameters.

A new method for computing output uncertainty estimates in DNNs for classification problems that matches state-of-the-art methods for uncertainty estimation and outperforms them in out-of-distribution detection tasks.

The authors present inhibited softmax, a modification of the softmax through adding a constant activation which provides a measure for uncertainty.

Inhibited Softmax for Uncertainty Estimation in Neural Networks


When deep learning is applied to sensitive data sets, many privacy-related implementation issues arise.

These issues are especially evident in the healthcare, finance, law and government industries.

Homomorphic encryption could allow a server to make inferences on inputs encrypted by a client, but to our best knowledge, there has been no complete implementation of common deep learning operations, for arbitrary model depths, using homomorphic encryption.

This paper demonstrates a novel approach, efficiently implementing many deep learning functions with bootstrapped homomorphic encryption.

As part of our implementation, we demonstrate Single and Multi-Layer Neural Networks, for the Wisconsin Breast Cancer dataset, as well as a Convolutional Neural Network for MNIST.

Our results give promising directions for privacy-preserving representation learning, and the return of data control to users.

We made a feature-rich system for deep learning with encrypted inputs, producing encrypted outputs, preserving privacy.

A framework for private deep learning model inference using FHE schemes that support fast bootstrapping and thus can reduce computation time.

The paper presents a means of evaluating a neural network securely using homomorphic encryption.

Deep Learning Inferences with Hybrid Homomorphic Encryption

deep learning
inference
homomorphic encryption


In this paper, we introduce a system called GamePad that can be used to explore the application of machine learning methods to theorem proving in the Coq proof assistant.

Interactive theorem provers such as Coq enable users to construct machine-checkable proofs in a step-by-step manner.

Hence, they provide an opportunity to explore theorem proving with human supervision.

We use GamePad to synthesize proofs for a simple algebraic rewrite problem and train baseline models for a formalization of the Feit-Thompson theorem.

We address position evaluation (i.e., predict the number of proof steps left) and tactic prediction (i.e., predict the next proof step) tasks, which arise naturally in tactic-based theorem proving.

We introduce a system called GamePad to explore the application of machine learning methods to theorem proving in the Coq proof assistant.

This paper describes a system for applying machine learning to interactive theorem proving, focuses on tasks of tactic prediction and position evaluation, and shows that a neural model outperforms an SVM on both tasks.

Proposes that machine learning techniques be used to help build proof in the theorem prover Coq.

GamePad: A Learning Environment for Theorem Proving

machine learn
theorem proving
coq
system
gamepad
proof


Deep neural networks are usually huge, which significantly limits the deployment on low-end devices.

weight-quantized models have  been proposed.

They have small storage and fast inference, but training can still be time-consuming.

This can be improved with distributed learning.

To reduce the high communication cost due to worker-server synchronization, recently gradient quantization has also been proposed to train deep networks with full-precision weights.

In this paper, we theoretically study how the combination of both weight and gradient quantization affects convergence.

(i) weight-quantized models converge to an error related to the weight quantization resolution and weight dimension;

(ii) quantizing gradients slows convergence by a factor related to the gradient quantization resolution and dimension; and

(iii) clipping the gradient before quantization renders this factor dimension-free, thus allowing the use of fewer bits for gradient quantization.

Empirical experiments confirm the theoretical convergence results, and demonstrate that quantized networks can speed up training and have comparable performance as full-precision networks.

In this paper, we studied efficient training of loss-aware weight-quantized  networks with  quantized gradient  in a distributed environment, both theoretically and empirically.

This paper studies convergence properties of loss-aware weight quantization with different gradient precisions in the distributed environment, and provides convergence analysis for weight quantization with full-precision, quantized and quantized clipped gradients.

The authors proposes an analysis of the effect of simultaneously quantizing the weights and gradients in training a parametrized model in a fully-synchronized distributed environment.

Analysis of Quantized Models

distribute environment
study
loss-aware
analysis
quantize
gradient
weight
model


Sequential learning, also called lifelong learning, studies the problem of learning tasks in a sequence with access restricted to only the data of the current task.

In this paper we look at a scenario with fixed model capacity, and postulate that the learning process should not be selfish, i.e. it should account for future tasks to be added and thus leave enough capacity for them.

To achieve Selfless Sequential Learning we study different regularization strategies and activation functions.

imposing sparsity at the level of the representation (i.e. neuron activations) is more beneficial for sequential learning than encouraging parameter sparsity.

In particular, we propose a novel regularizer, that encourages representation sparsity by means of neural inhibition.

It results in few active neurons which in turn leaves more free neurons to be utilized by upcoming tasks.

As neural inhibition over an entire layer can be too drastic, especially for complex tasks requiring strong representations,

our regularizer only inhibits other neurons in a local neighbourhood, inspired by lateral inhibition processes in the brain.

We combine our novel regularizer with state-of-the-art lifelong learning methods that penalize changes to important previously learned parts of the network.

We show that our new regularizer leads to increased sparsity which translates in consistent performance improvement on diverse datasets.

A regularization strategy for improving the performance of sequential learning

A novel, regularization based approach to the sequential learning problem using a fixed size model that adds extra terms to the loss, encouraging representation sparsity and combating catastrophic forgetting.

This paper deals with the problem of catastrophic forgetting in lifelong learning by proposing regularized learning strategies

Selfless Sequential Learning

sequential learning
problem
catastrophic forgetting
regularization
strategy


A Synaptic Neural Network (SynaNN) consists of synapses and neurons.

Inspired by the synapse research of neuroscience, we built a synapse model with a nonlinear synapse function of excitatory and inhibitory channel probabilities.

Introduced the concept of surprisal space and constructed a commutative diagram, we proved that the inhibitory probability function -log(1-exp(-x)) in surprisal space is the topologically conjugate function of the inhibitory complementary probability 1-x in probability space.

Furthermore, we found that the derivative of the synapse over the parameter in the surprisal space is equal to the negative Bose-Einstein distribution.

In addition, we constructed a fully connected synapse graph (tensor) as a synapse block of a synaptic neural network.

Moreover, we proved the gradient formula of a cross-entropy loss function over parameters, so synapse learning can work with the gradient descent and backpropagation algorithms.

In the proof-of-concept experiment, we performed an MNIST training and testing on the MLP model with synapse network as hidden layers.

A synaptic neural network with synapse graph and learning that has the feature of topological conjugation and Bose-Einstein distribution in surprisal space.

The authors propose a hybrid neural nework composed of a synapse graph that can be embedded into a standard neural network

Presents a biologically-inspired neural network model based on the excitatory and inhibitory ion channels in the membranes of real cells

A Synaptic Neural Network and Synapse Learning

synaptic neural network synapse
graph
learn


Many types of relations in physical, biological, social and information systems can be modeled as homogeneous or heterogeneous concept graphs.

Hence, learning from and with graph embeddings has drawn a great deal of research interest recently, but only ad hoc solutions have been obtained this far.

In this paper, we conjecture that the one-shot supervised learning mechanism is a bottleneck in improving the performance of the graph embedding learning algorithms, and propose to extend this by introducing a multi-shot unsupervised learning framework.

Empirical results on several real-world data set show that the proposed model consistently and significantly outperforms existing state-of-the-art approaches on knowledge base completion and graph based multi-label classification tasks.

A generalized knowledge graph embedding approach which learns the embeddings based on three different simultaneous objectives, and performs on par or even outperforms existing state-of-the art approaches.

Tackles the task of learning embeddings of multi-relational graphs using a neural network

Proposes a new method, GEN, to compute embeddings of multirelationship graphs, particularly that so-called E-Cells and R-Cells can answer queries of the form (h,r,?),(?r,t), and (h,?,t)

Generalized Graph Embedding Models

generalize graph embed model
learn embedding


We introduce and study minimax curriculum learning (MCL), a new method for adaptively selecting a sequence of training subsets for a succession of stages in machine learning.

The subsets are encouraged to be small and diverse early on, and then larger, harder, and allowably more homogeneous in later stages.

At each stage, model weights and training sets are chosen by solving a joint continuous-discrete minimax optimization, whose objective is composed of a continuous loss (reflecting training set hardness) and a discrete submodular promoter of diversity for the chosen subset.

MCL repeatedly solves a sequence of such optimizations with a schedule of increasing training set size and decreasing pressure on diversity encouragement.

We reduce MCL to the minimization of a surrogate function handled by submodular maximization and continuous gradient methods.

We show that MCL achieves better performance and, with a clustering trick, uses fewer labeled samples for both shallow and deep models while achieving the same performance.

Our method involves repeatedly solving constrained submodular maximization of an only slowly varying function on the same ground set.

Therefore, we develop a heuristic method that utilizes the previous submodular maximization solution as a warm start for the current submodular maximization process to reduce computation while still yielding a guarantee.

Minimax Curriculum Learning is a machine teaching method involving increasing desirable hardness and scheduled reducing diversity.

A curriculum learning approach using a submodular set function that captures the diversity of examples chosen during training.

The paper introduces MiniMax Curriculum learning as an approach for adaptively training models by providing it different subsets of data.

Minimax Curriculum Learning: Machine Teaching with Desirable Difficulties and Scheduled Diversity

minimax curriculum learn machine teaching


Progress in probabilistic generative models has accelerated, developing richer models with neural architectures, implicit densities, and with scalable algorithms for their Bayesian inference.

However, there has been limited progress in models that capture causal relationships, for example, how individual genetic factors cause major human diseases.

In this work, we focus on two challenges in particular: How do we build richer causal models, which can capture highly nonlinear relationships and interactions between multiple causes?

How do we adjust for latent confounders, which are variables influencing both cause and effect and which prevent learning of causal relationships?

To address these challenges, we synthesize ideas from causality and modern probabilistic modeling.

For the first, we describe implicit causal models, a class of causal models that leverages neural architectures with an implicit density.

For the second, we describe an implicit causal model that adjusts for confounders by sharing strength across examples.

In experiments, we scale Bayesian inference on up to a billion genetic measurements.

We achieve state of the art accuracy for identifying causal factors: we significantly outperform the second best result by an absolute difference of 15-45.3%.

Implicit models applied to causality and genetics

The authors propose to use the implicit model to tackle Genome-Wide Association problem.

This paper proposes solutions for the problems in genome-wide association studies of confounding due to population structure and the potential presence of non-linear interactions between different parts of the genome, and bridges statistical genetics and ML.

Presents a non-linear generative model for GWAS that models population structure where non-linearities are modeled using neural networks as non-linear function approximators and inference is performed using likelihood-free variational inference

Implicit Causal Models for Genome-wide Association Studies

genome-wide association study
population structure
genetic
implicit model
problem
non-linear


Few-shot learning trains image classifiers over datasets with few examples per category.

It poses challenges for the optimization algorithms, which typically require many examples to fine-tune the model parameters for new categories.

Distance-learning-based approaches avoid the optimization issue by embedding the images into a metric space and applying the nearest neighbor classifier for new categories.

In this paper, we propose to exploit the object-level relation to learn the image relation feature, which is converted into a distance directly.

For a new category, even though its images are not seen by the model, some objects may appear in the training images.

Hence, object-level relation is useful for inferring the relation of images from unseen categories.

Consequently, our model generalizes well for new categories without fine-tuning.

Experimental results on benchmark datasets show that our approach outperforms state-of-the-art methods.

Few-shot learning by exploiting the object-level relation to learn the image-level relation (similarity)

This paper deals with the problem of few-shot learning by proposing an embedding-based approach that learns to compare object-level features between support and query set examples

Proposes a few shot learning method that exploits the object-level relation between different images based on neared neighbor search and concatenates feature maps of two input images into one feature map

Few-Shot Learning by Exploiting Object Relation

exploit object-level relation
learn
few-shot learning
feature


Word embeddings are widely used in machine learning based natural language processing systems.

It is common to use pre-trained word embeddings which provide benefits such as reduced training time and improved overall performance.

There has been a recent interest in applying natural language processing techniques to programming languages.

However, none of this recent work uses pre-trained embeddings on code tokens.

Using extreme summarization as the downstream task, we show that using pre-trained embeddings on code tokens provides the same benefits as it does to natural languages, achieving: over 1.9x speedup, 5\% improvement in test loss, 4\% improvement in F1 scores, and resistance to over-fitting.

We also show that the choice of language used for the embeddings does not have to match that of the task to achieve these benefits and that even embeddings pre-trained on human languages provide these benefits to programming languages.

Researchers exploring natural language processing techniques applied to source code are not using any form of pre-trained embeddings, we show that they should be.

This paper sets to understand whether pretraining word embeddings for programming language code by using NLP-like language models has an impact on extreme code summarization task.

This work shows how pre-training word vectors using corpuses of code leads to representations that are more suitable than randomly initialized and trained representations for function/method name prediction

The Effectiveness of Pre-Trained Code Embeddings

code show pre-trained embedding language word


Recently, Approximate Policy Iteration (API) algorithms have achieved super-human proficiency in two-player zero-sum games such as Go, Chess, and Shogi without human data.

These API algorithms iterate between two policies: a slow policy (tree search), and a fast policy (a neural network).

In these two-player games, a reward is always received at the end of the game.

However, the Rubik’s Cube has only a single solved state, and episodes are not guaranteed to terminate.

This poses a major problem for these API algorithms since they rely on the reward received at the end of the game.

We introduce Autodidactic Iteration: an API algorithm that overcomes the problem of sparse rewards by training on a distribution of states that allows the reward to propagate from the goal state to states farther away.

Autodidactic Iteration is able to learn how to solve the Rubik’s Cube and the 15-puzzle without relying on human data.

Our algorithm is able to solve 100% of randomly scrambled cubes while achieving a median solve length of 30 moves — less than or equal to solvers that employ human domain knowledge.

We solve the Rubik's Cube with pure reinforcement learning

Solution to solving Rubik cube using reinforcement learning (RL) with Monte-Carlo tree search (MCTS) through autodidactic iteration.

This work solves Rubik's Cube using an approximate policy iteration method called Autodidactic iteration, overcoming the problem of sparse rewards by creating its own rewards system.

Introduces a deep RL algorithm to solve the Rubik's cube that handles the huge state space and very sparse reward of the Rubik's cube

Solving the Rubik's Cube with Approximate Policy Iteration

solve rubik 's cube approximate policy iteration


Answering compositional questions requiring multi-step reasoning is challenging for current models.

We introduce an end-to-end differentiable model for interpreting questions, which is inspired by formal approaches to semantics.

Each span of text is represented by a denotation in a knowledge graph, together with a vector that captures ungrounded aspects of meaning.

Learned composition modules recursively combine constituents, culminating in a grounding for the complete sentence which is an answer to the question.

For example, to interpret ‘not green’, the model will represent ‘green’ as a set of entities, ‘not’ as a trainable ungrounded vector, and then use this vector to parametrize a composition function to perform a complement operation.

For each sentence, we build a parse chart subsuming all possible parses, allowing the model to jointly learn both the composition operators and output structure by gradient descent.

We show the model can learn to represent a variety of challenging semantic operators, such as quantifiers, negation, disjunctions and composed relations on a synthetic question answering task.

The model also generalizes well to longer sentences than seen in its training data, in contrast to LSTM and RelNet baselines.

We describe an end-to-end differentiable model for QA that learns to represent spans of text in the question as denotations in knowledge graph, by learning both neural modules for composition and the syntactic structure of the sentence.

This paper presents a model for visual question answering that can learn both parameters and structure predictors for a modular neural network, without supervised structures or assistance from a syntactic parser.

Proposes for training a question answering model from answers only and a KB by learning latent trees that capture the syntax and learn the semantic of words

Neural Compositional Denotational Semantics for Question Answering

learn
syntactic
question answer
structure
model
semantic
neural


Deep learning software demands reliability and performance.

However, many of the existing deep learning frameworks are software libraries that act as an unsafe DSL in Python and a computation graph interpreter.

We present DLVM, a design and implementation of a compiler infrastructure with a linear algebra intermediate representation, algorithmic differentiation by adjoint code generation, domain- specific optimizations and a code generator targeting GPU via LLVM.

Designed as a modern compiler infrastructure inspired by LLVM, DLVM is more modular and more generic than existing deep learning compiler frameworks, and supports tensor DSLs with high expressivity.

With our prototypical staged DSL embedded in Swift, we argue that the DLVM system enables a form of modular, safe and performant frameworks for deep learning.

We introduce a novel compiler infrastructure that addresses shortcomings of existing deep learning frameworks.

Proposal to move from ad-hoc code generation in deep learning engines to compiler and languages best practices.

This paper presents a compiler framework that allows definition of domain-specific languages for deep learning systems, and defines compilation stages that can take advantage of standard optimizations and specialized optimizations for neural networks.

This paper introduces a DLVM to take advantage of the compiler aspects of a tensor compiler

DLVM: A modern compiler infrastructure for deep learning systems

take advantage
deep learning system
introduce
compiler infrastructure
dlvm
framework
language


In this work, we focus on the problem of grounding language by training an agent

to follow a set of natural language instructions and navigate to a target object

The agent receives visual information through raw

pixels and a natural language instruction telling what task needs to be achieved.

Other than these two sources of information, our model does not have any prior

information of both the visual and textual modalities and is end-to-end trainable.

We develop an attention mechanism for multi-modal fusion of visual and textual

modalities that allows the agent to learn to complete the navigation tasks and also

Our experimental results show that our attention

mechanism outperforms the existing multi-modal fusion mechanisms proposed in

order to solve the above mentioned navigation task.

visualization of attention weights that our model learns to correlate attributes of

the object referred in the instruction with visual representations and also show

that the learnt textual representations are semantically meaningful as they follow

vector arithmetic and are also consistent enough to induce translation between instructions

effectively to unseen scenarios and exhibit zero-shot generalization capabilities.

In order to simulate the above described challenges, we introduce a new 2D environment

for an agent to jointly learn visual and textual modalities

Attention based architecture for language grounding via reinforcement learning in a new customizable 2D grid environment

The paper tackles the problem of navigation given an instruction and proposes an approach to combine textual and visual information via an attention mechanism

This paper considers the problem of following natural language instructions given a first-person view of an a priori unknown environment, and proposes a neural architecture method.

Studies the problem of navigating to a target object in a 2D grid environment by following given natural language description and receiving visual information as raw pixels.

Learning to navigate by distilling visual information and natural language instructions

2d grid environment
problem
visual information
natural language instruction
navigate
give
follow
via
architecture
attention


Current end-to-end machine reading and question answering (Q\&A) models are primarily based on recurrent neural networks (RNNs) with attention.

Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs.

We propose a new Q\&A architecture called QANet, which does not require recurrent networks:  Its encoder consists exclusively of convolution and self-attention, where convolution models local interactions and self-attention models global interactions.

On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference, while achieving equivalent accuracy to recurrent models.

The speed-up gain allows us to train the model with much more data.

We hence combine our model with data generated by backtranslation from a neural machine translation model.

On the SQuAD dataset, our single model, trained with augmented data, achieves 84.6 F1 score on the test set, which is significantly better than the best published F1 score of 81.8.

A simple architecture consisting of convolutions and attention achieves results on par with the best documented recurrent models.

A fast high performance paraphrasing based data augmentation method and a non-recurrent reading comprehension model using only convolutions and attention.

This paper proposes applying CNNs+self-attention modules instead of LSTMs and enhancing the RC model training with passage paraphrases generated by a neural paraphrasing model in order to improve RC performance.

This paper presents a reading comprehension model using convolutions and attention and propose to augment additional training data by paraphrasing based on off-the-shelf neural machine translation

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

reading comprehension model convolution attention


Convolutional Neural Networks (CNNs) have become the method of choice for learning problems involving 2D planar images.

However, a number of problems of recent interest have created a demand for models that can analyze spherical images.

Examples include omnidirectional vision for drones, robots, and autonomous cars, molecular regression problems, and global weather and climate modelling.

A naive application of convolutional networks to a planar projection of the spherical signal is destined to fail, because the space-varying distortions introduced by such a projection will make translational weight sharing ineffective.

In this paper we introduce the building blocks for constructing spherical CNNs.

We propose a definition for the spherical cross-correlation that is both expressive and rotation-equivariant.

The spherical correlation satisfies a generalized Fourier theorem, which allows us to compute it efficiently using a generalized (non-commutative) Fast Fourier Transform (FFT) algorithm.

We demonstrate the computational efficiency, numerical accuracy, and effectiveness of spherical CNNs applied to 3D model recognition and atomization energy regression.

We introduce Spherical CNNs, a convolutional network for spherical signals, and apply it to 3D model recognition and molecular energy regression.

The paper proposes a framework for constructing spherical convolutional networks based on a novel synthesis of several existing concepts

This paper focuses on how to extend convolutional neural networks to have built-in spherical invariance, and adapts tools from non-Abelian harmonic analysis to achieve this goal.

The authors develop a novel scheme for representing spherical data from the ground up

Spherical CNNs

network
spherical cnn
novel
convolutional


We propose a novel method that makes use of deep neural networks and gradient decent to perform automated design on complex real world engineering tasks.

Our approach works by training a neural network to mimic the fitness function of a design optimization task and then, using the differential nature of the neural network, perform gradient decent to maximize the fitness.

We demonstrate this methods effectiveness by designing an optimized heat sink and both 2D and 3D airfoils that maximize the lift drag ratio under steady state flow conditions.

We highlight that our method has two distinct benefits over other automated design approaches.

First, evaluating the neural networks prediction of fitness can be orders of magnitude faster then simulating the system of interest.

Second, using gradient decent allows the design space to be searched much more efficiently then other gradient free methods.

These two strengths work together to overcome some of the current shortcomings of automated design.

A method for performing automated design on real world objects such as heat sinks and wing airfoils that makes use of neural networks and gradient descent.

Neural network (parameterization and prediction) and gradient descent (back propogation) to automatically design for engineering tasks.

This paper introduces using a deep network to approximate the behavior of a complex physical system, and then design optimal devices by optimizing this network with respect to its inputs.

AUTOMATED DESIGN USING NEURAL NETWORKS AND GRADIENT DESCENT

neural network gradient descent
automate design


Methods that align distributions by minimizing an adversarial distance between them have recently achieved impressive results.

However, these approaches are difficult to optimize with gradient descent and they often do not converge well without careful hyperparameter tuning and proper initialization.

We investigate whether turning the adversarial min-max problem into an optimization problem by replacing the maximization part with its dual improves the quality of the resulting alignment and explore its connections to Maximum Mean Discrepancy.

Our empirical results suggest that using the dual formulation for the restricted family of linear discriminators results in a more stable convergence to a desirable solution when compared with the performance of a primal min-max GAN-like objective and an MMD objective under the same restrictions.

We test our hypothesis on the problem of aligning two synthetic point clouds on a plane and on a real-image domain adaptation problem on digits.

In both cases, the dual formulation yields an iterative procedure that gives more stable and monotonic improvement over time.

We propose a dual version of the logistic adversarial distance for feature alignment and show that it yields more stable gradient step iterations than the min-max objective.

The paper deals with fixing GANs at the computational level

This paper studies a dual formulation of an adversarial loss based on an upper-bound of the logistic loss, and turns the standard min max problem of adversarial training into a single minimization problem.

Proposes to re-formulate the GAN saddle point objective (for a logistic regression discriminator) as a minimization problem by dualizing the maximum likelihood objective for regularized logistic regression

Stable Distribution Alignment Using the Dual of the Adversarial Distance

dual
logistic
adversarial distance
minimization problem
alignment
objective
gan
stable


  There are many applications scenarios for which the computational
  performance and memory footprint of the prediction phase of Deep
  Neural Networks (DNNs) need to be optimized.

Binary Deep Neural
  Networks (BDNNs) have been shown to be an effective way of achieving
  this objective.

In this paper, we show how Convolutional Neural
  Networks (CNNs) can be implemented using binary
  representations.

Espresso is a compact, yet powerful
  library written in C/CUDA that features all the functionalities
  required for the forward propagation of CNNs, in a binary file less
  than 400KB, without any external dependencies.

Although it is mainly
  designed to take advantage of massive GPU parallelism, Espresso also
  provides an equivalent CPU implementation for CNNs.

Espresso
  provides special convolutional and dense layers for BCNNs,
  leveraging bit-packing and bit-wise computations
  for efficient execution.

These techniques provide a speed-up of
  matrix-multiplication routines, and at the same time, reduce memory
  usage when storing parameters and activations.

We experimentally
  show that Espresso is significantly faster than existing
  implementations of optimized binary neural networks (~ 2
  orders of magnitude).

Espresso is released under the Apache 2.0
  license and is available at http://github.com/organization/project.

state-of-the-art computational performance implementation of binary neural networks

The paper presents a library written in C/CUDA that features all the functionalities required for the forward propagation of BCNNs

This paper builds on Binary-NET and expands it to CNN architectures, provides optimizations that improve the speed of the forward pass, and provides optimized code for Binary CNN.

Espresso: Efficient Forward Propagation for Binary Deep Neural Networks

forward
binary
neural network
propagation


Optimal selection of a subset of items from a given set is a hard problem that requires combinatorial optimization.

In this paper, we propose a subset selection algorithm that is trainable with gradient based methods yet achieves near optimal performance via submodular optimization.

We focus on the task of identifying a relevant set of sentences for claim verification in the context of the FEVER task.

Conventional methods for this task look at sentences on their individual merit and thus do not optimize the informativeness of sentences as a set.

We show that our proposed method which builds on the idea of unfolding a greedy algorithm into a computational graph allows both interpretability and gradient based training.

The proposed differentiable greedy network (DGN) outperforms discrete optimization algorithms as well as other baseline methods in terms of precision and recall.

We propose a subset selection algorithm that is trainable with gradient based methods yet achieves near optimal performance via submodular optimization.

Proposes a neural network based model that integrates submodular function by combining gradient based optimization technique with submodular framework named 'Differentiable Greedy Network' (DGN).

Proposes a neural network that aims to select a subset of elements (e.g. selecting k sentences that are mostly related to a claim from a set of retrieved docs)

Differentiable Greedy Networks

neural network
subset
gradient base
differentiable greedy
optimization
submodular


The joint optimization of representation learning and clustering in the embedding space has experienced a breakthrough in recent years.

In spite of the advance, clustering with representation learning has been limited to flat-level categories, which oftentimes involves cohesive clustering with a focus on instance relations.

To overcome the limitations of flat clustering, we introduce hierarchically clustered representation learning (HCRL), which simultaneously optimizes representation learning and hierarchical clustering in the embedding space.

Specifically, we place a nonparametric Bayesian prior on embeddings to handle dynamic mixture hierarchies under the variational autoencoder framework, and to adopt the generative process of a hierarchical-versioned Gaussian mixture model.

Compared with a few prior works focusing on unifying representation learning and hierarchical clustering, HCRL is the first model to consider a generation of deep embeddings from every component of the hierarchy, not just leaf components.

This generation process enables more meaningful separations and mergers of clusters via branches in a hierarchy.

In addition to obtaining hierarchically clustered embeddings, we can reconstruct data by the various abstraction levels, infer the intrinsic hierarchical structure, and learn the level-proportion features.

We conducted evaluations with image and text domains, and our quantitative analyses showed competent likelihoods and the best accuracies compared with the baselines.

We introduce hierarchically clustered representation learning (HCRL), which simultaneously optimizes representation learning and hierarchical clustering in the embedding space.

The paper proposes using the nested CRP as a clustering model rather than a topic model

Presents a novel hierarchical clustering method over an embedding space where both the embedding space and the heirarchical clustering are simultaneously learnt

Hierarchically Clustered Representation Learning

hierarchically cluster representation learning


We introduce a novel geometric perspective and unsupervised model augmentation framework for transforming traditional deep (convolutional) neural networks into adversarially robust classifiers.

Class-conditional probability densities based on Bayesian nonparametric mixtures of factor analyzers (BNP-MFA) over the input space are used to design soft decision labels for feature to label isometry.

Classconditional distributions over features are also learned using BNP-MFA to develop plug-in maximum a posterior (MAP) classifiers to replace the traditional multinomial logistic softmax classification layers.

This novel unsupervised augmented framework, which we call geometrically robust networks (GRN), is applied to CIFAR-10, CIFAR-100, and to Radio-ML (a time series dataset for radio modulation recognition).

We demonstrate the robustness of GRN models to adversarial attacks from fast gradient sign method, Carlini-Wagner, and projected gradient descent.

We develop a statistical-geometric unsupervised learning augmentation framework for deep neural networks to make them robust to adversarial attacks.

Transfroms traditional deep neural networks into adversarial robust calssifiers using GRNs

Proposes a defense based on class-conditional feature distributions to turn deep neural netwroks into robust classifiers

GEOMETRIC AUGMENTATION FOR ROBUST NEURAL NETWORK CLASSIFIERS

deep neural network
adversarial
robust
augmentation
classifier


Reinforcement learning in environments with large state-action spaces is challenging, as exploration can be highly inefficient.

Even if the dynamics are simple, the optimal policy can be combinatorially hard to discover.

In this work, we propose a hierarchical approach to structured exploration to improve the sample efficiency of on-policy exploration in large state-action spaces.

The key idea is to model a stochastic policy as a hierarchical latent variable model, which can learn low-dimensional structure in the state-action space, and to define exploration by sampling from the low-dimensional latent space.

This approach enables lower sample complexity, while preserving policy expressivity.

In order to make learning tractable, we derive a joint learning and exploration strategy by combining hierarchical variational inference with actor-critic learning.

The benefits of our learning approach are that

3) easily scalable to settings with many actions and

4) easily composable with existing deep learning approaches.

We demonstrate the effectiveness of our approach on learning a deep centralized multi-agent policy, as multi-agent environments naturally have an exponentially large state-action space.

In this setting, the latent hierarchy implements a form of multi-agent coordination during exploration and execution (MACE).

We demonstrate empirically that MACE can more efficiently learn optimal policies in challenging multi-agent games with a large number (~20) of agents, compared to conventional baselines.

Moreover, we show that our hierarchical structure leads to meaningful agent coordination.

Make deep reinforcement learning in large state-action spaces more efficient using structured exploration with deep hierarchical policies.

A method to coordinate agent behaviour by using policies that have shared latent structure, a variational policy optimization method to optimize the coordinated policies, and a derivation of the authors' variational, hierarchical update.

This paper suggests an algorithmic innovation consisting of hierarchical latent variables for coordinated exploration in multi-agent settings

Structured Exploration via Hierarchical Variational Policy Networks

policy exploration hierarchical structured latent variational


Much attention has been devoted recently to the generalization puzzle in deep learning: large, deep networks can generalize well, but existing theories bounding generalization error are exceedingly loose, and thus cannot explain this striking performance.

Furthermore, a major hope is that knowledge may transfer across tasks, so that multi-task learning can improve generalization on individual tasks.

However we lack analytic theories that can quantitatively predict how the degree of knowledge transfer depends on the relationship between the tasks.

We develop an analytic theory of the nonlinear dynamics of generalization in deep linear networks, both within and across tasks.

In particular, our theory provides analytic solutions to the training and testing error of deep networks as a function of training time, number of examples, network size and initialization, and the task structure and SNR.

Our theory reveals that deep networks progressively learn the most important task structure first, so that generalization error at the early stopping time primarily depends on task structure and is independent of network size.

This suggests any tight bound on generalization error must take into account task structure, and explains observations about real data being learned faster than random data.

Intriguingly our theory also reveals the existence of a learning algorithm that proveably out-performs neural network training through gradient descent.

Finally, for transfer learning, our theory reveals that knowledge transfer depends sensitively, but computably, on the SNRs and input feature alignments of pairs of tasks.

We provide many insights into neural network generalization from the theoretically tractable linear case.

The authors study a simple model of linear networks towards understanding generalization and transfer learning

An analytic theory of generalization dynamics and transfer learning in deep linear networks


We conduct a mathematical analysis on the Batch normalization (BN) effect on gradient backpropagation in residual network training in this work, which is believed to play a critical role in addressing the gradient vanishing/explosion problem.

Specifically, by analyzing the mean and variance behavior of the input and the gradient in the forward and backward passes through the BN and residual branches, respectively, we show that they work together to confine the gradient variance to a certain range across residual blocks in backpropagation.

As a result, the gradient vanishing/explosion problem is avoided.

Furthermore, we use the same analysis to discuss the tradeoff between depth and width of a residual network and demonstrate that shallower yet wider resnets have stronger learning performance than deeper yet thinner resnets.

Batch normalisation maintains gradient variance throughout training, thus stabilizing optimization.

This paper analyzed the effect of batch normalization on gradient backpropagation in residual networks

ANALYSIS ON GRADIENT PROPAGATION IN BATCH NORMALIZED RESIDUAL NETWORKS


To study how mental object representations are related to behavior, we estimated sparse, non-negative representations of objects using human behavioral judgments on images representative of 1,854 object categories.

These representations predicted a latent similarity structure between objects, which captured most of the explainable variance in human behavioral judgments.

Individual dimensions in the low-dimensional embedding were found to be highly reproducible and interpretable as conveying degrees of taxonomic membership, functionality, and perceptual attributes.

We further demonstrated the predictive power of the embeddings for explaining other forms of human behavior, including categorization, typicality judgments, and feature ratings, suggesting that the dimensions reflect human conceptual representations of objects beyond the specific task.

Human behavioral judgments are used to obtain sparse and interpretable representations of objects that generalize to other tasks

This paper describes a large-scale experiment on human object/sematic representations and a model of such representations.

This paper develops a new representation system for object representations from training on data collected from odd-one-out human judgements of images.

A new approach to learn a sparse, positive, interpretable semantic space that maximizes human similarity judgements by training to specifically maximize the prediction of human similarity judgements.

Revealing interpretable object representations from human behavior

object representation
sparse
interpretable
human
judgement
new


We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering.

We propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers.

The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer.

The reformulation system is trained end-to-end to maximize answer quality using policy gradient.

We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy!.

The agent outperforms a state-of-the-art base model, playing the role of the environment, and other benchmarks.

We also analyze the language that the agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting (tf-idf) and stemming.

We propose an agent that sits between the user and a black box question-answering system and which learns to reformulate questions to elicit the best possible answers

This paper proposes active question answering via a reinforcement learning approach that learns to rephrase questions in a way to provide the best possible answers.

Clearly describes how the researchers designed and actively trained two models for question reformulation and answer selection during question answering episodes

Ask the Right Questions: Active Question Reformulation with Reinforcement Learning

learn
active question
reinforcement learning
possible answer
reformulation


Most deep latent factor models choose simple priors for simplicity, tractability

the prior may have a profound effect on the expressiveness of the model,

especially when its generative network has limited capacity.

In this paper, we propose to learn a proper prior from data for adversarial autoencoders

We introduce the notion of code generators to transform manually selected

simple priors into ones that can better characterize the data distribution.

Experimental results show that the proposed model can generate better image quality and learn better disentangled representations than

AAEs in both supervised and unsupervised settings.

ability to do cross-domain translation in a  text-to-image synthesis task.

Learning Priors for Adversarial Autoencoders

Proposes a simple extension of adversarial auto-encoders for conditional image generation.

Focuses on adversarial autoencoders and introduces a code generator network to transform a simple prior into one that together with the generator can better fit the data distribution

learn prior adversarial autoencoder
simple


In the past few years, various advancements have been made in generative models owing to the formulation of Generative Adversarial Networks (GANs).

GANs have been shown to perform exceedingly well on a wide variety of tasks pertaining to image generation and style transfer.

In the field of Natural Language Processing, word embeddings such as word2vec and GLoVe are state-of-the-art methods for applying neural network models on textual data.

Attempts have been made for utilizing GANs with word embeddings for text generation.

This work presents an approach to text generation using Skip-Thought sentence embeddings in conjunction with GANs based on gradient penalty functions and f-measures.

The results of using sentence embeddings with GANs for generating text conditioned on input information are comparable to the approaches where word embeddings are used.

Generating text using sentence embeddings from Skip-Thought Vectors with the help of Generative Adversarial Networks.

Describes application of generative adversarial networks for modeling textual data with the help of ski-thought vectors and experiments with different flavors of GANs for two different datasets.

Generating Text through Adversarial Training using Skip-Thought Vectors

generative adversarial network
help
skip-thought vector
generate text


The novel \emph{Unbiased Online Recurrent Optimization} (UORO) algorithm allows for online learning of general recurrent computational graphs such as recurrent network models.

It works in a streaming fashion and avoids backtracking through past activations and inputs.

UORO is computationally as costly as \emph{Truncated Backpropagation Through Time} (truncated BPTT), a widespread algorithm for online learning of recurrent networks \cite{jaeger2002tutorial}.  UORO is a modification of \emph{NoBackTrack} \cite{DBLP:journals/corr/OllivierC15} that bypasses the need for model sparsity and makes implementation easy in current deep learning frameworks, even for complex models.

Like NoBackTrack, UORO provides unbiased gradient estimates; unbiasedness is the core hypothesis in stochastic gradient descent theory, without which convergence to a local optimum is not guaranteed.

On the contrary, truncated BPTT does not provide this property, leading to possible divergence.

On synthetic tasks where truncated BPTT is shown to diverge, UORO converges.

For instance, when a parameter has a positive short-term but negative long-term influence, truncated BPTT diverges unless the truncation span is very significantly longer than the intrinsic temporal range of the interactions, while UORO performs well thanks to the unbiasedness of its gradients.

Introduces an online, unbiased and easily implementable gradient estimate for recurrent models.

The authors introduce a novel approach to online learning of the parameters of recurrent neural networks from long sequences that overcomes the imitation of truncated backpropagation through time

This paper approaches online training of RNNs in a principled way, and proposes a modification to RTRL and to use forward approach for gradient calculation.

Unbiased Online Recurrent Optimization

approach online
recurrent
introduce
gradient
unbiased


We present a deep learning-based method for super-resolving coarse (low-resolution) labels assigned to groups of image pixels into pixel-level (high-resolution) labels, given the joint distribution between those low- and high-resolution labels.

This method involves a novel loss function that minimizes the distance between a distribution determined by a set of model outputs and the corresponding distribution given by low-resolution labels over the same set of outputs.

This setup does not require that the high-resolution classes match the low-resolution classes and can be used in high-resolution semantic segmentation tasks where high-resolution labeled data is not available.

Furthermore, our proposed method is able to utilize both data with low-resolution labels and any available high-resolution labels, which we show improves performance compared to a network trained only with the same amount of high-resolution data.

We test our proposed algorithm in a challenging land cover mapping task to super-resolve labels at a 30m resolution to a separate set of labels at a 1m resolution.

We compare our algorithm with models that are trained on high-resolution data and show that

1) we can achieve similar performance using only low-resolution data; and

2) we can achieve better performance when we incorporate a small amount of high-resolution data in our training.

We also test our approach on a medical imaging problem, resolving low-resolution probability maps into high-resolution segmentation of lymphocytes with accuracy equal to that of fully supervised models.

Super-resolving coarse labels into pixel-level labels, applied to aerial imagery and medical scans.

A method to super-resolve coarse low-res segmentation labels if the joint distribution of low-res and high-res labels are known.

Label super-resolution networks


We propose a novel framework for combining datasets via alignment of their associated intrinsic dimensions.

Our approach assumes that the two datasets are sampled from a common latent space, i.e., they measure equivalent systems.

Thus, we expect there to exist a natural (albeit unknown) alignment of the data manifolds associated with the intrinsic geometry of these datasets, which are perturbed by measurement artifacts in the sampling process.

Importantly, we do not assume any individual correspondence (partial or complete) between data points.

Instead, we rely on our assumption that a subset of data features have correspondence across datasets.

We leverage this assumption to estimate relations between intrinsic manifold dimensions, which are given by diffusion map coordinates over each of the datasets.

We compute a correlation matrix between diffusion coordinates of the datasets by considering graph (or manifold) Fourier coefficients of corresponding data features.

We then orthogonalize this correlation matrix to form an isometric transformation between the diffusion maps of the datasets.

Finally, we apply this transformation to the diffusion coordinates and construct a unified diffusion geometry of the datasets together.

We show that this approach successfully corrects misalignment artifacts, and allows for integrated data.

We propose a method for aligning the latent features learned from different datasets using harmonic correlations.

Proposes using feature correspondences to preform manifold alignment between batches of data from the same samples to avoid the collection of noisy measurements.

Manifold Alignment via Feature Correspondence

manifold alignment
feature correspondence


Reinforcement learning (RL) has proven to be a powerful paradigm for deriving complex behaviors from simple reward signals in a wide range of environments.

When applying RL to continuous control agents in simulated physics environments, the body is usually considered to be part of the environment.

However, during evolution the physical body of biological organisms and their controlling brains are co-evolved, thus exploring a much larger space of actuator/controller configurations.

Put differently, the intelligence does not reside only in the agent's mind, but also in the design of their body.

We propose a method for uncovering strong agents, consisting of a good combination of a body and policy, based on combining RL with an evolutionary procedure.

Given the resulting agent, we also propose an approach for identifying the body changes that contributed the most to the agent performance.

We use the Shapley value from cooperative game theory to find the fair contribution of individual components, taking into account synergies between components.

We evaluate our methods in an environment similar to the the recently proposed Robo-Sumo task, where agents in a 3D environment with simulated physics compete in tipping over their opponent or pushing them out of the arena.

Our results show that the proposed methods are indeed capable of generating strong agents, significantly outperforming baselines that focus on optimizing the agent policy alone.

A video is available at: www.youtube.com/watch?v=eei6Rgom3YY

Evolving the shape of the body in RL controlled agents improves their performance (and help learning)

PEOM algorithm that incorporates Shapley value to accelerate the evolution by identifying contribution of each body part

The Body is not a Given: Joint Agent Policy Learning and Morphology Evolution


Many practical reinforcement learning problems contain catastrophic states that the optimal policy visits infrequently or never.

Even on toy problems, deep reinforcement learners periodically revisit these states, once they are forgotten under a new policy.

In this paper, we introduce intrinsic fear, a learned reward shaping that accelerates deep reinforcement learning and guards oscillating policies against periodic catastrophes.

Our approach incorporates a second model trained via supervised learning to predict the probability of imminent catastrophe.

This score acts as a penalty on the Q-learning objective.

Our theoretical analysis demonstrates that the perturbed objective yields the same average return under strong assumptions and an $\epsilon$-close average return under weaker assumptions.

Our analysis also shows robustness to classification errors.

Equipped with intrinsic fear, our DQNs solve the toy environments and improve on the Atari games Seaquest, Asteroids, and Freeway.

Shape reward with intrinsic motivation to avoid catastrophic states and mitigate catastrophic forgetting.

An RL algorithm that combines the DQN algorithm with a fear model trained in parallel to predict catastropohic states.

The paper studies catastrophic forgetting in RL, by emphasizing tasks where a DQN is able to learn to avoid catastrophic events as long as it avoids forgetting.

Avoiding Catastrophic States with Intrinsic Fear

avoid catastrophic state
intrinsic
forgetting
dqn
rl
fear


Convolution is an efficient technique to obtain abstract feature representations using hierarchical layers in deep networks.

Although performing convolution in Euclidean geometries is fairly straightforward, its extension to other topological spaces---such as a sphere S^2 or a unit ball B^3---entails unique challenges.

In this work, we propose a novel `"volumetric convolution" operation that can effectively convolve arbitrary functions in B^3.

We develop a theoretical framework for "volumetric convolution" based on Zernike polynomials and efficiently implement it as a differentiable and an easily pluggable layer for deep networks.

Furthermore, our formulation leads to derivation of a  novel formula to measure the symmetry of a function in B^3 around an arbitrary axis, that is useful in 3D shape analysis tasks.

We demonstrate the efficacy of proposed volumetric convolution operation on a possible use-case i.e., 3D object recognition task.

A novel convolution operator for automatic representation learning inside unit ball

This work is related to the recent spherical CNN and SE(n) equivariant network papers and extends previous ideas to volumetric data in the unit ball.

Proposes using volumetric convolutions on convolutions networks in order to learn unit ball and discusses methodology and results of process.

Volumetric Convolution: Automatic Representation Learning in Unit Ball

automatic representation learning
unit ball
volumetric convolution
network


Learning in environments with large state and action spaces, and sparse rewards, can hinder a Reinforcement Learning (RL) agent’s learning through trial-and-error.

For instance, following natural language instructions on the Web (such as booking a flight ticket) leads to RL settings where input vocabulary and number of actionable elements on a page can grow very large.

Even though recent approaches improve the success rate on relatively simple environments with the help of human demonstrations to guide the exploration, they still fail in environments where the set of possible instructions can reach millions.

We approach the aforementioned problems from a different perspective and propose guided RL approaches that can generate unbounded amount of experience for an agent to learn from.

Instead of learning from a complicated instruction with a large vocabulary, we decompose it into multiple sub-instructions and schedule a curriculum in which an agent is tasked with a gradually increasing subset of these relatively easier sub-instructions.

In addition, when the expert demonstrations are not available, we propose a novel meta-learning framework that generates new instruction following tasks and trains the agent more effectively.

We train DQN, deep reinforcement learning agent, with Q-value function approximated with a novel QWeb neural network architecture on these smaller, synthetic instructions.

We evaluate the ability of our agent to generalize to new instructions onWorld of Bits benchmark, on forms with up to 100 elements, supporting 14 million possible instructions.

The QWeb agent outperforms the baseline without using any human demonstration achieving 100% success rate on several difficult environments.

We train reinforcement learning policies using reward augmentation, curriculum learning, and meta-learning  to successfully navigate web pages.

Develops a curriculum learning method for training an RL agent to navigate a web, based on the idea of decomposing an instruction in to multiple sub-instructions.

Learning to Navigate the Web


Labeled text classification datasets are typically only available in a few select languages.

In order to train a model for e.g news categorization in a language $L_t$ without a suitable text classification dataset there are two options.

The first option is to create a new labeled dataset by hand, and the second option is to transfer label information from an existing labeled dataset in a source language $L_s$ to the target language $L_t$. In this paper we propose a method for sharing label information across languages by means of a language independent text encoder.

The encoder will give almost identical representations to multilingual versions of the same text.

This means that labeled data in one language can be used to train a classifier that works for the rest of the languages.

The encoder is trained independently of any concrete classification task and can therefore subsequently be used for any classification task.

We show that it is possible to obtain good performance even in the case where only a comparable corpus of texts is available.

Cross Language Text Classification by universal encoding

This paper proposes an approach to cross-lingual text classification through the use of comparable corpora.

Learning cross-lingual embeddings and training a classifier using labelled data in the source language to address learning a cross-language text categorizer with no labelled information in the target language

Zero-shot Cross Language Text Classification

cross language text classification
cross-lingual


Syntax is a powerful abstraction for language understanding.

Many downstream tasks require segmenting input text into meaningful constituent chunks (e.g., noun phrases or entities); more generally, models for learning semantic representations of text benefit from integrating syntax in the form of parse trees (e.g., tree-LSTMs).

Supervised parsers have traditionally been used to obtain these trees, but lately interest has increased in unsupervised methods that induce syntactic representations directly from unlabeled text.

To this end, we propose the deep inside-outside recursive autoencoder (DIORA), a fully-unsupervised method for discovering syntax that simultaneously learns representations for constituents within the induced tree.

Unlike many prior approaches, DIORA does not rely on supervision from auxiliary downstream tasks and is thus not constrained to particular domains.

Furthermore, competing approaches do not learn explicit phrase representations along with tree structures, which limits their applicability to phrase-based tasks.

Extensive experiments on unsupervised parsing, segmentation, and phrase clustering demonstrate the efficacy of our method.

DIORA achieves the state of the art in unsupervised parsing (46.9 F1) on the benchmark WSJ dataset.

In this work we propose deep inside-outside recursive auto-encoders(DIORA)  a  fully  unsupervised  method  of  discovering  syntax  while  simultaneously learning representations for discovered constituents.

A neural latent tree model trained with an auto-encoding objective that achieves state of the art on unsupervised constituency parsing and captures syntactic structure better than other latent tree models.

The paper proposes a model for unsupervised dependency parsing (latent tree induction) that is based on a combination of the inside-outside algorithm with neural modeling (recursive auto-encoders).

Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Auto-Encoders

deep inside-outside recursive
latent tree induction
neural
unsupervised
parse
model
unsupervise
auto-encoder


Careful tuning of the learning rate, or even schedules thereof, can be crucial to effective neural net training.

There has been much recent interest in gradient-based meta-optimization, where one tunes hyperparameters, or even learns an optimizer, in order to minimize the expected loss when the training procedure is unrolled.

But because the training procedure must be unrolled thousands of times, the meta-objective must be defined with an orders-of-magnitude shorter time horizon than is typical for neural net training.

We show that such short-horizon meta-objectives cause a serious bias towards small step sizes, an effect we term short-horizon bias.

We introduce a toy problem, a noisy quadratic cost function, on which we analyze short-horizon bias by deriving and comparing the optimal schedules for short and long time horizons.

We then run meta-optimization experiments (both offline and online) on standard benchmark datasets, showing that meta-optimization chooses too small a learning rate by multiple orders of magnitude, even when run with a moderately long time horizon (100 steps) typical of work in the area.

We believe short-horizon bias is a fundamental problem that needs to be addressed if meta-optimization is to scale to practical neural net training regimes.

We investigate the bias in the short-horizon meta-optimization objective.

This paper proposes a simplified model and problem to demonstrate the short-horizon bias of the learning rate meta-optimization.

This paper studies the issue of truncated backpropagation for meta-optimization through a number of experiments on a toy problem

Understanding Short-Horizon Bias in Stochastic Meta-Optimization

bias short-horizon meta-optimization problem


Mainstream captioning models often follow a sequential structure to generate cap-

tions, leading to issues such as introduction of irrelevant semantics, lack of diversity

in the generated captions, and inadequate generalization performance.

we present an alternative paradigm for image captioning, which factorizes the

captioning procedure into two stages: (1) extracting an explicit semantic represen-

tation from the given image; and (2) constructing the caption based on a recursive

compositional procedure in a bottom-up manner.

our paradigm better preserves the semantic content through an explicit factorization

By using the compositional generation procedure, caption

construction follows a recursive structure, which naturally fits the properties of

Moreover, the proposed compositional procedure requires less

data to train, generalizes better, and yields more diverse captions.

a hierarchical and compositional way to generate captions

This paper presents a more interpretable method for image captioning.

A Neural Compositional Paradigm for Image Captioning


While many approaches to make neural networks more fathomable have been proposed, they are restricted to interrogating the network with input data.

Measures for characterizing and monitoring structural properties, however, have not been developed.

In this work, we propose neural persistence, a complexity measure for neural network architectures based on topological data analysis on weighted stratified graphs.

To demonstrate the usefulness of our approach, we show that neural persistence reflects best practices developed in the deep learning community such as dropout and batch normalization.

Moreover, we derive a neural persistence-based stopping criterion that shortens the training process while achieving comparable accuracies as early stopping based on validation loss.

We develop a new topological complexity measure for deep neural networks and demonstrate that it captures their salient properties.

This paper proposes the notion of neural persistence, a topological measure to assign scores to fully-connected layers in a neural network.

Paper proposes to analyze the complexity of a neural network using its zero-th persistent homology.

Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology

complexity measure deep neural network
persistence
topological


Deep neural networks (DNNs) are vulnerable to adversarial examples, which are carefully crafted instances aiming to cause prediction errors for DNNs.

Recent research on adversarial examples has examined local neighborhoods in the input space of DNN models.

However, previous work has limited what regions to consider, focusing either on low-dimensional subspaces or small balls.

In this paper, we argue that information from larger neighborhoods, such as from more directions and from greater distances, will better characterize the relationship between adversarial examples and the DNN models.

First, we introduce an attack, OPTMARGIN, which generates adversarial examples robust to small perturbations.

These examples successfully evade a defense that only considers a small ball around an input instance.

Second, we analyze a larger neighborhood around input instances by looking at properties of surrounding decision boundaries, namely the distances to the boundaries and the adjacent classes.

We find that the boundaries around these adversarial examples do not resemble the boundaries around benign examples.

Finally, we show that, under scrutiny of the surrounding decision boundaries, our OPTMARGIN examples do not convincingly mimic benign examples.

Although our experiments are limited to a few specific attacks, we hope these findings will motivate new, more evasive attacks and ultimately, effective defenses.

Looking at decision boundaries around an input gives you more information than a fixed small neighborhood

The authors present a novel attack for generating adversarial examples where they attack classifiers created by randomly classifying L2 small perturbations

A new approach to generate adversarial attacks to a neural network, and a method to defend a neural network from those attacks.

Decision Boundary Analysis of Adversarial Examples

generate adversarial
attack
decision boundary
small
example


Machine learning models are usually tuned by nesting optimization of model weights inside the optimization of hyperparameters.

We give a method to collapse this nested optimization into joint stochastic optimization of both weights and hyperparameters.

Our method trains a neural network to output approximately optimal weights as a function of hyperparameters.

We show that our method converges to locally optimal weights and hyperparameters for sufficiently large hypernets.

We compare this method to standard hyperparameter optimization strategies and demonstrate its effectiveness for tuning thousands of hyperparameters.

We train a neural network to output approximately optimal weights as a function of hyperparameters.

Hyper-networks for hyper-parameter optimization in neural networks.

Stochastic Hyperparameter Optimization through Hypernetworks

neural network
optimization
hyperparameter


Estimating covariances between financial assets plays an important role in risk management.

In practice, when the sample size is small compared to the number of variables, the empirical estimate is known to be very unstable.

Here, we propose a novel covariance estimator based on the Gaussian Process Latent Variable Model (GP-LVM).

Our estimator can be considered as a non-linear extension of standard factor models with readily interpretable parameters reminiscent of market betas.

Furthermore, our Bayesian treatment naturally shrinks the sample covariance matrix towards a more structured matrix given by the prior and thereby systematically reduces estimation errors.

Finally, we discuss some financial applications of the GP-LVM model.

Covariance matrix estimation of financial assets with Gaussian Process Latent Variable Models

Illustrates how the Gaussian Process Latent Variable Model (GP-LVM) can replace classical linear factor models for the estimation of covariance matrices in portfolio optimization problems.

This paper uses standard GPLVMs to model the covariance structure and a latent space representation of S&P500 financial time series, to optimize portfolios and predict missing values.

This paper proposes to use a GPLVM to model financial returns

Applications of Gaussian Processes in Finance

gaussian process latent variable model
estimation
covariance matrix
portfolio
financial