Incremental class learning involves sequentially learning classes in bursts of examples from the same class.

This violates the assumptions that underlie  methods for training standard deep neural networks, and will cause them to suffer from catastrophic forgetting.

Arguably, the best method for incremental class learning is iCaRL, but it requires storing  training examples for each class, making it challenging to scale.

Here, we propose FearNet for incremental class learning.

FearNet is a generative model that does not store previous examples, making it memory efficient.

FearNet uses a brain-inspired dual-memory system in which new memories are consolidated from a network for recent memories inspired by the mammalian hippocampal complex to a network for long-term storage inspired by medial prefrontal cortex.

Memory consolidation is inspired by mechanisms that occur during sleep.

FearNet also uses a module inspired by the basolateral amygdala for determining which memory system to use for recall.

FearNet achieves state-of-the-art performance at incremental class learning on image (CIFAR-100, CUB-200) and audio classification (AudioSet) benchmarks.

FearNet is a memory efficient neural-network, inspired by memory formation in the mammalian brain, that is capable of incremental class learning without catastrophic forgetting.

This paper presents a novel solution to an incremental classification problem based on a dual memory system.

FearNet: Brain-Inspired Model for Incremental Learning


Multi-view learning can provide self-supervision when different views are available of the same data.

Distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora.

Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we present two multi-view frameworks for learning sentence representations in an unsupervised fashion.

One framework uses a generative objective and the other a discriminative one.

In both frameworks, the final representation is an ensemble of two views, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model.

We show that, after learning, the vectors produced by our multi-view frameworks provide improved representations over their single-view learnt counterparts, and the combination of different views gives representational improvement over each view and demonstrates solid transferability on standard downstream tasks.

Multi-view learning improves unsupervised sentence representation learning

Approach uses different, complementary encoders of the input sentence and consensus maximization.

The paper presents a multi-view framework for improving sentence representation in NLP tasks using generative and discriminative objective architectures.

This paper shows that multi-view frameworks are more effective than using individual encoders for learning sentence representations.

Improving Sentence Representations with Multi-view Frameworks

improve sentence representation
multi-view framework
encoder


We show how discrete objects can be learnt in an unsupervised fashion from pixels, and how to perform reinforcement learning using this object representation.

More precisely, we construct a differentiable mapping from an image to a discrete tabular list of objects, where each object consists of a differentiable position, feature vector, and scalar presence value that allows the representation to be learnt using an attention mechanism.

Applying this mapping to Atari games, together with an interaction net-style architecture for calculating quantities from objects, we construct agents that can play Atari games using objects learnt in an unsupervised fashion.

During training, many natural objects emerge, such as the ball and paddles in Pong, and the submarine and fish in Seaquest.

This gives the first reinforcement learning agent for Atari with an interpretable object representation, and opens the avenue for agents that can conduct object-based exploration and generalization.

A method for learning object representations from pixels for doing reinforcement learning.

The paper proposes a neural architecture to map video streams to a discrete collection of objects, without human annotations, using an unsupervised pixel reconstruction loss.

Learning objects from pixels

pixel
reinforcement learning
object
unsupervised
learn
representation
discrete


Most recent gains in visual recognition have originated from the inclusion of attention mechanisms in deep convolutional networks (DCNs).

Because these networks are optimized for object recognition, they learn where to attend using only a weak form of supervision derived from image class labels.

Here, we demonstrate the benefit of using stronger supervisory signals by teaching DCNs to attend to image regions that humans deem important for object recognition.

We first describe a large-scale online experiment (ClickMe) used to supplement ImageNet with nearly half a million human-derived "top-down" attention maps.

Using human psychophysics, we confirm that the identified top-down features from ClickMe are more diagnostic than "bottom-up" saliency features for rapid image categorization.

As a proof of concept, we extend a state-of-the-art attention network and demonstrate that adding ClickMe supervision significantly improves its accuracy and yields visual features that are more interpretable and more similar to those used by human observers.

A large-scale dataset for training attention models for object recognition leads to more accurate, interpretable, and human-like object recognition.

Argues recent gains in visual recognition stem from using visual attention mechanisms in deep convolutional networks, which learn where to focus through a weak form of supervision based on image class labels.

Presents a new take on attention in which a large attention dataset is collected and used to train a NN in a supervised manner to exploit self-reported human attention.

This paper proposes a new approach to use more informative signals, specifically, regions humans deem important on images, to improve deep convolutional neural networks.

Learning what and where to attend

learn
dataset
train
deep convolutional
network
image
new
recognition
attention
human


In recent years, deep neural networks have demonstrated outstanding performancein many machine learning tasks.

However, researchers have discovered that thesestate-of-the-art models are vulnerable to adversarial examples:  legitimate examples added by small perturbations which are unnoticeable to human eyes.

Adversarial training, which augments the training data with adversarial examples duringthe training process,  is a well known defense to improve the robustness of themodel against adversarial attacks.

However, this robustness is only effective tothe same attack method used for adversarial training.

Madry et al. (2017) suggest that effectiveness of iterative multi-step adversarial attacks and particularlythat projected gradient descent (PGD) may be considered the universal first order adversary and applying the adversarial training with PGD implies resistanceagainst many other first order attacks.

However,  the computational cost of theadversarial training with PGD and other multi-step adversarial examples is muchhigher than that of the adversarial training with other simpler attack techniques.

In this paper, we show how strong adversarial examples can be generated only ata cost similar to that of two runs of the fast gradient sign method (FGSM), allowing defense against adversarial attacks with a robustness level comparable to thatof the adversarial training with multi-step adversarial examples.

We empiricallydemonstrate the effectiveness of the proposed two-step defense approach againstdifferent attack methods and its improvements over existing defense strategies.

We proposed a time-efficient defense method against one-step and iterative adversarial attacks.

Propose a novel, computationaly efficient method named e2SAD which generates sets of two training adversarial samples for each clean training sample.

The paper introduces a two-step adversarial defense method, to generate two adversarial examples per clean sample and include them in the actual training loop to achieve robustness and claiming it can outperform more expensive iterative methods.

The paper presents a 2-step approach to generate strong adversarial examples at a far lesser cost as compared to recent iterative multi-step adversarial attacks.

EFFICIENT TWO-STEP ADVERSARIAL DEFENSE FOR DEEP NEURAL NETWORKS

two-step adversarial defense
method
attack
generate
sample
iterative
clean
efficient
two
training
example


Recently several different deep learning architectures have been proposed that take a string of characters as the raw input signal and automatically derive features for text classification.

Little studies are available that compare the effectiveness of these approaches for character based text classification with each other.

In this paper we perform such an empirical comparison for the important cybersecurity problem of DGA detection: classifying domain names as either benign vs. produced by malware (i.e., by a Domain Generation Algorithm).

Training and evaluating on a dataset with 2M domain names shows that there is surprisingly little difference between various convolutional neural network (CNN) and recurrent neural network (RNN) based architectures in terms of accuracy, prompting a preference for the simpler architectures, since they are faster to train and less prone to overfitting.

A comparison of five deep neural network architectures for detection of malicious domain names shows surprisingly little difference.

Authors propose using five deep architectures for the cybersecurity task of domain generation algorithm detection.

Applies several NN architectures to classify url's between begign and malware related URLs.

This paper proposes to automatically recognize domain names as malicious or benign by deep networks trained to directly classify the character sequence as such.

Character Level Based Detection of DGA Domain Names

five deep
malicious
architecture
detection
domain name
classify
network
character


Recognizing the relationship between two texts is an important aspect of natural language understanding (NLU), and a variety of neural network models have been proposed for solving NLU tasks.

Unfortunately, recent work showed that the datasets these models are trained on often contain biases that allow models to achieve non-trivial performance without possibly learning the relationship between the two texts.

We propose a framework for building robust models by using adversarial learning to encourage models to learn latent, bias-free representations.

We test our approach in a Natural Language Inference (NLI) scenario, and show that our adversarially-trained models learn robust representations that ignore known dataset-specific biases.

Our experiments demonstrate that our models are more robust to new NLI datasets.

Adversarial learning methods encourage NLI models to ignore dataset-specific biases and help models transfer across datasets.

The paper proposes an adversarial setup to mitigate annotation artifacts in natural language inference data

This paper presents a method for removing bias of a textual entailment model through an adversarial training objective.

Mitigating Bias in Natural Language Inference Using Adversarial Learning

natural language inference
adversarial
model
bias
method
learning
mitigate


We study the problem of learning representations of entities and relations in knowledge graphs for predicting missing links.

The success of such a task heavily relies on the ability of modeling and inferring the patterns of (or between) the relations.

In this paper, we present a new approach for knowledge graph embedding called RotatE, which is able to model and infer various relation patterns including: symmetry/antisymmetry, inversion, and composition.

Specifically, the RotatE model defines each relation as a rotation from the source entity to the target entity in the complex vector space.

In addition, we propose a novel self-adversarial negative sampling technique for efficiently and effectively training the RotatE model.

Experimental results on multiple benchmark knowledge graphs show that the proposed RotatE model is not only scalable, but also able to infer and model various relation patterns and significantly outperform existing state-of-the-art models for link prediction.

A new state-of-the-art approach for knowledge graph embedding.

Presents a neural link prediction scoring function that can infer symmetry, anti-symmetry, inversion and composition patterns of relations in a knowledge base.

This paper proposes an approach to knowledge graph embedding by modeling relations as rotations in the complex vector space.

Proposes a method for graph embedding to be used for link prediction

RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space

approach knowledge graph embed
link prediction
relation
rotation
complex
space


Deep learning algorithms have been known to be vulnerable to adversarial perturbations in various tasks such as image classification.

This problem was addressed by employing several defense methods for detection and rejection of particular types of attacks.

However, training and manipulating networks according to particular defense schemes increases computational complexity of the learning algorithms.

In this work, we propose a simple yet effective method to improve robustness of convolutional neural networks (CNNs) to adversarial attacks by using data dependent adaptive convolution kernels.

To this end, we propose a new type of HyperNetwork in order to employ statistical properties of input data and features for computation of statistical adaptive maps.

Then, we filter convolution weights of CNNs with the learned statistical maps to compute dynamic kernels.

Thereby, weights and kernels are collectively optimized for learning of image classification models robust to

adversarial attacks without employment of additional target detection and rejection algorithms.

We empirically demonstrate that the proposed method enables CNNs to spontaneously defend against different types of attacks, e.g. attacks generated by Gaussian noise, fast gradient sign methods (Goodfellow et al., 2014) and a black-box attack (Narodytska & Kasiviswanathan, 2016).

We modified the CNN using HyperNetworks and observed better robustness against adversarial examples.

Improving the robustness and reliability of deep convolution neural networks by using data-dependent convolution kernels

HyperNetworks with statistical filtering for defending adversarial examples

adversarial example
robustness
hypernetwork


Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures.

Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent.

Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently.

In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning.

The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data.

This requires back-propagating errors through the solver steps.

While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage.

We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components.

Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.

We propose a meta-learning approach for few-shot classification that achieves strong performance at high-speed by back-propagating through the solution of fast solvers, such as ridge regression or logistic regression.

The paper proposes an algorithm for meta-learning which amounts to fixing the features (ie all hidden layers of a deep NN), and treating each task as having its own final layer which could be a ridge regression or a logistic regression.

This paper proposes a meta-learning approach for the problem of few-shot classification, they use a  method based on parametrizing the learner for each task by a closed-form solver.

Meta-learning with differentiable closed-form solvers

ridge regression logistic regression
meta-learning approach
task
few-shot classification
solver
meta-learne
closed-form


While many active learning papers assume that the learner can simply ask for a label and receive it, real annotation often presents a mismatch between the form of a label (say, one among many classes), and the form of an annotation (typically yes/no binary feedback).

To annotate examples corpora for multiclass classification, we might need to ask multiple yes/no questions, exploiting a label hierarchy if one is available.

To address this more realistic setting, we propose active learning with partial feedback (ALPF), where the learner must actively choose both which example to label and which binary question to ask.

At each step, the learner selects an example, asking if it belongs to a chosen (possibly composite) class.

Each answer eliminates some classes, leaving the learner with a partial label.

The learner may then either ask more questions about the same example (until an exact label is uncovered) or move on immediately, leaving the first example partially labeled.

Active learning with partial labels requires

(i) a sampling strategy to choose (example, class) pairs, and

(ii) learning from partial labels between rounds.

Experiments on Tiny ImageNet demonstrate that our most effective method improves 26% (relative) in top-1 classification accuracy compared to i.i.d. baselines and standard active learners given 30% of the annotation budget that would be required (naively) to annotate the dataset.

Moreover, ALPF-learners fully annotate TinyImageNet at 42% lower cost.

Surprisingly, we observe that accounting for per-example annotation costs can alter the conventional wisdom that active learners should solicit labels for hard examples.

We provide a new perspective on training a machine learning model from scratch in hierarchical label setting, i.e. thinking of it as two-way communication between human and algorithms, and study how we can both measure and improve the efficiency.

Introduces a new Active Learning setting where the oracle offers a partial or weak label instead of querying for a particular example's label, leading to a simpler retrieval of information.

This paper proposes a method of active learning with partial feedback that outperforms existing baselines under a limited budget.

The paper considers a multiclass classification problem in which labels are grouped in a given number M of subsets, which contain all individual labels as singletons.

Active Learning with Partial Feedback

active learning partial feedback
new
label


Despite their prevalence, Euclidean embeddings of data are fundamentally limited in their ability to capture latent semantic structures, which need not conform to Euclidean spatial assumptions.

Here we consider an alternative, which embeds data as discrete probability distributions in a Wasserstein space, endowed with an optimal transport metric.

Wasserstein spaces are much larger and more flexible than Euclidean spaces, in that they can successfully embed a wider variety of metric structures.

We propose to exploit this flexibility by learning an embedding that captures the semantic information in the Wasserstein distance between embedded distributions.

We examine empirically the representational capacity of such learned Wasserstein embeddings, showing that they can embed a wide variety of complex metric structures with smaller distortion than an equivalent Euclidean embedding.

We also investigate an application to word embedding, demonstrating a unique advantage of Wasserstein embeddings: we can directly visualize the high-dimensional embedding, as it is a probability distribution on a low-dimensional space.

This obviates the need for dimensionality reduction techniques such as t-SNE for visualization.

We show that Wasserstein spaces are good targets for embedding data with complex semantic structure.

Learns embeddings in a discrete space of probability distributions, using a minimized, regularised version of Wasserstein distances.

The paper describes a new embedding method that embeds data to the space of probability measures endowed with the Wasserstein distance.

The paper proposes embedding the data into low-dimensional Wasserstein spaces, which can capture the underlying structure of the data more accurately.

Learning Embeddings into Entropic Wasserstein Spaces

embed datum
wasserstein space
learn embedding
structure
distance
probability


Clustering high-dimensional datasets is hard because interpoint distances become less informative in high-dimensional spaces.

We present a clustering algorithm that performs nonlinear dimensionality reduction and clustering jointly.

The data is embedded into a lower-dimensional space by a deep autoencoder.

The autoencoder is optimized as part of the clustering process.

The resulting network produces clustered data.

The presented approach does not rely on prior knowledge of the number of ground-truth clusters.

Joint nonlinear dimensionality reduction and clustering are formulated as optimization of a global continuous objective.

We thus avoid discrete reconfigurations of the objective that characterize prior clustering algorithms.

Experiments on datasets from multiple domains demonstrate that the presented algorithm outperforms state-of-the-art clustering schemes, including recent methods that use deep networks.

A clustering algorithm that performs joint nonlinear dimensionality reduction and clustering by optimizing a global continuous objective.

Presents a clustering algorithm by jointly solving deep autoencoder and clustering as a global continuous objective, showing better results than state-of-the-art clustering schemas.

Deep Continuous Clustering is a clustering method that integrates the autoencoder objective with the clustering objective then train using SGD.

Deep Continuous Clustering

global continuous objective
clustering algorithm
cluster
autoencoder
deep


Deep convolutional neural networks (CNNs) are deployed in various applications but demand immense computational requirements.

Pruning techniques and Winograd convolution are two typical methods to reduce the CNN computation.

However, they cannot be directly combined because Winograd transformation fills in the sparsity resulting from pruning.

Li et al. (2017) propose sparse Winograd convolution in which weights are directly pruned in the Winograd domain, but this technique is not very practical because Winograd-domain retraining requires low learning rates and hence significantly longer training time.

Besides, Liu et al. (2018) move the ReLU function into the Winograd domain, which can help increase the weight sparsity but requires changes in the network structure.

To achieve a high Winograd-domain weight sparsity without changing network structures, we propose a new pruning method, spatial-Winograd pruning.

As the first step, spatial-domain weights are pruned in a structured way, which efficiently transfers the spatial-domain sparsity into the Winograd domain and avoids Winograd-domain retraining.

For the next step, we also perform pruning and retraining directly in the Winograd domain but propose to use an importance factor matrix to adjust weight importance and weight gradients.

This adjustment makes it possible to effectively retrain the pruned Winograd-domain network without changing the network structure.

For the three models on the datasets of CIFAR-10, CIFAR-100, and ImageNet, our proposed method can achieve the Winograd-domain sparsities of 63%, 50%, and 74%, respectively.

To accelerate the computation of convolutional neural networks, we propose a new two-step pruning technique which achieves a higher Winograd-domain weight sparsity without changing the network structure.

Proposes a spatial-Winograd pruning framework which allows pruned weight from the spatial domain to be kept in the Winograd domain and improves the sparsity of the Winograd domain.

Proposes two techniques for pruning convolutional layers which use the Winograd algorithm

Spatial-Winograd Pruning Enabling Sparse Winograd Convolution

winograd
technique
spatial-winograd pruning
weight
convolutional
sparsity


In federated learning problems, data is scattered across different servers and exchanging or pooling it is often impractical or prohibited.

We develop a Bayesian nonparametric framework for federated learning with neural networks.

Each data server is assumed to train local neural network weights, which are modeled through our framework.

We then develop an inference approach that allows us to synthesize a more expressive global network without additional supervision or data pooling.

We then demonstrate the efficacy of our approach on federated learning problems simulated from two popular image classification datasets.

We propose a Bayesian nonparametric model for federated learning with neural networks.

Uses beta process to do federated neural matching.

The paper considers federate learning of neural networks, where data is distributed on multiple machines and the allocation of data points is potentially inhomogenous and unbalanced.

Probabilistic Federated Neural Matching

learning neural network
matching
federate


We present a general-purpose method to train Markov chain Monte Carlo kernels, parameterized by deep neural networks, that converge and mix quickly to their target distribution.

Our method generalizes Hamiltonian Monte Carlo and is trained to maximize expected squared jumped distance, a proxy for mixing speed.

We demonstrate large empirical gains on a collection of simple but challenging distributions, for instance achieving a 106x improvement in effective sample size in one case, and mixing when standard HMC makes no measurable progress in a second.

Finally, we show quantitative and qualitative gains on a real-world task: latent-variable generative modeling.

Python source code will be open-sourced with the camera-ready paper.

General method to train expressive MCMC kernels parameterized with deep neural networks. Given a target distribution p, our method provides a fast-mixing sampler, able to efficiently explore the state space.

Proposes a generalized HMC by modifying the leapfrog integrator using neural networks to make the sampler to converge and mix quickly.

Generalizing Hamiltonian Monte Carlo with Neural Networks


This paper addresses the problem of evaluating learning systems in safety critical domains such as autonomous driving, where failures can have catastrophic consequences.

We focus on two problems: searching for scenarios when learned agents fail and assessing their probability of failure.

The standard method for agent evaluation in reinforcement learning, Vanilla Monte Carlo, can miss failures entirely, leading to the deployment of unsafe agents.

We demonstrate this is an issue for current agents, where even matching the compute used for training is sometimes insufficient for evaluation.

To address this shortcoming, we draw upon the rare event probability estimation literature and propose an adversarial evaluation approach.

Our approach focuses evaluation on adversarially chosen situations, while still providing unbiased estimates of failure probabilities.

The key difficulty is in identifying these adversarial situations -- since failures are rare there is little signal to drive optimization.

To solve this we propose a continuation approach that learns failure modes in related but less robust agents.

Our approach also allows reuse of data already collected for training the agent.

We demonstrate the efficacy of adversarial evaluation on two standard domains: humanoid control and simulated driving.

Experimental results show that our methods can find catastrophic failures and estimate failures rates of agents multiple orders of magnitude faster than standard evaluation schemes, in minutes to hours rather than days.

We show that rare but catastrophic failures may be missed entirely by random testing, which poses issues for safe deployment. Our proposed approach for adversarial testing fixes this.

Proposes a method which learns a failure probability predictor for a learned agent, leading to predictions of which initial states cause a system to fail.

This paper proposes an importance sampling approach to sampling failure cases for RL algorithms based on a function learned via a neural network on failures that occur during agent training

This paper proposed an adversarial approach to identifying catastrophic failure cases in reinforcement learning.

Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures

adversarial approach
failure case
agent
catastrophic
learn


The variational autoencoder (VAE) is a popular combination of deep latent variable model and accompanying variational learning technique.

By using a neural inference network to approximate the model's posterior on latent variables, VAEs efficiently parameterize a lower bound on marginal data likelihood that can be optimized directly via gradient methods.

In practice, however, VAE training often results in a degenerate local optimum known as "posterior collapse" where the model learns to ignore the latent variable and the approximate posterior mimics the prior.

In this paper, we investigate posterior collapse from the perspective of training dynamics.

We find that during the initial stages of training the inference network fails to approximate the model's true posterior, which is a moving target.

As a result, the model is encouraged to ignore the latent encoding and posterior collapse occurs.

Based on this observation, we propose an extremely simple modification to VAE training to reduce inference lag: depending on the model's current mutual information between latent variable and observation, we aggressively optimize the inference network before performing each model update.

Despite introducing neither new model components nor significant complexity over basic VAE, our approach is able to avoid the problem of collapse that has plagued a large amount of previous work.

Empirically, our approach outperforms strong autoregressive baselines on text and image benchmarks in terms of held-out likelihood, and is competitive with more complex techniques for avoiding collapse while being substantially faster.

To address posterior collapse in VAEs, we propose a novel yet simple training procedure that aggressively optimizes inference network with more updates. This new training procedure mitigates posterior collapse and leads to a better VAE model.

Looks into the phenomenon of posterior collapse, showing that increased training of the inference network can reduce the problem and lead to better optima.

Authors propose changing the training procedure of VAEs only as a solution to posterior collapse, leaving the model and objective untouched.

Lagging Inference Networks and Posterior Collapse in Variational Autoencoders

vaes
inference network
lead well
posterior collapse
training procedure
model


Online healthcare services can provide the general public with ubiquitous access to medical knowledge and reduce the information access cost for both individuals and societies.

To promote these benefits, it is desired to effectively expand the scale of high-quality yet novel relational medical entity pairs that embody rich medical knowledge in a structured form.

To fulfill this goal, we introduce a generative model called Conditional Relationship Variational Autoencoder (CRVAE), which can discover meaningful and novel relational medical entity pairs without the requirement of additional external knowledge.

Rather than discriminatively identifying the relationship between two given medical entities in a free-text corpus, we directly model and understand medical relationships from diversely expressed medical entity pairs.

The proposed model introduces the generative modeling capacity of variational autoencoder to entity pairs, and has the ability to discover new relational medical entity pairs solely based on the existing entity pairs.

Beside entity pairs, relationship-enhanced entity representations are obtained as another appealing benefit of the proposed method.

Both quantitative and qualitative evaluations on real-world medical datasets demonstrate the effectiveness of the proposed method in generating relational medical entity pairs that are meaningful and novel.

Generatively discover meaningful, novel entity pairs with a certain medical relationship by purely learning from the existing meaningful entity pairs, without the requirement of additional text corpus for discriminative extraction.

Presents a variational autoencoder for generating entity pairs given a relation in a medical setting.

In the medical context, this paper describes the classic problem of "knowledge base completion" from structured data only.

Generative Discovery of Relational Medical Entity Pairs


Although variational autoencoders (VAEs) represent a widely influential deep generative model, many aspects of the underlying energy function remain poorly understood.

In particular, it is commonly believed that Gaussian encoder/decoder assumptions reduce the effectiveness of VAEs in generating realistic samples

.  In this regard, we rigorously analyze the VAE objective, differentiating situations where this belief is and is not actually true

.  We then leverage the corresponding insights to develop a simple VAE enhancement that requires no additional hyperparameters or sensitive tuning

.  Quantitatively, this proposal produces crisp samples and stable FID scores that are actually competitive with a variety of GAN models, all while retaining desirable attributes of the original VAE architecture

. The code for our model is available at \url{https://github.com/daib13/TwoStageVAE}.

We closely analyze the VAE objective function and draw novel conclusions that lead to simple enhancements.

Proposes a two-stage VAE method to generate high-quality samples and avoid blurriness.

The paper provides a number of theoretical results on "vanilla" Gaussian Variational Auto-Encoders, which are then used to build a new algorithm called "2 stage VAEs".

Diagnosing and Enhancing VAE Models


We give a simple, fast algorithm for hyperparameter optimization inspired by techniques from the analysis of Boolean functions.

We focus on the high-dimensional regime where the canonical example is training a neural network with a large number of hyperparameters.

The algorithm --- an iterative application of compressed sensing techniques for orthogonal polynomials --- requires only uniform sampling of the hyperparameters and is thus easily parallelizable.

Experiments for training deep neural networks on Cifar-10 show that compared to state-of-the-art tools (e.g., Hyperband and Spearmint), our algorithm finds significantly improved solutions, in some cases better than what is attainable by hand-tuning.

In terms of overall running time (i.e., time required to sample various settings of hyperparameters plus additional computation time), we are at least an order of magnitude faster than Hyperband and Bayesian Optimization.

We also outperform Random Search $8\times$.

Our method is inspired by provably-efficient algorithms for learning decision trees using the discrete Fourier transform.

We obtain improved sample-complexty bounds for learning decision trees while matching state-of-the-art bounds on running time (polynomial and quasipolynomial, respectively).

A hyperparameter tuning algorithm using discrete Fourier analysis and compressed sensing

Investigates problem of optimizing hyperparameters under the assumption that the unknown function can be approximated, showing that the approximate minimization can be performed over the boolean hypercube.

The paper explores hyperparameter optimization by assuming structure in the unknown function mapping hyperparameters to classification accuracy

Hyperparameter optimization: a spectral approach

unknown function
hyperparameter optimization


Permutations and matchings are core building blocks in a variety of latent variable models, as they allow us to align, canonicalize, and sort data.

Learning in such models is difficult, however, because exact marginalization over these combinatorial objects is intractable.

In response, this paper introduces a collection of new methods for end-to-end learning in such models that approximate discrete maximum-weight matching using the continuous Sinkhorn operator.

Sinkhorn iteration is attractive because it functions as a simple, easy-to-implement analog of the softmax operator.

With this, we can define the Gumbel-Sinkhorn method, an extension of the Gumbel-Softmax method (Jang et al. 2016, Maddison2016 et al. 2016) to distributions over latent matchings.

We demonstrate the effectiveness of our method by outperforming competitive baselines on a range of qualitatively different tasks: sorting numbers, solving jigsaw puzzles, and identifying neural signals in worms.

A new method for gradient-descent inference of permutations, with applications to latent matching inference and supervised learning of permutations with neural networks

The paper utilizes finite approximation of the Sinkhorn operator to describe how one can construct a neural network for learning from permutation valued training data.

The paper proposes a new method that approximates the discrete max-weight for learning latent permutations

Learning Latent Permutations with Gumbel-Sinkhorn Networks

new method
learn latent permutation
neural network


Recent work in network quantization has substantially reduced the time and space complexity of neural network inference, enabling their deployment on embedded and mobile devices with limited computational and memory resources.

However, existing quantization methods often represent all weights and activations with the same precision (bit-width).

In this paper, we explore a new dimension of the design space: quantizing different layers with different bit-widths.

We formulate this problem as a neural architecture search problem and propose a novel differentiable neural architecture search (DNAS) framework to efficiently explore its exponential search space with gradient-based optimization.

Experiments show we surpass the state-of-the-art compression of ResNet on CIFAR-10 and ImageNet.

Our quantized models with 21.1x smaller model size or 103.9x lower computational cost can still outperform baseline quantized or even full precision models.

A novel differentiable neural architecture search framework for mixed quantization of ConvNets.

The authors introduce a new method for neural architecture search which selects the precision quantization of weights at each neural network layer, and use it in the context of network compression.

The paper presents a new approach in network quantization by quantizing different layers with different bit-widths and introduces a new differentiable neural architecture search framework.

Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search


The top-$k$ error is a common measure of performance in machine learning and computer vision.

In practice, top-$k$ classification is typically performed with deep neural networks trained with the cross-entropy loss.

Theoretical results indeed suggest that cross-entropy is an optimal learning objective for such a task in the limit of infinite data.

In the context of limited and noisy data however, the use of a loss function that is specifically designed for top-$k$ classification can bring significant improvements.

Our empirical evidence suggests that the loss function must be smooth and have non-sparse gradients in order to work well with deep neural networks.

Consequently, we introduce a family of smoothed loss functions that are suited to top-$k$ optimization via deep learning.

The widely used cross-entropy is a special case of our family.

Evaluating our smooth loss functions is computationally challenging: a na{\"i}ve algorithm would require $\mathcal{O}(\binom{n}{k})$ operations, where $n$ is the number of classes.

Thanks to a connection to polynomial algebra and a divide-and-conquer approach, we provide an algorithm with a time complexity of $\mathcal{O}(k n)$.

Furthermore, we present a novel approximation to obtain fast and stable algorithms on GPUs with single floating point precision.

We compare the performance of the cross-entropy loss and our margin-based losses in various regimes of noise and data size, for the predominant use case of $k=5$.

Our investigation reveals that our loss is more robust to noise and overfitting than cross-entropy.

Smooth Loss Function for Top-k Error Minimization

Proposes using top-k loss with deep models to address the problem of class confusion with similar classes both present or absent of the training dataset.

This paper introduces a smooth surrogate loss function for the top-k SVM, for the purpose of plugging the SVM to the deep neural networks.

Smooth Loss Functions for Deep Top-k Classification


Designing a molecule with desired properties is one of the biggest challenges in drug development, as it requires optimization of chemical compound structures with respect to many complex properties.

To augment the compound design process we introduce Mol-CycleGAN -- a CycleGAN-based model that generates optimized compounds with a chemical scaffold of interest.

Namely, given a molecule our model generates a structurally similar one with an optimized value of the considered property.

We evaluate the performance of the model on selected optimization objectives related to structural properties (presence of halogen groups, number of aromatic rings) and to a physicochemical property (penalized logP).

In the task of optimization of penalized logP of drug-like molecules our model significantly outperforms previous results.

We introduce Mol-CycleGAN - a new generative model for optimization of molecules to augment drug design.

The paper presents an approach for optimizing molecular properties based on the application of CycleGANs to variational autoencoders for molecules and employs a domain-specific VAE called Junction Tree VAE (JT-VAE).

This paper uses a variational autoencoders to learn a translation function, from the set of molecules without the interested property to the set of molecules with the property.

Mol-CycleGAN - a generative model for molecular optimization

variational autoencoder
generative model
mol-cyclegan -
molecule
optimization
molecular
property


Knowledge distillation is a potential solution for model compression.

The idea is to make a small student network imitate the target of a large teacher network, then the student network can be competitive to the teacher one.

Most previous studies focus on model distillation in the classification task, where they propose different architectures and initializations for the student network.

However, only the classification task is not enough, and other related tasks such as regression and retrieval are barely considered.

To solve the problem, in this paper, we take face recognition as a breaking point and propose model distillation with knowledge transfer from face classification to alignment and verification.

By selecting appropriate initializations and targets in the knowledge transfer, the distillation can be easier in non-classification tasks.

Experiments on the CelebA and CASIA-WebFace datasets demonstrate that the student network can be competitive to the teacher one in alignment and verification, and even surpasses the teacher network under specific compression rates.

In addition, to achieve stronger knowledge transfer, we also use a common initialization trick to improve the distillation performance of classification.

Evaluations on the CASIA-Webface and large-scale MS-Celeb-1M datasets show the effectiveness of this simple trick.

We take face recognition as a breaking point and propose model distillation with knowledge transfer from face classification to alignment and verification

This paper proposes to transfer the classifier from the model for face classification to the task of alignment and verification.

The manuscript presents experiments on distilling knowledge from a face classification model to student models for face alignment and verification.

Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification

knowledge transfer face classification alignment verification


RNNs have been shown to be excellent models for sequential data and in particular for session-based user behavior.

The use of RNNs provides impressive performance benefits over classical methods in session-based recommendations.

In this work we introduce a novel ranking loss function tailored for RNNs in recommendation settings.

The better performance of such loss over alternatives, along with further tricks and improvements described in this work, allow to achieve an overall improvement of up to 35% in terms of MRR and Recall@20 over previous session-based RNN solutions and up to 51% over classical collaborative filtering approaches.

Unlike data augmentation-based improvements, our method does not increase training times significantly.

Improving session-based recommendations with RNNs (GRU4Rec) by 35% using newly designed loss functions and sampling.

This paper analyzes existing loss functions for session-based recommendations and proposes two novel losses functions which add a weighting to existing ranking-based loss functions

Presents modifications on top of earlier work for session-based recommendation using RNN by weighting negative examples by their "relevance"

This paper discusses the issues for optimizing the loss functions in GRU4Rec, proposes tricks for optimizing, and suggests an enhanced version.

Recurrent Neural Networks with Top-k Gains for Session-based Recommendations

gru4rec
session-based recommendation
loss function


In representational lifelong learning an agent aims to continually learn to solve novel tasks while updating its representation in light of previous tasks.

Under the assumption that future tasks are related to previous tasks, representations should be learned in such a way that they capture the common structure across learned tasks, while allowing the learner sufficient flexibility to adapt to novel aspects of a new task.

We develop a framework for lifelong learning in deep neural networks that is based on generalization bounds, developed within the PAC-Bayes framework.

Learning takes place through the construction of a distribution over networks based on the tasks seen so far, and its utilization for learning a new task.

Thus, prior knowledge is incorporated through setting a history-dependent prior for novel tasks.

We develop a gradient-based algorithm implementing these ideas, based on minimizing an objective function motivated by generalization bounds, and demonstrate its effectiveness through numerical examples.

We develop a lifelong learning approach to transfer learning based on PAC-Bayes theory, whereby priors are adjusted as new tasks are encountered thereby facilitating the learning of novel tasks.

A novel PAC-Bayesian risk bound that serves as an objective function for multi-task machine learning, and an algorithm for minimizing a simplified version of that objective function.

Extends existing PAC-Bayes bounds to multi-task learning, to allow the prior to be adapted across different tasks.

Lifelong Learning by Adjusting Priors

lifelong learning
prior
task
novel
adjust
multi-task
pac-bayes


Optimization algorithms for training deep models not only affects the convergence rate and stability of the training process, but are also highly related to the generalization performance of trained models.

While adaptive algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in many scenarios, they often lead to worse generalization performance than SGD, when used for training deep neural networks (DNNs).

In this work, we identify two problems regarding the direction and step size for updating the weight vectors of hidden units, which may degrade the generalization performance of Adam.

As a solution, we propose the normalized direction-preserving Adam (ND-Adam) algorithm, which controls the update direction and step size more precisely, and thus bridges the generalization gap between Adam and SGD.

Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits.

By bridging the gap between SGD and Adam, we also shed some light on why certain optimization algorithms generalize better than others.

A tailored version of Adam for training DNNs, which bridges the generalization gap between Adam and SGD.

Proposes a variant of ADAM optimization algorithm that normalizes weights of each hidden unit using batch normalization

Extension of the Adam optimization algorithm to preserve the update direction by adapting the learning rate for the incoming weights to a hidden unit jointly using the L2 norm of the gradient vector

Normalized Direction-preserving Adam

weight hidden unit
adam optimization algorithm
normalize


Options in reinforcement learning allow agents to hierarchically decompose a task into subtasks, having the potential to speed up learning and planning.

However, autonomously learning effective sets of options is still a major challenge in the field.

In this paper we focus on the recently introduced idea of using representation learning methods to guide the option discovery process.

Specifically, we look at eigenoptions, options obtained from representations that encode diffusive information flow in the environment.

We extend the existing algorithms for eigenoption discovery to settings with stochastic transitions and in which handcrafted features are not available.

We propose an algorithm that discovers eigenoptions while learning non-linear state representations from raw pixels.

It exploits recent successes in the deep reinforcement learning literature and the equivalence between proto-value functions and the successor representation.

We use traditional tabular domains to provide intuition about our approach and Atari 2600 games to demonstrate its potential.

We show how we can use the successor representation to discover eigenoptions in stochastic domains, from raw pixels. Eigenoptions are options learned to navigate the latent dimensions of a learned representation.

Extends the idea of eigenoptions to domains with stochastic transitions and where state features are learned.

Shows equivalence between proto value functions and successor representations and derives the idea of eigen options as a mechanism in option discovery

The paper is a follow up on previous work by Machado et al. (2017) showing how proto-value functions can be used to define options called “eigenoptions”.

Eigenoption Discovery through the Deep Successor Representation

successor representation
eigenoption
show
domain
idea
learn
option
stochastic
function
discovery


One form of characterizing the expressiveness of a piecewise linear neural network is by the number of linear regions, or pieces, of the function modeled.

We have observed substantial progress in this topic through lower and upper bounds on the maximum number of linear regions and a counting procedure.

However, these bounds only account for the dimensions of the network and the exact counting may take a prohibitive amount of time, therefore making it infeasible to benchmark the expressiveness of networks.

In this work, we approximate the number of linear regions of specific rectifier networks with an algorithm for probabilistic lower bounds of mixed-integer linear sets.

In addition, we present a tighter upper bound that leverages network coefficients.

The algorithm for probabilistic lower bounds is several orders of magnitude faster than exact counting and the values reach similar orders of magnitude, hence making our approach a viable method to compare the expressiveness of such networks.

The refined upper bound is particularly stronger on networks with narrow layers.

We provide improved upper bounds for the number of linear regions used in network expressivity, and an highly efficient algorithm (w.r.t. exact counting) to obtain probabilistic lower bounds on the actual number of linear regions.

Contributes to the study of the number of linear regions in RELU neural networks by using an approximate probabilistic counting algorithm and analysis

Builds off previous work studying the counting of linear regions in deep neural networks, and improves the upper bound previously proposed by changing the dimensionality constraint

The paper deals with expressiveness of a piecewise linear neural network, characterized by the number of linear regions of the function modeled, and leverages probabilistic algorithms to compute the bounds faster, and proves tighter bounds.

Empirical Bounds on Linear Regions of Deep Rectifier Networks

number linear region
neural network
counting
bound
algorithm
study
deep
upper
improve
probabilistic


The ability to look multiple times through a series of pose-adjusted glimpses is fundamental to human vision.

This critical faculty allows us to understand highly complex visual scenes.

Short term memory plays an integral role in aggregating the information obtained from these glimpses and informing our interpretation of the scene.

Computational models have attempted to address glimpsing and visual attention but have failed to incorporate the notion of memory.

We introduce a novel, biologically inspired visual working memory architecture that we term the Hebb-Rosenblatt memory.

We subsequently introduce a fully differentiable Short Term Attentive Working Memory model (STAWM) which uses transformational attention to learn a memory over each image it sees.

The state of our Hebb-Rosenblatt memory is embedded in STAWM as the weights space of a layer.

By projecting different queries through this layer we can obtain goal-oriented latent representations for tasks including classification and visual reconstruction.

Our model obtains highly competitive classification performance on MNIST and CIFAR-10.

As demonstrated through the CelebA dataset, to perform reconstruction the model learns to make a sequence of updates to a canvas which constitute a parts-based representation.

Classification with the self supervised representation obtained from MNIST is shown to be in line with the state of the art models (none of which use a visual attention mechanism).

Finally, we show that STAWM can be trained under the dual constraints of classification and reconstruction to provide an interpretable visual sketchpad which helps open the `black-box' of deep learning.

A biologically inspired working memory that can be integrated in recurrent visual attention models for state of the art performance

Introduces a new network architecture inspired by visual attentive working memory and applies it to classification tasks and using it as a generative model

The paper augments the recurrent attention model with a novel Hebb-Rosenblatt working memory model and achieves competitive results on MNIST

A Biologically Inspired Visual Working Memory for Deep Networks

attention model
biologically inspire
working memory
recurrent
visual
network


Generative models have been successfully applied to image style transfer and domain translation.

However, there is still a wide gap in the quality of results when learning such tasks on musical audio.

Furthermore, most translation models only enable one-to-one or one-to-many transfer by relying on separate encoders or decoders and complex, computationally-heavy models.

In this paper, we introduce the Modulated Variational auto-Encoders (MoVE) to perform musical timbre transfer.

First, we define timbre transfer as applying parts of the auditory properties of a musical instrument onto another.

We show that we can achieve and improve this task by conditioning existing domain translation techniques with Feature-wise Linear Modulation (FiLM).

Then, by replacing the usual adversarial translation criterion by a Maximum Mean Discrepancy (MMD) objective, we alleviate the need for an auxiliary pair of discriminative networks.

This allows a faster and more stable training, along with a controllable latent space encoder.

By further conditioning our system on several different instruments, we can generalize to many-to-many transfer within a single variational architecture able to perform multi-domain transfers.

Our models map inputs to 3-dimensional representations, successfully translating timbre from one instrument to another and supporting sound synthesis on a reduced set of control parameters.

We evaluate our method in reconstruction and generation tasks while analyzing the auditory descriptor distributions across transferred domains.

We show that this architecture incorporates generative controls in multi-domain transfer, yet remaining rather light, fast to train and effective on small datasets.

The paper uses Variational Auto-Encoding and network conditioning for Musical Timbre Transfer, we develop and generalize our architecture for many-to-many instrument transfers together with visualizations and evaluations.

Proposes a Modulated Variational auto-Encoder to perform musical timbre transfer by replacing the usual adversarial translation criterion by a Maxiimum Mean Discrepancy

Describes a many-to-many model for musical timbre transfer which builds on recent developments in domain and style transfer

Proposes a hybrid VAE-based model to perform timbre transfer on recordings of musical instruments.

Modulated Variational Auto-Encoders for Many-to-Many Musical Timbre Transfer

musical timbre transfer
modulate variational auto-encoder
many-to-many
perform
model
instrument


Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena when the depth becomes large.

This, in particular, provides quantitative answers and insights to three questions that were yet fully understood in the literature.

Firstly, we provide a precise answer on how the random deep weight-tied autoencoder model performs “approximate inference” as posed by Scellier et al. (2018), and its connection to reversibility considered by several theoretical studies.

Secondly, we show that deep autoencoders display a higher degree of sensitivity to perturbations in the parameters, distinct from the shallow counterparts.

Thirdly, we obtain insights on pitfalls in training initialization practice, and demonstrate experimentally that it is possible to train a deep autoencoder, even with the tanh activation and a depth as large as 200 layers, without resorting to techniques such as layer-wise pre-training or batch normalization.

Our analysis is not specific to any depths or any Lipschitz activations, and our analytical techniques may have broader applicability.

We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena.

A theoretical analysis of autoencoders with weights tied between encoder and decoder (weight-tied) via mean field analysis

Analyses the performances of weighted tied auto-encoders by building on recent progress in analysis of high-dimensional statistics problems and specifically, the message passing algorithm

This paper studies auto-encoders under several assumptions, and points out that this model of random autoencoder can be elegantly and rigorously analysed with one-dimensional equations.

On Random Deep Weight-Tied Autoencoders: Exact Asymptotic Analysis, Phase Transitions, and Implications to Training

analysis
autoencoder
assumption
study
weight tie
exact
phase transition
via
weight-tied
random
analyse
auto-encoder


Assessing distance betweeen the true and the sample distribution is a key component of many state of the art generative models, such as Wasserstein Autoencoder (WAE).

CWAE cost function, based on introduced Cramer-Wold distance between samples, has a simple closed-form in the case of normal prior.

As a consequence, while simplifying the optimization procedure (no need of sampling necessary to evaluate the distance function in the training loop), CWAE performance matches quantitatively and qualitatively that of WAE-MMD (WAE using maximum mean discrepancy based distance function) and often improves upon SWAE.

Inspired by prior work on Sliced-Wasserstein Autoencoders (SWAE) and kernel smoothing we construct a new generative model – Cramer-Wold AutoEncoder (CWAE).

This paper proposes a WAE variant based on a new statistical distance between the encoded data distribution and the latent prior distribution

Introduces a variation on the Wasserstein AudoEncoders which is a novel regularized auto-encoder architecture that proposes a specific choice of the divergence penalty

This paper proposes the Cramer-Wold autoencoder, which uses the Cramer-Wold distance between two distributions based on the Cramer-Wold Theorem.

Cramer-Wold AutoEncoder

cramer-wold autoencoder
base
new
distance
prior
distribution


We propose a rejection sampling scheme using the discriminator of a GAN to

approximately correct errors in the GAN generator distribution.

under quite strict assumptions, this will allow us to recover the data distribution

We then examine where those strict assumptions break down and design a

practical algorithm—called Discriminator Rejection Sampling (DRS)—that can be

Finally, we demonstrate the efficacy of DRS on a mixture of

Gaussians and on the state of the art SAGAN model.

improved baseline that increases the best published Inception Score from 52.52 to

62.36 and reduces the Frechet Inception Distance from 18.65 to 14.79.

DRS to further improve on this baseline, improving the Inception Score to 76.08

We use a GAN discriminator to perform an approximate rejection sampling scheme on the output of the GAN generator.

Proposes a rejection sampling algorithm for sampling from the GAN generator.

This paper proposed a post-processing rejection sampling scheme for GANs, named Discriminator Rejection Sampling, to help filter ‘good’ samples from GANs’ generator.

Discriminator Rejection Sampling

gan generator
rejection sample scheme
discriminator


The quality of the features used in visual recognition is of fundamental importance for the overall system.

For a long time, low-level hand-designed feature algorithms as SIFT and HOG have obtained the best results on image recognition.

Visual features have recently been extracted from trained convolutional neural networks.

Despite the high-quality results, one of the main drawbacks of this approach, when compared with hand-designed features, is the training time required during the learning process.

In this paper, we propose a simple and fast way to train supervised convolutional models to feature extraction while still maintaining its high-quality.

This methodology is evaluated on different datasets and compared with state-of-the-art approaches.

A simple fast method for extracting visual features from convolutional neural networks

Proposes a fast way to learn convolutional features that later can be used with any classifier by using reduced numbers of training epocs and specific schedule delays of learning rate

Use a learning rate decay scheme that is fixed relative to the number of epochs used in training and extract the penultimate layer output as features to train a conventional classifier.

Simple Fast Convolutional Feature Learning

feature
extract
simple fast
classifier
training
number
convolutional
learn
rate


We develop a framework for understanding and improving recurrent neural networks (RNNs) using max-affine spline operators (MASOs).

We prove that RNNs using piecewise affine and convex nonlinearities can be written as a simple piecewise affine spline operator.

The resulting representation provides several new perspectives for analyzing RNNs, three of which we study in this paper.

First, we show that an RNN internally partitions the input space during training and that it builds up the partition through time.

Second, we show that the affine slope parameter of an RNN corresponds to an input-specific template, from which we can interpret an RNN as performing a simple template matching (matched filtering) given the input.

Third, by carefully examining the MASO RNN affine mapping, we prove that using a random initial hidden state corresponds to an explicit L2 regularization of the affine parameters, which can mollify exploding gradients and improve generalization.

Extensive experiments on several datasets of various modalities demonstrate and validate each of the above conclusions.

In particular, using a random initial hidden states elevates simple RNNs to near state-of-the-art performers on these datasets.

We provide new insights and interpretations of RNNs from a max-affine spline operators perspective.

Rewrites equations of Elman RNN in terms of so-called max-affine spline operators

Provide a novel approach toward understanding RNNs using max-affline spline operators (MASO) by rewriting them with piecewise affine and convex activations MASOs

The authors build upon max-affine spline operator interpetation of a substantial class of deep networks, focusing on Recurrent Neural Networks using noise in initial hidden state acts as regularization

A Max-Affine Spline Perspective of Recurrent Neural Networks

recurrent neural network
max-affine spline
provide
rnn
perspective
operator
rewrite


Reasoning over text and Knowledge Bases (KBs) is a major challenge for Artificial Intelligence, with applications in machine reading, dialogue, and question answering.

Transducing text to logical forms which can be operated on is a brittle and error-prone process

. Operating directly on text by jointly learning representations and transformations thereof by means of neural architectures that lack the ability to learn and exploit general rules can be very data-inefficient and not generalise correctly

. These issues are addressed by Neural Theorem Provers (NTPs) (Rocktäschel & Riedel, 2017), neuro-symbolic systems based on a continuous relaxation of Prolog’s backward chaining algorithm, where symbolic unification between atoms is replaced by a differentiable operator computing the similarity between their embedding representations

. In this paper, we first propose Neighbourhood-approximated Neural Theorem Provers (NaNTPs) consisting of two extensions toNTPs, namely

a) a method for drastically reducing the previously prohibitive time and space complexity during inference and learning, and

b) an attention mechanism for improving the rule learning process, deeming them usable on real-world datasets.

Then, we propose a novel approach for jointly reasoning over KB facts and textual mentions, by jointly embedding them in a shared embedding space.

The proposed method is able to extract rules and provide explanations—involving both textual patterns and KB relations—from large KBs and text corpora.

We show that NaNTPs perform on par with NTPs at a fraction of a cost, and can achieve competitive link prediction results on challenging large-scale datasets, including WN18, WN18RR, and FB15k-237 (with and without textual mentions) while being able to provide explanations for each prediction and extract interpretable rules.

We scale Neural Theorem Provers to large datasets, improve the rule learning process, and extend it to jointly reason over text and Knowledge Bases.

Proposes an extension of the Neural Theorem Provers system that addresses the main issues of this model by reducing the time and space complexity of the model

Scales NTPs by using approximate nearest neighbour search over facts and rules during unification and suggests parameterizing predicates using attention over known predicates

improves upon the previously proposed Neural Theorem Prover approach by using nearest neighbor search.

Scalable Neural Theorem Proving on Knowledge Bases and Natural Language

neural theorem prover
knowledge basis
rule
search
scale
improve
near


We investigate the methods by which a Reservoir Computing Network (RCN) learns concepts such as 'similar' and 'different' between pairs of images using a small training dataset and generalizes these concepts to previously unseen types of data.

Specifically, we show that an RCN trained to identify relationships between image-pairs drawn from a subset of digits from the MNIST database or the depth maps of subset of visual scenes from a moving camera generalizes the learned transformations to images of digits unseen during training or depth maps of different visual scenes.

We infer, using Principal Component Analysis, that the high dimensional reservoir states generated from an input image pair with a specific transformation converge over time to a unique relationship.

Thus, as opposed to training the entire high dimensional reservoir state, the reservoir only needs to train on these unique relationships, allowing the reservoir to perform well with very few training examples.

Thus, generalization of learning to unseen images is interpretable in terms of clustering of the reservoir state onto the attractor corresponding to the transformation in reservoir space.

We find that RCNs can identify and generalize linear and non-linear transformations, and combinations of transformations, naturally and be a robust and effective image classifier.

Additionally, RCNs perform significantly better than state of the art neural network classification techniques such as deep Siamese Neural Networks (SNNs) in generalization tasks both on the MNIST dataset and more complex depth maps of visual scenes from a moving camera.

This work helps bridge the gap between explainable machine learning and biological learning through analogies using small datasets, and points to new directions in the investigation of learning processes.

Generalization of the relationships learnt between pairs of images using a small training data to previously unseen types of images using an explainable dynamical systems model, Reservoir Computing, and a biologically plausible learning technique based on analogies.

Claims results of "combining transformations" in the context of RC by using an echo-state network with standard tanh acctivations with the difference that recurrent weights are not trained

Novel method of classifying different distorions of MNIST data

The paper uses an echo state network to learn to classify image transformations between pairs of images into one of fives classes.

Generalization of Learning using Reservoir Computing

pair image
reservoir computing
learn
generalization
network
classify
datum
transformation


We present Generative Adversarial Privacy and Fairness (GAPF), a data-driven framework for learning private and fair representations of the data.

GAPF leverages recent advances in adversarial learning to allow a data holder to learn "universal" representations that decouple a set of sensitive attributes from the rest of the dataset.

Under GAPF, finding the optimal decorrelation scheme is formulated as a constrained minimax game between a generative decorrelator and an adversary.

We show that for appropriately chosen adversarial loss functions, GAPF provides privacy guarantees against strong information-theoretic adversaries and enforces demographic parity.

We also evaluate the performance of GAPF on multi-dimensional Gaussian mixture models and real datasets, and show how a designer can certify that representations learned under an adversary with a fixed architecture perform well against more complex adversaries.

We present Generative Adversarial Privacy and Fairness (GAPF), a data-driven framework for learning private and fair representations with certified privacy/fairness guarantees

This paper uses a GAN model to provide an overview of the related work to Private/Fair Representation Learning (PRL).

This paper presents an adversarial-based approach for private and fair representations by learned distortion of data that minimises the dependency on sensitive variables while the degree of distortion is constrained.

The authors describe a framework of how to learn a demographic parity representation that can be used to train certain classifiers.

Generative Adversarial Models for Learning Private and Fair Representations

learn private fair representation
framework
generative adversarial
present
model
/


Current machine learning algorithms can be easily fooled by adversarial examples.

One possible solution path is to make models that use confidence thresholding to avoid making mistakes.

Such models refuse to make a prediction when they are not confident of their answer.

We propose to evaluate such models in terms of tradeoff curves with the goal of high success rate on clean examples and low failure rate on adversarial examples.

Existing untargeted attacks developed for models that do not use confidence thresholding tend to underestimate such models' vulnerability.

We propose the MaxConfidence family of attacks, which are optimal in a variety of theoretical settings, including one realistic setting: attacks against linear models.

Experiments show the attack attains good results in practice.

We show that simple defenses are able to perform well on MNIST but not on CIFAR, contributing further to previous calls that MNIST should be retired as a benchmarking dataset for adversarial robustness research.

We release code for these evaluations as part of the cleverhans (Papernot et al 2018) library  (ICLR reviewers should be careful not to look at who contributed these features to cleverhans to avoid de-anonymizing this submission).

We present metrics and an optimal attack for evaluating models that defend against adversarial examples using confidence thresholding

This paper introduces a family of attack on confidence thresholding algortihms, focusing mainly on evaluation methodologies.

Proposes an evaluation method for confidence thresholding defense models and an approach for generating adversarial examples by choosing the wrong class with the most confidence when using targeted attacks

The paper presents an evaluation methodology for evaluating attacks on confidence thresholding methods and proposes a new kind of attack.

Evaluation Methodology for Attacks Against Confidence Thresholding Models

attack confidence thresholding
evaluation methodology
adversarial example
model
evaluate
present
method


Deep learning has achieved remarkable successes in solving challenging reinforcement learning (RL) problems when dense reward function is provided.

However, in sparse reward environment it still often suffers from the need to carefully shape reward function to guide policy optimization.

This limits the applicability of RL in the real world since both reinforcement learning and domain-specific knowledge are required.

It is therefore of great practical importance to develop algorithms which can learn from a binary signal indicating successful task completion or other unshaped, sparse reward signals.

We propose a novel method called competitive experience replay, which efficiently supplements a sparse reward by placing learning in the context of an exploration competition between a pair of agents.

Our method complements the recently proposed hindsight experience replay (HER) by inducing an automatic exploratory curriculum.

We evaluate our approach on the tasks of reaching various goal locations in an ant maze and manipulating objects with a robotic arm.

Each task provides only binary rewards indicating whether or not the goal is achieved.

Our method asymmetrically augments these sparse rewards for a pair of agents each learning the same task, creating a competitive game designed to drive exploration.

Extensive experiments demonstrate that this method leads to faster converge and improved task performance.

a novel method to learn with sparse reward using adversarial reward re-labeling

Proposes to use a competitive multi-agent setting for encouraging exploration and shows that CER + HER > HER ~ CER

Propose a new method for learning from sparse rewards in model-free reinforcement learning settings and densifying reward

To address the sparse reward problems and encourage exploration in RL algorithms, the authors propose a relabeling strategy called Competitive Experience Reply (CER).

Competitive experience replay

method learn sparse reward
encourage exploration
setting
cer
competitive experience


This paper proposes a neural end-to-end text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions.

The model is formulated as a conditional generative model with two levels of hierarchical latent variables.

The first level is a categorical variable, which represents attribute groups (e.g. clean/noisy) and provides interpretability.

The second level, conditioned on the first, is a multivariate Gaussian variable, which characterizes specific attribute configurations (e.g. noise level, speaking rate) and enables disentangled fine-grained control over these attributes.

This amounts to using a Gaussian mixture model (GMM) for the latent distribution.

Extensive evaluation demonstrates its ability to control the aforementioned attributes.

In particular, it is capable of consistently synthesizing high-quality clean speech regardless of the quality of the training data for the target speaker.

Building a TTS model with Gaussian Mixture VAEs enables fine-grained control of speaking style, noise condition, and more.

Describes the conditioned GAN model to generate speaker conditioned Mel spectra by augmenting the z-space corresponding to the identification

This paper proposes a two layer latent variable model to obtain disentangled latent representation, thus facilitating fine-grained control over various attributes

This paper proposes a model that can control non-annotated attributes such as speaking style, accent, background noise, etc.

Hierarchical Generative Modeling for Controllable Speech Synthesis

speak style
fine-grained control
model
noise
condition
attribute


Visual Question Answering (VQA) models have struggled with counting objects in natural images so far.

We identify a fundamental problem due to soft attention in these models as a cause.

To circumvent this problem, we propose a neural network component that allows robust counting from object proposals.

Experiments on a toy task show the effectiveness of this component and we obtain state-of-the-art accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model.

On a difficult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.

Enabling Visual Question Answering models to count by handling overlapping object proposals.

This paper proposes a hand-designed network architecture on a graph of object proposals to perform soft non-maximum suppression to get object count.

Focuses on a counting problem in visual question answering using attention mechanism and propsoe a differentiable counting compoent which explicitly counts the number of objects

This paper tackles the object counting problem in visual question answering, it proposes many heuristics to find the correct count.

Learning to Count Objects in Natural Images for Visual Question Answering

counting problem visual question answer
object proposal
count


We propose a simple and robust training-free approach for building sentence representations.

Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and its surrounding context in a sentence.

We model the semantic meaning of a word in a sentence based on two aspects.

One is its relatedness to the word vector subspace already spanned by its contextual words.

The other is its novel semantic meaning which shall be introduced as a new basis vector perpendicular to this existing subspace.

Following this motivation, we develop an innovative method based on orthogonal basis to combine pre-trained word embeddings into sentence representation.

This approach requires zero training and zero parameters, along with efficient inference performance.

We evaluate our approach on 11 downstream NLP tasks.

Experimental results show that our model outperforms all existing zero-training alternatives in all the tasks and it is competitive to other approaches relying on either large amounts of labelled data or prolonged training time.

A simple and training-free approach for sentence embeddings with competitive performance compared with sophisticated models requiring either large amount of training data or prolonged training time.

Presented a new training-free way of generating sentence embedding with systematic analysis

Proposes a new geometry-based method for sentence embedding from word embedding vectors by quantifying the novelty, significance, and corpus-uniqueness of each word

This paper explores sentence embedding based on orthogonal decomposition of the spanned space by word embeddings

Zero-training Sentence Embedding via Orthogonal Basis

sentence embed
new
word
embedding
training-free
orthogonal


In few-shot classification, we are interested in learning algorithms that train a classifier from only a handful of labeled examples.

Recent progress in few-shot classification has featured meta-learning, in which a parameterized model for a learning algorithm is defined and trained on episodes representing different classification problems, each with a small labeled training set and its corresponding test set.

In this work, we advance this few-shot classification paradigm towards a scenario where unlabeled examples are also available within each episode.

We consider two situations: one where all unlabeled examples are assumed to belong to the same set of classes as the labeled examples of the episode, as well as the more challenging situation where examples from other distractor classes are also provided.

To address this paradigm, we propose novel extensions of Prototypical Networks (Snell et al., 2017) that are augmented with the ability to use unlabeled examples when producing prototypes.

These models are trained in an end-to-end way on episodes, to learn to leverage the unlabeled examples successfully.

We evaluate these methods on versions of the Omniglot and miniImageNet benchmarks, adapted to this new framework augmented with unlabeled examples.

We also propose a new split of ImageNet, consisting of a large set of classes, with a hierarchical structure.

Our experiments confirm that our Prototypical Networks can learn to improve their predictions due to unlabeled examples, much like a semi-supervised algorithm would.

We propose novel extensions of Prototypical Networks that are augmented with the ability to use unlabeled examples when producing prototypes.

This paper is an extension of a prototypical network that considers employing the unlabeled examples available to help train each episode

Studies the problem of semi-supervised few-shot classification by extending the prototypical networks into the setting of semi-supervised learning with example from distractor classes

Extends the Prototypical Network to the semi-supervised setting by updating prototypes using assigned pseudo-labels, dealing with distractors, and weighing samples using distance to the original prototypes.

Meta-Learning for Semi-Supervised Few-Shot Classification

extension prototypical network
semi-supervised few-shot classification
unlabeled example
prototype
setting
distractor
extend


We investigate the properties of multidimensional probability distributions in the context of latent space prior distributions of implicit generative models.

Our work revolves around the phenomena arising while decoding linear interpolations between two random latent vectors -- regions of latent space in close proximity to the origin of the space are oversampled, which restricts the usability of linear interpolations as a tool to analyse the latent space.

We show that the distribution mismatch can be eliminated completely by a proper choice of the latent probability distribution or using non-linear interpolations.

We prove that there is a trade off between the interpolation being linear, and the latent distribution having even the most basic properties required for stable training, such as finite mean.

We use the multidimensional Cauchy distribution as an example of the prior distribution, and also provide a general method of creating non-linear interpolations, that is easily applicable to a large family of commonly used latent distributions.

We theoretically prove that linear interpolations are unsuitable for analysis of trained implicit generative models.

Studies the problem of when the linear interpolant between two random variables follows the same distribution, related to prior distribution of an implicit generative model

This work asks how to interpolate in the latent space given a latent variable model.

Distribution-Interpolation Trade off in Generative Models

implicit generative model
linear
variable


Deep neural networks (DNN) have shown promising performance in computer vision.

In medical imaging, encouraging results have been achieved with deep learning for applications such as segmentation, lesion detection and classification.

Nearly all of the deep learning based image analysis methods work on reconstructed images, which are obtained from original acquisitions via solving inverse problems (reconstruction).

The reconstruction algorithms are designed for human observers, but not necessarily optimized for DNNs which can often observe features that are incomprehensible for human eyes.

Hence, it is desirable to train the DNNs directly from the original data which lie in a different domain with the images.

In this paper, we proposed an end-to-end DNN for abnormality detection in medical imaging.

To align the acquisition with the annotations made by radiologists in the image domain, a DNN was built as the unrolled version of iterative reconstruction algorithms to map the acquisitions to images, and followed by a 3D convolutional neural network (CNN) to detect the abnormality in the reconstructed images.

The two networks were trained jointly in order to optimize the entire DNN for the detection task from the original acquisitions.

The DNN was implemented for lung nodule detection in low-dose chest computed tomography (CT), where a numerical simulation was done to generate acquisitions from 1,018 chest CT images with radiologists' annotations.

The proposed end-to-end DNN demonstrated better sensitivity and accuracy for the task compared to a two-step approach, in which the reconstruction and detection DNNs were trained separately.

A significant reduction of false positive rate on suspicious lesions were observed, which is crucial for the known over-diagnosis in low-dose lung CT imaging.

The images reconstructed by the proposed end-to-end network also presented enhanced details in the region of interest.

Detection of lung nodule starting from projection data rather than images.

DNNs are used for patch-based lung nodule detection in CT projection data.

Jointly modeling computed tomography reconstruction and lesion detection in the lung by training the mapping from raw sinogram to detection outputs in an end-to-end manner

Presents an end to end training of a CNN architecture that combines CT image signal processing and image analysis.

End-to-End Abnormality Detection in Medical Imaging

projection datum
lung nodule
detection
image
ct
end-to-end


Deep reinforcement learning (DRL) algorithms have demonstrated progress in learning to find a goal in challenging environments.

As the title of the paper by Mirowski et al. (2016) suggests, one might assume that DRL-based algorithms are able to “learn to navigate” and are thus ready to replace classical mapping and path-planning algorithms, at least in simulated environments.

Yet, from experiments and analysis in this earlier work, it is not clear what strategies are used by these algorithms in navigating the mazes and finding the goal.

In this paper, we pose and study this underlying question: are DRL algorithms doing some form of mapping and/or path-planning?

Our experiments show that the algorithms are not memorizing the maps of mazes at the testing stage but, rather, at the training stage.

Hence, the DRL algorithms fall short of qualifying as mapping or path-planning algorithms with any reasonable definition of mapping.

We extend the experiments in Mirowski et al. (2016) by separating the set of training and testing maps and by a more ablative coverage of the space of experiments.

Our systematic experiments show that the NavA3C-D1-D2-L algorithm, when trained and tested on the same maps, is able to choose the shorter paths to the goal.

However, when tested on unseen maps the algorithm utilizes a wall-following strategy to find the goal without doing any mapping or path planning.

We quantitatively and qualitatively evaluate deep reinforcement learning based navigation methods under a variety of conditions to answer the question of how close they are to replacing classical path planners and mapping algorithms.

Evaluate a Deep RL-based model on training mazes by measuring repeated latency to goal and comparison to shortest route

Do Deep Reinforcement Learning Algorithms really Learn to Navigate?

deep reinforcement learning
evaluate
algorithm


In many robotic applications, it is crucial to maintain a belief about the state of

a system, like the location of a robot or the pose of an object.

These state estimates serve as input for planning and decision making and

Recursive Bayesian Filtering algorithms address the state estimation problem,

but they require a model of the process dynamics and the sensory observations as well as

noise estimates that quantify the accuracy of these models.

Recently, multiple works have demonstrated that the process and sensor models can be

learned by end-to-end training through differentiable versions of Recursive Filtering methods.

However, even if the predictive models are known, finding suitable noise models

Therefore, many practical applications rely on very simplistic noise

Our hypothesis is that end-to-end training through differentiable Bayesian

Filters enables us to learn more complex heteroscedastic noise models for

We evaluate learning such models with different types of

filtering algorithms and on two different robotic tasks.

for sampling-based filters like the Particle Filter, learning heteroscedastic noise

models can drastically improve the tracking performance in comparison to using

We evaluate learning heteroscedastic noise models within different Differentiable Bayes Filters

Proposes to learn heteroscedastic noise models from data by optimizing the prediction likelihood end-toend through differentiable Bayesian Filters and two different versions of the Unscented Kalman Filter

Revisits Bayes filters and evaluates the benefit of training the observation and process noise models while keeping all other models fixed

This paper presents a method to learn and use state and observation dependent noise in traditional Bayesian filtering algorithms. The approach consists of constructing a neural network model which takes as input the raw observation data and produces a compact representation and an associated diagonal covariance.

On Learning Heteroscedastic Noise Models within Differentiable Bayes Filters

learn heteroscedastic noise model within


Graph convolutional neural networks have recently shown great potential for the task of zero-shot learning.

These models are highly sample efficient as related concepts in the graph structure share statistical strength allowing generalization to new classes when faced with a lack of data.

However, we find that the extensive use of Laplacian smoothing at each layer in current approaches can easily dilute the knowledge from distant nodes and consequently decrease the performance in zero-shot learning.

In order to still enjoy the benefit brought by the graph structure while preventing the dilution of knowledge from distant nodes, we propose a Dense Graph Propagation (DGP) module with carefully designed direct links among distant nodes.

DGP allows us to exploit the hierarchical graph structure of the knowledge graph through additional connections.

These connections are added based on a node's relationship to its ancestors and descendants.

A weighting scheme is further used to weigh their contribution depending on the distance to the node.

Combined with finetuning of the representations in a two-stage training approach our method outperforms state-of-the-art zero-shot learning approaches.

We rethink the way information can be exploited more efficiently in the knowledge graph in order to improve performance on the Zero-Shot Learning task and propose a dense graph propagation (DGP) module for this purpose.

This authors propose a solution to the problem of over-smoothing in Graph conv networks by allowing dense propagation between all related nodes, weighted by the mutual distance.

Proposes a novel graph convolutional neural network to tackle the problem of zero-shot classification by using relational structures between classes as input of graph convolutional networks to learn classifiers of unseen classes

Rethinking Knowledge Graph Propagation for Zero-Shot Learning

graph propagation
problem
zero-shot
network
learn
knowledge
rethink
dense


In this paper, we propose a capsule-based neural network model to solve the semantic segmentation problem.

By taking advantage of the extractable part-whole dependencies available in capsule layers, we derive the probabilities of the class labels for individual capsules through a recursive, layer-by-layer procedure.

We model this procedure as a traceback pipeline and take it as a central piece to build an end-to-end segmentation network.

Under the proposed framework, image-level class labels and object boundaries are jointly sought in an explicit manner, which poses a significant advantage over the state-of-the-art fully convolutional network (FCN) solutions.

Experiments conducted on modified MNIST and neuroimages demonstrate that our model considerably enhance the segmentation performance compared to the leading FCN variant.

A capsule-based semantic segmentation, in which the probabilities of the class labels are traced back through capsule pipeline.

The authors present a trace-back mechanism to associate lowest level of Capsules with their respective classes

Proposes a traceback layer for capsule networks to do semantic segmentation and makes explicit use of part-whole relationship in the capsule layers

Proposes a trace-back method based on the CapsNet concept of Sabour to perform a semantic segmentation in parallel to classification.

Trace-back along capsules and its application on semantic segmentation

semantic segmentation
capsule
trace-back
class


Studying the evolution of information theoretic quantities during Stochastic Gradient Descent (SGD) learning of Artificial Neural Networks (ANNs) has gained popularity in recent years.

Nevertheless, these type of experiments require estimating mutual information and entropy which becomes intractable for moderately large problems.

In this work we propose a framework for understanding SGD learning in the information plane which consists of observing entropy and conditional entropy of the output labels of ANN.

Through experimental results and theoretical justifications it is shown that, under some assumptions, the SGD learning trajectories appear to be similar for different ANN architectures.

First, the SGD learning is modeled as a Hidden Markov Process (HMP) whose entropy tends to increase to the maximum.

Then, it is shown that the SGD learning trajectory appears to move close to the shortest path between the initial and final joint distributions in the space of probability measures equipped with the total variation metric.

Furthermore, it is shown that the trajectory of learning in the information plane can provide an alternative for observing the learning process, with potentially richer information about the learning than the trajectories in training and test error.

We look at SGD as a trajectory in the space of probability measures, show its connection to Markov processes, propose a simple Markov model of SGD learning, and experimentally compare it with SGD using information theoretic quantities.

Constructs a Markov chain that follows a shorted path in TV metric on P and shows that trajectories of SGD and \alpha-SMLC have similar conditional entropy

Studies the trajectory of H(\hat{y}) versus H(\hat{y}|y) on the information plane for stochastic gradient descent methods for training neural networks

Describes SGD from the point of view of the distribution p(y',y) where y is (a possibly corrupted) true class-label and y' a model prediction.

On the Trajectory of Stochastic Gradient Descent in the Information Plane

trajectory
stochastic gradient descent
information plane
sgd
show
markov
model