Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks.

Analogously, this suggests that learned update functions may similarly outperform current hand-designed optimizers, especially for specific tasks.

However, learned optimizers are notoriously difficult to train and have yet to demonstrate wall-clock speedups over hand-designed optimizers, and thus are rarely used in practice.

Typically, learned optimizers are trained by truncated backpropagation through an unrolled optimization process.

The resulting gradients are either strongly biased (for short truncations) or have exploding norm (for long truncations).

In this work we propose a training scheme which overcomes both of these difficulties, by dynamically weighting two unbiased gradient estimators for a variational loss on optimizer performance.

This allows us to train neural networks to perform optimization faster than well tuned first-order methods.

Moreover, by training the optimizer against validation loss, as opposed to training loss, we are able to use it to train models which generalize better than those trained by first order methods.

We demonstrate these results on problems where our learned optimizer trains convolutional networks in a fifth of the wall-clock time compared to tuned first-order methods, and with an improvement

We analyze problems when training learned optimizers, address those problems via variational optimization using two complementary gradient estimators, and train optimizers that are 5x faster in wall-clock time than baseline optimizers (e.g. Adam).

This paper uses un-rolled optimization to learn neural networks for optimization.

This paper tackles the problem of learning an optimizer, specifically the authors focus on obtaining cleaner gradients from the unrolled training procedure.

Presents a method for "learning an optimizer" by using a Variational Optimization for the "outer" optimizer loss and proposes the idea to combine both the reparametrized gradient and the score-function estimator for the Variational Objective and weights them using a product of Gaussians formula for the mean.

Learned optimizers that outperform on wall-clock and validation loss

estimator
variational optimization
learn optimizer
gradient
wall-clock
problem
training
loss


Asynchronous distributed gradient descent algorithms for training of deep neural

networks are usually considered as inefficient, mainly because of the Gradient delay

In this paper, we propose a novel asynchronous distributed algorithm

that tackles this limitation by well-thought-out averaging of model updates, computed

The algorithm allows computing gradients along the process

of gradient merge, thus, reducing or even completely eliminating worker idle time

due to communication overhead, which is a pitfall of existing asynchronous methods.

We provide theoretical analysis of the proposed asynchronous algorithm,

According to our analysis, the crucial parameter for

keeping high convergence rate is the maximal discrepancy between local parameter

As long as it is kept relatively small, the

convergence rate of the algorithm is shown to be the same as the one of a sequential

Furthermore, in our algorithm, this discrepancy is bounded

by an expression that involves the staleness parameter of the algorithm, and is

our approach and other solutions, such as Elastic Asynchronous SGD or Downpour

SGD, in which that maximal discrepancy is bounded by an expression that

depends on the number of workers, due to gradient delay problem.

effectiveness of our approach, we conduct a series of experiments on image

classification task on a cluster with 4 machines, equipped with a commodity communication

switch and with a single GPU card per machine.

show a linear scaling on 4-machine cluster without sacrificing the test accuracy,

while eliminating almost completely worker idle time.

using commodity communication switch, it paves a way for large scale distributed

training performed on commodity clusters.

A method for an efficient asynchronous distributed training of deep learning models along with theoretical regret bounds.

The paper proposes an algorithm to restrict the staleness in asynchronous SGD and provides theoretical analysis

Proposes a hybrid-algorithm to eliminate the gradient delay of asynchronous methods.

Asynchronous SGD without gradient delay for efficient distributed training

gradient delay
method
efficient
distribute training
asynchronous sgd
theoretical


Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs.

However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks.

This paper aims to raise people's awareness about the security of the quantized models, and we designed a novel quantization methodology to jointly optimize the efficiency and robustness of deep learning models.

We first conduct an empirical study to show that vanilla quantization suffers more from adversarial attacks.

We observe that the inferior robustness comes from the error amplification effect, where the quantization operation further enlarges the distance caused by amplified noise.

Then we propose a novel Defensive Quantization (DQ) method by controlling the Lipschitz constant of the network during quantization, such that the magnitude of the adversarial noise remains non-expansive during inference.

Extensive experiments on CIFAR-10 and SVHN datasets demonstrate that our new quantization method can defend neural networks against adversarial examples, and even achieves superior robustness than their full-precision counterparts, while maintaining the same hardware efficiency as vanilla quantization approaches.

As a by-product, DQ can also improve the accuracy of quantized models without adversarial attack.

We designed a novel quantization methodology to jointly optimize the efficiency and robustness of deep learning models.

Proposes a regularization scheme to protect quantized neural networks from adversarial attacks using a Lipschitz constant filitering of the inner layers' inpout-output.

Defensive Quantization: When Efficiency Meets Robustness


Recurrent Neural Networks (RNNs) continue to show  outstanding performance in sequence modeling tasks.

However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies.

In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time.

We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph.

This model can also be encouraged to perform fewer state updates through a budget constraint.

We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models.

Source code is publicly available at https://imatge-upc.github.io/skiprnn-2017-telecombcn/.

A modification for existing RNN architectures which allows them to skip state updates while preserving the performance of the original architectures.

Proposes the Skip RNN model which allows a recurrent network to selectively skip updating its hidden state for some inputs, leading to reduced computation at test-time.

Proposes a novel RNN model where both the input and the state update of the recurrent cells are skipped adaptively for some time steps.

Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks

skip state update
input
allow
rnn model
recurrent
network


We propose a fast second-order method that can be used as a drop-in replacement for current deep learning solvers.

Compared to stochastic gradient descent (SGD), it only requires two additional forward-mode automatic differentiation operations per iteration, which has a computational cost comparable to two standard forward passes and is easy to implement.

Our method addresses long-standing issues with current second-order solvers, which invert an approximate Hessian matrix every iteration exactly or by conjugate-gradient methods, procedures that are much slower than a SGD step.

Instead, we propose to keep a single estimate of the gradient projected by the inverse Hessian matrix, and update it once per iteration with just two passes over the network.

This estimate has the same size and is similar to the momentum variable that is commonly used in SGD.

No estimate of the Hessian is maintained.

We first validate our method, called CurveBall, on small problems with known solutions (noisy Rosenbrock function and degenerate 2-layer linear networks), where current deep learning solvers struggle.

We then train several large models on CIFAR and ImageNet, including ResNet and VGG-f networks, where we demonstrate faster convergence with no hyperparameter tuning.

We also show our optimiser's generality by testing on a large set of randomly-generated architectures.

A fast second-order solver for deep learning that works on ImageNet-scale problems with no hyper-parameter tuning

Choosing direction by using a single step of gradient descent "towards Newton step" from an original estimate, and then taking this direction instead of original gradient

A new approximate second-order optimization method with low computational cost that replaces the computation of the Hessian matrix with a single gradient step and a warm start strategy.

Small steps and giant leaps: Minimal Newton solvers for Deep Learning

solver deep learning
single
step
second-order
gradient
newton


The recently presented idea to learn heuristics for combinatorial optimization problems is promising as it can save costly development.

However, to push this idea towards practical implementation, we need better models and better ways of training.

We contribute in both directions: we propose a model based on attention layers with benefits over the Pointer Network and we show how to train this model using REINFORCE with a simple baseline based on a deterministic greedy rollout, which we find is more efficient than using a value function.

We significantly improve over recent learned heuristics for the Travelling Salesman Problem (TSP), getting close to optimal results for problems up to 100 nodes.

With the same hyperparameters, we learn strong heuristics for two variants of the Vehicle Routing Problem (VRP), the Orienteering Problem (OP) and (a stochastic variant of) the Prize Collecting TSP (PCTSP), outperforming a wide range of baselines and getting results close to highly optimized and specialized algorithms.

Attention based model trained with REINFORCE with greedy rollout baseline to learn heuristics with competitive results on TSP and other routing problems

Presents an attention-based approach to learning a policy for solving TSP and other routing-type combinatorial optimzation problems.

This paper trys to learn heuristics for solving combinatorial optimisation problems

Attention, Learn to Solve Routing Problems!

learn heuristic
tsp
solve
attention
problem
combinatorial


We propose an efficient online hyperparameter optimization method which uses a joint dynamical system to evaluate the gradient with respect to the hyperparameters.

While similar methods are usually limited to hyperparameters with a smooth impact on the model, we show how to apply it to the probability of dropout in neural networks.

Finally, we show its effectiveness on two distinct tasks.

An algorithm for optimizing regularization hyper-parameters during training

The paper proposes a way to re-initialize y at each update of lambda and a clipping procedure of y to maintain the stability of the dynamical system.

Proposes an algorithm for hyperparameter optimization that can be seen as an extension of Franceschi 2017 were some estimates are warm restarted to increase the stability of the method.

Proposes an extension to an existing method to optimize regularization hyperparameters.

Online Hyper-Parameter Optimization

stability
extension
optimize regularization
algorithm
method
hyperparameter
hyper-parameter
optimization


Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks.

However, these have been evaluated using differing codebases and limited computational resources, which represent uncontrolled sources of experimental variation.

We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models.

We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the Hutter Prize dataset.

Show that LSTMs are as good or better than recent innovations for LM and that model evaluation is often unreliable.

This paper describes a comprehensive validation of LSTM-based word and character language models, leading to a significant result in language modeling and a milestone in deep learning.

On the State of the Art of Evaluation in Neural Language Models


  Residual and skip connections play an important role in many current
  generative models.

Although their theoretical and numerical advantages
  are understood, their role in speech enhancement systems has not been
  investigated so far.

When performing spectral speech enhancement,
  residual connections are very similar in nature to spectral subtraction,
  which is the one of the most commonly employed speech enhancement approaches.

Highway networks, on the other hand, can be seen as a combination of spectral
  masking and spectral subtraction.

However, when using deep neural networks, such operations would
  normally happen in a transformed spectral domain, as opposed to traditional speech
  enhancement where all operations are often done directly on the spectrum.

In this paper, we aim to investigate the role of residual and highway
  connections in deep neural networks for speech enhancement, and verify whether
  or not they operate similarly to their traditional, digital signal processing
  counterparts.

We visualize the outputs of such connections, projected back to
  the spectral domain, in models trained for speech denoising, and show that while
  skip connections do not necessarily improve performance with regards to the
  number of parameters, they make speech enhancement models more interpretable.

We show how using skip connections can make speech enhancement models more interpretable, as it makes them use similar mechanisms that have been explored in the DSP literature.

The authors propose incorporating Residual, Highway and Masking blocks inside a fully convolutional pipeline in order to understand how iterative inference of the output and the masking is performed in a speech enhancement task

The authors interpret highway, residual and masking connections.

The authors generate their own noisy speech by artificially adding noise from a well established noise data-set to a less know clean speech data-set.

Investigating the effect of residual and highway connections in speech enhancement models

speech enhancement model
residual highway
connection
author
mask


Bayesian neural networks (BNNs) hold great promise as a flexible and principled solution to deal with uncertainty when learning from finite data.

Among approaches to realize probabilistic inference in deep neural networks, variational Bayes (VB) is theoretically grounded, generally applicable, and computationally efficient.

With wide recognition of potential advantages, why is it that variational Bayes has seen very limited practical use for BNNs in real applications?

We argue that variational inference in neural networks is fragile: successful implementations require careful initialization and tuning of prior variances, as well as controlling the variance of Monte Carlo gradient estimates.

We provide two innovations that aim to turn VB into a robust inference tool for Bayesian neural networks: first, we introduce a novel deterministic method to approximate moments in neural networks, eliminating gradient variance; second, we introduce a hierarchical prior for parameters and a novel Empirical Bayes procedure for automatically selecting prior variances.

Combining these two innovations, the resulting method is highly efficient and robust.

On the application of heteroscedastic regression we demonstrate good predictive performance over alternative approaches.

A method for eliminating gradient variance and automatically tuning priors for effective training of bayesian neural networks

Proposes a new approach to perform deterministic variational inference for feed-forward BNN with specific nonlinear activation functions by approximating layerwise moments.

The paper considers a purely deterministic approach to learning variational posterior approximations for Bayesian neural networks.

Deterministic Variational Inference for Robust Bayesian Neural Networks

bayesian neural network
deterministic variational inference
approach


Skills learned through (deep) reinforcement learning often generalizes poorly

across tasks and re-training is necessary when presented with a new task.

present a framework that combines techniques in formal methods with reinforcement

learning (RL) that allows for the convenient specification of complex temporal

dependent tasks with logical expressions and construction of new skills from existing

composition technique and evaluate on a simple grid world simulation as well as

A formal method's approach to skill composition in reinforcement learning tasks

The paper combines RL and constraints expressed by logical formulas by setting up an automation from scTLTL formulas.

Proposes a method that helps to construct policy from learned subtasks on the topic of combining RL tasks with linear temporal logic formulas.

Automata Guided Skill Composition

combine rl
skill composition
formula
method
task


The application of multi-modal generative models by means of a Variational Auto Encoder (VAE) is an upcoming research topic for sensor fusion and bi-directional modality exchange.

This contribution gives insights into the learned joint latent representation and shows that expressiveness and coherence are decisive properties for multi-modal datasets.

Furthermore, we propose a multi-modal VAE derived from the full joint marginal log-likelihood that is able to learn the most meaningful representation for ambiguous observations.

Since the properties of multi-modal sensor setups are essential for our approach but hardly available, we also propose a technique to generate correlated datasets from uni-modal ones.

Deriving a general formulation of a multi-modal VAE from the joint marginal log-likelihood.

Proposes a multi-modal VAE with a variational bound derived from the chain rule.

This paper proposes an objective, M^2VAE, for multi-modal VAEs, which is supposed to learn a more meaningful latent space representation.

Deep Generative Models for learning Coherent Latent Representations from Multi-Modal Data

multi-modal vae
derive
learn
representation
latent


We build on auto-encoding sequential Monte Carlo (AESMC): a method for model and proposal learning based on maximizing the lower bound to the log marginal likelihood in a broad family of structured probabilistic models.

Our approach relies on the efficiency of sequential Monte Carlo (SMC) for performing inference in structured probabilistic models and the flexibility of deep neural networks to model complex conditional probability distributions.

We develop additional theoretical insights and introduce a new training procedure which improves both model and proposal learning.

We demonstrate that our approach provides a fast, easy-to-implement and scalable means for simultaneous model learning and proposal adaptation in deep generative models.

We build on auto-encoding sequential Monte Carlo, gain new theoretical insights and develop an improved training procedure based on those insights.

The paper proposes a version of IWAE-style training that uses SMC instead of classical importance sampling.

This work proposes auto-encoding sequential Monte Carlo (SMC), extending the VAE framework to a new Monte Carto objective based on SMC.

Auto-Encoding Sequential Monte Carlo

auto-encode sequential monte carlo
smc
base
training
new


A key component for many reinforcement learning agents is to learn a value function, either for policy evaluation or control.

Many of the algorithms for learning values, however, are designed for linear function approximation---with a fixed basis or fixed representation.

Though there have been a few sound extensions to nonlinear function approximation, such as nonlinear gradient temporal difference learning, these methods have largely not been adopted, eschewed in favour of simpler but not sound methods like temporal difference learning and Q-learning.

In this work, we provide a two-timescale network (TTN) architecture that enables linear methods to be used to learn values, with a nonlinear representation learned at a slower timescale.

The approach facilitates the use of algorithms developed for the linear setting, such as data-efficient least-squares methods, eligibility traces and the myriad of recently developed linear policy evaluation algorithms, to provide nonlinear value estimates.

We prove convergence for TTNs, with particular care given to ensure convergence of the fast linear component under potentially dependent features provided by the learned representation.

We empirically demonstrate the benefits of TTNs, compared to other nonlinear value function approximation algorithms, both for policy evaluation and control.

We propose an architecture for learning value functions which allows the use of any linear policy evaluation algorithm in tandem with nonlinear feature learning.

The paper proposes a two-timescale framework for learning the value function and a state representation altogether with nonlinear approximators.

This paper proposes Two-Timescale Networks (TTNs) and prove the convergence of this method using methods from two time-scale stochastic approximation.

This paper presents a Two-Timescale Network (TTN) that enables linear methods to be used to learn values.

Two-Timescale Networks for Nonlinear Value Function Approximation

learn value function
two-timescale
method
nonlinear
network
linear
approximation


Large-scale Long Short-Term Memory (LSTM) cells are often the building blocks of many state-of-the-art algorithms for tasks in Natural Language Processing (NLP).

However, LSTMs are known to be computationally inefficient because the memory capacity of the models depends on the number of parameters, and the inherent recurrence that models the temporal dependency is not parallelizable.

In this paper, we propose simple, but effective, low-rank matrix factorization (MF) algorithms to compress network parameters and significantly speed up LSTMs with almost no loss of performance (and sometimes even gain).

To show the effectiveness of our method across different tasks, we examine two settings:

1) compressing core LSTM layers in Language Models,

2) compressing biLSTM layers of ELMo~\citep{ELMo} and evaluate in three downstream NLP tasks (Sentiment Analysis, Textual Entailment, and Question Answering).

The latter is particularly interesting as embeddings from large pre-trained biLSTM Language Models are often used as contextual word representations.

Finally, we discover that matrix factorization performs better in general, additive recurrence is often more important than multiplicative recurrence, and we identify an interesting correlation between matrix norms and compression performance.

We propose simple, but effective, low-rank matrix factorization (MF) algorithms to speed up in running time, save memory, and improve the performance of LSTMs.

Proposes to accelerate LSTM by using MF as the post-processing compression strategy and conducts extensive experiements to show the performance.

Low-Rank Matrix Factorization of LSTM as Effective Model Compression

low-rank matrix factorization
performance
mf
lstm
effective
compression


Manipulation and re-use of images in scientific publications is a recurring problem, at present lacking a scalable solution.

Existing tools for detecting image duplication are mostly manual or semi-automated, despite the fact that generating data for a learning-based approach is straightforward, as we here illustrate.

This paper addresses the problem of determining if, given two images, one is a manipulated version of the other by means of certain geometric and statistical manipulations, e.g. copy, rotation, translation, scale, perspective transform, histogram adjustment, partial erasing, and compression artifacts.

We propose a solution based on a 3-branch Siamese Convolutional Neural Network.

The ConvNet model is trained to map images into a 128-dimensional space, where the Euclidean distance between duplicate (respectively, unique) images is no greater (respectively, greater) than 1.

Our results suggest that such an approach can serve as tool to improve surveillance of the published and in-peer-review literature for image manipulation.

We also show that as a byproduct the network learns useful representations for semantic segmentation, with performance comparable to that of domain-specific models.

A forensic metric to determine if a given image is a copy (with possible manipulation) of another image from a given dataset.

Introduces the siamese network to identify duplicate and copied/modified images, which can be used to improve surveillance of the published and in-peer-review literature.

The paper presents an application of deep convolutional networks for the task of duplicate image detection

This work addresses the problem of finding duplicate/near duplicate images from biomedical publications and proposes a standard CNN and loss functions and apply it to this field.

A Forensic Representation to Detect Non-Trivial Image Duplicates, and How it Applies to Semantic Segmentation

apply image forensic copy duplicate network /


Training generative adversarial networks is unstable in high-dimensions as the true data distribution tends to be concentrated in a small fraction of the ambient space.

The discriminator is then quickly able to classify nearly all generated samples as fake, leaving the generator without meaningful gradients and causing it to deteriorate after a point in training.

In this work, we propose training a single generator simultaneously against an array of discriminators, each of which looks at a different random low-dimensional projection of the data.

Individual discriminators, now provided with restricted views of the input, are unable to reject generated samples perfectly and continue to provide meaningful gradients to the generator throughout training.

Meanwhile, the generator learns to produce samples consistent with the full data distribution to satisfy all discriminators simultaneously.

We  demonstrate the practical utility of this approach experimentally, and show that it is able to produce image samples with higher quality than traditional training with a single discriminator.

Stable GAN training in high dimensions by using an array of discriminators, each with a low dimensional view of generated samples

The paper proposes to stabilize GAN training by using an ensemble of discriminators, each working on a random projection of the input data, to provide the training signal for the generator model.

The paper proposes a GAN training method for improving the training stability.

The paper proposes a new approach to GAN training, which provides stable gradients to train the generator.

Stabilizing GAN Training with Multiple Random Projections

gan training
discriminator
provide
generator
random projection
stable
stabilize


We present a novel method to precisely impose tree-structured category information onto word-embeddings, resulting in ball embeddings in higher dimensional spaces (N-balls for short).

Inclusion relations among N-balls implicitly encode subordinate relations among categories.

The similarity measurement in terms of the cosine function is enriched by category information.

Using a geometric construction method instead of back-propagation, we create large N-ball embeddings that satisfy two conditions: (1) category trees are precisely imposed onto word embeddings at zero energy cost; (2) pre-trained word embeddings are well preserved.

A new benchmark data set is created for validating the category of unknown words.

Experiments show that N-ball embeddings, carrying category information, significantly outperform word embeddings in the test of nearest neighborhoods, and demonstrate surprisingly good performance in validating categories of unknown words.

Source codes and data-sets are free for public access \url{https://github.com/gnodisnait/nball4tree.git} and \url{https://github.com/gnodisnait/bp94nball.git}.

we show a geometric method to perfectly encode categroy tree information into pre-trained word-embeddings.

The paper proposes N-ball embedding for taxonomic data where an N-ball is a pair of a centroid vector and the radius from the center.

The paper presents a method for tweaking existing vector embeddings of categorical objects (such as words), to convert them to ball embeddings that follow hierarchies.

Focuses on adjusting the pretrained word embeddings so that they respect the hypernymy/hyponymy relationship by appropriate n-ball encapsulation.

Imposing Category Trees Onto Word-Embeddings Using A Geometric Construction

method word-embedding geometric embedding tree vector n-ball word


For the challenging semantic image segmentation task the best performing models

have traditionally combined the structured modelling capabilities of Conditional

Random Fields (CRFs) with the feature extraction power of CNNs.

works however, CRF post-processing has fallen out of favour.

is mainly due to the slow training and inference speeds of CRFs, as well as the

difficulty of learning the internal CRF parameters.

propose to add the assumption of conditional independence to the framework of

This allows us to reformulate the inference in terms of

convolutions, which can be implemented highly efficiently on GPUs.Doing so

speeds up inference and training by two orders of magnitude.

the convolutional CRFs can easily be optimized using backpropagation.

the goal of facilitating further CRF research we have made our implementations

We propose Convolutional CRFs a fast, powerful and trainable alternative to Fully Connected CRFs.

The authors replace the large filtering step in the permutohedral lattice with a spatially varying convolutional kernel and show that inference is more efficient and training is easier.

Proposes to perform message passing on a truncated Gaussian kernel CRF using a defined kernel and parallelized message passing on GPU.

Convolutional CRFs for Semantic Segmentation


Deep Learning NLP domain lacks procedures for the analysis of model robustness.

In this paper we propose a framework which validates robustness of any Question Answering model through model explainers.

We propose that output of a robust model should be invariant to alterations that do not change its semantics.

We test this property by manipulating question in two ways: swapping important question word for

2) for word vector that is close in embedding space.

We estimate importance of words in asked questions with Locally Interpretable Model Agnostic Explanations method (LIME).

With these two steps we compare state-of-the-art Q&A models.

We show that although accuracy of state-of-the-art models is high, they are very fragile to changes in the input.

We can choose architecture that is more immune to attacks and thus more robust and stable in production environment.

Morevoer, we propose 2 adversarial training scenarios which raise model sensitivity to true synonyms by up to 7% accuracy measure.

Our findings help to understand which models are more stable and how they can be improved.

In addition, we have created and published a new dataset that may be used for validation of robustness of a Q&A model.

We propose a model agnostic approach to validation of Q&A system robustness and demonstrate results on state-of-the-art Q&A models.

Addresses the problem of robustness to adversarial information in question answering.

Improving robustness of machine comprehension/question answering.

Are you tough enough? Framework for Robustness Validation of Machine Comprehension Systems

machine comprehension
validation
robustness
system
question


In this paper, we propose a mix-generator generative adversarial networks (PGAN) model that works in parallel by mixing multiple disjoint generators to approximate a complex real distribution.

In our model, we propose an adjustment component that collects all the generated data points from the generators, learns the boundary between each pair of generators, and provides error to separate the support of each of the generated distributions.

To overcome the instability in a multiplayer game, a shrinkage adjustment component method is introduced to gradually reduce the boundary between generators during the training procedure.

To address the linearly growing training time problem in a multiple generators model, we propose a method to train the generators in parallel.

This means that our work can be scaled up to large parallel computation frameworks.

We present an efficient loss function for the discriminator, an effective adjustment component, and a suitable generator.

We also show how to introduce the decay factor to stabilize the training procedure.

We have performed extensive experiments on synthetic datasets, MNIST, and CIFAR-10.

These experiments reveal that the error provided by the adjustment component could successfully separate the generated distributions and each of the generators can stably learn a part of the real distribution even if only a few modes are contained in the real distribution.

multi generator to capture Pdata, solve the competition and one-beat-all problem

Proposes parallel GANs to avoid mode collapse in GANs through a combination of multiple weak generators.

NOVEL AND EFFECTIVE PARALLEL MIX-GENERATOR GENERATIVE ADVERSARIAL NETWORKS


We capitalize on the natural compositional structure of images in order to learn object segmentation with weakly labeled images.

The intuition behind our approach is that removing objects from images will yield natural images, however removing random patches will yield unnatural images.

We leverage this signal to develop a generative model that decomposes an image into layers, and when all layers are combined, it reconstructs the input image.

However, when a layer is removed, the model learns to produce a different image that still looks natural to an adversary, which is possible by removing objects.

Experiments and visualizations suggest that this model automatically learns object segmentation on images labeled only by scene better than baselines.

Weakly-supervised image segmentation using compositional structure of images and generative models.

This paper creates a layered representation in order to better learn segmentation from unlabeled images.

This paper proposes a GAN-based generative model that decomposes images into multiple layers, where the objective of the GAN is to distinguish real images from images formed by combining the layers.

This paper proposes a neural network architecture around the idea of layered scene composition

Counterfactual Image Networks

generative model
image
segmentation
layered
network


Adversarial examples are a pervasive phenomenon of machine learning models where seemingly imperceptible perturbations to the input lead to misclassifications for otherwise statistically accurate models.

We propose a geometric framework, drawing on tools from the manifold reconstruction literature, to analyze the high-dimensional geometry of adversarial examples.

In particular, we highlight the importance of codimension: for low-dimensional data manifolds embedded in high-dimensional space there are many directions off the manifold in which to construct adversarial examples.

Adversarial examples are a natural consequence of learning a decision boundary that classifies the low-dimensional data manifold well, but classifies points near the manifold incorrectly.

Using our geometric framework we prove (1) a tradeoff between robustness under different norms, (2) that adversarial training in balls around the data is sample inefficient, and (3) sufficient sampling conditions under which nearest neighbor classifiers and ball-based adversarial training are robust.

We present a geometric framework for proving robustness guarantees and highlight the importance of codimension in adversarial examples.

This paper gives a theoretical analysis of adversarial examples, showing that  there exists a tradeoff between robustness in different norms, adversarial training is sample inefficient, and the nearest neighbor classifier can be robust under certain conditions.

On the Geometry of Adversarial Examples


Character-based neural machine translation (NMT) models alleviate out-of-vocabulary issues, learn morphology, and move us closer to completely end-to-end translation systems.

Unfortunately, they are also very brittle and easily falter when presented with noisy data.

In this paper, we confront NMT models with synthetic and natural sources of noise.

We find that state-of-the-art models fail to translate even moderately noisy texts that humans have no trouble comprehending.

We explore two approaches to increase model robustness: structure-invariant word representations and robust training on noisy texts.

We find that a model based on a character convolutional neural network is able to simultaneously learn representations robust to multiple kinds of noise.

This paper investigates the impact of character-level noise on 4 different neural machine translation systems

This paper empirically investigates the performance of character-level NMT systems in the face of character-level noise, both synthesized and natural.

This paper investigates the impact of noisy input on Machine Translation and tests ways to make NMT models more robust

Synthetic and Natural Noise Both Break Neural Machine Translation

investigate impact
character-level noise
neural machine translation
natural
system
nmt


As neural networks grow deeper and wider, learning networks with hard-threshold activations is becoming increasingly important, both for network quantization, which can drastically reduce time and energy requirements, and for creating large integrated systems of deep networks, which may have non-differentiable components and must avoid vanishing and exploding gradients for effective learning.

However, since gradient descent is not applicable to hard-threshold functions, it is not clear how to learn them in a principled way.

We address this problem by observing that setting targets for hard-threshold hidden units in order to minimize loss is a discrete optimization problem, and can be solved as such.

The discrete optimization goal is to find a set of targets such that each unit, including the output, has a linearly separable problem to solve.

Given these targets, the network decomposes into individual perceptrons, which can then be learned with standard convex approaches.

Based on this, we develop a recursive mini-batch algorithm for learning deep hard-threshold networks that includes the popular but poorly justified straight-through estimator as a special case.

Empirically, we show that our algorithm improves classification accuracy in a number of settings, including for AlexNet and ResNet-18 on ImageNet, when compared to the straight-through estimator.

We learn deep networks of hard-threshold units by setting hidden-unit targets using combinatorial optimization and weights by convex optimization, resulting in improved performance on ImageNet.

The paper explains and generalizes approaches for learning neural nets with hard activation.

This paper examines the problem of optimizing deep networks of hard-threshold units.

The paper discusses the problem of optimizing neural networks with hard threshold and proposes a novel solution to it with a collection of heuristics/approximations.

Deep Learning as a Mixed Convex-Combinatorial Optimization Problem

deep network hard-threshold unit
problem optimize
learn
hard
optimization
neural


The robust and efficient recognition of visual relations in images is a hallmark of biological vision.

Here, we argue that, despite recent progress in visual recognition, modern machine vision algorithms are severely limited in their ability to learn visual relations.

Through controlled experiments, we demonstrate that visual-relation problems strain convolutional neural networks (CNNs).

The networks eventually break altogether when rote memorization becomes impossible such as when the intra-class variability exceeds their capacity.

We further show that another type of feedforward network, called a relational network (RN), which was shown to successfully solve seemingly difficult visual question answering (VQA) problems on the CLEVR datasets, suffers similar limitations.

Motivated by the comparable success of biological vision, we argue that feedback mechanisms including working memory and attention are the key computational components underlying abstract visual reasoning.

Using a novel, controlled, visual-relation challenge, we show that same-different tasks critically strain the capacity of CNNs; we argue that visual relations can be better solved using attention-mnemonic strategies.

Demonstrates that convolutional and relational neural networks fail to solve visual relation problems by training networks on artificially generated visual relation data.

This paper explores how current CNN's and Relational Networks fail to recognize visual relations in images.

Not-So-CLEVR: Visual Relations Strain Feedforward Neural Networks

visual relation
network fail
solve
relational
cnn
strain
neural


Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.

Previous work has shown that the tracker can be trained in a simulator via reinforcement learning and deployed in real-world scenarios.

However, during training, such a method requires manually specifying the moving path of the target object to be tracked, which cannot ensure the tracker’s generalization on the unseen object moving patterns.

To learn a robust tracker for VAT, in this paper, we propose a novel adversarial RL method which adopts an Asymmetric Dueling mechanism, referred to as AD-VAT.

In AD-VAT, both the tracker and the target are approximated by end-to-end neural networks, and are trained via RL in a dueling/competitive manner: i.e., the tracker intends to lockup the target, while the target tries to escape from the tracker.

They are asymmetric in that the target is aware of the tracker, but not vice versa.

Specifically, besides its own observation, the target is fed with the tracker’s observation and action, and learns to predict the tracker’s reward as an auxiliary task.

We show that such an asymmetric dueling mechanism produces a stronger target, which in turn induces a more robust tracker.

To stabilize the training, we also propose a novel partial zero-sum reward for the tracker/target.

The experimental results, in both 2D and 3D environments, demonstrate that the proposed method leads to a faster convergence in training and yields more robust tracking behaviors in different testing scenarios.

For supplementary videos, see: https://www.youtube.com/playlist?list=PL9rZj4Mea7wOZkdajK1TsprRg8iUf51BS

The code is available at https://github.com/zfw1226/active_tracking_rl

We propose AD-VAT, where the tracker and the target object, viewed as two learnable agents, are opponents and can mutually enhance during training.

This work aims to address the visual active tracking problem with a training mechanism in which the tracker and target serve as mutual opponents

This paper presents a simple multi-agent Deep RL task where a moving tracker tries to follow a moving target.

Proposes a novel reward function - "partial zero sum", which only encourages the tracker-target competition when they are close and penalizes whey they are too far.

AD-VAT: An Asymmetric Dueling mechanism for learning Visual Active Tracking

tracker target
training
ad-vat
visual active tracking
mechanism
opponent


Identifying the hypernym relations that hold between words is a fundamental task in NLP.

Word embedding methods have recently shown some capability to encode hypernymy.

However, such methods tend not to explicitly encode the hypernym hierarchy that exists between words.

In this paper, we propose a method to learn a hierarchical word embedding in a speciﬁc order to capture the hypernymy.

To learn the word embeddings, the proposed method considers not only the hypernym relations that exists between words on a taxonomy, but also their contextual information in a large text corpus.

The experimental results on a supervised hypernymy detection and a newly-proposed hierarchical path completion tasks show the ability of the proposed method to encode the hierarchy.

Moreover, the proposed method outperforms previously proposed methods for learning word and hypernym-speciﬁc word embeddings on multiple benchmarks.

We presented a method to jointly learn a Hierarchical Word Embedding (HWE) using a corpus and a taxonomy for identifying the hypernymy relations between words.

The paper presents a method to jointly learn word embeddings using co-occurrence statistics as well as incorporating hierarchal information from semantic networks.

This paper proposed a joint learning method of hypernym from both raw text and supervised taxonomy data.

This paper proposes adding a measure of "distributional inclusion" difference to the GloVE objective for the purpose of representing hypernym relations.

Joint Learning of Hierarchical Word Embeddings from a Corpus and a Taxonomy

present method jointly learn
corpus taxonomy
hierarchical word
joint learning
relation
embedding
hypernym


While self-organizing principles have motivated much of early learning models, such principles have rarely been included in deep learning architectures.

Indeed, from a supervised learning perspective it seems that topographic constraints are rather decremental to optimal performance.

Here we study a network model that incorporates self-organizing maps into a supervised network and show how gradient learning results in a form of a self-organizing learning rule.

Moreover, we show that such a model is robust in the sense of its application to a variety of  areas, which is believed to be a hallmark of biological learning systems.

integration of self-organization and supervised learning in a hierarchical neural network

The paper discusses learning in a neural network with three layers, where the middle layer is topographically organized and investigates interplay between unsupervised and hierarchical supervised learning in biological context.

A supervised variant of Kohonen's self-organizing map (SOM), but where the linear output layer is replaced with squared error by a softmax layer with cross-entropy.

Proposes a model using hidden neurons with self-organising activation function, whose outputs feed to classifier with softmax output function.

Self-Organization adds application robustness to deep learners

supervised learning
neural network
hierarchical
layer
softmax
self-organization
output


Quantization of a neural network has an inherent problem called accumulated quantization error, which is the key obstacle towards ultra-low precision, e.g., 2- or 3-bit precision.

To resolve this problem, we propose precision highway, which forms an end-to-end high-precision information flow while performing the ultra-low-precision computation.

First, we describe how the precision highway reduce the accumulated quantization error in both convolutional and recurrent neural networks.

We also provide the quantitative analysis of the benefit of precision highway and evaluate the overhead on the state-of-the-art hardware accelerator.

In the experiments, our proposed method outperforms the best existing quantization methods while offering 3-bit weight/activation quantization with no accuracy loss and 2-bit quantization with a 2.45 % top-1 accuracy loss in ResNet-50.

We also report that the proposed method significantly outperforms the existing method in the 2-bit quantization of an LSTM for language modeling.

precision highway; a generalized concept of high-precision information flow for sub 4-bit quantization

Investigates the problem of neural network quantization by employing an end-to-end precision highway to reduce the accumulated quantization error and enable ultra-low precision in deep neural networks.

This paper studies methods to improve the performance of quantized neural networks

This paper proposes to keep a high activation/gradient flow in two kinds of networks structures, ResNet and LSTM.

Precision Highway for Ultra Low-precision Quantization

precision highway
flow
neural network
quantization


The vast majority of natural sensory data is temporally redundant.

For instance, video frames or audio samples which are sampled at nearby points in time tend to have similar values.

Typically, deep learning algorithms take no advantage of this redundancy to reduce computations.

We present a variant on backpropagation for neural networks in which computation scales with the rate of change of the data - not the rate at which we process the data.

We do this by implementing a form of Predictive Coding wherein neurons communicate a combination of their state, and their temporal change in state, and quantize this signal using Sigma-Delta modulation.

Intriguingly, this simple communication rule give rise to units that resemble biologically-inspired leaky integrate-and-fire neurons, and to a spike-timing-dependent weight-update similar to Spike-Timing Dependent Plasticity (STDP), a synaptic learning rule observed in the brain.

We demonstrate that on MNIST, on a temporal variant of MNIST, and on Youtube-BB, a dataset with videos in the wild, our algorithm performs about as well as a standard deep network trained with backpropagation, despite only communicating discrete values between layers.

An algorithm for training neural networks efficiently on temporally redundant data.

The paper describes a neural coding scheme for spike based learning in deep neural networks

This paper presents a method for spike based learning that aims at reducing the needed computation during learning and testing when classifying temporal redundant data.

This paper applies a predictive coding version of the Sigma-Delta encoding scheme to reduce a computational load on a deep learning network, combining the three components in a way not seen previously.

Temporally Efficient Deep Learning with Spikes

spike base learning
redundant datum
reduce
neural network
deep
scheme
temporally
code


Information bottleneck (IB) is a method for extracting information from one random variable X that is relevant for predicting another random variable Y. To do so, IB identifies an intermediate "bottleneck" variable T that has low mutual information I(X;T) and high mutual information I(Y;T).

The "IB curve" characterizes the set of bottleneck variables that achieve maximal I(Y;T) for a given I(X;T), and is typically explored by maximizing the "IB Lagrangian", I(Y;T) - βI(X;T).

In some cases, Y is a deterministic function of X, including many classification problems in supervised learning where the output class Y is a deterministic function of the input X. We demonstrate three caveats when using IB in any situation where Y is a deterministic function of X: (1) the IB curve cannot be recovered by maximizing the IB Lagrangian for different values of β; (2) there are "uninteresting" trivial solutions at all points of the IB curve; and (3) for multi-layer classifiers that achieve low prediction error, different layers cannot exhibit a strict trade-off between compression and prediction, contrary to a recent proposal.

We also show that when Y is a small perturbation away from being a deterministic function of X, these three caveats arise in an approximate way.

To address problem (1), we propose a functional that, unlike the IB Lagrangian, can recover the IB curve in all cases.

We demonstrate the three caveats on the MNIST dataset.

Information bottleneck behaves in surprising ways whenever the output is a deterministic function of the input.

Argues that most real classification problems show such a deterministic relation between the class labels and the inputs X and explores several issues that result from such pathologies.

Explores issues that arise when applying information bottlenext concepts to deterministic supervised learning models

The authors clarify several counter-intuitive behaviors of the information bottleneck method for supervised learning of a deterministic rule.

Caveats for information bottleneck in deterministic scenarios

information bottleneck
deterministic
input
issue
supervised learning
explore
several


We prove, under two sufficient conditions, that idealised models can have no adversarial examples.

We discuss which idealised models satisfy our conditions, and show that idealised Bayesian neural networks (BNNs) satisfy these.

We continue by studying near-idealised BNNs using HMC inference, demonstrating the theoretical ideas in practice.

We experiment with HMC on synthetic data derived from MNIST for which we know the ground-truth image density, showing that near-perfect epistemic uncertainty correlates to density under image manifold, and that adversarial images lie off the manifold in our setting.

This suggests why MC dropout, which can be seen as performing approximate inference, has been observed to be an effective defence against adversarial examples in practice; We highlight failure-cases of non-idealised BNNs relying on dropout, suggesting a new attack for dropout models and a new defence as well.

Lastly, we demonstrate the defence on a cats-vs-dogs image classification task with a VGG13 variant.

We prove that idealised Bayesian neural networks can have no adversarial examples, and give empirical evidence with real-world BNNs.

The paper studies the adversarial robustness of Bayesian classifiers and state two conditions that they show are provably sufficient for "idealised models" on "idealised datasets" to not have adversarial examples

Paper posit a class of discriminative BAyesian classifiers that do not have any adversarial examples.

Sufficient Conditions for Robustness to Adversarial Examples: a Theoretical and Empirical Study with Bayesian Neural Networks

adversarial example
bayesian neural network
classifier
robustness
condition
study
idealise
empirical
sufficient


Deep neural networks are susceptible to adversarial attacks.

In computer vision, well-crafted perturbations to images can cause neural networks to make mistakes such as confusing a cat with a computer.

Previous adversarial attacks have been designed to degrade performance of models or cause machine learning models to produce specific outputs chosen ahead of time by the attacker.

We introduce attacks that instead reprogram the target model to perform a task chosen by the attacker without the attacker needing to specify or compute the desired output for each test-time input.

This attack finds a single adversarial perturbation, that can be added to all test-time inputs to a machine learning model in order to cause the model to perform a task chosen by the adversary—even if the model was not trained to do this task.

These perturbations can thus be considered a program for the new task.

We demonstrate adversarial reprogramming on six ImageNet classification models, repurposing these models to perform a counting task, as well as classification tasks: classification of MNIST and CIFAR-10 examples presented as inputs to the ImageNet model.

We introduce the first instance of adversarial attacks that reprogram the target model to perform a task chosen by the attacker---without the attacker needing to specify or compute the desired output for each test-time input.

The authors present a novel adversarial attack scheme where a neural net is repurposed to accomplish a different task than the one it was originally trained on

This paper proposed "adversarial reprogramming" of well-trained and fixed neural networks and show that adversarial reprogramming is less effective on untrained networks.

The paper extends the idea of 'adversarial attacks' in supervised learning of NNs to a full repurposing of the solution of a trained net.

Adversarial Reprogramming of Neural Networks

adversarial attack
neural network
task
net
train
reprogramming


As shown in recent research, deep neural networks can perfectly fit randomly labeled data, but with very poor accuracy on held out data.

This phenomenon indicates that loss functions such as cross-entropy are not a reliable indicator of generalization.

This leads to the crucial question of how generalization gap should be predicted from the training data and network parameters.

In this paper, we propose such a measure, and conduct extensive empirical studies on how well it can predict the generalization gap.

Our measure is based on the concept of margin distribution, which are the distances of training points to the decision boundary.

We find that it is necessary to use margin distributions at multiple layers of a deep network.

On the CIFAR-10 and the CIFAR-100 datasets, our proposed measure correlates very strongly with the generalization gap.

In addition, we find the following other factors to be of importance: normalizing margin values for scale independence, using characterizations of margin distribution rather than just the margin (closest distance to decision boundary), and working in log space instead of linear space (effectively using a product of margins rather than a sum).

Our measure can be easily applied to feedforward deep networks with any architecture and may point towards new training loss functions that could enable better generalization.

We develop a new scheme to predict the generalization gap in deep networks with high accuracy.

Authors suggest using a geometric margin and layer-wise margin distribution for predicting generalization gap.

Empirically shows an interesting connection between the proposed margin statistics and the generalization gap, which can be used to provide some prescriptive insights towards understanding generalization in deep neural nets.

Predicting the Generalization Gap in Deep Networks with Margin Distributions

predict generalization gap deep network
margin distribution


We propose a new algorithm to learn a one-hidden-layer convolutional neural network where both the convolutional weights and the outputs weights are parameters to be learned.

Our algorithm works for a general class of (potentially overlapping) patches, including commonly used structures for computer vision tasks.

Our algorithm draws ideas from (1) isotonic regression for learning neural networks and (2) landscape analysis of non-convex matrix factorization problems.

We believe these findings may inspire further development in designing provable algorithms for learning neural networks and other complex models.

While our focus is theoretical, we also present experiments that illustrate our theoretical findings.

We propose an algorithm for provably recovering parameters (convolutional and output weights) of a convolutional network with overlapping patches.

This paper studies the theoretical learning of one-hidden-layer convolutional neural nets, resulting in a learning algorithm and provable guarantees using the algorithm.

This paper gives a new algorithm for learning a two layer neural network which involves a single convolutional filter and a weight vector for different locations.

Improved Learning of One-hidden-layer Convolutional Neural Networks with Overlaps

learning one-hidden-layer convolutional neural


Since their invention, generative adversarial networks (GANs) have become a popular approach for learning to model a distribution of real (unlabeled) data.

Convergence problems during training are overcome by Wasserstein GANs which minimize the distance between the model and the empirical distribution in terms of a different metric, but thereby introduce a Lipschitz constraint into the optimization problem.

A simple way to enforce the Lipschitz constraint on the class of functions, which can be modeled by the neural network, is weight clipping.

Augmenting the loss by a regularization term that penalizes the deviation of the gradient norm of the critic (as a function of the network's input) from one, was proposed as an alternative that improves training.

We present theoretical arguments why using a weaker regularization term enforcing the Lipschitz constraint is preferable.

These arguments are supported by experimental results on several data sets.

A new regularization term can improve your training of wasserstein gans

The paper proposes a regularization scheme for Wasserstein GAN based on relaxation of the constraints on the Lipschitz constant of 1.

The article deals with regularization/penalization in the fitting of GANs, when based on a L_1 Wasserstein metric.

On the regularization of Wasserstein GANs


We introduce a new method for training GANs by applying the Wasserstein-2 metric proximal on the generators.

The approach is based on the gradient operator induced by optimal transport, which connects the geometry of sample space and parameter space in implicit deep generative models.

From this theory, we obtain an easy-to-implement regularizer for the parameter updates.

Our experiments demonstrate that this method improves the speed and stability in training GANs in terms of wall-clock time and Fr\'echet Inception Distance (FID) learning curves.

We propose the Wasserstein proximal method for training GANs.

Proposes a new GAN procedure that takes into account points generated in the previous iteration and updates the generator to be carried out l times.

Considers natural gradient learning in GAN learning, where the Riemannian structure induced by the Wasserstein-2 distance is employed.

The paper intends to utilize natural gradient induced by Wasserstein-2 distance to train the generator in GAN and the authors propose the Wasserstein proximal operator as a regularization.

Wasserstein proximal of GANs

induce wasserstein-2 distance
wasserstein proximal
generator
train
gan
natural gradient


Influence diagrams provide a modeling and inference framework for sequential decision problems, representing the probabilistic knowledge by a Bayesian network and the preferences of an agent by utility functions over the random variables and decision variables.

MDPs and POMDPS, widely used for planning under uncertainty can also be represented by influence diagrams.

The time and space complexity of computing the maximum expected utility (MEU) and its maximizing policy is exponential in the induced width of the underlying graphical model, which is often prohibitively large due to the growth of the information set under the sequence of decisions.

In this paper, we develop a weighted mini-bucket approach for bounding the MEU.

These bounds can be used as a stand-alone approximation that can be improved as a function of a controlling i-bound parameter

.
They can also be used as heuristic  functions to guide search, especially for planning

We evaluate the scheme empirically against state-of-the-art, thus illustrating its potential.

This paper introduces an elimination based heuristic function for sequential decision making, suitable for guiding AND/OR search algorithms for solving influence diagrams.

generalizes minibuckets inference heuristic to influence diagrams.

A Weighted Mini-Bucket Bound Heuristic for Solving Influence Diagrams


Probabilistic Neural Networks deal with various sources of stochasticity: input noise, dropout, stochastic neurons, parameter uncertainties modeled as random variables, etc.

In this paper we revisit a feed-forward propagation approach that allows one to estimate for each neuron its mean and variance w.r.t. all mentioned sources of stochasticity.

In contrast, standard NNs propagate only point estimates, discarding the uncertainty.

Methods propagating also the variance have been proposed by several authors in different context.

The view presented here attempts to clarify the assumptions and derivation behind such methods, relate them to classical NNs and broaden their scope of applicability.

The main technical contributions are new approximations for the distributions of argmax and max-related transforms, which allow for fully analytic uncertainty propagation in networks with softmax and max-pooling layers as well as leaky ReLU activations.

We evaluate the accuracy of the approximation and suggest a simple calibration.

Applying the method to networks with dropout allows for faster training and gives improved test likelihoods without the need of sampling.

Approximating mean and variance of the NN output over noisy input / dropout / uncertain parameters. Analytic approximations for argmax, softmax and max layers.

The authors focus on the problem of uncertainty propagation DNN

This paper revisits the feed-forward propagation of mean and variance in neurons, by addressing the problem of propagating uncertainty through max-pooling layers and softmax.

Feed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers

mean variance
softmax
max layer
problem
feed-forward propagation
uncertainty


Generative Adversarial Networks are one of the leading tools in generative modeling, image editing and content creation.

However, they are hard to train as they require a delicate balancing act between two deep networks fighting a never ending duel.

Some of the most promising adversarial models today minimize a Wasserstein objective.

It is smoother and more stable to optimize.

In this paper, we show that the Wasserstein distance is just one out of a large family of objective functions that yield these properties.

By making the discriminator of a GAN robust to adversarial attacks we can turn any GAN objective into a smooth and stable loss.

We experimentally show that any GAN objective, including Wasserstein GANs, benefit from adversarial robustness both quantitatively and qualitatively.

The training additionally becomes more robust to suboptimal choices of hyperparameters, model architectures, or objective functions.

A discriminator that is not easily fooled by adversarial example makes GAN training more robust and leads to a smoother objective.

This paper proposes a new way to stabilize the training process of GAN by regularizing the Discriminator to be robust to adversarial examples.

The paper proposes a systematic way of training GANs with robustness regularization terms, allowing for smoother training of GANs.

Presents idea that making a discriminator robust to adversarial perturbations the GAN objective can be made smooth which results in better results both visually and in terms of FID.

Don't let your Discriminator  be fooled

discriminator
robust
adversarial example
gan
fool
objective
training
smoother
way
term
make


We propose a method to learn stochastic activation functions for use in probabilistic neural networks.

First, we develop a framework to embed stochastic activation functions based on Gaussian processes in probabilistic neural networks.

Second, we analytically derive expressions for the propagation of means and covariances in such a network, thus allowing for an efficient implementation and training without the need for sampling.

Third, we show how to apply variational Bayesian inference to regularize and efficiently train this model.

The resulting model can deal with uncertain inputs and implicitly provides an estimate of the confidence of its predictions.

Like a conventional neural network it can scale to datasets of arbitrary size and be extended with convolutional and recurrent connections, if desired.

We model the activation function of each neuron as a Gaussian Process and learn it alongside the weight with Variational Inference.

Propose placing Gaussian process priors on the functional form of each activation function in the neural net to learn the form of activation functions.

Gaussian Process Neurons

activation function
learn
gaussian process
neuron


Recent results from linear algebra stating that any matrix can be decomposed into products of diagonal and circulant matrices has lead to the design of compact deep neural network architectures that perform well in practice.

In this paper, we bridge the gap between these good empirical results

and the theoretical approximation capabilities of Deep diagonal-circulant ReLU networks.

More precisely, we first demonstrate  that a Deep diagonal-circulant ReLU networks of

bounded width and small depth can approximate a deep ReLU network in which the dense matrices are

Based on this result, we provide new bounds on the expressive power and universal approximativeness of this type of networks.

We support our experimental results with thorough experiments on a large, real world video classification problem.

We provide a theoretical study of the properties of Deep circulant-diagonal ReLU Networks and demonstrate that they are bounded width universal approximators.

The paper proposes using circulant and diagonal matrices to speed up computation and reduce memory requirements in eural networks.

This paper proves that bounded width diagonal-circulent ReLU networks (DC-ReLU) are universal approximators.

The Expressive Power of Deep Neural Networks with Circulant Matrices

relu network
bound width
universal approximator
deep
circulant
matrix


Camera drones, a rapidly emerging technology, offer people the ability to remotely inspect an environment with a high degree of mobility and agility.

However, manual remote piloting of a drone is prone to errors.

In contrast, autopilot systems can require a significant degree of environmental knowledge and are not necessarily designed to support flexible visual inspections.

Inspired by camera manipulation techniques in interactive graphics, we designed StarHopper, a novel touch screen interface for efficient object-centric camera drone navigation, in which a user directly specifies the navigation of a drone camera relative to a specified object of interest.

The system relies on minimal environmental information and combines both manual and automated control mechanisms to give users the freedom to remotely explore an environment with efficiency and accuracy.

A lab study shows that StarHopper offers an efficiency gain of 35.4% over manual piloting, complimented by an overall user preference towards our object-centric navigation system.

StarHopper is a novel touch screen interface for efficient and flexible object-centric camera drone navigation

The authors outline a new drone control interface StarHopper that they have developed, combining automated and manual piloting into a new hybrid navigation interface and gets rid of the assumption that the target object is already in the drone’s FOV by using an additional overhead camera.

This paper presents StarHopper, a system for semi-automatic drone navigation in the context of remote inspection.

Introduces StarHopper, an application that uses computer vision techniques with touch input to support drone piloting with an object-centric approach.

StarHopper: A Touch Interface for Remote Object-Centric Drone Navigation

starhopper
pilot
drone navigation
interface
remote
touch
camera
object-centric


Recurrent neural networks (RNN), convolutional neural networks (CNN) and self-attention networks (SAN) are commonly used to produce context-aware representations.

RNN can capture long-range dependency but is hard to parallelize and not time-efficient.

CNN focuses on local dependency but does not perform well on some tasks.

SAN can model both such dependencies via highly parallelizable computation, but memory requirement grows rapidly in line with sequence length.

In this paper, we propose a model, called "bi-directional block self-attention network (Bi-BloSAN)", for RNN/CNN-free sequence encoding.

It requires as little memory as RNN but with all the merits of SAN.

Bi-BloSAN splits the entire sequence into blocks, and applies an intra-block SAN to each block for modeling local context, then applies an inter-block SAN to the outputs for all blocks to capture long-range dependency.

Thus, each SAN only needs to process a short sequence, and only a small amount of memory is required.

Additionally, we use feature-level attention to handle the variation of contexts around the same word, and use forward/backward masks to encode temporal order information.

On nine benchmark datasets for different NLP tasks, Bi-BloSAN achieves or improves upon state-of-the-art accuracy, and shows better efficiency-memory trade-off than existing RNN/CNN/SAN.

A self-attention network for RNN/CNN-free sequence encoding with small memory consumption, highly parallelizable computation and state-of-the-art performance on several NLP tasks

Proposes applyting self-attention at two levels to limit the memory requirement in attention-based models with a negligible impact on speed.

This paper introduces bi-directional block self-attention model as a general-purpose encoder for various sequence modeling tasks in NLP

Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling

model
bi-directional block self-attention
sequence
memory
task
nlp


End-to-end neural models have made significant progress in question answering, however recent studies show that these models implicitly assume that the answer and evidence appear close together in a single document.

In this work, we propose the Coarse-grain Fine-grain Coattention Network (CFC), a new question answering model that combines information from evidence across multiple documents.

The CFC consists of a coarse-grain module that interprets documents with respect to the query then finds a relevant answer, and a fine-grain module which scores each candidate answer by comparing its occurrences across all of the documents with the query.

We design these modules using hierarchies of coattention and self-attention, which learn to emphasize different parts of the input.

On the Qangaroo WikiHop multi-evidence question answering task, the CFC obtains a new state-of-the-art result of 70.6% on the blind test set, outperforming the previous best by 3% accuracy despite not using pretrained contextual encoders.

A new state-of-the-art model for multi-evidence question answering using coarse-grain fine-grain hierarchical attention.

Proposes a method for multi-hop QA based on two separate modules (coarse-grained and fine-grained modules).

This paper proposes an interesting coarse-grain fine-grain coattention network architecture to address multi-evidence question answering

Focuses on multi-choice QA and proposes a coarse-to-fine scoring framework.

Coarse-grain Fine-grain Coattention Network for Multi-evidence Question Answering


Generative adversarial networks (GANs) are an expressive class of neural generative models with tremendous success in modeling high-dimensional continuous measures.

In this paper, we present a scalable method for unbalanced optimal transport (OT) based on the generative-adversarial framework.

We formulate unbalanced OT as a problem of simultaneously learning a transport map and a scaling factor that push a source measure to a target measure in a cost-optimal manner.

We provide theoretical justification for this formulation, showing that it is closely related to an existing static formulation by Liero et al. (2018).

We then propose an algorithm for solving this problem based on stochastic alternating gradient updates, similar in practice to GANs, and perform numerical experiments demonstrating how this methodology can be applied to population modeling.

We propose new methodology for unbalanced optimal transport using generative adversarial networks.

The authors consider the unbalanced optimal transport problem between two measures with different total mass using a stochastic min-max algorithm and local scaling

The authors propose an approach to estimate unbalanced optimal transport between sampled measures that scales well in the dimension and in the number of samples.

The paper introduces a static formulation for unbalanced optimal transport by learning simultaneously a transport map T and scaling factor xi.

Scalable Unbalanced Optimal Transport using Generative Adversarial Networks

unbalanced optimal transport generative adversarial network


Extracting saliency maps, which indicate parts of the image important to classification, requires many tricks to achieve satisfactory performance when using classifier-dependent methods.

Instead, we propose classifier-agnostic saliency map extraction, which finds all parts of the image that any classifier could use, not just one given in advance.

We observe that the proposed approach extracts higher quality saliency maps and outperforms existing weakly-supervised localization techniques, setting the new state of the art result on the ImageNet dataset.

We propose a new saliency map extraction method which results in extracting higher quality maps.

Proposes a classifier-agnostic method for saliency map extraction.

This paper introduces a new saliency map extractor that seems to improve state-of-the-art results.

The authors argue that when an extracted saliency map is directly dependent on a model, then it might not be useful for a different classifier, and suggests a scheme to approximate the solution.

Classifier-agnostic saliency map extraction

new saliency map
method
result
extract
extraction
classifier-agnostic


Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs).

However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images.

To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration.

The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances.

To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances.

We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances.

Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases.

Code and results are available in https://github.com/sangwoomo/instagan

We propose a novel method to incorporate the set of instance attributes for image-to-image translation.

This paper proposes a method -- InstaGAN -- which builds on CycleGAN by taking into account instance information in the form of per-instance segmentation masks, with results that compare favorably to CycleGAN and other baselines.

Proposes to add instance-aware segmentation masks for the problem of unpaired image-to-image translation.

InstaGAN: Instance-aware Image-to-Image Translation

segmentation mask
image-to-image translation
method
instance
instagan
instance-aware


Deep neural networks (DNNs) generalize remarkably well without explicit regularization even in the strongly over-parametrized regime  where classical learning theory would instead predict that they would severely overfit.

While many proposals for some kind of implicit regularization have been made to rationalise this success, there is no consensus for the fundamental reason why DNNs do not strongly overfit.

In this paper, we provide a new explanation.

By applying a very general probability-complexity bound recently derived from  algorithmic information theory (AIT), we argue that the parameter-function map of many DNNs should be exponentially biased towards simple functions.

We then provide clear evidence for this strong simplicity bias in a model DNN for Boolean functions, as well as in much larger fully connected and convolutional networks trained on CIFAR10 and MNIST.

As the target functions in many real problems are expected to be highly structured, this intrinsic simplicity bias helps explain why deep networks generalize well on real world problems.

This picture also facilitates a novel PAC-Bayes approach where the prior is taken over the DNN input-output function space, rather than  the more conventional prior over parameter space.

If we assume that the training algorithm samples parameters close to uniformly within the zero-error region then the PAC-Bayes theorem can be used to guarantee good expected generalization for target functions producing high-likelihood training sets.

By exploiting recently discovered connections between DNNs and Gaussian processes to estimate the marginal likelihood,  we produce relatively tight generalization PAC-Bayes error bounds which correlate well with the true error on realistic datasets such as MNIST and CIFAR10 and for architectures including convolutional and fully connected networks.

The parameter-function map of deep networks is hugely biased; this can explain why they generalize. We use PAC-Bayes and Gaussian processes to obtain nonvacuous bounds.

The paper studies the generalization capabilities of deep neural networks, with the help of the PAC-Bayesian learning theory and empirically backed intuitions.

This paper proposes an explaination of the generalization behaviors of large over-parameterized neural networks by claiming the parameter-function map in neural networks are biased towards "simple" functions and generalization behavior will be good if the target concept is also "simple".

Deep learning generalizes because the parameter-function map is biased towards simple functions

parameter-function map
neural network
generalize
bias towards
deep
generalization
learning
simple
function


We establish a theoretical link between evolutionary algorithms and variational parameter optimization of probabilistic generative models with binary hidden variables.

While the novel approach is independent of the actual generative model, here we use two such models to investigate its applicability and scalability: a noisy-OR Bayes Net (as a standard example of binary data) and Binary Sparse Coding (as a model for continuous data).

Learning of probabilistic generative models is first formulated as approximate maximum likelihood optimization using variational expectation maximization (EM).

We choose truncated posteriors as variational distributions in which discrete latent states serve as variational parameters.

optimized according to a tractable free-energy objective

. Given a data point, we can show that evolutionary algorithms can be used for the variational optimization loop by (A)~considering the bit-vectors of the latent states as genomes of individuals, and by (B)~defining the fitness of the

individuals as the (log) joint probabilities given by the used generative model.

As a proof of concept, we apply the novel evolutionary EM approach to the optimization of the parameters of noisy-OR Bayes nets and binary sparse coding on artificial and real data (natural image patches).

Using point mutations and single-point cross-over for the evolutionary algorithm, we find that scalable variational EM algorithms are obtained which efficiently improve the data likelihood.

In general we believe that, with the link established here, standard as well as recent results in the field of evolutionary optimization can be leveraged to address the difficult problem of parameter optimization in generative models.

We present Evolutionary EM as a novel algorithm for unsupervised training of generative models with binary latent variables that intimately connects variational EM with evolutionary optimization

The paper presents a combination of evolutionary computation and variational EM for models with binary latent variables represented via a particle-based approximation

The paper makes an attempt to tightly integrate expectation-maximization training algorithms with evolutionary algorithms.

Evolutionary Expectation Maximization for Generative Models with Binary Latents

generative model binary latent
variational
evolutionary
present
algorithm
training
variable


While deep neural networks have achieved groundbreaking prediction results in many tasks, there is a class of data where existing architectures are not optimal -- sequences of probability distributions.

Performing forward prediction on sequences of distributions has many important applications.

However, there are two main challenges in designing a network model for this task.

First, neural networks are unable to encode distributions compactly as each node encodes just a real value.

A recent work of Distribution Regression Network (DRN) solved this problem with a novel network that encodes an entire distribution in a single node, resulting in improved accuracies while using much fewer parameters than neural networks.

However, despite its compact distribution representation, DRN does not address the second challenge, which is the need to model time dependencies in a sequence of distributions.

In this paper, we propose our Recurrent Distribution Regression Network (RDRN) which adopts a recurrent architecture for DRN.

The combination of compact distribution representation and shared weights architecture across time steps makes RDRN suitable for modeling the time dependencies in a distribution sequence.

Compared to neural networks and DRN, RDRN achieves the best prediction performance while keeping the network compact.

We propose an efficient recurrent network model for forward prediction on time-varying distributions.

This paper proposes a method for creating neural nets that maps historical distributions onto distributions and applies the method to several distribution prediction tasks.

Proposes a Reccurent Distribution Regression Network which uses a recurrent architecture upon a previous model Distribution Regression Network.

This paper is on regressing over probability distributions by studying time varying distributions in a recurrent neural network setting

An Efficient Network for Predicting Time-Varying Distributions

recurrent
efficient
time-varying distribution
network
model
prediction
neural


We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations.

By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of computationally intensive matrix operation (such as inversion) or depending on knowing the graph structure upfront.

In this way, we address several key challenges of spectral-based graph neural networks simultaneously, and make our model readily applicable to inductive as well as transductive problems.

Our GAT models have achieved or matched state-of-the-art results across four established transductive and inductive graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, as well as a protein-protein interaction dataset (wherein test graphs remain unseen during training).

A novel approach to processing graph-structured data by neural networks, leveraging attention over a node's neighborhood. Achieves state-of-the-art results on transductive citation network tasks and an inductive protein-protein interaction task.

This paper proposes a new method for classifying nodes of a graph, which can be used in semi-supervised scenarios and on a completely new graph.

The paper introduces a neural network architecture to operate on graph-structured data named Graph Attention Networks.

Provides a fair and almost comprehensive discussion of the state of art approaches to learning vector representations for the nodes of a graph.

Graph Attention Networks

node graph
neural network
graph-structured datum
approach
attention


While bigger and deeper neural network architectures continue to advance the state-of-the-art for many computer vision tasks, real-world adoption of these networks is impeded by hardware and speed constraints.

Conventional model compression methods attempt to address this problem by modifying the architecture manually or using pre-defined heuristics.

Since the space of all reduced architectures is very large, modifying the architecture of a deep neural network in this way is a difficult task.

In this paper, we tackle this issue by introducing a principled method for learning reduced network architectures in a data-driven way using reinforcement learning.

Our approach takes a larger 'teacher' network as input and outputs a compressed 'student' network derived from the 'teacher' network.

In the first stage of our method, a recurrent policy network aggressively removes layers from the large 'teacher' model.

In the second stage, another  recurrent policy network carefully reduces the size of each remaining layer.

The resulting network is then evaluated to obtain a reward -- a score based on the accuracy and compression of the network.

Our approach uses this reward signal with policy gradients to train the policies to find a locally optimal student network.

Our experiments show that we can achieve compression rates of more than 10x for models such as ResNet-34 while maintaining similar performance to the input 'teacher' network.

We also present a valuable transfer learning result which shows that policies which are pre-trained on smaller 'teacher' networks can be used to rapidly speed up training on larger 'teacher' networks.

A novel reinforcement learning based approach to compress deep neural networks with knowledge distillation

This paper proposes to use reinforcement learning instead of pre-defined heuristics to determine the structure of the compressed model in the knowledge distillation process

Introduces a principled way of network to network compression, which uses policy gradients for optimizing two policies which compress a strong teacher into a strong but smaller student model.

N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning

network network compression
knowledge distillation
compress
reinforcement learning
model
policy gradient


Recent advances in conditional image generation tasks, such as image-to-image translation and image inpainting, are largely accounted to the success of conditional GAN models, which are often optimized by the joint use of the GAN loss with the reconstruction loss.

However, we reveal that this training recipe shared by almost all existing methods causes one critical side effect: lack of diversity in output samples.

In order to accomplish both training stability and multimodal output generation, we propose novel training schemes with a new set of losses named moment reconstruction losses that simply replace the reconstruction loss.

We show that our approach is applicable to any conditional generation tasks by performing thorough experiments on image-to-image translation, super-resolution and image inpainting using Cityscapes and CelebA dataset.

Quantitative evaluations also confirm that our methods achieve a great diversity in outputs while retaining or even improving the visual fidelity of generated samples.

We prove that the mode collapse in conditional GANs is largely attributed to a mismatch between reconstruction loss and GAN loss and introduce a set of novel loss functions as alternatives for reconstruction loss.

The paper proposes a modification to the traditional conditional GAN objective in order to promote diverse, multimodal generation of images.

This paper proposes an alternative to L1/L2 errors that are used to augment adversarial losses when training conditional GANs.

Harmonizing Maximum Likelihood with GANs for Multimodal Conditional Generation

alternative gan loss multimodal conditional generation


Generative models are important tools to capture and investigate the properties of complex empirical data.

Recent developments such as Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) use two very similar, but \textit{reverse}, deep convolutional architectures, one to generate and one to extract information from data.

Does learning the parameters of both architectures obey the same rules?

We exploit the causality principle of independence of mechanisms to quantify how the weights of successive layers adapt to each other.

Using the recently introduced Spectral Independence Criterion, we quantify the dependencies between the kernels of successive convolutional layers and show that those are more independent for the generative process than for information extraction, in line with results from the field of causal inference.

In addition, our experiments on generation of human faces suggest that more independence between successive layers of generators results in improved performance of these architectures.

We use causal inference to characterise the architecture of generative models

This paper examines the nature of convolutional filters in the encoder and a decoder of a VAE, and a generator and a discriminator of a GAN.

This work exploits the causality principle to quantify how the weights of successive layers adapt to each other.

On the difference between building and extracting patterns: a causal analysis of deep generative models.


Many deep reinforcement learning approaches use graphical state representations,

this means visually distinct games that share the same underlying structure cannot

This paper outlines a new approach for learning

underlying game state embeddings irrespective of the visual rendering of the game

We utilise approaches from multi-task learning and domain adaption in

order to place visually distinct game states on a shared embedding manifold.

present our results in the context of deep reinforcement learning agents.

An approach to learning a shared embedding space between visually distinct games.

A new approach for learning underlying structure of visually distinct games combining convolutional layers for processing input images, Asynchronous Advantage Actor Critic for deep reinforcement learning and adversarial approach to force the embedding representation to be independent of the visual representation of games

Introduces a method to learn a policy on visually distinct games by adapting deep reinforcement learning.

This paper discusses an agent architecture which uses a shared representation to train multiple tasks with different sprite level visual statistics

Domain Adaptation for Deep Reinforcement Learning in Visually Distinct Games

visually distinct game
deep reinforcement learning
approach learn
share
representation
embed
visual


We study discrete time dynamical systems governed by the state equation $h_{t+1}=ϕ(Ah_t+Bu_t)$.

Here A,B are weight matrices, ϕ is an activation function, and $u_t$ is the input data.

This relation is the backbone of recurrent neural networks (e.g. LSTMs) which have broad applications in sequential learning tasks.

We utilize stochastic gradient descent to learn the weight matrices from a finite input/state trajectory $(u_t,h_t)_{t=0}^N$.

We prove that SGD estimate linearly converges to the ground truth weights while using near-optimal sample size.

Our results apply to increasing activations whose derivatives are bounded away from zero.

i) an SGD convergence result with nonlinear activations and

ii) careful statistical characterization of the state vector.

Numerical experiments verify the fast convergence of SGD on ReLU and leaky ReLU in consistence with our theory.

We study the state equation of a recurrent neural network. We show that SGD can efficiently learn the unknown dynamics from few input/output observations under proper assumptions.

The paper studies discrete-time dynamical systems with a non-linear state equation, proving that running SGD on a fixed-length trajectory gives logarithmic convergence.

This work considers the problem of learning a non-linear dynamical system in which the output equals the state.

This paper studies the ability of SGD to learn dynamics of a linear system and non-linear activation.

Stochastic Gradient Descent Learns State Equations with Nonlinear Activations

dynamical system
sgd
dynamic
state equation
study
learn
non-linear
output
activation


Although deep neural networks show their extraordinary power in various tasks, they are not feasible for deploying such large models on embedded systems due to high computational cost and storage space limitation.

The recent work knowledge distillation (KD) aims at transferring model knowledge from a well-trained teacher model to a small and fast student model which can significantly help extending the usage of large deep neural networks on portable platform.

In this paper, we show that, by properly defining the neuron manifold of deep neuron network (DNN), we can significantly improve the performance of student DNN networks through approximating neuron manifold of powerful teacher network.

To make this, we propose several novel methods for learning neuron manifold from DNN model.

Empowered with neuron manifold knowledge, our experiments show the great improvement across a variety of DNN architectures and training data.

Compared with other KD methods, our Neuron Manifold Transfer (NMT) has best transfer ability of the learned features.

A new knowledge distill method for transfer learning

The work introduces a knowledge distillation method using the proposed neuron manifold concept.

Proposes a knowledge distilling method in which neural manifold is taken as the transferred knowledge.

KNOWLEDGE DISTILL VIA LEARNING NEURON MANIFOLD

method
knowledge distill
transfer
neuron manifold