Supervised machine learning models for high-value computer vision applications such as medical image classification often require large datasets labeled by domain experts, which are slow to collect, expensive to maintain, and static with respect to changes in the data distribution.

In this context, we assess the utility of observational supervision, where we take advantage of passively-collected signals such as eye tracking or “gaze” data, to reduce the amount of hand-labeled data needed for model training.

Specifically, we leverage gaze information to directly supervise a visual attention layer by penalizing disagreement between the spatial regions the human labeler looked at the longest and those that most heavily influence model output.

We present evidence that constraining the model in this way can reduce the number of labeled examples required to achieve a given performance level by as much as 50%, and that gaze information is most helpful on more difficult tasks.

We explore using passively collected eye-tracking data to reduce the amount of labeled data needed during training.

A method to use gaze information to reduce the sample complexity of a model and the needed labeling effort to get a target performance, with improved results in middle-sized samples and harder tasks.

A method to incorporate gaze signals into standard CNNs for image classification, adding a loss function term based in the difference between the model's Class Activation Map and the map constructed from eye tracking information.

Improving Sample Complexity with Observational Supervision

reduce
method
sample complexity
model
need
information
gaze
improve


We study the robustness to symmetric label noise of GNNs training procedures.

By combining the nonlinear neural message-passing models (e.g. Graph Isomorphism Networks, GraphSAGE, etc.) with loss correction methods, we present a noise-tolerant approach for the graph classification task.

Our experiments show that test accuracy can be improved under the artificial symmetric noisy setting.

We apply loss correction to graph neural networks to train a more robust to noise model.

This paper introduces loss correction for Graph Neural Networks to deal with symmetric graph label noise, focused on a graph classification task.

This paper proposes the use of a noise correction loss in the context of graph neural networks to deal with noisy labels.

Learning Graph Neural Networks with Noisy Labels

loss correction graph neural network
deal
noisy label
noise


Through many recent advances in graph representation learning, performance achieved on tasks involving graph-structured data has substantially increased in recent years---mostly on tasks involving node-level predictions.

The setup of prediction tasks over entire graphs (such as property prediction for a molecule, or side-effect prediction for a drug), however, proves to be more challenging, as the algorithm must combine evidence about several structurally relevant patches of the graph into a single prediction.

Most prior work attempts to predict these graph-level properties while considering only one graph at a time---not allowing the learner to directly leverage structural similarities and motifs across graphs.

Here we propose a setup in which a graph neural network receives pairs of graphs at once, and extend it with a co-attentional layer that allows node representations to easily exchange structural information across them.

We first show that such a setup provides natural benefits on a pairwise graph classification task (drug-drug interaction prediction), and then expand to a more generic graph regression setup: enhancing predictions over QM9, a standard molecular prediction benchmark.

Our setup is flexible, powerful and makes no assumptions about the underlying dataset properties, beyond anticipating the existence of multiple training graphs.

We use graph co-attention in a paired graph training system for graph classification and regression.

This paper injects a multi-head co-attention mechanism in GCN that allows one drug to attend to another drug during drug side effect prediction.

A method to extend graph-based learning with a co-attentional layer, which outperforms other previous ones on a pairwise graph classification task.

Empowering Graph Representation Learning with Paired Training and Graph Co-Attention

graph co-attention
pair
classification
training
one


In this paper we study image captioning as a conditional GAN training, proposing both a context-aware LSTM captioner and co-attentive discriminator, which enforces semantic alignment between images and captions.

We investigate the viability of two discrete GAN training methods: Self-critical Sequence Training (SCST) and Gumbel Straight-Through (ST) and demonstrate that SCST shows more stable gradient behavior and improved results over Gumbel ST.

Image captioning as a conditional GAN training with novel architectures, also study two discrete GAN training methods.

An improved GAN model for image captioning that proposes a context-aware LSTM captioner, introduces a stronger co-attentive discriminator with better performance, and uses SCST for GAN training.

Improved Adversarial Image Captioning


We present Newtonian Monte Carlo (NMC), a method to improve Markov Chain Monte Carlo (MCMC) convergence by analyzing the first and second order gradients of the target density to determine a suitable proposal density at each point.

Existing first order gradient-based methods suffer from the problem of determining an appropriate step size.

Too small a step size and it will take a large number of steps to converge, while a very large step size will cause it to overshoot the high density region.

NMC is similar to the Newton-Raphson update in optimization where the second order gradient is used to automatically scale the step size in each dimension.

However, our objective is not to find a maxima but instead to find a parameterized density that can best match the local curvature of the target density.

This parameterized density is then used as a single-site Metropolis-Hastings proposal.

As a further improvement on first order methods, we show that random variables with constrained supports don't need to be transformed before taking a gradient step.

NMC directly matches constrained random variables to a proposal density with the same support thus keeping the curvature of the target density intact.

We demonstrate the efficiency of NMC on a number of different domains.

For statistical models where the prior is conjugate to the likelihood, our method recovers the posterior quite trivially in one step.

However, we also show results on fairly large non-conjugate models, where NMC performs better than adaptive first order methods such as NUTS or other inexact scalable inference methods such as Stochastic Variational Inference or bootstrapping.

Exploit curvature to make MCMC methods converge faster than state of the art.

Newtonian Monte Carlo: a second-order gradient method for speeding up MCMC


Neural Tangents is a library designed to enable research into infinite-width neural networks.

It provides a high-level API for specifying complex and hierarchical neural network architectures.

These networks can then be trained and evaluated either at finite-width as usual or in their infinite-width limit.

Infinite-width networks can be trained analytically using exact Bayesian inference or using gradient descent via the Neural Tangent Kernel.

Additionally, Neural Tangents provides tools to study gradient descent training dynamics of wide but finite networks in either function space or weight space.

The entire library runs out-of-the-box on CPU, GPU, or TPU.

All computations can be automatically distributed over multiple accelerators with near-linear scaling in the number of devices.

We also provide an accompanying interactive Colab notebook at

Neural Tangents: Fast and Easy Infinite Neural Networks in Python


Deep neural networks have achieved great success in classiﬁcation tasks during the last years.

However, one major problem to the path towards artiﬁcial intelligence is the inability of neural networks to accurately detect samples from novel class distributions and therefore, most of the existent classiﬁcation algorithms assume that all classes are known prior to the training stage.

In this work, we propose a methodology for training a neural network that allows it to efﬁciently detect out-of-distribution (OOD) examples without compromising much of its classiﬁcation accuracy on the test examples from known classes.

Based on the Outlier Exposure (OE) technique, we propose a novel loss function that achieves state-of-the-art results in out-of-distribution detection with OE both on image and text classiﬁcation tasks.

Additionally, the way this method was constructed makes it suitable for training any classiﬁcation algorithm that is based on Maximum Likelihood methods.

We propose a novel loss function that achieves state-of-the-art results in out-of-distribution detection with Outlier Exposure both on image and text classiﬁcation tasks.

This paper tackles the problems of out-of-distribution detection and model calibration by adapting the loss function of the Outlier Exposure technique, with results demonstrating increased performance over OE on vision and text benchmarks and improved model calibration.

Proposal for a new loss function to train the network with Outlier Exposure which leads to better OOD detection compared to simple loss functions using KL divergence.

Simultaneous Classification and Out-of-Distribution Detection Using Deep Neural Networks

out-of-distribution detection
outli exposure
loss function
result
text
network


Navigation is crucial for animal behavior and is assumed to require an internal representation of the external environment, termed a cognitive map.

The precise form of this representation is often considered to be a metric representation of space.

An internal representation, however, is judged by its contribution to performance on a given task, and may thus vary between different types of navigation tasks.

Here we train a recurrent neural network that controls an agent performing several navigation tasks in a simple environment.

To focus on internal representations, we split learning into a task-agnostic pre-training stage that modifies internal connectivity and a task-specific Q learning stage that controls the network's output.

We show that pre-training shapes the attractor landscape of the networks, leading to either a continuous attractor, discrete attractors or a disordered state.

These structures induce bias onto the Q-Learning phase, leading to a performance pattern across the tasks corresponding to metric and topological regularities.

Our results show that, in recurrent networks, inductive bias takes the form of attractor landscapes -- which can be shaped by pre-training and analyzed using dynamical systems methods.

Furthermore, we demonstrate that non-metric representations are useful for navigation tasks.

Task agnostic pre-training can shape RNN's attractor landscape, and form diverse inductive bias for different navigation tasks

This paper studies the internal representations of recurrent neural networks trained on navigation tasks, and finds that RNNs pre-trained to use path integration contain 2D continuous attractors while RNNs pre-trained for landmark memory contain discrete attractors.

This paper explores how pre-training recurrent networks on different navigational objectives confers different benefits for solving downstream tasks, and shows how different pretraining manifests as different dynamical structures in the networks after pre-training.

Implementing Inductive bias for different navigation tasks through diverse RNN attrractors

inductive bias different navigation task


Formal verification of machine learning models has attracted attention recently, and significant progress has been made on proving simple properties like robustness to small perturbations of the input features.

In this context, it has also been observed that folding the verification procedure into training makes it easier to train verifiably robust models.

In this paper, we extend the applicability of verified training by extending it to (1) recurrent neural network architectures and (2) complex specifications that go beyond simple adversarial robustness, particularly specifications that capture temporal properties like requiring that a robot periodically visits a charging station or that a language model always produces sentences of bounded length.

Experiments show that while models trained using standard training often violate desired specifications, our verified training method produces models that both perform well (in terms of test error or reward) and can be shown to be provably consistent with specifications.

Neural Network Verification for Temporal Properties and Sequence Generation Models

This paper extends interval bound propagation to recurrent computation and auto-regressive models, introduces and extends Signal Temporal Logic for specifying temporal contraints, and provides proof that STL with bound propagation can ensure neural models conform to temporal specification.

A way to train time-series regressors verifiably with respect to a set of rules defined by signal temporal logic, and work in deriving bound propagation rules for the STL language.

Scalable Neural Learning for Verifiable Consistency with Temporal Specifications

signal temporal logic
stl
bound propagation
neural
model
specification


Neural Network (NN) has achieved state-of-the-art performances in many tasks within image, speech, and text domains.

Such great success is mainly due to special structure design to fit the particular data patterns, such as CNN capturing spatial locality and RNN modeling sequential dependency.

Essentially, these specific NNs achieve good performance by leveraging the prior knowledge over corresponding domain data.

Nevertheless, there are many applications with all kinds of tabular data in other domains.

Since there are no shared patterns among these diverse tabular data, it is hard to design specific structures to fit them all.

Without careful architecture design based on domain knowledge, it is quite challenging for NN to reach satisfactory performance in these tabular data domains.

To fill the gap of NN in tabular data learning, we propose a universal neural network solution, called TabNN, to derive effective NN architectures for tabular data in all kinds of tasks automatically.

Specifically, the design of TabNN follows two principles: \emph{to explicitly leverages expressive feature combinations} and \emph{to reduce model complexity}.

Since GBDT has empirically proven its strength in modeling tabular data, we use GBDT to power the implementation of TabNN.

Comprehensive experimental analysis on a variety of tabular datasets demonstrate that TabNN can achieve much better performance than many baseline solutions.

We propose a universal neural network solution to derive effective NN architectures for tabular data automatically.

A new Neural Network training procedure, designed for tabular data, that seeks to leverage feature clusters extracted from GBDTs.

Proposal for a hybrid machine learning algorithm using Gradient Boosted Decision Trees and Deep Neural Networks, with intended research direction on tabular data.

TabNN: A Universal Neural Network Solution for Tabular Data

universal neural network solution
tabular datum


Knowledge Bases (KBs) are becoming increasingly large, sparse and probabilistic.

These KBs are typically used to perform query inferences and rule mining.

But their efficacy is only as high as their completeness.

Efficiently utilizing incomplete KBs remains a major challenge as the current KB completion techniques either do not take into account the inherent uncertainty associated with each KB tuple or do not scale to large KBs.

Probabilistic rule learning not only considers the probability of every KB tuple but also tackles the problem of KB completion in an explainable way.

For any given probabilistic KB, it learns probabilistic first-order rules from its relations to identify interesting patterns.

But, the current probabilistic rule learning techniques perform grounding to do probabilistic inference for evaluation of candidate rules.

It does not scale well to large KBs as the time complexity of inference using grounding is exponential over the size of the KB.

In this paper, we present SafeLearner -- a scalable solution to probabilistic KB completion that performs probabilistic rule learning using lifted probabilistic inference -- as faster approach instead of grounding.

We compared SafeLearner to the state-of-the-art probabilistic rule learner ProbFOIL+ and to its deterministic contemporary AMIE+ on standard probabilistic KBs of NELL (Never-Ending Language Learner) and Yago.

Our results demonstrate that SafeLearner scales as good as AMIE+ when learning simple rules and is also significantly faster than ProbFOIL+.

Probabilistic Rule Learning system using Lifted Inference

A model for probabilistic rule learning to automate the completion of probabilistic databases that uses AMIE+ and lifted inference to help computational efficiency.

Scalable Rule Learning in Probabilistic Knowledge Bases


Recent efforts in Dialogue State Tracking (DST) for task-oriented dialogues have progressed toward open-vocabulary or generation-based approaches where the models can generate slot value candidates from the dialogue history itself.

These approaches have shown good performance gain, especially in complicated dialogue domains with dynamic slot values.

However, they fall short in two aspects: (1) they do not allow models to explicitly learn signals across domains and slots to detect potential dependencies among \textit{(domain, slot)} pairs; and (2) existing models follow auto-regressive approaches which incur high time cost when the dialogue evolves over multiple domains and multiple turns.

In this paper, we propose a novel framework of Non-Autoregressive Dialog State Tracking (NADST) which can factor in potential dependencies among domains and slots to optimize the models towards better prediction of dialogue states as a complete set rather than separate slots.

In particular, the non-autoregressive nature of our method not only enables decoding in parallel to significantly reduce the latency of DST for real-time dialogue response generation, but also detect dependencies among slots at token level in addition to slot and domain level.

Our empirical results show that our model achieves the state-of-the-art joint accuracy across all domains on the MultiWOZ 2.1 corpus, and the latency of our model is an order of magnitude lower than the previous state of the art as the dialogue history extends over time.

We propose the first non-autoregressive neural model for Dialogue State Tracking (DST), achieving the SOTA accuracy (49.04%) on MultiWOZ2.1 benchmark, and reducing inference latency by an order of magnitude.

A new model for the DST task that reduces inference time complexity with a non-autoregressive decoder, obtains competitive DST accuracy, and shows improvements over other baselines.

Proposal for a model that is capable of tracking dialogue states in a non-recursive fashion.

dialogue state
dst
reduce inference
model
accuracy
tracking
non-autoregressive


The 3D-zoom operation is the positive translation of the camera in the Z-axis, perpendicular to the image plane.

In contrast, the optical zoom changes the focal length and the digital zoom is used to enlarge a certain region of an image to the original image size.

In this paper, we are the first to formulate an unsupervised 3D-zoom learning problem where images with an arbitrary zoom factor can be generated from a given single image.

An unsupervised framework is convenient, as it is a challenging task to obtain a 3D-zoom dataset of natural scenes due to the need for special equipment to ensure camera movement is restricted to the Z-axis.

Besides, the objects in the scenes should not move when being captured, which hinders the construction of a large dataset of outdoor scenes.

We present a novel unsupervised framework to learn how to generate arbitrarily 3D-zoomed versions of a single image, not requiring a 3D-zoom ground truth, called the Deep 3D-Zoom Net.

The Deep 3D-Zoom Net incorporates the following features:

(i) transfer learning from a pre-trained disparity estimation network via a back re-projection reconstruction loss;

(ii) a fully convolutional network architecture that models depth-image-based rendering (DIBR), taking into account high-frequency details without the need for estimating the intermediate disparity; and

(iii) incorporating a discriminator network that acts as a no-reference penalty for unnaturally rendered areas.

Even though there is no baseline to fairly compare our results, our method outperforms previous novel view synthesis research in terms of realistic appearance on large camera baselines.

We performed extensive experiments to verify the effectiveness of our method on the KITTI and Cityscapes datasets.

A novel network architecture to perform Deep 3D Zoom or close-ups.

A method for creating a "zoomed image" for a given input image,and a novel back re-projection reconstruction loss that allows the network to learn underlying 3D structure and maintain a natural appearance.

An algorithm for synthesizing 3D-zoom behavior when the camera is moving forward, a network structure incorporating disparity estimation in a GANs framework to synthesize novel views, and a proposed new computer vision task.

Deep 3D-Zoom Net: Unsupervised Learning of Photo-Realistic 3D-Zoom

network novel deep 3d zoom structure 3d-zoom


The universal approximation theorem, in one of its most general versions, says that if we consider only continuous activation functions σ, then a standard feedforward neural network with one hidden layer is able to approximate any continuous multivariate function f to any given approximation threshold ε, if and only if σ is non-polynomial.

In this paper, we give a direct algebraic proof of the theorem.

Furthermore we shall explicitly quantify the number of hidden units required for approximation.

Specifically, if X in R^n is compact, then a neural network with n input units, m output units, and a single hidden layer with {n+d choose d} hidden units (independent of m and ε), can uniformly approximate any polynomial function f:X -> R^m whose total degree is at most d for each of its m coordinate functions.

In the general case that f is any continuous function, we show there exists some N in O(ε^{-n}) (independent of m), such that N hidden units would suffice to approximate f.

We also show that this uniform approximation property (UAP) still holds even under seemingly strong conditions imposed on the weights.

(i) For any δ > 0, the UAP still holds if we restrict all non-bias weights w in the last layer to satisfy |w| < δ.

(ii) There exists some λ>0 (depending only on f and σ), such that the UAP still holds if we restrict all non-bias weights w in the first layer to satisfy |w|>λ.

(iii) If the non-bias weights in the first layer are *fixed* and randomly chosen from a suitable range, then the UAP holds with probability 1.

A quantitative refinement of the universal approximation theorem via an algebraic approach.

The authors derive the universal approximation property proofs algebraically and assert that the results are general to other kinds of neural networks and similar learners.

A new proof of Leshno's version of the universal approximation property for neural networks, and new insights into the universal approximation property.

A closer look at the approximation capabilities of neural networks

universal approximation
neural network
proof
property


In this paper, we design a generic framework for learning a robust text classification model that achieves accuracy comparable to standard full models under test-time

We take a different approach from existing methods and learn to dynamically delete a large fraction of unimportant words by a low-complexity selector such that the high-complexity classifier only needs to process a small fraction of important words.

In addition, we propose a new data aggregation method to train the classifier, allowing it to make accurate predictions even on fragmented sequence of words.

Our end-to-end method achieves state-of-the-art performance while its computational complexity scales linearly with the small fraction of important words in the whole corpus.

Besides, a single deep neural network classifier trained by our framework can be dynamically tuned to different budget levels at inference time.

Modular framework for document classification and data aggregation technique for making the framework robust to various distortion, and noise and focus only on the important words.

The authors consider training a RNN-based text classification where there is a resource restriction on test-time prediction, and provide an approach using a masking mechanism to reduce words/phrases/sentences used in prediction followed by a classifier to handle those components.

Robust Text Classifier on Test-Time Budgets

classification classifier test-time word robust text


Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-net and searching for an optimal architecture.

In this paper, we present a novel approach, namely  Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance.

In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut.

This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels.

We solve it by introducing  edge normalization, which adds a new set of edge-level hyper-parameters to reduce uncertainty in search.

Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoy both faster speed and higher training stability.

Experiment results demonstrate the effectiveness of the proposed method.

Specifically, we achieve an error rate of 2.57% on CIFAR10 within merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet (under the mobile setting) within 3.8 GPU-days for search.

Our code has been made available at https://www.dropbox.com/sh/on9lg3rpx1r6dkf/AABG5mt0sMHjnEJyoRnLEYW4a?dl=0

Allowing partial channel connection in super-networks to regularize and accelerate differentiable architecture search

An extension of the neural architecture search method DARTS that addresses its shortcoming of immense memory cost by using a random subset of channels and a method to normalize edges.

This paper proposes to improve DARTS in terms of training efficiency, from large memory and computing overheads, and proposes a partially-connected DARTS with partial channel connection and edge normalization.

PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search

partial channel connection
architecture search
edge
dart
memory


Dialogue research tends to distinguish between chit-chat and goal-oriented tasks.

While the former is arguably more naturalistic and has a wider use of language, the latter has clearer metrics and a more straightforward learning signal.

Humans effortlessly combine the two, and engage in chit-chat for example with the goal of exchanging information or eliciting a specific response.

Here, we bridge the divide between these two domains in the setting of a rich multi-player text-based fantasy environment where agents and humans engage in both actions and dialogue.

Specifically, we train a goal-oriented model with reinforcement learning via self-play against an imitation-learned chit-chat model with two new approaches: the policy either learns to pick a topic or learns to pick an utterance given the top-k utterances.

We show that both models outperform a strong inverse model baseline and can converse naturally with their dialogue partner in order to achieve goals.

Agents interact (speak, act) and can achieve goals in a rich world with diverse language, bridging the gap between chit-chat and goal-oriented dialogue.

This paper studies a multiagent dialog task in which the learning agent aims to generate natural language actions that elicit a particular action from the other agent, and shows RL-agents can achieve higher task completion levels than imitation learning baselines.

This paper explores the goal-oriented dialogue setting with reinforcement learning in a Fantasy Text Adventure Game and observes that the RL approaches outperform supervised learning models.

I love your chain mail! Making knights smile in a fantasy game world

goal-oriented dialogue
fantasy
achieve
world
agent
language
game


We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies.

Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation.

We propose estimated mixture policy (EMP), a novel class of partially policy-agnostic methods to accurately estimate those quantities.

With careful analysis, we show that EMP gives rise to estimates with reduced variance for estimating the state stationary distribution correction while it also offers a useful induction bias for estimating the state-action stationary distribution correction.

In extensive experiments with both continuous and discrete environments, we demonstrate that our algorithm offers significantly improved accuracy compared to the state-of-the-art methods.

A new partially policy-agnostic method for infinite-horizon off-policy policy evalution with multiple known or unknown behavior policies.

An estimated mixture policy which takes ideas from off-policy policy evaluation infinite horizon estimators and regression importance sampling for importance weight, and extends them to many policies and unknown policies.

An algorithm to solve infinite horizon off policy evaluation with multiple behavior policies by estimating a mixed policy under regression, and theoretical proof that an estimated policy ratio can reduce variance.

Infinite-horizon Off-Policy Policy Evaluation with Multiple Behavior Policies

policy evaluation multiple behavior policy


We introduce a more efficient neural architecture for amortized inference, which combines continuous and conditional normalizing flows using a principled choice of structure.

Our gradient flow derives its sparsity pattern from the minimally faithful inverse of its underlying graphical model.

We find that this factorization reduces the necessary numbers both of parameters in the neural network and of adaptive integration steps in the ODE solver.

Consequently, the throughput at training time and inference time is increased, without decreasing performance in comparison to unconstrained flows.

By expressing the structural inversion and the flow construction as compilation passes of a probabilistic programming language, we demonstrate their applicability to the stochastic inversion of realistic models such as convolutional neural networks (CNN).

We introduce a more efficient neural architecture for amortized inference, which combines continuous and conditional normalizing flows using a principled choice of sparsity structure.

Efficient Inference Amortization in Graphical Models using Structured Continuous Conditional Normalizing Flows

conditional normalizing flow
efficient
inference
continuous


We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way.

By defining the combinatorial search space of NAS to be the set of different edge-partitionings (colorings) into same-weight classes, we represent compact architectures via efficient learned edge-partitionings.

For several RL tasks, we manage to learn colorings translating to effective policies parameterized by as few as 17 weight parameters, providing >90 % compression over vanilla policies and 6x compression over state-of-the-art compact policies based on Toeplitz matrices, while still maintaining good reward.

We believe that our work is one of the first attempts to propose a rigorous approach to training structured neural network architectures for RL problems that are of interest especially in mobile robotics with limited storage and computational resources.

We show that ENAS with ES-optimization in RL is highly scalable, and use it to compactify neural network policies by weight sharing.

The authors construct reinforcement learning policies with very few parameters by compressing a feed-forward neural network, forcing it to share weights, and using a reinforcement learning method to learn the mapping of shared weights.

This paper combines ideas from ENAS and ES methods for optimisation, and introduces the chromatic network architecture, which partitions weights of the RL network into tied sub-groups.

Reinforcement Learning with Chromatic Networks

ena
rl
reinforcement learning
weight
chromatic network
policy
method
neural


Deep approaches to anomaly detection have recently shown promising results over shallow methods on large and complex datasets.

Typically anomaly detection is treated as an unsupervised learning problem.

In practice however, one may have---in addition to a large set of unlabeled samples---access to a small pool of labeled samples, e.g. a subset verified by some domain expert as being normal or anomalous.

Semi-supervised approaches to anomaly detection aim to utilize such labeled samples, but most proposed methods are limited to merely including labeled normal samples.

Only a few methods take advantage of labeled anomalies, with existing deep approaches being domain-specific.

In this work we present Deep SAD, an end-to-end deep methodology for general semi-supervised anomaly detection.

Using an information-theoretic perspective on anomaly detection, we derive a loss motivated by the idea that the entropy of the latent distribution for normal data should be lower than the entropy of the anomalous distribution.

We demonstrate in extensive experiments on MNIST, Fashion-MNIST, and CIFAR-10, along with other anomaly detection benchmark datasets, that our method is on par or outperforms shallow, hybrid, and deep competitors, yielding appreciable performance improvements even when provided with only little labeled data.

We introduce Deep SAD, a deep method for general semi-supervised anomaly detection that especially takes advantage of labeled anomalies.

A new method to find anomaly data, when some labeled anomalies are given, that applies information theory-derived loss based on normal data usuallly having lower entropy than abnormal data.

Proposal for an abnormal detection framework under settings where unlabeled data, labeled positive data, and labeled negative data are available, and proposal to approach semi-supervised AD from an information theoretic perspective.

Deep Semi-Supervised Anomaly Detection

label anomaly
datum
method
abnormal
deep
semi-supervised
detection
information


To analyze deep ReLU network, we adopt a student-teacher setting in which an over-parameterized student network learns from the output of a fixed teacher network of the same depth, with Stochastic Gradient Descent (SGD).

First, we prove that when the gradient is zero (or bounded above by a small constant) at every data point in training, a situation called  \emph{interpolation setting}, there exists many-to-one \emph{alignment} between student and teacher nodes in the lowest layer under mild conditions.

This suggests that generalization in unseen dataset is achievable, even the same condition often leads to zero training error.

Second, analysis of noisy recovery and training dynamics in 2-layer network shows that strong teacher nodes (with large fan-out weights) are learned first and subtle teacher nodes are left unlearned until late stage of training.

As a result, it could take a long time to converge into these small-gradient critical points.

Our analysis shows that over-parameterization plays two roles: (1) it is a necessary condition for alignment to happen at the critical points, and (2) in training dynamics, it helps student nodes cover more teacher nodes with fewer iterations.

This paper analyzes training dynamics and critical points of training deep ReLU network via SGD in the teacher-student setting.

Study of over-parametrization in student-teacher multilayer ReLU networks, a theoretical part about SGD critical points for the teacher-student setting, and a heuristic and empirical part on dynamics of the SDG algorithm as a function of teacher networks.

Toward Understanding Generalization of Over-parameterized Deep ReLU network trained with SGD in Student-teacher Setting

teacher-student setting
critical point
deep relu network
sgd
dynamic
train
student-teacher


We study the convergence of gradient descent (GD) and stochastic gradient descent (SGD) for training $L$-hidden-layer linear residual networks (ResNets).

We prove that for training deep residual networks with certain linear transformations at input and output layers, which are fixed throughout training, both GD and SGD with zero initialization on all hidden weights can converge to the global minimum of the training loss.

Moreover, when specializing to appropriate Gaussian random linear transformations, GD and SGD provably optimize wide enough deep linear ResNets.

Compared with the global convergence result of GD for training standard deep linear networks \citep{du2019width}, our condition on the neural network width is sharper by a factor of $O(\kappa L)$, where $\kappa$ denotes the condition number of the covariance matrix of the training data.

In addition, for the first time we establish the global convergence of SGD for training deep linear ResNets and prove a linear convergence rate when the global minimum is $0$.

Under certain condition on the input and output linear transformations, both GD and SGD can achieve global convergence for training deep linear ResNets.

The authors study the convergence of gradient descent in training deep linear residual networks, and establish a global convergence of GD/SGD and linear convergence rates of SG/SGD.

Study of convergence properties of GD and SGD on deep linear resnets, and proof that under certain conditions on the input and output transformations and with zero initialization, GD and SGD converges to global minima.

On the Global Convergence  of Training Deep Linear ResNets

certain condition input output
gd sgd
train deep linear resnet
global convergence
transformation
study


In this paper, we empirically investigate the training journey of deep neural networks relative to fully trained shallow machine learning models.

We observe that the deep neural networks (DNNs) train by learning to correctly classify shallow-learnable examples in the early epochs before learning the harder examples.

We build on this observation this to suggest a way for partitioning the dataset into hard and easy subsets that can be used for improving the overall training process.

Incidentally, we also found evidence of a subset of intriguing examples across all the datasets we considered, that were shallow learnable but not deep-learnable.

In order to aid reproducibility, we also duly release our code for this work at https://github.com/karttikeya/Shallow_to_Deep/

We analyze the training process for Deep Networks and show that they start from rapidly learning shallow classifiable examples and slowly generalize to harder data points.

Do deep neural networks learn shallow learnable examples first?


While much recent work has targeted learning deep discrete latent variable models with variational inference, this setting remains challenging, and it is often necessary to make use of potentially high-variance gradient estimators in optimizing the ELBO.

As an alternative, we propose to optimize a non-ELBO objective derived from the Bethe free energy approximation to an MRF's partition function.

This objective gives rise to a saddle-point learning problem, which we train inference networks to approximately optimize.

The derived objective requires no sampling, and can be efficiently computed for many MRFs of interest.

We evaluate the proposed approach in learning high-order neural HMMs on text, and find that it often outperforms other approximate inference schemes in terms of true held-out log likelihood.

At the same time, we find that all the approximate inference-based approaches to learning high-order neural HMMs we consider underperform learning with exact inference by a significant margin.

Learning deep latent variable MRFs with a saddle-point objective derived from the Bethe partition function approximation.

A method for learning deep latent-variable MRF with an optimization objective that utilizes Bethe free energy, that also solves the underlying constraints of Bethe free energy optimizations.

An objective for learning latent variable MRFs based on Bethe free energy and amortized inference, different from optimizing the standard ELBO.

Learning Deep Latent-variable MRFs with Amortized Bethe Free Energy Minimization

bethe free energy
learn deep latent-variable mrf
latent variable
objective
amortized


In an explanation generation problem, an agent needs to identify and explain the reasons for its decisions to another agent.

Existing work in this area is mostly confined to planning-based systems that use automated planning approaches to solve the problem.

In this paper, we approach this problem from a new perspective, where we propose a general logic-based framework for explanation generation.

In particular, given a knowledge base $KB_1$ that entails a formula $\phi$ and a second knowledge base $KB_2$ that does not entail $\phi$, we seek to identify an explanation $\epsilon$ that is a subset of $KB_1$ such that the union of $KB_2$ and $\epsilon$ entails $\phi$.

We define two types of explanations, model- and proof-theoretic explanations, and use cost functions to reflect preferences between explanations.

Further, we present our algorithm implemented for propositional logic that compute such explanations and empirically evaluate it in random knowledge bases and a planning domain.

A general framework for explanation generation using Logic.

This paper researches explanation generation from a KR point of view and conducts experiments measuring explanation size and runtime on random formulas and formulas from a Blocksworld instance.

This paper provides a perspective on explanations between two knowledge bases, and runs parallel to work on model reconciliation in planning literature.

A General Logic-based Approach for Explanation Generation


Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem.

This problem can be typically resolved by the rectified linear unit (ReLU) activation.

However, here we show that even for such activation, deep and narrow neural networks (NNs) will converge to erroneous mean or median states of the target function depending on the loss with high probability.

Deep and narrow NNs are encountered in solving partial differential equations with high-order derivatives.

We demonstrate this collapse of such NNs both numerically and theoretically, and provide estimates of the probability of collapse.

We also construct a diagram of a safe region for designing NNs that avoid the collapse to erroneous states.

Finally, we examine different ways of initialization and normalization that may avoid the collapse problem.

Asymmetric initializations may reduce the probability of collapse but do not totally eliminate it.

Deep and narrow neural networks will converge to erroneous mean or median states of the target function depending on the loss with high probability.

This paper studies failure modes of deep and narrow networks, focusing on as small as possible models for which the undesired behavior occurs.

This paper shows that the training of deep ReLU neural networks will converge to a constant classifier with high probability over random initialization if hidden layer widths are too small.

Collapse of deep and narrow neural nets

neural network converge
high probability
small
deep narrow


We study adversarial robustness of neural networks from a margin maximization perspective, where margins are defined as the distances from inputs to a classifier's decision boundary.

Our study shows that maximizing margins can be achieved by minimizing the adversarial loss on the decision boundary at the "shortest successful perturbation", demonstrating a close connection between adversarial losses and the margins.

We propose Max-Margin Adversarial (MMA) training to directly maximize the margins to achieve adversarial robustness.

Instead of adversarial training with a fixed $\epsilon$, MMA offers an improvement by enabling adaptive selection of the "correct" $\epsilon$ as the margin individually for each datapoint.

In addition, we rigorously analyze adversarial training with the perspective of margin maximization, and provide an alternative interpretation for adversarial training, maximizing either a lower or an upper bound of the margins.

Our experiments empirically confirm our theory and demonstrate MMA training's efficacy on the MNIST and CIFAR10 datasets w.r.t. $\ell_\infty$ and $\ell_2$ robustness.

We propose MMA training to directly maximize input space margin in order to improve adversarial robustness primarily by removing the requirement of specifying a fixed distortion bound.

An adaptive margin-based adversarial training approach to train robust DNNs, by maximizing the shortest margin of inputs to the decision boundary, that makes adversarial training with large perturbation possible.

A method for robust learning against adversarial attacks where the input space margin is directly maximized and a softmax variant of the max-margin is introduced.

MMA Training: Direct Input Space Margin Maximization through Adversarial Training

input space margin
mma training
directly maximize
adversarial
robust


Many anomaly detection methods exist that perform well on low-dimensional problems however there is a notable lack of effective methods for high-dimensional spaces, such as images.

Inspired by recent successes in deep learning we propose a novel approach to anomaly detection using generative adversarial networks.

Given a sample under consideration, our method is based on searching for a good representation of that sample in the latent space of the generator; if such a representation is not found, the sample is deemed anomalous.

We achieve state-of-the-art performance on standard image benchmark datasets and visual inspection of the most anomalous samples reveals that our method does indeed return anomalies.

We propose a method for anomaly detection with GANs by searching the generator's latent space for good sample representations.

The authors propose using GAN for anomaly detection, a gradient-descent based method to iteratively update latent representations, and a novel parameter update to the generators.

A GAN based approach to doing anomaly detection for image data where the generator's latent space is explored to find a representation for a test image.

Anomaly Detection with Generative Adversarial Networks

generator 's latent space
anomaly detection
representation
gan
method
base


Variational inference (VI) and Markov chain Monte Carlo (MCMC) are approximate posterior inference algorithms that are often said to have complementary strengths, with VI being fast but biased and MCMC being slower but asymptotically unbiased.

In this paper, we analyze gradient-based MCMC and VI procedures and find theoretical and empirical evidence that these procedures are not as different as one might think.

In particular, a close examination of the Fokker-Planck equation that governs the Langevin dynamics (LD) MCMC procedure reveals that LD implicitly follows a gradient flow that corresponds to a variational inference procedure based on optimizing a nonparametric normalizing flow.

This result suggests that the transient bias of LD (due to too few warmup steps) may track that of VI (due to too few optimization steps), up to differences due to VI’s parameterization and asymptotic bias.

Empirically, we find that the transient biases of these algorithms (and momentum-accelerated versions) do evolve similarly.

This suggests that practitioners with a limited time budget may get more accurate results by running an MCMC procedure (even if it’s far from burned in) than a VI procedure, as long as the variance of the MCMC estimator can be dealt with (e.g., by running many parallel chains).

The transient behavior of gradient-based MCMC and variational inference algorithms is more similar than one might think, calling into question the claim that variational inference is faster than MCMC.

Langevin Dynamics as Nonparametric Variational Inference


Graph Convolutional Networks (GCNs) have recently been shown to be quite successful in modeling graph-structured data.

However, the primary focus has been on handling simple undirected graphs.

Multi-relational graphs are a more general and prevalent form of graphs where each edge has a label and direction associated with it.

Most of the existing approaches to handle such graphs suffer from over-parameterization and are restricted to learning representations of nodes only.

In this paper, we propose CompGCN, a novel Graph Convolutional framework which jointly embeds both nodes and relations in a relational graph.

CompGCN leverages a variety of entity-relation composition operations from Knowledge Graph Embedding techniques and scales with the number of relations.

It also generalizes several of the existing multi-relational GCN methods.

We evaluate our proposed method on multiple tasks such as node classification, link prediction, and graph classification, and achieve demonstrably superior results.

We make the source code of CompGCN available to foster reproducible research.

A Composition-based Graph Convolutional framework for multi-relational graphs.

The authors develop GCN on multi-relational graphs and propose CompGCN, which leverages insights from knowledge graph embeddings and learns node and relation representations to alleviate the problem of over-parameterization.

This paper introduces a GCN framework for multi-relational graphs and generalizes several existing approaches to Knowledge Graph embedding into one framework.

Composition-based Multi-Relational Graph Convolutional Networks

framework multi-relational graph
gcn
knowledge
composition-based
convolutional


State-of-the-art neural machine translation methods employ massive amounts of parameters.

Drastically reducing computational costs of such methods without affecting performance has been up to this point unsolved.

In this work, we propose a quantization strategy tailored to the Transformer architecture.

We evaluate our method on the WMT14 EN-FR and WMT14 EN-DE translation tasks and achieve state-of-the-art quantization results for the Transformer, obtaining no loss in BLEU scores compared to the non-quantized baseline.

We further compress the Transformer by showing that, once the model is trained, a good portion of the nodes in the encoder can be removed without causing any loss in BLEU.

We fully quantize the Transformer to 8-bit and improve translation quality compared to the full precision model.

An 8-bit quantization method to quantize the machine translation model Transformer, proposing to use uniform min-max quantization during inference and bucketing weigts before quantization to reduce quantization error.

A method for reducing the required memory space by a quantization technique, focused on reducing it for Transformer architecture.

Fully Quantized Transformer for Improved Translation

fully quantize transformer
quantization
8-bit
method
reduce
model
translation


Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems.

However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes.

We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space.

The resulting approach, latent embedding optimization (LEO), decouples the gradient-based adaptation procedure from the underlying high-dimensional space of model parameters.

Our evaluation shows that LEO can achieve state-of-the-art performance on the competitive miniImageNet and tieredImageNet few-shot classification tasks.

Further analysis indicates LEO is able to capture uncertainty in the data, and can perform adaptation more effectively by optimizing in latent space.

Latent Embedding Optimization (LEO) is a novel gradient-based meta-learner with state-of-the-art performance on the challenging 5-way 1-shot and 5-shot miniImageNet and tieredImageNet classification tasks.

A new meta-learning framework that learns data-dependent latent space, performs fast adaptation in the latent space, is effective for few-shot learning, has task-dependent initialization for adaptation, and works well for multimodal task distribution.

This paper proposes a latent embedding optimization method for meta-learning, and claims the contribution is to decouple optimization-based meta-learning techniques from high-dimensional space of model parameters.

Meta-Learning with Latent Embedding Optimization

latent embed optimization
space
task
meta-learning


We introduce an approach for augmenting model-free deep reinforcement learning agents with a mechanism for relational reasoning over structured representations, which improves performance, learning efficiency, generalization, and interpretability.

Our architecture encodes an image as a set of vectors, and applies an iterative message-passing procedure to discover and reason about relevant entities and relations in a scene.

In six of seven StarCraft II Learning Environment mini-games, our agent achieved state-of-the-art performance, and surpassed human grandmaster-level on four.

In a novel navigation and planning task, our agent's performance and learning efficiency far exceeded non-relational baselines, it was able to generalize to more complex scenes than it had experienced during training.

Moreover, when we examined its learned internal representations, they reflected important structure about the problem and the agent's intentions.

The main contribution of this work is to introduce techniques for representing and reasoning about states in model-free deep reinforcement learning agents via relational inductive biases.

Our experiments show this approach can offer advantages in efficiency, generalization, and interpretability, and can scale up to meet some of the most challenging test environments in modern artificial intelligence.

Relational inductive biases improve out-of-distribution generalization capacities in model-free reinforcement learning agents

A shared relational network architecture for parameterizing the actor and critic network, focused on distributed advantage actor-critic algorithms, that enhances model-free deep reinforcement techniques with relational knowledge about the environment so agents can learn interpretable state representations.

A quantitative and qualitative analysis and evaluation of the self-attention mechanism combined with relation network in the context of model-free RL.

Deep reinforcement learning with relational inductive biases

relational inductive bias
model-free
network
deep reinforcement
learn
agent


Image translation between two domains is a class of problems aiming to learn mapping from an input image in the source domain to an output image in the target domain.

It has been applied to numerous applications, such as data augmentation, domain adaptation, and unsupervised training.

When paired training data is not accessible, image translation becomes an ill-posed problem.

We constrain the problem with the assumption that the translated image needs to be perceptually similar to the original image and also appears to be drawn from the new domain, and propose a simple yet effective image translation model consisting of a single generator trained with a self-regularization term and an adversarial term.

We further notice that existing image translation techniques are agnostic to the subjects of interest and often introduce unwanted changes or artifacts to the input.

Thus we propose to add an attention module to predict an attention map to guide the image translation process.

The module learns to attend to key parts of the image while keeping everything else unaltered, essentially avoiding undesired artifacts or changes.

The predicted attention map also opens door to applications such as unsupervised segmentation and saliency detection.

Extensive experiments and evaluations show that our model while being simpler, achieves significantly better performance than existing image translation methods.

We propose a simple generative model for unsupervised image translation and saliency detection.

Show, Attend and Translate: Unsupervised Image Translation with Self-Regularization and Attention


Building deep neural networks to control autonomous agents which have to interact in real-time with the physical world, such as robots or automotive vehicles, requires a seamless integration of time into a network’s architecture.

The central question of this work is, how the temporal nature of reality should be reflected in the execution of a deep neural network and its components.

Most artificial deep neural networks are partitioned into a directed graph of connected modules or layers and the layers themselves consist of elemental building blocks, such as single units.

For most deep neural networks, all units of a layer are processed synchronously and in parallel, but layers themselves are processed in a sequential manner.

In contrast, all elements of a biological neural network are processed in parallel.

In this paper, we define a class of networks between these two extreme cases.

These networks are executed in a streaming or synchronous layerwise-parallel manner, unlocking the layers of such networks for parallel processing.

Compared to the standard layerwise-sequential deep networks, these new layerwise-parallel networks show a fundamentally different temporal behavior and flow of information, especially for networks with skip or recurrent connections.

We argue that layerwise-parallel deep networks are better suited for future challenges of deep neural network design, such as large functional modularized and/or recurrent architectures as well as networks allocating different network capacities dependent on current stimulus and/or task complexity.

We layout basic properties and discuss major challenges for layerwise-parallel networks.

Additionally, we provide a toolbox to design, train, evaluate, and online-interact with layerwise-parallel networks.

We define a concept of layerwise model-parallel deep neural networks, for which layers operate in parallel, and provide a toolbox to design, train, evaluate, and on-line interact with these networks.

A GPU-accelerated toolbox for parallel neuron updating, written in Theano, that supports different update orders in recurrent networks and networks with connections that skip layers.

A new toolbox for deep neural networks learning and evaluation, and proposal for a paradigm switch from layerwise-sequential networks to layer-wise parallel networks.

Statestream: A toolbox to explore layerwise-parallel deep neural networks

deep neural network
toolbox
layer
parallel


Deep neural networks are known to be vulnerable to adversarial perturbations.

In this paper, we bridge adversarial robustness of neural nets with Lyapunov stability of dynamical systems.

From this viewpoint, training neural nets is equivalent to finding an optimal control of the discrete dynamical system, which allows one to utilize methods of successive approximations, an optimal control algorithm based on Pontryagin's maximum principle, to train neural nets.

This decoupled training method allows us to add constraints to the optimization, which makes the deep model more robust.

The constrained optimization problem can be formulated as a semi-definite programming problem and hence can be solved efficiently.

Experiments show that our method effectively improves deep model's adversarial robustness.

An adversarial defense method bridging robustness of deep neural nets with Lyapunov stability

The authors formulate training NNs as finding an optimal controller for a discrete dynamical system, allowing them to use method of successive approximations to train a NN in a way to be more robust to adversarial attacks.

This paper uses the theoretical view of a neural network as a discretized ODE to develop a robust control theory aimed at training the network while enforcing robustness.

Adversarially Robust Neural Networks via Optimal Control: Bridging Robustness with Lyapunov Stability

lyapunov stability
bridge robustness
train
adversarial
robust
neural network
method
optimal
control


In this paper, we propose a method named Dimensional reweighting Graph Convolutional Networks (DrGCNs), to tackle the problem of variance between dimensional information in the node representations of GCNs.

We prove that DrGCNs can reduce the variance of the node representations by connecting our problem to the theory of the mean field.

However, practically, we find that the degrees DrGCNs help vary severely on different datasets.

We revisit the problem and develop a new measure K to quantify the effect.

This measure guides when we should use dimensional reweighting in GCNs and how much it can help.

Moreover, it offers insights to explain the improvement obtained by the proposed DrGCNs.

The dimensional reweighting block is light-weighted and highly flexible to be built on most of the GCN variants.

Carefully designed experiments, including several fixes on duplicates, information leaks, and wrong labels of the well-known node classification benchmark datasets, demonstrate the superior performances of DrGCNs over the existing state-of-the-art approaches.

Significant improvements can also be observed on a large scale industrial dataset.

We propose a simple yet effective reweighting scheme for GCNs, theoretically supported by the mean field theory.

A method, known as DrGCN, for reweighting the different dimensions of the node representations in graph convolutional networks by reducing variance between dimensions.

Dimensional Reweighting Graph Convolution Networks


Knowledge-grounded dialogue is a task of generating an informative response based on both discourse context and external knowledge.

As we focus on better modeling the knowledge selection in the multi-turn knowledge-grounded dialogue, we propose a sequential latent variable model as the first approach to this matter.

The model named sequential knowledge transformer (SKT) can keep track of the prior and posterior distribution over knowledge; as a result, it can not only reduce the ambiguity caused from the diversity in knowledge selection of conversation but also better leverage the response information for proper choice of knowledge.

Our experimental results show that the proposed model improves the knowledge selection accuracy and subsequently the performance of utterance generation.

We achieve the new state-of-the-art performance on Wizard of Wikipedia (Dinan et al., 2019) as one of the most large-scale and challenging benchmarks.

We further validate the effectiveness of our model over existing conversation methods in another knowledge-based dialogue Holl-E dataset (Moghe et al., 2018).

Our approach is the first attempt to leverage a sequential latent variable model for knowledge selection in the multi-turn knowledge-grounded dialogue. It achieves the new state-of-the-art performance on Wizard of Wikipedia benchmark.

A sequential latent variable model for knowledge selection in dialogue generation that extends the posterior attention model to the latent knowledge selection problem and achieves higher performances than previous state-of-the-art models.

A novel architecture for selecting knowledge-grounded multi-turn dialogue that yields state of the art on relevant benchmarks datasets, and scores higher in human evaluations.

Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue

sequential latent variable model knowledge selection


Meta-learning, or learning-to-learn, has proven to be a successful strategy in attacking problems in supervised learning and reinforcement learning that involve small amounts of data.

State-of-the-art solutions involve learning an initialization and/or learning algorithm using a set of training episodes so that the meta learner can generalize to an evaluation episode quickly.

These methods perform well but often lack good quantification of uncertainty, which can be vital to real-world applications when data is lacking.

We propose a meta-learning method which efficiently amortizes hierarchical variational inference across tasks, learning a prior distribution over neural network weights so that a few steps of Bayes by Backprop will produce a good task-specific approximate posterior.

We show that our method produces good uncertainty estimates on contextual bandit and few-shot learning benchmarks.

We propose a meta-learning method which efficiently amortizes hierarchical variational inference across training episodes.

An adaptation to MAML-type models that accounts for posterior uncertainty in task specific latent variables by employing variational inference for task-specific parameters in a hierarchical Bayesian view of MAML.

The authors consider meta-learning to learn a prior over neural network weights, done via amortized variational inference.

Amortized Bayesian Meta-Learning

variational inference
amortize
hierarchical
bayesian
meta-learne


Often we wish to transfer representational knowledge from one neural network to another.

Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator.

Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network.

We demonstrate that this objective ignores important structural knowledge of the teacher network.

This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data.

We formulate this objective as contrastive learning.

Experiments demonstrate that our resulting new objective outperforms knowledge distillation on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer.

When combined with knowledge distillation, our method sets a state of the art in many transfer tasks, sometimes even outperforming the teacher network.

Representation/knowledge distillation by maximizing mutual information between teacher and student

This paper combines a contrastive objective measuring the mutual information between the representations learned by teacher and student networks for model distillation, and proposes a model with improvement over existing alternatives on distillation tasks.

Contrastive Representation Distillation

teacher student
mutual information
distillation
representation
contrastive


Developing effective biologically plausible learning rules for deep neural networks is important for advancing connections between deep learning and neuroscience.

To date, local synaptic learning rules like those employed by the brain have failed to match the performance of backpropagation in deep networks.

In this work, we employ meta-learning to discover networks that learn using feedback connections and local, biologically motivated learning rules.

Importantly, the feedback connections are not tied to the feedforward weights, avoiding any biologically implausible weight transport.

It can be shown mathematically that this approach has sufficient expressivity to approximate any online learning algorithm.

Our experiments show that the meta-trained networks effectively use feedback connections to perform online credit assignment in multi-layer architectures.

Moreover, we demonstrate empirically that this model outperforms a state-of-the-art gradient-based meta-learning algorithm for continual learning on regression and classification benchmarks.

This approach represents a step toward biologically plausible learning mechanisms that can not only match gradient descent-based learning, but also overcome its limitations.

Networks that learn with feedback connections and local plasticity rules can be optimized for using meta learning.

Learning to Learn with Feedback and Local Plasticity


In the visual system, neurons respond to a patch of the input known as their classical receptive field (RF), and can be modulated by stimuli in the surround.

These interactions are often mediated by lateral connections, giving rise to extra-classical RFs.

We use supervised learning via backpropagation to learn feedforward connections, combined with an unsupervised learning rule to learn lateral connections between units within a convolutional neural network.

These connections allow each unit to integrate information from its surround, generating extra-classical receptive fields for the units in our new proposed model (CNNEx).

We demonstrate that these connections make the network more robust and achieve better performance on noisy versions of the MNIST and CIFAR-10 datasets.

Although the image statistics of MNIST and CIFAR-10 differ greatly, the same unsupervised learning rule generalized to both datasets.

Our framework can potentially be applied to networks trained on other tasks, with the learned lateral connections aiding the computations implemented by feedforward connections when the input is unreliable.

CNNs with biologically-inspired lateral connections learned in an unsupervised manner are more robust to noisy inputs.

Convolutional neural networks with extra-classical receptive fields


Deep learning (DL) has in recent years been widely used in natural

language processing (NLP) applications due to its superior

However, while natural languages are rich in

grammatical structure, DL has not been able to explicitly

architecture to bridge this gap by exploiting tensor product

representations (TPR), a structured neural-symbolic framework

developed in cognitive science over the past 20 years, with the

aim of integrating DL with explicit language structures and rules.

We call it the Tensor Product Generation Network

(TPGN), and apply it to image captioning.

1) unsupervised learning of
role-unbinding vectors of words via a TPR-based deep neural
network, and

2) integration of TPR with typical DL architectures

including Long Short-Term Memory (LSTM) models.

approach lies in its ability to generate a sentence and extract

partial grammatical structure of the sentence by using

role-unbinding vectors, which are obtained in an unsupervised

Experimental results demonstrate the effectiveness of the

This paper is intended to develop a tensor product representation approach for deep-learning-based natural language processinig applications.

A Neural-Symbolic Approach to Natural Language Tasks


It is well-known that  classifiers are vulnerable to adversarial perturbations.

To defend against adversarial perturbations, various certified robustness results have been derived.

However, existing certified robustnesses are limited to top-1 predictions.

In many real-world applications, top-$k$ predictions are more relevant.

In this work, we aim to derive certified robustness for top-$k$ predictions.

In particular, our certified robustness is based on randomized smoothing, which turns any classifier to a new classifier via adding noise to an input example.

We adopt randomized smoothing because it is scalable to large-scale neural networks and applicable to any classifier.

We derive a tight robustness in $\ell_2$ norm for top-$k$ predictions  when using randomized smoothing with Gaussian noise.

We find that generalizing the certified robustness  from top-1 to top-$k$ predictions faces significant technical challenges.

We also empirically evaluate our method on CIFAR10 and ImageNet.

For example, our method can obtain an ImageNet classifier with a certified top-5 accuracy of 62.8\% when the $\ell_2$-norms of the adversarial perturbations are less than 0.5 (=127/255).

Our code is publicly available at: \url{https://github.com/jjy1994/Certify_Topk}.

We study the certified robustness for top-k predictions via randomized smoothing under Gaussian noise and derive a tight robustness bound in L_2 norm.

This paper extends work on deducing a certified radius using randomized smoothing, and shows the radius at which a smoothed classifier under Gaussian perturbations is certified for the top k predictions.

This paper builds upon the random smoothing technique for top-1 prediction, and aims to provide certification on top-k predictions.

Certified Robustness for Top-k Predictions against Adversarial Perturbations via Randomized Smoothing

certify robustness top-k prediction
via randomized smoothing
gaussian
perturbation


Recent work has shown increased interest in using the Variational Autoencoder (VAE) framework to discover interpretable representations of data in an unsupervised way.

These methods have focussed largely on modifying the variational cost function to achieve this goal.

However, we show that methods like beta-VAE simplify the tendency of variational inference to underfit causing pathological over-pruning and over-orthogonalization of learned components.

In this paper we take a complementary approach: to modify the probabilistic model to encourage structured latent variable representations to be discovered.

Specifically, the standard VAE probabilistic model is unidentifiable: the likelihood of the parameters is invariant under rotations of the latent space.

This means there is no pressure to identify each true factor of variation with a latent variable.

We therefore employ a rich prior distribution, akin to the ICA model, that breaks the rotational symmetry.

Extensive quantitative and qualitative experiments demonstrate that the proposed prior mitigates the trade-off introduced by modified cost functions like beta-VAE and TCVAE between reconstruction loss and disentanglement.

The proposed prior allows to improve these approaches with respect to both disentanglement and reconstruction quality significantly over the state of the art.

We present structured priors for unsupervised learning of disentangled representations in VAEs that significantly mitigate the trade-off between disentanglement and reconstruction loss.

A general framework to use the family of L^p-nested distributions as the prior for the code vector of VAE, demonstrating a higher MIG.

The authors point out issues in current VAE approaches and provide a new perspective on the tradeoff between reconstruction and orthogonalization for VAE, beta-VAE, and beta-TCVAE.

ISA-VAE: Independent Subspace Analysis with Variational Autoencoders


Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks.

The explanations in the literature for the apparent effectiveness of shortcuts are varied and often contradictory.

We hypothesize that shortcuts work primarily because they act as linear counterparts to nonlinear layers.

We test this hypothesis by using several variations on the standard residual block, with different types of linear connections, to build small (100k--1.2M parameter) image classification networks.

Our experiments show that other kinds of linear connections can be even more effective than the identity shortcuts.

Our results also suggest that the best type of linear connection for a given application may depend on both network width and depth.

We generalize residual blocks to tandem blocks, which use arbitrary linear maps instead of shortcuts, and improve performance over ResNets.

This paper performs an analysis of shortcut connections in ResNet-like architectures, and proposes to substitute the identity shortcuts with an alternative convolutional one referred to as tandem block.

This paper investigates the effect of replacing identity skip connections with trainable convolutional skip connections in ResNet and finds that performance improves.

Tandem Blocks in Deep Convolutional Neural Networks

tandem block
shortcut
resnet
performance
improve
connection
identity
convolutional


Adam-typed optimizers, as a class of adaptive moment estimation methods with the exponential moving average scheme, have been successfully used in many applications of deep learning.

Such methods are appealing for capability on large-scale sparse datasets.

On top of that, they are computationally efficient and insensitive to the hyper-parameter settings.

In this paper, we present a new framework for adapting Adam-typed methods, namely AdamT.

Instead of applying a simple exponential weighted average, AdamT also includes the trend information when updating the parameters with the adaptive step size and gradients.

The newly added term is expected to efficiently capture the non-horizontal moving patterns on the cost surface, and thus converge more rapidly.

We show empirically the importance of the trend component, where AdamT outperforms the conventional Adam method constantly in both convex and non-convex settings.

We present a new framework for adapting Adam-typed methods, namely AdamT, to include the trend information when updating the parameters with the adaptive step size and gradients.

A new type of Adam variant that uses Holt's linear method to compute the smoothed first order and second order momentum instead of using exponential weighted average.

AdamT: A Stochastic Optimization with Trend Correction Scheme


As machine learning methods see greater adoption and implementation in high stakes applications such
as medical image diagnosis, the need for model interpretability and explanation has become more
critical.

Classical approaches that assess feature importance (eg saliency maps) do not explain how and why a particular region of an image is relevant to the prediction.

We propose
a method that explains the outcome of a classification black-box by gradually exaggerating
the semantic effect of a given class.

Given a query input to a classifier, our method produces a
progressive set of plausible variations of that query, which gradually change the posterior probability
from its original class to its negation.

These counter-factually generated samples preserve features
unrelated to the classification decision, such that a user can employ our method as a ``tuning knob'' to traverse a data manifold while crossing the decision boundary.

Our method is model agnostic and only requires the output value and gradient of the predictor with respect to its input.

A method to explain a classifier, by generating visual perturbation of an image by exaggerating  or diminishing the semantic features that the classifier associates with a target label.

A model that when given a query input to a black-box, aims to explain the outcome by providing plausible and progressive variations to the query that can result in a change to the output.

A method for explaining the output of black box classification of images, that generates gradual perturbation of outputs in response to gradually perturbed input queries.

Explanation  by Progressive  Exaggeration

method explain
image
generate
output
perturbation
query
progressive
input


We study the problem of explaining a rich class of behavioral properties of deep neural networks.

Our influence-directed explanations approach this problem by peering inside the network to identify neurons with high influence on the property of interest using an axiomatically justified influence measure, and then providing an interpretation for the concepts these neurons represent.

We evaluate our approach by training convolutional neural networks on Pubfig, ImageNet, and Diabetic Retinopathy datasets.

Our evaluation demonstrates that influence-directed explanations (1) localize features used by the network, (2) isolate features distinguishing related instances, (3) help extract the essence of what the network learned about the class, and (4) assist in debugging misclassifications.

We present an influence-directed approach to constructing explanations for the behavior of deep convolutional networks, and show how it can be used to answer a broad set of questions that could not be addressed by prior work.

A way to measure influence that satisfies certain axioms, and a notion of influence that can be used to identify what input part is most influential for the output of a neuron in a deep neural network.

This paper proposes to measure the influence of single neurons with regard to a quantity of interest represented by another neuron.

Influence-Directed Explanations for Deep Convolutional Networks

deep convolutional network
measure influence
explanation
neuron
influence-directed


Standard deep learning systems require thousands or millions of examples to learn a concept, and cannot integrate new concepts easily.

By contrast, humans have an incredible ability to do one-shot or few-shot learning.

For instance, from just hearing a word used in a sentence, humans can infer a great deal about it, by leveraging what the syntax and semantics of the surrounding words tells us.

Here, we draw inspiration from this to highlight a simple technique by which deep recurrent networks can similarly exploit their prior knowledge to learn a useful representation for a new word from little data.

This could make natural language processing systems much more flexible, by allowing them to learn continually from the new words they encounter.

We highlight a technique by which natural language processing systems can learn a new word from context, allowing them to be much more flexible.

A technique for exploiting prior knowledge to learn embedding representations for new words with minimal data.

One-shot and few-shot learning of word embeddings


Recent research developing neural network architectures with external memory have often used the benchmark bAbI question and answering dataset which provides a challenging number of tasks requiring reasoning.

Here we employed a classic associative inference task from the human neuroscience literature in order to more carefully probe the reasoning capacity of existing memory-augmented architectures.

This task is thought to capture the essence of reasoning -- the appreciation of distant relationships among elements distributed across multiple facts or memories.

Surprisingly, we found that current architectures struggle to reason over long distance associations.

Similar results were obtained on a more complex task involving finding the shortest path between nodes in a path.

We therefore developed a novel architecture, MEMO, endowed with the capacity to reason over longer distances.

This was accomplished with the addition of two novel components.

First, it introduces a separation between memories/facts stored in external memory and the items that comprise these facts in external memory.

Second, it makes use of an adaptive retrieval mechanism, allowing a variable number of ‘memory hops’ before the answer is produced.

MEMO is capable of solving our novel reasoning tasks, as well as all 20 tasks in bAbI.

A memory architecture that support inferential reasoning.

This paper proposes changes to the End2End Memory Network architecture, introduces a new Paired Associative Inference task that most existing models struggle to solve, and shows that their proposed architecture solves the task better.

A new task (paired associate inference) drawn from cognitive psychology, and proposal for a new memory architecture with features that allow for better performance on the paired associate task.

MEMO: A Deep Network for Flexible Combination of Episodic Memories

memory architecture
task
well
new
network
pair
inference


Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency.

They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures).

Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results.

In this work, we study how depthwise separable convolutions can be applied to neural machine translation.

We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves better results.

In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation.

We also introduce a new super-separable convolution operation that further reduces the number of parameters and computational cost of the models.

Depthwise separable convolutions improve neural machine translation: the more separable the better.

This paper proposes to use depthwise separable convolution layers in a fully convolutional neural machine translation model, and introduces a new super-separable convolution layer which further reduces computational cost.

Depthwise Separable Convolutions for Neural Machine Translation

neural machine translation
depthwise separable convolution


Interpreting generative adversarial network (GAN) training as approximate divergence minimization has been

theoretically insightful, has spurred discussion, and has lead to theoretically and practically interesting

extensions such as f-GANs and Wasserstein GANs.

For both classic GANs and f-GANs, there is an original variant of training and a "non-saturating" variant which uses an alternative form of generator gradient.

The original variant is theoretically easier to study, but for GANs the alternative variant performs better in practice.

The non-saturating scheme is often regarded as a simple modification to deal with optimization issues, but we show that in fact the non-saturating scheme for GANs is effectively optimizing a reverse KL-like f-divergence.

We also develop a number of theoretical tools to help compare and classify f-divergences.

We hope these results may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.

Non-saturating GAN training effectively minimizes a reverse KL-like f-divergence.

This paper proposes a useful expression of the class of f-divergences, investigates theoretical properties of popular f-divergences from newly developed tools, and investigates GANs with the non-saturating training scheme.

The divergences minimized by non-saturating GAN training

non-saturating gan training
minimize
f-divergence


We introduce a novel method for converting text data into abstract image representations, which allows image-based processing techniques (e.g. image classification networks) to be applied to text-based comparison problems.

We apply the technique to entity disambiguation of inventor names in US patents.

The method involves converting text from each pairwise comparison between two inventor name records into a 2D RGB (stacked) image representation.

We then train an image classification neural network to discriminate between such pairwise comparison images, and use the trained network to label each pair of records as either matched (same inventor) or non-matched (different inventors), obtaining highly accurate results (F1: 99.09%, precision: 99.41%, recall: 98.76%).

Our new text-to-image representation method could potentially be used more broadly for other NLP comparison problems, such as disambiguation of academic publications, or for problems that require simultaneous classification of both text and images.

We introduce a novel text representation method which enables image classifiers to be applied to text classification problems, and apply the method to inventor name disambiguation.

A method to map a pair of textual information into a 2D RGB image that can be fed to 2D convoutional neural networks (image classifiers).

The authors consider the problem of names disambiguisation for patent names inventors and propose to build an image page representation of the two name strings to compare and to apply an image classifier.

A novel text representation which enables image classifiers to perform text classification

enable image classifier
novel text representation
apply
problem
method
inventor
classification
name


We propose a novel algorithm, Difference-Seeking Generative Adversarial Network (DSGAN), developed from traditional GAN.

DSGAN considers the scenario that the training samples of target distribution, $p_{t}$, are difficult to collect.

Suppose there are two distributions  $p_{\bar{d}}$ and $p_{d}$ such that the density of the target distribution can be the differences between the densities of $p_{\bar{d}}$ and $p_{d}$.

We show how to learn the target distribution $p_{t}$ only via samples from $p_{d}$ and $p_{\bar{d}}$ (relatively easy to obtain).

DSGAN has the flexibility to produce samples from various target distributions (e.g. the out-of-distribution).

Two key applications, semi-supervised learning and adversarial training, are taken as examples to validate the effectiveness of DSGAN.

We also provide theoretical analyses about the convergence of DSGAN.

We proposed "Difference-Seeking Generative Adversarial Network" (DSGAN) model to learn the target distribution which is hard to collect training data.

This paper presents DS-GAN, which aims to learn the difference between any two distributions whose samples are difficult or impossible to collect, and shows its effectiveness on semi-supervised learning and adversarial training tasks.

This paper considers the problem of learning a GAN to capture a target distribution with only very few training samples from that distribution available.

target distribution
learn
generative adversarial network
collect
sample
training


Recently, Generative Adversarial Network (GAN) and numbers of its variants have been widely used to solve the image-to-image translation problem and achieved extraordinary results in both a supervised and unsupervised manner.

However, most GAN-based methods suffer from the imbalance problem between the generator and discriminator in practice.

Namely, the relative model capacities of the generator and discriminator do not match, leading to mode collapse and/or diminished gradients.

To tackle this problem, we propose a GuideGAN based on attention mechanism.

More specifically, we arm the discriminator with an attention mechanism so not only it estimates the probability that its input is real, but also does it create an attention map that highlights the critical features for such prediction.

This attention map then assists the generator to produce more plausible and realistic images.

We extensively evaluate the proposed GuideGAN framework on a  number of image transfer tasks.

Both qualitative results and quantitative comparison demonstrate the superiority of our proposed approach.

A general method that improves the image translation performance of GAN framework by using an attention embedded discriminator

A feedback mechanism in the GAN framework which improves the quality of generated images in image-to-image translation, and whose discriminator outputs a map indicating where the generator should focus to make its results more convincing.

Proposal for a GAN with an attention-based discriminator for I2I translation which provides the probability of real/fake and an attention map which reflects salience for image generation.

GUIDEGAN:  ATTENTION  BASED  SPATIAL  GUIDANCE FOR  IMAGE-TO-IMAGE TRANSLATION

gan framework
improve
attention
image
image-to-image translation
map
discriminator


The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation.

However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains unexplored.

This paper specifically aims to study the fact verification given semi-structured data as evidence.

To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED.

TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning.

To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA).

Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification.

LPA parses statements into LISP-like programs and executes them against the tables to obtain the returned binary value for verification.

Both methods achieve similar accuracy but still lag far behind human performance.

We also perform a comprehensive analysis to demonstrate great future opportunities.

We propose a new dataset to investigate the entailment problem under semi-structured table as premise

This paper proposes a new dataset for table-based fact verification and introduces methods for the task.

The authors propose the problem of fact verification with semi-structured data sources such as tables, create a new dataset, and evaluate baseline models with variations.

TabFact: A Large-scale Dataset for Table-based Fact Verification

new dataset
table-based fact verification
table
semi-structured
problem


This work presents a two-stage neural architecture for learning and refining structural correspondences between graphs.

First, we use localized node embeddings computed by a graph neural network to obtain an initial ranking of soft correspondences between nodes.

Secondly, we employ synchronous message passing networks to iteratively re-rank the soft correspondences to reach a matching consensus in local neighborhoods between graphs.

We show, theoretically and empirically, that our message passing scheme computes a well-founded measure of consensus for corresponding neighborhoods, which is then used to guide the iterative re-ranking process.

Our purely local and sparsity-aware architecture scales well to large, real-world inputs while still being able to recover global correspondences consistently.

We demonstrate the practical effectiveness of our method on real-world tasks from the fields of computer vision and entity alignment between knowledge graphs, on which we improve upon the current state-of-the-art.

We develop a deep graph matching architecture which refines initial correspondences in order to reach neighborhood consensus.

A framework for answering graph matching questions consisting of local node embeddings with a message passing refinement step.

A two-stage GNN-based architecture to establish correspondences between two graphs that performs well on real-world tasks of image matching and knowledge graph entity alignment.

Deep Graph Matching Consensus

architecture
graph matching
correspondence
deep
consensus


This paper extends the proof of density of neural networks in the space of continuous (or even measurable) functions on Euclidean spaces to functions on compact sets of probability measures.

By doing so the work parallels a more then a decade old results on mean-map embedding of probability measures in reproducing kernel Hilbert spaces.

The work has wide practical consequences for multi-instance learning, where it theoretically justifies some recently proposed constructions.

The result is then extended to Cartesian products, yielding universal approximation theorem for tree-structured domains, which naturally occur in data-exchange formats like JSON, XML, YAML, AVRO, and ProtoBuffer.

This has important practical implications, as it enables to automatically create an architecture of neural networks for processing structured data (AutoML paradigms), as demonstrated by an accompanied library for JSON format.

This paper investigates the approximation properties of a family of neural networks designed to address multi-instance learning problems, and shows that results for standard one layer architectures extend to these models.

This paper generalizes the universal approximation theorem to real functions on the space of measures.

Approximation capability of neural networks on sets of probability measures and tree-structured data

set probability measure
neural network
space
extend
function
approximation