Lifelong learning is the problem of learning multiple consecutive tasks in a sequential manner where knowledge gained from previous tasks is retained and used for future learning.

It is essential towards the development of intelligent machines that can adapt to their surroundings.

In this work we focus on a lifelong learning approach to generative modeling where we continuously incorporate newly observed streaming distributions into our learnt model.

We do so through a student-teacher architecture which allows us to learn and preserve all the distributions seen so far without the need to retain the past data nor the past models.

Through the introduction of a novel cross-model regularizer, the student model leverages the information learnt by the teacher, which acts as a summary of everything seen till now.

The regularizer has the additional benefit of reducing the effect of catastrophic interference that appears when we learn over streaming data.

We demonstrate its efficacy on streaming distributions as well as its ability to learn a common latent representation across a complex transfer learning scenario.

Lifelong distributional learning through a student-teacher architecture coupled with a cross model posterior regularizer.

Lifelong Generative Modeling


Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling.

In this paper, we look at geometric data represented as point clouds.

We introduce a deep autoencoder (AE) network with excellent reconstruction quality and generalization ability.

The learned representations outperform the state of the art in 3D recognition tasks and enable basic shape editing applications via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation.

We also perform a thorough study of different generative models including GANs operating on the raw point clouds, significantly improved GANs trained in the fixed latent space our AEs and, Gaussian mixture models (GMM).

Interestingly, GMMs trained in the latent space of our AEs produce samples of the best fidelity and diversity.

To perform our quantitative evaluation of generative models, we propose simple measures of fidelity and diversity based on optimally matching between sets point clouds.

Deep autoencoders to learn a good representation for geometric 3D point-cloud data; Generative models for point clouds.

Approaches to learn GAN-type generative models using PointNet architecture and latent-space GAN.

Learning Representations and Generative Models for 3D Point Clouds

generative model
point cloud
learn
representation
3d


Despite the remarkable performance of deep neural networks (DNNs) on various tasks, they are susceptible to adversarial perturbations which makes it difficult to deploy them in real-world safety-critical applications.

In this paper, we aim to obtain robust networks by sparsifying DNN's latent features sensitive to adversarial perturbation.

Specifically, we define vulnerability at the latent feature space and then propose a Bayesian framework to prioritize/prune features based on their contribution to both the original and adversarial loss.

We also suggest regularizing the features' vulnerability during training to improve robustness further.

While such network sparsification has been primarily studied in the literature for computational efficiency and regularization effect of DNNs, we confirm that it is also useful to design a defense mechanism through quantitative evaluation and qualitative analysis.

We validate our method, \emph{Adversarial Neural Pruning (ANP)} on multiple benchmark datasets, which results in an improvement in test accuracy and leads to state-of-the-art robustness.

ANP also tackles the practical problem of obtaining sparse and robust networks at the same time, which could be crucial to ensure adversarial robustness on lightweight networks deployed to computation and memory-limited devices.

We propose a novel method for suppressing the vulnerability of latent feature space to achieve robust and compact networks.

This paper proposes "adversarial neural pruning" method of training a pruning mask and a new vulnerability suppression loss to improve accuracy and adversarial robustness.

Adversarial Neural Pruning

adversarial neural pruning
method
vulnerability


In anomaly detection (AD), one seeks to identify whether a test sample is abnormal,  given a data set of normal samples.

A recent and promising approach to AD relies on deep generative models, such as variational autoencoders (VAEs),for unsupervised learning of the normal data distribution.

In semi-supervised AD (SSAD), the data also includes a small sample of labeled anomalies.

In this work,we propose two variational methods for training VAEs for SSAD.

The intuitive idea in both methods is to train the encoder to ‘separate’ between latent vectors for normal and outlier data.

We show that this idea can be derived from principled probabilistic formulations of the problem, and propose simple and effective algorithms.

Our methods can be applied to various data types, as we demonstrate on SSAD datasets ranging from natural images to astronomy and medicine, and can be combined with any VAE model architecture.

When comparing to state-of-the-art SSAD methods that are not specific to particular data types, we obtain marked improvement in outlier detection.

We proposed two VAE modifications that account for negative data examples, and used them for semi-supervised anomaly detection.

The papers propose two methods of VAE-like approaches for semi-supervised novelty detection, MML-VAE and DP-VAE.

Deep Variational Semi-Supervised Novelty Detection


We introduce dynamic instance hardness (DIH) to facilitate the training of machine learning models.

DIH is a property of each training sample and is computed as the running mean of the sample's instantaneous hardness as measured over the training history.

We use DIH to evaluate how well a model retains knowledge about each training sample over time.

We find that for deep neural nets (DNNs), the DIH of a sample in relatively early training stages reflects its DIH in later stages and as a result, DIH can be effectively used to reduce the set of training samples in future epochs.

Specifically, during each epoch, only samples with high DIH are trained (since they are historically hard) while samples with low DIH can be safely ignored.

DIH is updated each epoch only for the selected samples, so it does not require additional computation.

Hence, using DIH during training leads to an appreciable speedup.

Also, since the model is focused on the historically more challenging samples, resultant models are more accurate.

The above, when formulated as an algorithm, can be seen as a form of curriculum learning, so we call our framework DIH curriculum learning (or DIHCL).

The advantages of DIHCL, compared to other curriculum learning approaches, are: (1) DIHCL does not require additional inference steps over the data not selected by DIHCL in each epoch, (2) the dynamic instance hardness, compared to static instance hardness (e.g., instantaneous loss), is more stable as it integrates information over the entire training history up to the present time.

Making certain mathematical assumptions, we formulate the problem of DIHCL as finding a curriculum that maximizes a multi-set function $f(\cdot)$, and derive an approximation bound for a DIH-produced curriculum relative to the optimal curriculum.

Empirically, DIHCL-trained DNNs significantly outperform random mini-batch SGD and other recently developed curriculum learning methods in terms of efficiency, early-stage convergence, and final performance, and this is shown in training several state-of-the-art DNNs on 11 modern datasets.

New understanding of training dynamics and metrics of memorization hardness lead to efficient and provable curriculum learning.

This paper formulates DIH as a curriculum leaning problem that can more effectively utilize the data to train DNNs, and derives theory on the approximation bound.

Dynamic Instance Hardness


This paper explores many immediate connections between adaptive control and machine learning, both through common update laws as well as common concepts.

Adaptive control as a field has focused on mathematical rigor and guaranteed convergence.

The rapid advances in machine learning on the other hand have brought about a plethora of new techniques and problems for learning.

This paper elucidates many of the numerous common connections between both fields such that results from both may be leveraged together to solve new problems.

In particular, a specific problem related to higher order learning is solved through insights obtained from these intersections.

History of parallel developments in update laws and concepts between adaptive control and optimization in machine learning.

Connections Between Optimization in Machine Learning and Adaptive Control

optimization machine learning
adaptive control


Recurrent convolution (RC) shares the same convolutional kernels and unrolls them multiple times, which is originally proposed to model time-space signals.

We suggest that RC can be viewed as a model compression strategy for deep convolutional neural networks.

RC reduces the redundancy across layers and is complementary to most existing model compression approaches.

However, the performance of an RC network can't match the performance of its corresponding standard one, i.e. with the same depth but independent convolutional kernels.

This reduces the value of RC for model compression.

In this paper, we propose a simple variant which improves RC networks: The batch normalization layers of an RC module are learned independently (not shared) for different unrolling steps.

Experiments on CIFAR show that unrolling a convolutional layer several steps can improve the performance, thus indirectly plays a role in model compression.

Recurrent convolution for model compression and a trick for training it, that is learning independent BN layres over steps.

The author modifies the recurrent convolution neural network (RCNN) with independent batch normalization, with the experimental results on RCNN compatible with the ResNet neural network architecture when it contains the same number of layers.

Recurrent Convolutions: A Model Compression Point of View

recurrent convolution model compression
independent


The visual world is vast and varied, but its variations divide into structured and unstructured factors.

Structured factors, such as scale and orientation, admit clear theories and efficient representation design.

Unstructured factors, such as what it is that makes a cat look like a cat, are too complicated to model analytically, and so require free-form representation learning.

We compose structured Gaussian filters and free-form filters, optimized end-to-end, to factorize the representation for efficient yet general learning.

Our experiments on dynamic structure, in which the structured filters vary with the input, equal the accuracy of dynamic inference with more degrees of freedom while improving efficiency.

(Please see https://arxiv.org/abs/1904.11487 for the full edition.)

Dynamic receptive fields with spatial Gaussian structure are accurate and efficient.

This paper proposes a structured convolution operator to model deformations of local regions of an image, which significantly reduced the number of parameters.

Efficient Receptive Field Learning by Dynamic Gaussian Structure

gaussian structure
receptive field
efficient
dynamic


It is widely known that well-designed perturbations can cause state-of-the-art machine learning classifiers to mis-label an image, with sufficiently small perturbations that are imperceptible to the human eyes.

However, by detecting the inconsistency between the image and wrong label, the human observer would be alerted of the attack.

In this paper, we aim to design attacks that not only make classifiers generate wrong labels, but also make the wrong labels imperceptible to human observers.

To achieve this, we propose an algorithm called LabelFool which identifies a target label similar to the ground truth label and finds a perturbation of the image for this target label.

We first find the target label for an input image by a probability model, then move the input in the feature space towards the target label.

Subjective studies on ImageNet show that in the label space, our attack is much less recognizable by human observers, while objective experimental results on ImageNet show that we maintain similar performance in the image space as well as attack rates to state-of-the-art attack algorithms.

A trick on adversarial samples so that the mis-classified labels are imperceptible in the label space to human observers

A method for constructing adversarial attacks that are less detectable by humans without cost in image space by changing the target class to be similar to the original class of the image.

LabelFool: A Trick in the Label Space


This paper presents noise type/position classification of various impact noises generated in a building which is a serious conflict issue in apartment complexes.

For this study, a collection of floor impact noise dataset is recorded with a single microphone.

Noise types/positions are selected based on a report by the Floor Management Center under Korea Environmental Corporation.

Using a convolutional neural networks based classifier, the impact noise signals converted to log-scaled Mel-spectrograms are classified into noise types or positions.

Also, our model is evaluated on a standard environmental sound dataset ESC-50 to show extensibility on environmental sound classification.

This work describes the use of convolutional neural networks in a novel application area of building noise type and noise position classification.

Classification of Building Noise Type/Position via Supervised Learning

noise type / position
building
classification


Recordings of neural circuits in the brain reveal extraordinary dynamical richness and high variability.

At the same time, dimensionality reduction techniques generally uncover low-dimensional structures underlying these dynamics.

What determines the dimensionality of activity in neural circuits?

What is the functional role of dimensionality in behavior and task learning?

In this work we address these questions using recurrent neural network (RNN) models.

We find that, depending on the dynamics of the initial network, RNNs learn to increase and reduce dimensionality in a way that matches task demands.

These findings shed light on fundamental dynamical mechanisms by which neural networks solve tasks with robust representations that generalize to new cases.

Recurrent Neural Networks learn to  increase and reduce the dimensionality of their internal representation in a way that matches the task, depending on the dynamics of the initial network.

Recurrent neural networks learn robust representations by dynamically balancing compression and expansion

recurrent neural network learn
representation


Domain adaptation addresses the common problem when the target distribution generating our test data drifts from the source (training) distribution.

While absent assumptions, domain adaptation is impossible, strict conditions, e.g. covariate or label shift, enable principled algorithms.

Recently-proposed domain-adversarial approaches consist of aligning source and target encodings, often motivating this approach as minimizing two (of three) terms in a theoretical bound on target error.

Unfortunately, this minimization can cause arbitrary increases in the third term, e.g. they can break down under shifting label distributions.

We propose asymmetrically-relaxed distribution alignment, a new approach that overcomes some limitations of standard domain-adversarial algorithms.

Moreover, we characterize precise assumptions under which our algorithm is theoretically principled and demonstrate empirical benefits on both synthetic and real datasets.

Instead of strict distribution alignments in traditional deep domain adaptation objectives, which fails when target label distribution shifts, we propose to optimize a relaxed objective with new analysis, new algorithms, and experimental validation.

This paper suggests relaxed metrics for domain adaptation which give new theoretical bounds on the target error.

Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment

domain adaptation
target
distribution alignment
relaxed
new


In this paper, we explore \textit{summary-to-article generation}: the task of generating long articles given a short summary, which provides finer-grained content control for the generated text.

To prevent sequence-to-sequence (seq2seq) models from degenerating into language models and better controlling the long text to be generated, we propose a hierarchical generation approach which first generates a sketch of intermediate length based on the summary and then completes the article by enriching the generated sketch.

To mitigate the discrepancy between the ``oracle'' sketch used during training and the noisy sketch generated during inference, we propose an end-to-end joint training framework based on multi-agent reinforcement learning.

For evaluation, we use text summarization corpora by reversing their inputs and outputs, and introduce a novel evaluation method that employs a summarization system to summarize the generated article and test its match with the original input summary.

Experiments show that our proposed hierarchical generation approach can generate a coherent and relevant article based on the given summary, yielding significant improvements upon conventional seq2seq models.

we explore the task of summary-to-article generation and propose a hierarchical generation scheme together with a jointly end-to-end reinforcement learning framework to train the hierarchical model.

To address the issue of degeneration in summary-to-article generation, this paper proposes a hierarchical generation approach which first generates an intermediate sketch of the article and then the full article.

Hierarchical Summary-to-Article Generation


When training a deep neural network for supervised image classification, one can broadly distinguish between two types of latent features of images that will drive the classification of class Y. Following the notation of Gong et al. (2016), we can divide features broadly into the classes of

(i) “core” or “conditionally invariant” features X^ci whose distribution P(X^ci | Y) does not change substantially across domains and

(ii) “style” or “orthogonal” features X^orth whose distribution P(X^orth | Y) can change substantially across domains.

These latter orthogonal features would generally include features such as position, rotation, image quality or brightness but also more complex ones like hair color or posture for images of persons.

We try to guard against future adversarial domain shifts by ideally just using the “conditionally invariant” features for classification.

In contrast to previous work, we assume that the domain itself is not observed and hence a latent variable.

We can hence not directly see the distributional change of features across different domains.

We do assume, however, that we can sometimes observe a so-called identifier or ID variable.

We might know, for example, that two images show the same person, with ID referring to the identity of the person.

In data augmentation, we generate several images from the same original image, with ID referring to the relevant original image.

The method requires only a small fraction of images to have an ID variable.

We provide a causal framework for the problem by adding the ID variable to the model of Gong et al. (2016).

However, we are interested in settings where we cannot observe the domain directly and we treat domain as a latent variable.

If two or more samples share the same class and identifier, (Y, ID)=(y,i), then we treat those samples as counterfactuals under different style interventions on the orthogonal or style features.

Using this grouping-by-ID approach, we regularize the network to provide near constant output across samples that share the same ID by penalizing with an appropriate graph Laplacian.

This is shown to substantially improve performance in settings where domains change in terms of image quality, brightness, color changes, and more complex changes such as changes in movement and posture.

We show links to questions of interpretability, fairness and transfer learning.

We propose counterfactual regularization to guard against adversarial domain shifts arising through shifts in the distribution of latent "style features" of images.

The paper discusses ways to guard against adversarial domain shifts with counterfactual regularization by learning a classifier that is invariant to superficial changes (or "style" features) in imagess.

This paper aims at robust image classification against adversarial domain shifts and the goal is achieved by avoiding using the changing style features.

Grouping-By-ID: Guarding Against Adversarial Domain Shifts

guard adversarial domain shift
counterfactual regularization
change
"
style feature
image


Gradient-based meta-learning algorithms require several steps of gradient descent to adapt to newly incoming tasks.

This process becomes more costly as the number  of  samples  increases.

Moreover,  the  gradient  updates  suffer  from  several sources of noise leading to a degraded performance

.   In this work,  we propose a meta-learning algorithm equipped with the GradiEnt Component COrrections, aGECCO cell for short, which generates a multiplicative corrective low-rank matrix which (after vectorization) corrects the estimated gradients

. GECCO contains a simple decoder-like network with learnable parameters, an attention module and a so-called context input parameter

. The context parameter of GECCO is updated to  generate  a  low-rank  corrective  term  for  the  network  gradients

.   As  a  result, meta-learning requires only a few of gradient updates to absorb new task (often, a single update is sufficient in the few-shot scenario

). While previous approaches address this problem by altering the learning rates, factorising network parameters or directly learning feature corrections from features and/or gradients, GECCO is an off-the-shelf generator-like unit that performs element-wise gradient corrections without the need to ‘observe’ the features and/or the gradients directly

(ii) performs robust corrections of the gradients corrupted by a noise, and

(iii) leads to notable improvements over existing gradient-based meta-learning algorithms.

We propose a meta-learner to adapt quickly on multiple tasks even one step in a few-shot setting.

This paper proposes a method to meta-learn a gradient correction module in which preconditioning is parameterized by a neural network, and builds in a two-stage gradient update process during adaptation.

Learning to Learn via Gradient Component Corrections


Discriminative  question  answering  models  can  overfit  to  superficial  biases  in datasets,  because their loss function saturates when any clue makes the answer likely.

We introduce generative models of the joint distribution of questions and answers, which are trained to explain the whole question, not just to answer it.

Our  question  answering  (QA)  model  is  implemented  by  learning  a  prior  over answers,  and  a  conditional  language  model  to  generate  the  question  given  the answer—allowing scalable and interpretable many-hop reasoning as the question is generated word-by-word.

Our model achieves competitive performance with specialised discriminative models on the SQUAD and CLEVR benchmarks, indicating that it is a more general architecture for language understanding and reasoning than previous work.

The model greatly improves generalisation both from biased training data and to adversarial testing data, achieving a new state-of-the-art on ADVERSARIAL SQUAD.

Question answering models that model the joint distribution of questions and answers can learn more than discriminative models

This paper proposes a generative approach to textual and visual QA, where a joint distribution over the question and answer space given the context is learned, which captures more complex relationships.

This paper introduces a generative model for question answering and proposes to model p(q,a|c), factorized as p(a|c) * p(q|a,c).

The authors proposes a generative QA model, which optimizes jointly the distribution of questions and answering given a document/context.

Generative Question Answering: Learning to Answer the Whole Question

joint distribution question answer
learn
model
generative
give
context
qa


In this paper, we turn our attention to the interworking between the activation functions and the batch normalization, which is a virtually mandatory technique to train deep networks currently.

We propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization.

Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of standardized VGG and Residual Networks state-of-the-art models.

These convolutional neural networks were trained on CIFAR-100 and CIFAR-10, the most commonly used deep learning computer vision datasets.

The results showed DReLU speeded up learning in all models and datasets.

Besides, statistical significant performance assessments (p<0.05) showed DReLU enhanced the test accuracy presented by ReLU in all scenarios.

Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception, in which case it presented the second best performance.

Therefore, this work demonstrates that it is possible to increase performance replacing ReLU by an enhanced activation function.

A new activation function called Displaced Rectifier Linear Unit is proposed. It is showed to enhance the training and inference performance of batch normalized convolutional neural networks.

The paper compares and suggests against the usage of batch normalization after using rectifier linear units

This paper proposes an activation function, called displaced ReLU, to improve the performance of CNNs that use batch normalization.

Enhancing Batch Normalized Convolutional Networks using Displaced Rectifier Linear Units: A Systematic Comparative Study

displace rectifi linear unit
batch normalization
normalize convolutional
performance
network
activation function
call
enhance


Encoding the input scale information explicitly into the representation learned by a convolutional neural network (CNN) is beneficial for many vision tasks especially when dealing with multiscale input signals.

We study, in this paper, a scale-equivariant CNN architecture with joint convolutions across the space and the scaling group, which is shown to be both sufficient and necessary to achieve scale-equivariant representations.

To reduce the model complexity and computational burden, we decompose the convolutional filters under two pre-fixed separable bases and truncate the expansion to low-frequency components.

A further benefit of the truncated filter expansion is the improved deformation robustness of the equivariant representation.

Numerical experiments demonstrate that the proposed scale-equivariant neural network with decomposed convolutional filters (ScDCFNet) achieves significantly improved performance in multiscale image classification and better interpretability than regular CNNs at a reduced model size.

We construct scale-equivariant convolutional neural networks in the most general form with both computational efficiency and proved deformation robustness.

The authors propose a CNN architecture that is theoretically equivariant to isotropic scalings and translations by adding an extra scale-dimension to activation tensors.

Scale-Equivariant Neural Networks with Decomposed Convolutional Filters

neural network
scale-equivariant
convolutional


In this paper, we diagnose deep neural networks for 3D point cloud processing to explore the utility of different network architectures.

We propose a number of hypotheses on the effects of specific network architectures on the representation capacity of DNNs.

In order to prove the hypotheses, we design five metrics to diagnose various types of DNNs from the following perspectives, information discarding, information concentration, rotation robustness, adversarial robustness, and neighborhood inconsistency.

We conduct comparative studies based on such metrics to verify the hypotheses, which may shed new lights on the architectural design of neural networks.

Experiments demonstrated the effectiveness of our method.

The code will be released when this paper is accepted.

We diagnose deep neural networks for 3D point cloud processing to explore the utility of different network architectures.

The paper investigates different neural network architectures for 3D point cloud processing and proposes metrics for adversarial robustness, rotational robustness, and neighborhood consistency.

Utility Analysis of Network Architectures for 3D Point Cloud Processing

network architecture 3d point cloud processing


In this work we construct flexible joint distributions from low-dimensional conditional semi-implicit distributions.

Explicitly defining the structure of the approximation allows to make the variational lower bound tighter, resulting in more accurate inference.

Utilizing the structure of distributions improves semi-implicit variational inference

Structured Semi-Implicit Variational Inference

semi-implicit variational inference
structure


Imitation learning from human-expert demonstrations has been shown to be greatly helpful for challenging reinforcement learning problems with sparse environment rewards.

However, it is very difficult to achieve similar success without relying on expert demonstrations.

Recent works on self-imitation learning showed that imitating the agent's own past good experience could indirectly drive exploration in some environments, but these methods often lead to sub-optimal and myopic behavior.

To address this issue, we argue that exploration in diverse directions by imitating diverse trajectories, instead of focusing on limited good trajectories, is more desirable for the hard-exploration tasks.

We propose a new method of learning a trajectory-conditioned policy to imitate diverse trajectories from the agent's own past experiences and show that such self-imitation helps avoid myopic behavior and increases the chance of finding a globally optimal solution for hard-exploration tasks, especially when there are misleading rewards.

Our method significantly outperforms existing self-imitation learning and count-based exploration methods on various hard-exploration tasks with local optima.

In particular, we report a state-of-the-art score of more than 20,000 points on Montezumas Revenge without using expert demonstrations or resetting to arbitrary states.

Self-imitation learning of diverse trajectories with trajectory-conditioned policy

This paper addresses hard exploration tasks by applying self-imitation to a diverse selection of trajectories from past experience, to drive more efficient exploration in sparse-reward problems, achieving SOTA results.

Self-Imitation Learning via Trajectory-Conditioned Policy for Hard-Exploration Tasks

trajectory-conditioned policy
diverse
self-imitation learning
trajectory
task


We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost.

This is achieved by gating the deep-learning architecture on a fine-grained-level.

Individual convolutional maps are turned on/off conditionally on features in the network.

To achieve this, we introduce a new residual block architecture that gates convolutional channels in a fine-grained manner.

We also introduce a generally applicable tool batch-shaping that matches the marginal aggregate posteriors of features in a neural network to a pre-specified prior distribution.

We use this novel technique to force gates to be more conditional on the data.

We present results on CIFAR-10 and ImageNet datasets for image classification, and Cityscapes for semantic segmentation.

Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy.

In particular, on ImageNet, our ResNet50 and ResNet34 gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity.

We also show that the resulting networks automatically learn to use more features for difficult examples and fewer features for simple examples.

A method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost

A method to train a network with large capacity, only parts of which are used at inference time dependent on input, using fine-grained conditional selection and a new method of regularization, "batch shaping."

Batch-shaping for learning conditional channel gated networks

method train
network
large capacity
conditional


With a view to bridging the gap between deep learning and symbolic AI, we present a novel end-to-end neural network architecture that learns to form propositional representations with an explicitly relational structure from raw pixel data.

In order to evaluate and analyse the architecture, we introduce a family of simple visual relational reasoning tasks of varying complexity.

We show that the proposed architecture, when pre-trained on a curriculum of such tasks, learns to generate reusable representations that better facilitate subsequent learning on previously unseen tasks when compared to a number of baseline architectures.

The workings of a successfully trained model are visualised to shed some light on how the architecture functions.

We present an end-to-end differentiable architecture that learns to map pixels to predicates, and evaluate it on a suite of simple relational reasoning tasks

A network architecture based on the multi-head self-attention module to learn a new form of relational representations, which improves data efficiency and generalization ability on curriculum learning.

An Explicitly Relational Neural Network Architecture


In natural language inference, the semantics of some words do not affect the inference.

Such information is considered superficial and brings overfitting.

How can we represent and discard such superficial information?

In this paper, we use first order logic (FOL) - a classic technique from meaning representation language – to explain what information is superficial for a given sentence pair.

Such explanation also suggests two inductive biases according to its properties.

We proposed a neural network-based approach that utilizes the two inductive biases.

We obtain substantial improvements over extensive experiments.

We use neural networks to project superficial information out for natural language inference by defining and identifying the superficial information from the perspective of first-order logic.

This paper tries to reduce superficial information in natural language inference to prevent overfitting, and introduces a graph neural network to model relation between premise and hypothesis.

An approach to treat natural language inference using first-order logic and to infuse NLI models with logical information to be more robust at inference.

Robust Natural Language Representation Learning for Natural Language Inference by Projecting Superficial Words out

superficial information natural language inference


We propose an approach to training machine learning models that are fair in the sense that their performance is invariant under certain perturbations to the features.

For example, the performance of a resume screening system should be invariant under changes to the name of the applicant.

We formalize this intuitive notion of fairness by connecting it to the original  notion of individual fairness put forth by Dwork et al and show that the proposed approach achieves this notion of fairness.

We also demonstrate the effectiveness of the approach on two machine learning tasks that are susceptible to gender and racial biases.

Algorithm for training individually fair classifier using adversarial robustness

This paper proposes a new definition of algorithmic fairness and an algorithm to provably find an ML model that satisfies the fairness contraint.

Training individually fair ML models with sensitive subspace robustness

train individually fair
ml model
algorithm
robustness


In this paper, we propose a Seed-Augment-Train/Transfer (SAT) framework that contains a synthetic seed image dataset generation procedure for languages with different numeral systems using freely available open font file datasets.

This seed dataset of images is then augmented to create a purely synthetic training dataset, which is in turn used to train a deep neural network and test on held-out real world handwritten digits dataset spanning five Indic scripts, Kannada, Tamil, Gujarati, Malayalam, and Devanagari.

We showcase the efficacy of this approach both qualitatively, by training a Boundary-seeking GAN (BGAN) that generates realistic digit images in the five languages, and also qualitatively by testing a CNN trained on the synthetic data on the real-world datasets.

This establishes not only an interesting nexus between the font-datasets-world and transfer learning but also provides a recipe for universal-digit classification in any script.

Is seeding and augmentation all you need for classifying digits in any language?

This paper presents new datasets for five languages and proposes a new framework (SAT) for font image datasets generation for universal digit classification.

A Seed-Augment-Train Framework for Universal Digit Classification

universal digit classification
language
framework


An important research direction in machine learning has centered around developing meta-learning algorithms to tackle few-shot learning.

An especially successful algorithm has been Model Agnostic Meta-Learning (MAML), a method that consists of two optimization loops, with the outer loop finding a meta-initialization, from which the inner loop can efficiently learn new tasks.

Despite MAML's popularity, a fundamental open question remains -- is the effectiveness of MAML due to the meta-initialization being primed for rapid learning (large, efficient changes in the representations) or due to feature reuse,  with the meta initialization already containing high quality features?

We investigate this question, via ablation studies and analysis of the latent representations, finding that feature reuse is the dominant factor.

This leads to the ANIL (Almost No Inner Loop) algorithm, a simplification of MAML where we remove the inner loop for all but the (task-specific) head of the underlying neural network.

ANIL matches MAML's performance on benchmark few-shot image classification and RL and offers computational improvements over MAML.

We further study the precise contributions of the head and body of the network, showing that performance on the test tasks is entirely determined by the quality of the learned features, and we can remove even the head of the network (the NIL algorithm).

We conclude with a discussion of the rapid learning vs feature reuse question for meta-learning algorithms more broadly.

The success of MAML relies on feature reuse from the meta-initialization, which also yields a natural simplification of the algorithm, with the inner loop removed for the network body, as well as other insights on the head and body.

The paper finds that feature reuse is the dominant factor in the success of MAML, and propose new algorithms which spend much less computation than MAML.

Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML


Model training remains a dominant financial cost and time investment in machine learning applications.

Developing and debugging models often involve iterative training, further exacerbating this issue.

With growing interest in increasingly complex models, there is a need for techniques that help to reduce overall training effort.

While incremental training can save substantial time and cost by training an existing model on a small subset of data, little work has explored policies for determining when incremental training provides adequate model performance versus full retraining.

We provide a method-agnostic algorithm for deciding when to incrementally train versus fully train.

We call this setting of non-deterministic full- or incremental training ``Mixed Setting Training".

Upon evaluation in slot-filling tasks, we find that this algorithm provides a bounded error, avoids catastrophic forgetting, and results in a significant speedup over a policy of always fully training.

We provide a method-agnostic algorithm for deciding when to incrementally train versus fully train and it provides a significant speedup over fully training and avoids catastrophic forgetting

This paper proposes an approach for deciding when to incrementally vs. fully retrain a model in the setting of iterative model development in slot filling tasks.

Mixed Setting Training Methods for Incremental Slot-Filling Tasks

decide incrementally
training
fully
task


Neural networks have succeeded in many reasoning tasks.

Empirically, these tasks require specialized network structures, e.g., Graph Neural Networks (GNNs) perform well on many such tasks, while less structured networks fail.

Theoretically, there is limited understanding of why and when a network structure generalizes better than other equally expressive ones.

We develop a framework to characterize which reasoning tasks a network can learn well, by studying how well its structure aligns with the algorithmic structure of the relevant reasoning procedure.

We formally define algorithmic alignment and derive a sample complexity bound that decreases with better alignment.

This framework explains the empirical success of popular reasoning models and suggests their limitations.

We unify seemingly different reasoning tasks, such as intuitive physics, visual question answering, and shortest paths, via the lens of a powerful algorithmic paradigm, dynamic programming (DP).

We show that GNNs can learn DP and thus solve these tasks.

On several reasoning tasks, our theory aligns with empirical results.

We develop a theoretical framework to characterize which reasoning tasks a neural network can learn well.

The paper proposes a measure of classes of algorithmic alignment that measure how "close" neural networks are to known algorithms, proving the link between several classes of known algorithms and neural network architectures.

What Can Neural Networks Reason About?


Cell-cell interactions have an integral role in tumorigenesis as they are critical in governing immune responses.

As such, investigating specific cell-cell interactions has the potential to not only expand upon the understanding of tumorigenesis, but also guide clinical management of patient responses to cancer immunotherapies.

A recent imaging technique for exploring cell-cell interactions, multiplexed ion beam imaging by time-of-flight (MIBI-TOF), allows for cells to be quantified in 36 different protein markers at sub-cellular resolutions in situ as high resolution multiplexed images.

To explore the MIBI images, we propose a GAN for multiplexed data with protein specific attention.

By conditioning image generation on cell types, sizes, and neighborhoods through semantic segmentation maps, we are able to observe how these factors affect cell-cell interactions simultaneously in different protein channels.

Furthermore, we design a set of metrics and offer the first insights towards cell spatial orientations, cell protein expressions, and cell neighborhoods.

Our model, cell-cell interaction GAN (CCIGAN), outperforms or matches existing image synthesis methods on all conventional measures and significantly outperforms on biologically motivated metrics.

To our knowledge, we are the first to systematically model multiple cellular protein behaviors and interactions under simulated conditions through image synthesis.

We explore cell-cell interactions across tumor environment contexts observed in highly multiplexed images, by image synthesis using a novel attention GAN architecture.

A new method to model the data generated by multiplexed ion beam imaging by time-of-flight (MIBI-TOF) by learning the many-to-many mapping between cell types and protein markers' expression levels.

Exploring Cellular Protein Localization Through Semantic Image Synthesis

image synthesis
explore
multiplexed
protein


Machine learning models for question-answering (QA), where given a question and a passage, the learner must select some span in the passage as an answer, are known to be brittle.

By inserting a single nuisance sentence into the passage, an adversary can fool the model into selecting the wrong span.

A promising new approach for QA decomposes the task into two stages:

(i) select relevant sentences from the passage; and

(ii) select a span among those sentences.

Intuitively, if the sentence selector excludes the offending sentence, then the downstream span selector will be robust.

While recent work has hinted at the potential robustness of two-stage QA, these methods have never, to our knowledge, been explicitly combined with adversarial training.

This paper offers a thorough empirical investigation of adversarial robustness, demonstrating that although the two-stage approach lags behind single-stage span selection, adversarial training improves its performance significantly, leading to an improvement of over 22 points in F1 score over the adversarially-trained single-stage model.

A two-stage approach consisting of sentence selection followed by span selection can be made more robust to adversarial attacks in comparison to a single-stage model trained on full context.

This paper investigates an existing model and finds that a two-stage trained QA method is not more robust to adversarial attacks compared to other methods.

On the Effectiveness of Minimal Context Selection for Robust Question Answering

robust adversarial attack
two-stage
selection
model
context
train


The aim of this study is to introduce a formal framework for analysis and synthesis of driver assistance systems.

It applies formal methods to the verification of a stochastic human driver model built using the cognitive architecture ACT-R, and then bootstraps safety in semi-autonomous vehicles through the design of provably correct Advanced Driver Assistance Systems.

The main contributions include the integration of probabilistic ACT-R models in the formal analysis of semi-autonomous systems and an abstraction technique that enables a finite representation of a large dimensional, continuous system in the form of a Markov model.

The effectiveness of the method is illustrated in several case studies under various conditions.

Verification of a human driver model based on a cognitive architecture and synthesis of a correct-by-construction ADAS from it.

Towards Provably Correct Driver Assistance Systems through Stochastic Cognitive Modeling


In contrast to the older writing system of the 19th century, modern Hawaiian orthography employs characters for long vowels and glottal stops.

These extra characters account for about one-third of the phonemes in Hawaiian, so including them makes a big difference to reading comprehension and pronunciation.

However, transliterating between older and newer texts is a laborious task when performed manually.

We introduce two related methods to help solve this transliteration problem automatically, given that there were not enough data to train an end-to-end deep learning model.

One approach is implemented, end-to-end, using finite state transducers (FSTs).

The other is a hybrid deep learning approach which approximately composes an FST with a recurrent neural network (RNN).

We find that the hybrid approach outperforms the end-to-end FST by partitioning the original problem into one part that can be modelled by hand, using an FST, and into another part, which is easily solved by an RNN trained on the available data.

A novel, hybrid deep learning approach provides the best solution to a limited-data problem (which is important to the conservation of the Hawaiian language)

Composing RNNs and FSTs for Small Data: Recovering Missing Characters in Old Hawaiian Text


In many real-world settings, a learning model must perform few-shot classification: learn to classify examples from unseen classes using only a few labeled examples per class.

Additionally, to be safely deployed, it should have the ability to detect out-of-distribution inputs: examples that do not belong to any of the classes.

While both few-shot classification and out-of-distribution detection are popular topics,
their combination has not been studied.

In this work, we propose tasks for out-of-distribution detection in the few-shot setting and establish benchmark datasets, based on four popular few-shot classification datasets.

Then, we propose two new methods for this task and investigate their performance.

In sum, we establish baseline out-of-distribution detection results using standard metrics on new benchmark datasets and show improved results with our proposed methods.

We quantitatively study out-of-distribution detection in few-shot setting, establish baseline results with ProtoNet, MAML, ABML, and improved upon them.

The paper proposes two new confidence scores which are more suitable for out-of-distribution detection of few-shot classification and shows that a distance metric-based approach improves performance.

Out-of-distribution Detection in Few-shot Classification

out-of-distribution detection few-shot classification


While modern  generative  models are able to synthesize high-fidelity, visually appealing images, successfully generating examples that are useful for recognition tasks remains an elusive goal.

To this end, our key insight is that the examples should be synthesized to recover classifier decision boundaries that would be learned from a large amount of real examples.

More concretely, we treat a classifier trained on synthetic examples as ''student'' and a classifier trained on real examples as ''teacher''.

By introducing knowledge distillation into a meta-learning framework, we encourage the generative model to produce examples in a way that enables the student classifier to mimic the behavior of the teacher.

To mitigate the potential gap between student and teacher classifiers, we further propose to distill the knowledge in a progressive manner, either by gradually strengthening the teacher or weakening the student.

We demonstrate the use of our model-agnostic distillation approach to deal with data scarcity, significantly improving few-shot learning performance on miniImageNet and ImageNet1K benchmarks.

This paper introduces progressive knowledge distillation for learning generative models that are recognition task oriented

This paper demonstrates easy-to-hard curriculum learning to train a generative model to improve few-shot classification.

Progressive Knowledge Distillation For Generative Modeling

generative model
progressive knowledge distillation
learn


Deep neural networks provide state-of-the-art performance for many applications of interest.

Unfortunately they are known to be vulnerable to adversarial examples, formed by applying small but malicious perturbations to the original inputs.

Moreover, the perturbations can transfer across models: adversarial examples generated for a specific model will often mislead other unseen models.

Consequently  the adversary can leverage it to attack against the deployed black-box systems.

In this work, we demonstrate that the adversarial perturbation can be decomposed into two components: model-specific and data-dependent one, and it is the latter that mainly contributes to the transferability.

Motivated by this understanding, we propose to craft adversarial examples by utilizing the noise reduced gradient (NRG) which approximates the data-dependent component.

Experiments on various classification models trained on ImageNet demonstrates that the new approach enhances the transferability dramatically.

We also find that low-capacity models have more powerful attack capability than high-capacity counterparts, under the condition that they have comparable test performance.

These insights give rise to a principled manner to construct adversarial examples with high success rates and could potentially provide us guidance for designing effective defense approaches against black-box attacks.

We propose a new method for enhancing the transferability of adversarial examples by using the noise-reduced gradient.

This paper postulates that an adversarial perturbation consists of a model-specific and data-specific component, and that amplification of the latter is best suited for adversarial attacks.

This paper focuses on enhancing the transferability of adversarial examples from one model to another model.

Enhancing the Transferability of Adversarial Examples with Noise Reduced Gradient

enhance transferability adversarial example


We present the iterative two-pass decomposition flow to accelerate existing convolutional neural networks (CNNs).

The proposed rank selection algorithm can effectively determine the proper ranks of the target convolutional layers for the low rank approximation.

Our two-pass CP-decomposition helps prevent from the instability problem.

The iterative flow makes the decomposition of the deeper networks systematic.

The experiment results shows that VGG16 can be accelerated with a 6.2x measured speedup while the accuracy drop remains only 1.2%.

We present the iterative two-pass CP decomposition flow to effectively accelerate existing convolutional neural networks (CNNs).

The paper proposes a novel workflow for acceleration and compression of CNNs and also proposes a way to determine the target rank of each layer given the target overall acceleration.

This paper addresses the problem of learning a low rank tensor filter operation for filtering layers in deep neural networks (DNNs).

Accelerating Convolutional Neural Networks using Iterative Two-Pass Decomposition

convolutional neural network
iterative two-pass
layer
accelerate
decomposition
rank


We introduce LiPopt, a polynomial optimization framework for computing increasingly tighter upper bound on the Lipschitz constant of neural networks.

The underlying optimization problems boil down to either linear (LP) or semidefinite (SDP) programming.

We show how to use the sparse connectivity of a network, to significantly reduce the complexity of computation.

This is specially useful for convolutional as well as pruned neural networks.

We conduct experiments on networks with random weights as well as networks trained on MNIST, showing that in the particular case of the $\ell_\infty$-Lipschitz constant, our approach yields superior estimates as compared to other baselines available in the literature.

LP-based upper bounds on the Lipschitz constant of Neural Networks

The authors study the problem of estimating the Lipschitz constant of a deep neural network with ELO activation function, formulating it as a polynomial optimisation problem.

Lipschitz constant estimation of Neural Networks via sparse polynomial optimization

lipschitz constant
neural network
polynomial


Although few-shot learning research has advanced rapidly with the help of meta-learning, its practical usefulness is still limited because most of the researches assumed that all meta-training and meta-testing examples came from a single domain.

We propose a simple but effective way for few-shot classification in which a task distribution spans multiple domains including previously unseen ones during meta-training.

The key idea is to build a pool of embedding models which have their own metric spaces and to learn to select the best one for a particular task through multi-domain meta-learning.

This simplifies task-specific adaptation over a complex task distribution as a simple selection problem rather than modifying the model with a number of parameters at meta-testing time.

Inspired by common multi-task learning techniques, we let all models in the pool share a base network and add a separate modulator to each model to refine the base network in its own way.

This architecture allows the pool to maintain representational diversity and each model to have domain-invariant representation as well.

Experiments show that our selection scheme outperforms other few-shot classification algorithms when target tasks could come from many different domains.

They also reveal that aggregating outputs from all constituent models is effective for tasks from unseen domains showing the effectiveness of our framework.

We address multi-domain few-shot classification by building multiple models to represent this complex task distribution in a collective way and simplifying task-specific adaptation as a selection problem from these pre-trained models.

This paper tackles few-shot classification with many different domains by building a pool of embedding models to capture domain-invariant and domain-specific features without a significant increase in the number of parameters.

Domain-Agnostic Few-Shot Classification by Learning Disparate Modulators


Still in 2019, many scanned documents come into businesses in non-digital format.

Text to be extracted from real world documents is often nestled inside rich formatting, such as tabular structures or forms with fill-in-the-blank boxes or underlines whose ink often touches or even strikes through the ink of the text itself.

Such ink artifacts can severely interfere with the performance of recognition algorithms or other downstream processing tasks.

In this work, we propose DeepErase, a neural preprocessor to erase ink artifacts from text images.

We devise a method to programmatically augment text images with real artifacts, and use them to train a segmentation network in an weakly supervised manner.

In additional to high segmentation accuracy, we show that our cleansed images achieve a significant boost in downstream recognition accuracy by popular OCR software such as Tesseract 4.0.

We test DeepErase on out-of-distribution datasets (NIST SDB) of scanned IRS tax return forms and achieve double-digit improvements in recognition accuracy over baseline for both printed and handwritten text.

Neural-based removal of document ink artifacts (underlines, smudges, etc.) using no manually annotated training data

DeepErase: Weakly Supervised Ink Artifact Removal in Document Text Images


Black-box adversarial attacks require a large number of attempts before finding successful adversarial examples that are visually indistinguishable from the original input.

Current approaches relying on substitute model training, gradient estimation or genetic algorithms often require an excessive number of queries.

Therefore, they are not suitable for real-world systems where the maximum query number is limited due to cost.

We propose a query-efficient black-box attack which uses Bayesian optimisation in combination with Bayesian model selection to optimise over the adversarial perturbation and the optimal degree of search space dimension reduction.

We demonstrate empirically that our method can achieve comparable success rates with 2-5 times fewer queries compared to previous state-of-the-art black-box attacks.

The authors propose to use Bayesian optimization with a GP surrogate for adversarial image generation, by exploiting additive structure and using Bayesian model selection to determine an optimal dimensionality reduction.

BayesOpt Adversarial Attack

bayesian model selection
adversarial
optimal
reduction
attack


Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information.

Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data:

1) models must learn the complex intra-modal and cross-modal interactions for prediction and

2) models must be robust to unexpected missing or noisy modalities during testing.

In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels.

We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors.

Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction.

Modality-specific generative factors are unique for each modality and contain the information required for generating data.

Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets.

Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance.

Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning.

We propose a model to learn factorized multimodal representations that are discriminative, generative, and interpretable.

This paper presents 'Multimodal Factorization model' that factorizes representations into shared multimodal discriminative factors and modality specific generative factors.

Learning Factorized Multimodal Representations

learn factorize multimodal representation


The successful application of flexible, general learning algorithms to real-world robotics applications is often limited by their poor data-efficiency.

To address the challenge, domains with more than one dominant task of interest encourage the sharing of information across tasks to limit required experiment time.

To this end, we investigate compositional inductive biases in the form of hierarchical policies as a mechanism for knowledge transfer across tasks in reinforcement learning (RL).

We demonstrate that this type of hierarchy enables positive transfer while mitigating negative interference.

Furthermore, we demonstrate the benefits of additional incentives to efficiently decompose task solutions.

Our experiments show that these incentives are naturally given in multitask learning and can be easily introduced for single objectives.

We design an RL algorithm that enables stable and fast learning of structured policies and the effective reuse of both behavior components and transition data across tasks in an off-policy setting.

Finally, we evaluate our algorithm in simulated environments as well as physical robot experiments and  demonstrate substantial improvements in data data-efficiency over competitive baselines.

We develop a hierarchical, actor-critic algorithm for compositional transfer by sharing policy components and demonstrate component specialization and related direct benefits in multitask domains as well as its adaptation for single tasks.

A combination of different learning techniques for acquiring structure and learning with asymmetric data, used to train an HRL policy.

The authors introduce a hierarchical policy structure for use in both single task and multitask reinforcement learning, and assess the structure's usefulness on complex robotic tasks.

Compositional Transfer in Hierarchical Reinforcement Learning

single task
compositional transfer
multitask
hierarchical
structure
reinforcement learning
policy


In this paper, we study the representational power of deep neural networks (DNN) that belong to the family of piecewise-linear (PWL) functions, based on PWL activation units such as rectifier or maxout.

We investigate the complexity of such networks by studying the number of linear regions of the PWL function.

Typically, a PWL function from a DNN can be seen as a large family of linear functions acting on millions of such regions.

We directly build upon the work of Mont´ufar et al. (2014), Mont´ufar (2017), and Raghu et al. (2017) by refining the upper and lower bounds on the number of linear regions for rectified and maxout networks.

In addition to achieving tighter bounds, we also develop a novel method to perform exact numeration or counting of the number of linear regions with a mixed-integer linear formulation that maps the input space to output.

We use this new capability to visualize how the number of linear regions change while training DNNs.

We empirically count the number of linear regions of rectifier networks and refine upper and lower bounds.

This paper presents improved bounds for counting the number of linear regions in ReLU networks.

Bounding and Counting Linear Regions of Deep Neural Networks

count number linear region
bound
network


Convolutional neural networks memorize part of their training data, which is why strategies such as data augmentation and drop-out are employed to mitigate over- fitting.

This paper considers the related question of “membership inference”, where the goal is to determine if an image was used during training.

We con- sider membership tests over either ensembles of samples or individual samples.

First, we show how to detect if a dataset was used to train a model, and in particular whether some validation images were used at train time.

Then, we introduce a new approach to infer membership when a few of the top layers are not available or have been fine-tuned, and show that lower layers still carry information about the training samples.

To support our findings, we conduct large-scale experiments on Imagenet and subsets of YFCC-100M with modern architectures such as VGG and Resnet.

We analyze the memorization properties by a convnet of the training set and propose several use-cases where we can extract some information about the training set.

Illuminates the generalization/memorization properties of large and deep ConvNets and tries to develop procedures related to identifying whether an input to a trained ConvNet has actually been used to train the network.

Déjà Vu: An Empirical Evaluation of the Memorization Properties of Convnets


While Generative Adversarial Networks (GANs) have empirically produced impressive results on learning complex real-world distributions, recent works have shown that they suffer from lack of diversity or mode collapse.

The theoretical work of Arora et al. (2017a) suggests a dilemma about GANs’ statistical properties: powerful discriminators cause overfitting, whereas weak discriminators cannot detect mode collapse.

By contrast, we show in this paper that GANs can in principle learn distributions in Wasserstein distance (or KL-divergence in many cases) with polynomial sample complexity, if the discriminator class has strong distinguishing power against the particular generator class (instead of against all possible generators).

For various generator classes such as mixture of Gaussians, exponential families, and invertible and injective neural networks generators, we design corresponding discriminators (which are often neural nets of specific architectures) such that the Integral Probability Metric (IPM) induced by the discriminators can provably approximate the Wasserstein distance and/or KL-divergence.

This implies that if the training is successful, then the learned distribution is close to the true distribution in Wasserstein distance or KL divergence, and thus cannot drop modes.

Our preliminary experiments show that on synthetic datasets the test IPM is well correlated with KL divergence or the Wasserstein distance, indicating that the lack of diversity in GANs may be caused by the sub-optimality in optimization instead of statistical inefficiency.

GANs can in principle learn distributions sample-efficiently, if the discriminator class is compact and has strong distinguishing power against the particular generator class.

Proposes the notion of restricted approximability, and provides a sample complexity bound, polynomial in the dimension, which is useful in investigating lack of diversity in GANs.

Analyzes that the Integral Probability Metric can be a good approximation of Wasserstein distance under some mild assumptions.

Approximability of Discriminators Implies Diversity in GANs

gan discriminator approximability diversity


Understanding the optimization trajectory is critical to understand training of deep neural networks.

We show how the hyperparameters of stochastic gradient descent influence the covariance of the gradients (K) and the Hessian of the training loss (H) along this trajectory.

Based on a theoretical model, we predict that using a high learning rate or a small batch size in the early phase of training leads SGD to regions of the parameter space with (1) reduced spectral norm of K, and (2) improved conditioning of K and H. We show that the point on the trajectory after which these effects hold, which we refer to as the break-even point, is reached early during training.

We demonstrate these effects empirically for a range of deep neural networks applied to multiple different tasks.

Finally, we apply our analysis to networks with batch normalization (BN) layers and find that it is necessary to use a high learning rate to achieve loss smoothing effects attributed previously to BN alone.

In the early phase of training of deep neural networks there exists a "break-even point" which determines properties of the entire optimization trajectory.

This work analyzes the optimization of deep neural networks by considering how the batch size and step-size hyper-parameters modify learning trajectories.

The Break-Even Point on the Optimization Trajectories of Deep Neural Networks

deep neural network
optimization trajectory
break-even point


Graph Convolution Network (GCN) has been recognized as one of the most effective graph models for semi-supervised learning, but it extracts merely the first-order or few-order neighborhood information through information propagation, which suffers performance drop-off for deeper structure.

Existing approaches that deal with the higher-order neighbors tend to take advantage of adjacency matrix power.

In this paper, we assume a seemly trivial condition that the higher-order neighborhood information may be similar to that of the first-order neighbors.

Accordingly, we present an unsupervised approach to describe such similarities and learn the weight matrices of higher-order neighbors automatically through Lasso that minimizes the feature loss between the first-order and higher-order neighbors, based on which we formulate the new convolutional filter for GCN to learn the better node representations.

Our model, called higher-order weighted GCN (HWGCN), has achieved the state-of-the-art results on a number of node classification tasks over Cora, Citeseer and Pubmed datasets.

We propose HWGCN to mix the relevant neighborhood information at different orders to better learn node representations.

The authors propose a variant of GCN, HWGCN, to consider convolution beyond 1-step neighbors, which is comparable to state-of-the-art methods.

Higher-order Weighted Graph Convolutional Networks


The performance of deep neural networks is often attributed to their automated, task-related feature construction.

It remains an open question, though, why this leads to solutions with good generalization, even in cases where the number of parameters is larger than the number of samples.

Back in the 90s, Hochreiter and Schmidhuber observed that flatness of the loss surface around a local minimum correlates with low generalization error.

For several flatness measures, this correlation has been empirically validated.

However, it has recently been shown that existing measures of flatness cannot theoretically be related to generalization: if a network uses ReLU activations, the network function can be reparameterized without changing its output in such a way that flatness is changed almost arbitrarily.

This paper proposes a natural modification of existing flatness measures that results in invariance to reparameterization.

The proposed measures imply a robustness of the network to changes in the input and the hidden layers.

Connecting this feature robustness to generalization leads to a generalized definition of the representativeness of data.

With this, the generalization error of a model trained on representative data can be bounded by its feature robustness which depends on our novel flatness measure.

We introduce a novel measure of flatness at local minima of the loss surface of deep neural networks which is invariant with respect to layer-wise reparameterizations and we connect flatness to feature robustness and generalization.

The authors propose a notion of feature robustness which is invariant with respect to rescaling the weight and discuss the notion's relationship to generalization.

This paper defines a notion of feature-robustness and combines it with epsilon representativeness of a function to describe a connection between flatness of minima and generalization in deep neural networks.

Feature-Robustness, Flatness and Generalization Error for Deep Neural Networks

invariant respect
deep neural network
feature robustness
flatness
minima
generalization
notion
feature-robustness


Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e.

We apply and further develop this approach for gated recurrent architectures.

Specifically, in addition to sparsification of individual weights and neurons, we propose to sparsify preactivations of gates and information flow in LSTM.

It makes some gates and information flow components constant, speeds up forward pass and improves compression.

Moreover, the resulting structure of gate sparsity is interpretable and depends on the task.

We propose to sparsify preactivations of gates and information flow in LSTM to make them constant and boost the neuron sparsity level

This paper proposed a sparsification method for recurrent neural networks by eliminating neurons with zero preactivations to obtain compact networks.

Bayesian Sparsification of Gated Recurrent Neural Networks

recurrent neural network
gate
neuron
preactivation
sparsification


Improving the accuracy of numerical methods remains a central challenge in many disciplines and is especially important for nonlinear simulation problems.

A representative example of such problems is fluid flow, which has been thoroughly studied to arrive at efficient simulations of complex flow phenomena.

This paper presents a data-driven approach that learns to improve the accuracy of numerical solvers.

The proposed method utilizes an advanced numerical scheme with a fine simulation resolution to acquire reference data.

We, then, employ a neural network that infers a correction to move a coarse thus quickly obtainable result closer to the reference data.

We provide insights into the targeted learning problem with different learning approaches: fully supervised learning methods with a naive and an optimized data acquisition as well as an unsupervised learning method with a differentiable Navier-Stokes solver.

While our approach is very general and applicable to arbitrary partial differential equation models, we specifically highlight gains in accuracy for fluid flow simulations.

We introduce a neural network approach to assist partial differential equation solvers.

The authors aim at improving the accuracy of numerical solvers by training a neural network on simulated reference data which corrects the numerical solver.

Learning Time-Aware Assistance Functions for Numerical Fluid Solvers


A patient’s health information is generally fragmented across silos.

Though it is technically feasible to unite data for analysis in a manner that underpins a rapid learning healthcare system, privacy concerns and regulatory barriers limit data centralization.

Machine learning can be conducted in a federated manner on patient datasets with the same set of variables, but separated across sites of care.

But federated learning cannot handle the situation where different data types for a given

patient are separated vertically across different organizations.

We call methods that enable machine learning model training on data separated by two or more degrees “confederated machine learning.”

We built and evaluated a confederated machine
learning

model to stratify the risk of accidental falls among the elderly.

a confederated learning method that train model from horizontally and vertically separated medical data

A "confederated" machine learning method that learns across divides in medical data separated both horizontally and vertically.

CONFEDERATED MACHINE LEARNING ON HORIZONTALLY AND VERTICALLY SEPARATED MEDICAL DATA FOR LARGE-SCALE HEALTH SYSTEM INTELLIGENCE

horizontally vertically separate medical datum


Existing neural networks are vulnerable to "adversarial examples"---created by adding maliciously designed small perturbations in inputs to induce a misclassification by the networks.

The most investigated defense strategy is adversarial training which augments training data with adversarial examples.

However, applying single-step adversaries in adversarial training does not support the robustness of the networks, instead, they will even make the networks to be overfitted.

In contrast to the single-step, multi-step training results in the state-of-the-art performance on MNIST and CIFAR10, yet it needs a massive amount of time.

Therefore, we propose a method, Stochastic Quantized Activation (SQA) that solves overfitting problems in single-step adversarial training and fastly achieves the robustness comparable to the multi-step.

SQA attenuates the adversarial effects by providing random selectivity to activation functions and allows the network to learn robustness with only single-step training.

Throughout the experiment, our method demonstrates the state-of-the-art robustness against one of the strongest white-box attacks as PGD training, but with much less computational cost.

Finally, we visualize the learning process of the network with SQA to handle strong adversaries, which is different from existing methods.

This paper proposes Stochastic Quantized Activation that solves overfitting problems in FGSM adversarial training and fastly achieves the robustness comparable to multi-step training.

The paper proposes a model to improve adversarial training by introducing random perturbations in the activations of one of the hidden layers

Stochastic Quantized Activation: To prevent Overfitting in Fast Adversarial Training

stochastic quantize activation
adversarial training


Neural activity is highly variable in response to repeated stimuli.

We used an open dataset, the Allen Brain Observatory, to quantify the distribution of responses to repeated natural movie presentations.

A large fraction of responses are best fit by log-normal distributions or Gaussian mixtures with two components.

These distributions are similar to those from units in deep neural networks with dropout.

Using a separate set of electrophysiological recordings, we constructed a population coupling model as a control for state-dependent activity fluctuations and found that the model residuals also show non-Gaussian distributions.

We then analyzed responses across trials from multiple sections of different movie clips and observed that the noise in cortex aligns better with in-clip versus out-of-clip stimulus variations.

We argue that noise is useful for generalization when it moves along representations of different exemplars in-class, similar to the structure of cortical noise.

We study the structure of noise in the brain and find it may help generalization by moving representations along in-class stimulus variations.

Does the neuronal noise in cortex help generalization?


Unsupervised domain adaptation has received significant attention in recent years.

Most of existing works tackle the closed-set scenario, assuming that the source and target domains share the exactly same categories.

In practice, nevertheless, a target domain often contains samples of classes unseen in source domain (i.e., unknown class).

The extension of domain adaptation from closed-set to such open-set situation is not trivial since the target samples in unknown class are not expected to align with the source.

In this paper, we address this problem by augmenting the state-of-the-art domain adaptation technique, Self-Ensembling, with category-agnostic clusters in target domain.

Specifically, we present Self-Ensembling with Category-agnostic Clusters (SE-CC) --- a novel architecture that steers domain adaptation with the additional guidance of category-agnostic clusters that are specific to target domain.

These clustering information provides domain-specific visual cues, facilitating the generalization of Self-Ensembling for both closed-set and open-set scenarios.

Technically, clustering is firstly performed over all the unlabeled target samples to obtain the category-agnostic clusters, which reveal the underlying data space structure peculiar to target domain.

A clustering branch is capitalized on to ensure that the learnt representation preserves such underlying structure by matching the estimated assignment distribution over clusters to the inherent cluster distribution for each target sample.

Furthermore, SE-CC enhances the learnt representation with mutual information maximization.

Extensive experiments are conducted on Office and VisDA datasets for both open-set and closed-set domain adaptation, and superior results are reported when comparing to the state-of-the-art approaches.

We present a new design, i.e., Self-Ensembling with Category-agnostic Clusters, for both closed-set and open-set domain adaptation.

A new approach to open set domain adaptation, where the source domain categories are contained in the target domain categories in order to filter out outlier categories and enable adaptation within the shared classes.

Open-Set Domain Adaptation with Category-Agnostic Clusters

open-set domain adaptation
category-agnostic cluster
new


We present Spectral Inference Networks, a framework for learning eigenfunctions of linear operators by stochastic optimization.

Spectral Inference Networks generalize Slow Feature Analysis to generic symmetric operators, and are closely related to Variational Monte Carlo methods from computational physics.

As such, they can be a powerful tool for unsupervised representation learning from video or graph-structured data.

We cast training Spectral Inference Networks as a bilevel optimization problem, which allows for online learning of multiple eigenfunctions.

We show results of training Spectral Inference Networks on problems in quantum mechanics and feature learning for videos on synthetic datasets.

Our results demonstrate that Spectral Inference Networks accurately recover eigenfunctions of linear operators and can discover interpretable representations from video in a fully unsupervised manner.

We show how to learn spectral decompositions of linear operators with deep learning, and use it for unsupervised learning without a generative model.

The authors propose to use a deep learning framework to solve the computation of the largest eigenvectors.

This paper presents a framework to learn eigenfunctions via a stochastic process and proposes to tackle the challenge of computing eigenfunctions in a large-scale context by approximating then using a two-phase stochastic optimization process.

Spectral Inference Networks: Unifying Deep and Spectral Learning


The Tensor-Train factorization (TTF) is an efficient way to compress large weight matrices of fully-connected layers and recurrent layers in recurrent neural networks (RNNs).

However, high Tensor-Train ranks for all the core tensors of parameters need to be element-wise fixed, which results in an unnecessary redundancy of model parameters.

This work applies Riemannian stochastic gradient descent (RSGD) to train core tensors of parameters in the Riemannian Manifold before finding vectors of lower Tensor-Train ranks for parameters.

The paper first presents the RSGD algorithm with a convergence analysis and then tests it on more advanced Tensor-Train RNNs such as bi-directional GRU/LSTM and Encoder-Decoder RNNs with a Tensor-Train attention model.

The experiments on digit recognition and machine translation tasks suggest the effectiveness of the RSGD algorithm for Tensor-Train RNNs.

Applying the Riemannian SGD (RSGD) algorithm for training Tensor-Train RNNs to further reduce model parameters.

The paper proposes to use Riemannian stochastic gradient algorithm for low-rank tensor train learning in deep networks.

Proposes an algorithm for optimizing neural networks parametrized by Tensor Train decomposition based on the Riemannian optimization and rank adaptation, and designs a bidirectional TT LSTM architecture.

Riemannian Stochastic Gradient Descent for Tensor-Train Recurrent Neural Networks

algorithm
riemannian stochastic gradient
tensor train
neural network
tensor-train


In this paper, we consider the problem of learning control policies that optimize areward function while satisfying constraints due to considerations of safety, fairness, or other costs.

We propose a new algorithm - Projection Based ConstrainedPolicy Optimization (PCPO), an iterative method for optimizing policies in a two-step process - the first step performs an unconstrained update while the secondstep reconciles the constraint violation by projection the policy back onto the constraint set.

We theoretically analyze PCPO and provide a lower bound on rewardimprovement, as well as an upper bound on constraint violation for each policy update.

We further characterize the convergence of PCPO with projection basedon two different metrics - L2 norm and Kullback-Leibler divergence.

Our empirical results over several control tasks demonstrate that our algorithm achievessuperior performance, averaging more than 3.5 times less constraint violation andaround 15% higher reward compared to state-of-the-art methods.

We propose a new algorithm that learns constraint-satisfying policies, and provide theoretical analysis and empirical demonstration in the context of reinforcement learning with constraints.

This paper introduces a constrained policy optimization algorithm using a two-step optimization process, where policies that do not satisfy the constraint can be projected back into the constraint set.

Projection Based Constrained Policy Optimization

constraint policy algorithm optimization constrained


Deep networks face challenges of ensuring their robustness against inputs that cannot be effectively represented by information learned from training data.

We attribute this vulnerability to the limitations inherent to activation-based representation.

To complement the learned information from activation-based representation, we propose utilizing a gradient-based representation that explicitly focuses on missing information.

In addition, we propose a directional constraint on the gradients as an objective during training to improve the characterization of missing information.

To validate the effectiveness of the proposed approach, we compare the anomaly detection performance of gradient-based and activation-based representations.

We show that the gradient-based representation outperforms the activation-based representation by 0.093 in CIFAR-10 and 0.361 in CURE-TSR datasets in terms of AUROC averaged over all classes.

Also, we propose an anomaly detection algorithm that uses the gradient-based representation, denoted as GradCon, and validate its performance on three benchmarking datasets.

The proposed method outperforms the majority of the state-of-the-art algorithms in CIFAR-10, MNIST, and fMNIST datasets with an average AUROC of 0.664, 0.973, and 0.934, respectively.

We propose a gradient-based representation for characterizing information that deep networks have not learned.

The authors present creating representations based on gradients with respect to the weights to supplement information missing from the training dataset for deep networks.

Characterizing Missing Information in Deep Networks Using Backpropagated Gradients

information deep network
characterize
representation
gradient


Medical images may contain various types of artifacts with different patterns and mixtures, which depend on many factors such as scan setting, machine condition, patients’ characteristics, surrounding environment, etc.

However, existing deep learning based artifact reduction methods are restricted by their training set with specific predetermined artifact type and pattern.

As such, they have limited clinical adoption.

In this paper, we introduce a “Zero-Shot” medical image Artifact Reduction (ZSAR) framework, which leverages the power of deep learning but without using general pre-trained networks or any clean image reference.

Specifically, we utilize the low internal visual entropy of an image and train a light-weight image-specific artifact reduction network to reduce artifacts in an image at test-time.

We use Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) as vehicles to show that ZSAR can reduce artifacts better than state-of-the-art both qualitatively and quantitatively, while using shorter execution time.

To the best of our knowledge, this is the first deep learning framework that reduces artifacts in medical images without using a priori training set.

We introduce a “Zero-Shot” medical image Artifact Reduction framework, which leverages the power of deep learning but without using general pre-trained networks or any clean image reference.

Zero-Shot Medical Image Artifact Reduction

medical image artifact reduction
zero-shot


Attribution methods provide insights into the decision-making of machine learning models like artificial neural networks.

For a given input sample, they assign a relevance score to each individual input variable, such as the pixels of an image.

In this work we adapt the information bottleneck concept for attribution.

By adding noise to intermediate feature maps we restrict the flow of information and can quantify (in bits) how much information image regions provide.

We compare our method against ten baselines using three different metrics on VGG-16 and ResNet-50, and find that our methods outperform all baselines in five out of six settings.

The method’s information-theoretic foundation provides an absolute frame of reference for attribution values (bits) and a guarantee that regions scored close to zero are not necessary for the network's decision.

We apply the informational bottleneck concept to attribution.

The paper proposes a novel perturbation-based method for computing attribution/saliency maps for deep neural network based image classifiers, by injecting crafted noise into an early layer of the network.

Restricting the Flow: Information Bottlenecks for Attribution


Recurrent Neural Networks (RNNs) are used in state-of-the-art models in domains such as speech recognition, machine translation, and language modelling.

Sparsity is a technique to reduce compute and memory requirements of deep learning models.

Sparse RNNs are easier to deploy on devices and high-end server processors.

Even though sparse operations need less compute and memory relative to their dense counterparts, the speed-up observed by using sparse operations is less than expected on different hardware platforms.

In order to address this issue, we investigate two different approaches to induce block sparsity in RNNs: pruning blocks of weights in a layer and using group lasso regularization with pruning to create blocks of weights with zeros.

Using these techniques, we can create block-sparse RNNs with sparsity ranging from 80% to 90% with a small loss in accuracy.

This technique allows us to reduce the model size by roughly 10x.

Additionally, we can prune a larger dense network to recover this loss in accuracy while maintaining high block sparsity and reducing the overall parameter count.

Our technique works with a variety of block sizes up to 32x32.

Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.

We show the RNNs can be pruned to induce block sparsity which improves speedup for sparse operations on existing hardware

The authors propose a block sparsity pruning approach to compress RNNs, using group LASSO to promote sparsity and to prune, but with a very specialized schedule as to the pruning and pruning weight.

Block-Sparse Recurrent Neural Networks


Value iteration networks are an approximation of the value iteration (VI) algorithm implemented with convolutional neural networks to make VI fully differentiable.

In this work, we study these networks in the context of robot motion planning, with a focus on applications to planetary rovers.

The key challenging task in learning-based motion planning is to learn a transformation from terrain observations to a suitable navigation reward function.

In order to deal with complex terrain observations and policy learning, we propose a value iteration recurrence, referred to as the soft value iteration network (SVIN).

SVIN is designed to produce more effective training gradients through the value iteration network.

It relies on a soft policy model, where the policy is represented with a probability distribution over all possible actions, rather than a deterministic policy that returns only the best action.

We demonstrate the effectiveness of the proposed method in robot motion planning scenarios.

In particular, we study the application of SVIN to very challenging problems in planetary rover navigation and present early training results on data gathered by the  Curiosity rover that is currently operating on Mars.

We propose an improvement to value iteration networks, with applications to planetary rover path planning.

This paper learns a reward function based on expert trajectories using a Value Iteration Module to make the planning step differentiable

Soft Value Iteration Networks for Planetary Rover Path Planning

planetary rover path planning
value iteration network


Transformer networks have lead to important progress in language modeling and machine translation.

These models include two consecutive modules, a feed-forward layer and a self-attention layer.

The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers.

Building upon this intuition, we propose a new model that solely consists of attention layers.

More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer.

Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer.

Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.

A novel attention layer that combines self-attention and feed-forward sublayers of Transformer networks.

This paper proposes a modification to the Transformer model by incorporating attention over "persistent" memory vectors into the self-attention layer, resulting in performance on par with existing models while using fewer parameters.

Augmenting Self-attention with Persistent Memory

layer transformer self-attention attention persistent memory


This work views neural networks as data generating systems and applies anomalous pattern detection techniques on that data in order to detect when a network is processing a group of anomalous inputs.

Detecting anomalies is a critical component for multiple machine learning problems including detecting the presence of adversarial noise added to inputs.

More broadly, this work is a step towards giving neural networks the ability to detect groups of out-of-distribution samples.

This work introduces ``Subset Scanning methods from the anomalous pattern detection domain to the task of detecting anomalous inputs to neural networks.

Subset Scanning allows us to answer the question: "``Which subset of inputs have larger-than-expected activations at which subset of nodes?

"  Framing the adversarial detection problem this way allows us to identify systematic patterns in the activation space that span multiple adversarially noised images.

Leveraging this common anomalous pattern, we show increased detection power as the proportion of noised images increases in a test set.

Detection power and accuracy results are provided for targeted adversarial noise added to CIFAR-10 images on a 20-layer ResNet using the Basic Iterative Method attack.

We efficiently find a subset of images that have higher than expected activations for some subset of nodes.  These images appear more anomalous and easier to detect when viewed as a group.

The paper proposed a scheme to detect the presence of anomalous inputs based on a "subset scanning" approach to detect anomalous activations in the deep learning network.

Deep Mining: Detecting Anomalous Patterns in Neural Network Activations with Subset Scanning

detect activation anomalous subset deep network


Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks.

In this work, we conduct a thorough investigation of stable recurrent models.

Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent.

Empirically, we demonstrate stable recurrent models often perform as well as their unstable counterparts on benchmark sequence tasks.

Taken together, these findings shed light on the effective power of recurrent networks and suggest much of sequence learning happens, or can be made to happen, in the stable regime.

Moreover, our results help to explain why in many cases practitioners succeed in replacing recurrent models by feed-forward models.

Stable recurrent models can be approximated by feed-forward networks and empirically perform as well as unstable models on benchmark tasks.

Studies the stability of RNNs and investigation of spectral normalization to sequential predictions.

Stable Recurrent Models