Mixed precision training (MPT) is becoming a practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs.

MPT is typically used in combination with a technique called loss scaling, that works by scaling up the loss value up before the start of backpropagation in order to minimize the impact of numerical underflow on training.

Unfortunately, existing methods make this loss scale value a hyperparameter that needs to be tuned per-model, and a single scale cannot be adapted to different layers at different training stages.

We introduce a loss scaling-based training method called adaptive loss scaling that makes MPT easier and more practical to use, by removing the need to tune a model-specific loss scale hyperparameter.

We achieve this by introducing layer-wise loss scale values which are automatically computed during training to deal with underflow more effectively than existing methods.

We present experimental results on a variety of networks and tasks that show our approach can shorten the time to convergence and improve accuracy, compared with using the existing state-of-the-art MPT and single-precision floating point.

We devise adaptive loss scaling to improve mixed precision training that surpass the state-of-the-art results.

Proposal for an adaptive loss scaling method during backpropagation for mix precision training where scale rate is decided automatically to reduce the underflow.

The authors propose a method to train models in FP16 precision that adopts a more elaborate way to minimize underflow in every layer simultaneously and automatically.

Adaptive Loss Scaling for Mixed Precision Training

adaptive loss scaling
mixed precision training
automatically
method
underflow


Many real-world problems, e.g. object detection, have outputs that are naturally expressed as sets of entities.

This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors.

We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks.

Specifically, in our formulation we incorporate the permutation as unobservable variable and estimate its distribution during the learning process using alternating optimization.

We demonstrate the validity of this new formulation on two relevant vision problems: object detection, for which our formulation outperforms state-of-the-art detectors such as Faster R-CNN and YOLO, and a complex CAPTCHA test, where we observe that, surprisingly, our set based network acquired the ability of mimicking arithmetics without any rules being coded.

We present a novel approach for learning to predict sets with unknown permutation and cardinality using feed-forward deep neural networks.

A formulation to learn the distribution over unobservable permutation variables based on deep networks for the set prediction problem.

Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks

learn predict set unknown permutation cardinality


Foveation is an important part of human vision, and a number of deep networks have also used foveation.

However, there have been few systematic comparisons between foveating and non-foveating deep networks, and between different variable-resolution downsampling methods.

Here we define several such methods, and compare their performance on ImageNet recognition with a Densenet-121 network.

The best variable-resolution method slightly outperforms uniform downsampling.

Thus in our experiments, foveation does not substantially help or hinder object recognition in deep networks.

We compare object recognition performance on images that are downsampled uniformly and with three different foveation schemes.

Foveated Downsampling Techniques


We explore the concept of co-design in the context of neural network verification.

Specifically, we aim to train deep neural networks that not only are robust to adversarial perturbations but also whose robustness can be verified more easily.

To this end, we identify two properties of network models - weight sparsity and so-called ReLU stability - that turn out to significantly impact the complexity of the corresponding verification task.

We demonstrate that improving weight sparsity alone already enables us to turn computationally intractable verification problems into tractable ones.

Then, improving ReLU stability leads to an additional 4-13x speedup in verification times.

An important feature of our methodology is its "universality," in the sense that it can be used with a broad range of training procedures and verification approaches.

We develop methods to train deep neural models that are both robust to adversarial perturbations and whose robustness is significantly easier to verify.

The paper presents several ways to regularize plain ReLU networks to opimize the adversarial robustness, provable adversarial robustness, and the verification speed.

This paper proposes methods to train robust neural networks that can be verified faster, using pruning methods to encourage weight sparsity and regularization to encourage ReLU stability.

Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability

verify
method train
adversarial
robustness
network
fast
relu stability
neural
robust
verification


Batch Normalization (BatchNorm) has shown to be effective for improving and accelerating the training of deep neural networks.

However, recently it has been shown that it is also vulnerable to adversarial perturbations.

In this work, we aim to investigate the cause of adversarial vulnerability of the BatchNorm.

We hypothesize that the use of different normalization statistics during training and inference (mini-batch statistics for training and moving average of these values at inference) is the main cause of this adversarial vulnerability in the BatchNorm layer.

We empirically proved this by experiments on various neural network architectures and datasets.

Furthermore, we introduce Robust Normalization (RobustNorm) and experimentally show that it is not only resilient to adversarial perturbation but also inherit the benefits of BatchNorm.

Investigation of how BatchNorm causes adversarial vulnerability and how to avoid it.

This paper addresses vulnerability to adversarial perturbations in BatchNorm, and proposes an alternative called RobustNorm, using min-max rescaling instead of normalization.

This paper investigates the reason behind the vulnerability of BatchNorm and proposes Robust Normalization, a normalization method that achieves significantly better results under a variety of attack methods.

Towards an Adversarially Robust Normalization Approach

batchnorm vulnerability normalization adversarial robust


Electronic Health Records (EHR) comprise of longitudinal clinical observations portrayed with sparsity, irregularity, and high-dimensionality which become the major obstacles in drawing reliable downstream outcome.

Despite greatly numbers of imputation methods are being proposed to tackle these issues, most of the existing methods ignore correlated features or temporal dynamics and entirely put aside the uncertainty.

In particular, since the missing values estimates have the risk of being imprecise, it motivates us to pay attention to reliable and less certain information differently.

In this work, we propose a novel variational-recurrent imputation network (V-RIN), which unified imputation and prediction network, by taking into account the correlated features, temporal dynamics, and further utilizing the uncertainty to alleviate the risk of biased missing values estimates.

Specifically, we leverage the deep generative model to estimate the missing values based on the distribution among variables and a recurrent imputation network to exploit the temporal relations in conjunction with utilization of the uncertainty.

We validated the effectiveness of our proposed model with publicly available real-world EHR dataset, PhysioNet Challenge 2012, and compared the results with other state-of-the-art competing methods in the literature.

Our variational-recurrent imputation network (V-RIN) takes into account the correlated features, temporal dynamics, and further utilizes the uncertainty to alleviate the risk of biased missing values estimates.

A missing data imputation network to incorporate correlation, temporal relationships, and data uncertainty for the problem of data sparsity in EHRs, which yields higher AUC on mortality rate classification tasks.

The paper presented a method that combines VAE and uncertainty aware GRU for sequential missing data imputation and outcome prediction.

Uncertainty-aware Variational-Recurrent Imputation Network for Clinical Time Series


Despite the state-of-the-art accuracy of Deep Neural Networks (DNN) in various classification problems, their deployment onto resource constrained edge computing devices remains challenging due to their large size and complexity.

Several recent studies have reported remarkable results in reducing this complexity through quantization of DNN models.

However, these studies usually do not consider the changes in the loss function when performing quantization, nor do they take the different importances of DNN model parameters to the accuracy into account.

We address these issues in this paper by proposing a new method, called adaptive quantization, which simplifies a trained DNN model by finding a unique, optimal precision for each network parameter such that the increase in loss is minimized.

The optimization problem at the core of this method iteratively uses the loss function gradient to determine an error margin for each parameter and assigns it a precision accordingly.

Since this problem uses linear functions, it is computationally cheap and, as we will show, has a closed-form approximate solution.

Experiments on MNIST, CIFAR, and SVHN datasets showed that the proposed method can achieve near or better than state-of-the-art reduction in model size with similar error rates.

Furthermore, it can achieve compressions close to floating-point model compression methods without loss of accuracy.

An adaptive method for fixed-point quantization of neural networks based on theoretical analysis rather than heuristics.

Proposes a method for quantizing neural networks that allow weights to be quantized with different precision depending on their importance, taking into account the loss.

The paper proposes a technique for quantizing the weights of a neural network with bit-depth/precision varying on a per-parameter basis.

Adaptive Quantization of Neural Networks

quantization neural network
quantize
weight
method
adaptive
precision


We study the problem of learning permutation invariant representations that can capture containment relations.

We propose training a model on a novel task: predicting the size of the symmetric difference between pairs of multisets, sets which may contain multiple copies of the same object.

With motivation from fuzzy set theory, we formulate both multiset representations and how to predict symmetric difference sizes given these representations.

We model multiset elements as vectors on the standard simplex and multisets as the summations of such vectors, and we predict symmetric difference as the l1-distance between multiset representations.

We demonstrate that our representations more effectively predict the sizes of symmetric differences than DeepSets-based approaches with unconstrained object representations.

Furthermore, we demonstrate that the model learns meaningful representations, mapping objects of different classes to different standard basis vectors.

Based on fuzzy set theory, we propose a model that given only the sizes of symmetric differences between pairs of multisets, learns representations of such multisets and their elements.

This paper proposes a new task of set learning, predicting the size of the symmetric difference between multisets, and gives a method to solve the task based on fuzzy set theory.

Representation Learning with Multisets

size symmetric difference
base fuzzy set theory
give
multiset
learn
representation


It is important to collect credible training samples $(x,y)$ for building data-intensive learning systems (e.g., a deep learning system).

In the literature, there is a line of studies on eliciting distributional information from self-interested agents who hold a relevant information.

Asking people to report complex distribution $p(x)$, though theoretically viable, is challenging in practice.

This is primarily due to the heavy cognitive loads required for human agents to reason and report this high dimensional information.

Consider the example where we are interested in building an image classifier via first collecting a certain category of  high-dimensional image data.

While classical elicitation results apply to eliciting a complex and generative (and continuous) distribution $p(x)$ for this image data, we are interested in eliciting samples $x_i \sim p(x)$ from agents.

This paper introduces a deep learning aided method to incentivize credible sample contributions from selfish and rational agents.

The challenge to do so is to design an incentive-compatible score function to score each reported sample to induce truthful reports, instead of an arbitrary or even adversarial one.

We show that with accurate estimation of a certain $f$-divergence function we are able to achieve approximate incentive compatibility in eliciting truthful samples.

We then present an efficient estimator with theoretical guarantee via studying the variational forms of $f$-divergence function.

Our work complements the literature of information elicitation via introducing the problem of \emph{sample elicitation}.  We also show a connection between this sample elicitation problem and $f$-GAN, and how this connection can help reconstruct an estimator of the distribution based on collected samples.

This paper proposes a deep learning aided method to elicit credible samples from self-interested agents.

The authors propose a sample elicitation framework for the problem of eliciting credible samples from agents for complex distributions, suggest that deep neural frameworks can be applied in this framework, and connect sample elicitation and f-GAN.

This paper studies the sample elicitation problem, proposing a deep learning approach that relies on the dual expression of the f-divergence which writes as a maximum over a set of functions t.

Credible Sample Elicitation by Deep Learning, for Deep Learning

elicit credible sample
deep learning
problem
agent
elicitation


The celebrated Sequence to Sequence learning (Seq2Seq) technique and its numerous variants achieve excellent performance on many tasks.

However, many machine learning tasks have inputs naturally represented as graphs; existing Seq2Seq models face a significant challenge in achieving accurate conversion from graph form to the appropriate sequence.

To address this challenge, we introduce a general end-to-end graph-to-sequence neural encoder-decoder architecture that maps an input graph to a sequence of vectors and uses an attention-based LSTM method to decode the target sequence from these vectors.

Our method first generates the node and graph embeddings using an improved graph-based neural network with a novel aggregation strategy to incorporate edge direction information in the node embeddings.

We further introduce an attention mechanism that aligns node embeddings and the decoding sequence to better cope with large graphs.

Experimental results on bAbI, Shortest Path, and Natural Language Generation tasks demonstrate that our model achieves state-of-the-art performance and significantly outperforms existing graph neural networks, Seq2Seq, and Tree2Seq models; using the proposed bi-directional node embedding aggregation strategy, the model can converge rapidly to the optimal performance.

A graph2seq architecture that combines a graph encoder mixing GGNN and GCN components with an attentional sequence encoder, and that shows improvements over baselines.

This work proposes an end-to-end graph encoder to sequence decoder models with an attention mechanism in between.

Graph2Seq: Graph to Sequence Learning with Attention-Based Neural Networks

graph sequence learn attention-based neural network


We address the problem of learning to discover 3D parts for objects in unseen categories.

Being able to learn the geometry prior of parts and transfer this prior to unseen categories pose fundamental challenges on data-driven shape segmentation approaches.

Formulated as a contextual bandit problem, we propose a learning-based iterative grouping framework which learns a grouping policy to progressively merge small part proposals into bigger ones in a bottom-up fashion.

At the core of our approach is to restrict the local context for extracting part-level features, which encourages the generalizability to novel categories.

On a recently proposed large-scale fine-grained 3D part dataset, PartNet, we demonstrate that our method can transfer knowledge of parts learned from 3 training categories to 21 unseen testing categories without seeing any annotated samples.

Quantitative comparisons against four strong shape segmentation baselines show that we achieve the state-of-the-art performance.

A zero-shot segmentation framework for 3D object part segmentation. Model the segmentation as a decision-making process and solve as a contextual bandit problem.

A method for segmenting 3D point clouds of objects into component parts, focused on generalizing part groupings to novel object categories unseen during training, that shows strong performance relative to baselines.

This paper proposes a method for part segmentation in object point clouds.

Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories

framework 3d
part segmentation
point cloud
method
unseen
object
category


This paper presents the ballistic graph neural network.

Ballistic graph neural network tackles the weight distribution from a transportation perspective and has many different properties comparing to the traditional graph neural network pipeline.

The ballistic graph neural network does not require to calculate any eigenvalue.

The filters propagate exponentially faster($\sigma^2 \sim T^2$) comparing to traditional graph neural network($\sigma^2 \sim T$).

We use a perturbed coin operator to perturb and optimize the diffusion rate.

Our results show that by selecting the diffusion speed, the network can reach a similar accuracy with fewer parameters.

We also show the perturbed filters act as better representations comparing to pure ballistic ones.

We provide a new perspective of training graph neural network, by adjusting the diffusion rate, the neural network's performance can be improved.

A new perspective on how to collect the correlation between nodes based on diffusion properties.

A new diffusion operation for graph neural networks that does not require eigenvalue calculation and can propagate exponentially faster compared to traditional graph neural networks.

The paper proposes to cope with the speed of diffusion problem by introducing ballistic walk.

Beyond Classical Diffusion: Ballistic Graph Neural Network

graph neural network
diffusion
new
ballistic


In this paper, we propose a \textit{weak supervision} framework for neural ranking tasks based on the data programming paradigm \citep{Ratner2016}, which enables us to leverage multiple weak supervision signals from different sources.

Empirically, we consider two sources of weak supervision signals, unsupervised ranking functions and semantic feature similarities.

We train a BERT-based passage-ranking model (which achieves new state-of-the-art performances on two benchmark datasets with full supervision) in our weak supervision framework.

Without using ground-truth training labels, BERT-PR models outperform BM25 baseline by a large margin on all three datasets and even beat the previous state-of-the-art results with full supervision on two of datasets.

We propose a weak supervision training pipeline based on the data programming framework for ranking tasks, in which we train a BERT-base ranking model and establish the new SOTA.

The authors propose a combination of BERT and the weak supervision framework to tackle the problem of passage ranking, obtaining results better than the fully supervised state-of-the-art.

Passage Ranking with Weak Supervision


We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective.

We demonstrate a very universal Frequency Principle (F-Principle) --- DNNs often fit target functions from low to high frequencies --- on high-dimensional benchmark datasets, such as MNIST/CIFAR10, and deep networks, such as VGG16.

This F-Principle of DNNs is opposite to the learning behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibits faster convergence for higher frequencies, for various scientific computing problems.

With a naive theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions.

The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function.

This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.

In real problems, we found that DNNs often fit target functions from low to high frequencies during the training process.

This paper analyzes the loss of neural networks in the Fourier domain and finds that DNNs tend to learn low-frequency components before high-frequency ones.

The paper studies the training process of NNs through Fourier analysis, concluding that NNs learn low frequency components before high frequency components.

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

training process
find dnn
high frequency
fourier
neural network
component
low
learn
analysis


The problem of accelerating drug discovery relies heavily on automatic tools to optimize precursor molecules to afford them with better biochemical properties.

Our work in this paper substantially extends prior state-of-the-art on graph-to-graph translation methods for molecular optimization.

In particular, we realize coherent multi-resolution representations by interweaving the encoding of substructure components with the atom-level encoding of the original molecular graph.

Moreover, our graph decoder is fully autoregressive, and interleaves each step of adding a new substructure with the process of resolving its attachment to the emerging molecule.

We evaluate our model on multiple molecular optimization tasks and show that our model significantly outperforms previous state-of-the-art baselines.

We propose a multi-resolution, hierarchically coupled encoder-decoder for graph-to-graph translation.

A hierarchical graph-to-graph translation model to generate molecular graphs using chemical substructures as building blocks that is fully autoregressive and learns coherent multi-resolution representations, outperforming previous models.

The authors present a hierarchical graph-to-graph translation method for generating novel organic molecules.

Hierarchical Graph-to-Graph Translation for Molecules

hierarchical graph-to-graph translation
generate
molecule
multi-resolution


Equivariance is a nice property to have as it produces much more parameter efficient neural architectures and preserves the structure of the input through the feature mapping.

Even though some combinations of transformations might never appear (e.g. an upright face with a horizontal nose), current equivariant architectures consider the set of all possible transformations in a transformation group when learning feature representations.

Contrarily, the human visual system is able to attend to the set of relevant transformations occurring in the environment and utilizes this information to assist and improve object recognition.

Based on this observation, we modify conventional equivariant feature mappings such that they are able to attend to the set of co-occurring transformations in data and generalize this notion to act on groups consisting of multiple symmetries.

We show that our proposed co-attentive equivariant neural networks consistently outperform conventional rotation equivariant and rotation & reflection equivariant neural networks on rotated MNIST and CIFAR-10.

We utilize attention to restrict equivariant neural networks to the set or co-occurring transformations in data.

This paper combines attention with group equivariance, specifically looking at the p4m group of rotations, translations, and flips, and derives a form of self-attention that doesn't destroy the equivariance property.

The authors propose a self-attention mechanism for rotation-equivariant neural nets that improves classification performance over regular rotation-equivariant nets.

Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring in Data

equivariant neural network
self-attention
transformation
co-occurre
datum
attention
equivariance


The fast generation and refinement of protein backbones would constitute a major advancement to current methodology for the design and development of de novo proteins.

In this study, we train Generative Adversarial Networks (GANs) to generate fixed-length full-atom protein backbones, with the goal of sampling from the distribution of realistic 3-D backbone fragments.

We represent protein structures by pairwise distances between all backbone atoms, and present a method for directly recovering and refining the corresponding backbone coordinates in a differentiable manner.

We show that interpolations in the latent space of the generator correspond to smooth deformations of the output backbones, and that test set structures not seen by the generator during training exist in its image.

Finally, we perform sequence design, relaxation, and ab initio folding of a subset of generated structures, and show that in some cases we can recover the generated folds after forward-folding.

Together, these results suggest a mechanism for fast protein structure refinement and folding using external energy functions.

We train a GAN to generate and recover full-atom protein backbones , and we show that in select cases we can recover the generated proteins after sequence design and ab initio forward-folding.

A generative model for protein backbone which uses a GAN, autoencoder-like network, and refinement process, and a set of qualitative evaluations suggesting positive results.

This paper presents an end-to-end approach for generating protein backbones using generative adversarial networks.

Fully differentiable full-atom protein backbone generation

protein backbone
gan
generate
network
generative
full-atom


Few-Shot Learning (learning with limited labeled data) aims to overcome the limitations of traditional machine learning approaches which require thousands of labeled examples to train an effective model.

Considered as a hallmark of human intelligence, the community has recently witnessed several contributions on this topic, in particular through meta-learning, where a model learns how to learn an effective model for few-shot learning.

The main idea is to acquire prior knowledge from a set of training tasks, which is then used to perform (few-shot) test tasks.

Most existing work assumes that both training and test tasks are drawn from the same distribution, and a large amount of labeled data is available in the training tasks.

This is a very strong assumption which restricts the usage of meta-learning strategies in the real world where ample training tasks following the same distribution as test tasks may not be available.

In this paper, we propose a novel meta-learning paradigm wherein a few-shot learning model is learnt, which simultaneously overcomes domain shift between the train and test tasks via adversarial domain adaptation.

We demonstrate the efficacy the proposed method through extensive experiments.

Meta Learning for Few Shot learning assumes that training tasks and test tasks are drawn from the same distribution. What do you do if they are not? Meta Learning with task-level Domain Adaptation!

This paper proposes a model combining unsupervised adversarial domain adaptation with prototypical networks that performs better than few-shot learning baselines on few-shot learning tasks with domain shift.

The authors proposed meta domain adaptation to address domain shift scenario in meta learning setup, demonstrating performance improvements in several experiments.

Meta-Learning with Domain Adaptation for Few-Shot Learning under Domain Shift

meta learning
domain adaptation
task
few-shot
shift


Universal probabilistic programming systems (PPSs) provide a powerful framework for specifying rich and complex probabilistic models.

However, this expressiveness comes at the cost of substantially complicating the process of drawing inferences from the model.

In particular, inference can become challenging when the support of the model varies between executions.

Though general-purpose inference engines have been designed to operate in such settings, they are typically inefficient, often relying on proposing from the prior to make transitions.

To address this, we introduce a new inference framework: Divide, Conquer, and Combine (DCC).

DCC divides the program into separate straight-line sub-programs, each of which has a fixed support allowing more powerful inference algorithms to be run locally, before recombining their outputs in a principled fashion.

We show how DCC can be implemented as an automated and general-purpose PPS inference engine, and empirically confirm that it can provide substantial performance improvements over previous approaches.

Divide, Conquer, and Combine is a new inference scheme that can be performed on the probabilistic programs with stochastic support, i.e. the very existence of variables is stochastic.

Divide, Conquer, and Combine: a New Inference Strategy for Probabilistic Programs with Stochastic Support

probabilistic program stochastic support


Detecting communities or the modular structure of real-life networks (e.g. a social

network or a product purchase network) is an important task because the way a

network functions is often determined by its communities.

The traditional approaches to community detection involve modularity-based approaches,

which generally speaking, construct partitions based on heuristics that

seek to maximize the ratio of the edges within the partitions to those between

Node embedding approaches, which represent each node in a graph as a

real-valued vector, transform the problem of community detection in a graph to

Existing node embedding approaches are primarily

based on first initiating uniform random walks from each node to construct

a context of a node and then seeks to make the vector representation of

However, standard node embedding approaches do

not directly take into account the community structure of a network while constructing

To alleviate this, we explore two different

First, we investigate the use of biased random walks (specifically,

maximum entropy based walks) to obtain more centrality preserving embedding

of nodes, which we hypothesize may lead to more effective clusters in the embedded

Second, we propose a community structure aware node embedding

approach where we incorporate modularity-based partitioning heuristics into

the objective function of node embedding.

We demonstrate that our proposed approach

for community detection outperforms a number of modularity-based baselines

as well as K-means on a standard node-embedded vector space (specifically,

node2vec) on a wide range of real-life networks of different sizes and densities.

A community preserving node embedding algorithm that results in more effective detection of communities with a clustering on the embedded space

COMMUNITY PRESERVING NODE EMBEDDING


A point cloud is an agile 3D representation, efficiently modeling an object's surface geometry.

However, these surface-centric properties also pose challenges on designing tools to recognize and synthesize point clouds.

This work presents a novel autoregressive model, PointGrow, which generates realistic point cloud samples from scratch or conditioned from given semantic contexts.

Our model operates recurrently, with each point sampled according to a conditional distribution given its previously-generated points.

Since point cloud object shapes are typically encoded by long-range interpoint dependencies, we augment our model with dedicated self-attention modules to capture these relations.

Extensive evaluation demonstrates that PointGrow achieves satisfying performance on both unconditional and conditional point cloud generation tasks, with respect to fidelity, diversity and semantic preservation.

Further, conditional PointGrow learns a smooth manifold of given images where 3D shape interpolation and arithmetic calculation can be performed inside.

An autoregressive deep learning model for generating diverse point clouds.

An approach for generating 3D shapes as point clouds which considers the lexicographic ordering of points according to coordinates and trains a model to predict points in order.

The paper introduces a generative model for point clouds using a pixel RNN-like auto-regressive model and an attention model to handle longer-range interactions.

PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention


Reinforcement learning and evolutionary algorithms can be used to create sophisticated control solutions.

Unfortunately explaining how these solutions work can be difficult to due to their "black box" nature.

In addition, the time-extended nature of control algorithms often prevent direct applications of explainability techniques used for standard supervised learning algorithms.

This paper attempts to address explainability of blackbox control algorithms through six different techniques:

3) Single time step integrated gradients,

5) Sensitivity analysis combined with temporal modeling with LSTMs, and

These techniques are tested on a simple 2d domain, where a simulated rover attempts to navigate through obstacles to reach a goal.

For control, this rover uses an evolved multi-layer perception that maps an 8d field of obstacle and goal sensors to an action determining where it should go in the next time step.

Results show that some simple insights in explaining the neural network are possible, but that good explanations are difficult.

Describes a series of explainability techniques applied to a simple neural network controller used for navigation.

This paper provides insights and explanations for the problem of providing explanations for a multilayer perceptron used as an inverse controller for rover movement, and ideas on how to explain a black-box model.

Challenges of Explaining Control


The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments.

This challenging task demands that the agent be aware of which instruction was completed, which instruction is needed next, which way to go, and its navigation progress towards the goal.

In this paper, we introduce a self-monitoring agent with two complementary components: (1) visual-textual co-grounding module to locate the instruction completed in the past, the instruction required for the next action, and the next moving direction from surrounding images and (2) progress monitor to ensure the grounded instruction correctly reflects the navigation progress.

We test our self-monitoring agent on a standard benchmark and analyze our proposed approach through a series of ablation studies that elucidate the contributions of the primary components.

Using our proposed method, we set the new state of the art by a significant margin (8% absolute increase in success rate on the unseen test set).

Code is available at https://github.com/chihyaoma/selfmonitoring-agent.

We propose a self-monitoring agent for the Vision-and-Language Navigation task.

A method for vision+language navigation which tracks progress on the instruction using a progress monitor and a visual-textual co-grounding module, and performs well on standard benchmarks.

This paper describes a model for vision-and-language navigation with a panoramic visual attention and an auxillary progress monitoring loss, giving state-of-the-art results.

Self-Monitoring Navigation Agent via Auxiliary Progress Estimation

vision-and-language navigation
progress monitor
self-monitoring
agent


Environments in Reinforcement Learning (RL) are usually only partially observable.

To address this problem, a possible solution is to provide the agent with information about past  observations.

While common methods represent this history using a Recurrent Neural Network (RNN), in this paper we propose an alternative representation which is based on the record of the past events observed in a given episode.

Inspired by the human memory, these events describe only important changes in the environment and, in our approach, are automatically discovered using self-supervision.

We evaluate our history representation method using two challenging RL benchmarks: some games of the Atari-57 suite and the 3D environment Obstacle Tower.

Using these benchmarks we show the advantage of our solution with respect to common RNN-based approaches.

event discovery to represent the history for the agent in RL

The authors study the problem of RL under partially observed settings, and propose a solution that uses a FFNN but provides a history representation, outperforming PPO.

This paper proposes a new way to represent past history as input to an RL agent, showing to perform better than PPO and an RNN variant of PPO.

Event Discovery for History Representation in Reinforcement Learning

history representation
event discovery
rl
agent
represent
ppo


The unconditional generation of high fidelity images is a longstanding benchmark

for testing the performance of image decoders.

have been able to generate small images unconditionally, but the extension of

these methods to large images where fidelity can be more readily assessed has

Among the major challenges are the capacity to encode

the vast previous context and the sheer difficulty of learning a distribution that

preserves both global semantic coherence and exactness of detail.

former challenge, we propose the Subscale Pixel Network (SPN), a conditional

decoder architecture that generates an image as a sequence of image slices of equal

The SPN compactly captures image-wide spatial dependencies and requires a

fraction of the memory and the computation.

propose to use multidimensional upscaling to grow an image in both size and depth

via intermediate stages corresponding to distinct SPNs.

unconditional generation of CelebAHQ of size 256 and of ImageNet from size 32

We achieve state-of-the-art likelihood results in multiple settings, set up

new benchmark results in previously unexplored settings and are able to generate

very high fidelity large scale samples on the basis of both datasets.

We show that autoregressive models can generate high fidelity images.

An architecture utilizing decoder, size-upscaling decoder, and depth-upscaling decoder components to tackle the problem of learning long-range dependencies in images in order to obtain high fidelity images.

This paper addresses the problem of generation to high fidelity images, successfully showing convincing Imagenet samples with 128x128 resolution for a likelihood density model.

GENERATING HIGH FIDELITY IMAGES WITH SUBSCALE PIXEL NETWORKS AND MULTIDIMENSIONAL UPSCALING

generate high fidelity image
problem
model
show


Real-world dynamical systems often consist of multiple stochastic subsystems that interact with each other.

Modeling and forecasting the behavior of such dynamics are generally not easy, due to the inherent hardness in understanding the complicated interactions and evolutions of their constituents.

This paper introduces the relational state-space model (R-SSM), a sequential hierarchical latent variable model that makes use of graph neural networks (GNNs) to simulate the joint state transitions of multiple correlated objects.

By letting GNNs cooperate with SSM, R-SSM provides a flexible way to incorporate relational information into the modeling of multi-object dynamics.

We further suggest augmenting the model with normalizing flows instantiated for vertex-indexed random variables and propose two auxiliary contrastive objectives to facilitate the learning.

The utility of R-SSM is empirically evaluated on synthetic and real time series datasets.

A deep hierarchical state-space model in which the state transitions of correlated objects are coordinated by graph neural networks.

A hierarchical latent variable model of sequential dynamic processes of multiple objects when each object exhibits significant stochasticity.

The paper presents a relational state-space model that simulates the joint state transitions of correlated objects which are hierarchically coordinated in a graph structure.

Relational State-Space Model for Stochastic Multi-Object Systems

state transition correlate object
relational state-space model
graph
hierarchical


Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses).

When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed.

While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents.

This paper proposes to add such inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated.

Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

We introduce a new inductive bias that integrates tree structures in recurrent neural networks.

This paper proposes ON-LSTM, a new RNN unit that integrates the latent tree structure into recurrent models and that has good results on language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

integrate tree structure recurrent neural network


Skip connections made the training of very deep networks possible and have become an indispensable component in a variety of neural architectures.

A completely satisfactory explanation for their success remains elusive.

Here, we present a novel explanation for the benefits of skip connections in training very deep networks.

The difficulty of training deep networks is partly due to the singularities caused by the non-identifiability of the model.

Several such singularities have been identified in previous works:

(i) overlap singularities caused by the permutation symmetry of nodes in a given layer,

(ii) elimination singularities corresponding to the elimination, i.e. consistent deactivation, of nodes,

(iii) singularities generated by the linear dependence of the nodes.

These singularities cause degenerate manifolds in the loss landscape that slow down learning.

We argue that skip connections eliminate these singularities by breaking the permutation symmetry of nodes, by reducing the possibility of node elimination and by making the nodes less linearly dependent.

Moreover, for typical initializations, skip connections move the network away from the "ghosts" of these singularities and sculpt the landscape around them to alleviate the learning slow-down.

These hypotheses are supported by evidence from simplified models, as well as from experiments with deep networks trained on real-world datasets.

Degenerate manifolds arising from the non-identifiability of the model slow down learning in deep networks; skip connections help by breaking degeneracies.

The authors show that elimination singularities and overlap singularities impede learning in deep neural networks, and demonstrate that skip connections can reduce the prevalence of these singularities, speeding up learning.

Paper examines the use of skip connections in deep networks as a way of alleviating singularities in the Hessian matrix during training.

Skip Connections Eliminate Singularities

skip connection
deep network
learning
singularity


Representation learning is a central challenge across a range of machine learning areas.

In reinforcement learning, effective and functional representations have the potential to tremendously accelerate learning progress and solve more challenging problems.

Most prior work on representation learning has focused on generative approaches, learning representations that capture all the underlying factors of variation in the observation space in a more disentangled or well-ordered manner.

In this paper, we instead aim to learn functionally salient representations: representations that are not necessarily complete in terms of capturing all factors of variation in the observation space, but rather aim to capture those factors of variation that are important for decision making -- that are "actionable".

These representations are aware of the dynamics of the environment, and capture only the elements of the observation that are necessary for decision making rather than all factors of variation, eliminating the need for explicit reconstruction.

We show how these learned representations can be useful to improve exploration for sparse reward problems, to enable long horizon hierarchical reinforcement learning, and as a state representation for learning policies for downstream tasks.

We evaluate our method on a number of simulated environments, and compare it to prior methods for representation learning, exploration, and hierarchical reinforcement learning.

Learning state representations which capture factors necessary for control

An approach to representation learning in the context of reinforcement learning that distinguishes two stages functionally in terms of the actions that are needed to reach them.

The paper presents a method to learn representations where proximity in euclidean distance represents states that are achieved by similar policies.

Learning Actionable Representations with Goal Conditioned Policies


We explore the behavior of a standard convolutional neural net in a setting that introduces classification tasks sequentially and requires the net to master new tasks while preserving mastery of previously learned tasks.

This setting corresponds to that which human learners face as they acquire domain expertise, for example, as an individual reads a textbook chapter-by-chapter.

Through simulations involving sequences of 10 related tasks, we find reason for optimism that nets will scale well as they advance from having a single skill to becoming domain experts.

First, forward facilitation---the accelerated learning of task n+1 having learned n previous tasks---grows with n. Second, backward interference---the forgetting of the n previous tasks when learning task n+1---diminishes with n.  Forward facilitation is the goal of research on metalearning, and reduced backward interference is the goal of research on ameliorating catastrophic forgetting.

We find that both of these goals are attained simply through broader exposure to a domain.

We study the behavior of a CNN as it masters new tasks while preserving mastery for previously learned tasks

Scaling characteristics of sequential multitask learning: Networks naturally learn to learn


We demonstrate a low effort method that unsupervisedly constructs task-optimized embeddings from existing word embeddings to gain performance on a supervised end-task.

This avoids additional labeling or building more complex model architectures by instead providing specialized embeddings better fit for the end-task(s).

Furthermore, the method can be used to roughly estimate whether a specific kind of end-task(s) can be learned form, or is represented in, a given unlabeled dataset, e.g. using publicly available probing tasks.

We evaluate our method for diverse word embedding probing tasks and by size of embedding training corpus -- i.e. to explore its use in reduced (pretraining-resource) settings.

Morty refits pretrained word embeddings to either: (a) improve overall embedding performance (for Multi-task settings) or improve Single-task performance, while requiring only minimal effort.

MORTY Embedding: Improved Embeddings without Supervision


Data augmentation is commonly used to encode invariances in learning methods.

However, this process is often performed in an inefficient manner, as artificial examples are created by applying a number of transformations to all points in the training set.

The resulting explosion of the dataset size can be an issue in terms of storage and training costs, as well as in selecting and tuning the optimal set of transformations to apply.

In this work, we demonstrate that it is possible to significantly reduce the number of data points included in data augmentation while realizing the same accuracy and invariance benefits of augmenting the entire dataset.

We propose a novel set of subsampling policies, based on model influence and loss, that can achieve a 90% reduction in augmentation set size while maintaining the accuracy gains of standard data augmentation.

Selectively augmenting difficult to classify points results in efficient training.

The authors study the problem of identifying subsampling strategies for data augmentation and propose strategies based on model influence and loss as well as empirical benchmarking of the proposed methods.

The authors propose to use influence or loss-based methods to select a subset of points to use in augmenting data sets for training models where the loss is additive over data points.

Efficient Augmentation via Data Subsampling

loss training author influence method model point augment efficient augmentation datum subsampling


Over the last few years exciting work in deep generative models has produced models able to suggest new organic molecules by generating strings, trees, and graphs representing their structure.

While such models are able to generate molecules with desirable properties, their utility in practice is limited due to the difficulty in knowing how to synthesize these molecules.

We therefore propose a new molecule generation model, mirroring a more realistic real-world process, where reactants are selected and combined to form more complex molecules.

More specifically, our generative model proposes a bag of initial reactants (selected from a pool of commercially-available molecules) and uses a reaction model to predict how they react together to generate new molecules.

Modeling the entire process of constructing a molecule during generation offers a number of advantages.

First, we show that such a model has the ability to generate a wide, diverse set of valid and unique molecules due to the useful inductive biases of modeling reactions.

Second, modeling synthesis routes rather than final molecules offers practical advantages to chemists who are not only interested in new molecules but also suggestions on stable and safe synthetic routes.

Third, we demonstrate the capabilities of our model to also solve one-step retrosynthesis problems, predicting a set of reactants that can produce a target product.

A deep generative model for organic molecules that first generates reactant building blocks before combining these using a reaction predictor.

A molecular generative model that generates molecules via a two-step process that provides synthesis routes of the generated molecules, allowing users to examine the synthetic accessibility of generated compounds.

Generating Molecules via Chemical Reactions

generative model
generate molecule via
reaction


Deep neural networks are complex non-linear models used as predictive analytics tool and have demonstrated state-of-the-art performance on many classification tasks.

However, they have no inherent capability to recognize when their predictions might go wrong.

There have been several efforts in the recent past to detect natural errors i.e.  misclassified inputs but these mechanisms pose additional energy requirements.

To address this issue, we present a novel post-hoc framework to detect natural errors in an energy efficient way.

We achieve this by appending relevant features based linear classifiers per class referred as Relevant features based Auxiliary Cells (RACs).

The proposed technique makes use of the consensus between RACs appended at few selected hidden layers to distinguish the correctly classified inputs from misclassified inputs.

The combined confidence of RACs is utilized to determine if classification should terminate at an early stage.

We demonstrate the effectiveness of our technique on various image classification datasets such as CIFAR10, CIFAR100 and Tiny-ImageNet.

Our results show that for CIFAR100 dataset trained on VGG16 network, RACs can detect 46% of the misclassified examples along with 12% reduction in energy compared to the baseline network while 69% of the examples are correctly classified.

Improve the robustness and energy efficiency of a deep neural network using the hidden representations.

This paper aims to reduce the misclassifications of deep neural networks in an energy efficient way by adding Relevant feature based Auxiliary Cells after one or more hidden layers to decide whether to end classification early.

Relevant-features based Auxiliary Cells for Robust and Energy Efficient Deep Learning

deep neural network
base auxiliary cell
hidden
energy efficient


Many methods have been developed to represent knowledge graph data, which implicitly exploit low-rank latent structure in the data to encode known information and enable unknown facts to be inferred.

To predict whether a relationship holds between entities, their embeddings are typically compared in the latent space following a relation-specific mapping.

Whilst link prediction has steadily improved, the latent structure, and hence why such models capture semantic information, remains unexplained.

We build on recent theoretical interpretation of word embeddings as a basis to consider an explicit structure for representations of relations between entities.

For identifiable relation types, we are able to predict properties and justify the relative performance of leading knowledge graph representation methods, including their often overlooked ability to make independent predictions.

Understanding the structure of knowledge graph representation using insight from word embeddings.

This paper attempts to understand the latent structure underlying knowledge graph embedding methods, and demonstrates that a model's ability to represent a relation type depends on the model architecture's limitations with respect to relation conditions.

This paper proposes a detailed study on the explainability of link prediction (LP) models by utilizing a recent interpretation of word embeddings to provide a better understanding of LPs' model performance.

On Understanding Knowledge Graph Representation

word embedding
knowledge graph representation
model
understand
structure


Many real-world applications involve multivariate, geo-tagged time series data: at each location, multiple sensors record corresponding measurements.

For example, air quality monitoring system records PM2.5, CO, etc.

The resulting time-series data often has missing values due to device outages or communication errors.

In order to impute the missing values, state-of-the-art methods are built on Recurrent Neural Networks (RNN), which process each time stamp sequentially, prohibiting the direct modeling of the relationship between distant time stamps.

Recently, the self-attention mechanism has been proposed for sequence modeling tasks such as machine translation, significantly outperforming RNN because the relationship between each two time stamps can be modeled explicitly.

In this paper, we are the first to adapt the self-attention mechanism for multivariate, geo-tagged time series data.

In order to jointly capture the self-attention across different dimensions (i.e. time, location and sensor measurements) while keep the size of attention maps reasonable, we propose a novel approach called Cross-Dimensional Self-Attention (CDSA) to process each dimension sequentially, yet in an order-independent manner.

On three real-world datasets, including one our newly collected NYC-traffic dataset, extensive experiments demonstrate the superiority of our approach compared to state-of-the-art methods for both imputation and forecasting tasks.

A novel self-attention mechanism for multivariate, geo-tagged time series imputation.

This paper proposes the problem of applying the transformer network to spatiotemporal data in a compuationally efficient way, and investigates ways of implementing 3D attention.

This paper empirically studies the effectiveness of transformer models for time series data imputation across dimensions of the input.

Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation

multivariate geo-tagged time series imputation


The conversion of scanned documents to digital forms is performed using an Optical Character Recognition (OCR) software.

This work focuses on improving the quality of scanned documents in order to improve the OCR output.

We create an end-to-end document enhancement pipeline which takes in a set of noisy documents and produces clean ones.

Deep neural network based denoising auto-encoders are trained to improve the OCR quality.

We train a blind model that works on different noise levels of scanned text documents.

Results are shown for blurring and watermark noise removal from noisy scanned documents.

We designed and tested a REDNET (ResNet Encoder-Decoder) with 8 skip connections to remove noise from documents, including blurring and watermarks, resulting in a high performance deep network for document image cleanup.

Document Enhancement System Using Auto-encoders


The existence of adversarial examples, or intentional mis-predictions constructed from small changes to correctly predicted examples, is one of the most significant challenges in neural network research today.

Ironically, many new defenses are based on a simple observation - the adversarial inputs themselves are not robust and small perturbations to the attacking input often recover the desired prediction.

While the intuition is somewhat clear, a detailed understanding of this phenomenon is missing from the research literature.

This paper presents a comprehensive experimental analysis of when and why perturbation defenses work and potential mechanisms that could explain their effectiveness (or ineffectiveness) in different settings.

We identify a family of defense techniques and show that both deterministic lossy compression and randomized perturbations to the input lead to similar gains in robustness.

This paper discusses ways of destabilizing a given adversarial attack, what makes adversarial images non-robust, and if it's possible for attackers to use a universal model of perturbations to make their adversarial examples robust against such perturbations.

The paper studies the robustness of adversarial attacks to transformations of their input.

A Perturbation Analysis of Input Transformations for Adversarial Attacks

adversarial attack
robustness
perturbation
input
transformation


There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD.

In this work, we fill in the important, yet ambiguous concept of ‘ease-of-use’ by defining an optimizer’s tunability:  How easy is it to find good hyperparameter configurations using automatic random hyperparameter search?

We propose a practical and universal quantitative measure for optimizer tunability that can form the basis for a fair optimizer benchmark.

Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, we find  that Adam is the most tunable for the majority of problems, especially with a low budget for hyperparameter tuning.

We provide a method to benchmark optimizers that is cognizant to the hyperparameter tuning process.

Introduction of a novel metric to capture the tunability of an optimizer, and a comprehensive empirical comparison of deep learning optimizers under different amounts of hyper-parameter tuning.

This paper introduces a simple measure of tunability that allows to compare optimizers under resource constraints, finding that tuning Adam optimizers' learning rate is easiest to find well-performing hyperparameter configurations.

On the Tunability of Optimizers in Deep Learning

tunability optimizer
deep
hyperparameter
tuning
learning


The phase problem in diffraction physics is one of the oldest inverse problems in all of science.

The central difficulty that any approach to solving this inverse problem must overcome is that half of the information, namely the phase of the diffracted beam, is always missing.

In the context of electron microscopy, the phase problem is generally non-linear and solutions provided by phase-retrieval techniques are known to be poor approximations to the physics of electrons interacting with matter.

Here, we show that a semi-supervised learning approach can effectively solve the phase problem in electron microscopy/scattering.

In particular, we introduce a new Deep Neural Network (DNN), Y-net, which simultaneously learns a reconstruction algorithm via supervised training in addition to learning a physics-based regularization via unsupervised training.

We demonstrate that this constrained, semi-supervised approach is an order of magnitude more data-efficient and accurate than the same model trained in a purely supervised fashion.

In addition, the architecture of the Y-net model provides for a straightforward evaluation of the consistency of the model's prediction during inference and is generally applicable to the phase problem in other settings.

We introduce a semi-supervised deep neural network to approximate the solution of the phase problem in electron microscopy

Y-net: A Physics-constrained and Semi-supervised Learning Approach to the Phase Problem in Computational Electron Imaging


Word embeddings extract semantic features of words from large datasets of text.

Most embedding methods rely on a log-bilinear model to predict the occurrence

replaces their linear parametrization with neural networks.

vocabulary, word2net posits a neural network that takes the context as input and

Further, word2net can use the hierarchical

organization of its word networks to incorporate additional meta-data, such as

syntactic features, into the embedding model.

parameters across word networks to develop an embedding model that includes

We study word2net with two datasets, a collection

of Wikipedia articles and a corpus of U.S. Senate speeches.

found that word2net outperforms popular embedding methods on predicting held-

out words and that sharing parameters based on part of speech further boosts

Qualitatively, word2net learns interpretable semantic representations

and, compared to vector-based methods, better incorporates syntactic information.

Word2net is a novel method for learning neural network representations of words that can use syntactic information to learn better semantic features.

This paper extends SGNS with an architectural change from a bag-of-words model to a feedforward model, and contributes a new form of regularization by tying a subset of layers between different associated networks.

A method to use non-linear combination of context vectors for learning vector representation of words, where the main idea is to replace each word embedding by a neural network.

Word2net: Deep Representations of Language

representation word
method
learn
word2net
neural network


A key goal in neuroscience is to understand brain mechanisms of cognitive functions.

An emerging approach is to study “brain states” dynamics using functional magnetic resonance imaging (fMRI).

So far in the literature, brain states have typically been studied using 30 seconds of fMRI data or more, and it is unclear to which extent brain states can be reliably identified from very short time series.

In this project, we applied graph convolutional networks (GCN) to decode brain activity over short time windows in a task fMRI dataset, i.e. associate a given window of fMRI time series with the task used.

Starting with a populational brain graph with nodes defined by a parcellation of cerebral cortex and the adjacent matrix extracted from functional connectome, GCN takes a short series of fMRI volumes as input, generates high-level domain-specific graph representations, and then predicts the corresponding cognitive state.

We investigated the performance of this GCN "cognitive state annotation" in the Human Connectome Project (HCP) database, which features 21 different experimental conditions spanning seven major cognitive domains, and high temporal resolution task fMRI data.

Using a 10-second window, the 21 cognitive states were identified with an excellent average test accuracy of 89% (chance level 4.8%).

As the HCP task battery was designed to selectively activate a wide range of specialized functional networks, we anticipate the GCN annotation to be applicable as a base model for other transfer learning applications, for instance, adapting to new task domains.

Using a 10s window of fMRI signals, our GCN model identified 21 different task conditions from HCP dataset with a test accuracy of 89%.

Functional Annotation of Human Cognitive States using Graph Convolution Networks


Modern deep neural networks (DNNs) require high memory consumption and large computational loads.

In order to deploy DNN algorithms efficiently on edge or mobile devices, a series of DNN compression algorithms have been explored, including the line of works on factorization methods.

Factorization methods approximate the weight matrix of a DNN layer with multiplication of two or multiple low-rank matrices.

However, it is hard to measure the ranks of DNN layers during the training process.

Previous works mainly induce low-rank through implicit approximations or via costly singular value decomposition (SVD) process on every training step.

The former approach usually induces a high accuracy loss while the latter prevents DNN factorization from efficiently reaching a high compression rate.

In this work, we propose SVD training, which first applies SVD to decompose DNN's layers and then performs training on the full-rank decomposed weights.

To improve the training quality and convergence, we add orthogonality regularization to the singular vectors, which ensure the valid form of SVD and avoid gradient vanishing/exploding.

Low-rank is encouraged by applying sparsity-inducing regularizers on the singular values of each layer.

Singular value pruning is applied at the end to reach a low-rank model.

We empirically show that SVD training can significantly reduce the rank of DNN layers and achieve higher reduction on computation load under the same accuracy, comparing to not only previous factorization methods but also state-of-the-art filter pruning methods.

Efficiently inducing low-rank deep neural networks via SVD training with sparse singular values and orthogonal singular vectors.

This paper introduces an approach to network compression by encouraging the weight matrix in each layer to have a low rank and explicitly factorizing the weight matrices into an SVD-like factorization for treatment as new parameters.

Proposal to parametrize each layer of a deep neural network, before training, with a low-rank matrix decomposition, accordingly replace convolutions with two consecutive convolutions, and then train the decomposed method.

Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification

low-rank deep neural network via
layer
singular value
training
vector
matrix


The recent rise in popularity of few-shot learning algorithms has enabled models to quickly adapt to new tasks based on only a few training samples.

Previous few-shot learning works have mainly focused on classification and reinforcement learning.

In this paper, we propose a few-shot meta-learning system that focuses exclusively on regression tasks.

Our model is based on the idea that the degree of freedom of the unknown function can be significantly reduced if it is represented as a linear combination of a set of appropriate basis functions.

This enables a few labelled samples to approximate the function.

We design a Feature Extractor network to encode basis functions for a task distribution, and a  Weights Generator to generate the weight vector for a novel task.

We show that our model outperforms the current state of the art meta-learning methods in various regression tasks.

We propose a few-shot learning model that is tailored specifically for regression tasks

This paper proposes a novel shot-learning method for small sample regression problems.

A method that learns a regression model with a few samples and outperforms other methods.

Few-Shot Regression via Learned Basis Functions

model regression method few-shot sample learn


Most classification and segmentation datasets assume a closed-world scenario in which predictions are expressed as distribution over a predetermined set of visual classes.

However, such assumption implies unavoidable and often unnoticeable failures in presence of out-of-distribution (OOD) input.

These failures are bound to happen in most real-life applications since current visual ontologies are far from being comprehensive.

We propose to address this issue by discriminative detection

Different from recent approaches, we avoid to bring any decisions by only observing the training dataset of the primary model trained to solve the desired computer vision task.

which discriminates the primary training set from a much larger "background" dataset which approximates the variety of the visual world.

We perform our experiments on high resolution natural images in a dense prediction setup.

We use several road driving datasets as our training distribution, while we approximate the background distribution with the ILSVRC dataset.

We evaluate our approach on WildDash test, which is currently the only public test dataset with out-of-distribution images.

The obtained results show that the proposed approach succeeds to identify out-of-distribution pixels while outperforming previous work by a wide margin.

We present a novel approach for detecting out-of-distribution pixels in semantic segmentation.

This paper addresses out-of-distribution detection for helping the segmentation process, and proposes an approach of training a binary classifier that distinguishes image patches from a known set of classes from those of an unknown.

This paper aims to detect out-of-distribution pixels for semantic segmentation, and this work utilizes data from other domains to detect undetermined classes to model uncertainty better.

Discriminative out-of-distribution detection for semantic segmentation

detect out-of-distribution pixel semantic segmentation


Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices.

Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy.

The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead.

To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs.

However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large.

The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results.

It is difficult for even deep reinforcement learning (DRL) DDPG-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy.

In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer.

Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06%, and decrease the inference energy consumption by 50.69%, while achieving the same inference accuracy.

Accurate, Fast and Automated Kernel-Wise Neural Network Quantization with Mixed Precision using Hierarchical Deep Reinforcement Learning

A method for quantizing neural network weights and activations that uses deep reinforcement learning to select bitwidth for individual kernels in a layer and that achieves better performance, or latency, than prior approaches.

This paper proposes to automatically search quantization schemes for each kernel in the neural network, using hierarchial RL to guide the search.

AutoQ: Automated Kernel-Wise Neural Network Quantization

kernel-wise neural network quantization
deep reinforcement learning
kernel


Recent visual analytics systems make use of multiple machine learning models to better fit the data as opposed to traditional single, pre-defined model systems.

However, while multi-model visual analytic systems can be effective, their added complexity poses usability concerns, as users are required to interact with the parameters of multiple models.

Further, the advent of various model algorithms and associated hyperparameters creates an exhaustive model space to sample models from.

This poses complexity to navigate this model space to find the right model for the data and the task.

In this paper, we present Gaggle, a multi-model visual analytic system that enables users to interactively navigate the model space.

Further translating user interactions into inferences, Gaggle simplifies working with multiple models by automatically finding the best model from the high-dimensional model space to support various user tasks.

Through a qualitative user study, we show how our approach helps users to find a best model for a classification and ranking task.

The study results confirm that Gaggle is intuitive and easy to use, supporting interactive model space navigation and automated model selection without requiring any technical expertise from users.

Gaggle, an interactive visual analytic system to help users interactively navigate model space for classification and ranking tasks.

A new visual analytic system which aims to enable non-expert users to interactively navigate a model space by using a demonstration-based approach.

A visual analytics system that helps novice analysts navigate model space in performing classification and ranking tasks.

Gaggle: Visual Analytics for Model Space Navigation

interactively navigate model space
visual analytic system help
classification ranking task
gaggle
user


Chinese text classification has received more and more attention today.

However, the problem of Chinese text representation still hinders the improvement of Chinese text classification, especially the polyphone and the homophone in social media.

To cope with it effectively, we propose a new structure, the Extractor, based on attention mechanisms and design novel attention networks named Extractor-attention network (EAN).

Unlike most of previous works, EAN uses a combination of a word encoder and a Pinyin character encoder instead of a single encoder.

It improves the capability of Chinese text representation.

Moreover, compared with the hybrid encoder methods, EAN has more complex combination architecture and more reducing parameters structures.

Thus, EAN can take advantage of a large amount of information that comes from multi-inputs and alleviates efficiency issues.

The proposed model achieves the state of the art results on 5 large datasets for Chinese text classification.

We propose a novel attention networks with the hybird encoder to solve the text representation issue of Chinese text classification, especially the language phenomena about pronunciations such as the polyphone and the homophone.

This paper proposes an attention-based model consisting of the word encoder and Pinyin encoder for the Chinese text classification task, and extends the architecture for the Pinyin character encoder.

Proposal for an attention network where both word and pinyin are considered for Chinese representation, with improved results shown in several datasets for text classification.

Extractor-Attention Network: A New Attention Network with Hybrid Encoders for Chinese Text Classification

chinese text classification
attention network
pinyin
word
encoder
representation


Recent advances in learning from demonstrations (LfD) with deep neural networks have enabled learning complex robot skills that involve high dimensional perception such as raw image inputs.

LfD algorithms generally assume learning from single task demonstrations.

In practice, however, it is more efficient for a teacher to demonstrate a multitude of tasks without careful task set up, labeling, and engineering.

Unfortunately in such cases, traditional imitation learning techniques fail to represent the multi-modal nature of the data, and often result in sub-optimal behavior.

In this paper we present an LfD approach for learning multiple modes of behavior from visual data.

Our approach is based on a stochastic deep neural network (SNN), which represents the underlying intention in the demonstration as a stochastic activation in the network.

We present an efficient algorithm for training SNNs, and for learning with vision inputs, we also propose an architecture that associates the intention with a stochastic attention module.

We demonstrate our method on real robot visual object reaching tasks, and show that

it can reliably learn the multiple behavior modes in the demonstration data.

Video results are available at https://vimeo.com/240212286/fd401241b9.

multi-modal imitation learning from unstructured demonstrations using stochastic neural network modeling intention.

A new sampling-based approach for inference in latent variable models that applies to multi-modal imitation learning and works better than deterministic neural networks and stochastic neural networks for a real visual robotics task.

This paper shows how to learn several modalities using imitation learning from visual data using stochastic Neural Networks, and a method for learning from demonstrations where several modalities of the same task are given.

Imitation Learning from Visual Data with Multiple Intentions

stochastic neural network
imitation learn visual datum
demonstration
task
intention
multi-modal


The interpretability of neural networks has become crucial for their applications in real world with respect to the reliability and trustworthiness.

Existing explanation generation methods usually provide important features by scoring their individual contributions to the model prediction and ignore the interactions between features, which eventually provide a bag-of-words representation as explanation.

In natural language processing, this type of explanations is challenging for human user to understand the meaning of an explanation and draw the connection between explanation and model prediction, especially for long texts.

In this work, we focus on detecting the interactions between features, and propose a novel approach to build a hierarchy of explanations based on feature interactions.

The proposed method is evaluated with three neural classifiers, LSTM, CNN, and BERT, on two benchmark text classification datasets.

The generated explanations are assessed by both automatic evaluation measurements and human evaluators.

Experiments show the effectiveness of the proposed method in providing explanations that are both faithful to models, and understandable to humans.

A novel approach to construct hierarchical explanations for text classification by detecting feature interactions.

A novel method for providing explanations for predicitions made by text classifiers that outperforms baselines on word level importance scores, and a new metric, cohesion loss, to evaluate span-level importance.

An interpretation method based on feature interactions and feature importance score as compared to independent feature contributions.

Building Hierarchical Interpretations in Natural Language via Feature Interaction Detection

feature interaction
importance score
text
explanation
novel
interpretation
hierarchical
method


Making deep convolutional neural networks more accurate typically comes at the cost of increased computational and memory resources.

In this paper, we reduce this cost by exploiting the fact that the importance of features computed by convolutional layers is highly input-dependent, and propose feature boosting and suppression (FBS), a new method to predictively amplify salient convolutional channels and skip unimportant ones at run-time.

FBS introduces small auxiliary connections to existing convolutional layers.

In contrast to channel pruning methods which permanently remove channels, it preserves the full network structures and accelerates convolution by dynamically skipping unimportant input and output channels.

FBS-augmented networks are trained with conventional stochastic gradient descent, making it readily available for many state-of-the-art CNNs.

We compare FBS to a range of existing channel pruning and dynamic execution schemes and demonstrate large improvements on ImageNet classification.

Experiments show that FBS can respectively provide 5× and 2× savings in compute on VGG-16 and ResNet-18, both with less than 0.6% top-5 accuracy loss.

We make convolutional layers run faster by dynamically boosting and suppressing channels in feature computation.

A feature boosting and suppression method for dynamic channel pruning that predicts the importance of each channel and then uses an affine function to amplify/suppress channel importance.

Proposal for a channel pruning method for dynamically selecting channels during testing.

Dynamic Channel Pruning: Feature Boosting and Suppression

feature boosting suppression
channel
dynamically
method
suppress
dynamic
pruning


We propose a novel way of reducing the number of parameters in the storage-hungry fully connected layers of a neural network by using pre-defined sparsity, where the majority of connections are absent prior to starting training.

Our results indicate that convolutional neural networks can operate without any loss of accuracy at less than 0.5% classification layer connection density, or less than 5% overall network connection density.

We also investigate the effects of pre-defining the sparsity of networks with only fully connected layers.

Based on our sparsifying technique, we introduce the `scatter' metric to characterize the quality of a particular connection pattern.

As proof of concept, we show results on CIFAR, MNIST and a new dataset on classifying Morse code symbols, which highlights some interesting trends and limits of sparse connection patterns.

Neural networks can be pre-defined to have sparse connectivity without performance degradation.

This paper examines sparse connection patterns in upper layers of convolutional image classification networks, and introduces heuristics for distributing connections among windows/groups and a measure called scatter to construct connectivity masks.

Proposal to reduce the number of parameters learned by a deep network by setting up sparse connection weights in classiication layers, and introduction of a concept of "scatter."

Characterizing Sparse Connectivity Patterns in Neural Networks

network sparse pattern layer scatter neural connectivity connection


Deep neural networks are vulnerable to adversarial examples, which becomes one of the most important problems in the development of deep learning.

While a lot of efforts have been made in recent years, it is of great significance to perform correct and complete evaluations of the adversarial attack and defense algorithms.

In this paper, we establish a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness on image classification tasks.

After briefly reviewing plenty of representative attack and defense methods, we perform large-scale experiments with two robustness curves as the fair-minded evaluation criteria to fully understand the performance of these methods.

Based on the evaluation results, we draw several important findings and provide insights for future research.

We provide a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness of deep learning models.

This paper presents an evaluation of different kinds of classification models under various adversarial attack methods.

A large-scale empirical study comparing different adversarial attack and defense techniques, and use of accuracy vs. perturbation budget and accuracy vs. attack strength curves to evaluate attacks and defenses.

Benchmarking Adversarial Robustness

model
evaluate
adversarial robustness
attack
different


We propose a modification to traditional Artificial Neural Networks (ANNs), which provides the ANNs with new aptitudes motivated by biological neurons.

Biological neurons work far beyond linearly summing up synaptic inputs and then transforming the integrated information.

A biological neuron change firing modes accordingly to peripheral factors (e.g., neuromodulators) as well as intrinsic ones.

Our modification connects a new type of ANN nodes, which mimic the function of biological neuromodulators and are termed modulators, to enable other traditional ANN nodes to adjust their activation sensitivities in run-time based on their input patterns.

In this manner, we enable the slope of the activation function to be context dependent.

This modification produces statistically significant improvements in comparison with traditional ANN nodes in the context of Convolutional Neural Networks and Long Short-Term Memory networks.

We propose a modification to traditional Artificial Neural Networks motivated by the biology of neurons to enable the shape of the activation function to be context dependent.

A method to scale the activations of a layer of neurons in an ANN depending on inputs to that layer that reports improvements above the baselines.

Introduction of an architectural change for basic neurons in a neural network, and the idea to multiply neuron linear combination output by a modulator prior to feeding it into the activation function.

Context Dependent Modulation of Activation Function

activation function
neuron
context dependent
neural network


In this work, we study how the large-scale pretrain-finetune framework changes the behavior of a neural language generator.

We focus on the transformer encoder-decoder model for the open-domain dialogue response generation task.

We find that after standard fine-tuning, the model forgets important language generation skills acquired during large-scale pre-training.

We demonstrate the forgetting phenomenon through a detailed behavior analysis from the perspectives of context sensitivity and knowledge transfer.

Adopting the concept of data mixing, we propose an intuitive fine-tuning strategy named "mix-review''.

We find that mix-review effectively regularize the fine-tuning process, and the forgetting problem is largely alleviated.

Finally, we discuss interesting behavior of the resulting dialogue model and its implications.

We identify the forgetting problem in fine-tuning of pre-trained NLG models, and propose the mix-review strategy to address it.

This paper analyzes the forgetting problem in the pretraining-finetuning framework from the perspective of context sensitivity and knowledge transfer, and proposes a fine-tuning strategy which outperforms the weight decay method.

Study of the forgetting problem in the pretrain-finetune framework, specifically in dialogue response generation tasks, and proposal of a mix-review strategy to alleviate the forgetting issue.

Mix-review: Alleviate Forgetting in the Pretrain-Finetune Framework for Neural Language Generation Models

mix-review strategy
forgetting problem
pretrain-finetune framework
alleviate forget
fine-tuning
model
generation


Combining domain knowledge models with neural models has been challenging.

End-to-end trained neural models often perform better (lower Mean Square Error) than domain knowledge models or domain/neural combinations, and the combination is inefficient to train.

In this paper, we demonstrate that by composing domain models with machine learning models, by using extrapolative testing sets, and invoking decorrelation objective functions, we create models which can predict more complex systems.

The models are interpretable, extrapolative, data-efficient, and capture predictable but complex non-stochastic behavior such as unmodeled degrees of freedom and systemic measurement noise.

We apply this improved modeling paradigm to several simulated systems and an actual physical system in the context of system identification.

Several ways of composing domain models with neural models are examined for time series, boosting, bagging, and auto-encoding on various systems of varying complexity and non-linearity.

Although this work is preliminary, we show that the ability to combine models is a very promising direction for neural modeling.

Improved modeling of complex systems uses hybrid neural/domain model composition, new decorrelation loss functions and extrapolative test sets

This paper conducts experiments to compare the extrapolative predictions of various hybrid models which compose physical models, neural networks and stochastic models, and tackles the challenge of unmodeled dynamics being a bottleneck.

This paper presents approaches for combining neural network with non-NN models to predict behavior of complex physical systems.

Improved Modeling of Complex Systems Using Hybrid Physics/Machine Learning/Stochastic Models

improve modeling complex system hybrid
neural network
extrapolative
stochastic model
/
physical


Humans can learn task-agnostic priors from interactive experience and utilize the priors for novel tasks without any finetuning.

In this paper, we propose Scoring-Aggregating-Planning (SAP), a framework that can learn task-agnostic semantics and dynamics priors from arbitrary quality interactions as well as the corresponding sparse rewards and then plan on unseen tasks in zero-shot condition.

The framework finds a neural score function for local regional state and action pairs that can be aggregated to approximate the quality of a full trajectory; moreover, a dynamics model that is learned with self-supervision can be incorporated for planning.

Many of previous works that leverage interactive data for policy learning either need massive on-policy environmental interactions or assume access to expert data while we can achieve a similar goal with pure off-policy imperfect data.

Instantiating our framework results in a generalizable policy to unseen tasks.

Experiments demonstrate that the proposed method can outperform baseline methods on a wide range of applications including gridworld, robotics tasks and video games.

We learn dense scores and dynamics model as priors from exploration data and use them to induce a good policy in new tasks in zero-shot condition.

This paper discusses zero shot generalization into new environments, and proposes an approach with results on Grid-World, Super Mario Bros, and 3D Robotics.

A method aiming to learn task-agnostic priors for zero-shot generalization, with the idea to employ a modeling approach on top of the model-based RL framework.

Scoring-Aggregating-Planning: Learning task-agnostic priors from interactions and sparse rewards for zero-shot generalization

learn task-agnostic prior
zero-shot generalization
new
approach


Particle-based inference algorithm is a promising method to efficiently generate samples for an intractable target distribution by iteratively updating a set of particles.

As a noticeable example, Stein variational gradient descent (SVGD) provides a deterministic and computationally efficient update, but it is known to underestimate the variance in high dimensions, the mechanism of which is poorly understood.

In this work we explore a connection between SVGD and MMD-based inference algorithm via Stein's lemma.

By comparing the two update rules, we identify the source of bias in SVGD as a combination of high variance and deterministic bias, and empirically demonstrate that the removal of either factors leads to accurate estimation of the variance.

In addition, for learning high-dimensional Gaussian target, we analytically derive the converged variance for both algorithms, and confirm that only SVGD suffers from the "curse of dimensionality".

Analyze the underlying mechanisms of variance collapse of SVGD in high dimensions.

Towards Characterizing the High-dimensional Bias of Kernel-based Particle Inference Algorithms


We describe an approach to understand the peculiar and counterintuitive generalization properties of deep neural networks.

The approach involves going beyond worst-case theoretical capacity control frameworks that have been popular in machine learning in recent years to revisit old ideas in the statistical mechanics of neural networks.

Within this approach, we present a prototypical Very Simple Deep Learning (VSDL) model, whose behavior is controlled by two control parameters, one describing an effective amount of data, or load, on the network (that decreases when noise is added to the input), and one with an effective temperature interpretation (that increases when algorithms are early stopped).

Using this model, we describe how a very simple application of ideas from the statistical mechanics theory of generalization provides a strong qualitative description of recently-observed empirical results regarding the inability of deep neural networks not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc.

Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior

The authors suggest that statistical mechanics ideas will help to understand generalization properties of deep neural networks, and give an approach that provides strong qualitative descriptions of empirical results regarding deep neural networks and learning algorithms.

A set of ideas related to theoretical understanding generalization properties of multilayer neural networks, and a qualitative analogy between behaviours in deep learning and results from quantitative statistical physics analysis of single and two-layer neural networks.

generalization require revisit old idea statistical mechanic approach


Computations for the softmax function in neural network models are expensive when the number of output classes is large.

This can become a significant issue in both training and inference for such models.

In this paper, we present Doubly Sparse Softmax (DS-Softmax), Sparse Mixture of Sparse of Sparse Experts, to improve the efficiency for softmax inference.

During training, our method learns a two-level class hierarchy by dividing entire output class space into several partially overlapping experts.

Each expert is responsible for a learned subset of the output class space and each output class only belongs to a small number of those experts.

During inference, our method quickly locates the most probable expert to compute small-scale softmax.

Our method is learning-based and requires no knowledge of the output class partition space a priori.

We empirically evaluate our method on several real-world tasks and demonstrate that we can achieve significant computation reductions without loss of performance.

We present doubly sparse softmax, the sparse mixture of sparse of sparse experts, to improve the efficiency for softmax inference through exploiting the two-level overlapping hierarchy.

This paper proposes a fast approximation to the softmax computation when the number of classes is very large.

This paper proposes a sparse mixture of sparse experts that learns a two-level class hierarchy for efficient softmax inference.

Doubly Sparse: Sparse Mixture of Sparse Experts for Efficient Softmax Inference

sparse mixture sparse
efficient softmax inference
hierarchy
expert
two-level
doubly
class