We explore the match prediction problem where one seeks to estimate the likelihood of a group of M items preferred over another, based on partial group comparison data.

As existing state-of-the-art algorithms are tailored to certain statistical models, we have different best algorithms across distinct scenarios.

Worse yet, we have no prior knowledge on the underlying model for a given scenario.

These call for a unified approach that can be universally applied to a wide range of scenarios and achieve consistently high performances.

To this end, we incorporate deep learning architectures so as to reflect the key structural features that most state-of-the-art algorithms, some of which are optimal in certain settings, share in common.

This enables us to infer hidden models underlying a given dataset, which govern in-group interactions and statistical patterns of comparisons, and hence to devise the best algorithm tailored to the dataset at hand.

Through extensive experiments on synthetic and real-world datasets, we evaluate our framework in comparison to state-of-the-art algorithms.

It turns out that our framework consistently leads to the best performance across all datasets in terms of cross entropy loss and prediction accuracy, while the state-of-the-art algorithms suffer from inconsistent performances across different datasets.

Furthermore, we show that it can be easily extended to attain satisfactory performances in rank aggregation tasks, suggesting that it can be adaptable for other tasks as well.

We investigate the merits of employing neural networks in the match prediction problem where one seeks to estimate the likelihood of a group of M items preferred over another, based on partial group comparison data.

This paper proposes a deep neural network solution to the set ranking problem and designs a architecture for this task inspired by previous manually designed algorithms.

This paper provides a technique to solve the match prediction problem using a deep learning architecture.

Match prediction from group comparison data using neural networks

match prediction problem
group comparison datum
neural network
deep
architecture


Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients.

Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs).

We propose a simpler and novel update scheme to maintain orthogonal recurrent weight matrices without using complex valued matrices.

This is done by parametrizing with a skew-symmetric matrix using the Cayley transform.

Such a parametrization is unable to represent matrices with negative one eigenvalues, but this limitation is overcome by scaling the recurrent weight matrix by a diagonal matrix consisting of ones and negative ones.

The proposed training scheme involves a straightforward gradient calculation and update step.

In several experiments, the proposed scaled Cayley orthogonal recurrent neural network (scoRNN) achieves superior results with fewer trainable parameters than other unitary RNNs.

A novel approach to maintain orthogonal recurrent weight matrices in a RNN.

Introduces a scheme for learning the recurrent parameter matrix in a neural network that uses the Cayley transform and a scaling weight matrix.

This paper suggests an RNN reparametrization of the recurrent weights with a skew-symmetric matrix using Cayley transform to keep the recurrent weight matrix orthogonal.

Novel parametrization of RNNs allows representing orthogonal weight matrices relatively easily.

Orthogonal Recurrent Neural Networks with Scaled Cayley Transform

recurrent weight matrix
cayley transform
neural network
rnn
scale
orthogonal
novel


A large number of natural language processing tasks exist to analyze syntax, semantics, and information content of human language.

These seemingly very different tasks are usually solved by specially designed architectures.

In this paper, we provide the simple insight that a great variety of tasks can be represented in a single unified format consisting of labeling spans and relations between spans, thus a single task-independent model can be used across different tasks.

We perform extensive experiments to test this insight on 10 disparate tasks as broad as dependency parsing (syntax), semantic role labeling (semantics), relation extraction (information content), aspect based sentiment analysis (sentiment), and many others, achieving comparable performance as state-of-the-art specialized models.

We further demonstrate benefits in multi-task learning.

We convert these datasets into a unified format to build a benchmark, which provides a holistic testbed for evaluating future models for generalized natural language analysis.

We use a single model to solve a great variety of natural language analysis tasks by formulating them in a unified span-relation format.

This paper generalizes a wide range of natural language processing tasks as a single span-based framework and proposes a general architecture to solve all these problems.

This work presents a unified formulation of various phrase and token level NLP tasks.

Generalizing Natural Language Analysis through Span-relation Representations

natural language analysis
single
solve
task
unified
span-relation
generalize


Large matrix inversions have often been cited as a major impediment to scaling Gaussian process (GP) models.

With the use of GPs as building blocks for ever more sophisticated Bayesian deep learning models, removing these impediments is a necessary step for achieving large scale results.

We present a variational approximation for a wide range of GP models that does not require a matrix inverse to be performed at each optimisation step.

Our bound instead directly parameterises a free matrix, which is an additional variational parameter.

At the local maxima of the bound, this matrix is equal to the matrix inverse.

We prove that our bound gives the same guarantees as earlier variational approximations.

We demonstrate some beneficial properties of the bound experimentally, although significant wall clock time speed improvements will require future improvements in optimisation and implementation.

We present a variational lower bound for GP models that can be optimised without computing expensive matrix operations like inverses, while providing the same guarantees as existing variational approximations.

Variational Gaussian Process Models without Matrix Inverses

inverse model matrix without variational


It has been shown that using geometric spaces with non-zero curvature instead of plain Euclidean spaces with zero curvature improves performance on a range of Machine Learning tasks for learning representations.

Recent work has leveraged these geometries to learn latent variable models like Variational Autoencoders (VAEs) in spherical and hyperbolic spaces with constant curvature.

While these approaches work well on particular kinds of data that they were designed for e.g.~tree-like data for a hyperbolic VAE, there exists no generic approach unifying all three models.

We develop a Mixed-curvature Variational Autoencoder, an efficient way to train a VAE whose latent space is a product of constant curvature Riemannian manifolds, where the per-component curvature can be learned.

This generalizes the Euclidean VAE to curved latent spaces, as the model essentially reduces to the Euclidean VAE if curvatures of all latent space components go to 0.

Variational Autoencoders with latent spaces modeled as products of constant curvature Riemannian manifolds improve on image reconstruction over single-manifold variants.

This paper introduces a general formulation of the notion of a VAE with a latent space composed by a curved manifold.

This paper is about developing VAEs in non-Euclidean spaces.

Mixed-curvature Variational Autoencoders

latent space
variational autoencoder
manifold


Machine learning algorithms for generating molecular structures offer a promising new approach to drug discovery.

We cast molecular optimization as a translation problem, where the goal is to map an input compound to a target compound with improved biochemical properties.

Remarkably, we observe that when generated molecules are iteratively fed back into the translator, molecular compound attributes improve with each step.

We show that this finding is invariant to the choice of translation model, making this a "black box" algorithm.

We call this method Black Box Recursive Translation (BBRT), a new inference method for molecular property optimization.

This simple, powerful technique operates strictly on the inputs and outputs of any translation model.

We obtain new state-of-the-art results for molecular property optimization tasks using our simple drop-in replacement with well-known sequence and graph-based models.

Our method provides a significant boost in performance relative to its non-recursive peers with just a simple "``for" loop.

Further, BBRT is highly interpretable, allowing users to map the evolution of newly discovered compounds from known starting points.

We introduce a black box algorithm for repeated optimization of compounds using a translation framework.

The authors frame molecule optimization as a sequence-to-sequence problem, and extend existing methods for improving molecules, showing that it is beneficial for optimizing logP but not QED.

The paper builds on existing translation models developed for molecular optimization, making an iterative use of sequence to sequence or graph to graph translation models.

Black Box Recursive Translations for Molecular Optimization

molecular optimization
black box
translation
exist


Deep Neural Networks (DNNs) are increasingly deployed in cloud servers and autonomous agents due to their superior performance.

The deployed DNN is either leveraged in a white-box setting (model internals are publicly known) or a black-box setting (only model outputs are known) depending on the application.

A practical concern in the rush to adopt DNNs is protecting the models against Intellectual Property (IP) infringement.

We propose BlackMarks, the first end-to-end multi-bit watermarking framework that is applicable in the black-box scenario.

BlackMarks takes the pre-trained unmarked model and the owner’s binary signature as inputs.

The output is the corresponding marked model with specific keys that can be later used to trigger the embedded watermark.

To do so, BlackMarks first designs a model-dependent encoding scheme that maps all possible classes in the task to bit ‘0’ and bit ‘1’.

Given the owner’s watermark signature (a binary string), a set of key image and label pairs is designed using targeted adversarial attacks.

The watermark (WM) is then encoded in the distribution of output activations of the DNN by fine-tuning the model with a WM-specific regularized loss.

To extract the WM, BlackMarks queries the model with the WM key images and decodes the owner’s signature from the corresponding predictions using the designed encoding scheme.

We perform a comprehensive evaluation of BlackMarks’ performance on MNIST, CIFAR-10, ImageNet datasets and corroborate its effectiveness and robustness.

BlackMarks preserves the functionality of the original DNN and incurs negligible WM embedding overhead as low as 2.054%.

Proposing the first watermarking framework for multi-bit signature embedding and extraction using the outputs of the DNN.

Proposes a method for multi-bit watermarking of neural networks in a black-box setting and demonstrate that the predictions of existing models can carry a multi-bit string that can later be used to verify ownership.

The paper proposes an approach for model watermarking where the watermark is a bit string embedded in the model as part of a fine-tuning procedure

BlackMarks: Black-box Multi-bit Watermarking for Deep Neural Networks

multi-bit watermarking
embed
neural network
model
black-box
string


Adversarial training provides a principled approach for training robust neural networks.

From an optimization perspective, the adversarial training is essentially solving a minmax robust optimization problem.

The outer minimization is trying to learn a robust classifier, while the inner maximization is trying to generate adversarial samples.

Unfortunately, such a minmax problem is very difficult to solve due to the lack of convex-concave structure.

This work proposes a new adversarial training method based on a general learning-to-learn framework.

Specifically, instead of applying the existing hand-design algorithms for the inner problem, we learn an optimizer, which is parametrized as a convolutional neural network.

At the same time, a robust classifier is learned to defense the adversarial attack generated by the learned optimizer.

From the perspective of generative learning, our proposed method can be viewed as learning a deep generative model for generating adversarial samples, which is adaptive to the robust classification.

Our experiments demonstrate that our proposed method significantly outperforms existing adversarial training methods on CIFAR-10 and CIFAR-100 datasets.

Don't know how to optimize? Then just learn to optimize!

This paper proposes a way to train image classification models to be resistant to L-infinity perturbation attacks.

This paper proposes using the learning-to-learn framework to learn an attacker.

Learning to Defense by Learning to Attack


In this work we introduce a new framework for performing temporal predictions

It is based on a simple idea of disentangling com-

ponents of the future state which are predictable from those which are inherently

unpredictable, and encoding the unpredictable components into a low-dimensional

latent variable which is fed into the forward model.

pervised training objective which is fast and easy to train.

context of video prediction on multiple datasets and show that it is able to consi-

tently generate diverse predictions without the need for alternating minimization

over a latent space or adversarial training.

A simple and easy to train method for multimodal prediction in time series.

This paper introduces a times-series prediction model that that learns a deterministic mapping and trains another net to predict future frames given the input and residual error from the first network.

The paper proposes a model for prediction under uncertainty where they separate out deterministic component prediction and uncertain component prediction.

Prediction Under Uncertainty with Error Encoding Networks

prediction uncertainty
model
train
deterministic
error
network


Conducting reinforcement-learning experiments can be a complex and timely process.

A full experimental pipeline will typically consist of a simulation of an environment, an implementation of one or many learning algorithms, a variety of additional components designed to facilitate the agent-environment interplay, and any requisite analysis, plotting, and logging thereof.

In light of this complexity, this paper introduces simple rl, a new open source library for carrying out reinforcement learning experiments in Python 2 and 3 with a focus on simplicity.

The goal of simple_rl is to support seamless, reproducible methods for running reinforcement learning experiments.

This paper gives an overview  of the core design philosophy of the package, how it differs from existing libraries, and showcases its central features.

This paper introduces and motivates simple_rl, a new open source library for carrying out reinforcement learning experiments in Python 2 and 3 with a focus on simplicity.

simple_rl: Reproducible Reinforcement Learning in Python


Wasserstein GAN(WGAN) is a model that minimizes the Wasserstein distance between a data distribution and sample distribution.

Recent studies have proposed stabilizing the training process for the WGAN and implementing the Lipschitz constraint.

In this study, we prove the local stability of optimizing the simple gradient penalty $\mu$-WGAN(SGP $\mu$-WGAN) under suitable assumptions regarding the equilibrium and penalty measure $\mu$.

The measure valued differentiation concept is employed to deal with the derivative of the penalty terms, which is helpful for handling abstract singular measures with lower dimensional support.

Based on this analysis, we claim that penalizing the data manifold or sample manifold is the key to regularizing the original WGAN with a gradient penalty.

Experimental results obtained with unintuitive penalty measures that satisfy our assumptions are also provided to support our theoretical results.

This paper deals with stability of simple gradient penalty $\mu$-WGAN optimization by introducing a concept of measure valued differentiation.

WGAN with a squared zero centered gradient penalty term w.r.t. to a general measure is studied.

Characterizes the convergence of gradient penalized Wasserstein GAN.

Local Stability and Performance of Simple Gradient Penalty $\mu$-Wasserstein GAN

simple gradient penalty $
stability
measure
gan


We present Random Partition Relaxation (RPR), a method for strong quantization of the parameters of convolutional neural networks to binary (+1/-1) and ternary (+1/0/-1) values.

Starting from a pretrained model, we first quantize the weights and then relax random partitions of them to their continuous values for retraining before quantizing them again and switching to another weight partition for further adaptation.

We empirically evaluate the performance of RPR with ResNet-18, ResNet-50 and GoogLeNet on the ImageNet classification task for binary and ternary weight networks.

We show accuracies beyond the state-of-the-art for binary- and ternary-weight GoogLeNet and competitive performance for ResNet-18 and ResNet-50 using a SGD-based training method that can easily be integrated into existing frameworks.

State-of-the-art training method for binary and ternary weight networks based on alternating optimization of randomly relaxed weight partitions

The paper proposes a new training scheme of optimizing a ternary neural network.

Authors propose RPR, a way to randomly partition and quantize weights and train the remaining parameters followed by relaxation in alternate cycles to train quantized models.

Random Partition Relaxation for Training Binary and Ternary Weight Neural Network

binary ternary weight
alternate
randomly
neural network
relaxation
train
training
partition


Learning long-term dependencies is a key long-standing challenge of recurrent neural networks (RNNs).

Hierarchical recurrent neural networks (HRNNs) have been considered a promising approach as long-term dependencies are resolved through shortcuts up and down the hierarchy.

Yet, the memory requirements of Truncated Backpropagation Through Time (TBPTT) still prevent training them on very long sequences.

In this paper, we empirically show that in (deep) HRNNs, propagating gradients back from higher to lower levels can be replaced by locally computable losses, without harming the learning capability of the network, over a wide range of tasks.

This decoupling by local losses reduces the memory requirements of training by a factor exponential in the depth of the hierarchy in comparison to standard TBPTT.

We replace some gradients paths in hierarchical RNN's by an auxiliary loss. We show that this can reduce the memory cost while preserving performance.

The paper introduces a hierarchical RNN architecture that could be trained more memory efficiently.

The proposed paper suggests to decouple the different layers of hierarchy in RNN using auxiliary losses.

Decoupling Hierarchical Recurrent Neural Networks With Locally Computable Losses

auxiliary loss
hierarchical rnn
memory
decouple


In a typical deep learning approach to a computer vision task, Convolutional Neural Networks (CNNs) are used to extract features at varying levels of abstraction from an image and compress a high dimensional input into a lower dimensional decision space through a series of transformations.

In this paper, we investigate how a class of input images is eventually compressed over the course of these transformations.

In particular, we use singular value decomposition to analyze the relevant variations in feature space.

These variations are formalized as the effective dimension of the embedding.

We consider how the effective dimension varies across layers within class.

We show that across datasets and architectures, the effective dimension of a class increases before decreasing further into the network, suggesting some sort of initial whitening transformation.

Further, the decrease rate of the effective dimension deeper in the network corresponds with training performance of the model.

Neural networks that do a good job of classification project points into more spherical shapes before compressing them into fewer dimensions.

Using effective dimension to analyze feature transformations in deep neural networks


Deep learning methods have achieved high performance in sound recognition tasks.

Deciding how to feed the training data is important for further performance improvement.

We propose a novel learning method for deep sound recognition: Between-Class learning (BC learning).

Our strategy is to learn a discriminative feature space by recognizing the between-class sounds as between-class sounds.

We generate between-class sounds by mixing two sounds belonging to different classes with a random ratio.

We then input the mixed sound to the model and train the model to output the mixing ratio.

The advantages of BC learning are not limited only to the increase in variation of the training data; BC learning leads to an enlargement of Fisher’s criterion in the feature space and a regularization of the positional relationship among the feature distributions of the classes.

The experimental results show that BC learning improves the performance on various sound recognition networks, datasets, and data augmentation schemes, in which BC learning proves to be always beneficial.

Furthermore, we construct a new deep sound recognition network (EnvNet-v2) and train it with BC learning.

As a result, we achieved a performance surpasses the human level.

We propose an novel learning method for deep sound recognition named BC learning.

Authors defined a new learning task that requires a DNN to predict mixing ratio between sounds from two different classes to increase discriminitive power of the final learned network.

Proposes a method to improve the performance of a generic learning method by generating "in between class" training samples and presents the basic intuition and necessity of the proposed technique.

Learning from Between-class Examples for Deep Sound Recognition

deep sound recognition
learning method
class


Spatiotemporal forecasting has become an increasingly important prediction task in machine learning and statistics due to its vast applications, such as climate modeling, traffic prediction, video caching predictions, and so on.

While numerous studies have been conducted, most existing works assume that the data from different sources or across different locations are equally reliable.

Due to cost, accessibility, or other factors, it is inevitable that the data quality could vary, which introduces significant biases into the model and leads to unreliable prediction results.

The problem could be exacerbated in black-box prediction models, such as deep neural networks.

In this paper, we propose a novel solution that can automatically infer data quality levels of different sources through local variations of spatiotemporal signals without explicit labels.

Furthermore, we integrate the estimate of data quality level with graph convolutional networks to exploit their efficient structures.

We evaluate our proposed method on forecasting temperatures in Los Angeles.

We propose a method that infers the time-varying data quality level for spatiotemporal forecasting without explicitly assigned labels.

Introduces a new definition of data quality that relies on the notion of local variation defined in (Zhou and Scholkopf) and extends it to multiple heterogenous data sources.

This work proposed a new way to evaluate the quality of different data sources with the time-vary graph model, with the quality level used as a regularization term in the objective function

Automatically Inferring Data Quality for Spatiotemporal Forecasting

spatiotemporal forecasting
quality level
datum source
new
infer
data


Human perception of 3D shapes goes beyond reconstructing them as a set of points or a composition of geometric primitives: we also effortlessly understand higher-level shape structure such as the repetition and reflective symmetry of object parts.

In contrast, recent advances in 3D shape sensing focus more on low-level geometry but less on these higher-level relationships.

In this paper, we propose 3D shape programs, integrating bottom-up recognition systems with top-down, symbolic program structure to capture both low-level geometry and high-level structural priors for 3D shapes.

Because there are no annotations of shape programs for real shapes, we develop neural modules that not only learn to infer 3D shape programs from raw, unannotated shapes, but also to execute these programs for shape reconstruction.

After initial bootstrapping, our end-to-end differentiable model learns 3D shape programs by reconstructing shapes in a self-supervised manner.

Experiments demonstrate that our model accurately infers and executes 3D shape programs for highly complex shapes from various categories.

It can also be integrated with an image-to-shape module to infer 3D shape programs directly from an RGB image, leading to 3D shape reconstructions that are both more accurate and more physically plausible.

We propose 3D shape programs, a structured, compositional shape representation. Our model learns to infer and execute shape programs to explain 3D shapes.

An approach to infer shape programs given 3D models, with architecture consisting of a recurrent network that encodes a 3D shape and outputs instructions, and a second module that renders the program to 3D.

This paper introduces a high-level semantic description for 3D shapes, given by the ShapeProgram.

Learning to Infer and Execute 3D Shape Programs

learn infer execute
3d shape
model
program
give


Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention  thanks to its encouraging performance on a variety of control tasks.

Yet, conventional regularization techniques in training neural networks (e.g., $L_2$ regularization, dropout) have been largely ignored in RL methods, possibly because agents are typically trained and evaluated in the same environment.

In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks.

Interestingly, we find conventional regularization techniques on the policy networks can often bring large improvement on the task performance, and the improvement is typically more significant when the task is more difficult.

We also compare with the widely used entropy regularization and find $L_2$ regularization is generally better.

Our findings are further confirmed to be robust against the choice of training hyperparameters.

We also study the effects of regularizing different components and find that only regularizing the policy network is typically enough.

We hope our study provides guidance for future practices in regularizing policy optimization algorithms.

We show that conventional regularization methods (e.g., $L_2$, dropout), which have been largely ignored in RL methods, can be very effective in policy optimization.

The authors study a set of existing direct policy optimization methods in the field of reinforcement learning and provide a detailed investigation on the effect of regulations on the performance and behavior of agents following these methods.

This paper provides a study on the effect of regularization on performance in training environments in policy optimization methods in multiple continuous control tasks.

Regularization Matters in Policy Optimization

policy optimization method
effect
provide
performance
study
regularization


We introduce FigureQA, a visual reasoning corpus of over one million question-answer pairs grounded in over 100,000 images.

The images are synthetic, scientific-style figures from five classes: line plots, dot-line plots, vertical and horizontal bar graphs, and pie charts.

We formulate our reasoning task by generating questions from 15 templates; questions concern various relationships between plot elements and examine characteristics like the maximum, the minimum, area-under-the-curve, smoothness, and intersection.

To resolve, such questions often require reference to multiple plot elements and synthesis of information distributed spatially throughout a figure.

To facilitate the training of machine learning systems, the corpus also includes side data that can be used to formulate auxiliary objectives.

In particular, we provide the numerical data used to generate each figure as well as bounding-box annotations for all plot elements.

We study the proposed visual reasoning task by training several models, including the recently proposed Relation Network as strong baseline.

Preliminary results indicate that the task poses a significant machine learning challenge.

We envision FigureQA as a first step towards developing models that can intuitively recognize patterns from visual representations of data.

We present a question-answering dataset, FigureQA, as a first step towards developing models that can intuitively recognize patterns from visual representations of data.

This paper introduces a dataset of templated question answering on figures, involving reasoning about figure elements.

The paper introduces a new visual reasoning dataset called Figure-QA which consists of 140K figure images and 1.55M QA pairs, which can help in developing models that can extract useful information from visual representations of data.

FigureQA: An Annotated Figure Dataset for Visual Reasoning

visual representation datum
develop model
introduce
figureqa
dataset
figure
reasoning


In this paper, I discuss some varieties of explanation that can arise

I distinguish between process accounts, which

address the detailed decisions made during heuristic search, and

preference accounts, which clarify the ordering of alternatives

which types of users will appreciate which types of explanation.

In addition, I discuss three facets of multi-step decision making

-- conceptual inference, plan generation, and plan execution --

to present questions to agents and for them provide their answers.

This position paper analyzes different types of self explanation that can arise in planning and related systems.

Discusses different aspects of explanations, particularly in the context of sequential decision making.

Varieties of Explainable Agency


Generative deep learning has sparked a new wave of Super-Resolution (SR) algorithms that enhance single images with impressive aesthetic results, albeit with imaginary details.

Multi-frame Super-Resolution (MFSR) offers a more grounded approach to the ill-posed problem, by conditioning on multiple low-resolution views.

This is important for satellite monitoring of human impact on the planet -- from deforestation, to human rights violations -- that depend on reliable imagery.

To this end, we present HighRes-net, the first deep learning approach to MFSR that learns its sub-tasks in an end-to-end fashion:

Co-registration of low-res views is learned implicitly through a reference-frame channel, with no explicit registration mechanism.

We learn a global fusion operator that is applied recursively on an arbitrary number of low-res pairs.

We introduce a registered loss, by learning to align the SR output to a ground-truth through ShiftNet.

We show that by learning deep representations of multiple views, we can super-resolve low-resolution signals and enhance Earth observation data at scale.

Our approach recently topped the European Space Agency's MFSR competition on real-world satellite imagery.

The first deep learning approach to MFSR to solve registration, fusion, up-sampling in an end-to-end manner.

This paper proposes an end-to-end multi-frame super-resolution algorithm, that relies on a pair-wise co-registrations and fusing blocks (convolutional residual blocks), embedded in a encoder-decoder network 'HighRes-net' that estimates the super-resolution image.

This paper proposes a framework including recursive fusion to co-registration loss to solve the problem of super-resolution results and high-resolution labels not being pixel aligned.

HighRes-net: Multi-Frame Super-Resolution by Recursive Fusion

end-to-end
recursive fusion
solve
super-resolution
co-registration
highres-net
multi-frame


Large mini-batch parallel SGD is commonly used for distributed training of deep networks.

Approaches that use tightly-coupled exact distributed averaging based on AllReduce are sensitive to slow nodes and high-latency communication.

In this work we show the applicability of Stochastic Gradient Push (SGP) for distributed training.

SGP uses a gossip algorithm called PushSum for approximate distributed averaging, allowing for much more loosely coupled communications which can be beneficial in high-latency or high-variability scenarios.

The tradeoff is that approximate distributed averaging injects additional noise in the gradient which can affect the train and test accuracies.

We prove that SGP converges to a stationary point of smooth, non-convex objective functions.

Furthermore, we validate empirically the potential of SGP.

For example, using 32 nodes with 8 GPUs per node to train ResNet-50 on ImageNet, where nodes communicate over 10Gbps Ethernet, SGP completes 90 epochs in around 1.5 hours while AllReduce SGD takes over 5 hours, and the top-1 validation accuracy of SGP remains within 1.2% of that obtained using AllReduce SGD.

For distributed training over high-latency networks, use gossip-based approximate distributed averaging instead of exact distribute averaging like AllReduce.

The authors propose using gossip algorithms as a general method of computing approximate average over a set of workers approximately

The paper proves the convergence of SGP for nonconvex smooth functions and shows the SGP can achieve a significant speed-up in the low-latency environment without sacrificing too much predictive performance.

Stochastic Gradient Push for Distributed Deep Learning


In this paper, we extend the persona-based sequence-to-sequence (Seq2Seq) neural network conversation model to a multi-turn dialogue scenario by modifying the state-of-the-art hredGAN architecture to simultaneously capture utterance attributes such as speaker identity, dialogue topic, speaker sentiments and so on.

The proposed system, phredGAN has a persona-based HRED generator (PHRED) and a conditional discriminator.

We also explore two approaches to accomplish the conditional discriminator: (1) $phredGAN_a$, a system that passes the attribute representation as an additional input into a traditional adversarial discriminator, and (2) $phredGAN_d$, a dual discriminator system which in addition to the adversarial discriminator, collaboratively predicts the attribute(s) that generated the input utterance.

To demonstrate the superior performance of phredGAN over the persona SeqSeq model, we experiment with two conversational datasets, the Ubuntu Dialogue Corpus (UDC) and TV series transcripts from the Big Bang Theory and Friends.

Performance comparison is made with respect to a variety of quantitative measures as well as crowd-sourced human evaluation.

We also explore the trade-offs from using either variant of $phredGAN$ on datasets with many but weak attribute modalities (such as with Big Bang Theory and Friends) and ones with few but strong attribute modalities (customer-agent interactions in Ubuntu dataset).

This paper develops an adversarial learning framework for neural conversation models with persona

This paper proposes an extension to hredGAN to simultaneously learn a set of attribute embeddings that represent the persona of each speaker and generate persona-based responses

An Adversarial Learning Framework for a Persona-based Multi-turn Dialogue Model

adversarial learning framework
persona
model
persona-based


We introduce bio-inspired artificial neural networks consisting of neurons that are additionally characterized by spatial positions.

To simulate properties of biological systems we add the costs penalizing long connections and the proximity of neurons in a two-dimensional space.

Our experiments show that in the case where the network performs two different tasks, the neurons naturally split into clusters, where each cluster is responsible for processing a different task.

This behavior not only corresponds to the biological systems, but also allows for further insight into interpretability or continual learning.

Bio-inspired artificial neural networks, consisting of neurons positioned in a two-dimensional space, are capable of forming independent groups for performing different tasks.

Biologically-Inspired Spatial Neural Networks


The transformer has become a central model for many NLP tasks from translation to language modeling to representation learning.

Its success demonstrates the effectiveness of stacked attention as a replacement for recurrence for many tasks.

In theory attention also offers more insights into the model’s internal decisions; however, in practice when stacked it quickly becomes nearly as fully-connected as recurrent models.

In this work, we propose an alternative transformer architecture, discrete transformer, with the goal of better separating out internal model decisions.

The model uses hard attention to ensure that each step only depends on a fixed context.

Additionally, the model uses a separate “syntactic” controller to separate out network structure from decision making.

Finally we show that this approach can be further sparsified with direct regularization.

Empirically, this approach is able to maintain the same level of performance on several datasets, while discretizing reasoning decisions over the data.

Discrete transformer which uses hard attention to ensure that each step only depends on a fixed context.

This paper presents modifications to the standard transformer architecture with the goal of improving interpretability while retaining performance in NLP tasks.

This paper proposes three Discrete Transformers: a discrete and stochastic Gumbel-softmax based attention module, a two-stream syntactic and semantic transformer, and sparsity regularization.

Discrete Transformer


Deep predictive coding networks are neuroscience-inspired unsupervised learning models that learn to predict future sensory states.

We build upon the PredNet implementation by Lotter, Kreiman, and Cox (2016) to investigate if predictive coding representations are useful to predict brain activity in the visual cortex.

We use representational similarity analysis (RSA) to compare PredNet representations to functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data from the Algonauts Project (Cichy et al., 2019).

In contrast to previous findings in the literature (Khaligh-Razavi & Kriegeskorte, 2014), we report empirical data suggesting that unsupervised models trained to predict frames of videos without further fine-tuning may outperform supervised image classification baselines in terms of correlation to spatial (fMRI) and temporal (MEG) data.

We show empirical evidence that predictive coding models yield representations more correlated to brain data than supervised image recognition models.

Learning to predict visual brain activity by predicting future sensory states


The incorporation of prior knowledge into learning is essential in achieving good performance based on small noisy samples.

Such knowledge is often incorporated through the availability of related data arising from domains and tasks similar to the one of current interest.

Ideally one would like to allow both the data for the current task and for previous related tasks to self-organize the learning system in such a way that commonalities and differences between the tasks are learned in a data-driven fashion.

We develop a framework for learning multiple tasks simultaneously, based on sharing features that are common to all tasks, achieved through the use of a modular deep feedforward neural network consisting of shared branches, dealing with the common features of all tasks, and private branches, learning the specific unique aspects of each task.

Once an appropriate weight sharing architecture has been established, learning takes place through standard algorithms for feedforward networks, e.g., stochastic gradient descent and its variations.

The method deals with meta-learning (such as domain adaptation, transfer and multi-task learning) in a unified fashion, and can easily deal with data arising from different types of sources.

Numerical experiments demonstrate the effectiveness of learning in domain adaptation and transfer learning setups, and provide evidence for the flexible and task-oriented representations arising in the network.

A generic framework for handling transfer and multi-task learning using pairs of autoencoders with task-specific and shared weights.

Proposes a generic framework for end-to-end transfer learning / domain adaptation with deep neural networks.

This paper proposes a model for allowing deep neural network architectures to share parameters across different datasets, and applies it to transfer learning.

The paper focuses on learning common features from multiple domains data and ends up with a general architecture for multi-task, semi-supervised and transfer learning

Joint autoencoders: a flexible meta-learning framework

generic framework
autoencoder
deep neural network
learning
multi-task
transfer
architecture
learn
domain


Deep neural networks and decision trees operate on largely separate paradigms; typically, the former performs representation learning with pre-specified architectures, while the latter is characterised by learning hierarchies over pre-specified features with data-driven architectures.

We unite the two via adaptive neural trees (ANTs), a model that incorporates representation learning into edges, routing functions and leaf nodes of a decision tree, along with a backpropagation-based training algorithm that adaptively grows the architecture from primitive modules (e.g., convolutional layers).

ANTs allow increased interpretability via hierarchical clustering, e.g., learning meaningful class associations, such as separating natural vs. man-made objects.

We demonstrate this on classification and regression tasks, achieving over 99% and 90% accuracy on the MNIST and CIFAR-10 datasets, and outperforming standard neural networks, random forests and gradient boosted trees on the SARCOS dataset.

Furthermore, ANT optimisation naturally adapts the architecture to the size and complexity of the training data.

We propose a framework to combine decision trees and neural networks, and show on image classification tasks that it enjoys the complementary benefits of the two approaches, while addressing the limitations of prior work.

The authors proposed a new model, Adaptive Neural Trees, by combining the representation learning and gradient optimization of neural networks with architecture learning of decision trees

This paper proposes the Adaptive Neural Trees approach to combine the two learning paradigms of deep neural nets and decision trees

Adaptive Neural Trees

adaptive neural tree
combine
two
network
approach
decision


While natural language processing systems often focus on a single language, multilingual transfer learning has the potential to improve performance, especially for low-resource languages.

We introduce XLDA, cross-lingual data augmentation, a method that replaces a segment of the input text with its translation in another language.

XLDA enhances performance of all 14 tested languages of the cross-lingual natural language inference (XNLI) benchmark.

With improvements of up to 4.8, training with XLDA achieves state-of-the-art performance for Greek, Turkish, and Urdu.

XLDA is in contrast to, and performs markedly better than, a more naive approach that aggregates examples in various languages in a way that each example is solely in one language.

On the SQuAD question answering task, we see that XLDA provides a 1.0 performance increase on the English evaluation set.

Comprehensive experiments suggest that most languages are effective as cross-lingual augmentors, that XLDA is robust to a wide range of translation quality, and that XLDA is even more effective for randomly initialized models than for pretrained models.

Translating portions of the input during training can improve cross-lingual performance.

The paper proposes a cross-lingual data augmentation method to improve the language inference and question answering tasks.

This paper proposes to augment crosslingual data with heuristic swaps using aligned translations, like bilingual humans do in code-switching.

XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering

cross-lingual datum augmentation
language inference question
improve


Training conditional generative latent-variable models is challenging in scenarios where the conditioning signal is very strong and the decoder is expressive enough to generate a plausible output given only the condition; the generative model tends to ignore the latent variable, suffering from posterior collapse.

We find, and empirically show, that one of the major reasons behind posterior collapse is rooted in the way that generative models are conditioned, i.e., through concatenation of the latent variable and the condition

. To mitigate this problem, we propose to explicitly make the latent variables depend on the condition by unifying the conditioning and latent variable sampling, thus coupling them so as to prevent the model from discarding the root of variations

. To achieve this, we develop a conditional Variational Autoencoder architecture that learns a distribution not only of the latent variables, but also of the condition, the latter acting as prior on the former

. Our experiments on the challenging tasks of conditional human motion prediction and image captioning demonstrate the effectiveness of our approach at avoiding posterior collapse

. Video results of our approach are anonymously provided in http://bit.ly/iclr2020

We propose a conditional variational autoencoder framework that mitigates the posterior collapse in scenarios where the conditioning signal strong enough for an expressive decoder to generate a plausible output from it.

This paper considers strongly conditioned generative models, and proposes an objective function and a parameterisation of the variational distribution such that latent variables explicitly depend on input conditions.

This paper argues that when the decoder is conditioned on the concatenation of latent variables and auxiliary information, then posterior collapse is more likely than in vanilla VAE.

Mitigating Posterior Collapse in Strongly Conditioned Variational Autoencoders

posterior collapse
latent variable
decoder
variational autoencoder
condition
mitigate
strongly


We propose a study of the stability of several few-shot learning algorithms subject to variations in the hyper-parameters and optimization schemes while controlling the random seed.

We propose a methodology for testing for statistical differences in model performances under several replications.

To study this specific design, we attempt to reproduce results from three prominent papers: Matching Nets, Prototypical Networks, and TADAM.

We analyze on the miniImagenet dataset on the standard classification task in the 5-ways, 5-shots learning setting at test time.

We find that the selected implementations exhibit stability across random seed, and repeats.

This paper studies reproducibility for few-shot learning.

Reproducibility and Stability Analysis in Metric-Based Few-Shot Learning

study
few-shot learning
stability
reproducibility


We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning.

In such hierarchical structures, a higher-level controller solves tasks by iteratively communicating goals which a lower-level policy is trained to reach.

Accordingly, the choice of representation -- the mapping of observation space to goal space -- is crucial.

To study this problem, we develop a notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation.

We derive expressions which bound the sub-optimality and show how these expressions can be translated to representation learning objectives which may be optimized in practice.

Results on a number of difficult continuous-control tasks show that our approach to representation learning yields qualitatively better representations as well as quantitatively better hierarchical policies, compared to existing methods.

We translate a bound on sub-optimality of representations to a practical training objective in the context of hierarchical reinforcement learning.

The authors proposes a novel approach in learning a representation for HRL and state an intriguing connection between representation learning and bounding the sub-optimality which results in a gradient based algorithm

This paper proposes a way to handle sub-optimality in the context of learning representations which refer to the sub-optimality of hierarchical polity with respect to the task reward.

Near-Optimal Representation Learning for Hierarchical Reinforcement Learning

hierarchical reinforcement learning
context
learn representation
sub-optimality


Heuristic search research often deals with finding algorithms for offline planning which aim to minimize the number of expanded nodes or planning time.

In online planning, algorithms for real-time search or deadline-aware search have been considered before.

However, in this paper, we are interested in the problem of {\em situated temporal planning} in which an agent's plan can depend on exogenous events in the external world, and thus it becomes important to take the passage of time into account during the planning process.

Previous work on situated temporal planning has proposed simple pruning strategies, as well as complex schemes for a simplified version of the associated metareasoning problem.

In this paper, we propose a simple metareasoning technique,  called the crude greedy scheme, which can be applied in a situated temporal planner.

Our empirical evaluation shows that the crude greedy scheme outperforms standard heuristic search based on cost-to-go estimates.

Metareasoning in a Situated Temporal Planner

This paper addresses the problem of situated temporal planning, proposing a further simplification on  greedy strategies previously proposed by Shperberg.

Beyond Cost-to-go Estimates in Situated Temporal Planning


Neural networks are vulnerable to small adversarial perturbations.

Existing literature largely focused on understanding and mitigating the vulnerability of learned models.

In this paper, we demonstrate an intriguing phenomenon about the most popular robust training method in the literature, adversarial training: Adversarial robustness, unlike clean accuracy, is sensitive to the input data distribution.

Even a semantics-preserving transformations on the input data distribution can cause a significantly different robustness for the adversarial trained model that is both trained and evaluated on the new distribution.

Our discovery of such sensitivity on data distribution is based on a study which disentangles the behaviors of clean accuracy and robust accuracy of the Bayes classifier.

Empirical investigations further confirm our finding.

We construct semantically-identical variants for MNIST and CIFAR10 respectively, and show that standardly trained models achieve comparable clean accuracies on them, but adversarially trained models achieve significantly different robustness accuracies.

This counter-intuitive phenomenon indicates that input data distribution alone can affect the adversarial robustness of trained neural networks, not necessarily the tasks themselves.

Lastly, we discuss the practical implications on evaluating adversarial robustness, and make initial attempts to understand this complex phenomenon.

Robustness performance of PGD trained models are sensitive to semantics-preserving transformation of image datasets, which implies the trickiness of evaluation of robust learning algorithms in practice.

Paper clarifies the difference between clean and robust accuracy and shows that changing the marginal distribution of the input data P(x) while preserving its semantic P(y|x) affects the robustness of the model.

This paper investigates the origin of the lack of robustness of classifiers to perturbations of adversarial inputs under l-inf bounded perturbations.

On the Sensitivity of Adversarial Robustness to Input Data Distributions

robustness model robust input adversarial distribution


Many tasks in natural language processing involve comparing two sentences to compute some notion of relevance, entailment, or similarity.

Typically this comparison is done either at the word level or at the sentence level, with no attempt to leverage the inherent structure of the sentence.

When sentence structure is used for comparison, it is obtained during a non-differentiable pre-processing step, leading to propagation of errors.

We introduce a model of structured alignments between sentences, showing how to compare two sentences by matching their latent structures.

Using a structured attention mechanism, our model matches possible spans in the first sentence to possible spans in the second sentence, simultaneously discovering the tree structure of each sentence and performing a comparison, in a model that is fully differentiable and is trained only on the comparison objective.

We evaluate this model on two sentence comparison tasks: the Stanford natural language inference dataset and the TREC-QA dataset.

We find that comparing spans results in superior performance to comparing words individually, and that the learned trees are consistent with actual linguistic structures.

Matching sentences by learning the latent constituency tree structures with a variant of the inside-outside algorithm embedded as a neural network layer.

This paper introduces a structured attention mechanisms to compute alignment scores among all possible spans in two given sentences

This paper proposes a model of structured alignments between sentences as a means of comparing sentences by matching their latent structures.

STRUCTURED ALIGNMENT NETWORKS

structure sentence latent structured network match alignment


Learning disentangled representation from any unlabelled data is a non-trivial problem.

In this paper we propose Information Maximising Autoencoder (InfoAE) where the encoder learns powerful disentangled representation through maximizing the mutual information between the representation and given information in an unsupervised fashion.

We have evaluated our model on MNIST dataset and achieved approximately 98.9 % test accuracy while using complete unsupervised training.

Learn disentangle representation in an unsupervised manner.

The authors present a framework in which an auto encoder (E, D) is regularized such that its latent representation to share mutual information with a generated latent space representation.

Disentangled Representation Learning with Information Maximizing Autoencoder

disentangle representation
information
learn


Effective training of neural networks requires much data.

parameters are underdetermined, and learnt networks generalise poorly.

Augmentation (Krizhevsky et al., 2012) alleviates this by using existing data

However standard data augmentation produces only limited

Given there is potential to generate a much broader set

of augmentations, we design and train a generative model to do data augmentation.

The model, based on image conditional Generative Adversarial Networks, takes

data from a source domain and learns to take any data item and generalise it

to generate other within-class data items.

depend on the classes themselves, it can be applied to novel unseen classes of data.

We show that a Data Augmentation Generative Adversarial Network (DAGAN)

augments standard vanilla classifiers well.

few-shot learning systems such as Matching Networks.

approaches on Omniglot, on EMNIST having learnt the DAGAN on Omniglot, and

In our experiments we can see over 13% increase in accuracy in

the low-data regime experiments in Omniglot (from 69% to 82%), EMNIST (73.9%

to 76%) and VGG-Face (4.5% to 12%); in Matching Networks for Omniglot we

observe an increase of 0.5% (from 96.9% to 97.4%) and an increase of 1.8% in

Conditional GANs trained to generate data augmented samples of their conditional inputs used to enhance vanilla classification and one shot learning systems such as matching networks and pixel distance

The authors propose a method to conduct data augmentation where the cross-class transformations are mapped to a low dimensional latent space using conditional GAN

Data Augmentation Generative Adversarial Networks


Answering questions about data can require understanding what parts of an input X influence the response Y. Finding such an understanding can be built by testing relationships between variables through a machine learning model.

For example, conditional randomization tests help determine whether a variable relates to the response given the rest of the variables.

However, randomization tests require users to specify test statistics.

We formalize a class of proper test statistics that are guaranteed to select a feature when it provides information about the response even when the rest of the features are known.

We show that f-divergences provide a broad class of proper test statistics.

In the class of f-divergences, the KL-divergence yields an easy-to-compute proper test statistic that relates to the AMI.

Questions of feature importance can be asked at the level of an individual sample.

We show that estimators from the same AMI test can also be used to find important features in a particular instance.

We provide an example to show that perfect predictive models are insufficient for instance-wise feature selection.

We evaluate our method on several simulation experiments, on a genomic dataset, a clinical dataset for hospital readmission, and on a subset of classes in ImageNet.

Our method outperforms several baselines in various simulated datasets, is able to identify biologically significant genes, can select the most important predictors of a hospital readmission event, and is able to identify distinguishing features in an image-classification task.

We develop a simple regression-based model-agnostic feature selection method to interpret data generating processes with FDR control, and outperform several popular baselines on several simulated, medical, and image datasets.

This paper proposes a practical improvement of the conditional randomization test and a new test statistic, proves f-divergence is one possible choice, and shows that KL-divergence cancels out some conditional distributions.

This paper addresses the problem of finding useful features in an input that are dependent on a response variable even when conditioning on all other input variables.

A model agnostic method to provide interpretation on the influence of input features on the response of a machine level model down to instance level, and proper test statistics for model agnostic feature selection.

Model-Agnostic Feature Selection with Additional Mutual Information

model-agnostic feature selection
test statistic
response
method
input


Supervised learning depends on annotated examples, which are taken to be the ground truth.

But these labels often come from noisy crowdsourcing platforms, like Amazon Mechanical Turk.

Practitioners typically collect multiple labels per example and aggregate the results to mitigate noise (the classic crowdsourcing problem).

Given a fixed annotation budget and unlimited unlabeled data, redundant annotation comes at the expense of fewer labeled examples.

This raises two fundamental questions: (1) How can we best learn from noisy workers?

(2) How should we allocate our labeling budget to maximize the performance of a classifier?

We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data.

The alternating minimization proceeds in rounds, estimating worker quality from disagreement with the current model and then updating the model by optimizing a loss function that accounts for the current estimate of worker quality.

Unlike previous approaches, even with only one annotation per example, our algorithm can estimate worker quality.

We establish a generalization error bound for models learned with our algorithm and establish theoretically that it's better to label many examples once (vs less multiply) when worker quality exceeds a threshold.

Experiments conducted on both ImageNet (with simulated noisy workers) and MS-COCO (using the real crowdsourced labels) confirm our algorithm's benefits.

A new approach for learning a model from noisy crowdsourced annotations.

This paper proposes a method for learning from noisy labels, focusing on the case when data isn't redundantly labeled with theoretical and experimental validation

This paper focuses on the learning-from-crowds problem, where jointly updating the classifier weights and the confusion matrices of workers can help on the estimation problem with rare crowdsourced labels.

Proposes a supervised learning algorithm for modeling label and worker quality and utilizes algorithm to study how much redundancy is required in crowdsourcing and whether low redundancy with abundant noise examples lead to better labels.

Learning From Noisy Singly-labeled Data

focus learn model noisy label worker crowdsource datum


The reason why a mistake is made often remains a mystery.

As such neural networks often are considered a black box.

It would be useful to have a method that can give an explanation that is intuitive to a user as to why an image is misclassified.

In this paper we develop a method for explaining the mistakes of a classifier model by visually showing what must be added to an image such that it is correctly classified.

Our work combines the fields of adversarial examples, generative modeling and a correction technique based on difference target propagation to create an technique that creates explanations of why an image is misclassified.

In this paper we explain our method and demonstrate it on MNIST and CelebA.

This approach could aid in demystifying neural networks for a user.

New way of explaining why a neural network has misclassified an image

This paper proposes a method for explaining the classification mistakes of neural networks.

Aims to better understand the classification of neural networks and explores the latent space of a variational auto encoder and considers the perturbations of the latent space in order to obtain the correct classification.

Explaining the Mistakes of Neural Networks with Latent Sympathetic Examples

mistake neural network
explain
classification
latent


In the context of multi-task learning, neural networks with branched architectures have often been employed to jointly tackle the tasks at hand.

Such ramified networks typically start with a number of shared layers, after which different tasks branch out into their own sequence of layers.

Understandably, as the number of possible network configurations is combinatorially large, deciding what layers to share and where to branch out becomes cumbersome.

Prior works have either relied on ad hoc methods to determine the level of layer sharing, which is suboptimal, or utilized neural architecture search techniques to establish the network design, which is considerably expensive.

In this paper, we go beyond these limitations and propose a principled approach to automatically construct branched multi-task networks, by leveraging the employed tasks' affinities.

Given a specific budget, i.e. number of learnable parameters, the proposed approach generates architectures, in which shallow layers are task-agnostic, whereas deeper ones gradually grow more task-specific.

Extensive experimental analysis across numerous, diverse multi-tasking datasets shows that, for a given budget, our method consistently yields networks with the highest performance, while for a certain performance threshold it requires the least amount of learnable parameters.

A method for the automated construction of branched multi-task networks with strong experimental evaluation on diverse multi-tasking datasets.

This paper proposes a novel soft parameter sharing Multi-task Learning framework based on a tree-like structure.

This paper presents a method to infer multi-task networks architecture to determine which part of the network should be shared among different tasks.

Branched Multi-Task Networks: Deciding What Layers To Share


Typical recent neural network designs are primarily convolutional layers, but the tricks enabling structured efficient linear layers (SELLs) have not yet been adapted to the convolutional setting.

We present a method to express the weight tensor in a convolutional layer using diagonal matrices, discrete cosine transforms (DCTs) and permutations that can be optimised using standard stochastic gradient methods.

A network composed of such structured efficient convolutional layers (SECL) outperforms existing low-rank networks and demonstrates competitive computational efficiency.

It's possible to substitute the weight matrix in a convolutional layer to train it as a structured efficient layer; performing as well as low-rank decomposition.

This work applies previous Structured Efficient Linear Layers to conv layers and proposes Structured Efficient Convolutional Layers as substitution of original conv layers.

Training Structured Efficient Convolutional Layers

structure efficient convolutional layer
structured


Blind document deblurring is a fundamental task in the field of document processing and restoration, having wide enhancement applications in optical character recognition systems, forensics, etc.

Since this problem is highly ill-posed, supervised and unsupervised learning methods are well suited for this application.

Using various techniques, extensive work has been done on natural-scene deblurring.

However, these extracted features are not suitable for document images.

We present SVDocNet, an end-to-end trainable U-Net based spatial recurrent neural network (RNN) for blind document deblurring where the weights of the RNNs are determined by different convolutional neural networks (CNNs).

This network achieves state of the art performance in terms of both quantitative measures and qualitative results.

We present SVDocNet, an end-to-end trainable U-Net based spatial recurrent neural network (RNN) for blind document deblurring.

SVDocNet: Spatially Variant U-Net for Blind Document Deblurring


In contrast to the monolithic deep architectures used in deep learning today for computer vision, the visual cortex processes retinal images via two functionally distinct but interconnected networks: the ventral pathway for processing object-related information and the dorsal pathway for processing motion and transformations.

Inspired by this cortical division of labor and properties of the magno- and parvocellular systems, we explore an unsupervised approach to feature learning that jointly learns object features and their transformations from natural videos.

We propose a new convolutional bilinear sparse coding model that (1) allows independent feature transformations and (2) is capable of processing large images.

Our learning procedure leverages smooth motion in natural videos.

Our results show that our model can learn groups of features and their transformations directly from natural videos in a completely unsupervised manner.

The learned "dynamic filters" exhibit certain equivariance properties, resemble cortical spatiotemporal filters, and capture the statistics of transitions between video frames.

Our model can be viewed as one of the first approaches to demonstrate unsupervised learning of primary "capsules" (proposed by Hinton and colleagues for supervised learning) and has strong connections to the Lie group approach to visual perception.

We extend bilinear sparse coding and leverage video sequences to learn dynamic filters.

Learning a Convolutional Bilinear Sparse Code for Natural Videos


Conventional out-of-distribution (OOD) detection schemes based on variational autoencoder or Random Network Distillation (RND) are known to assign lower uncertainty to the OOD data than the target distribution.

In this work, we discover that such conventional novelty detection schemes are also vulnerable to the blurred images.

Based on the observation, we construct a novel RND-based OOD detector, SVD-RND, that utilizes blurred images during training.

Our detector is simple, efficient in test time, and outperforms baseline OOD detectors in various domains.

Further results show that SVD-RND learns a better target distribution representation than the baselines.

Finally, SVD-RND combined with geometric transform achieves near-perfect detection accuracy in CelebA domain.

We propose a novel OOD detector that employ blurred images as adversarial examples . Our model achieve significant OOD detection performance in various domains.

This paper presents the idea to use blurred images as regularizing examples to improve out-of-distribution detection performance based on Random Network Distillation.

This paper tackles out-of-data distribution by leveraging RND applied to data augmentations by training a model to match the outputs of a random network with an augmentation as input.

Novelty Detection Via Blurring

blur image
model
random network
example
detection performance


Training large deep neural networks on massive datasets is  computationally very challenging.

There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue.

The most prominent algorithm in this line of research is LARS, which by  employing layerwise adaptive learning rates trains ResNet on ImageNet in a few minutes.

However, LARS performs poorly for attention models like BERT, indicating that its performance gains are not consistent across tasks.

In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches.

Using this strategy, we develop a new layerwise adaptive large batch optimization technique called LAMB; we then provide convergence analysis of LAMB as well as LARS, showing convergence to a stationary point in general nonconvex settings.

Our empirical results demonstrate the superior performance of LAMB across various tasks such as BERT and ResNet-50 training with very little hyperparameter tuning.

In particular, for BERT training, our optimizer enables use of very large batch sizes of 32868 without any degradation of performance.

By increasing the batch size to the memory limit of a TPUv3 Pod, BERT training time can be reduced from 3 days to just 76 minutes (Table 1).

A fast optimizer for general applications and large-batch training.

In this paper, the authors made a study on large-batch training for the BERT, and successfully trained a BERT model in 76 minutes.

This paper develops a layerwise adaptation strategy that allows training BERT models with large 32k mini-batches vs baseline 512.

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

large-batch training
train bert model
76 minute
large


Model-agnostic meta-learning (MAML) is known as a powerful meta-learning method.

However, MAML is notorious for being hard to train because of the existence of two learning rates.

Therefore, in this paper, we derive the conditions that inner learning rate $\alpha$ and meta-learning rate $\beta$ must satisfy for MAML to converge to minima with some simplifications.

We find that the upper bound of $\beta$ depends on $ \alpha$, in contrast to the case of using the normal gradient descent method.

Moreover, we show that the threshold of $\beta$ increases as $\alpha$ approaches its own upper bound.

This result is verified by experiments on various few-shot tasks and architectures; specifically, we perform sinusoid regression and classification of Omniglot and MiniImagenet datasets with a multilayer perceptron and a convolutional neural network.

Based on this outcome, we present a guideline for determining the learning rates: first, search for the largest possible $\alpha$; next, tune $\beta$ based on the chosen value of $\alpha$.

We analyzed the role of two learning rates in model-agnostic meta-learning in convergence.

The authors tackled the optimization instability problem in MAML by investigating the two learning rates.

This paper studies a method to help tune the two learning rates used in the MAML training algorithm.

Role of two learning rates in convergence of model-agnostic meta-learning

role two learning rate
model-agnostic meta-learning
convergence
maml


We present a neural framework for learning associations between interrelated groups of words such as the ones found in Subject-Verb-Object (SVO) structures.

Our model induces a joint function-specific word vector space, where vectors of e.g. plausible SVO compositions lie close together.

The model retains information about word group membership even in the joint space, and can thereby effectively be applied to a number of tasks reasoning over the SVO structure.

We show the robustness and versatility of the proposed framework by reporting state-of-the-art results on the tasks of estimating selectional preference (i.e., thematic fit) and event similarity.

The results indicate that the combinations of representations learned with our task-independent model outperform task-specific architectures from prior work, while reducing the number of parameters by up to 95%.

The proposed framework is versatile and holds promise to support learning function-specific representations beyond the SVO structures.

Task-independent neural model for learning associations between interrelated groups of words.

The paper proposed a method for training function-specific word vectors, in which each word is represented with three vectors each in a different category (Subject-Verb-Object).

This paper proposes a neural network to learn function-specific work representations and demonstrates the advantage over alternatives.

Learning Function-Specific Word Representations

word learn neural function-specific representation


The fabrication of semiconductor involves etching process to remove selected areas from wafers.

However, the measurement of etched structure in micro-graph heavily relies on time-consuming manual routines.

Traditional image processing usually demands on large number of annotated data and the performance is still poor.

We treat this challenge as segmentation problem and use deep learning approach to detect masks of objects in etched structure of wafer.

Then, we use simple image processing to carry out automatic measurement on the objects.

We attempt Generative Adversarial Network (GAN) to generate more data to overcome the problem of very limited dataset.

We download 10 SEM (Scanning Electron Microscope) images of 4 types from Internet, based on which we carry out our experiments.

Our deep learning based method demonstrates superiority over image processing approach with  mean accuracy reaching over 96% for the measurements, compared with the ground truth.

To the best of our knowledge, it is the first time that deep learning has been applied in semiconductor industry for automatic measurement.

Using deep learning method to carry out automatic measurement of SEM images in semiconductor industry

Automatic Measurement on Etched Structure in Semiconductor Using Deep Learning Approach

deep learning
automatic measurement
semiconductor


Generating and scheduling activities is particularly challenging

when considering both consumptive resources and

complex resource interactions such as time-dependent resource

usage.We present three methods of determining valid

temporal placement intervals for an activity in a temporally

grounded plan in the presence of such constraints.

the Max Duration and Probe algorithms which are

sound, but incomplete, and the Linear algorithm which is

sound and complete for linear rate resource consumption.

We apply these techniques to the problem of scheduling

awakes for a planetary rover where the awake durations

Probe algorithm performs competitively with the Linear algorithm

given an advantageous problem space and well-defined

We show that the Probe and Linear algorithms

outperform the Max Duration algorithm empirically.

We then empirically present the runtime differences between

The Probe algorithm is currently base-lined

for use in the onboard scheduler for NASA’s next planetary

This paper describes and analyzes three methods to schedule non-fixed duration activities in the presence of consumptive resources.

The paper presents three approaches for on-board scheduling of activities in a planetary rover under reservoir resource constraints.

Scheduling with Complex Consumptive Resources for a Planetary Rover

consumptive resource
planetary rover
activity
scheduling
three


A disentangled representation of a data set should be capable of recovering the underlying factors that generated it.

One question that arises is whether using Euclidean space for latent variable models can produce a disentangled representation when the underlying generating factors have a certain geometrical structure.

Take for example the images of a car seen from different angles.

The angle has a periodic structure but a 1-dimensional representation would fail to capture this topology.

The submissions presented for the first stage of the  NeurIPS2019 Disentanglement Challenge consist of a Diffusion Variational Autoencoder ($\Delta$VAE) with a hyperspherical latent space which can for example recover periodic true factors.

The training of the $\Delta$VAE is enhanced by incorporating a modified version of the Evidence Lower Bound (ELBO) for tailoring the encoding capacity of the posterior approximate.

Description of submission to NeurIPS2019 Disentanglement Challenge based on hyperspherical variational autoencoders

Disentanglement with Hyperspherical Latent Spaces using Diffusion Variational Autoencoders

hyperspherical
variational autoencoder
disentanglement


Anomaly detection, finding patterns that substantially deviate from those seen previously, is one of the fundamental problems of artificial intelligence.

Recently, classification-based methods were shown to achieve superior results on this task.

In this work, we present a unifying view and propose an open-set method to relax current generalization assumptions.

Furthermore, we extend the applicability of transformation-based methods to non-image data using random affine transformations.

Our method is shown to obtain state-of-the-art accuracy and is applicable to broad data types.

The strong performance of our method is extensively validated on multiple datasets from different domains.

An anomaly detection that:  uses random-transformation classification for generalizing to non-image data.

This paper proposes a deep method for anomaly detection that unifies recent deep one-class classification and transformation-based classification approaches.

This paper proposes an approach to classification-based anomaly detection for general data by using the affine transformation y = Wx+b.

Classification-Based Anomaly Detection for General Data

anomaly detection general datum
classification
approach


Recent improvements in large-scale language models have driven progress on automatic generation of syntactically and semantically consistent text for many real-world applications.

Many of these advances leverage the availability of large corpora.

While training on such corpora encourages the model to understand long-range dependencies in text, it can also result in the models internalizing the social biases present in the corpora.

This paper aims to quantify and reduce biases exhibited by language models.

Given a conditioning context (e.g. a writing prompt) and a language model, we analyze if (and how) the sentiment of the generated text is affected by changes in values of sensitive attributes (e.g. country names, occupations, genders, etc.) in the conditioning context, a.k.a. counterfactual evaluation.

We quantify these biases by adapting individual and group fairness metrics from the fair machine learning literature.

Extensive evaluation on two different corpora (news articles and Wikipedia) shows that state-of-the-art Transformer-based language models exhibit biases learned from data.

We propose embedding-similarity and sentiment-similarity regularization methods that improve both individual and group fairness metrics without sacrificing perplexity and semantic similarity---a positive step toward development and deployment of fairer language models for real-world applications.

We reduce sentiment biases based on counterfactual evaluation of text generation using language models.

This paper measures sentiment bias in language models as reflected by text generated by the models, and adds other objective terms to the usual language modeling objective to reduce bias.

This paper proposes to evaluate bias in pre-trained language models by using a fixed sentiment system and tests several different prefix templates.

A method based on semantic simiilarity and a method based on sentiment similarity to debias the neural language models trained from large datasets.

Reducing Sentiment Bias in Language Models via Counterfactual Evaluation

sentiment bias language model
base
text
counterfactual evaluation
reduce


Topic modeling of text documents is one of the most important tasks in representation learning.

In this work, we propose iTM-VAE, which is a Bayesian nonparametric (BNP) topic model with variational auto-encoders.

On one hand, as a BNP topic model, iTM-VAE potentially has infinite topics and can adapt the topic number to data automatically.

On the other hand, different with the other BNP topic models, the inference of iTM-VAE is modeled by neural networks, which has rich representation capacity and can be computed in a simple feed-forward manner.

Two variants of iTM-VAE are also proposed in this paper, where iTM-VAE-Prod models the generative process in products-of-experts fashion for better performance and iTM-VAE-G places a prior over the concentration parameter such that the model can adapt a suitable concentration parameter to data automatically.

Experimental results on 20News and Reuters RCV1-V2 datasets show that the proposed models outperform the state-of-the-arts in terms of perplexity, topic coherence and document retrieval tasks.

Moreover, the ability of adjusting the concentration parameter to data is also confirmed by experiments.

A Bayesian Nonparametric Topic Model with Variational Auto-Encoders which achieves the state-of-the-arts on public benchmarks in terms of perplexity, topic coherence and retrieval tasks.

This paper constructs an infinite Topic Model with Variational Auto-Encoders by combining Nalisnick & Smith's stick-breaking variational auto-encoder with latent Dirichlet allocation and several inference techniques used in Miao.

bayesian nonparametric topic model variational auto-encoder


Knowledge Distillation (KD) is a widely used technique in recent deep learning research to obtain small and simple models whose performance is on a par with their large and complex counterparts.

Standard Knowledge Distillation tends to be time-consuming because of the training time spent to obtain a teacher model that would then provide guidance for the student model.

It might be possible to cut short the time by training a teacher model on the fly, but it is not trivial to have such a high-capacity teacher that gives quality guidance to student models this way.

To improve this, we present a novel framework of Knowledge Distillation exploiting dark knowledge from the whole training set.

In this framework, we propose a simple and effective implementation named Distillation by Utilizing Peer Samples (DUPS) in one generation.

We verify our algorithm on numerous experiments.

Compared with standard training on modern architectures, DUPS achieves an average improvement of 1%-2% on various tasks with nearly zero extra cost.

Considering some typical Knowledge Distillation methods which are much more time-consuming, we also get comparable or even better performance using DUPS.

We present a novel framework of Knowledge Distillation utilizing peer samples as the teacher

Proposes a method for improving the effectiveness of knowledge distillation by softening the labels used and employing a dataset instead of a single sample.

This paper proposes to address the extra computational cost of training with knowledge distillation, building on the recently proposed Snapshot Distillation technique.

One Generation Knowledge Distillation by Utilizing Peer Samples

knowledge distillation utilize peer sample


We develop a metalearning approach for learning hierarchically structured poli- cies, improving sample efficiency on unseen tasks through the use of shared primitives—policies that are executed for large numbers of timesteps.

Specifi- cally, a set of primitives are shared within a distribution of tasks, and are switched between by task-specific policies.

We provide a concrete metric for measuring the strength of such hierarchies, leading to an optimization problem for quickly reaching high reward on unseen tasks.

We then present an algorithm to solve this problem end-to-end through the use of any off-the-shelf reinforcement learning method, by repeatedly sampling new tasks and resetting task-specific policies.

We successfully discover meaningful motor primitives for the directional movement of four-legged robots, solely by interacting with distributions of mazes.

We also demonstrate the transferability of primitives to solve long-timescale sparse-reward obstacle courses, and we enable 3D humanoid robots to robustly walk and crawl with the same policy.

learn hierarchal sub-policies through end-to-end training over a distribution of tasks

The authors consider the problem of learning a useful set of ‘sub policies’ that can be shared between tasks so as to jump start learning on new tasks drawn from the task distribution.

This paper proposes a novel method for inducing temporal hierarchical structure in a specialized multi-task setting.

META LEARNING SHARED HIERARCHIES


This paper proposes a new model for document embedding.

Existing approaches either require complex inference or use recurrent neural networks that are difficult to parallelize.

We take a different route and use recent advances in language modeling to develop a convolutional neural network embedding model.

This allows us to train deeper architectures that are fully parallelizable.

Stacking layers together increases the receptive filed allowing each successive layer to model increasingly longer range semantic dependences within the document.

Empirically we demonstrate superior results on two publicly available benchmarks.

Full code will be released with the final version of this paper.

Convolutional neural network model for unsupervised document embedding.

Introduces a new model for the general task of inducing document representations (embeddings) which uses a CNN architecture to improve computational efficiency.

This paper proposes using CNNs with a skip-gram like objective as a fast way to output document embeddings

Learning Document Embeddings With CNNs


We prove bounds on the generalization error of convolutional networks.

The bounds are in terms of the training loss, the number of

parameters, the Lipschitz constant of the loss and the distance from

number of pixels in the input, and the height and width of hidden

We present experiments with CIFAR-10, along with varying

hyperparameters of a deep convolutional network, comparing our bounds

We prove generalization bounds for convolutional neural networks that take account of weight-tying

Studies the generalization power of CNNs and improves the upper bounds of generalization errors, showing correlation between the generalization error of learned CNNs and the upper bound's dominant term.

This paper presents a generalization bound for convolutional neural networks based on the number of parameters, the Lipschitz constant, and the distance of the final weights from initialization.

Size-free generalization bounds for convolutional neural networks

generalization bound convolutional neural network


MobileNets family of computer vision neural networks have fueled tremendous progress in the design and organization of resource-efficient architectures in recent years.

New applications with stringent real-time requirements in highly constrained devices require further compression of MobileNets-like already computeefficient networks.

Model quantization is a widely used technique to compress and accelerate neural network inference and prior works have quantized MobileNets to 4 − 6 bits albeit with a modest to significant drop in accuracy.

While quantization to sub-byte values (i.e. precision ≤ 8 bits) has been valuable, even further quantization of MobileNets to binary or ternary values is necessary to realize significant energy savings and possibly runtime speedups on specialized hardware, such as ASICs and FPGAs.

Under the key observation that convolutional filters at each layer of a deep neural network may respond differently to ternary quantization, we propose a novel quantization method that generates per-layer hybrid filter banks consisting of full-precision and ternary weight filters for MobileNets.

The layer-wise hybrid filter banks essentially combine the strengths of full-precision and ternary weight filters to derive a compact, energy-efficient architecture for MobileNets.

Using this proposed quantization method, we quantized a substantial portion of weight filters of MobileNets to ternary values resulting in 27.98% savings in energy, and a 51.07% reduction in the model size, while achieving comparable accuracy and no degradation in throughput on specialized hardware in comparison to the baseline full-precision MobileNets.

2x savings in model size, 28% energy reduction for MobileNets on ImageNet at no loss in accuracy using hybrid layers composed of conventional full-precision filters and ternary filters

Focuses on quantizing the MobileNets architecture to ternary values, lowering the required space and computation in order to make neural networks more energy efficient.

The paper proposes layer-wise hybrid filter bank which only quantizes a fraction of convolutional filters to ternary values towards the MobileNets architecture.

Ternary MobileNets via Per-Layer Hybrid Filter Banks

mobilenet architecture
filter ternary
quantize
hybrid
energy
value
bank


Performing controlled experiments on noisy data is essential in thoroughly understanding deep learning across a spectrum of noise levels.

Due to the lack of suitable datasets, previous research have only examined deep learning on controlled synthetic noise, and real-world noise has never been systematically studied in a controlled setting.

To this end, this paper establishes a benchmark of real-world noisy labels at 10 controlled noise levels.

As real-world noise possesses unique properties, to understand the difference, we conduct a large-scale study across a variety of noise levels and types, architectures, methods, and training settings.

Our study shows that: (1) Deep Neural Networks (DNNs) generalize much better on real-world noise.

(2) DNNs may not learn patterns first on real-world noisy data.

(3) When networks are fine-tuned, ImageNet architectures generalize well on noisy data.

(4) Real-world noise appears to be less harmful, yet it is more difficult for robust DNN methods to improve.

(5) Robust learning methods that work well on synthetic noise may not work as well on real-world noise, and vice versa.

We hope our benchmark, as well as our findings, will facilitate deep learning research on noisy data.

We establish a benchmark of controlled real noise and reveal several interesting findings about real-world noisy data.

This paper compares 6 existing noisy label learning methods in two training settings: from scratch, and finetuning.

The authors establish a large dataset and benchmark of controlled real-world noise for performing controlled experiments on noisy data in deep learning.

Synthetic vs Real: Deep Learning on Controlled Noise

benchmark control
deep learning
noisy datum
noise
establish
real
real-world


Designing RNA molecules has garnered recent interest in medicine, synthetic biology, biotechnology and bioinformatics since many functional RNA molecules were shown to be involved in regulatory processes for transcription, epigenetics and translation.

Since an RNA's function depends on its structural properties, the RNA Design problem is to find an RNA sequence which satisfies given structural constraints.

Here, we propose a new algorithm for the RNA Design problem, dubbed LEARNA.

LEARNA uses deep reinforcement learning to train a policy network to sequentially design an entire RNA sequence given a specified target structure.

By meta-learning across 65000 different RNA Design tasks for one hour on 20 CPU cores, our extension Meta-LEARNA constructs an RNA Design policy that can be applied out of the box to solve novel RNA Design tasks.

Methodologically, for what we believe to be the first time, we jointly optimize over a rich space of architectures for the policy network, the hyperparameters of the training procedure and the formulation of the decision process.

Comprehensive empirical results on two widely-used RNA Design benchmarks, as well as a third one that we introduce, show that our approach achieves new state-of-the-art performance on the former while also being orders of magnitudes faster in reaching the previous state-of-the-art performance.

In an ablation study, we analyze the importance of our method's different components.

We learn to solve the RNA Design problem with reinforcement learning using meta learning and autoML approaches.

Used policy gradient optimization for generating RNA sequences which fold into a target secondary structure, resulting in clear accuracy and runtime improvements.

Learning to Design RNA


Pruning is a popular technique for compressing a neural network: a large pre-trained network is fine-tuned while connections are successively removed.

However, the value of pruning has largely evaded scrutiny.

In this extended abstract, we examine residual networks obtained through Fisher-pruning and make two interesting observations.

First, when time-constrained, it is better to train a simple, smaller network from scratch than prune a large network.

Second, it is the architectures obtained through the pruning process  --- not the learnt weights --- that prove valuable.

Such architectures are powerful when trained from scratch.

Furthermore, these architectures are easy to approximate without any further pruning: we can prune once and obtain a family of new, scalable network architectures for different memory requirements.

Training small networks beats pruning, but pruning finds good small networks to train that are easy to copy.

Pruning neural networks: is it time to nip it in the bud?


Supervised learning problems---particularly those involving social data---are often subjective.

That is, human readers, looking at the same data, might come to legitimate but completely different conclusions based on their personal experiences.

Yet in machine learning settings feedback from multiple human annotators is often reduced to a single ``ground truth'' label, thus hiding the true, potentially rich and diverse interpretations of the data found across the social spectrum.

We explore the rewards and challenges of discovering and learning representative distributions of the labeling opinions of a large human population.

A major, critical cost to this approach is the number of humans needed to provide enough labels not only to obtain representative samples but also to train a machine to predict representative distributions on unlabeled data.

We propose aggregating label distributions over, not just individuals, but also data items, in order to maximize the costs of humans in the loop.

We test different aggregation approaches on state-of-the-art deep learning models.

Our results suggest that careful label aggregation methods can greatly reduce the number of samples needed to obtain representative distributions.

We study the problem of learning to predict the underlying diversity of beliefs present in supervised learning domains.

Aggregating Crowdsourced Labels in Subjective Domains


Recent advancements in deep learning techniques such as Convolutional Neural Networks(CNN) and Generative Adversarial Networks(GAN) have achieved breakthroughs in the problem of semantic image inpainting, the task of reconstructing missing pixels in given images.

While much more effective than conventional approaches, deep learning models require large datasets and great computational resources for training, and inpainting quality varies considerably when training data vary in size and diversity.

To address these problems, we present in this paper a inpainting strategy of \textit{Comparative Sample Augmentation}, which enhances the quality of training set by filtering out irrelevant images and constructing additional images using information about the surrounding regions of the images to be inpainted.

Experiments on multiple datasets demonstrate that our method extends the applicability of deep inpainting models to training sets with varying sizes, while maintaining inpainting quality as measured by qualitative and quantitative metrics for a large class of deep models, with little need for model-specific consideration.

We introduced a strategy which enables inpainting models on datasets of various sizes

Help image inpainting using GANs by using a comparative augmenting filter and adding random noise to each pixel.

Deep Generative Inpainting with Comparative Sample Augmentation