We present a novel approach for training neural abstract architectures which in- corporates (partial) supervision over the machine’s interpretable components.

To cleanly capture the set of neural architectures to which our method applies, we introduce the concept of a differential neural computational machine (∂NCM) and show that several existing architectures (e.g., NTMs, NRAMs) can be instantiated as a ∂NCM and can thus benefit from any amount of additional supervision over their interpretable components.

Based on our method, we performed a detailed experimental evaluation with both, the NTM and NRAM architectures, and showed that the approach leads to significantly better convergence and generalization capabilities of the learning phase than when training using only input-output examples.

We increase the amount of trace supervision possible to utilize when training fully differentiable neural machine architectures.

Training Neural Machines with Partial Traces


Bayesian learning of model parameters in neural networks is important in scenarios where estimates with well-calibrated uncertainty are important.

In this paper, we propose Bayesian quantized networks (BQNs), quantized neural networks (QNNs) for which we learn a posterior distribution over their discrete parameters.

We provide a set of efficient algorithms for learning and prediction in BQNs without the need to sample from their parameters or activations, which not only allows for differentiable learning in quantized models but also reduces the variance in gradients estimation.

We evaluate BQNs on MNIST, Fashion-MNIST and KMNIST classification datasets compared against bootstrap ensemble of QNNs (E-QNN).

We demonstrate BQNs achieve both lower predictive errors and better-calibrated uncertainties than E-QNN (with less than 20% of the negative log-likelihood).

We propose Bayesian quantized networks, for which we learn a posterior distribution over their quantized parameters.

Sampling-Free Learning of Bayesian Quantized Neural Networks


Injecting adversarial examples during training, known as adversarial training, can improve robustness against one-step attacks, but not for unknown iterative attacks.

To address this challenge, we first show iteratively generated adversarial images easily transfer between networks trained with the same strategy.

Inspired by this observation, we propose cascade adversarial training, which transfers the knowledge of the end results of adversarial training.

We train a network from scratch by injecting iteratively generated adversarial images crafted from already defended networks in addition to one-step adversarial images from the network being trained.

We also propose to utilize embedding space for both classification and low-level (pixel-level) similarity learning to ignore unknown pixel level perturbation.

During training, we inject adversarial images without replacing their corresponding clean images and penalize the distance between the two embeddings (clean and adversarial).

Experimental results show that cascade adversarial training together with our proposed low-level similarity learning efficiently enhances the robustness against iterative attacks, but at the expense of decreased robustness against one-step attacks.

We show that combining those two techniques can also improve robustness under the worst case black box attack scenario.

Cascade adversarial training + low level similarity learning improve robustness against both white box and black box attacks.

Cascade Adversarial Machine Learning Regularized with a Unified Embedding


Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities ``solve'' the exploding gradient problem, we show that this is not the case and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice.

We explain why exploding gradients occur and highlight the {\it collapsing domain problem}, which can arise in architectures that avoid exploding gradients.

ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks, which we show is a consequence of a surprising mathematical property.

By noticing that {\it any neural network is a residual network}, we devise the {\it residual trick}, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.

We show that in contras to popular wisdom, the exploding gradient problem has not been solved and that it limits the depth to which MLPs can be effectively trained. We show why gradients explode and how ResNet handles them.

Gradients explode - Deep Networks are shallow - ResNet explained


In this paper, we are interested in two seemingly different concepts: \textit{adversarial training} and \textit{generative adversarial networks (GANs)}.

Particularly, how these techniques work to improve each other.

To this end, we analyze the limitation of adversarial training as a defense method, starting from questioning how well the robustness of a model can generalize.

Then, we successfully improve the generalizability via data augmentation by the ``fake'' images sampled from generative adversarial network.

After that, we are surprised to see that the resulting robust classifier leads to a better generator, for free.

We intuitively explain this interesting phenomenon and leave the theoretical analysis for future work.

Motivated by these observations, we propose a system that combines generator, discriminator, and adversarial attacker together in a single network.

After end-to-end training and fine tuning, our method can simultaneously improve the robustness of classifiers, measured by accuracy under strong adversarial attacks, and the quality of generators, evaluated both aesthetically and quantitatively.

In terms of the classifier, we achieve better robustness than the state-of-the-art adversarial training algorithm proposed in (Madry \textit{et al.}, 2017), while our generator achieves competitive performance compared with SN-GAN (Miyato and Koyama, 2018).

We found adversarial training not only speeds up the GAN training but also increases the image quality

From Adversarial Training to Generative Adversarial Networks


We present a method to train self-binarizing neural networks, that is, networks that evolve their weights and activations during training to become binary.

To obtain similar binary networks, existing methods rely on the sign activation function.

This function, however, has no gradients for non-zero values, which makes standard backpropagation impossible.

To circumvent the difficulty of training a network relying on the sign activation function, these methods alternate between floating-point and binary representations of the network during training, which is sub-optimal and inefficient.

We approach the binarization task by training on a unique representation involving a smooth activation function, which is iteratively sharpened during training until it becomes a binary representation equivalent to the sign activation function.

Additionally, we introduce a new technique to perform binary batch normalization that simplifies the conventional batch normalization by transforming it into a simple comparison operation.

This is unlike existing methods, which are forced to the retain the conventional floating-point-based batch normalization.

Our binary networks, apart from displaying advantages of lower memory and computation as compared to conventional floating-point and binary networks, also show higher classification accuracy than existing state-of-the-art methods on multiple benchmark datasets.

A method to binarize both weights and activations of a deep neural network that is efficient in computation and memory usage and performs better than the state-of-the-art.

Self-Binarizing Networks


In many applications, the training data for a machine learning task is partitioned across multiple nodes, and aggregating this data may be infeasible due to storage, communication, or privacy constraints.

In this work, we present Good-Enough Model Spaces (GEMS), a novel framework for learning a global satisficing (i.e. "good-enough") model within a few communication rounds by carefully combining the space of local nodes' satisficing models.

In experiments on benchmark and medical datasets, our approach outperforms other baseline aggregation techniques such as ensembling or model averaging, and performs comparably to the ideal non-distributed models.

We present Good-Enough Model Spaces (GEMS), a framework for learning an aggregate model over distributed nodes within a small number of communication rounds.

Model Aggregation via Good-Enough Model Spaces


We propose the Wasserstein Auto-Encoder (WAE)---a new algorithm for building a generative model of the data distribution.

WAE minimizes a penalized form of the Wasserstein distance between the model distribution and the target distribution, which leads to a different regularizer than the one used by the Variational Auto-Encoder (VAE).

This regularizer encourages the encoded training distribution to match the prior.

We compare our algorithm with several other techniques and show that it is a generalization of adversarial auto-encoders (AAE).

Our experiments show that WAE shares many of the properties of VAEs (stable training, encoder-decoder architecture, nice latent manifold structure) while generating samples of better quality.

We propose a new auto-encoder based on the Wasserstein distance, which improves on the sampling properties of VAE.

Wasserstein Auto-Encoders


In this paper, we aim to develop a novel mechanism to preserve differential privacy (DP) in adversarial learning for deep neural networks, with provable robustness to adversarial examples.

We leverage the sequential composition theory in DP, to establish a new connection between DP preservation and provable robustness.

To address the trade-off among model utility, privacy loss, and robustness, we design an original, differentially private, adversarial objective function, based on the post-processing property in DP, to tighten the sensitivity of our model.

An end-to-end theoretical analysis and thorough evaluations show that our mechanism notably improves the robustness of DP deep neural networks.

Preserving Differential Privacy in Adversarial Learning with Provable Robustness to Adversarial Examples

differential privacy adversarial learning provable robustness


In high-dimensional reinforcement learning settings with sparse rewards, performing

effective exploration to even obtain any reward signal is an open challenge.

While model-based approaches hold promise of better exploration via planning, it

is extremely difficult to learn a reliable enough Markov Decision Process (MDP)

in high dimensions (e.g., over 10^100 states).

an abstract MDP over a much smaller number of states (e.g., 10^5), which we can

We assume we have an abstraction function

that maps concrete states (e.g., raw pixels) to abstract states (e.g., agent position,

In our approach, a manager maintains an abstract

MDP over a subset of the abstract states, which grows monotonically through targeted

exploration (possible due to the abstract MDP).

worker policy to travel between abstract states; the worker deals with the messiness

of concrete states and presents a clean abstraction to the manager.

the hardest games from the Arcade Learning Environment (Montezuma's,

Pitfall!, and Private Eye), our approach outperforms the previous

state-of-the-art by over a factor of 2 in each game.

the first to achieve superhuman performance without demonstrations.

We automatically construct and explore a small abstract Markov Decision Process, enabling us to achieve state-of-the-art results on Montezuma's Revenge, Pitfall!, and Private Eye by a significant margin.

Learning Abstract Models for Long-Horizon Exploration


Most deep reinforcement learning (RL) systems are not able to learn effectively from off-policy data, especially if they cannot explore online in the environment.

This is a critical shortcoming for applying RL to real-world problems where collecting data is expensive, and models must be tested offline before being deployed to interact with the environment -- e.g. systems that learn from human interaction.

Thus, we develop a novel class of off-policy batch RL algorithms which use KL-control to penalize divergence from a pre-trained prior model of probable actions.

This KL-constraint reduces extrapolation error, enabling effective offline learning, without exploration, from a fixed batch of data.

We also use dropout-based uncertainty estimates to lower bound the target Q-values as a more efficient alternative to Double Q-Learning.

This Way Off-Policy (WOP) algorithm is tested on both traditional RL tasks from OpenAI Gym, and on the problem of open-domain dialog generation; a challenging reinforcement learning problem with a 20,000 dimensional action space.

WOP allows for the extraction of multiple different reward functions post-hoc from collected human interaction data, and can learn effectively from all of these.

We test real-world generalization by deploying dialog models live to converse with humans in an open-domain setting, and demonstrate that WOP achieves significant improvements over state-of-the-art prior methods in batch deep RL.

We show that KL-control from a pre-trained prior can allow RL models to learn from a static batch of collected data, without the ability to explore online in the environment.

Way Off-Policy Batch Deep Reinforcement Learning of Human Preferences in Dialog


Transfer and adaptation to new unknown environmental dynamics is a key challenge for reinforcement learning (RL).

An even greater challenge is performing near-optimally in a single attempt at test time, possibly without access to dense rewards, which is not addressed by current methods that require multiple experience rollouts for adaptation.

To achieve single episode transfer in a family of environments with related dynamics, we propose a general algorithm that optimizes a probe and an inference model to rapidly estimate underlying latent variables of test dynamics, which are then immediately used as input to a universal control policy.

This modular approach enables integration of state-of-the-art algorithms for variational inference or RL.

Moreover, our approach does not require access to rewards at test time, allowing it to perform in settings where existing adaptive approaches cannot.

In diverse experimental domains with a single episode test constraint, our method significantly outperforms existing adaptive approaches and shows favorable performance against baselines for robust transfer.

Single episode policy transfer in a family of environments with related dynamics, via optimized probing for rapid inference of latent variables and immediate execution of a universal policy.

Single Episode Policy Transfer in Reinforcement Learning


Domain specific goal-oriented dialogue systems typically require modeling three types of inputs, viz.,

(i) the knowledge-base associated with the domain,

(ii) the history of the conversation, which is a sequence of utterances and

(iii) the current utterance for which the response needs to be generated.

While modeling these inputs, current state-of-the-art models such as Mem2Seq typically ignore the rich structure inherent in the knowledge graph and the sentences in the conversation context.

Inspired by the recent success of structure-aware Graph Convolutional Networks (GCNs) for various NLP tasks such as machine translation, semantic role labeling and document dating, we propose a memory augmented GCN for goal-oriented dialogues.

(i) the entity relation graph in a knowledge-base  and

(ii) the dependency graph associated with an utterance to compute richer representations for words and entities.

Further, we take cognizance of the fact that in certain situations, such as, when the conversation is in a code-mixed language, dependency parsers may not be available.

We show that in such situations we could use the global word co-occurrence graph and use it to enrich the representations of utterances.

We experiment with the modified DSTC2 dataset and its recently released code-mixed versions in four languages and show that our method outperforms existing state-of-the-art methods, using a wide range of evaluation metrics.

We propose a Graph Convolutional Network based encoder-decoder model with sequential attention for goal-oriented dialogue systems.

Graph Convolutional Network with Sequential Attention For Goal-Oriented Dialogue Systems

sequential attention goal-oriented dialogue system


Effectively capturing graph node sequences in the form of vector embeddings is critical to many applications.

(i) first learning vector embeddings of single graph nodes and

(ii) then composing them to compactly represent node sequences.

Specifically, we propose SENSE-S (Semantically Enhanced Node Sequence Embedding - for Single nodes), a skip-gram based novel embedding mechanism, for single graph nodes that co-learns graph structure as well as their textual descriptions.

We demonstrate that SENSE-S vectors increase the accuracy of multi-label classification tasks by up to 50% and link-prediction tasks by up to 78% under a variety of scenarios using real datasets.

Based on SENSE-S, we next propose generic SENSE to compute composite vectors that represent a sequence of nodes, where preserving the node order is important.

We prove that this approach is efficient in embedding node sequences, and our experiments on real data confirm its high accuracy in node order decoding.

Node sequence embedding mechanism that captures both graph and text properties.

SENSE: SEMANTICALLY ENHANCED NODE SEQUENCE EMBEDDING


Recent evidence shows that convolutional neural networks (CNNs) are biased towards textures so that CNNs are non-robust to adversarial perturbations over textures, while traditional robust visual features like SIFT (scale-invariant feature transforms) are designed to be robust across a substantial range of affine distortion, addition of noise, etc with the mimic of human perception nature.

This paper aims to leverage good properties of SIFT to renovate CNN architectures towards better accuracy and robustness.

We borrow the scale-space extreme value idea from SIFT, and propose EVPNet (extreme value preserving network) which contains three novel components to model the extreme values: (1) parametric differences of Gaussian (DoG) to extract extrema, (2) truncated ReLU to suppress non-stable extrema and (3) projected normalization layer (PNL) to mimic PCA-SIFT like feature normalization.

Experiments demonstrate that EVPNets can achieve similar or better accuracy than conventional CNNs, while achieving much better robustness on a set of adversarial attacks (FGSM,PGD,etc) even without adversarial training.

This paper aims to leverage good properties of robust visual features like SIFT to renovate CNN architectures towards better accuracy and robustness.

Extreme Values are Accurate and Robust in Deep Networks


Compressed representations generalize better (Shamir et al., 2010), which may be crucial when learning from limited or noisy labeled data.

The Information Bottleneck (IB) method (Tishby et al. (2000)) provides an insightful and principled approach for balancing compression and prediction in representation learning.

The IB objective I(X; Z) − βI(Y ; Z) employs a Lagrange multiplier β to tune this trade-off.

However, there is little theoretical guidance for how to select β.

There is also a lack of theoretical understanding about the relationship between β, the dataset, model capacity, and learnability.

In this work, we show that if β is improperly chosen, learning cannot happen: the trivial representation P(Z|X) = P(Z) becomes the global minimum of the IB objective.

We show how this can be avoided, by identifying a sharp phase transition between the unlearnable and the learnable which arises as β varies.

This phase transition defines the concept of IB-Learnability.

We prove several sufficient conditions for IB-Learnability, providing theoretical guidance for selecting β.

We further show that IB-learnability is determined by the largest confident, typical, and imbalanced subset of the training examples.

We give a practical algorithm to estimate the minimum β for a given dataset.

We test our theoretical results on synthetic datasets, MNIST, and CIFAR10 with noisy labels, and make the surprising observation that accuracy may be non-monotonic in β.

Theory predicts the phase transition between unlearnable and learnable values of beta for the Information Bottleneck objective

Learnability for the Information Bottleneck


We consider a new class of \emph{data poisoning} attacks on neural networks, in which the attacker takes control of a model by making small perturbations to a subset of its training data.

We formulate the task of finding poisons as a bi-level optimization problem, which can be solved using methods borrowed from the meta-learning community.

Unlike previous poisoning strategies, the meta-poisoning can poison networks that are trained from scratch using an initialization unknown to the attacker and transfer across hyperparameters.

Further we show that our attacks are more versatile: they can cause misclassification of the target image into an arbitrarily chosen class.

Our results show above 50% attack success rate when poisoning just 3-10% of the training dataset.

Generate corrupted training images that are imperceptible yet change CNN behavior on a target during any new training.

MetaPoison:   Learning to craft adversarial poisoning examples via meta-learning


We give a new algorithm for learning a two-layer neural network under a very general class of input distributions.

Assuming there is a ground-truth two-layer network

where A, W are weight matrices, \xi represents noise, and the number of neurons in the hidden layer is no larger than the input or output,  our algorithm is guaranteed to recover the parameters A, W of the ground-truth network.

The only requirement on the input x is that it is symmetric, which still allows highly complicated and structured input.

Our algorithm is based on the method-of-moments framework and extends several results in tensor decompositions.

We use spectral algorithms to avoid the complicated non-convex optimization in learning neural networks.

Experiments show that our algorithm can robustly learn the ground-truth neural network with a small number of samples for many symmetric input distributions.

We give an algorithm for learning a two-layer neural network with symmetric input distribution.

Learning Two-layer Neural Networks with Symmetric Inputs

two-layer neural network symmetric input


Teachers intentionally pick the most informative examples to show their students.

However, if the teacher and student are neural networks, the examples that the teacher network learns to give, although effective at teaching the student, are typically uninterpretable.

We show that training the student and teacher iteratively, rather than jointly, can produce interpretable teaching strategies.

We evaluate interpretability by (1) measuring the similarity of the teacher's emergent strategies to intuitive strategies in each domain and (2) conducting human experiments to evaluate how effective the teacher's strategies are at teaching humans.

We show that the teacher network learns to select or generate interpretable, pedagogical examples to teach rule-based, probabilistic, boolean, and hierarchical concepts.

We show that training a student and teacher network iteratively, rather than jointly, can produce emergent, interpretable teaching strategies.

Interpretable and Pedagogical Examples


Stochastic gradient descent (SGD), which  trades off noisy gradient updates for computational efficiency, is the de-facto optimization algorithm to solve large-scale machine learning problems.

SGD can make rapid learning progress by performing updates using subsampled training data, but the noisy updates also lead to slow asymptotic convergence.

Several variance reduction algorithms, such as SVRG, introduce control variates to obtain a lower variance gradient estimate and faster convergence.

Despite their appealing asymptotic guarantees, SVRG-like algorithms have not been widely adopted in deep learning.

The traditional asymptotic analysis in stochastic optimization provides limited insight into training deep learning models under a fixed number of epochs.

In this paper, we present a non-asymptotic analysis of SVRG under a noisy least squares regression problem.

Our primary focus is to compare the exact loss of SVRG to that of SGD at each iteration t.

We show that the learning dynamics of our regression model closely matches with that of neural networks on MNIST and CIFAR-10 for both the underparameterized and the overparameterized models.

Our analysis and experimental results suggest there is a trade-off between the computational cost and the convergence speed in underparametrized neural networks.

SVRG outperforms SGD after a few epochs in this regime.

However, SGD is shown to always outperform SVRG in the overparameterized regime.

Non-asymptotic analysis of SGD and SVRG, showing the strength of each algorithm in convergence speed and computational cost, in both under-parametrized and over-parametrized settings.

A Non-asymptotic comparison of SVRG and SGD: tradeoffs between compute and speed


Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models.

Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance.

Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear.

In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training.

We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data.

Furthermore, a strong correlation is observed between the capacity of an NAT model and the optimal complexity of the distilled data for the best translation quality.

Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models.

We achieve the state-of-the-art performance for the NAT-based models, and close the gap with the autoregressive baseline on WMT14 En-De benchmark.

We systematically examine why knowledge distillation is crucial to the training of non-autoregressive translation (NAT) models, and propose methods to further improve the distilled data to best match the capacity of an NAT model.

Understanding Knowledge Distillation in Non-autoregressive Machine Translation

knowledge distillation
non-autoregressive
translation


Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP), achieving state-of-the-art results in domains such as language modeling and machine translation.

Harnessing the transformer's ability to process long time horizons of information could provide a similar performance boost in partially-observable reinforcement learning (RL) domains, but the large-scale transformers used in NLP have yet to be successfully applied to the RL setting.

In this work we demonstrate that the standard transformer architecture is difficult to optimize, which was previously observed in the supervised learning setting but becomes especially pronounced with RL objectives.

We propose architectural modifications that substantially improve the stability and learning speed of the original Transformer and XL variant.

The proposed architecture, the Gated Transformer-XL (GTrXL), surpasses LSTMs on challenging memory environments and achieves state-of-the-art results on the multi-task DMLab-30 benchmark suite, exceeding the performance of an external memory architecture.

We show that the GTrXL, trained using the same losses, has stability and performance that consistently matches or exceeds a competitive LSTM baseline, including on more reactive tasks where memory is less critical.

GTrXL offers an easy-to-train, simple-to-implement but substantially more expressive architectural alternative to the standard multi-layer LSTM ubiquitously used for RL agents in partially-observable environments.

We succeed in stabilizing transformers for training in the RL setting and demonstrate a large improvement over LSTMs on DMLab-30, matching an external memory architecture.

Stabilizing Transformers for Reinforcement Learning


Knowledge distillation is an effective model compression technique in which a smaller model is trained to mimic a larger pretrained model.

However in order to make these compact models suitable for real world deployment, not only do

we need to reduce the performance gap but also we need to make them more robust to commonly occurring and adversarial perturbations.

Noise permeates every level of the nervous system, from the perception of sensory signals to the

We therefore believe that noise could be a crucial element in improving neural networks training and addressing the apparently contradictory goals of improving both the generalization and robustness of the

Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise at either the input level or the supervision signals.

Our results show that noise can improve both the generalization and robustness of the model.

”Fickle Teacher” which uses dropout in teacher model as a source of response variation leads to significant generalization improvement.

”Soft Randomization”, which matches the output distribution of

the student model on the image with Gaussian noise to the output of the teacher on original image, improves the adversarial robustness manifolds compared to the student model trained with Gaussian noise.

We further show the surprising effect of random label corruption on a model’s adversarial robustness.

The study highlights the benefits of adding constructive noise in the knowledge distillation framework and hopes to inspire further work in the area.

Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise in the knowledge distillation framework and studied their effect on generalization and robustness.

Noisy Collaboration in Knowledge Distillation


Our clustering objective is based on optimizing normalized cuts, a criterion which measures both intra-cluster similarity as well as inter-cluster dissimilarity.

Unlike much of the work in unsupervised deep learning, our trained model directly outputs final cluster assignments, rather than embeddings that need further processing to be usable.

Our approach generalizes to unseen datasets across a wide variety of domains, including text, and image.

Specifically, we achieve state-of-the-art results on popular unsupervised clustering benchmarks (e.g., MNIST, Reuters, CIFAR-10, and CIFAR-100), outperforming the strongest baselines by up to 10.9%.

Our generalization results are superior (by up to 21.9%) to the recent top-performing clustering approach with the ability to generalize.

We introduce a novel end-to-end approach for learning to cluster in the absence of labeled examples. We define a differentiable loss function equivalent to the expected normalized cuts.

Generalized Clustering by Learning to Optimize Expected Normalized Cuts


We introduce the largest (among publicly available) dataset for Cyrillic Handwritten Text Recognition and the first dataset for Cyrillic Text in the Wild Recognition, as well as suggest a method for recognizing Cyrillic Handwritten Text and Text in the Wild.

Based on this approach, we develop a system that can reduce the document processing time for one of the largest mathematical competitions in Ukraine by 12 days and the amount of used paper by 0.5 ton.

We introduce several datasets for Cyrillic OCR and a method for its recognition

On recognition of Cyrillic Text


Most deep learning for NLP represents each word with a single point or single-mode region in semantic space, while the existing multi-mode word embeddings cannot represent longer word sequences like phrases or sentences.

We introduce a phrase representation (also applicable to sentences) where each phrase has a distinct set of multi-mode codebook embeddings to capture different semantic facets of the phrase's meaning.

The codebook embeddings can be viewed as the cluster centers which summarize the distribution of possibly co-occurring words in a pre-trained word embedding space.

We propose an end-to-end trainable neural model that directly predicts the set of cluster centers from the input text sequence (e.g., a phrase or a sentence) during test time.

We find that the per-phrase/sentence codebook embeddings not only provide a more interpretable semantic representation but also outperform strong baselines (by a large margin in some tasks) on benchmark datasets for unsupervised phrase similarity, sentence similarity, hypernym detection, and extractive summarization.

We propose an unsupervised way to learn multiple embeddings for sentences and phrases

Learning Multi-facet Embeddings of Phrases and Sentences using Sparse Coding for Unsupervised Semantic Applications

sentence phrase embedding unsupervised learn


We present a meta-learning approach for adaptive text-to-speech (TTS) with few data.

During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker.

The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system.

Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers.

We introduce and benchmark three strategies:

(i) learning the speaker embedding while keeping the WaveNet core fixed,

(ii) fine-tuning the entire architecture with stochastic gradient descent, and

(iii) predicting the speaker embedding with a trained neural network encoder.

The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art results in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers.

Sample efficient algorithms to adapt a text-to-speech model to a new voice style with the state-of-the-art performance.

Sample Efficient Adaptive Text-to-Speech


This paper introduces a probabilistic framework for k-shot image classification.

The goal is to generalise from an initial large-scale classification task to a separate task comprising new  classes and small numbers of examples.

The new approach not only leverages the feature-based representation learned by a neural network from the initial task (representational transfer), but also information about the classes (concept transfer).

The concept information is encapsulated in a probabilistic model for the final layer weights of the neural network which acts as a prior for probabilistic k-shot learning.

We show that even a simple probabilistic model achieves state-of-the-art on a standard k-shot learning dataset by a large margin.

Moreover, it is able to accurately model uncertainty, leading to well calibrated classifiers, and is easily extensible and flexible, unlike many recent approaches to k-shot learning.

This paper introduces a probabilistic framework for k-shot image classification that achieves state-of-the-art results

Discriminative k-shot learning using probabilistic models


Building on the recent successes of distributed training of RL agents, in this paper we investigate the training of RNN-based RL agents from distributed prioritized experience replay.

We study the effects of parameter lag resulting in representational drift and recurrent state staleness and empirically derive an improved training strategy.

Using a single network architecture and fixed set of hyper-parameters, the resulting agent, Recurrent Replay Distributed DQN, quadruples the previous state of the art on Atari-57, and matches the state of the art on DMLab-30.

It is the first agent to exceed human-level performance in 52 of the 57 Atari games.

Investigation on combining recurrent neural networks and experience replay leading to state-of-the-art agent on both Atari-57 and DMLab-30 using single set of hyper-parameters.

Recurrent Experience Replay in Distributed Reinforcement Learning


The current state-of-the-art end-to-end semantic role labeling (SRL) model is a deep neural network architecture with no explicit linguistic features.

However, prior work has shown that gold syntax trees can dramatically improve SRL, suggesting that neural network models could see great improvements from explicit modeling of syntax.

In this work, we present linguistically-informed self-attention (LISA): a new neural network model that combines

multi-head self-attention with multi-task learning across dependency parsing, part-of-speech, predicate detection and SRL.

For example, syntax is incorporated by training one of the attention heads to attend to syntactic parents for each token.

Our model can predict all of the above tasks, but it is also trained such that if a high-quality syntactic parse is already available, it can be beneficially injected at test time without re-training our SRL model.

In experiments on the CoNLL-2005 SRL dataset LISA achieves an increase of 2.5 F1 absolute over the previous state-of-the-art on newswire with predicted predicates and more than 2.0 F1 on out-of-domain data.

On ConLL-2012 English SRL we also show an improvement of more than 3.0 F1, a 13% reduction in error.

Our combination of multi-task learning and self-attention, training the model to attend to parents in a syntactic parse tree, achieves state-of-the-art CoNLL-2005 and CoNLL-2012 SRL results for models using predicted predicates.

Linguistically-Informed Self-Attention for Semantic Role Labeling


Bottleneck structures with identity (e.g., residual) connection are now emerging popular paradigms for designing deep convolutional neural networks (CNN), for processing large-scale features efficiently.

In this paper, we focus on the information-preserving nature of identity connection and utilize this to enable a convolutional layer to have a new functionality of channel-selectivity, i.e., re-distributing its computations to important channels.

In particular, we propose Selective Convolutional Unit (SCU), a widely-applicable architectural unit that improves parameter efficiency of various modern CNNs with bottlenecks.

During training, SCU gradually learns the channel-selectivity on-the-fly via the alternative usage of

(b) rewiring the pruned parameters to important channels.

The rewired parameters emphasize the target channel in a way that selectively enlarges the convolutional kernels corresponding to it.

Our experimental results demonstrate that the SCU-based models without any postprocessing generally achieve both model compression and accuracy improvement compared to the baselines, consistently for all tested architectures.

We propose a new module that improves any ResNet-like architectures by enforcing "channel selective" behavior to convolutional layers

Selective Convolutional Units: Improving CNNs via Channel Selectivity


Predictive models that generalize well under distributional shift are often desirable and sometimes crucial to machine learning applications.

One example is the estimation of treatment effects from observational data, where a subtask is to predict the effect of a treatment on subjects that are systematically different from those who received the treatment in the data.

A related kind of distributional shift appears in unsupervised domain adaptation, where we are tasked with generalizing to a distribution of inputs that is different from the one in which we observe labels.

We pose both of these problems as prediction under a shift in design.

Popular methods for overcoming distributional shift are often heuristic or rely on assumptions that are rarely true in practice, such as having a well-specified model or knowing the policy that gave rise to the observed data.

Other methods are hindered by their need for a pre-specified metric for comparing observations, or by poor asymptotic properties.

In this work, we devise a bound on the generalization error under design shift, based on integral probability metrics and sample re-weighting.

We combine this idea with representation learning, generalizing and tightening existing results in this space.

Finally, we propose an algorithmic framework inspired by our bound and verify is effectiveness in causal effect estimation.

A theory and algorithmic framework for prediction under distributional shift, including causal effect estimation and domain adaptation

Learning Weighted Representations for Generalization Across Designs


Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty  and out-of-distribution robustness of deep learning models.

While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well.

Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift.

One possible explanation for this gap between theory and practice is that popular scalable approximate Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space.

We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions.

Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space.

We demonstrate that while low-loss connectors between modes exist, they are not connected in the space of predictions.

Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods.

We study deep ensembles through the lens of loss landscape and the space of predictions, demonstrating that the decorrelation power of random initializations is unmatched by subspace sampling that only explores a single mode.

Deep Ensembles: A Loss Landscape Perspective


Existing deep learning approaches for learning visual features tend to extract more information than what is required for the task at hand.

From a privacy preservation perspective, the input visual information is not protected from the model; enabling the model to become more intelligent than it is trained to be.

Existing approaches for suppressing additional task learning assume the presence of ground truth labels for the tasks to be suppressed during training time.

In this research, we propose a three-fold novel contribution:

(i) a novel metric to measure the trust score of a trained deep learning model,

(ii) a model-agnostic solution framework for trust score improvement by suppressing all the unwanted tasks, and

(iii) a simulated benchmark dataset, PreserveTask, having five different fundamental image classification tasks to study the generalization nature of models.

In the first set of experiments, we measure and improve the trust scores of five popular deep learning models: VGG16, VGG19, Inception-v1, MobileNet, and DenseNet and demonstrate that Inception-v1 is having the lowest trust score.

Additionally, we show results of our framework on color-MNIST dataset and practical applications of face attribute preservation in Diversity in Faces (DiF) and IMDB-Wiki dataset.

Can we trust our deep learning models? A framework to measure and improve a deep learning model's trust during training.

Is my Deep Learning Model Learning more than I want it to?


Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training.

In this work, we propose to train a policy while explicitly penalizing the mismatch between these two distributions over a fixed time horizon.

We do this by using a learned model of the environment dynamics which is unrolled for multiple time steps, and training a policy network to minimize a differentiable cost over this rolled-out trajectory.

This cost contains two terms: a policy cost which represents the objective the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on.

We propose to measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks.

We evaluate our approach using a large-scale observational dataset of driving behavior recorded from traffic cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.

A model-based RL approach which uses a differentiable uncertainty penalty to learn driving policies from purely observational data.

Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic


Dynamical system models (including RNNs) often lack the ability to adapt the sequence generation or prediction to a given context, limiting their real-world application.

In this paper we show that hierarchical multi-task dynamical systems (MTDSs) provide direct user control over sequence generation, via use of a latent  code z that specifies the customization to the

This enables style transfer, interpolation and morphing within generated sequences.

We show the MTDS can improve predictions via latent code interpolation, and avoid the long-term performance degradation of standard RNN approaches.

Tailoring predictions from sequence models (such as LDSs and RNNs) via an explicit latent code.

Customizing Sequence Generation with Multi-Task Dynamical Systems


By injecting adversarial examples into training data, adversarial training is promising for improving the robustness of deep learning models.

However, most existing adversarial training approaches are based on a specific type of adversarial attack.

It may not provide sufficiently representative samples from the adversarial domain, leading to a weak generalization ability on adversarial examples from other attacks.

Moreover, during the adversarial training, adversarial perturbations on inputs are usually crafted by fast single-step adversaries so as to scale to large datasets.

This work is mainly focused on the adversarial training yet efficient FGSM adversary.

In this scenario, it is difficult to train a model with great generalization due to the lack of representative adversarial samples, aka the samples are unable to accurately reflect the adversarial domain.

To alleviate this problem, we propose a novel Adversarial Training with Domain Adaptation (ATDA) method.

Our intuition is to regard the adversarial training on FGSM adversary as a domain adaption task with limited number of target domain samples.

The main idea is to learn a representation that is semantically meaningful and domain invariant on the clean domain as well as the adversarial domain.

Empirical evaluations on Fashion-MNIST, SVHN, CIFAR-10 and CIFAR-100 demonstrate that ATDA can greatly improve the generalization of adversarial training and the smoothness of the learned models, and outperforms state-of-the-art methods on standard benchmark datasets.

To show the transfer ability of our method, we also extend ATDA to the adversarial training on iterative attacks such as PGD-Adversial Training (PAT) and the defense performance is improved considerably.

We propose a novel adversarial training with domain adaptation method that significantly improves the generalization ability on adversarial examples from different attacks.

Improving the Generalization of Adversarial Training with Domain Adaptation

adversarial training domain adaptation
improve generalization


Character-level language modeling is an essential but challenging task in Natural Language Processing.

Prior works have focused on identifying long-term dependencies between characters and have built deeper and wider networks for better performance.

However, their models require substantial computational resources, which hinders the usability of character-level language models in applications with limited resources.

In this paper, we propose a lightweight model, called Group-Transformer, that reduces the resource requirements for a Transformer, a promising method for modeling sequence with long-term dependencies.

Specifically, the proposed method partitions linear operations to reduce the number of parameters and computational cost.

As a result, Group-Transformer only uses 18.2\% of parameters compared to the best performing LSTM-based model, while providing better performance on two benchmark tasks, enwik8 and text8.

When compared to Transformers with a comparable number of parameters and time complexity, the proposed model shows better performance.

The implementation code will be available.

This paper proposes a novel lightweight Transformer for character-level language modeling, utilizing group-wise operations.

Group-Transformer: Towards A Lightweight Character-level Language Model


Domain adaptation tackles the problem of transferring knowledge from a label-rich source domain to an unlabeled or label-scarce target domain.

Recently domain-adversarial training (DAT) has shown promising capacity to learn a domain-invariant feature space by reversing the gradient propagation of a domain classifier.

However, DAT is still vulnerable in several aspects including (1) training instability due to the overwhelming discriminative ability of the domain classifier in adversarial training, (2) restrictive feature-level alignment, and (3) lack of interpretability or systematic explanation of the learned feature space.

In this paper, we propose a novel Max-margin Domain-Adversarial Training (MDAT) by designing an Adversarial Reconstruction Network (ARN).

The proposed MDAT stabilizes the gradient reversing in ARN by replacing the domain classifier with a reconstruction network, and in this manner ARN conducts both feature-level and pixel-level domain alignment without involving extra network structures.

Furthermore, ARN demonstrates strong robustness to a wide range of hyper-parameters settings, greatly alleviating the task of model selection.

Extensive empirical results validate that our approach outperforms other state-of-the-art domain alignment methods.

Additionally, the reconstructed target samples are visualized to interpret the domain-invariant feature space which conforms with our intuition.

A stable domain-adversarial training approach for robust and comprehensive domain adaptation

Towards Stable and comprehensive Domain Alignment: Max-Margin Domain-Adversarial Training

comprehensive domain
stable
domain-adversarial training


We present an approach for expanding taxonomies with synonyms, or aliases.

We target large shopping taxonomies, with thousands of nodes.

A comprehensive set of entity aliases is an important component of identifying entities in unstructured text such as product reviews or search queries.

Our method consists of two stages: we generate synonym candidates from WordNet and shopping search queries, then use a binary classiﬁer to ﬁlter candidates.

We process taxonomies with thousands of synonyms in order to generate over 90,000 synonyms.

We show that using the taxonomy to derive contextual features improves classiﬁcation performance over using features from the target node alone.We show that our approach has potential for transfer learning between diﬀerent taxonomy domains, which reduces the need to collect training data for new taxonomies.

We use machine learning to generate synonyms for large shopping taxonomies.

Synonym Expansion for Large Shopping Taxonomies


Relational reasoning, the ability to model interactions and relations between objects, is valuable for robust multi-object tracking and pivotal for trajectory prediction.

In this paper, we propose MOHART, a class-agnostic, end-to-end multi-object tracking and trajectory prediction algorithm, which explicitly accounts for permutation invariance in its relational reasoning.

We explore a number of permutation invariant architectures and show that multi-headed self-attention outperforms the provided baselines and better accounts for complex physical interactions in a challenging toy experiment.

We show on three real-world tracking datasets that adding relational reasoning capabilities in this way increases the tracking and trajectory prediction performance, particularly in the presence of ego-motion, occlusions, crowded scenes, and faulty sensor inputs.

To the best of our knowledge, MOHART is the first fully end-to-end multi-object tracking from vision approach applied to real-world data reported in the literature.

MOHART uses a self-attention mechanism to perform relational reasoning in multi-object tracking.

Improving End-to-End Object Tracking Using Relational Reasoning


We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges:

We employ those insights to accelerate hyper-parameter sweeps in which all participating agents run concurrently and share their experience via a common replay module.

To this end we analyze the bias-variance tradeoffs in V-trace, a form of importance sampling for actor-critic methods.

Based on our analysis, we then argue for mixing experience sampled from replay with on-policy experience, and propose a new trust region scheme that scales effectively to data distributions where V-trace becomes unstable.

We provide extensive empirical validation of the proposed solution.

We further show the benefits of this setup by demonstrating state-of-the-art data efficiency on Atari among agents trained up until 200M environment frames.

We investigate and propose solutions for two challenges in reinforcement learning: (a) efficient actor-critic learning with experience replay (b) stability of very off-policy learning.

Off-Policy Actor-Critic with Shared Experience Replay

experience replay
actor-critic
off-policy


Stochastic gradient descent (SGD) is the workhorse of modern machine learning.

Sometimes, there are many different potential gradient estimators that can be used.

When so, choosing the one with the best tradeoff between cost and variance is important.

This paper analyzes the convergence rates of SGD as a function of time, rather than iterations.

This results in a simple rule to select the estimator that leads to the best optimization convergence guarantee.

This choice is the same for different variants of SGD, and with different assumptions about the objective (e.g. convexity or smoothness).

Inspired by this principle, we propose a technique to automatically select an estimator when a finite pool of estimators is given.

Then, we extend to infinite pools of estimators, where each one is indexed by control variate weights.

This is enabled by a reduction to a mixed-integer quadratic program.

Empirically, automatically choosing an estimator performs comparably to the best estimator chosen with hindsight.

We propose a gradient estimator selection algorithm with the aim on improving optimization efficiency.

Automatically Trading off Time and Variance when Selecting Gradient Estimators


We study the problem of alleviating the instability issue in the GAN training procedure via new architecture design.

The discrepancy between the minimax and maximin objective values could serve as a proxy for the difficulties that the alternating gradient descent encounters in the optimization of GANs.

In this work, we give new results on the benefits of multi-generator architecture of GANs.

We show that the minimax gap shrinks to \epsilon as the number of generators increases with rate O(1/\epsilon).

This improves over the best-known result of O(1/\epsilon^2).

At the core of our techniques is a novel application of Shapley-Folkman lemma to the generic minimax problem, where in the literature the technique was only known to work when the objective function is restricted to the Lagrangian function of a constraint optimization problem.

Our proposed Stackelberg GAN performs well experimentally in both synthetic and real-world datasets, improving Frechet Inception Distance by 14.61% over the previous multi-generator GANs on the benchmark datasets.

We study the problem of alleviating the instability issue in the GAN training procedure via new architecture design, with theoretical guarantees.

Stackelberg GAN: Towards Provable Minimax Equilibrium via Multi-Generator Architectures


Nesterov SGD is widely used for training modern neural networks and other machine learning models.

Yet, its advantages over SGD have not been theoretically clarified.

Indeed, as we show  in this paper, both theoretically and empirically, Nesterov SGD with any parameter selection does not in general provide acceleration over ordinary SGD.

Furthermore, Nesterov SGD may diverge for step sizes that ensure convergence of ordinary SGD.

This is in contrast to the classical results in the deterministic setting, where the same step size ensures accelerated convergence of the Nesterov's method over optimal gradient descent.

To address the non-acceleration issue, we  introduce a compensation term to Nesterov SGD.

The resulting  algorithm, which we call MaSS, converges  for same step sizes as SGD.

We prove that MaSS obtains an accelerated convergence rates over SGD for any mini-batch size in the linear setting.

For full batch, the convergence rate of MaSS matches the well-known accelerated rate of the Nesterov's method.

We also analyze the  practically important question of the dependence of the convergence rate and  optimal hyper-parameters on the mini-batch size, demonstrating three distinct regimes: linear scaling, diminishing returns and saturation.

Experimental evaluation of MaSS for several standard  architectures of deep networks, including ResNet and convolutional networks, shows improved performance over SGD, Nesterov SGD  and Adam.

This work proves the non-acceleration of Nesterov SGD with any hyper-parameters, and proposes new algorithm which provably accelerates SGD in the over-parameterized setting.

Accelerating SGD with momentum for over-parameterized learning


We propose the Fixed Grouping Layer (FGL); a novel feedforward layer designed to incorporate the inductive bias of structured smoothness into a deep learning model.

FGL achieves this goal by connecting nodes across layers based on spatial similarity.

The use of structured smoothness, as implemented by FGL, is motivated by applications to structured spatial data, which is, in turn, motivated by domain knowledge.

The proposed model architecture outperforms conventional neural network architectures across a variety of simulated and real datasets with structured smoothness.

A feedforward layer to incorporate structured smoothness into a deep learning model

Towards a Deep Network Architecture for Structured Smoothness


Convolutional networks are not aware of an object's geometric variations, which leads to inefficient utilization of model and data capacity.

To overcome this issue, recent works on deformation modeling seek to spatially reconfigure the data towards a common arrangement such that semantic recognition suffers less from deformation.

This is typically done by augmenting static operators with learned free-form sampling grids in the image space, dynamically tuned to the data and task for adapting the receptive field.

Yet adapting the receptive field does not quite reach the actual goal -- what really matters to the network is the *effective* receptive field (ERF), which reflects how much each pixel contributes.

It is thus natural to design other approaches to adapt the ERF directly during runtime.

In this work, we instantiate one possible solution as Deformable Kernels (DKs), a family of novel and generic convolutional operators for handling object deformations by directly adapting the ERF while leaving the receptive field untouched.

At the heart of our method is the ability to resample the original kernel space towards recovering the deformation of objects.

This approach is justified with theoretical insights that the ERF is strictly determined by data sampling locations and kernel values.

We implement DKs as generic drop-in replacements of rigid kernels and conduct a series of empirical studies whose results conform with our theories.

Over several tasks and standard base models, our approach compares favorably against prior works that adapt during runtime.

In addition, further experiments suggest a working mechanism orthogonal and complementary to previous works.

Don't deform your convolutions -- deform your kernels.

Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation


Inferring the structural properties of a protein from its amino acid sequence is a challenging yet important problem in biology.

Structures are not known for the vast majority of protein sequences, but structure is critical for understanding function.

Existing approaches for detecting structural similarity between proteins from sequence are unable to recognize and exploit structural patterns when sequences have diverged too far, limiting our ability to transfer knowledge between structurally related proteins.

We newly approach this problem through the lens of representation learning.

We introduce a framework that maps any protein sequence to a sequence of vector embeddings --- one per amino acid position --- that encode structural information.

We train bidirectional long short-term memory (LSTM) models on protein sequences with a two-part feedback mechanism that incorporates information from

(i) global structural similarity between proteins and

(ii) pairwise residue contact maps for individual proteins.

To enable learning from structural similarity information, we define a novel similarity measure between arbitrary-length sequences of vector embeddings based on a soft symmetric alignment (SSA) between them.

Our method is able to learn useful position-specific embeddings despite lacking direct observations of position-level correspondence between sequences.

We show empirically that our multi-task framework outperforms other sequence-based methods and even a top-performing structure-based alignment method when predicting structural similarity, our goal.

Finally, we demonstrate that our learned embeddings can be transferred to other protein sequence problems, improving the state-of-the-art in transmembrane domain prediction.

We present a method for learning protein sequence embedding models using structural information in the form of global structural similarity between proteins and within protein residue-residue contacts.

Learning protein sequence embeddings using information from structure


Inspired by the adaptation phenomenon of biological neuronal firing, we propose regularity normalization: a reparameterization of the activation in the neural network that take into account the statistical regularity in the implicit space.

By considering the neural network optimization process as a model selection problem, the implicit space is constrained by the normalizing factor, the minimum description length of the optimal universal code.

We introduce an incremental version of computing this universal code as normalized maximum likelihood and demonstrated its flexibility to include data prior such as top-down attention and other oracle information and its compatibility to be incorporated into batch normalization and layer normalization.

The preliminary results showed that the proposed method outperforms existing normalization methods in tackling the limited and imbalanced data from a non-stationary distribution benchmarked on computer vision task.

As an unsupervised attention mechanism given input data, this biologically plausible normalization has the potential to deal with other complicated real-world scenarios as well as reinforcement learning setting where the rewards are sparse and non-uniform.

Further research is proposed to discover these scenarios and explore the behaviors among different variants.

Considering neural network optimization process as a model selection problem, we introduce a biological plausible normalization method that extracts statistical regularity under MDL principle to tackle imbalanced and limited data issue.

Regularity Normalization: Constraining Implicit Space with Minimum Description Length


In this paper we study generative modeling via autoencoders while using the elegant geometric properties of the optimal transport (OT) problem and the Wasserstein distances.

We introduce Sliced-Wasserstein Autoencoders (SWAE), which are generative models that enable one to shape the distribution of the latent space into any samplable probability distribution without the need for training an adversarial network or defining a closed-form for the distribution.

In short, we regularize the autoencoder loss with the sliced-Wasserstein distance between the distribution of the encoded training samples and a predefined samplable distribution.

We show that the proposed formulation has an efficient numerical solution that provides similar capabilities to Wasserstein Autoencoders (WAE) and Variational Autoencoders (VAE), while benefiting from an embarrassingly simple implementation.

"Generative modeling with no need for adversarial training"

Sliced-Wasserstein Autoencoder: An Embarrassingly Simple Generative Model


The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations.

However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs.

IL in such situations can be challenging, especially when the level of demonstrators' expertise is unknown.

We propose a new IL paradigm called Variational Imitation Learning with Diverse-quality demonstrations (VILD), where we explicitly model the level of demonstrators' expertise with a probabilistic graphical model and estimate it along with a reward function.

We show that a naive estimation approach is not suitable to large state and action spaces, and fix this issue by using a variational approach that can be easily implemented using existing reinforcement learning methods.

Experiments on continuous-control benchmarks demonstrate that VILD outperforms state-of-the-art methods.

Our work enables scalable and data-efficient IL under more realistic settings than before.

We propose an imitation learning method to learn from diverse-quality demonstrations collected by demonstrators with different level of expertise.

VILD: Variational Imitation Learning with Diverse-quality Demonstrations

learn diverse-quality demonstration
imitation


With a growing number of available services, each having slightly different parameters, preconditions and effects, automated planning on general semantic services become highly relevant.

However, most exiting planners only consider PDDL, or if they claim to use OWL-S, they usually translate it to PDDL, losing much of the semantics on the way.

In this paper, we propose a new domain-independent heuristic based on a semantic distance that can be used by generic planning algorithms such as A* for automated planning of semantic services described with OWL-S.

For the heuristic to include more relevant information we calculate the heuristic at runtime.

Using this heuristic, we are able to produce better results (fewer expanded states) in less time than with established techniques.

Describing a semantic heuristics which builds upon an OWL-S service description and uses word and sentence distance measures to evaluate the usefulness of services for a given goal.

Using Semantic Distance as a Heuristic for Service Planning


In colored graphs, node classes are often associated with either their neighbors class or with information not incorporated in the graph associated with each node.

We here propose that node classes are also associated with topological features of the nodes.

We use this association to improve Graph machine learning in general and specifically, Graph Convolutional Networks (GCN).

First, we show that even in the absence of any external information on nodes, a good accuracy can be obtained on the prediction of the node class using either topological features, or using the neighbors class as an input to a GCN.

This accuracy is slightly less than the one that can be obtained using content based GCN.

Secondly, we show that explicitly adding the topology as an input to the GCN does not improve the accuracy when combined with external information on nodes.

However,  adding an additional adjacency matrix with edges between distant nodes with similar topology to the GCN does significantly improve its accuracy, leading to results better than all state of the art methods in multiple datasets.

Topology-Based Graph Convolutional Network (GCN)

Topological based classification using graph convolutional networks


Flies and mice are species separated by 600 million years of evolution, yet have evolved olfactory systems that share many similarities in their anatomic and functional organization.

What functions do these shared anatomical and functional features serve, and are they optimal for odor sensing?

In this study, we address the optimality of evolutionary design in olfactory circuits by studying artificial neural networks trained to sense odors.

We found that artificial neural networks quantitatively recapitulate structures inherent in the olfactory system, including the formation of glomeruli onto a compression layer and sparse and random connectivity onto an expansion layer.

Finally, we offer theoretical justifications for each result.

Our work offers a framework to explain the evolutionary convergence of olfactory circuits, and gives insight and logic into the anatomic and functional structure of the olfactory system.

Artificial neural networks evolved the same structures present in the olfactory systems of flies and mice after being trained to classify odors

Evolving the Olfactory System


The recent direction of unpaired image-to-image translation is on one hand very exciting as it alleviates the big burden in obtaining label-intensive pixel-to-pixel supervision, but it is on the other hand not fully satisfactory due to the presence of artifacts and degenerated transformations.

In this paper, we take a manifold view of the problem by introducing a smoothness term over the sample graph to attain harmonic functions to enforce consistent mappings during the translation.

We develop HarmonicGAN to learn bi-directional translations between the source and the target domains.

With the help of similarity-consistency, the inherent self-consistency property of samples can be maintained.

Distance metrics defined on two types of features including histogram and CNN are exploited.

Under an identical problem setting as CycleGAN, without additional manual inputs and only at a small training-time cost, HarmonicGAN demonstrates a significant qualitative and quantitative improvement over the state of the art, as well as improved interpretability.

We show experimental results in a number of applications including medical imaging, object transfiguration, and semantic labeling.

We outperform the competing methods in all tasks, and for a medical imaging task in particular our method turns CycleGAN from a failure to a success, halving the mean-squared error, and generating images that radiologists prefer over competing methods in 95% of cases.

Smooth regularization over sample graph for unpaired image-to-image translation results in significantly improved consistency

Harmonic Unpaired Image-to-image Translation


Multiview stereo aims to reconstruct scene depth from images acquired by a camera under arbitrary motion.

Recent methods address this problem through deep learning, which can utilize semantic cues to deal with challenges such as textureless and reflective regions.

In this paper, we present a convolutional neural network called DPSNet (Deep Plane Sweep Network) whose design is inspired by best practices of traditional geometry-based approaches.

Rather than directly estimating depth and/or optical flow correspondence from image pairs as done in many previous deep learning methods, DPSNet takes a plane sweep approach that involves building a cost volume from deep features using the plane sweep algorithm, regularizing the cost volume via a context-aware cost aggregation, and regressing the depth map from the cost volume.

The cost volume is constructed using a differentiable warping process that allows for end-to-end training of the network.

Through the effective incorporation of conventional multiview stereo concepts within a deep learning framework, DPSNet achieves state-of-the-art reconstruction results on a variety of challenging datasets.

A convolution neural network for multi-view stereo matching whose design is inspired by best practices of traditional geometry-based approaches

DPSNet: End-to-end Deep Plane Sweep Stereo


The ability to design biological structures such as DNA or proteins would have considerable medical and industrial impact.

Doing so presents a challenging black-box optimization problem characterized by the large-batch, low round setting due to the need for labor-intensive wet lab evaluations.

In response, we propose using reinforcement learning (RL) based on proximal-policy optimization (PPO) for biological sequence design.

RL provides a flexible framework for optimization generative sequence models to achieve specific criteria, such as diversity among the high-quality sequences discovered.

We propose a model-based variant of PPO, DyNA-PPO, to improve sample efficiency, where the policy for a new round is trained offline using a simulator fit on functional measurements from prior rounds.

To accommodate the growing number of observations across rounds, the simulator model is automatically selected at each round from a pool of diverse models of varying capacity.

On the tasks of designing DNA transcription factor binding sites, designing antimicrobial proteins, and optimizing the energy of Ising models based on protein structure, we find that DyNA-PPO performs significantly better than existing methods in settings in which modeling is feasible, while still not performing worse in situations in which a reliable model cannot be learned.

We augment model-free policy learning with a sequence-level surrogate reward functions and count-based visitation bonus and demonstrate effectiveness in the large batch, low-round regime seen in designing DNA and protein sequences.

Model-based reinforcement learning for biological sequence design


Achieving machine intelligence requires a smooth integration of perception and reasoning, yet models developed to date tend to specialize in one or the other; sophisticated manipulation of symbols acquired from rich perceptual spaces has so far proved elusive.

Consider a visual arithmetic task, where the goal is to carry out simple arithmetical algorithms on digits presented under natural conditions (e.g. hand-written, placed randomly).

We propose a two-tiered architecture for tackling this kind of problem.

The lower tier consists of a heterogeneous collection of information processing modules, which can include pre-trained deep neural networks for locating and extracting characters from the image, as well as modules performing symbolic transformations on the representations extracted by perception.

The higher tier consists of a controller, trained using reinforcement learning, which coordinates the modules in order to solve the high-level task.

For instance, the controller may learn in what contexts to execute the perceptual networks and what symbolic transformations to apply to their outputs.

The resulting model is able to solve a variety of tasks in the visual arithmetic domain,and has several advantages over standard, architecturally homogeneous feedforward networks including improved sample efficiency.

We use reinforcement learning to train an agent to solve a set of visual arithmetic tasks using provided pre-trained perceptual modules and transformations of internal representations created by those modules.

Sequential Coordination of Deep Models for Learning Visual Arithmetic


Animals develop novel skills not only through the interaction with the environment but also from the influence of the others.

In this work we model the social influence into the scheme of reinforcement learning, enabling the agents to learn both from the environment and from their peers.

Specifically, we first define a metric to measure the distance between policies then quantitatively derive the definition of uniqueness.

Unlike previous precarious joint optimization approaches, the social uniqueness motivation in our work is imposed as a constraint to encourage the agent to learn a policy different from the existing agents while still solve the primal task.

The resulting algorithm, namely Interior Policy Differentiation (IPD), brings about performance improvement as well as a collection of policies that solve a given task with distinct behaviors

A new RL algorithm called Interior Policy Differentiation is proposed to learn a collection of diverse policies for a given primal task.

Learning with Social Influence through  Interior Policy Differentiation


Generative adversarial networks (GANs) have been extremely effective in approximating complex distributions of high-dimensional, input data samples, and substantial progress has been made in understanding and improving GAN performance in terms of both theory and application.

However, we currently lack quantitative methods for model assessment.

Because of this, while many GAN variants being proposed, we have relatively little understanding of their relative abilities.

In this paper, we evaluate the performance of various types of GANs using divergence and distance functions typically used only for training.

We observe consistency across the various proposed metrics and, interestingly, the test-time metrics do not favour networks that use the same training-time criterion.

We also compare the proposed metrics to human perceptual scores.

An empirical evaluation on generative adversarial networks

Quantitatively Evaluating GANs With Divergences Proposed for Training


Knowledge bases (KB) are often represented as a collection of facts in the form (HEAD, PREDICATE, TAIL), where HEAD and TAIL are entities while PREDICATE is a binary relationship that links the two.

It is a well-known fact that knowledge bases are far from complete, and hence the plethora of research on KB completion methods, specifically on link prediction.

However, though frequently ignored, these repositories also contain numerical facts.

Numerical facts link entities to numerical values via numerical predicates; e.g., (PARIS, LATITUDE, 48.8).

Likewise, numerical facts also suffer from the incompleteness problem.

To address this issue, we introduce the numerical attribute prediction problem.

This problem involves a new type of query where the relationship is a numerical predicate.

Consequently, and contrary to link prediction, the answer to this query is a numerical value.

We argue that the numerical values associated with entities explain, to some extent, the relational structure of the knowledge base.

Therefore, we leverage knowledge base embedding methods to learn representations that are useful predictors for the numerical attributes.

An extensive set of experiments on benchmark versions of FREEBASE and YAGO show that our approaches largely outperform sensible baselines.

We make the datasets available under a permissive BSD-3 license.

Prediction of numerical attribute values associated with entities in knowledge bases.

Learning Numerical Attributes in Knowledge Bases


We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT.

In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek.

Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer.

Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, e.g. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure.

These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.

We propose procedures for evaluating and strengthening contextual embedding alignment and show that they both improve multilingual BERT's zero-shot XNLI transfer and provide useful insights into the model.

Multilingual Alignment of Contextual Word Representations


Neural networks have reached outstanding performance for solving various ill-posed inverse problems in imaging.

However, drawbacks of end-to-end learning approaches in comparison to classical variational methods are the requirement of expensive retraining for even slightly different problem statements and the lack of provable error bounds during inference.

Recent works tackled the first problem by using networks trained for Gaussian image denoising as generic plug-and-play regularizers in energy minimization algorithms.

Even though this obtains state-of-the-art results on many tasks, heavy restrictions on the network architecture have to be made if provable convergence of the underlying fixed point iteration is a requirement.

More recent work has proposed to train networks to output descent directions with respect to a given energy function with a provable guarantee of convergence to a minimizer of that energy.

However, each problem and energy requires the training of a separate network.

In this paper we consider the combination of both approaches by projecting the outputs of a plug-and-play denoising network onto the cone of descent directions to a given energy.

This way, a single pre-trained network can be used for a wide variety of reconstruction tasks.

Our results show improvements compared to classical energy minimization methods while still having provable convergence guarantees.

We use neural networks trained for image denoising as plug-and-play priors in energy minimization algorithms for image reconstruction problems with provable convergence.

Energy Dissipation with Plug-and-Play Priors


Knowledge graph embedding research has overlooked the problem of probability calibration.

We show popular embedding models are indeed uncalibrated.

That means probability estimates associated to predicted triples are unreliable.

We present a novel method to calibrate a model when ground truth negatives are not available, which is the usual case in knowledge graphs.

We propose to use Platt scaling and isotonic regression alongside our method.

Experiments on three datasets with ground truth negatives show our contribution leads to well calibrated models when compared to the gold standard of using negatives.

We get significantly better results than the uncalibrated models from all calibration methods.

We show isotonic regression offers the best the performance overall, not without trade-offs.

We also show that calibrated models reach state-of-the-art accuracy without the need to define relation-specific decision thresholds.

We propose a novel method to calibrate knowledge graph embedding models without the need of negative examples.

Probability Calibration for Knowledge Graph Embedding Models


As an emerging topic in face recognition, designing margin-based loss functions can increase the feature margin between different classes for enhanced discriminability.

More recently, absorbing the idea of mining-based strategies is adopted to emphasize the misclassified samples and achieve promising results.

However, during the entire training process, the prior methods either do not explicitly emphasize the sample based on its importance that renders the hard samples not fully exploited or explicitly emphasize the effects of semi-hard/hard samples even at the early training stage that may lead to convergence issues.

In this work, we propose a novel Adaptive Curriculum Learning loss (CurricularFace) that embeds the idea of curriculum learning into the loss function to achieve a novel training strategy for deep face recognition, which mainly addresses easy samples in the early training stage and hard ones in the later stage.

Specifically, our CurricularFace adaptively adjusts the relative importance of easy and hard samples during different training stages.

In each stage, different samples are assigned with different importance according to their corresponding difficultness.

Extensive experimental results on popular benchmarks demonstrate the superiority of our CurricularFace over the state-of-the-art competitors.

Code will be available upon publication.

A novel  Adaptive Curriculum Learning loss for deep face recognition

CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition

adaptive curriculum learn loss deep face recognition


Adversarial examples are perturbed inputs designed to fool machine learning models.

Adversarial training injects such examples into training data to increase robustness.

To scale this technique to large datasets, perturbations are crafted using fast single-step methods that maximize a linear approximation of the model's loss.

We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss.

The model thus learns to generate weak perturbations, rather than defend against strong ones.

As a result, we find that adversarial training remains vulnerable to black-box attacks, where we transfer perturbations computed on undefended models, as well as to a powerful novel single-step attack that escapes the non-smooth vicinity of the input data via a small random step.

We further introduce Ensemble Adversarial Training, a technique that augments training data with perturbations transferred from other models.

On ImageNet, Ensemble Adversarial Training yields models with strong robustness to black-box attacks.

In particular, our most robust model won the first round of the NIPS 2017 competition on Defenses against Adversarial Attacks.

Adversarial training with single-step methods overfits, and remains vulnerable to simple black-box and white-box attacks. We show that including adversarial examples from multiple sources helps defend against black-box attacks.

Ensemble Adversarial Training: Attacks and Defenses


An adversarial feature learning (AFL) is a powerful framework to learn representations invariant to a nuisance attribute, which uses an adversarial game between a feature extractor and a categorical attribute classifier.

It theoretically sounds in term of it maximize conditional entropy between attribute and representation.

However, as shown in this paper, the AFL often causes unstable behavior that slows down the convergence.

We propose an {\em attribute perception matching} as an alternative approach, based on the reformulation of conditional entropy maximization as {\em pair-wise distribution matching}. Although the naive approach for realizing the pair-wise distribution matching requires the significantly large number of parameters, the proposed method requires the same number of parameters with AFL but has a better convergence property.

Experiments on both toy and real-world dataset prove that our proposed method converges to better invariant representation significantly faster than AFL.

This paper proposes a new approach to incorporating desired invariance to representations learning, based on the observations that the current state-of-the-art AFL has practical issues.

Invariant Feature Learning by Attribute Perception Matching


We study the properties of common loss surfaces through their Hessian matrix.

In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian is composed of two parts: (1) the bulk centered near zero, (2) and outliers away from the bulk.

We present numerical evidence and mathematical justifications to the following conjectures laid out by Sagun et.

al. (2016): Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making the data less separable) only affects the outliers.

We believe that our observations have striking implications for non-convex optimization in high dimensions.

First, the *flatness* of such landscapes (which can be measured by the singularity of the Hessian) implies that classical notions of basins of attraction may be quite misleading.

And that the discussion of wide/narrow basins may be in need of a new perspective around over-parametrization and redundancy that are able to create *large* connected components at the bottom of the landscape.

Second, the dependence of a small number of large eigenvalues to the data distribution can be linked to the spectrum of the covariance matrix of gradients of model outputs.

With this in mind, we may reevaluate the connections within the data-architecture-algorithm framework of a model, hoping that it would shed light on the geometry of high-dimensional and non-convex spaces in modern applications.

In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.

The loss surface is *very* degenerate, and there are no barriers between large batch and small batch solutions.

Empirical Analysis of the Hessian of Over-Parametrized Neural Networks


The allocation of computation resources in the backbone is a crucial issue in object detection.

However, classification allocation pattern is usually adopted directly to object detector, which is proved to be sub-optimal.

In order to reallocate the engaged computation resources in a more efficient way, we present CR-NAS (Computation Reallocation Neural Architecture Search) that can learn computation reallocation strategies across different feature resolution and spatial position diectly on the target detection dataset.

A two-level reallocation space is proposed for both stage and spatial reallocation.

A novel hierarchical search procedure is adopted to cope with the complex search space.

We apply CR-NAS to multiple backbones and achieve consistent improvements.

Our CR-ResNet50 and CR-MobileNetV2 outperforms the baseline by 1.9% and 1.7% COCO AP respectively without any additional computation budget.

The models discovered by CR-NAS can be equiped to other powerful detection neck/head and be easily transferred to other dataset, e.g. PASCAL VOC, and other vision tasks, e.g. instance segmentation.

Our CR-NAS can be used as a plugin to improve the performance of various networks, which is demanding.

We propose CR-NAS to reallocate engaged  computation resources in different resolution and spatial position.

Computation Reallocation for Object Detection


Uncertainty is a very important feature of the intelligence and helps the brain become a flexible, creative and powerful intelligent system.

The crossbar-based neuromorphic computing chips, in which the computing is mainly performed by analog circuits, have the uncertainty and can be used to imitate the brain.

However, most of the current deep neural networks have not taken the uncertainty of the neuromorphic computing chip into consideration.

Therefore, their performances on the neuromorphic computing chips are not as good as on the original platforms (CPUs/GPUs).

In this work, we proposed the uncertainty adaptation training scheme (UATS) that tells the uncertainty to the neural network in the training process.

The experimental results show that the neural networks can achieve comparable inference performances on the uncertain neuromorphic computing chip compared to the results on the original platforms, and much better than the performances without this training scheme.

A training method that can make deep learning algorithms work better on neuromorphic computing chips with uncertainty

A Training Scheme for the Uncertain Neuromorphic Computing Chips


An important property of image classification systems in the real world is that they both accurately classify objects from target classes (``knowns'') and safely reject unknown objects (``unknowns'') that belong to classes not present in the training data.

Unfortunately, although the strong generalization ability of existing CNNs ensures their accuracy when classifying known objects, it also causes them to often assign an unknown to a target class with high confidence.

As a result, simply using low-confidence detections as a way to detect unknowns does not work well.

In this work, we propose an Unknown-aware Deep Neural Network (UDN for short) to solve this challenging problem.

The key idea of UDN is to enhance existing CNNs to support a product operation that models the product relationship among the features produced by convolutional layers.

This way, missing a single key feature of a target class will greatly reduce the probability of assigning an object to this class.

UDN uses a learned ensemble of these product operations, which allows it to balance the contradictory requirements of accurately classifying known objects and correctly rejecting unknowns.

To further improve the performance of UDN at detecting unknowns, we propose an information-theoretic regularization strategy that incorporates the objective of rejecting unknowns into the learning process of UDN.

We experiment on benchmark image datasets including MNIST, CIFAR-10, CIFAR-100, and SVHN, adding unknowns by injecting one dataset into another.

Our results demonstrate that UDN significantly outperforms state-of-the-art methods at rejecting unknowns by 25 percentage points improvement in accuracy, while still preserving the classification accuracy.

A CNN architecture that can effective rejects the unknowns in test objects

Unknown-Aware Deep Neural Network


Multivariate time series with missing values are common in areas such as healthcare and finance, and have grown in number and complexity over the years.

This raises the question whether deep learning methodologies can outperform classical data imputation methods in this domain.

However, naive applications of deep learning fall short in giving reliable confidence estimates and lack interpretability.

We propose a new deep sequential latent variable model for dimensionality reduction and data imputation.

Our modeling assumption is simple and interpretable: the high dimensional time series has a lower-dimensional representation which evolves smoothly in time according to a Gaussian process.

The non-linear dimensionality reduction in the presence of missing data is achieved using a VAE approach with a novel structured variational approximation.

We demonstrate that our approach outperforms several classical and deep learning-based data imputation methods on high-dimensional data from the domains of computer vision and healthcare, while additionally improving the smoothness of the imputations and providing interpretable uncertainty estimates.

We perform amortized variational inference on a latent Gaussian process model to achieve superior imputation performance on multivariate time series with missing data.

Deep Amortized Variational Inference for Multivariate Time Series Imputation with Latent Gaussian Process Models

latent gaussian process model
multivariate time series
variational inference
imputation


In practice it is often found that large over-parameterized neural networks generalize better than their smaller counterparts, an observation that appears to conflict with classical notions of function complexity, which typically favor smaller models.

In this work, we investigate this tension between complexity and generalization through an extensive empirical exploration of two natural metrics of complexity related to sensitivity to input perturbations.

Our experiments survey thousands of models with different architectures, optimizers, and other hyper-parameters, as well as four different image classification datasets.

We find that trained neural networks are more robust to input perturbations in the vicinity of the training data manifold, as measured by the input-output Jacobian of the network, and that this correlates well with generalization.

We further establish that factors associated with poor generalization -- such as full-batch training or using random labels -- correspond to higher sensitivity, while factors associated with good generalization  -- such as data augmentation and ReLU non-linearities -- give rise to more robust functions.

Finally, we demonstrate how the input-output Jacobian norm can be predictive of generalization at the level of individual test points.

We perform massive experimental studies characterizing the relationships between Jacobian norms, linear regions, and generalization.

Sensitivity and Generalization in Neural Networks: an Empirical Study