While adversarial training can improve robust accuracy (against an adversary), it sometimes hurts standard accuracy (when there is no adversary).

Previous work has studied this tradeoff between standard and robust accuracy, but only in the setting where no predictor performs well on both objectives in the infinite data limit.

In this paper, we show that even when the optimal predictor with infinite data performs well on both objectives, a tradeoff can still manifest itself with finite data.

Furthermore, since our construction is based on a convex learning problem, we rule out optimization concerns, thus laying bare a fundamental tension between robustness and generalization.

Finally, we show that robust self-training mostly eliminates this tradeoff by leveraging unlabeled data.

Even if there is no tradeoff in the infinite data limit, adversarial training can have worse standard accuracy even in a convex problem.

Adversarial Training Can Hurt Generalization


Skip connections are increasingly utilized by deep neural networks to improve accuracy and cost-efficiency.

In particular, the recent DenseNet is efficient in computation and parameters, and achieves state-of-the-art predictions by directly connecting each feature layer to all previous ones.

However, DenseNet's extreme connectivity pattern may hinder its scalability to high depths, and in applications like fully convolutional networks, full DenseNet connections are prohibitively expensive.

This work first experimentally shows that one key advantage of skip connections is to have short distances among feature layers during backpropagation.

Specifically, using a fixed number of skip connections, the connection patterns with shorter backpropagation distance among layers have more accurate predictions.

Following this insight, we propose a connection template, Log-DenseNet, which, in comparison to DenseNet,  only slightly increases the backpropagation distances among layers from 1 to  ($1 + \log_2 L$), but uses only $L\log_2 L$ total connections instead of $O(L^2)$.

Hence, \logdenses are easier to scale than DenseNets, and no longer require careful GPU memory management.

We demonstrate the effectiveness of our design principle by showing better performance than DenseNets on tabula rasa semantic segmentation, and competitive results on visual recognition.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.

Log-DenseNet: How to Sparsify a DenseNet


Building agents to interact with the web would allow for significant improvements in knowledge understanding and representation learning.

However, web navigation tasks are difficult for current deep reinforcement learning (RL) models due to the large discrete action space and the varying number of actions between the states.

In this work, we introduce DOM-Q-NET, a novel architecture for RL-based web navigation to address both of these problems.

It parametrizes Q functions with separate networks for different action categories: clicking a DOM element and typing a string input.

Our model utilizes a graph neural network to represent the tree-structured HTML of a standard web page.

We demonstrate the capabilities of our model on the MiniWoB environment where we can match or outperform existing work without the use of expert demonstrations.

Furthermore, we show 2x improvements in sample efficiency when training in the multi-task setting, allowing our model to transfer learned behaviours across tasks.

Graph-based Deep Q Network for Web Navigation

DOM-Q-NET:  Grounded RL on Structured Language


Learning disentangled representations that correspond to factors of variation in real-world data is critical to interpretable and human-controllable machine learning.

Recently, concerns about the viability of learning disentangled representations in a purely unsupervised manner has spurred a shift toward the incorporation of weak supervision.

However, there is currently no formalism that identifies when and how weak supervision will guarantee disentanglement.

To address this issue, we provide a theoretical framework—including a calculus of disentanglement— to assist in analyzing the disentanglement guarantees (or lack thereof) conferred by weak supervision when coupled with learning algorithms based on distribution matching.

We empirically verify the guarantees and limitations of several weak supervision methods (restricted labeling, match-pairing, and rank-pairing), demonstrating the predictive power and usefulness of our theoretical framework.

We construct a theoretical framework for weakly supervised disentanglement and conducted lots of experiments to back up the theory.

Weakly Supervised Disentanglement with Guarantees


Despite the growing interest in continual learning, most of its contemporary works have been studied in a rather restricted setting where tasks are clearly distinguishable, and task boundaries are known during training.

However, if our goal is to develop an algorithm that learns as humans do, this setting is far from realistic, and it is essential to develop a methodology that works in a task-free manner.

Meanwhile, among several branches of continual learning, expansion-based methods have the advantage of eliminating catastrophic forgetting by allocating new resources to learn new data.

In this work, we propose an expansion-based approach for task-free continual learning.

Our model, named Continual Neural Dirichlet Process Mixture (CN-DPM), consists of a set of neural network experts that are in charge of a subset of the data.

CN-DPM expands the number of experts in a principled way under the Bayesian nonparametric framework.

With extensive experiments, we show that our model successfully performs task-free continual learning for both discriminative and generative tasks such as image classification and image generation.

We propose an expansion-based approach for task-free continual learning for the first time. Our model consists of a set of neural network experts and expands the number of experts under the Bayesian nonparametric principle.

A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning

task-free continual learning
neural
model


Stochastic Gradient Descent (SGD) with Nesterov's momentum is a widely used optimizer in deep learning, which is observed to have excellent generalization performance.

However, due to the large stochasticity, SGD with Nesterov's momentum is not robust, i.e., its performance may deviate significantly from the expectation.

In this work, we propose Amortized Nesterov's Momentum, a special variant of Nesterov's momentum which has more robust iterates, faster convergence in the early stage and higher efficiency.

Our experimental results show that this new momentum achieves similar (sometimes better) generalization performance with little-to-no tuning.

In the convex case, we provide optimal convergence rates for our new methods and discuss how the theorems explain the empirical results.

Amortizing Nesterov's momentum for more robust, lightweight and fast deep learning training.

Amortized Nesterov's Momentum: Robust and Lightweight  Momentum for Deep Learning

nesterov 's momentum robust
lightweight
deep


A state-of-the-art generative model, a ”factorized action variational autoencoder (FAVAE),” is presented for learning disentangled and interpretable representations from sequential data via the information bottleneck without supervision.

The purpose of disentangled representation learning is to obtain interpretable and transferable representations from data.

We focused on the disentangled representation of sequential data because there is a wide range of potential applications if disentanglement representation is extended to sequential data such as video, speech, and stock price data.

Sequential data is characterized by dynamic factors and static factors: dynamic factors are time-dependent, and static factors are independent of time.

Previous works succeed in disentangling static factors and dynamic factors by explicitly modeling the priors of latent variables to distinguish between static and dynamic factors.

However, this model can not disentangle representations between dynamic factors, such as disentangling ”picking” and ”throwing” in robotic tasks.

In this paper, we propose new model that can disentangle multiple dynamic factors.

Since our method does not require modeling priors, it is capable of disentangling ”between” dynamic factors.

In experiments, we show that FAVAE can extract the disentangled dynamic factors.

We propose new model that can disentangle multiple dynamic factors in sequential data

FAVAE: SEQUENCE DISENTANGLEMENT USING IN- FORMATION BOTTLENECK PRINCIPLE


Generative networks are promising models for specifying visual transformations.

Unfortunately, certification of generative models is challenging as one needs to capture sufficient non-convexity so to produce precise bounds on the output.

Existing verification methods either fail to scale to generative networks or do not capture enough non-convexity.

In this work, we present a new verifier, called ApproxLine, that can certify non-trivial properties of generative networks.

ApproxLine performs both deterministic and probabilistic abstract interpretation and captures infinite sets of outputs of generative networks.

We show that ApproxLine can verify interesting interpolations in the network's latent space.

We verify deterministic and probabilistic properties of neural networks using non-convex relaxations over visible transformations specified by generative models

Verification of Generative-Model-Based Visual Transformations


Multi-view video summarization (MVS) lacks researchers’ attention due to their major challenges of inter-view correlations and overlapping of cameras.

Most of the prior MVS works are offline, relying on only summary, needing extra communication bandwidth and transmission time with no focus on uncertain environments.

Different from the existing methods, we propose edge intelligence based MVS and spatio-temporal features based activity recognition for IoT environments.

We segment the multi-view videos on each slave device over edge into shots using light-weight CNN object detection model and compute mutual information among them to generate summary.

Our system does not rely on summary only but encode and transmit it to a master device with neural computing stick (NCS) for intelligently computing inter-view correlations and efficiently recognizing activities, thereby saving computation resources, communication bandwidth, and transmission time.

Experiments report an increase of 0.4 in F-measure score on MVS Office dataset as well as 0.2% and 2% increase in activity recognition accuracy over UCF-50 and YouTube 11 datasets, respectively, with lower storage and transmission time compared to state-of-the-art.

The time complexity is decreased from 1.23 to 0.45 secs for a single frame processing, thereby generating 0.75 secs faster MVS.

Furthermore, we made a new dataset by synthetically adding fog to an MVS dataset to show the adaptability of our system for both certain and uncertain surveillance environments.

An efficient multi-view video summarization scheme advanced to activity recognition in IoT environments.

Multi-View Summarization and Activity Recognition Meet Edge Computing in IoT Environments

activity recognition
iot environment
multi-view
summarization


A central goal in the study of the primate visual cortex and hierarchical models for object recognition is understanding how and why single units trade off invariance versus sensitivity to image transformations.

For example, in both deep networks and visual cortex there is substantial variation from layer-to-layer and unit-to-unit in the degree of translation invariance.

Here, we provide theoretical insight into this variation and its consequences for encoding in a deep network.

Our critical insight comes from the fact that rectification simultaneously decreases response variance and correlation across responses to transformed stimuli, naturally inducing a positive relationship between invariance and dynamic range.

Invariant input units then tend to drive the network more than those sensitive to small image transformations.

We discuss consequences of this relationship for AI: deep nets naturally weight invariant units over sensitive units, and this can be strengthened with training, perhaps contributing to generalization performance.

Our results predict a signature relationship between invariance and dynamic range that can now be tested in future neurophysiological studies.

Rectification in deep neural networks naturally leads them to favor an invariant representation.

The Natural Tendency of Feed Forward Neural Networks to Favor Invariant Units


We study the use of the Wave-U-Net architecture for speech enhancement, a model introduced by Stoller et al for the separation of music vocals and accompaniment.

This end-to-end learning method for audio source separation operates directly in the time domain, permitting the integrated modelling of phase information and being able to take large  temporal contexts into account.

Our experiments show that the proposed method improves several metrics, namely PESQ, CSIG, CBAK, COVL and SSNR, over the  state-of-the-art with respect to the speech enhancement task on the Voice Bank corpus (VCTK) dataset.

We find that a reduced number of hidden layers is sufficient for speech enhancement in comparison to the original system designed for singing voice separation in music.

We see this initial result as an encouraging signal to further explore speech enhancement in the time-domain, both as an end in itself and as a pre-processing step to speech recognition systems.

The Wave-U-Net architecture, recently introduced by Stoller et al for music source separation, is highly effective for speech enhancement, beating the state of the art.

Improved Speech Enhancement with the Wave-U-Net


Reinforcement learning (RL) is a powerful framework for solving problems by exploring and learning from mistakes.

However, in the context of autonomous vehicle (AV) control, requiring an agent to make mistakes, or even allowing mistakes, can be quite dangerous and costly in the real world.

For this reason, AV RL is generally only viable in simulation.

Because these simulations have imperfect representations, particularly with respect to graphics, physics, and human interaction, we find motivation for a framework similar to RL, suitable to the real world.

To this end, we formulate a learning framework that learns from restricted exploration by having a human demonstrator do the exploration.

Existing work on learning from demonstration typically either assumes the collected data is performed by an optimal expert, or requires potentially dangerous exploration to find the optimal policy.

We propose an alternative framework that learns continuous control from only safe behavior.

One of our key insights is that the problem becomes tractable if the feedback score that rates the demonstration applies to the atomic action, as opposed to the entire sequence of actions.

We use human experts to collect driving data as well as to label the driving data through a framework we call ``Backseat Driver'', giving us state-action pairs matched with scalar values representing the score for the action.

We call the more general learning framework ReNeg, since it learns a regression from states to actions given negative as well as positive examples.

We empirically validate several models in the ReNeg framework, testing on lane-following with limited data.

We find that the best solution in this context outperforms behavioral cloning has strong connections to stochastic policy gradient approaches.

We introduce a novel framework for learning from demonstration that uses continuous human feedback; we evaluate this framework on continuous control for autonomous vehicles.

ReNeg and Backseat Driver: Learning from demonstration with continuous human feedback

learn demonstration continuous human feedback


We explore the use of Vector Quantized Variational AutoEncoder (VQ-VAE) models for large scale image generation.

To this end, we scale and enhance the autoregressive priors used in VQ-VAE to generate synthetic samples of much higher coherence and fidelity than possible before.

We use simple feed-forward encoder and decoder networks, thus our model is an attractive candidate for applications where the encoding and decoding speed is critical.

Additionally, this  allows us to only sample autoregressively in the compressed latent space, which is an order of magnitude faster than sampling in the pixel space, especially for large images.

We demonstrate that a multi-scale hierarchical organization of  VQ-VAE, augmented with powerful priors over the latent codes, is able to generate samples with quality that rivals that of state of the art Generative Adversarial Networks on multifaceted datasets such as ImageNet, while not suffering from GAN's known shortcomings such as mode collapse and lack of diversity.

scale and enhance VQ-VAE with powerful priors to generate near realistic images.

Generating Diverse High-Resolution Images with VQ-VAE


We present a graph neural network assisted Monte Carlo Tree Search approach for the classical traveling salesman problem (TSP).

We adopt a greedy algorithm framework to construct the optimal solution to TSP by adding the nodes successively.

A graph neural network (GNN) is trained to capture the local and global graph structure and give the prior probability of selecting each vertex every step.

The prior probability provides a heuristics for MCTS, and the MCTS output is an improved probability for selecting the successive vertex, as it is the feedback information by fusing the prior with the scouting procedure.

Experimental results on TSP up to 100 nodes demonstrate that the proposed method obtains shorter tours than other learning-based methods.

A Graph Neural Network Assisted Monte Carlo Tree Search Approach to Traveling Salesman Problem

graph neural network assist monte carlo tree search approach


Graph neural networks have recently achieved great successes in predicting quantum mechanical properties of molecules.

These models represent a molecule as a graph using only the distance between atoms (nodes) and not the spatial direction from one atom to another.

However, directional information plays a central role in empirical potentials for molecules, e.g. in angular potentials.

To alleviate this limitation we propose directional message passing, in which we embed the messages passed between atoms instead of the atoms themselves.

Each message is associated with a direction in coordinate space.

These directional message embeddings are rotationally equivariant since the associated directions rotate with the molecule.

We propose a message passing scheme analogous to belief propagation, which uses the directional information by transforming messages based on the angle between them.

Additionally, we use spherical Bessel functions to construct a theoretically well-founded, orthogonal radial basis that achieves better performance than the currently prevalent Gaussian radial basis functions while using more than 4x fewer parameters.

We leverage these innovations to construct the directional message passing neural network (DimeNet).

DimeNet outperforms previous GNNs on average by 77% on MD17 and by 41% on QM9.

Directional message passing incorporates spatial directional information to improve graph neural networks.

Directional Message Passing for Molecular Graphs


Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult.

The agent needs to learn  a latent representation together with a control policy to perform the task.

Fitting a high-capacity encoder using a scarce reward signal is not only extremely sample inefficient, but also prone to suboptimal convergence.

Two ways to improve sample efficiency are to learn a good feature representation and use off-policy algorithms.

We dissect various approaches of learning good latent features, and conclude that the image reconstruction loss is the essential ingredient that enables efficient and stable representation learning in image-based RL.

Following these findings, we devise an off-policy actor-critic algorithm with an auxiliary decoder that trains end-to-end and matches state-of-the-art performance across both model-free and model-based algorithms on many challenging control tasks.

We release our code to encourage future research on image-based RL.

We design a simple and efficient model-free off-policy method for image-based reinforcement learning that matches the state-of-the-art model-based methods in sample efficiency

Improving Sample Efficiency in Model-Free Reinforcement Learning from Images

reinforcement learning
sample efficiency
model-free


Large deep neural networks require huge memory to run and their running speed is sometimes too slow for real applications.

Therefore network size reduction with keeping accuracy is crucial for practical applications.

We present a novel neural network operator, chopout, with which neural networks are trained, even in a single training process, so as to truncated sub-networks perform as well as possible.

Chopout is easy to implement and integrate into most type of existing neural networks.

Furthermore it enables to reduce size of networks and latent representations even after training just by truncating layers.

We show its effectiveness through several experiments.

We present a novel simple operator, chopout, with which neural networks are trained, even in a single training process, so as to truncated sub-networks perform as well as possible.

Chopout: A Simple Way to Train Variable Sized Neural Networks at Once


Generative Adversarial Networks (GANs) have been shown to produce realistically looking synthetic images with remarkable success, yet their performance seems less impressive when the training set is highly diverse.

In order to provide a better fit to the target data distribution when the dataset includes many different classes, we propose a variant of the basic GAN model, a Multi-Modal Gaussian-Mixture GAN (GM-GAN), where the probability distribution over the latent space is a mixture of Gaussians.

We also propose a supervised variant which is capable of conditional sample synthesis.

In order to evaluate the model's performance, we propose a new scoring method which separately takes into account two (typically conflicting) measures - diversity vs. quality of the generated data.

Through a series of experiments, using both synthetic and real-world datasets, we quantitatively show that GM-GANs outperform baselines, both when evaluated using the commonly used Inception Score, and when evaluated using our own alternative scoring method.

In addition, we qualitatively demonstrate how the unsupervised variant of GM-GAN tends to map latent vectors sampled from different Gaussians in the latent space to samples of different classes in the data space.

We show how this phenomenon can be exploited for the task of unsupervised clustering, and provide quantitative evaluation showing the superiority of our method for the unsupervised clustering of image datasets.

Finally, we demonstrate a feature which further sets our model apart from other GAN models: the option to control the quality-diversity trade-off by altering, post-training, the probability distribution of the latent space.

This allows one to sample higher quality and lower diversity samples, or vice versa, according to one's needs.

Multi modal Guassian distribution of latent space in GAN models improves performance and allows to trade-off quality vs. diversity

Multi-Modal Generative Adversarial Networks for Diverse Datasets


Distributed optimization is essential for training large models on large datasets.

Multiple approaches have been proposed to reduce the communication overhead in distributed training, such as synchronizing only after performing multiple local SGD steps, and decentralized methods (e.g., using gossip algorithms) to decouple communications among workers.

Although these methods run faster than AllReduce-based methods, which use blocking communication before every update, the resulting models may be less accurate after the same number of updates.

Inspired by the BMUF method of Chen & Huo (2016), we propose a slow momentum (SloMo) framework, where workers periodically synchronize and perform a momentum update, after multiple iterations of a base optimization algorithm.

Experiments on image classification and machine translation tasks demonstrate that SloMo consistently yields improvements in optimization and generalization performance relative to the base optimizer, even when the additional overhead is amortized over many updates so that the SloMo runtime is on par with that of the base optimizer.

We provide theoretical convergence guarantees showing that SloMo converges to a stationary point of smooth non-convex losses.

Since BMUF is a particular instance of the SloMo framework, our results also correspond to the first theoretical convergence guarantees for BMUF.

SlowMo improves the optimization and generalization performance of communication-efficient decentralized algorithms without sacrificing speed.

SloMo: Improving Communication-Efficient Distributed SGD with Slow Momentum


Structural planning is important for producing long sentences, which is a missing part in current language generation models.

In this work, we add a planning phase in neural machine translation to control the coarse structure of output sentences.

The model first generates some planner codes, then predicts real output words conditioned on them.

The codes are learned to capture the coarse structure of the target sentence.

In order to learn the codes, we design an end-to-end neural network with a discretization bottleneck, which predicts the simplified part-of-speech tags of target sentences.

Experiments show that the translation performance are generally improved by planning ahead.

We also find that translations with different structures can be obtained by manipulating the planner codes.

Plan the syntactic structural of translation using codes

Discrete Structural Planning for Neural Machine Translation


Interpolation of data in deep neural networks has become a subject of significant research interest.

We prove that over-parameterized single layer fully connected autoencoders do not merely interpolate, but rather, memorize training data: they produce outputs in (a non-linear version of) the span of the training examples.

In contrast to fully connected autoencoders, we prove that depth is necessary for memorization in convolutional autoencoders.

Moreover, we observe that adding nonlinearity to deep convolutional autoencoders results in a stronger form of memorization: instead of outputting points in the span of the training images, deep convolutional autoencoders tend to output individual training images.

Since convolutional autoencoder components are building blocks of deep convolutional networks, we envision that our findings will shed light on the important question of the inductive bias in over-parameterized deep networks.

We identify memorization as the inductive bias of interpolation in overparameterized fully connected and convolutional auto-encoders.

Memorization in Overparameterized Autoencoders


The tremendous success of deep neural networks has motivated the need to better understand the fundamental properties of these networks, but many of the theoretical results proposed have only been for shallow networks.

In this paper, we study an important primitive for understanding the meaningful input space of a deep network: span recovery.

For $k<n$, let $\mathbf{A} \in \mathbb{R}^{k \times n}$ be the innermost weight matrix of an arbitrary feed forward neural network $M: \mathbb{R}^n \to  \mathbb{R}$, so $M(x)$ can be written as $M(x) = \sigma(\mathbf{A} x)$, for some network $\sigma: \mathbb{R}^k \to  \mathbb{R}$.

The goal is then to recover the row span of $\mathbf{A}$ given only oracle access to the value of $M(x)$. We show that if $M$ is a multi-layered network with ReLU activation functions, then partial recovery is possible: namely, we can provably recover $k/2$ linearly independent vectors in the row span of $\mathbf{A}$ using poly$(n)$ non-adaptive queries to $M(x)$.  Furthermore, if $M$ has differentiable activation functions, we demonstrate that \textit{full} span recovery is possible even when the output is first passed through a sign or $0/1$ thresholding function; in this case our algorithm is adaptive.

Empirically, we confirm that full span recovery is not always possible, but only for unrealistically thin layers.

For reasonably wide networks, we obtain full span recovery on both random networks and networks trained on MNIST data.

Furthermore, we demonstrate the utility of span recovery as an attack by inducing neural networks to misclassify data obfuscated by controlled random noise as sensical inputs.

We provably recover the span of a deep multi-layered neural network with latent structure and empirically apply efficient span recovery algorithms to attack networks by obfuscating inputs.

Span Recovery for Deep Neural Networks with Applications to Input Obfuscation


In this paper, we study the implicit regularization of the gradient descent algorithm in homogeneous neural networks, including fully-connected and convolutional neural networks with ReLU or LeakyReLU activations.

In particular, we study the gradient descent or gradient flow (i.e., gradient descent with infinitesimal step size) optimizing the logistic loss or cross-entropy loss of any homogeneous model (possibly non-smooth), and show that if the training loss decreases below a certain threshold, then we can define a smoothed version of the normalized margin which increases over time.

We also formulate a natural constrained optimization problem related to margin maximization, and prove that both the normalized margin and its smoothed version converge to the objective value at a KKT point of the optimization problem.

Our results generalize the previous results for logistic regression with one-layer or multi-layer linear networks, and provide more quantitative convergence results with weaker assumptions than previous results for homogeneous smooth neural networks.

We conduct several experiments to justify our theoretical finding on MNIST and CIFAR-10 datasets.

Finally, as margin is closely related to robustness, we discuss potential benefits of training longer for improving the robustness of the model.

We study the implicit bias of gradient descent and prove under a minimal set of assumptions that the parameter direction of homogeneous models converges to KKT points of a natural margin maximization problem.

Gradient Descent Maximizes the Margin of Homogeneous Neural Networks


Long-term video prediction is highly challenging since it entails simultaneously capturing spatial and temporal information across a long range of image frames.Standard recurrent models are ineffective since they are prone to error propagation and cannot effectively capture higher-order correlations.

A potential solution is to extend to higher-order spatio-temporal recurrent models.

However, such a model requires  a  large number of parameters and operations, making it intractable  to learn in practice and is prone to overfitting.

In this work, we propose convolutional tensor-train LSTM (Conv-TT-LSTM), which  learns higher-orderConvolutional LSTM (ConvLSTM) efficiently using convolutional  tensor-train decomposition (CTTD).

Our proposed model naturally incorporates higher-order spatio-temporal information at a small cost of memory and computation by using efficient low-rank tensor representations.

We evaluate our model on Moving-MNIST and KTH datasets and show improvements over standard ConvLSTM and better/comparable results to other ConvLSTM-based approaches, but with much fewer parameters.

we propose convolutional tensor-train LSTM,  which learns higher-order Convolutional LSTM efficiently using convolutional tensor-train decomposition.

Convolutional Tensor-Train LSTM for Long-Term Video Prediction


While deep neural networks have shown outstanding results in a wide range of applications,

learning from a very limited number of examples is still a challenging

Despite the difficulties of the few-shot learning, metric-learning techniques

showed the potential of the neural networks for this task.

perform well, they don’t provide satisfactory results.

metric-learning is extended with Support Vector Machines (SVM) working mechanism,

which is well known for generalization capabilities on a small dataset.

Furthermore, this paper presents an end-to-end learning framework for training

adaptive kernel SVMs, which eliminates the problem of choosing a correct kernel

Next, the one-shot learning problem is redefined

Then the model was tested on vision task (using Omniglot

dataset) and speech task (using TIMIT dataset) as well.

using Omniglot dataset improved accuracy from 98.1% to 98.5% on the one-shot

classification task and from 98.9% to 99.3% on the few-shot classification task.

The proposed method is an end-to-end neural SVM, which is optimized for few-shot learning.

Make SVM great again with Siamese kernel for  few-shot learning


Most existing 3D CNN structures for video representation learning are clip-based methods, and do not consider video-level temporal evolution of spatio-temporal features.

In this paper, we propose Video-level 4D Convolutional Neural Networks, namely V4D, to model the evolution of long-range spatio-temporal representation with 4D convolutions, as well as preserving 3D spatio-temporal representations with residual connections.

We further introduce the training and inference methods for the proposed V4D.

Extensive experiments are conducted on three video recognition benchmarks, where V4D achieves excellent results, surpassing recent 3D CNNs by a large margin.

A novel 4D CNN structure for video-level representation learning, surpassing  recent 3D CNNs.

V4D: 4D Convonlutional Neural Networks for Video-level Representation Learning


We study the problem of learning and optimizing through physical simulations via differentiable programming.

We present DiffSim, a new differentiable programming language tailored for building high-performance differentiable physical simulations.

We demonstrate the performance and productivity of our language in gradient-based learning and optimization tasks on 10 different physical simulators.

For example, a differentiable elastic object simulator written in our language is 4.6x faster than the hand-engineered CUDA version yet runs as fast, and is 188x faster than TensorFlow.

Using our differentiable programs, neural network controllers are typically optimized within only tens of iterations.

Finally, we share the lessons learned from our experience developing these simulators, that is, differentiating physical simulators does not always yield useful gradients of the physical system being simulated.

We systematically study the underlying reasons and propose solutions to improve gradient quality.

We study the problem of learning and optimizing through physical simulations via differentiable programming, using our proposed DiffSim programming language and compiler.

Differentiable Programming for Physical Simulation

physical simulation
differentiable programming


Local explanation frameworks aim to rationalize particular decisions made by a black-box prediction model.

Existing techniques are often restricted to a specific type of predictor or based on input saliency, which may be undesirably sensitive to factors unrelated to the model's decision making process.

We instead propose sufficient input subsets that identify minimal subsets of features whose observed values alone suffice for the same decision to be reached, even if all other input feature values are missing.

General principles that globally govern a model's decision-making can also be revealed by searching for clusters of such input patterns across many data points.

Our approach is conceptually straightforward, entirely model-agnostic, simply implemented using instance-wise backward selection, and able to produce more concise rationales than existing techniques.

We demonstrate the utility of our interpretation method on neural network models trained on text and image data.

We present a method for interpreting black-box models by using instance-wise backward selection to identify minimal subsets of features that alone suffice to justify a particular decision made by the model.

Local and global model interpretability via backward selection and clustering


In this paper, we tackle the problem of detecting samples that are not drawn from the training distribution, i.e., out-of-distribution (OOD) samples, in classification.

Many previous studies have attempted to solve this problem by regarding samples with low classification confidence as OOD examples using deep neural networks (DNNs).

However, on difficult datasets or models with low classification ability, these methods incorrectly regard in-distribution samples close to the decision boundary as OOD samples.

This problem arises because their approaches use only the features close to the output layer and disregard the uncertainty of the features.

Therefore, we propose a method that extracts the uncertainties of features in each layer of DNNs using a reparameterization trick and combines them.

In experiments, our method outperforms the existing methods by a large margin, achieving state-of-the-art detection performance on several datasets and classification models.

For example, our method increases the AUROC score of prior work (83.8%) to 99.8% in DenseNet on the CIFAR-100 and Tiny-ImageNet datasets.

We propose a method that extracts the uncertainties of features in each layer of DNNs and combines them for detecting OOD samples when solving classification tasks.

Out-of-Distribution Detection Using Layerwise Uncertainty in Deep Neural Networks


In this paper, we explore new approaches to combining information encoded within the learned representations of autoencoders.

We explore models that are capable of combining the attributes of multiple inputs such that a resynthesised output is trained to fool an adversarial discriminator for real versus synthesised data.

Furthermore, we explore the use of such an architecture in the context of semi-supervised learning, where we learn a mixing function whose objective is to produce interpolations of hidden states, or masked combinations of latent representations that are consistent with a conditioned class label.

We show quantitative and qualitative evidence that such a formulation is an interesting avenue of research.

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.

Adversarial Mixup Resynthesizers


We outline the problem of concept drifts for time series data.

In this work, we analyze the temporal inconsistency of streaming wireless signals in the context of device-free passive indoor localization.

We show that data obtained from WiFi channel state information (CSI) can be used to train a robust system capable of performing room level localization.

One of the most challenging issues for such a system is the movement of input data distribution to an unexplored space over time, which leads to an unwanted shift in the learned boundaries of the output space.

In this work, we propose a phase and magnitude augmented feature space along with a standardization technique that is little affected by drifts.

We show that this robust representation of the data yields better learning accuracy and requires less number of retraining.

We introduce an augmented robust feature space for streaming wifi data that is capable of tackling concept drift for indoor localization

HANDLING CONCEPT DRIFT  IN WIFI-BASED INDOOR LOCALIZATION USING REPRESENTATION LEARNING


Federated learning improves data privacy and efficiency in machine learning performed over networks of distributed devices, such as mobile phones, IoT and wearable devices, etc.

Yet models trained with federated learning can still fail to generalize to new devices due to the problem of domain shift.

Domain shift occurs when the labeled data collected by source nodes statistically differs from the target node's unlabeled data.

In this work, we present a principled approach to the problem of federated domain adaptation, which aims to align the representations learned among the different nodes with the data distribution of the target node.

Our approach extends adversarial adaptation techniques to the constraints of the federated setting.

In addition, we devise a dynamic attention mechanism and leverage feature disentanglement to enhance knowledge transfer.

Empirically, we perform extensive experiments on several image and text classification tasks and show promising results under unsupervised federated domain adaptation setting.

Federated Adversarial Domain Adaptation


A capsule is a group of neurons whose outputs represent different properties of the same entity.

Each layer in a capsule network contains many capsules.

We describe a version of capsules in which each capsule has a logistic unit to represent the presence of an entity and a 4x4 matrix which could learn to represent the relationship between that entity and the viewer (the pose).

A capsule in one layer votes for the pose matrix of many different capsules in the layer above by multiplying its own pose matrix by trainable viewpoint-invariant transformation matrices that could learn to represent part-whole relationships.

Each of these votes is weighted by an assignment coefficient.

These coefficients are iteratively updated for each image using the Expectation-Maximization algorithm such that the output of each capsule is routed to a capsule in the layer above that receives a cluster of similar votes.

The transformation matrices are trained discriminatively by backpropagating through the unrolled iterations of EM between each pair of adjacent capsule layers.

On the smallNORB benchmark, capsules reduce the number of test errors by 45\% compared to the state-of-the-art.

Capsules also show far more resistance to white box adversarial attacks than our baseline convolutional neural network.

Capsule networks with learned pose matrices and EM routing improves state of the art classification on smallNORB, improves generalizability to new view points, and white box adversarial robustness.

Matrix capsules with EM routing


We study a general formulation of program synthesis called syntax-guided synthesis(SyGuS) that concerns synthesizing a program that follows a given grammar and satisfies a given logical specification.

Both the logical specification and the grammar have complex structures and can vary from task to task, posing significant challenges for learning across different tasks.

Furthermore, training data is often unavailable for domain specific synthesis tasks.

To address these challenges, we propose a meta-learning framework that learns a transferable policy from only weak supervision.

Our framework consists of three components:

1) an encoder, which embeds both the logical specification and grammar at the same time using a graph neural network;

2) a grammar adaptive policy network which enables learning a transferable policy; and

3) a reinforcement learning algorithm that jointly trains the embedding and adaptive policy.

We evaluate the framework on 214 cryptographic circuit synthesis tasks.

It solves 141 of them in the out-of-box solver setting, significantly outperforming a similar search-based approach but without learning, which solves only 31.

The result is comparable to two state-of-the-art classical synthesis engines, which solve 129 and 153 respectively.

In the meta-solver setting, the framework can efficiently adapt to unseen tasks and achieves speedup ranging from 2x up to 100x.

We propose a meta-learning framework that learns a transferable policy from only weak supervision to solve synthesis tasks with different logical specifications and grammars.

Learning a Meta-Solver for Syntax-Guided Program Synthesis


Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence.

Recent approach for this task either apply a fixed policy on transformer, or a learnable monotonic attention on a weaker recurrent neural network based structure.

In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which introduced the monotonic attention mechanism to multihead attention.

We also introduced two novel interpretable approaches for latency control that are specifically designed for multiple attentions.

We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach.

Make the transformer streamable with monotonic attention.

Monotonic Multihead Attention


This paper presents a novel two-step approach for the fundamental problem of learning an optimal map from one distribution to another.

First, we learn an optimal transport (OT) plan, which can be thought as a one-to-many map between the two distributions.

To that end, we propose a stochastic dual approach of regularized OT, and show empirically that it scales better than a recent related approach when the amount of samples is very large.

Second, we estimate a Monge map as a deep neural network learned by approximating the barycentric projection of the previously-obtained OT plan.

This parameterization allows generalization of the mapping outside the support of the input measure.

We prove two theoretical stability results of regularized OT which show that our estimations converge to the OT and Monge map between the underlying continuous measures.

We showcase our proposed approach on two applications: domain adaptation and generative modeling.

Learning optimal mapping with deepNN between distributions along with theoretical guarantees.

Large Scale Optimal Transport and Mapping Estimation


Learning effective text representations is a key foundation for numerous machine learning and NLP applications.

While the celebrated Word2Vec technique yields semantically rich word representations, it is less clear whether sentence or document representations should be built upon word representations or from scratch.

Recent work has demonstrated that a distance measure between documents called \emph{Word Mover's Distance} (WMD) that aligns semantically similar words, yields unprecedented KNN classification accuracy.

However, WMD is very expensive to compute, and is harder to apply beyond simple KNN than feature embeddings.

In this paper, we propose the \emph{Word Mover's Embedding } (WME), a novel approach to building an unsupervised document (sentence) embedding from pre-trained word embeddings.

Our technique extends the theory of \emph{Random Features} to show convergence of the inner product between WMEs to a positive-definite kernel that can be interpreted as a soft version of (inverse) WMD.

The proposed embedding is more efficient and flexible than WMD in many situations.

As an example, WME with a simple linear classifier reduces the computational cost of WMD-based KNN \emph{from cubic to linear} in document length and \emph{from quadratic to linear} in number of samples, while simultaneously improving accuracy.

In experiments on 9 benchmark text classification datasets and 22 textual similarity tasks the proposed technique consistently matches or outperforms state-of-the-art techniques, with significantly higher accuracy on problems of short length.

A novel approach to building an unsupervised document (sentence) embeddings from pre-trainedword embeddings

Word Mover's Embedding: From Word2Vec to Document Embedding


We present a new approach to assessing the robustness of neural networks based on estimating the proportion of inputs for which a property is violated.

Specifically, we estimate the probability of the event that the property is violated under an input model.

Our approach critically varies from the formal verification framework in that when the property can be violated, it provides an informative notion of how robust the network is, rather than just the conventional assertion that the network is not verifiable.

Furthermore, it provides an ability to scale to larger networks than formal verification approaches.

Though the framework still provides a formal guarantee of satisfiability whenever it successfully finds one or more violations, these advantages do come at the cost of only providing a statistical estimate of unsatisfiability whenever no violation is found.

Key to the practical success of our approach is an adaptation of multi-level splitting, a Monte Carlo approach for estimating the probability of rare events, to our statistical robustness framework.

We demonstrate that our approach is able to emulate formal verification procedures on benchmark problems, while scaling to larger networks and providing reliable additional information in the form of accurate estimates of the violation probability.

We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated.

A Statistical Approach to Assessing Neural Network Robustness

statistical approach assess neural network robustness


Recent pretrained transformer-based language models have set state-of-the-art performances on various NLP datasets.

However, despite their great progress, they suffer from various structural and syntactic biases.

In this work, we investigate the lexical overlap bias, e.g., the model classifies two sentences that have a high lexical overlap as entailing regardless of their underlying meaning.

To improve the robustness, we enrich input sentences of the training data with their automatically detected predicate-argument structures.

This enhanced representation allows the transformer-based models to learn different attention patterns by focusing on and recognizing the major semantically and syntactically important parts of the sentences.

We evaluate our solution for the tasks of natural language inference and grounded commonsense inference using the BERT, RoBERTa, and XLNET models.

We evaluate the models' understanding of syntactic variations, antonym relations, and named entities in the presence of lexical overlap.

Our results show that the incorporation of predicate-argument structures during fine-tuning considerably improves the robustness, e.g.,  about 20pp on discriminating different named entities, while it incurs no additional cost at the test time and does not require changing the model or the training procedure.

Enhancing the robustness of pretrained transformer models against the lexical overlap bias by extending the input sentences of the training data with their corresponding predicate-argument structures

Overcoming the Lexical Overlap Bias Using Predicate-Argument Structures

lexical overlap bias
predicate-argument structure


We propose a method for quantifying uncertainty in neural network regression models when the targets are real values on a $d$-dimensional simplex, such as probabilities.

We show that each target can be modeled as a sample from a Dirichlet distribution, where the parameters of the Dirichlet are provided by the output of a neural network, and that the combined model can be trained using the gradient of the data likelihood.

This approach provides interpretable predictions in the form of multidimensional distributions, rather than point estimates, from which one can obtain confidence intervals or quantify risk in decision making.

Furthermore, we show that the same approach can be used to model targets in the form of empirical counts as samples from the Dirichlet-multinomial compound distribution.

In experiments, we verify that our approach provides these benefits without harming the performance of the point estimate predictions on two diverse applications: (1) distilling deep convolutional networks trained on CIFAR-100, and (2) predicting the location of particle collisions in the XENON1T Dark Matter detector.

Neural network regression should use Dirichlet output distribution when targets are probabilities in order to quantify uncertainty of predictions.

Neural Network Regression with Beta, Dirichlet, and Dirichlet-Multinomial Outputs

neural network regression
dirichlet
output


Deep neural networks have achieved outstanding performance in many real-world applications with the expense of huge computational resources.

The DenseNet, one of the recently proposed neural network architecture, has achieved the state-of-the-art performance in many visual tasks.

However, it has great redundancy due to the dense connections of the internal structure, which leads to high computational costs in training such dense networks.

To address this issue,  we design a reinforcement learning framework to search for efficient DenseNet architectures with layer-wise pruning (LWP) for different tasks, while retaining the original advantages of DenseNet, such as feature reuse, short paths, etc.

In this framework, an agent evaluates the importance of each connection between any two block layers, and prunes the redundant connections.

In addition, a novel reward-shaping trick is introduced to make DenseNet reach a better trade-off between accuracy and float point operations (FLOPs).

Our experiments show that DenseNet with LWP is more compact and efficient than existing alternatives.

Learning to Search Efficient DenseNet with Layer-wise Pruning

learn search efficient densenet layer-wise pruning


We consider the setting of an agent with a fixed body interacting with an unknown and uncertain external world.

We show that models trained to predict proprioceptive information about the agent's body come to represent objects in the external world.

In spite of being trained with only internally available signals, these dynamic body models come to represent external objects through the necessity of predicting their effects on the agent's own body.

That is, the model learns holistic persistent representations of objects in the world, even though the only training signals are body signals.

Our dynamics model is able to successfully predict distributions over 132 sensor readings over 100 steps into the future and we demonstrate that even when the body is no longer in contact with an object, the latent variables of the dynamics model continue to represent its shape.

We show that active data collection by maximizing the entropy of predictions about the body---touch sensors, proprioception and vestibular information---leads to learning of dynamic models that show superior performance when used for control.

We also collect data from a real robotic hand and show that the same models can be used to answer questions about properties of objects in the real world.

Videos with qualitative results of our models are available at https://goo.gl/mZuqAV.

We train predictive models on proprioceptive information and show they represent properties of external objects.

Learning Awareness Models


Inspired by the success of generative adversarial networks (GANs) in image domains, we introduce a novel hierarchical architecture for learning characteristic topological features from a single arbitrary input graph via GANs.

The hierarchical architecture consisting of multiple GANs preserves both local and global topological features, and automatically partitions the input graph into representative stages for feature learning.

The stages facilitate reconstruction and can be used as indicators of the importance of the associated topological structures.

Experiments show that our method produces subgraphs retaining a wide range of topological features, even in early reconstruction stages.

This paper contains original research on combining the use of GANs and graph topological analysis.

A GAN based method to learn important topological features of an arbitrary input graph.

Graph Topological Features via GAN


In Chinese societies, superstition is of paramount importance, and vehicle license plates with desirable numbers can fetch very high prices in auctions.

Unlike other valuable items, license plates are not allocated an estimated price before auction.

I propose that the task of predicting plate prices can be viewed as a natural language processing (NLP) task, as the value depends on the meaning of each individual character on the plate and its semantics.

I construct a deep recurrent neural network (RNN) to predict the prices of vehicle license plates in Hong Kong, based on the characters on a plate.

I demonstrate the importance of having a deep network and of retraining.

Evaluated on 13 years of historical auction prices, the deep RNN's predictions can explain over 80 percent of price variations, outperforming previous models by a significant margin.

I also demonstrate how the model can be extended to become a search engine for plates and to provide estimates of the expected price distribution.

Predicting auction price of vehicle license plates in Hong Kong with deep recurrent neural network, based on the characters on the plates.

Predicting Auction Price of Vehicle License Plate with Deep Recurrent Neural Network

predict auction price vehicle license plate


We present a new latent model of natural images that can be learned on large-scale datasets.

The learning process provides a latent embedding for every image in the training dataset, as well as a deep convolutional network that maps the latent space to the image space.

After training, the new model provides a strong and universal image prior for a variety of image restoration tasks such as large-hole inpainting, superresolution, and colorization.

To model high-resolution natural images, our approach uses latent spaces of very high dimensionality (one to two orders of magnitude higher than previous latent image models).

To tackle this high dimensionality, we use latent spaces with a special manifold structure (convolutional manifolds) parameterized by a ConvNet of a certain architecture.

In the experiments, we compare the learned latent models with latent models learned by autoencoders, advanced variants of generative adversarial networks, and a strong baseline system using simpler parameterization of the latent space.

Our model outperforms the competing approaches over a range of restoration tasks.

We present a new deep latent model of natural images that can be trained from unlabeled datasets and can be utilized to solve various image restoration tasks.

Latent Convolutional Models


Automatic Essay Scoring (AES) has been an active research area as it can greatly reduce the workload of teachers and prevents subjectivity bias .

Most recent AES solutions apply deep neural network (DNN)-based models with regression, where the neural neural-based encoder learns an essay representation that helps differentiate among the essays and the corresponding essay score is inferred by a regressor.

Such DNN approach usually requires a lot of expert-rated essays as training data in order to learn a good essay representation for accurate scoring.

However, such data is usually expensive and thus is sparse.

Inspired by the observation that human usually scores an essay by comparing it with some references, we propose a Siamese framework called Referee Network (RefNet) which allows the model to compare the quality of two essays by capturing the relative features that can differentiate the essay pair.

The proposed framework can be applied as an extension to regression models as it can capture additional relative features on top of internal information.

Moreover, it intrinsically augment the data by pairing thus is ideal for handling data sparsity.

Experiment shows that our framework can significantly improve the existing regression models and achieve acceptable performance even when the training data is greatly reduced.

Automatically score essays on sparse data by comparing new essays with known samples with Referee Network.

RefNet: Automatic Essay Scoring by Pairwise Comparison


One of the fundamental tasks in understanding genomics is the problem of predicting Transcription Factor Binding Sites (TFBSs).

With more than hundreds of Transcription Factors (TFs) as labels, genomic-sequence based TFBS prediction is a challenging multi-label classification task.

There are two major biological mechanisms for TF binding: (1) sequence-specific binding patterns on genomes known as “motifs” and (2) interactions among TFs known as co-binding effects.

In this paper, we propose a novel deep architecture, the Prototype Matching Network (PMN) to mimic the TF binding mechanisms.

Our PMN model automatically extracts prototypes (“motif”-like features) for each TF through a novel prototype-matching loss.

Borrowing ideas from few-shot matching models, we use the notion of support set of prototypes and an LSTM to learn how TFs interact and bind to genomic sequences.

On a reference TFBS dataset with 2.1 million genomic sequences, PMN significantly outperforms baselines and validates our design choices empirically.

To our knowledge, this is the first deep learning architecture that introduces prototype learning and considers TF-TF interactions for large scale TFBS prediction.

Not only is the proposed architecture accurate, but it also models the underlying biology.

We combine the matching network framework for few shot learning into a large scale multi-label model for genomic sequence classification.

Prototype Matching Networks for Large-Scale Multi-label  Genomic Sequence Classification

genomic sequence classification
network
multi-label


Previous work shows that adversarially robust generalization requires larger sample complexity, and the same dataset, e.g., CIFAR-10, which enables good standard accuracy may not suffice to train robust models.

Since collecting new training data could be costly, we focus on better utilizing the given data by inducing the regions with high sample density in the feature space, which could lead to locally sufficient samples for robust learning.

We first formally show that the softmax cross-entropy (SCE) loss and its variants convey inappropriate supervisory signals, which encourage the learned feature points to spread over the space sparsely in training.

This inspires us to propose the Max-Mahalanobis center (MMC) loss to explicitly induce dense feature regions in order to benefit robustness.

Namely, the MMC loss encourages the model to concentrate on learning ordered and compact representations, which gather around the preset optimal centers for different classes.

We empirically demonstrate that applying the MMC loss can significantly improve robustness even under strong adaptive attacks, while keeping state-of-the-art accuracy on clean inputs with little extra computation compared to the SCE loss.

Applying the softmax function in training leads to indirect and unexpected supervision on features. We propose a new training objective to explicitly induce dense feature regions for locally sufficient samples to benefit adversarial robustness.

Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness


We present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level.

We first identify a group of interpretable units that are closely related to object concepts with a segmentation-based network dissection method.

Then, we examine the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.

Finally, we examine the contextual relationship between these units and their surrounding by inserting the discovered object concepts into new images.

We show several practical applications enabled by our framework, from comparing internal representations across different layers and models, to improving GANs by locating and removing artifact-causing units, to interactively manipulating objects in the scene.

GAN representations are examined in detail, and sets of representation units are found that control the generation of semantic concepts in the output.

Visualizing and Understanding GANs


We present a simple nearest-neighbor (NN) approach that synthesizes high-frequency photorealistic images from an ``incomplete'' signal such as a low-resolution image, a surface normal map, or edges.

Current state-of-the-art deep generative models designed for such conditional image synthesis lack two important things: (1) they are unable to generate a large set of diverse outputs, due to the mode collapse problem.

(2) they are not interpretable, making it difficult to control the synthesized output.

We demonstrate that NN approaches potentially address such limitations, but suffer in accuracy on small datasets.

We design a simple pipeline that combines the best of both worlds:  the first stage uses a convolutional neural network (CNN) to map the input to a (overly-smoothed) image, and the second stage uses a pixel-wise nearest neighbor method to map the smoothed output to multiple high-quality, high-frequency outputs in a controllable manner.

Importantly, pixel-wise matching allows our method to compose novel high-frequency content by cutting-and-pasting pixels from different training exemplars.

We demonstrate our approach for various input modalities, and for various domains ranging from human faces, pets, shoes, and handbags.

Pixel-wise nearest neighbors used for generating multiple images from incomplete priors such as a low-res images, surface normals, edges etc.

PixelNN: Example-based Image Synthesis


Neural networks are vulnerable to adversarial examples and researchers have proposed many heuristic attack and defense mechanisms.

We address this problem through the principled lens of distributionally robust optimization, which guarantees performance under adversarial input perturbations.

By considering a Lagrangian penalty formulation of perturbing the underlying data distribution in a Wasserstein ball, we provide a training procedure that augments model parameter updates with worst-case perturbations of training data.

For smooth losses, our procedure provably achieves moderate levels of robustness with little computational or statistical cost relative to empirical risk minimization.

Furthermore, our statistical guarantees allow us to efficiently certify robustness for the population loss.

For imperceptible perturbations, our method matches or outperforms heuristic approaches.

We provide a fast, principled adversarial training procedure with computational and statistical performance guarantees.

Certifying Some Distributional Robustness with Principled Adversarial Training


Amortized inference has led to efficient approximate inference for large datasets.

The quality of posterior inference is largely determined by two factors:

a) the ability of the variational distribution to model the true posterior and

b) the capacity of the recognition network to generalize inference over all datapoints.

We analyze approximate inference in variational autoencoders in terms of these factors.

We find that suboptimal inference is often due to amortizing inference rather than the limited complexity of the approximating distribution.

We show that this is due partly to the generator learning to accommodate the choice of approximation.

Furthermore, we show that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation.

We decompose the gap between the marginal log-likelihood and the evidence lower bound and study the effect of the approximate posterior on the true posterior distribution in VAEs.

Inference Suboptimality in Variational Autoencoders


In this paper, we propose a framework that leverages semi-supervised models to improve unsupervised clustering performance.

To leverage semi-supervised models, we first need to automatically generate labels, called pseudo-labels.

We find that prior approaches for generating pseudo-labels hurt clustering performance because of their low accuracy.

Instead, we use an ensemble of deep networks  to construct a similarity graph, from which we extract high accuracy pseudo-labels.

The approach of finding high quality pseudo-labels using ensembles and training the semi-supervised model is iterated, yielding continued improvement.

We show that our approach outperforms state of the art clustering results for multiple image and text datasets.

For example, we achieve 54.6% accuracy for CIFAR-10 and 43.9% for 20news, outperforming state of the art by 8-12% in absolute terms.

Using ensembles and pseudo labels for unsupervised clustering

Unsupervised Clustering using Pseudo-semi-supervised Learning


This paper concerns dictionary learning, i.e., sparse coding, a fundamental representation learning problem.

We show that a subgradient descent algorithm, with random initialization, can recover orthogonal dictionaries on a natural nonsmooth, nonconvex L1 minimization formulation of the problem, under mild statistical assumption on the data.

This is in contrast to previous provable methods that require either expensive computation or delicate initialization schemes.

Our analysis develops several tools for characterizing landscapes of nonsmooth functions, which might be of independent interest for provable training of deep networks with nonsmooth activations (e.g., ReLU), among other applications.

Preliminary synthetic and real experiments corroborate our analysis and show that our algorithm works well empirically in recovering orthogonal dictionaries.

Efficient dictionary learning by L1 minimization via a novel analysis of the non-convex non-smooth geometry.

Subgradient Descent Learns Orthogonal Dictionaries


We study model recovery for data classification, where the training labels are generated from a one-hidden-layer fully -connected neural network with sigmoid activations, and the goal is to recover the weight vectors of the neural network.

We prove that under Gaussian inputs, the empirical risk function using cross entropy exhibits strong convexity and smoothness uniformly in a local neighborhood of the ground truth, as soon as the sample complexity is sufficiently large.

This implies that if initialized in this neighborhood, which can be achieved via the tensor method, gradient descent converges linearly to a critical point that is provably close to the ground truth without requiring a fresh set of samples at each iteration.

To the best of our knowledge, this is the first global convergence guarantee established for the empirical risk minimization using cross entropy via gradient descent for learning one-hidden-layer neural networks, at the near-optimal sample and computational complexity with respect to the network input dimension.

We provide the first theoretical analysis of guaranteed recovery of one-hidden-layer neural networks under cross entropy loss for classification problems.

Guaranteed Recovery of One-Hidden-Layer Neural Networks via Cross Entropy

guarantee recovery one-hidden-layer neural network


With the deployment of neural networks on mobile devices and the necessity of transmitting neural networks over limited or expensive channels, the file size of trained model was identified as bottleneck.

of neural networks which is based on transform coding for convolutional and dense layers and on clustering for biases and normalizations.

With this codec, we achieve average compression factors between 7.9–9.3 while the accuracy of the compressed networks for image classification decreases only by 1%–2%, respectively.

Our neural network codec (which is based on transform coding and clustering) enables a low complexity and high efficient transparent compression of neural networks.

Neural Network Compression using Transform Coding and Clustering

transform code
neural network
compression


As Machine Learning (ML) gets applied to security-critical or sensitive domains, there is a growing need for integrity and privacy for outsourced ML computations.

A pragmatic solution comes from Trusted Execution Environments (TEEs), which use hardware and software protections to isolate sensitive computations from the untrusted software stack.

However, these isolation guarantees come at a price in performance, compared to untrusted alternatives.

This paper initiates the study of high performance execution of Deep Neural Networks (DNNs) in TEEs by efficiently partitioning DNN computations between trusted and untrusted devices.

Building upon an efficient outsourcing scheme for matrix multiplication, we propose Slalom, a framework that securely delegates execution of all linear layers in a DNN from a TEE (e.g., Intel SGX or Sanctum) to a faster, yet untrusted, co-located processor.

We evaluate Slalom by running DNNs in an Intel SGX enclave, which selectively delegates work to an untrusted GPU.

For canonical DNNs (VGG16, MobileNet and ResNet variants) we obtain 6x to 20x increases in throughput for verifiable inference, and 4x to 11x for verifiable and private inference.

We accelerate secure DNN inference in trusted execution environments (by a factor 4x-20x) by selectively outsourcing the computation of linear layers to a faster yet untrusted co-processor.

Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware


While deep neural networks are a highly successful model class, their large memory footprint puts considerable strain on energy consumption, communication bandwidth, and storage requirements.

Consequently, model size reduction has become an utmost goal in deep learning.

A typical approach is to train a set of deterministic weights, while applying certain techniques such as pruning and quantization, in order that the empirical weight distribution becomes amenable to Shannon-style coding schemes.

However, as shown in this paper, relaxing weight determinism and using a full variational distribution over weights allows for more efficient coding schemes and consequently higher compression rates.

In particular, following the classical bits-back argument, we encode the network weights using a random sample, requiring only a number of bits corresponding to the Kullback-Leibler divergence between the sampled variational distribution and the encoding distribution.

By imposing a constraint on the Kullback-Leibler divergence, we are able to explicitly control the compression rate, while optimizing the expected loss on the training set.

The employed encoding scheme can be shown to be close to the optimal information-theoretical lower bound, with respect to the employed variational family.

Our method sets new state-of-the-art in neural network compression, as it strictly dominates previous approaches in a Pareto sense: On the benchmarks LeNet-5/MNIST and VGG-16/CIFAR-10, our approach yields the best test performance for a fixed memory budget, and vice versa, it achieves the highest compression rates for a fixed test performance.

This paper proposes an effective method to compress neural networks based on recent results in information theory.

Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters


Most existing neural networks for learning graphs deal with the issue of permutation invariance by conceiving of the network as a message passing scheme, where each node sums the feature vectors coming from its neighbors.

We argue that this imposes a limitation on their representation power, and instead propose a new general architecture for representing objects consisting of a hierarchy of parts, which we call Covariant Compositional Networks (CCNs).

Here covariance means that the activation of each neuron must transform in a specific way under permutations, similarly to steerability in CNNs.

We achieve covariance by making each activation transform according to a tensor representation of the permutation group, and derive the corresponding tensor aggregation rules that each neuron must implement.

Experiments show that CCNs can outperform competing methods on some standard graph learning benchmarks.

A general framework for creating covariant graph neural networks

Covariant Compositional Networks For Learning Graphs


In recent years, three-dimensional convolutional neural network (3D CNN) are intensively applied in the video analysis and action recognition and receives good performance.

However, 3D CNN leads to massive computation and storage consumption, which hinders its deployment on mobile and embedded devices.

In this paper, we propose a three-dimensional regularization-based pruning method to assign different regularization parameters to different weight groups based on their importance to the network.

Our experiments show that the proposed method outperforms other popular methods in this area.

In this paper, we propose a three-dimensional regularization-based pruning method to accelerate the 3D-CNN.

Three Dimensional Convolutional Neural Network Pruning with Regularization-Based Method


In this paper, we propose data statements as a design solution and professional practice for natural language processing technologists, in both research and development — through the adoption and widespread use of data statements, the field can begin to address critical scientific and ethical issues that result from the use of data from certain populations in the development of technology for other populations.

We present a form that data statements can take and explore the implications of adopting them as part of regular practice.

We argue that data statements will help alleviate issues related to exclusion and bias in language technology; lead to better precision in claims about how NLP research can generalize and thus better engineering results; protect companies from public embarrassment; and ultimately lead to language technology that meets its users in their own preferred linguistic style and furthermore does not mis- represent them to others.

A practical proposal for more ethical and responsive NLP technology, operationalizing transparency of test and training data

Data Statements for NLP: Toward Mitigating System Bias and Enabling Better Science


We introduce CGNN, a framework to learn functional causal models as generative neural networks.

These networks are trained using backpropagation to minimize the maximum mean discrepancy to the observed data.

Unlike previous approaches, CGNN leverages both conditional independences and distributional asymmetries to seamlessly discover bivariate and multivariate
 causal structures, with or without hidden variables.

CGNN does not only estimate the causal structure, but a full and differentiable generative model of the data.

Throughout an extensive variety of experiments, we illustrate the competitive  esults of CGNN w.r.t state-of-the-art alternatives in observational causal discovery on both simulated and real data, in the tasks of cause-effect inference, v-structure identification, and multivariate causal discovery.

Discover the structure of functional causal models with generative neural networks

Causal Generative Neural Networks


Network pruning is widely used for reducing the heavy computational cost of deep models.

A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning.

In this work, we make a rather surprising observation: fine-tuning a pruned model only gives comparable or even worse performance than training that model with randomly initialized weights.

1) training a large, over-parameterized model is not necessary to obtain an efficient final model,

2) learned "important" weights of the large model are not necessarily useful for the small pruned model,

3) the pruned architecture itself, rather than a set of inherited weights, is what leads to the efficiency benefit in the final model, which suggests that some pruning algorithms could be seen as performing network architecture search.

In network pruning, fine-tuning a pruned model only gives comparable or worse performance than training it from scratch. This advocate a rethinking of existing pruning algorithms.

Rethinking the Value of Network Pruning


Clustering is the central task in unsupervised learning and data mining.

k-means is one of the most widely used clustering algorithms.

Unfortunately, it is generally non-trivial to extend k-means to cluster data points beyond Gaussian distribution, particularly, the clusters with non-convex shapes (Beliakov & King, 2006).

To this end, we, for the first time, introduce Extreme Value Theory (EVT) to improve the clustering ability of k-means.

Particularly, the Euclidean space was transformed into a novel probability space denoted as extreme value space by EVT.

We thus propose a novel algorithm called Extreme Value k-means (EV k-means), including GEV k-means and GPD k-means.

In addition, we also introduce the tricks to accelerate Euclidean distance computation in improving the computational efficiency of classical k-means.

Furthermore, our EV k-means is extended to an online version, i.e., online Extreme Value k-means, in utilizing the Mini Batch k-means to cluster streaming data.

Extensive experiments are conducted to validate our EV k-means and online EV k-means on synthetic datasets and real datasets.

Experimental results show that our algorithms significantly outperform competitors in most cases.

This paper introduces Extreme Value Theory into k-means to measure similarity and proposes a novel algorithm called Extreme Value k-means for clustering.

Extreme Value k-means Clustering


Deep reinforcement learning algorithms have proven successful in a variety of domains.

However, tasks with sparse rewards remain challenging when the state space is large.

Goal-oriented tasks are among the most typical problems in this domain, where a reward can only be received when the final goal is accomplished.

In this work, we propose a potential solution to such problems with the introduction of an experience-based tendency reward mechanism, which provides the agent with additional hints based on a discriminative learning on past experiences during an automated reverse curriculum.

This mechanism not only provides dense additional learning signals on what states lead to success, but also allows the agent to retain only this tendency reward instead of the whole histories of experience during multi-phase curriculum learning.

We extensively study the advantages of our method on the standard sparse reward domains like Maze and Super Mario Bros and show that our method performs more efficiently and robustly than prior approaches in tasks with long time horizons and large state space.

In addition, we demonstrate that using an optional keyframe scheme with very small quantity of key states, our approach can solve difficult robot manipulation challenges directly from perception and sparse rewards.

We propose Tendency RL to efficiently solve goal-oriented tasks with large state space using automated curriculum learning and discriminative shaping reward, which has the potential to tackle robot manipulation tasks with perception.

TRL: Discriminative Hints for Scalable Reverse Curriculum Learning


Human scene perception goes beyond recognizing a collection of objects and their pairwise relations.

We understand higher-level, abstract regularities within the scene such as symmetry and repetition.

Current vision recognition modules and scene representations fall short in this dimension.

In this paper, we present scene programs, representing a scene via a symbolic program for its objects, attributes, and their relations.

We also propose a model that infers such scene programs by exploiting a hierarchical, object-based scene representation.

Experiments demonstrate that our model works well on synthetic data and transfers to real images with such compositional structure.

The use of scene programs has enabled a number of applications, such as complex visual analogy-making and scene extrapolation.

We present scene programs, a structured scene representation that captures both low-level object appearance and high-level regularity in the scene.

Learning to Describe Scenes with Programs


Modern neural networks often require deep compositions of high-dimensional nonlinear functions (wide architecture) to achieve high test accuracy, and thus can have overwhelming number of parameters.

Repeated high cost in prediction at test-time makes neural networks ill-suited for devices with constrained memory or computational power.

We introduce an efficient mechanism, reshaped tensor decomposition, to compress neural networks by exploiting three types of invariant structures: periodicity, modulation and low rank.

Our reshaped tensor decomposition method exploits such invariance structures using a technique called tensorization (reshaping the layers into higher-order tensors) combined with higher order tensor decompositions on top of the tensorized layers.

Our compression method improves low rank approximation methods and can be incorporated to (is complementary to) most of the existing compression methods for neural networks to achieve better compression.

Experiments on LeNet-5 (MNIST), ResNet-32 (CI- FAR10) and ResNet-50 (ImageNet) demonstrate that our reshaped tensor decomposition outperforms (5% test accuracy improvement universally on CIFAR10) the state-of-the-art low-rank approximation techniques under same compression rate, besides achieving orders of magnitude faster convergence rates.

Compression of neural networks which improves the state-of-the-art low rank approximation techniques and is complementary to most of other compression techniques.

Exploiting Invariant Structures for Compression in Neural Networks


With the rise in employment of deep learning methods in safety-critical scenarios, interpretability is more essential than ever before.

Although many different directions regarding interpretability have been explored for visual modalities, time-series data has been neglected with only a handful of methods tested due to their poor intelligibility.

We approach the problem of interpretability in a novel way by proposing TSInsight where we attach an auto-encoder with a sparsity-inducing norm on its output to the classifier and fine-tune it based on the gradients from the classifier and a reconstruction penalty.

The auto-encoder learns to preserve features that are important for the prediction by the classifier and suppresses the ones that are irrelevant i.e. serves as a feature attribution method to boost interpretability.

In other words, we ask the network to only reconstruct parts which are useful for the classifier i.e. are correlated or causal for the prediction.

In contrast to most other attribution frameworks, TSInsight is capable of generating both instance-based and model-based explanations.

We evaluated TSInsight along with other commonly used attribution methods on a range of different time-series datasets to validate its efficacy.

Furthermore, we analyzed the set of properties that TSInsight achieves out of the box including adversarial robustness and output space contraction.

The obtained results advocate that TSInsight can be an effective tool for the interpretability of deep time-series models.

We present an attribution technique leveraging sparsity inducing norms to achieve interpretability.

TSInsight: A local-global attribution framework for interpretability in time-series data


Variance reduction methods which use a mixture of large and small batch gradients, such as SVRG (Johnson & Zhang, 2013) and SpiderBoost (Wang et al., 2018), require significantly more computational resources per update than SGD (Robbins & Monro, 1951).

We reduce the computational cost per update of variance reduction methods by introducing a sparse gradient operator blending the top-K operator (Stich et al., 2018; Aji & Heafield, 2017) and the randomized coordinate descent operator.

While the computational cost of computing the derivative of a model parameter is constant, we make the observation that the gains in variance reduction are proportional to the magnitude of the derivative.

In this paper, we show that a sparse gradient based on the magnitude of past gradients reduces the computational cost of model updates without a significant loss in variance reduction.

Theoretically, our algorithm is at least as good as the best available algorithm (e.g. SpiderBoost) under appropriate settings of parameters and can be much more efficient if our algorithm succeeds in capturing the sparsity of the gradients.

Empirically, our algorithm consistently outperforms SpiderBoost using various models to solve various image classification tasks.

We also provide empirical evidence to support the intuition behind our algorithm via a simple gradient entropy computation, which serves to quantify gradient sparsity at every iteration.

We use sparsity to improve the computational complexity of variance reduction methods.

Variance Reduction With Sparse Gradients


Despite promising progress on unimodal data imputation (e.g. image inpainting), models for multimodal data imputation are far from satisfactory.

In this work, we propose variational selective autoencoder (VSAE) for this task.

Learning only from partially-observed data, VSAE can model the joint distribution of observed/unobserved modalities and the imputation mask, resulting in a unified model for various down-stream tasks including data generation and imputation.

Evaluation on synthetic high-dimensional and challenging low-dimensional multimodal datasets shows significant improvement over state-of-the-art imputation models.

We propose a novel VAE-based framework learning from partially-observed data for imputation and generation.

Variational Selective Autoencoder


In many domains, especially enterprise text analysis, there is an abundance of data which can be used for the development of new AI-powered intelligent experiences to improve people's productivity.

However, there are strong-guarantees of privacy which prevent broad sampling and  labeling of personal text data to learn or evaluate models of interest.

Fortunately, in some cases like enterprise email, manual annotation is possible on certain public datasets.

The hope is that models trained on these public datasets would perform well on the target private datasets of interest.

In this paper, we study the challenges of transferring information from one email dataset to another, for predicting user intent.

In particular, we present approaches to characterizing the transfer gap in text corpora from both an intrinsic and extrinsic point-of-view, and evaluate several proposed methods in the literature for bridging this gap.

We conclude with raising issues for further discussion in this arena.

Insights on the domain adaptation challenge, when predicting user intent in enterprise email.

On Domain Transfer When Predicting Intent in Text


Hierarchical Reinforcement Learning is a promising approach to long-horizon decision-making problems with sparse rewards.

Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task.

Treating the skills as fixed can lead to significant sub-optimality in the transfer setting.

In this work, we propose a novel algorithm to discover a set of skills, and continuously adapt them along with the higher level even when training on a new task.

First, we derive a new hierarchical policy gradient, as well as an unbiased latent-dependent baseline.

We introduce Hierarchical Proximal Policy Optimization (HiPPO), an on-policy method to efficiently train all levels of the hierarchy simultaneously.

Second, we propose a method of training time-abstractions that improves the robustness of the obtained skills to environment changes.

Code and results are available at sites.google.com/view/hippo-rl.

We propose HiPPO, a stable Hierarchical Reinforcement Learning algorithm that can train several levels of the hierarchy simultaneously, giving good performance both in skill discovery and adaptation.

Sub-policy Adaptation for Hierarchical Reinforcement Learning

hierarchical reinforcement learning
adaptation


Learning can be framed as trying to encode the mutual information between input and output while discarding other information in the input.

Since the distribution between input and output is unknown, also the true mutual information is.

To quantify how difficult it is to learn a task, we calculate a observed mutual information score by dividing the estimated mutual information by the entropy of the input.

We substantiate this score analytically by showing that the estimated mutual information has an error that increases with the entropy of the data.

Intriguingly depending on how the data is represented the observed entropy and mutual information can vary wildly.

There needs to be a match between how data is represented and how a model encodes it.

Experimentally we analyze image-based input data representations and demonstrate that performance outcomes of extensive network architectures searches are well aligned to the calculated score.

Therefore to ensure better learning outcomes, representations may need to be tailored to both task and model to align with the implicit distribution of the model.

We take a step towards measuring learning task difficulty and demonstrate that in practice performance strongly depends on the match of the representation of the information and the model interpreting it.

Information lies in the eye of the beholder: The effect of representations on observed mutual information


Sepsis is a life-threatening complication from infection and a leading cause of mortality in hospitals.

While early detection of sepsis improves patient outcomes, there is little consensus on exact treatment guidelines, and treating septic patients remains an open  problem.

In this work we present a new deep reinforcement learning method that we use to learn optimal personalized treatment policies for septic patients.

We model patient continuous-valued physiological time series using multi-output Gaussian processes, a probabilistic model that easily handles missing values and irregularly spaced observation times while maintaining estimates of uncertainty.

The Gaussian process is directly tied to a deep recurrent Q-network that learns clinically interpretable treatment policies, and both models are learned together end-to-end.

We evaluate our approach on a heterogeneous dataset of septic spanning 15 months from our university health system, and find that our learned policy could reduce patient mortality by as much as 8.2\% from an overall baseline mortality rate of 13.3\%.

Our algorithm could be used to make treatment recommendations to physicians as part of a decision support tool, and the framework readily applies to other reinforcement learning problems that rely on sparsely sampled and frequently missing multivariate time series data.

We combine Multi-output Gaussian processes with deep recurrent Q-networks to learn optimal treatments for sepsis and show improved performance over standard deep reinforcement learning methods,

Learning to Treat Sepsis with Multi-Output Gaussian Process Deep Recurrent Q-Networks

multi-output gaussian process deep recurrent q-network


Unsupervised and semi-supervised learning are important problems that are especially challenging with complex data like natural images.

Progress on these problems would accelerate if we had access to appropriate generative models under which to pose the associated inference tasks.

Inspired by the success of Convolutional Neural Networks (CNNs) for supervised prediction in images, we design the Neural Rendering Model (NRM), a new hierarchical probabilistic generative model whose inference calculations correspond to those in a CNN.

The NRM introduces a small set of latent variables at each level of the model and enforces dependencies among all the latent variables via a conjugate prior distribution.

The conjugate prior yields a new regularizer for learning based on the paths rendered in the generative model for training CNNs–the Rendering Path Normalization (RPN).

We demonstrate that this regularizer improves generalization both in theory and in practice.

Likelihood estimation in the NRM yields the new Max-Min cross entropy training loss, which suggests a new deep network architecture–the Max- Min network–which exceeds or matches the state-of-art for semi-supervised and supervised learning on SVHN, CIFAR10, and CIFAR100.

We develop a new deep generative model for semi-supervised learning and propose a new Max-Min cross-entropy for training CNNs.

Neural Rendering Model: Joint Generation and Prediction for Semi-Supervised Learning


Deep reinforcement learning (RL) agents often fail to generalize to unseen environments (yet semantically similar to trained agents), particularly when they are trained on high-dimensional state spaces, such as images.

In this paper, we propose a simple technique to improve a generalization ability of deep RL agents by introducing a randomized (convolutional) neural network that randomly perturbs input observations.

It enables trained agents to adapt to new domains by learning robust features invariant across varied and randomized environments.

Furthermore, we consider an inference method based on the Monte Carlo approximation to reduce the variance induced by this randomization.

We demonstrate the superiority of our method across 2D CoinRun, 3D DeepMind Lab exploration and 3D robotics control tasks: it significantly outperforms various regularization and data augmentation methods for the same purpose.

We propose a simple randomization technique for improving generalization in deep reinforcement learning across tasks with various unseen visual patterns.

Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning

generalization deep reinforcement learning