Touch interactions with current mobile devices have limited expressiveness.

Augmenting devices with additional degrees of freedom can add power to the interaction, and several augmentations have been proposed and tested.

However, there is still little known about the effects of learning multiple sets of augmented interactions that are mapped to different applications.

To better understand whether multiple command mappings can interfere with one another, or affect transfer and retention, we developed a prototype with three pushbuttons on a smartphone case that can be used to provide augmented input to the system.

The buttons can be chorded to provide seven possible shortcuts or transient mode switches.

We mapped these buttons to three different sets of actions, and carried out a study to see if multiple mappings affect learning and performance, transfer, and retention.

Our results show that all of the mappings were quickly learned and there was no reduction in performance with multiple mappings.

Transfer to a more realistic task was successful, although with a slight reduction in accuracy.

Retention after one week was initially poor, but expert performance was quickly restored.

Our work provides new information about the design and use of augmented input in mobile interactions.

Describes a study investigating interference, transfer, and retention of multiple mappings with the same set of chorded buttons

Learning Multiple Mappings: an Evaluation of Interference, Transfer, and Retention with Chorded Shortcut Buttons

interference transfer retention
multiple mapping
chorded
button


Neural networks are widely used in Natural Language Processing, yet despite their empirical successes, their behaviour is brittle: they are both over-sensitive to small input changes, and under-sensitive to deletions of large fractions of input text.

This paper aims to tackle under-sensitivity in the context of natural language inference by ensuring that models do not become more confident in their predictions as arbitrary subsets of words from the input text are deleted.

We develop a novel technique for formal verification of this specification for models based on the popular decomposable attention mechanism by employing the efficient yet effective interval bound propagation (IBP) approach.

Using this method we can efficiently prove, given a model, whether a particular sample is free from the under-sensitivity problem.

We compare different training methods to address under-sensitivity, and compare metrics to measure it.

In our experiments on the SNLI and MNLI datasets, we observe that IBP training leads to a significantly improved verified accuracy.

On the SNLI test set, we can verify 18.4% of samples, a substantial improvement over only 2.8% using standard training.

Formal verification of a specification on a model's prediction undersensitivity using Interval Bound Propagation

Towards Verified Robustness under Text Deletion Interventions


Training with larger number of parameters while keeping fast iterations is an increasingly

adopted strategy and trend for developing better performing Deep Neural

This necessitates increased memory footprint and

computational requirements for training.

for training deep neural networks using 8-bit floating point (FP8) numbers.

Reduced bit precision allows for a larger effective memory and increased computational

We name this method Shifted and Squeezed FP8 (S2FP8).

show that, unlike previous 8-bit precision training methods, the proposed method

works out of the box for representative models: ResNet50, Transformer and NCF.

The method can maintain model accuracy without requiring fine-tuning loss scaling

parameters or keeping certain layers in single precision.

learnable statistics of the DNN tensors - shifted and squeezed factors that are used

to optimally adjust the range of the tensors in 8-bits, thus minimizing the loss in

We propose a novel 8-bit format that eliminates the need for loss scaling, stochastic rounding, and other low precision techniques

Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks


Variational Auto Encoders (VAE) are capable of generating realistic images, sounds and video sequences.

From practitioners point of view, we are usually interested in solving problems where tasks are learned sequentially, in a way that avoids revisiting all previous data at each stage.

We address this problem by introducing a conceptually simple and scalable end-to-end approach of incorporating past knowledge by learning prior directly from the data.

We consider scalable boosting-like approximation for intractable theoretical optimal prior.

We provide empirical studies on two commonly used benchmarks, namely  MNIST and Fashion MNIST on disjoint sequential image generation tasks.

For each dataset proposed method delivers the best results among comparable approaches,  avoiding catastrophic forgetting in a fully automatic way with a fixed model architecture.

Novel algorithm for Incremental learning of VAE with fixed architecture

BooVAE: A Scalable Framework for Continual VAE Learning under Boosting Approach


The field of few-shot learning has recently seen substantial advancements.

Most of these advancements came from casting few-shot learning as a meta-learning problem.Model Agnostic Meta Learning or MAML is currently one of the best approaches for few-shot learning via meta-learning.

MAML is simple, elegant and very powerful, however, it has a variety of issues, such as being very sensitive to neural network architectures, often leading to instability during training, requiring arduous hyperparameter searches to stabilize training and achieve high generalization and being very computationally expensive at both training and inference times.

In this paper, we propose various modifications to MAML that not only stabilize the system, but also substantially improve the generalization performance, convergence speed and computational overhead of MAML, which we call MAML++.

MAML is great, but it has many problems, we solve many of those problems and as a result we learn most hyper parameters end to end, speed-up training and inference and set a new SOTA in few-shot learning

How to train your MAML


Community detection in graphs is of central importance in graph mining, machine learning and network science.

Detecting overlapping communities is especially challenging, and remains an open problem

.  Motivated by the success of graph-based  deep  learning  in  other  graph-related  tasks,  we  study  the  applicability  of this framework for overlapping community detection

. We propose a probabilistic model for overlapping community detection based on the graph neural network architecture

.  Despite its simplicity, our model outperforms the existing approaches in the community recovery task by a large margin

.  Moreover, due to the inductive formulation, the proposed model is able to perform out-of-sample community detection for nodes that were not present at training time

Detecting overlapping communities in graphs using graph neural networks

Overlapping Community Detection with Graph Neural Networks


Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks.

Existing techniques rely on two stages: searching over the architecture space and validating the best architecture.

NAS algorithms are currently compared solely based on their results on the downstream task.

While intuitive, this fails to explicitly evaluate the effectiveness of their search strategies.

In this paper, we propose to evaluate the NAS search phase.

To this end, we compare the quality of the solutions obtained by NAS search policies with that of random architecture selection.

(i) On average, the state-of-the-art NAS algorithms perform similarly to the random policy;

(ii) the widely-used weight sharing strategy degrades the ranking of the NAS candidates to the point of not reflecting their true performance, thus reducing the effectiveness of the search process.

We believe that our evaluation framework will be key to designing NAS strategies that consistently discover architectures superior to random ones.

We empirically disprove a fundamental hypothesis of the widely-adopted weight sharing strategy in neural architecture search and explain why the state-of-the-arts NAS algorithms performs similarly to random search.

Evaluating The Search Phase of Neural Architecture Search


In this paper we use the geometric properties of the optimal transport (OT) problem and the Wasserstein distances to define a prior distribution for the latent space of an auto-encoder.

We introduce Sliced-Wasserstein Auto-Encoders (SWAE), that enable one to shape the distribution of the latent space into any samplable probability distribution without the need for training an adversarial network or having a likelihood function specified.

In short, we regularize the auto-encoder loss with the sliced-Wasserstein distance between the distribution of the encoded training samples and a samplable prior distribution.

We show that the proposed formulation has an efficient numerical solution that provides similar capabilities to Wasserstein Auto-Encoders (WAE) and Variational Auto-Encoders (VAE), while benefiting from an embarrassingly simple implementation.

We provide extensive error analysis for our algorithm, and show its merits on three benchmark datasets.

In this paper we use the sliced-Wasserstein distance to shape the latent distribution of an auto-encoder into any samplable prior distribution.

Sliced Wasserstein Auto-Encoders


The Hamiltonian formalism plays a central role in classical and quantum physics.

Hamiltonians are the main tool for modelling the continuous time evolution of systems with conserved quantities, and they come equipped with many useful properties, like time reversibility and smooth interpolation in time.

These properties are important for many machine learning problems - from sequence prediction to reinforcement learning and density modelling - but are not typically provided out of the box by standard tools such as recurrent neural networks.

In this paper, we introduce the Hamiltonian Generative Network (HGN), the first approach capable of consistently learning Hamiltonian dynamics from high-dimensional observations (such as images) without restrictive domain assumptions.

Once trained, we can use HGN to sample new trajectories, perform rollouts both forward and backward in time, and even speed up or slow down the learned dynamics.

We demonstrate how a simple modification of the network architecture turns HGN into a powerful normalising flow model, called Neural Hamiltonian Flow (NHF), that uses Hamiltonian dynamics to model expressive densities.

Hence, we hope that our work serves as a first practical demonstration of the value that the Hamiltonian formalism can bring to machine learning.

More results and video evaluations are available at: http://tiny.cc/hgn

We introduce a class of generative models that reliably learn Hamiltonian dynamics from high-dimensional observations. The learnt Hamiltonian can be applied to sequence modeling or as a normalising flow.

Hamiltonian Generative Networks


Cloud Migration transforms customer’s data, application and services from original IT platform to one or more cloud en- vironment, with the goal of improving the performance of the IT system while reducing the IT management cost.

The enterprise level Cloud Migration projects are generally com- plex, involves dynamically planning and replanning various types of transformations for up to 10k endpoints.

Currently the planning and replanning in Cloud Migration are generally done manually or semi-manually with heavy dependency on the migration expert’s domain knowledge, which takes days to even weeks for each round of planning or replanning.

As a result, automated planning engine that is capable of gener- ating high quality migration plan in a short time is particu- larly desirable for the migration industry.

In this short paper, we briefly introduce the advantages of using AI planning in Cloud Migration, a preliminary prototype, as well as the challenges the requires attention from the planning and scheduling society.

In this short paper, we briefly introduce the advantages of using AI planning in Cloud Migration, a preliminary prototype, as well as the chal- lenges the requires attention from the planning and schedul- ing society.

Advantages and Challenges of Using AI Planning in Cloud Migration


Unsupervised domain adaptation aims to generalize the hypothesis trained in a source domain to an unlabeled target domain.

One popular approach to this problem is to learn a domain-invariant representation for both domains.

In this work, we study, theoretically and empirically, the explicit effect of the embedding on generalization to the target domain.

In particular, the complexity of the class of embeddings affects an upper bound on the target domain's risk.

This is reflected in our experiments, too.

A general upper bound on the target domain's risk that reflects the role of embedding-complexity.

The Role of Embedding Complexity in Domain-invariant Representations


The domain of time-series forecasting has been extensively studied because it is of fundamental importance in many real-life applications.

Weather prediction, traffic flow forecasting or sales are compelling examples of sequential phenomena.

Predictive models generally make use of the relations between past and future values.

However, in the case of stationary time-series, observed values also drastically depend on a number of exogenous features that can be used to improve forecasting quality.

In this work, we propose a change of paradigm which consists in learning such features in embeddings vectors within recurrent neural networks.

We apply our framework to forecast smart cards tap-in logs in the Parisian subway network.

Results show that context-embedded models perform quantitatively better in one-step ahead and multi-step ahead forecasting.

In order to forecast multivariate stationary time-series we learn embeddings containing contextual features within a RNN; we apply the framework on public transportation data

Context-aware Forecasting for Multivariate Stationary Time-series


The ability of an agent to {\em discover} its own learning objectives has long been considered a key ingredient for artificial general intelligence.

Breakthroughs in autonomous decision making and reinforcement learning have primarily been in domains where the agent's goal is outlined and clear: such as playing a game to win, or driving safely.

Several studies have demonstrated that learning extramural sub-tasks and auxiliary predictions can improve (1) single human-specified task learning, (2) transfer of learning, (3) and the agent's learned representation of the world.

In all these examples, the agent was instructed what to learn about.

We investigate a framework for discovery: curating a large collection of predictions, which are used to construct the agent's representation of the world.

Specifically, our system maintains a large collection of predictions, continually pruning and replacing predictions.

We highlight the importance of considering stability rather than convergence for such a system, and develop an adaptive, regularized algorithm towards that aim.

We provide several experiments in computational micro-worlds demonstrating that this simple approach can be effective for discovering useful predictions autonomously.

We investigate a framework for discovery: curating a large collection of predictions, which are used to construct the agent’s representation in partially observable domains.

Discovery of Predictive Representations With a Network of General Value Functions


Estimating the location where an image was taken based solely on the contents of the image is a challenging task, even for humans, as properly labeling an image in such a fashion relies heavily on contextual information, and is not as simple as identifying  a single object in the image.

Thus any methods which attempt to do so must somehow account for these complexities, and no single model to date is completely capable of addressing all challenges.

This work contributes to the state of research in image geolocation inferencing by introducing a novel global meshing strategy, outlining a variety of training procedures to overcome the considerable data limitations when training these models, and demonstrating how incorporating additional information can be used to improve the overall performance of a geolocation inference model.

In this work, it is shown that Delaunay triangles are an effective type of mesh for geolocation in relatively low volume scenarios when compared to results from state of the art models which use quad trees and an order of magnitude more training data.

In addition, the time of posting, learned user albuming, and other meta data are easily incorporated to improve geolocation by up to 11% for country-level (750 km) locality accuracy to 3% for city-level (25 km) localities.

A global geolocation inferencing strategy with novel meshing strategy and demonstrating incorporating additional information can be used to improve the overall performance of a geolocation inference model.

MULTI-MODAL GEOLOCATION ESTIMATION USING DEEP NEURAL NETWORKS


Hierarchical Bayesian methods have the potential to unify many related tasks (e.g. k-shot classification, conditional, and unconditional generation) by framing each as inference within a single generative model.

We show that existing approaches for learning such models can fail on expressive generative networks such as PixelCNNs, by describing the global distribution with little reliance on latent variables.

To address this, we develop a modification of the Variational Autoencoder in which encoded observations are decoded to new elements from the same class; the result, which we call a Variational Homoencoder (VHE), may be understood as training a hierarchical latent variable model which better utilises latent variables in these cases.

Using this framework enables us to train a hierarchical PixelCNN for the Omniglot dataset, outperforming all existing models on test set likelihood.

With a single model we achieve both strong one-shot generation and near human-level classification, competitive with state-of-the-art discriminative classifiers.

The VHE objective extends naturally to richer dataset structures such as factorial or hierarchical categories, as we illustrate by training models to separate character content from simple variations in drawing style, and to generalise the style of an alphabet to new characters.

Technique for learning deep generative models with shared latent variables, applied to Omniglot with a PixelCNN decoder.

The Variational Homoencoder: Learning to Infer High-Capacity Generative Models from Few Examples


Common-sense or background knowledge is required to understand natural language, but in most neural natural language understanding (NLU) systems, the requisite background knowledge is indirectly acquired from static corpora.

We develop a new reading architecture for the dynamic integration of explicit background knowledge in NLU models.

A new task-agnostic reading module provides refined word representations to a task-specific NLU architecture by processing background knowledge in the form of free-text statements, together with the task-specific inputs.

Strong performance on the tasks of document question answering (DQA) and recognizing textual entailment (RTE) demonstrate the effectiveness and flexibility of our approach.

Analysis shows that our models learn to exploit knowledge selectively and in a semantically appropriate way.

In this paper we present a task-agnostic reading architecture for the dynamic integration of explicit background knowledge in neural NLU models.

Dynamic Integration of Background Knowledge in Neural NLU Systems

background knowledge neural nlu
dynamic integration


Mixed-precision arithmetic combining both single- and half-precision operands in the same operation have been successfully applied to train deep neural networks.

Despite the advantages of mixed-precision arithmetic in terms of reducing the need for key resources like memory bandwidth or register file size, it has a limited capacity for diminishing computing costs and requires 32 bits to represent its output operands.

This paper proposes two approaches to replace mixed-precision for half-precision arithmetic during a large portion of the training.

The first approach achieves accuracy ratios slightly slower than the state-of-the-art by using half-precision arithmetic during more than 99% of training.

The second approach reaches the same accuracy as the state-of-the-art by dynamically switching between half- and mixed-precision arithmetic during training.

It uses half-precision during more than 94% of the training process.

This paper is the first in demonstrating that half-precision can be used for a very large portion of DNNs training and still reach state-of-the-art accuracy.

Dynamic precision technique to train deep neural networks

A Dynamic Approach to Accelerate Deep Learning Training


We introduce the “inverse square root linear unit” (ISRLU) to speed up learning in deep neural networks.

ISRLU has better performance than ELU but has many of the same benefits.

ISRLU and ELU have similar curves and characteristics.

Both have negative values, allowing them to push mean unit activation closer to zero, and bring the normal gradient closer to the unit natural gradient, ensuring a noise- robust deactivation state, lessening the over fitting risk.

The significant performance advantage of ISRLU on traditional CPUs also carry over to more efficient HW implementations on HW/SW codesign for CNNs/RNNs.

In experiments with TensorFlow, ISRLU leads to faster learning and better generalization than ReLU on CNNs.

This work also suggests a computationally efficient variant called the “inverse square root unit” (ISRU) which can be used for RNNs.

Many RNNs use either long short-term memory (LSTM) and gated recurrent units (GRU) which are implemented with tanh and sigmoid activation functions.

ISRU has less computational complexity but still has a similar curve to tanh and sigmoid.

We introduce the ISRLU activation function which is continuously differentiable and faster than ELU. The related ISRU replaces tanh & sigmoid.

Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)


A generally intelligent learner should generalize to more complex tasks than it has previously encountered, but the two common paradigms in machine learning -- either training a separate learner per task or training a single learner for all tasks -- both have difficulty with such generalization because they do not leverage  the compositional structure of the task distribution.

This paper introduces the compositional problem graph as a broadly applicable formalism to relate tasks of different complexity in terms of problems with shared subproblems.

We propose the compositional generalization problem for measuring how readily old knowledge can be reused and hence built upon.

As a first step for tackling compositional generalization, we introduce the compositional recursive learner, a domain-general framework for learning algorithmic procedures for composing representation transformations, producing a learner that reasons about what computation to execute by making analogies to previously seen problems.

We show on a symbolic and a high-dimensional domain that our compositional approach can generalize to more complex problems than the learner has previously encountered, whereas baselines that are not explicitly compositional do not.

We explore the problem of compositional generalization and propose a means for endowing neural network architectures with the ability to compose themselves to solve these problems.

Automatically Composing Representation Transformations as a Means for Generalization


The information bottleneck method provides an information-theoretic method for representation learning, by training an encoder to retain all information which is relevant for predicting the label, while minimizing the amount of other, superfluous information in the representation.

The original formulation, however, requires labeled data in order to identify which information is superfluous.

In this work, we extend this ability to the multi-view unsupervised setting, in which two views of the same underlying entity are provided but the label is unknown.

This enables us to identify superfluous information as that which is not shared by both views.

A theoretical analysis leads to the definition of a new multi-view model that produces state-of-the-art results on the Sketchy dataset and on label-limited versions of the MIR-Flickr dataset.

We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to traditional unsupervised approaches for representation learning.

We extend the information bottleneck method to the unsupervised multiview setting and show state of the art results on standard datasets

Learning Robust Representations via Multi-View Information Bottleneck


The biological plausibility of the backpropagation algorithm has long been doubted by neuroscientists.

Two major reasons are that neurons would need to send two different types of signal in the forward and backward phases, and that pairs of neurons would need to communicate through symmetric bidirectional connections.

We present a simple two-phase learning procedure for fixed point recurrent networks that addresses both these issues.

In our model, neurons perform leaky integration and synaptic weights are updated through a local mechanism.

Our learning method extends the framework of Equilibrium Propagation to general dynamics, relaxing the requirement of an energy function.

As a consequence of this generalization, the algorithm does not compute the true gradient of the objective function,

but rather approximates it at a precision which is proven to be directly related to the degree of symmetry of the feedforward and feedback weights.

We show experimentally that the intrinsic properties of the system lead to alignment of the feedforward and feedback weights, and that our algorithm optimizes the objective function.

We describe a biologically plausible learning algorithm for fixed point recurrent networks without tied weights

Extending the Framework of Equilibrium Propagation to General Dynamics


In this paper we present, to the best of our knowledge, the first method to learn a generative model of 3D shapes from natural images in a fully unsupervised way.

For example, we do not use any ground truth 3D or 2D annotations, stereo video, and ego-motion during the training.

Our approach follows the general strategy of Generative Adversarial Networks, where an image generator network learns to create image samples that are realistic enough to fool a discriminator network into believing that they are natural images.

In contrast, in our approach the image gen- eration is split into 2 stages.

In the first stage a generator network outputs 3D ob- jects.

In the second, a differentiable renderer produces an image of the 3D object from a random viewpoint.

The key observation is that a realistic 3D object should yield a realistic rendering from any plausible viewpoint.

Thus, by randomizing the choice of the viewpoint our proposed training forces the generator network to learn an interpretable 3D representation disentangled from the viewpoint.

In this work, a 3D representation consists of a triangle mesh and a texture map that is used to color the triangle surface by using the UV-mapping technique.

We provide analysis of our learning approach, expose its ambiguities and show how to over- come them.

Experimentally, we demonstrate that our method can learn realistic 3D shapes of faces by using only the natural images of the FFHQ dataset.

We train a generative 3D model of shapes from natural images in an fully unsupervised way.

Unsupervised Generative 3D Shape Learning from Natural Images


We focus on the problem of black-box adversarial attacks, where the aim is to generate adversarial examples using information limited to loss function evaluations of input-output pairs.

We use Bayesian optimization (BO) to specifically

cater to scenarios involving low query budgets to develop query efficient adversarial attacks.

We alleviate the issues surrounding BO in regards to optimizing high dimensional deep learning models by effective dimension upsampling techniques.

Our proposed approach achieves performance comparable to the state of the art black-box adversarial attacks albeit with a much lower average query count.

In particular, in low query budget regimes, our proposed method reduces the query count up to 80% with respect to the state of the art methods.

We show that a relatively simple black-box adversarial attack scheme using Bayesian optimization and dimension upsampling  is preferable to existing methods when the number of available queries is very low.

Black-box Adversarial Attacks with Bayesian Optimization

bayesian optimization
black-box adversarial attack


State-of-the-art deep neural networks (DNNs) typically have tens of millions of parameters, which might not fit into the upper levels of the memory hierarchy, thus increasing the inference time and energy consumption significantly, and prohibiting their use on edge devices such as mobile phones.

The compression of DNN models has therefore become an active area of research recently, with \emph{connection pruning} emerging as one of the most successful strategies.

A very natural approach is to prune connections of DNNs via $\ell_1$ regularization, but recent empirical investigations have suggested that this does not work as well in the context of DNN compression.

In this work, we revisit this simple strategy and analyze it rigorously, to show that:

(a) any \emph{stationary point} of an $\ell_1$-regularized layerwise-pruning objective has its number of non-zero elements bounded by the number of penalized prediction logits, regardless of the strength of the regularization;

(b) successful pruning highly relies on an accurate optimization solver, and there is a trade-off between compression speed and distortion of prediction accuracy, controlled by the strength of regularization.

Our theoretical results thus suggest that $\ell_1$ pruning could be successful provided we use an accurate optimization solver.

We corroborate this in our experiments, where we show that simple $\ell_1$ regularization with an Adamax-L1(cumulative) solver gives pruning ratio competitive to the state-of-the-art.

We revisit the simple idea of pruning connections of DNNs through $\ell_1$ regularization achieving state-of-the-art results on multiple datasets with theoretic guarantees.

DEEP-TRIM: REVISITING L1 REGULARIZATION FOR CONNECTION PRUNING OF DEEP NETWORK

pruning revisit connection regularization


Autonomy and adaptation of machines requires that they be able to measure their own errors.

We consider the advantages and limitations of such an approach when a machine has to measure the error in a regression task.

How can a machine measure the error of regression sub-components when it does not have the ground truth for the correct predictions?

compressed sensing approach applied to the error signal of the regressors can recover their precision error without any ground truth.

It allows for some regressors to be strongly correlated as long as not too many are so related.

Its solutions, however, are not unique - a property of ground truth inference solutions.

as a condition can recover the correct solution in settings where error correction is possible.

We briefly discuss the similarity of the mathematics of ground truth inference for regressors to that for classifiers.

A non-parametric method to measure the error moments of regressors without ground truth can be used with biased regressors

Error Correcting Algorithms for Sparsely Correlated Regressors


We propose a new representation, one-pixel signature, that can be used to reveal the characteristics of the convolution neural networks (CNNs).

Here, each CNN classifier is associated with a signature that is created by generating, pixel-by-pixel, an adversarial value that is the result of the largest change to the class prediction.

The one-pixel signature is agnostic to the design choices of CNN architectures such as type, depth, activation function, and how they were trained.

It can be computed efficiently for a black-box classifier without accessing the network parameters.

Classic networks such as LetNet, VGG, AlexNet, and ResNet demonstrate different characteristics in their signature images.

For application, we focus on the classifier backdoor detection problem where a CNN classifier has been maliciously inserted with an unknown Trojan.

We show the effectiveness of the one-pixel signature in detecting backdoored CNN.

Our proposed one-pixel signature representation is general and it can be applied in problems where discriminative classifiers, particularly neural network based, are to be characterized.

Cnvolutional neural networks characterization for backdoored classifier detection and understanding.

Characterizing convolutional neural networks with one-pixel signature


We extend the learning from demonstration paradigm by providing a method for learning unknown constraints shared across tasks, using demonstrations of the tasks, their cost functions, and knowledge of the system dynamics and control constraints.

Given safe demonstrations, our method uses hit-and-run sampling to obtain lower cost, and thus unsafe, trajectories.

Both safe and unsafe trajectories are used to obtain a consistent representation of the unsafe set via solving a mixed integer program.

Additionally, by leveraging a known parameterization of the constraint, we modify our method to learn parametric constraints in high dimensions.

We show that our method can learn a six-dimensional pose constraint for a 7-DOF robot arm.

We can learn high-dimensional constraints from demonstrations by sampling unsafe trajectories and leveraging a known constraint parameterization.

Learning Parametric Constraints in High Dimensions from Demonstrations


We propose a metric-learning framework for computing distance-preserving maps that generate low-dimensional embeddings for a certain class of manifolds.

We employ Siamese networks to solve the problem of least squares multidimensional scaling for generating mappings that preserve geodesic distances on the manifold.

In contrast to previous parametric manifold learning methods we show a substantial reduction in training effort enabled by the computation of geodesic distances in a farthest point sampling strategy.

Additionally, the use of a network to model the distance-preserving map reduces the complexity of the multidimensional scaling problem and leads to an improved non-local generalization of the manifold compared to analogous non-parametric counterparts.

We demonstrate our claims on point-cloud data and on image manifolds and show a numerical analysis of our technique to facilitate a greater understanding of the representational power of neural networks in modeling manifold data.

Parametric Manifold Learning with Neural Networks in a Geometric Framework

Parametric Manifold Learning Via Sparse Multidimensional Scaling


Exploration is a fundamental aspect of Reinforcement Learning, typically implemented using stochastic action-selection.

Exploration, however, can be more efficient if directed toward gaining new world knowledge.

Visit-counters have been proven useful both in practice and in theory for directed exploration.

However, a major limitation of counters is their locality.

While there are a few model-based solutions to this shortcoming, a model-free approach is still missing.

We propose $E$-values, a generalization of counters that can be used to evaluate the propagating exploratory value over state-action trajectories.

We compare our approach to commonly used RL techniques, and show that using $E$-values improves learning and performance over traditional counters.

We also show how our method can be implemented with function approximation to efficiently learn continuous MDPs.

We demonstrate this by showing that our approach surpasses state of the art performance in the Freeway Atari 2600 game.

We propose a generalization of visit-counters that evaluate the propagating exploratory value over trajectories, enabling efficient exploration for model-free RL

DORA The Explorer: Directed Outreaching Reinforcement Action-Selection


Deep neuroevolution and deep reinforcement learning (deep RL) algorithms are two popular approaches to policy search.

The former is widely applicable and rather stable, but suffers from low sample efficiency.

By contrast, the latter is more sample efficient, but the most sample efficient variants are also rather unstable and highly sensitive to hyper-parameter setting.

So far, these families of methods have mostly been compared as competing tools.

However, an emerging approach consists in combining them so as to get the best of both worlds.

Two previously existing combinations use either an ad hoc evolutionary algorithm or a goal exploration process together with the Deep Deterministic Policy Gradient (DDPG) algorithm, a sample efficient off-policy deep RL algorithm.

In this paper, we propose a different combination scheme using the simple cross-entropy

method (CEM) and Twin Delayed Deep Deterministic policy gradient (TD3), another off-policy deep RL algorithm which improves over DDPG.

We evaluate the resulting method, CEM-RL, on a set of benchmarks classically used in deep RL.

We show that CEM-RL benefits from several advantages over its competitors and offers a satisfactory trade-off between performance and sample efficiency.

We propose a new combination of evolution strategy and deep reinforcement learning which takes the best of both worlds

CEM-RL: Combining evolutionary and gradient-based methods for policy search


Recent advances in deep reinforcement learning have made significant strides in performance on applications such as Go and Atari games.

However, developing practical methods to balance exploration and exploitation in complex domains remains largely unsolved.

Thompson Sampling and its extension to reinforcement learning provide an elegant approach to exploration that only requires access to posterior samples of the model.

At the same time, advances in approximate Bayesian methods have made posterior approximation for flexible neural network models practical.

Thus, it is attractive to consider approximate Bayesian neural networks in a Thompson Sampling framework.

To understand the impact of using an approximate posterior on Thompson Sampling, we benchmark well-established and recently developed methods for approximate posterior sampling combined with Thompson Sampling over a series of contextual bandit problems.

We found that many approaches that have been successful in the supervised learning setting underperformed in the sequential decision-making scenario.

In particular, we highlight the challenge of adapting slowly converging uncertainty estimates to the online setting.

Deep Bayesian Bandits Showdown:  An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling

empirical comparison bayesian deep network thompson sampling


Class labels have been empirically shown useful in improving the sample quality of generative adversarial nets (GANs).

In this paper, we mathematically study the properties of the current variants of GANs that make use of class label information.

With class aware gradient and cross-entropy decomposition, we reveal how class labels and associated losses influence GAN's training.

Based on that, we propose Activation Maximization Generative Adversarial Networks (AM-GAN) as an advanced solution.

Comprehensive experiments have been conducted to validate our analysis and evaluate the effectiveness of our solution, where AM-GAN outperforms other strong baselines and achieves state-of-the-art Inception Score (8.91) on CIFAR-10.

In addition, we demonstrate that, with the Inception ImageNet classifier, Inception Score mainly tracks the diversity of the generator, and there is, however, no reliable evidence that it can reflect the true sample quality.

We thus propose a new metric, called AM Score, to provide more accurate estimation on the sample quality.

Our proposed model also outperforms the baseline methods in the new metric.

Understand how class labels help GAN training. Propose a new evaluation metric for generative models.

Activation Maximization Generative Adversarial Nets


Modern neural networks are over-parametrized.

In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and out- put weights, without changing the rest of the network.

Inspired by the Sinkhorn-Knopp algorithm, we introduce a fast iterative method for minimizing the l2 norm of the weights, equivalently the weight decay regularizer.

It provably converges to a unique solution.

Interleaving our algorithm with SGD during training improves the test accuracy.

For small batches, our approach offers an alternative to batch- and group- normalization on CIFAR-10 and ImageNet with a ResNet-18.

Fast iterative algorithm to balance the energy of a network while staying in the same functional equivalence class

Equi-normalization of Neural Networks


Reinforcement learning agents are typically trained and evaluated according to their performance averaged over some distribution of environment settings.

But does the distribution over environment settings contain important biases, and do these lead to agents that fail in certain cases despite high average-case performance?

In this work, we consider worst-case analysis of agents over environment settings in order to detect whether there are directions in which agents may have failed to generalize.

Specifically, we consider a 3D first-person task where agents must navigate procedurally generated mazes, and where reinforcement learning agents have recently achieved human-level average-case performance.

By optimizing over the structure of mazes, we find that agents can suffer from catastrophic failures, failing to find the goal even on surprisingly simple mazes, despite their impressive average-case performance.

Additionally, we find that these failures transfer between different agents and even significantly different architectures.

We believe our findings highlight an important role for worst-case analysis in identifying whether there are directions in which agents have failed to generalize.

Our hope is that the ability to automatically identify failures of generalization will facilitate development of more general and robust agents.

To this end, we report initial results on enriching training with settings causing failure.

We find environment settings in which SOTA agents trained on navigation tasks display extreme failures suggesting failures in generalization.

Uncovering Surprising Behaviors in Reinforcement Learning via Worst-case Analysis


In some misspecified settings, the posterior distribution in Bayesian statistics may lead to inconsistent estimates.

To fix this issue, it has been suggested to replace the likelihood by a pseudo-likelihood, that is the exponential of a loss function enjoying suitable robustness properties.

In this paper, we build a pseudo-likelihood based on the Maximum Mean Discrepancy, defined via an embedding of probability distributions into a reproducing kernel Hilbert space.

We show that this MMD-Bayes posterior is consistent and robust to model misspecification.

As the posterior obtained in this way might be intractable, we also prove that reasonable variational approximations of this posterior enjoy the same properties.

We provide details on a stochastic gradient algorithm to compute these variational approximations.

Numerical simulations indeed suggest that our estimator is more robust to misspecification than the ones based on the likelihood.

MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy

robust bayesian estimation via maximum mean discrepancy


The standard variational lower bounds used to train latent variable models produce biased estimates of most quantities of interest.

We introduce an unbiased estimator of the log marginal likelihood and its gradients for latent variable models based on randomized truncation of infinite series.

If parameterized by an encoder-decoder architecture, the parameters of the encoder can be optimized to minimize its variance of this estimator.

We show that models trained using our estimator give better test-set likelihoods than a standard importance-sampling based approach for the same average computational cost.

This estimator also allows use of latent variable models for tasks where unbiased estimators, rather than marginal likelihood lower bounds, are preferred, such as minimizing reverse KL divergences and estimating score functions.

We create an unbiased estimator for the log probability of latent variable models, extending such models to a larger scope of applications.

SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models

probability latent variable model
log
unbiased


This paper makes two contributions towards understanding how the hyperparameters of stochastic gradient descent affect the final training loss and test accuracy of neural networks.

First, we argue that stochastic gradient descent exhibits two regimes with different behaviours; a noise dominated regime which typically arises for small or moderate batch sizes, and a curvature dominated regime which typically arises when the batch size is large.

In the noise dominated regime, the optimal learning rate increases as the batch size rises, and the training loss and test accuracy are independent of batch size under a constant epoch budget.

In the curvature dominated regime, the optimal learning rate is independent of batch size, and the training loss and test accuracy degrade as the batch size rises.

We support these claims with experiments on a range of architectures including ResNets, LSTMs and autoencoders.

We always perform a grid search over learning rates at all batch sizes.

Second, we demonstrate that small or moderately large batch sizes continue to outperform very large batches on the test set, even when both models are trained for the same number of steps and reach similar training losses.

Furthermore, when training Wide-ResNets on CIFAR-10 with a constant batch size of 64, the optimal learning rate to maximize the test accuracy only decays by a factor of 2 when the epoch budget is increased by a factor of 128, while the optimal learning rate to minimize the training loss decays by a factor of 16.

These results confirm that the noise in stochastic gradients can introduce beneficial implicit regularization.

Smaller batch sizes can outperform very large batches on the test set under constant step budgets and with properly tuned learning rate schedules.

Hyperparameter Tuning and Implicit Regularization in Minibatch SGD


Counterfactual Regret Minimization (CFR) is the most successful algorithm for finding approximate Nash equilibria in imperfect information games.

However, CFR's reliance on full game-tree traversals limits its scalability and generality.

Therefore, the game's state- and action-space is often abstracted (i.e. simplified) for CFR, and the resulting strategy is then mapped back to the full game.

This requires extensive expert-knowledge, is not practical in many games outside of poker, and often converges to highly exploitable policies.

A recently proposed method, Deep CFR, applies deep learning directly to CFR, allowing the agent to intrinsically abstract and generalize over the state-space from samples, without requiring expert knowledge.

In this paper, we introduce Single Deep CFR (SD-CFR), a variant of Deep CFR that has a lower overall approximation error by avoiding the training of an average strategy network.

We show that SD-CFR is more attractive from a theoretical perspective and empirically outperforms Deep CFR with respect to exploitability and one-on-one play in poker.

Better Deep Reinforcement Learning algorithm to approximate Counterfactual Regret Minimization

Single Deep Counterfactual Regret Minimization


While generative models have shown great success in generating high-dimensional samples conditional on low-dimensional descriptors (learning e.g. stroke thickness in MNIST, hair color in CelebA, or speaker identity in Wavenet), their generation out-of-sample poses fundamental problems.

The conditional variational autoencoder (CVAE) as a simple conditional generative model does not explicitly relate conditions during training and, hence, has no incentive of learning a compact joint distribution across conditions.

We overcome this limitation by matching their distributions using maximum mean discrepancy (MMD) in the decoder layer that follows the bottleneck.

This introduces a strong regularization both for reconstructing samples within the same condition and for transforming samples across conditions, resulting in much improved generalization.

We refer to the architecture as transformer VAE (trVAE).

Benchmarking trVAE on high-dimensional image and tabular data, we demonstrate higher robustness and higher accuracy than existing approaches.

In particular, we show qualitatively improved predictions for cellular perturbation response to treatment and disease based on high-dimensional single-cell gene expression data, by tackling previously problematic minority classes and multiple conditions.

For generic tasks, we improve Pearson correlations of high-dimensional estimated means and variances with their ground truths from 0.89 to 0.97 and 0.75 to 0.87, respectively.

Generates never seen data during training from a desired condition

Conditional Out-of-Sample Generation For Unpaired Data using trVAE


We analyze speed of convergence to global optimum for gradient descent training a deep linear neural network by minimizing the L2 loss over whitened data.

Convergence at a linear rate is guaranteed when the following hold: (i) dimensions of hidden layers are at least the minimum of the input and output dimensions; (ii) weight matrices at initialization are approximately balanced; and (iii) the initial loss is smaller than the loss of any rank-deficient solution.

The assumptions on initialization (conditions (ii) and (iii)) are necessary, in the sense that violating any one of them may lead to convergence failure.

Moreover, in the important case of output dimension 1, i.e. scalar regression, they are met, and thus convergence to global optimum holds, with constant probability under a random initialization scheme.

Our results significantly extend previous analyses, e.g., of deep linear residual networks (Bartlett et al., 2018).

We analyze gradient descent for deep linear neural networks, providing a guarantee of convergence to global optimum at a linear rate.

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

gradient descent deep linear neural network


One of the fundamental problems in supervised classification and in machine learning in general, is the modelling of non-parametric invariances that exist in data.

Most prior art has focused on enforcing priors in the form of invariances to parametric nuisance transformations that are expected to be present in data.

However, learning non-parametric invariances directly from data remains an important open problem.

In this paper, we introduce a new architectural layer for convolutional networks which is capable of learning general invariances from data itself.

This layer can learn invariance to non-parametric transformations and interestingly, motivates and incorporates permanent random connectomes there by being called Permanent Random Connectome Non-Parametric Transformation Networks (PRC-NPTN).

PRC-NPTN networks are initialized with random connections (not just weights) which are a small subset of the connections in a fully connected convolution layer.

Importantly, these connections in PRC-NPTNs once initialized remain permanent throughout training and testing.

Random connectomes makes these architectures loosely more biologically plausible than many other mainstream network architectures which require highly ordered structures.

We motivate randomly initialized connections as a simple method to learn invariance from data itself while invoking invariance towards multiple nuisance transformations simultaneously.

We find that these randomly initialized permanent connections have positive effects on generalization, outperform much larger ConvNet baselines and the recently proposed Non-Parametric Transformation Network (NPTN) on benchmarks that enforce learning invariances from the data itself.

A layer modelling local random connectomes in the cortex within deep networks capable of learning general non-parametric invariances from the data itself.

Learning Non-Parametric Invariances from Data with Permanent Random Connectomes

non-parametric invariance datum
random connectome
learn


We propose a fully-convolutional conditional generative model, the latent transformation neural network (LTNN), capable of view synthesis using a light-weight neural network suited for real-time applications.

generative models which incorporate conditioning information via concatenation, we introduce a dedicated network component, the conditional transformation unit (CTU), designed to learn the latent space transformations corresponding to specified target views.

In addition, a consistency loss term is defined to guide the network toward learning the desired latent space mappings, a task-divided decoder is constructed to refine the quality of generated views, and an adaptive discriminator is introduced to improve the adversarial training process.

The generality of the proposed methodology is demonstrated on a collection of three diverse tasks: multi-view reconstruction on real hand depth images, view synthesis of real and synthetic faces, and the rotation of rigid objects.

The proposed model is shown to exceed state-of-the-art results in each category while simultaneously achieving a reduction in the computational demand required for inference by 30% on average.

We introduce an effective, general framework for incorporating conditioning information into inference-based generative models.

Latent Transformations for Object  View Points Synthesis


We describe three approaches to enabling an extremely computationally limited embedded scheduler to consider a small number of alternative activities based on resource availability.

We consider the case where the scheduler is so computationally limited that it cannot backtrack search.

The first two approaches precompile resource checks (called guards) that only enable selection of a preferred alternative activity if sufficient resources are estimated to be available to schedule the remaining activities.

The final approach mimics backtracking by invoking the scheduler multiple times with the alternative activities.

We present an evaluation of these techniques on mission scenarios (called sol types) from NASA's next planetary rover where these techniques are being evaluated for inclusion in an onboard scheduler.

This paper describes three techniques to allow a non-backtracking,  computationally limited scheduler to consider a small number of alternative activities based on resource availability.

Enabling Limited Resource-Bounded Disjunction in Scheduling


Inspired by the modularity and the life-cycle of biological neurons,we introduce Continual Learning via Neural Pruning (CLNP), a new method aimed at lifelong learning in fixed capacity models based on the pruning of neurons of low activity.

In this method, an L1 regulator is used to promote the presence of neurons of zero or low activity whose connections to previously active neurons is permanently severed at the end of training.

Subsequent tasks are trained using these pruned neurons after reinitialization and cause zero deterioration to the performance of previous tasks.

We show empirically that this biologically inspired method leads to state of the art results beating or matching current methods of higher computational complexity.

We use simple and biologically motivated modifications of standard learning techniques to achieve state of the art performance on catastrophic forgetting benchmarks.

Continual Learning via Neural Pruning


There has been an increasing use of neural networks for music information retrieval tasks.

In this paper, we empirically investigate different ways of improving the performance of convolutional neural networks (CNNs) on spectral audio features.

More specifically, we explore three aspects of CNN design: depth of the network, the use of residual blocks along with the use of grouped convolution, and global aggregation over time.

The application context is singer classification and singing performance embedding and we believe the conclusions extend to other types of music analysis using convolutional neural networks.

The results show that global time aggregation helps to improve the performance of CNNs the most.

Another contribution of this paper is the release of a singing recording dataset that can be used for training and evaluation.

Using deep learning techniques on singing voice related tasks.

Learning Audio Features for Singer Identification and Embedding


The Softmax function is used in the final layer of nearly all existing sequence-to-sequence models for language generation.

However, it is usually the slowest layer to compute which limits the vocabulary size to a subset of most frequent types; and it has a large memory footprint.

We propose a general technique for replacing the softmax layer with a continuous embedding layer.

Our primary innovations are a novel probabilistic loss, and a training and inference procedure in which we generate a probability distribution over pre-trained word embeddings, instead of a multinomial distribution over the vocabulary obtained via softmax.

We evaluate this new class of sequence-to-sequence models with continuous outputs on the task of neural machine translation.

We show that our models obtain upto 2.5x speed-up in training time while performing on par with the state-of-the-art models in terms of translation quality.

These models are capable of handling very large vocabularies without compromising on translation quality.

They also produce more meaningful errors than in the softmax-based models, as these errors typically lie in a subspace of the vector space of the reference translations.

Language generation using seq2seq models which produce word embeddings instead of a softmax based distribution over the vocabulary at each step enabling much faster training while maintaining generation quality

Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs


We propose a framework to understand the unprecedented performance and robustness of deep neural networks using field theory.

Correlations between the weights within the same layer can be described by symmetries in that layer, and networks generalize better if such symmetries are broken to reduce the redundancies of the weights.

Using a two parameter field theory, we find that the network can break such symmetries itself towards the end of training in a process commonly known in physics as spontaneous symmetry breaking.

This corresponds to a network generalizing itself without any user input layers to break the symmetry, but by communication with adjacent layers.

In the layer decoupling limit applicable to residual networks (He et al., 2015), we show that the remnant symmetries that survive the non-linear layers are spontaneously broken based on empirical results.

The Lagrangian for the non-linear and weight layers together has striking similarities with the one in quantum field theory of a scalar.

Using results from quantum field theory we show that our framework is able to explain many experimentally observed phenomena, such as training on random labels with zero error (Zhang et al., 2017), the information bottleneck and the phase transition out of it (Shwartz-Ziv & Tishby, 2017), shattered gradients (Balduzzi et al., 2017), and many more.

Closed form results for deep learning in the layer decoupling limit applicable to Residual Networks

Spontaneous Symmetry Breaking in Deep Neural Networks


During the last years, a remarkable breakthrough has been made in AI domain

thanks to artificial deep neural networks that achieved a great success in many

machine learning tasks in computer vision, natural language processing, speech

recognition, malware detection and so on.

Many investigations have pointed out this

fact and different approaches have been proposed to generate attacks while adding

a limited perturbation to the original data.

Nonetheless, a countermeasure known as fea-

ture squeezing coupled with ensemble defense showed that most of these attacks

In this paper, we present a new method we call Centered

Initial Attack (CIA) whose advantage is twofold : first, it insures by construc-

without the clipping process that degrades the quality of attacks.

robust against recently introduced defenses such as feature squeezing, JPEG en-

coding and even against a voting ensemble of defenses.

not limited to images, we illustrate this using five of the current best classifiers

on ImageNet dataset among which two are adversarialy retrained on purpose to

With a fixed maximum perturbation of only 1.5% on

any pixel, around 80% of attacks (targeted) fool the voting ensemble defense and

nearly 100% when the perturbation is only 6%.

to defend against CIA attacks, the last section of the paper gives some guidelines

In this paper, a new method we call Centered Initial Attack (CIA) is provided. It insures by construction the maximum perturbation to be smaller than a threshold fixed beforehand, without the clipping process.

Clipping Free Attacks Against Neural Networks


Answering complex logical queries on large-scale incomplete knowledge graphs (KGs) is a fundamental yet challenging task.

Recently, a promising approach to this problem has been to embed KG entities as well as the query into a vector space such that entities that answer the query are embedded close to the query.

However, prior work models queries as single points in the vector space, which is problematic because a complex query represents a potentially large set of its answer entities, but it is unclear how such a set can be represented as a single point.

Furthermore, prior work can only handle queries that use conjunctions ($\wedge$) and existential quantifiers ($\exists$).

Handling queries with logical disjunctions ($\vee$) remains an open problem.

Here we propose query2box, an embedding-based framework for reasoning over arbitrary queries with $\wedge$, $\vee$, and $\exists$ operators in massive and incomplete KGs.

Our main insight is that queries can be embedded as boxes (i.e., hyper-rectangles), where a set of points inside the box corresponds to a set of answer entities of the query.

We show that conjunctions can be naturally represented as intersections of boxes and also prove a negative result that handling disjunctions would require embedding with dimension proportional to the number of KG entities.

However, we show that by transforming queries into a Disjunctive Normal Form, query2box is capable of handling arbitrary logical queries with $\wedge$, $\vee$, $\exists$ in a scalable manner.

We demonstrate the effectiveness of query2box on two large KGs and show that query2box achieves up to 25% relative improvement over the state of the art.

Answering a wide class of logical queries over knowledge graphs with box embeddings in vector space

Query2box: Reasoning over Knowledge Graphs in Vector Space Using Box Embeddings

knowledge graph
box embedding
vector space


Driven by the need for parallelizable hyperparameter optimization methods, this paper studies open loop search methods: sequences that are predetermined and can be generated before a single configuration is evaluated.

Examples include grid search, uniform random search, low discrepancy sequences, and other sampling distributions.

In particular, we propose the use of k-determinantal point processes in  hyperparameter optimization via random search.

Compared to conventional uniform random search where hyperparameter settings are sampled independently, a k-DPP promotes diversity.

We describe an approach that transforms hyperparameter search spaces for efficient use with a k-DPP.

In addition, we introduce a novel Metropolis-Hastings algorithm which can sample from k-DPPs defined over any space from which uniform samples can be drawn, including spaces with a mixture of discrete and continuous dimensions or tree structure.

Our experiments show significant benefits  in realistic scenarios with a limited budget for training supervised learners, whether in serial or parallel.

We address fully parallel hyperparameter optimization with Determinantal Point Processes.

Open Loop Hyperparameter Optimization and Determinantal Point Processes

hyperparameter optimization determinantal point process


Graph embedding techniques have been increasingly deployed in a multitude of different applications that involve learning on non-Euclidean data.

However, existing graph embedding models either fail to incorporate node attribute information during training or suffer from node attribute noise, which compromises the accuracy.

Moreover, very few of them scale to large graphs due to their high computational complexity and memory usage.

In this paper we propose GraphZoom, a multi-level framework for improving both accuracy and scalability of unsupervised graph embedding algorithms.

GraphZoom first performs graph fusion to generate a new graph that effectively encodes the topology of the original graph and the node attribute information.

This fused graph is then repeatedly coarsened into a much smaller graph by merging nodes with high spectral similarities.

GraphZoom allows any existing embedding methods to be applied to the coarsened graph, before it progressively refine the embeddings obtained at the coarsest level to increasingly finer graphs.

We have evaluated our approach on a number of popular graph datasets for both transductive and inductive tasks.

Our experiments show that GraphZoom increases the classification accuracy and significantly reduces the run time compared to state-of-the-art unsupervised embedding methods.

A multi-level spectral approach to improving the quality and scalability of unsupervised graph embedding.

GraphZoom: A Multi-level Spectral Approach for Accurate and Scalable Graph Embedding

multi-level spectral approach
graph embed


Deep generative models have been enjoying success in modeling continuous data.

However it remains challenging to capture the representations for discrete structures with formal grammars and semantics, e.g., computer programs and molecular structures.

How to generate both syntactically and semantically correct data still remains largely an open problem.

Inspired by the theory of compiler where syntax and semantics check is done via syntax-directed translation (SDT), we propose a novel syntax-directed variational autoencoder (SD-VAE) by introducing stochastic lazy attributes.

This approach converts the offline SDT check into on-the-fly generated guidance for constraining the decoder.

Comparing to the state-of-the-art methods, our approach enforces constraints on the output space so that the output will be not only syntactically valid, but also semantically reasonable.

We evaluate the proposed model with applications in programming language and molecules, including reconstruction and program/molecule optimization.

The results demonstrate the effectiveness in incorporating syntactic and semantic constraints in discrete generative models, which is significantly better than current state-of-the-art approaches.

A new generative model for discrete structured data. The proposed stochastic lazy attribute converts the offline semantic check into online guidance for stochastic decoding, which effectively addresses the constraints in syntax and semantics, and also achieves superior performance

Syntax-Directed Variational Autoencoder for Structured Data


Modeling informal inference in natural language is very challenging.

With the recent availability of large annotated data, it has become feasible to train complex models such as neural networks to perform natural language inference (NLI), which have achieved state-of-the-art performance.

Although there exist relatively large annotated data, can machines learn all knowledge needed to perform NLI from the data?

If not, how can NLI models benefit from external knowledge and how to build NLI models to leverage it?

In this paper, we aim to answer these questions by enriching the state-of-the-art neural natural language inference models with external knowledge.

We demonstrate that the proposed models with external knowledge further improve the state of the art on the Stanford Natural Language Inference (SNLI) dataset.

the proposed models with external knowledge further improve the state of the art on the SNLI dataset.

Natural Language Inference with External Knowledge


High intra-class diversity and inter-class similarity is a characteristic of remote sensing scene image data sets currently posing significant difficulty for deep learning algorithms on classification tasks.

To improve accuracy, post-classification

methods have been proposed for smoothing results of model predictions.

However, those approaches require an additional neural network to perform the smoothing operation, which adds overhead to the task.

We propose an approach that involves learning deep features directly over neighboring scene images without requiring use of a cleanup model.

Our approach utilizes a siamese network to improve the discriminative power of convolutional neural networks on a pair

It then exploits semantic coherence between this pair to enrich the feature vector of the image for which we want to predict a label.

Empirical results show that this approach provides a viable alternative to existing methods.

For example, our model improved prediction accuracy by 1 percentage point and dropped the mean squared error value by 0.02 over the baseline, on a disease density estimation task.

These performance gains are comparable with results from existing post-classification methods, moreover without implementation overheads.

Approach for improving prediction accuracy by learning deep features over neighboring scene images in satellite scene image analysis.

EXPLOITING SEMANTIC COHERENCE TO IMPROVE PREDICTION IN SATELLITE SCENE IMAGE ANALYSIS: APPLICATION TO DISEASE DENSITY ESTIMATION

satellite scene image analysis
improve prediction


Sparsely available data points cause a numerical error on finite differences which hinder to modeling the dynamics of physical systems.

The discretization error becomes even larger when the sparse data are irregularly distributed so that the data defined on an unstructured grid, making it hard to build deep learning models to handle physics-governing observations on the unstructured grid.

In this paper, we propose a novel architecture named Physics-aware Difference Graph Networks (PA-DGN) that exploits neighboring information to learn finite differences inspired by physics equations.

PA-DGN further leverages data-driven end-to-end learning to discover underlying dynamical relations between the spatial and temporal differences in given observations.

We demonstrate the superiority of PA-DGN in the approximation of directional derivatives and the prediction of graph signals on the synthetic data and the real-world climate observations from weather stations.

We propose physics-aware difference graph networks designed to effectively learn spatial differences to modeling sparsely-observed dynamics.

Physics-aware Difference Graph Networks for Sparsely-Observed Dynamics

physics-aware difference graph network
sparsely-observed dynamic


Solving long-horizon sequential decision making tasks in environments with sparse rewards is a longstanding problem in reinforcement learning (RL) research.

Hierarchical Reinforcement Learning (HRL) has held the promise to enhance the capabilities of RL agents via operation on different levels of temporal abstraction.

Despite the success of recent works in dealing with inherent nonstationarity and sample complexity, it remains difficult to generalize to unseen environments and to transfer different layers of the policy to other agents.

In this paper, we propose a novel HRL architecture, Hierarchical Decompositional Reinforcement Learning (HiDe), which allows decomposition of the hierarchical layers into independent subtasks, yet allows for joint training of all layers in end-to-end manner.

The main insight is to combine a control policy on a lower level with an image-based planning policy on a higher level.

We evaluate our method on various complex continuous control tasks for navigation, demonstrating that generalization across environments and transfer of higher level policies can be achieved.

Learning Functionally Decomposed Hierarchies for Continuous Navigation Tasks

learn functionally decompose hierarchy continuous navigation task


Sound correspondence patterns play a crucial role for linguistic reconstruction.

use them to prove language relationship, to reconstruct proto-forms, and for classical

phylogenetic reconstruction based on shared innovations.Cognate words which fail to

conform with expected patterns can further point to various kinds of exceptions in sound

change, such as analogy or assimilation of frequent words.

method for the inference of sound correspondence patterns across multiple languages based

The core idea is to represent all columns in aligned cognate sets as

nodes in a network with edges representing the degree of compatibility between the nodes.

The task of inferring all compatible correspondence sets can then be handled as the well-known minimum clique cover problem in graph theory, which essentially seeks to split the graph into the smallest number of cliques in which each node is represented by exactly one clique.

The resulting partitions represent all correspondence patterns which can be

By excluding those patterns which occur in only a few cognate

sets, the core of regularly recurring sound correspondences can be inferred.

idea, the paper presents a method for automatic correspondence pattern recognition, which

is implemented as part of a Python library which supplements the paper.

usefulness of the method, various tests are presented, and concrete examples of the output

In addition to the source code, the study is supplemented by

a short interactive tutorial that illustrates how to use the new method and how to inspect

The paper describes a new algorithm by which sound correspondence patterns for multiple languages can be inferred.

Automatic Inference of Sound Correspondence Patterns Across Multiple Languages

sound correspondence pattern
multiple language


We describe Kernel RNN Learning (KeRNL), a reduced-rank, temporal eligibility trace-based approximation to backpropagation through time (BPTT) for training recurrent neural networks (RNNs) that gives competitive performance to BPTT on long time-dependence tasks.

The approximation replaces a rank-4 gradient learning tensor, which describes how past hidden unit activations affect the current state, by a simple reduced-rank product of a sensitivity weight and a temporal eligibility trace.

In this structured approximation motivated by node perturbation, the sensitivity weights and eligibility kernel time scales are themselves learned by applying perturbations.

The rule represents another step toward biologically plausible or neurally inspired ML, with lower complexity in terms of relaxed architectural requirements (no symmetric return weights), a smaller memory demand (no unfolding and storage of states over time), and a shorter feedback time.

A biologically plausible learning rule for training recurrent neural networks

Kernel RNN Learning (KeRNL)


We present Deep Graph Infomax (DGI), a general approach for learning node representations within graph-structured data in an unsupervised manner.

DGI relies on maximizing mutual information between patch representations and corresponding high-level summaries of graphs---both derived using established graph convolutional network architectures.

The learnt patch representations summarize subgraphs centered around nodes of interest, and can thus be reused for downstream node-wise learning tasks.

In contrast to most prior approaches to unsupervised learning with GCNs, DGI does not rely on random walk objectives, and is readily applicable to both transductive and inductive learning setups.

We demonstrate competitive performance on a variety of node classification benchmarks, which at times even exceeds the performance of supervised learning.

A new method for unsupervised representation learning on graphs, relying on maximizing mutual information between local and global representations in a graph. State-of-the-art results, competitive with supervised learning.

Deep Graph Infomax


In many environments only a tiny subset of all states yield high reward.

In these cases, few of the interactions with the environment provide a relevant learning signal.

Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them.

To this end, we advocate for the use of a \textit{backtracking model} that predicts the preceding states that terminate at a given high-reward state.

We can train a model which, starting from a high value state (or one that is estimated to have high value), predicts and samples which (state, action)-tuples may have led to that high value state.

These traces of (state, action) pairs, which we refer to as Recall Traces, sampled from this backtracking model starting from a high value state, are informative as they terminate in good states, and hence we can use these traces to improve a policy.

We provide a variational interpretation for this idea and a practical algorithm in which the backtracking model samples from an approximate posterior distribution over trajectories which lead to large rewards.

Our method improves the sample efficiency of both on- and off-policy RL algorithms across several environments and tasks.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.

Recall Traces: Backtracking Models for Efficient Reinforcement Learning


Prepositions are among the most frequent words.

Good prepositional representation  is of great syntactic and semantic interest  in computational linguistics.

Existing methods on preposition representation either treat prepositions as content words (e.g., word2vec and GloVe) or depend heavily on external linguistic resources including syntactic parsing, training task and dataset-specific representations.

In this paper we use word-triple counts (one of the words is a preposition) to  capture the preposition's interaction with its head and children.

Prepositional  embeddings are derived via tensor decompositions on a large unlabeled corpus.

We reveal a new geometry involving Hadamard products and empirically demonstrate its utility in paraphrasing of phrasal verbs.

Furthermore, our prepositional  embeddings are used as simple features to two challenging downstream tasks: preposition selection and prepositional attachment disambiguation.

We achieve comparable to or better results than state of the art on  multiple standardized datasets.

This work is about tensor-based method for preposition representation training.

Tensor-Based Preposition Representation


A promising class of generative models maps points from a simple distribution to a complex distribution through an invertible neural network.

Likelihood-based training  of  these  models  requires  restricting  their  architectures  to  allow  cheap computation of Jacobian determinants.

Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation.

In this paper, we use Hutchinson’s trace estimator to give a scalable unbiased estimate of the log-density.

The result is a continuous-time invertible generative model with unbiased density estimation and one-pass sampling, while allowing unrestricted neural network architectures.

We demonstrate our approach on high-dimensional density  estimation,  image  generation,  and  variational  inference,  achieving  the state-of-the-art among exact likelihood methods with efficient sampling.

We use continuous time dynamics to define a generative model with exact likelihoods and efficient sampling that is parameterized by unrestricted neural networks.

FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models


We propose a non-adversarial feature matching-based approach to train generative models.

Our approach, Generative Feature Matching Networks (GFMN), leverages pretrained neural networks such as autoencoders and ConvNet classifiers to perform feature extraction.

We perform an extensive number of experiments with different challenging datasets, including ImageNet.

Our experimental results demonstrate that, due to the expressiveness of the features from pretrained ImageNet classifiers, even by just matching first order statistics, our approach can achieve state-of-the-art results for challenging benchmarks such as CIFAR10 and STL10.

A new non-adversarial feature matching-based approach to train generative models that achieves state-of-the-art results.

Generative Feature Matching Networks


We propose a novel architecture for k-shot classification on the Omniglot dataset.

Building on prototypical networks, we extend their architecture to what we call Gaussian prototypical networks.

Prototypical networks learn a map between images and embedding vectors, and use their clustering for classification.

In our model, a part of the encoder output is interpreted as a confidence region estimate about the embedding point, and expressed as a Gaussian covariance matrix.

Our network then constructs a direction and class dependent distance metric on the embedding space, using uncertainties of individual data points as weights.

We show that Gaussian prototypical networks are a preferred architecture over vanilla prototypical networks with an equivalent number of parameters.

We report results consistent with state-of-the-art performance in 1-shot and 5-shot classification both in 5-way and 20-way regime on the Omniglot dataset.

We explore artificially down-sampling a fraction of images in the training set, which improves our performance.

Our experiments therefore lead us to hypothesize that Gaussian prototypical networks might perform better in less homogeneous, noisier datasets, which are commonplace in real world applications.

A novel architecture for few-shot classification capable of dealing with uncertainty.

Gaussian Prototypical Networks for Few-Shot Learning on Omniglot


We show that the output of a (residual) CNN with an appropriate prior over the weights and biases is a GP in the limit of infinitely many convolutional filters, extending similar results for dense networks.

For a CNN, the equivalent kernel can be computed exactly and, unlike "deep kernels", has very few parameters: only the hyperparameters of the original CNN.

Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer.

The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GP with a comparable number of parameters.

We show that CNNs and ResNets with appropriate priors on the parameters are Gaussian processes in the limit of infinitely many convolutional filters.

Deep Convolutional Networks as shallow Gaussian Processes


We present an end-to-end design methodology for efficient deep learning deployment.

Unlike previous methods that separately optimize the neural network architecture, pruning policy, and quantization policy, we jointly optimize them in an end-to-end manner.

To deal with the larger design space it brings, we train a quantization-aware accuracy predictor that fed to the evolutionary search to select the best fit.

We first generate a large dataset of <NN architecture, ImageNet accuracy> pairs without training each architecture, but by sampling a unified supernet.

Then we use these data to train an accuracy predictor without quantization, further using predictor-transfer technique to get the quantization-aware predictor, which reduces the amount of post-quantization fine-tuning time.

Extensive experiments on ImageNet show the benefits of the end-to-end methodology: it maintains the same accuracy (75.1%) as ResNet34 float model while saving 2.2× BitOps comparing with the 8-bit model; we obtain the same level accuracy as MobileNetV2+HAQ while achieving 2×/1.3× latency/energy saving; the end-to-end optimization outperforms separate optimizations using ProxylessNAS+AMC+HAQ by 2.3% accuracy while reducing orders of magnitude GPU hours and CO2 emission.

EMS: End-to-End Model Search for Network Architecture, Pruning and Quantization


Distributed optimization is vital in solving large-scale machine learning problems.

A widely-shared feature of distributed optimization techniques is the requirement that all nodes complete their assigned tasks in each computational epoch before the system can proceed to the next epoch.

In such settings, slow nodes, called stragglers, can greatly slow progress.

To mitigate the impact of stragglers, we propose an online distributed optimization method called Anytime Minibatch.

In this approach, all nodes are given a fixed time to compute the gradients of as many data samples as possible.

The result is a variable per-node minibatch size.

Workers then get a fixed communication time to average their minibatch gradients via several rounds of consensus, which are then used to update primal variables via dual averaging.

Anytime Minibatch prevents stragglers from holding up the system without wasting the work that stragglers can complete.

We present a convergence analysis and analyze the wall time performance.

Our numerical results show that our approach is up to 1.5 times faster in Amazon EC2 and it is up to five times faster when there is greater variability in compute node performance.

Accelerate distributed optimization by exploiting stragglers.

ANYTIME MINIBATCH: EXPLOITING STRAGGLERS IN ONLINE DISTRIBUTED OPTIMIZATION

exploit straggler
distribute optimization


Many machine learning image classifiers are vulnerable to adversarial attacks, inputs with perturbations designed to intentionally trigger misclassification.

Current adversarial methods directly alter pixel colors and evaluate against pixel norm-balls: pixel perturbations smaller than a specified magnitude, according to a measurement norm.

This evaluation, however, has limited practical utility since perturbations in the pixel space do not correspond to underlying real-world phenomena of image formation that lead to them and has no security motivation attached.

Pixels in natural images are measurements of light that has interacted with the geometry of a physical scene.

As such, we propose a novel evaluation measure, parametric norm-balls, by directly perturbing physical parameters that underly image formation.

One enabling contribution we present is a physically-based differentiable renderer that allows us to propagate pixel gradients to the parametric space of lighting and geometry.

Our approach enables physically-based adversarial attacks, and our differentiable renderer leverages models from the interactive rendering literature to balance the performance and accuracy trade-offs necessary for a memory-efficient and scalable adversarial data augmentation workflow.

Enabled by a novel differentiable renderer, we propose a new metric that has real-world implications for evaluating adversarial machine learning algorithms, resolving the lack of realism of the existing metric based on pixel norms.

Beyond Pixel Norm-Balls: Parametric Adversaries using an Analytically Differentiable Renderer


Alternatives to recurrent neural networks, in particular, architectures based on attention or convolutions, have been gaining momentum for processing input sequences.

In spite of their relevance, the computational properties of these alternatives have not yet been fully explored.

We study the computational power of two of the most paradigmatic architectures exemplifying these mechanisms: the Transformer (Vaswani et al., 2017) and the Neural GPU (Kaiser & Sutskever, 2016).

We show both models to be Turing complete exclusively based on their capacity to compute and access internal dense representations of the data.

In particular, neither the Transformer nor the Neural GPU requires access to an external memory to become Turing complete.

Our study also reveals some minimal sets of elements needed to obtain these completeness results.

We show that the Transformer architecture and the Neural GPU are Turing complete.

On the Turing Completeness of Modern Neural Network Architectures


It is usually hard for a learning system to predict correctly on the rare events, and there is no exception for segmentation algorithms.

Therefore, we hope to build an alarm system to set off alarms when the segmentation result is possibly unsatisfactory.

One plausible solution is to project the segmentation results into a low dimensional feature space, and then learn classifiers/regressors in the feature space to predict the qualities of segmentation results.

In this paper, we form the feature space using shape feature which is a strong prior information shared among different data, so it is capable to predict the qualities of segmentation results given different segmentation algorithms on different datasets.

The shape feature of a segmentation result is captured using the value of loss function when the segmentation result is tested using a Variational Auto-Encoder(VAE).

The VAE is trained using only the ground truth masks, therefore the bad segmentation results with bad shapes become the rare events for VAE and will result in large loss value.

By utilizing this fact, the VAE is able to detect all kinds of shapes that are out of the distribution of normal shapes in ground truth (GT).

Finally, we learn the representation in the one-dimensional feature space to predict the qualities of segmentation results.

We evaluate our alarm system on several recent segmentation algorithms for the medical segmentation task.

The segmentation algorithms perform differently on different datasets, but our system consistently provides reliable prediction on the qualities of segmentation results.

We use VAE to capture the shape feature for automatic segmentation evaluation

An Alarm System for Segmentation Algorithm Based on Shape Model


The linear transformations in converged deep networks show fast eigenvalue decay.

The distribution of eigenvalues looks like a Heavy-tail distribution, where the vast majority of eigenvalues is small, but not actually zero, and only a few spikes of large eigenvalues exist.

We use a stochastic approximator to generate histograms of eigenvalues.

This allows us to investigate layers with hundreds of thousands of dimensions.

We show how the distributions change over the course of image net training, converging to a similar heavy-tail spectrum across all intermediate layers.

We investigate the eigenvalues of the linear layers in deep networks and show that the distributions develop heavy-tail behavior during training.

Evolution of Eigenvalue Decay in Deep Networks


Capturing long-range feature relations has been a central issue on convolutional neural networks(CNNs).

To tackle this, attempts to integrate end-to-end trainable attention module on CNNs are widespread.

Main goal of these works is to adjust feature maps considering spatial-channel correlation inside a convolution layer.

In this paper, we focus on modeling relationships among layers and propose a novel structure, 'Recurrent Layer Attention network,' which stores the hierarchy of features into recurrent neural networks(RNNs) that concurrently propagating with CNN and adaptively scales feature volumes of all layers.

We further introduce several structural derivatives for demonstrating the compatibility on recent attention modules and the expandability of proposed network.

For semantic understanding on learned features, we also visualize intermediate layers and plot the curve of layer scaling coefficients(i.e., layer attention).

Recurrent Layer Attention network achieves significant performance enhancement requiring a slight increase on parameters in an image classification task with CIFAR and ImageNet-1K 2012 dataset and an object detection task with Microsoft COCO 2014 dataset.

We propose a new type of end-to-end trainable attention module, which applies global weight balances among layers by utilizing co-propagating RNN with CNN.

Recurrent Layer Attention Network


We seek to auto-generate stronger  input features  for ML methods faced with limited training data.

Biological neural nets (BNNs) excel at fast learning, implying that they extract highly informative features.

In particular, the insect olfactory network  learns new odors very rapidly, by means of three key elements: A competitive inhibition layer; randomized, sparse connectivity into a high-dimensional sparse plastic layer; and Hebbian updates of synaptic weights.

In this work we deploy MothNet, a computational model of the moth olfactory network, as an automatic feature generator.

Attached as a front-end pre-processor, MothNet's readout neurons provide new features, derived from the original features, for use by standard ML classifiers.

These ``insect cyborgs'' (part BNN and part ML method) have significantly better performance than baseline ML methods alone on vectorized MNIST and Omniglot data sets, reducing test set error averages 20% to 55%.

The MothNet feature generator also substantially out-performs other feature generating methods including PCA, PLS, and NNs.

These results highlight the potential value of BNN-inspired feature generators in the ML context.

Features auto-generated by the bio-mimetic MothNet model significantly improve the test accuracy of standard ML methods on vectorized MNIST. The MothNet-generated features also outperform standard feature generators.

Insect Cyborgs: Bio-mimetic Feature Generators Improve ML Accuracy  on Limited Data

bio-mimetic
feature generator
accuracy
ml
improve


We present an approach for anytime predictions in deep neural networks (DNNs).

For each test sample, an anytime predictor produces a coarse result quickly, and then continues to refine it until the test-time computational budget is depleted.

Such predictors can address the growing computational problem of DNNs by automatically adjusting to varying test-time budgets.

In this work, we study a \emph{general} augmentation to feed-forward networks to form anytime neural networks (ANNs) via auxiliary predictions and losses.

Specifically, we point out a blind-spot in recent studies in such ANNs: the importance of high final accuracy.

In fact, we show on multiple recognition data-sets and architectures that by having near-optimal final predictions in small anytime models, we can effectively double the speed of large ones to reach corresponding accuracy level.

We achieve such speed-up with simple weighting of anytime losses that oscillate during training.

We also assemble a sequence of exponentially deepening ANNs, to achieve both theoretically and practically near-optimal anytime results at any budget, at the cost of a constant fraction of additional consumed budget.

By focusing more on the final predictions in anytime predictors (such as the very recent Multi-Scale-DenseNets), we make small anytime models to outperform large ones that don't have such focus.

Anytime Neural Network: a Versatile Trade-off Between Computation and Accuracy


Computational imaging systems jointly design computation and hardware to retrieve information which is not traditionally accessible with standard imaging systems.

Recently, critical aspects such as experimental design and image priors are optimized through deep neural networks formed by the unrolled iterations of classical physics-based reconstructions (termed physics-based networks).

However, for real-world large-scale systems, computing gradients via backpropagation restricts learning due to memory limitations of graphical processing units.

In this work, we propose a memory-efficient learning procedure that exploits the reversibility of the network’s layers to enable data-driven design for large-scale computational imaging.

We demonstrate our methods practicality on two large-scale systems: super-resolution optical microscopy and multi-channel magnetic resonance imaging.

We propose a memory-efficient learning procedure that exploits the reversibility of the network’s layers to enable data-driven design for large-scale computational imaging.

Memory-efficient Learning for Large-scale Computational Imaging

large-scale computational imaging
memory-efficient learning