We present a simple and general method to train a single neural network executable at different widths (number of channels in a layer), permitting instant and adaptive accuracy-efficiency trade-offs at runtime.

Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization.

At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models.

Our trained networks, named slimmable neural networks, achieve similar (and in many cases better) ImageNet classification accuracy than individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths respectively.

We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters.

Lastly we visualize and discuss the learned features of slimmable networks.

Code and models are available at: https://github.com/JiahuiYu/slimmable_networks

The paper proposes an idea of combining different size models together into one shared net, greatly improving performance for detection

This paper trains a single network executable at different widths.

Slimmable Neural Networks

network executable different width
train single
neural


Measuring visual (dis)similarity between two or more instances within a data distribution is a fundamental task in many applications, specially in image retrieval.

Theoretically, non-metric distances are able to generate a more complex and accurate similarity model than metric distances, provided that the non-linear data distribution is precisely captured by the similarity model.

In this work, we analyze a simple approach for deep learning networks to be used as an approximation of non-metric similarity functions and we study how these models generalize across different image retrieval datasets.

Similarity network to learn a non-metric visual similarity estimation between a pair of images

The authors propose learning similarity measure for visual similarity and obtain by this an improvement in very well-known datasets of Oxford and Paris for image retrival.

The paper argues that it is more suitable to use non-metric distances instead of metric distances.

Learning Non-Metric Visual Similarity for Image Retrieval

learn non-metric visual similarity
image


Training a model to perform a task typically requires a large amount of data from the domains in which the task will be applied.

However, it is often the case that data are abundant in some domains but scarce in others.

Domain adaptation deals with the challenge of adapting a model trained from a data-rich source domain to perform well in a data-poor target domain.

In general, this requires learning plausible mappings between domains.

CycleGAN is a powerful framework that efficiently learns to map inputs from one domain to another using adversarial training and a cycle-consistency constraint.

However, the conventional approach of enforcing cycle-consistency via reconstruction may be overly restrictive in cases where one or more domains have limited training data.

In this paper, we propose an augmented cyclic adversarial learning model that enforces the cycle-consistency constraint via an external task specific model, which encourages the preservation of task-relevant content as opposed to exact reconstruction.

We explore digit classification in a low-resource setting in supervised, semi and unsupervised situation, as well as high resource unsupervised.

In low-resource supervised setting, the results show that our approach improves absolute performance by 14% and 4% when adapting SVHN to MNIST and vice versa, respectively, which outperforms unsupervised domain adaptation methods that require high-resource unlabeled target domain.

Moreover, using only few unsupervised target data, our approach can still outperforms many high-resource unsupervised models.

Our model also outperforms on USPS to MNIST and synthetic digit to SVHN for high resource unsupervised adaptation.

In speech domains, we similarly adopt a speech recognition model from each domain as the task specific model.

Our approach improves absolute performance of speech recognition by 2% for female speakers in the TIMIT dataset, where the majority of training samples are from male voices.

A new cyclic adversarial learning augmented with auxiliary task model which improves domain adaptation performance in low resource supervised and unsupervised situations

Proposes an extension of cycle-consistent adversatial adaptation methods in order to tackle domain adaptation where limited supervised target data is available.

This paper introduces a domain adaptation approach based on the idea of Cyclic GAN and proposes two different algorithms.

Augmented Cyclic Adversarial Learning for Low Resource Domain Adaptation

low resource
cyclic adversarial learning
domain adaptation


Nodes residing in different parts of a graph can have similar structural roles within their local network topology.

The identification of such roles provides key insight into the organization of networks and can also be used to inform machine learning on graphs.

However, learning structural representations of nodes is a challenging unsupervised-learning task, which typically involves manually specifying and tailoring topological features for each node.

Here we develop GraphWave, a method that represents each node’s local network neighborhood via a low-dimensional embedding by leveraging spectral graph wavelet diffusion patterns.

We prove that nodes with similar local network neighborhoods will have similar GraphWave embeddings even though these nodes may reside in very different parts of the network.

Our method scales linearly with the number of edges and does not require any hand-tailoring of topological features.

We evaluate performance on both synthetic and real-world datasets, obtaining improvements of up to 71% over state-of-the-art baselines.

We develop a method for learning structural signatures in networks based on the diffusion of spectral graph wavelets.

Using spectral graph wavelet diffusion patterns of a node's local meighbothood to embed the node in a low-dimensional space

The paper derived a way to compare nodes in graph based on wavelet analysis of graph laplacian.

Spectral Graph Wavelets for Structural Role Similarity in Networks

spectral graph wavelet
base
network
node
diffusion
structural


Driving simulators play an important role in vehicle research.

However, existing virtual reality simulators do not give users a true sense of presence.

UniNet is our driving simulator, designed to allow users to interact with and visualize simulated traffic in mixed reality.

UniNet's modular architecture allows us to investigate interdisciplinary research topics such as vehicular ad-hoc networks, human-computer interaction, and traffic management.

We accomplish this by giving users the ability to observe and interact with simulated traffic in a high fidelity driving simulator.

We present a user study that subjectively measures user's sense of presence in UniNet.

Our findings suggest that our novel mixed reality system does increase this sensation.

A mixed reality driving simulator using stereo cameras and passthrough VR evaluated in a user study with 24 participants.

Proposes a complicated system for driving simulation.

This paper presents a mixed reality driving simulator setup to enhance the sensation of presence

Proposes a mixed reality driving simulator that incorporates traffic generation and claims an enhanced "presence" due to an MR system.

UniNet: A Mixed Reality Driving Simulator

mixed reality drive simulator
system
presence


We consider the problem of improving kernel approximation via feature maps.

These maps arise as Monte Carlo approximation to integral representations of kernel functions and scale up kernel methods for larger datasets.

We propose to use more efficient numerical integration technique to obtain better estimates of the integrals compared to the state-of-the-art methods.

Our approach allows to use information about the integrand to enhance approximation and facilitates fast computations.

We derive the convergence behavior and conduct an extensive empirical study that supports our hypothesis.

Quadrature rules for kernel approximation.

The paper proposes improving the kernel approximation of random features by using quadrature rules like stochastic spherical-radial rules.

The authors propose a novel version of the random feature map approach to approximately solve large-scale kernel problems.

This paper shows that techniques due to Genz & Monahan (1998) can be used to achieve low kernel approximation error under the framework of random fourier feature, a new way to apply quadrature rules to improve kernel approximation.

Quadrature-based features for kernel approximation

improve kernel approximation
random feature
quadrature rule


Human world knowledge is both structured and flexible.

When people see an object, they represent it not as a pixel array but as a meaningful arrangement of semantic parts.

Moreover, when people refer to an object, they provide descriptions that are not merely true but also relevant in the current context.

Here, we combine these two observations in order to learn fine-grained correspondences between language and contextually relevant geometric properties of 3D objects.

To do this, we employed an interactive communication task with human participants to construct a large dataset containing natural utterances referring to 3D objects from ShapeNet in a wide variety of contexts.

Using this dataset, we developed neural listener and speaker models with strong capacity for generalization.

By performing targeted lesions of visual and linguistic input, we discovered that the neural listener depends heavily on part-related words and associates these words correctly with the corresponding geometric properties of objects, suggesting that it has learned task-relevant structure linking the two input modalities.

We further show that a neural speaker that is `listener-aware' --- that plans its utterances according to how an imagined listener would interpret its words in context --- produces more discriminative referring expressions than an `listener-unaware' speaker, as measured by human performance in identifying the correct object.

How to build neural-speakers/listeners that learn fine-grained characteristics of 3D objects, from referential language.

The authors provide a study on learning to refer to 3D objects, collecting a dataset of referential expressions and training several models by experimenting with a number of architectural choices

Learning to Refer to 3D Objects with Natural Language

learn refer 3d object
referential
language


Object-based factorizations provide a useful level of abstraction for interacting with the world.

Building explicit object representations, however, often requires supervisory signals that are difficult to obtain in practice.

We present a paradigm for learning object-centric representations for physical scene understanding without direct supervision of object properties.

Our model, Object-Oriented Prediction and Planning (O2P2), jointly learns a perception function to map from image observations to object representations, a pairwise physics interaction function to predict the time evolution of a collection of objects, and a rendering function to map objects back to pixels.

For evaluation, we consider not only the accuracy of the physical predictions of the model, but also its utility for downstream tasks that require an actionable representation of intuitive physics.

After training our model on an image prediction task, we can use its learned representations to build block towers more complicated than those observed during training.

We present a framework for learning object-centric representations suitable for planning in tasks that require an understanding of physics.

The paper presents a platform for predicting images of objects interacting with each other under the effect of gravitational forces.

The paper presents a method that learns to reproduce 'block towers' from a given image.

Proposes a method which learns to reason on physical interaction of different objects with no supervison of object properties.

Reasoning About Physical Interactions with Object-Oriented Prediction and Planning

method learn
reason physical interaction
present
object
image


We study the error landscape of deep linear and nonlinear neural networks with the squared error loss.

Minimizing the loss of a deep linear neural network is a nonconvex problem, and despite recent progress, our understanding of this loss surface is still incomplete.

For deep linear networks, we present necessary and sufficient conditions for a critical point of the risk function to be a global minimum.

Surprisingly, our conditions provide an efficiently checkable test for global optimality, while such tests are typically intractable in nonconvex optimization.

We further extend these results to deep nonlinear neural networks and prove similar sufficient conditions for global optimality, albeit in a more limited function space setting.

We provide efficiently checkable necessary and sufficient conditions for global optimality in deep linear neural networks, with some initial extensions to nonlinear settings.

The paper gives conditions for the global optimality of the loss function of deep linear neural networks

The paper gives theoretical results regarding the existence of local minima in the objective function of deep neural networks.

Studies some theoretical properties of deep linear networks.

Global Optimality Conditions for Deep Neural Networks

condition global optimality
deep linear neural network
give
function
theoretical


Recurrent auto-encoder model can summarise sequential data through an encoder structure into a fixed-length vector and then reconstruct into its original sequential form through the decoder structure.

The summarised information can be used to represent time series features.

In this paper, we propose relaxing the dimensionality of the decoder output so that it performs partial reconstruction.

The fixed-length vector can therefore represent features only in the selected dimensions.

In addition, we propose using rolling fixed window approach to generate samples.

The change of time series features over time can be summarised as a smooth trajectory path.

The fixed-length vectors are further analysed through additional visualisation and unsupervised clustering techniques.

This proposed method can be applied in large-scale industrial processes for sensors signal analysis purpose where clusters of the vector representations can be used to reflect the operating states of selected aspects of the industrial system.

Using recurrent auto-encoder model to extract multidimensional time series features

This writeup describes an application of recurrent autoencoder to analyze of multidimensional time series

The paper describes a sequence to sequence auto-encoder model which is used to learn sequence representations, showing that for their application, better performance is obtained when the network is only trained to reconstruct a subset of the data measurements.

Proposes a strategy inspired by the recurrent auto-encoder model such that clustering multidimensional time series data can be performed based on context vectors.

Recurrent Auto-Encoder Model for Multidimensional Time Series Representation

recurrent auto-encoder model
multidimensional time series
application
datum
describe
representation


We view molecule optimization as a graph-to-graph translation problem.

The goal is to learn to map from one molecular graph to another with better properties based on an available corpus of paired molecules.

Since molecules can be optimized in different ways, there are multiple viable translations for each input graph.

A key challenge is therefore to model diverse translation outputs.

Our primary contributions include a junction tree encoder-decoder for learning diverse graph translations along with a novel adversarial training method for aligning distributions of molecules.

Diverse output distributions in our model are explicitly realized by low-dimensional latent vectors that modulate the translation process.

We evaluate our model on multiple molecule optimization tasks and show that our model outperforms previous state-of-the-art baselines by a significant margin.

We introduce a graph-to-graph encoder-decoder framework for learning diverse graph translations.

Proposes a graph-to-graph translation model for molecule optimization inspired by matched molecular pair analysis.

Extension of JT-VAE into the graph to graph translation scenario by adding the latent variable to capture multi-modality and an adversarial regularization in the latent space

Proposes a quite complex system, involving many different choices and components, for obtaining chemical compouds with improved properties starting from a given corpora.

Learning Multimodal Graph-to-Graph Translation for Molecule Optimization

graph translation
molecule optimization
graph-to-graph
learn


Partial differential equations (PDEs) are widely used across the physical and computational sciences.

Decades of research and engineering went into designing fast iterative solution methods.

Existing solvers are general purpose, but may be sub-optimal for specific classes of problems.

In contrast to existing hand-crafted solutions, we propose an approach to learn a fast iterative solver tailored to a specific domain.

We achieve this goal by learning to modify the updates of an existing solver using a deep neural network.

Crucially, our approach is proven to preserve strong correctness and convergence guarantees.

After training on a single geometry, our model generalizes to a wide variety of geometries and boundary conditions, and achieves 2-3 times speedup compared to state-of-the-art solvers.

We learn a fast neural solver for PDEs that has convergence guarantees.

Develops a method to accelerate the finite difference method in solving PDEs and proposes a revised framework for fixed point iteration after discretization.

The authors propose a linear method for speeding up PDE solvers.

Learning Neural PDE Solvers with Convergence Guarantees

pde
convergence guarantee
solver
method
neural
learn


Variational Bayesian neural networks (BNN) perform variational inference over weights, but it is difficult to specify meaningful priors and approximating posteriors in a high-dimensional weight space.

We introduce functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund (ELBO) defined directly on stochastic processes, i.e. distributions over functions.

We prove that the KL divergence between stochastic processes is equal to the supremum of marginal KL divergences over all finite sets of inputs.

Based on this, we introduce a practical training objective which approximates the functional ELBO using finite measurement sets and the spectral Stein gradient estimator.

With fBNNs, we can specify priors which entail rich structure, including Gaussian processes and implicit stochastic processes.

Empirically, we find that fBNNs extrapolate well using various structured priors, provide reliable uncertainty estimates, and can scale to large datasets.

We perform functional variational inference on the stochastic processes defined by Bayesian neural networks.

Fitting of variational Bayesian Neural Network approximations in functional form and considering matching to a stochastic process prior implicitly via samples.

Presents a novel ELBO objective for training BNNs which allows for more meaningful priors to be encoded in the model rather than the less informative weight priors features in the literature.

Presents a new variational inference algorithm for Bayesian neural network models where the prior is specified functionally rather than via a prior over weights.

FUNCTIONAL VARIATIONAL BAYESIAN NEURAL NETWORKS

bayesian neural network
stochastic process
prior
model
functional variational
rather
present
inference
via
weight


In fact, they form an aristocratic graph with a latent hierarchical structure that the next generation of unsupervised learned word embeddings should reveal.

In this paper, justified by the notion of delta-hyperbolicity or tree-likeliness of a space, we propose to embed words in a Cartesian product of hyperbolic spaces which we theoretically connect to the Gaussian word embeddings and their Fisher geometry.

This connection allows us to introduce a novel principled hypernymy score for word embeddings.

Moreover, we adapt the well-known Glove algorithm to learn unsupervised word embeddings in this type of Riemannian manifolds.

We further explain how to solve the analogy task using the Riemannian parallel transport that generalizes vector arithmetics to this new type of geometry.

Empirically, based on extensive experiments, we prove that our embeddings, trained unsupervised, are the first to simultaneously outperform strong and popular baselines on the tasks of similarity, analogy and hypernymy detection.

In particular, for word hypernymy, we obtain new state-of-the-art on fully unsupervised WBLESS classification accuracy.

We embed words in the hyperbolic space and make the connection  with the Gaussian word embeddings.

This paper adapts the Glove word embedding to a hyperbolic space given by the Poincare half-plane model

This paper proposes an approach to implement a GLOVE-based hyperbolic word embedding model, which is optimized via the Riemannian Optimization methods.

Poincare Glove: Hyperbolic Word Embeddings

hyperbolic space
word embedding
glove
model
poincare
embed


Answering questions about a text frequently requires aggregating information from multiple places in that text.

End-to-end neural network models, the dominant approach in the current literature, can theoretically learn how to distill and manipulate representations of the text without explicit supervision about how to do so.

We investigate a canonical architecture for this task, the memory network, and analyze how effective it really is in the context of three multi-hop reasoning settings.

In a simple synthetic setting, the path-finding task of the bAbI dataset, the model fails to learn the correct reasoning without additional supervision of its attention mechanism.

However, with this supervision, it can perform well.

On a real text dataset, WikiHop, the memory network gives nearly state-of-the-art performance, but does so without using its multi-hop capabilities.

A tougher anonymized version of the WikiHop dataset is qualitatively similar to bAbI: the model fails to perform well unless it has additional supervision.

We hypothesize that many "multi-hop" architectures do not truly learn this reasoning as advertised, though they could learn this reasoning if appropriately supervised.

Memory Networks do not learn multi-hop reasoning unless we supervise them.

Claims multi-hop reasoning is not easy to learn directly and requires direct supervision and doing well on WikiHop doesn't necessarily mean the model is actually learning to hop.

The paper proposes to investigate the well-known problem of memory network learning and more precisely the difficulty of the attention learning supervision with such models.

This paper argues that memory network fails to learn reasonable multi-hop reasoning.

How to learn (and how not to learn) multi-hop reasoning with memory networks

memory network learn
multi-hop reasoning
model
supervision


Generative Adversarial Nets (GANs) and Variational Auto-Encoders (VAEs) provide impressive image generations from Gaussian white noise, but the underlying mathematics are not well understood.

We compute deep convolutional network generators by inverting a fixed embedding operator.

Therefore, they do not require to be optimized with a discriminator or an encoder.

The embedding is Lipschitz continuous to deformations so that generators transform linear interpolations between input white noise vectors into deformations between output images.

This embedding is computed with a wavelet Scattering transform.

Numerical experiments demonstrate that the resulting Scattering generators have similar properties as GANs or VAEs, without learning a discriminative network or an encoder.

We introduce generative networks that do not require to be learned with a discriminator or an encoder; they are obtained by inverting a special embedding operator defined by a wavelet Scattering transform.

Introduces scattering transforms as image generative models in the context of Generative Adversarial Networks and suggest why they could be seen as Gaussianization transforms with controlled information loss and invertibility.

The paper proposes a generative model for images that does no require to learn a discriminator (as in GAN’s) or learned embedding.

Generative networks as inverse problems with Scattering transforms

require learn discriminator
generative model
scatter transform
network
image
introduce
embed


Recurrent neural networks (RNNs) can model natural language by sequentially ''reading'' input tokens and outputting a distributed representation of each token.

Due to the sequential nature of RNNs, inference time is linearly dependent on the input length, and all inputs are read regardless of their importance.

Efforts to speed up this inference, known as ''neural speed reading'', either ignore or skim over part of the input.

We present Structural-Jump-LSTM: the first neural speed reading model to both skip and jump text during inference.

The model consists of a standard LSTM and two agents: one capable of skipping single words when reading, and one capable of exploiting punctuation structure (sub-sentence separators (,:), sentence end symbols (.!?), or end of text markers) to jump ahead after reading a word.

A comprehensive experimental evaluation of our model against all five state-of-the-art neural reading models shows that

Structural-Jump-LSTM achieves the best overall floating point operations (FLOP) reduction (hence is faster), while keeping the same accuracy or even improving it compared to a vanilla LSTM that reads the whole text.

We propose a new model for neural speed reading that utilizes the inherent punctuation structure of a text to define effective jumping and skipping behavior.

The paper proposes a Structural-Jump-LSTM model to speed up machine reading with two agents instead of one

Proposes a novel model for neural speed reading in which the new reader has the ability to skip a word or sequence of words.

The paper proposes a fast-reading method using skip and jump actions, showing that the proposed method is as accurate as LSTM but uses much less computation.

Neural Speed Reading with Structural-Jump-LSTM

model neural speed reading
new
skip
structural-jump-lstm


One of the key challenges of session-based recommender systems is to enhance users’ purchase intentions.

In this paper, we formulate the sequential interactions between user sessions and a recommender agent as a Markov Decision Process (MDP).

In practice, the purchase reward is delayed and sparse, and may be buried by clicks, making it an impoverished signal for policy learning.

Inspired by the prediction error minimization (PEM) and embodied cognition, we propose a simple architecture to augment reward, namely Imagination Reconstruction Network (IRN).

Speciﬁcally, IRN enables the agent to explore its environment and learn predictive representations via three key components.

The imagination core generates predicted trajectories, i.e., imagined items that users may purchase.

The trajectory manager controls the granularity of imagined trajectories using the planning strategies, which balances the long-term rewards and short-term rewards.

To optimize the action policy, the imagination-augmented executor minimizes the intrinsic imagination error of simulated trajectories by self-supervised reconstruction, while maximizing the extrinsic reward using model-free algorithms.

Empirically, IRN promotes quicker adaptation to user interest, and shows improved robustness to the cold-start scenario and ultimately higher purchase performance compared to several baselines.

Somewhat surprisingly, IRN using only the purchase reward achieves excellent next-click prediction performance, demonstrating that the agent can "guess what you like" via internal planning.

We propose the IRN architecture to augment sparse and delayed purchase reward for session-based recommendation.

The paper proposes improving the performance of recommendation systems through reinforcement learning by using an Imagination Reconstruction Network.

The paper presents a session-based recommendation approach by focusing on user purchases instead of clicks.

Purchase as Reward : Session-based  Recommendation by Imagination Reconstruction

purchase reward
session-based recommendation
imagination reconstruction


The question why deep learning algorithms generalize so well has attracted increasing

However, most of the well-established approaches,

such as hypothesis capacity, stability or sparseness, have not provided complete

explanations (Zhang et al., 2016; Kawaguchi et al., 2017).

on the robustness approach (Xu & Mannor, 2012), i.e., if the error of a hypothesis

will not change much due to perturbations of its training examples, then it

As most deep learning algorithms are stochastic (e.g.,

Stochastic Gradient Descent, Dropout, and Bayes-by-backprop), we revisit the robustness

arguments of Xu & Mannor, and introduce a new approach – ensemble

robustness – that concerns the robustness of a population of hypotheses.

the lens of ensemble robustness, we reveal that a stochastic learning algorithm can

generalize well as long as its sensitiveness to adversarial perturbations is bounded

Moreover, an algorithm may be sensitive to

some adversarial examples (Goodfellow et al., 2015) but still generalize well.

support our claims, we provide extensive simulations for different deep learning

algorithms and different network architectures exhibiting a strong correlation between

ensemble robustness and the ability to generalize.

Explaining the generalization of stochastic deep learning algorithms, theoretically and empirically, via ensemble robustness

This paper presents an adaptation of the algorithmic robustness of Xu&Mannor'12 and presents learning bounds and an experimental showing correlation between empirical ensemble robustness and generalization error.

Proposes a study of the generalization ability of deep learning algorithms using an extension of notion of stability called ensemble robustness and gives bounds on generalization error of a randomized algorithm in terms of stability parameter and provides empirical study attempting to connect theory with practice.

The paper studied the generalization ability of learning algorithms from the robustness viewpoint in a deep learning context

Ensemble Robustness and Generalization of Stochastic Deep Learning Algorithms

generalization stochastic deep learning algorithm


Deep autoregressive models have shown state-of-the-art performance in density estimation for natural images on large-scale datasets such as ImageNet.

However, such models require many thousands of gradient-based weight updates and unique image examples for training.

Ideally, the models would rapidly learn visual concepts from only a handful of examples, similar to the manner in which humans learns across many vision tasks.

In this paper, we show how 1) neural attention and 2) meta learning techniques can be used in combination with autoregressive models to enable effective few-shot density estimation.

Our proposed modifications to PixelCNN result in state-of-the art few-shot density estimation on the Omniglot dataset.

Furthermore, we visualize the learned attention policy and find that it learns intuitive algorithms for simple tasks such as image mirroring on ImageNet and handwriting on Omniglot without supervision.

Finally, we extend the model to natural images and demonstrate few-shot image generation on the Stanford Online Products dataset.

The paper proposes on using density estimation when the availability of training data is low by using a meta-learning model.

This paper considers the problem of one/few-shot density estimation, using metalearning techniques that have been applied to one/few-shot supervised learning

The paper focuses on few shot learning with autoregressive density estimation and improves PixelCNN with neural attention and meta learning techniques.

Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions

density estimation
technique
learn
learning
pixelcnn
few-shot
autoregressive


Neural networks exhibit good generalization behavior in the

over-parameterized regime, where the number of network parameters

current generalization bounds for neural networks fail to explain this

In an attempt to bridge this gap, we study the problem of

learning a two-layer over-parameterized neural network, when the data is generated by a linearly separable function.

ReLU activations, we provide both optimization and generalization guarantees for over-parameterized networks.

Specifically, we prove convergence rates of SGD to a global

minimum and provide generalization guarantees for this global minimum

that are independent of the network size.

Therefore, our result clearly shows that the use of SGD for optimization both finds a global minimum, and avoids overfitting despite the high capacity of the model.

This is the first theoretical demonstration that SGD can avoid overfitting, when learning over-specified neural network classifiers.

We show that SGD learns two-layer over-parameterized neural networks with Leaky ReLU activations that provably generalize on linearly separable data.

The paper studies overparameterised models being able to learn well-generalising solutions by using a 1-hidden layer network with fixed output layer.

This paper shows that on linearly seperabel data, SGD on an overparameterized network can still lean a classifier that provably generalizes.

SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data

provably generalize linearly separable datum


A central challenge in reinforcement learning is discovering effective policies for tasks where rewards are sparsely distributed.

We postulate that in the absence of useful reward signals, an effective exploration strategy should seek out {\it decision states}.

These states lie at critical junctions in the state space from where the agent can transition to new, potentially unexplored regions.

We propose to learn about decision states from prior experience.

By training a goal-conditioned model with an information bottleneck, we can identify decision states by examining where the model accesses the goal state through the bottleneck.

We find that this simple mechanism effectively identifies decision states, even in partially observed settings.

In effect, the model learns the sensory cues that correlate with potential subgoals.

In new environments, this model can then identify novel subgoals for further exploration, guiding the agent through a sequence of potential decision  states and through new regions of the state space.

Training agents with goal-policy information bottlenecks promotes transfer and yields a powerful exploration bonus

Proposes regularizing standard RL losses with the negative conditional mutual information for policy search in a multi-goal RL setting.

This paper proposes the concept of decision state and proposes a KL divergence regularization to learn the structure of the tasks to use this information to encourage the policy to visit the decision states.

The paper proposes a method of regularising goal-conditioned policies with a mutual information term.

InfoBot: Transfer and Exploration via the Information Bottleneck

mutual information
policy
transfer
bottleneck
exploration


Many applications in machine learning require optimizing a function whose true gradient is unknown, but where surrogate gradient information (directions that may be correlated with, but not necessarily identical to, the true gradient) is available instead.

This arises when an approximate gradient is easier to compute than the full gradient (e.g. in meta-learning or unrolled optimization), or when a true gradient is intractable and is replaced with a surrogate (e.g. in certain reinforcement learning applications or training networks with discrete variables).

We propose Guided Evolutionary Strategies, a method for optimally using surrogate gradient directions along with random search.

We define a search distribution for evolutionary strategies that is elongated along a subspace spanned by the surrogate gradients.

This allows us to estimate a descent direction which can then be passed to a first-order optimizer.

We analytically and numerically characterize the tradeoffs that result from tuning how strongly the search distribution is stretched along the guiding subspace, and use this to derive a setting of the hyperparameters that works well across problems.

Finally, we apply our method to example problems including truncated unrolled optimization and training neural networks with discrete variables, demonstrating improvement over both standard evolutionary strategies and first-order methods (that directly follow the surrogate gradient).

We provide a demo of Guided ES at: redacted URL

We propose an optimization method for when only biased gradients are available--we define a new gradient estimator for this scenario, derive the bias and variance of this estimator, and apply it to example problems.

The authors propose an approach that combines random search with the surrogate gradient information and give a discussion on variance-bias trade-off as well as a discussion on hyperparameter optimization.

The paper proposes a method to improve random search by building a subspace of the previous k surrogate gradients.

This paper attempts accelerating the OpenAI type evolution by introducing a non-isotrophic distribution with a covariance matrix in the form I + UU^t and external information such as a surrogate gradient to determine U

Guided Evolutionary Strategies: Escaping the curse of dimensionality in random search

method
surrogate gradient
random search
optimization
information


Point clouds are an important type of geometric data and have widespread use in computer graphics and vision.

However, learning representations for point clouds is particularly challenging due to their nature as being an unordered collection of points irregularly distributed in 3D space.

Graph convolution, a generalization of the convolution operation for data defined over graphs, has been recently shown to be very successful at extracting localized features from point clouds in supervised or semi-supervised tasks such as classification or segmentation.

This paper studies the unsupervised problem of a generative model exploiting graph convolution.

We focus on the generator of a GAN and define methods for graph convolution when the graph is not known in advance as it is the very output of the generator.

The proposed architecture learns to generate localized features that approximate graph embeddings of the output geometry.

We also study the problem of defining an upsampling layer in the graph-convolutional generator, such that it learns to exploit a self-similarity prior on the data distribution to sample more effectively.

A GAN using graph convolution operations with dynamically computed graphs from hidden features

The paper proposes a version of GANs specifically designed for generating point clouds with the core contribution of the work the upsampling operation.

This paper proposes graph-convolutional GANs for irregular 3D point clouds that learn domain and features at the same time.

Learning Localized Generative Models for 3D Point Clouds via Graph Convolution

3d point cloud
gan
operation
graph convolution
feature
learn


Memorization in over-parameterized neural networks can severely hurt generalization in the presence of mislabeled examples.

However, mislabeled examples are to hard avoid in extremely large datasets.

We address this problem using the implicit regularization effect of stochastic gradient descent with large learning rates, which we find to be able to separate clean and mislabeled examples with remarkable success using loss statistics.

We leverage this to identify and on-the-fly discard mislabeled examples using a threshold on their losses.

This leads to On-the-fly Data Denoising (ODD), a simple yet effective algorithm that is robust to mislabeled examples, while introducing almost zero computational overhead.

Empirical results demonstrate the effectiveness of ODD on several datasets containing artificial and real-world mislabeled examples.

We introduce a fast and easy-to-implement algorithm that is robust to dataset noise.

The paper aims to remove potential examples with label noise by discarding the ones with large losses in the training procedure.

Better Generalization with On-the-fly Dataset Denoising


Binarized Neural Networks (BNNs) have recently attracted significant interest due to their computational efficiency.

Concurrently, it has been shown that neural networks may be overly sensitive to ``attacks" -- tiny adversarial changes in the input -- which may be detrimental to their use in safety-critical domains.

Designing attack algorithms that effectively fool trained models is a key step towards learning robust neural networks.

The discrete, non-differentiable nature of BNNs, which distinguishes them from their full-precision counterparts, poses a challenge to gradient-based attacks.

In this work, we study the problem of attacking a BNN through the lens of combinatorial and integer optimization.

We propose a Mixed Integer Linear Programming (MILP) formulation of the problem.

While exact and flexible, the MILP quickly becomes intractable as the network and perturbation space grow.

To address this issue, we propose IProp, a decomposition-based algorithm that solves a sequence of much smaller MILP problems.

Experimentally, we evaluate both proposed methods against the standard gradient-based attack (PGD) on MNIST and Fashion-MNIST, and show that IProp performs favorably compared to PGD, while scaling beyond the limits of the MILP.

Gradient-based attacks on binarized neural networks are not effective due to the non-differentiability of such networks; Our IPROP algorithm solves this problem using integer optimization

Proposes a new target propagation style algorithm to generate strong adversarial attacks on binarized neural networks.

This paper proposed a new attack algorithm based on MILP on binary neural networks.

This paper presents an algorithm to find adversarial attacks to binary neural networks which iteratively finds desired representations layer by layer from the top to the input and is more efficient than solving the full mixed integer linear programming (MILP) solver.

Combinatorial Attacks on Binarized Neural Networks

attack binarize neural network
new
milp
solve
algorithm
binary
integer
adversarial


Highly regularized LSTMs achieve impressive results on several benchmark datasets in language modeling.

We propose a new regularization method based on decoding the last token in the context using the predicted distribution of the next token.

This biases the model towards retaining more contextual information, in turn improving its ability to predict the next token.

With negligible overhead in the number of parameters and training time, our Past Decode Regularization (PDR) method achieves a word level perplexity of 55.6 on the Penn Treebank and 63.5 on the WikiText-2 datasets using a single softmax.

We also show gains by using PDR in combination with a mixture-of-softmaxes, achieving a word level perplexity of 53.8 and 60.5 on these datasets.

In addition, our method achieves 1.169 bits-per-character on the Penn Treebank Character dataset for character level language modeling.

These results constitute a new state-of-the-art in their respective settings.

Decoding the last token in the context using the predicted next token distribution acts as a regularizer and improves language modeling.

The authors introduce the idea of past decoding for the purpose of regularization for improved perplexity on Penn Treebank

Proposes an additional loss term to use when training an LSTM LM and shows that by adding this loss term they can achieve SOTA perplexity on a number of LM benchmarks.

Suggests a new regularization technique which can be added on top of those used in AWD-LSTM of Merity et al. (2017) with little overhead.

Improved Language Modeling by Decoding the Past

add
decode
improve language
past
perplexity
regularization


The assumption that data samples are independently identically distributed is the backbone of many learning algorithms.

Nevertheless, datasets often exhibit rich structures in practice, and we argue that there exist some unknown orders within the data instances.

Aiming to find such orders, we introduce a novel Generative Markov Network (GMN) which we use to extract the order of data instances automatically.

Specifically, we assume that the instances are sampled from a Markov chain.

Our goal is to learn the transitional operator of the chain as well as the generation order by maximizing the generation probability under all possible data permutations.

One of our key ideas is to use neural networks as a soft lookup table for approximating the possibly huge, but discrete transition matrix.

This strategy allows us to amortize the space complexity with a single model and make the transitional operator generalizable to unseen instances.

To ensure the learned Markov chain is ergodic, we propose a greedy batch-wise permutation scheme that allows fast training.

Empirically, we evaluate the learned Markov chain by showing that GMNs are able to discover orders among data instances and also perform comparably well to state-of-the-art methods on the one-shot recognition benchmark task.

Propose to observe implicit orders in datasets in a generative model viewpoint.

The authors deal with the problem of implicit ordering in a dataset and the challenge of recovering it and propose to learn a distance-metric-free model that assumes a Markov chain as the generative mechanism of the data

The paper proposes “Generative Markov Networks” - a deep-learning-based approach to modeling sequences and discovering order in datasets.

Proposes learning the order of an unordered data sample by learning a Markov chain.

Discovering Order in Unordered Datasets: Generative Markov Networks

dataset
markov chain
generative
learn
discover order
unordered
model
implicit
network


We present a Neural Program Search, an algorithm to generate programs from natural language description and a small number of input / output examples.

The algorithm combines methods from Deep Learning and Program Synthesis fields by designing rich domain-specific language (DSL) and defining efficient search algorithm guided by a Seq2Tree model on it.

To evaluate the quality of the approach we also present a semi-synthetic dataset of descriptions with test examples and corresponding programs.

We show that our algorithm significantly outperforms sequence-to-sequence model with attention baseline.

Program synthesis from natural language description and input / output examples via Tree-Beam Search over Seq2Tree model

Presents a seq2Tree model to translate a problem statement in natural language to the corresponding functional program in DSL, which has shown an improvement over the seq2seq baseline approach.

This paper tackles the problem of doing program synthesis when given a problem description and a small number of input-output examples.

The paper introduces a technique for program synthesis involving a restricted grammar of problems that is beam-searched using an attentional encoder-decoder network.

Neural Program Search: Solving Data Processing Tasks from Description and Examples

problem
program synthesis
natural language
seq2tree model
description
search
example


Generative adversarial training can be generally understood as minimizing certain moment matching loss defined by a set of discriminator functions, typically  neural networks.

The discriminator set should be large enough to be able to uniquely identify the true distribution (discriminative), and also be small enough to go beyond memorizing samples (generalizable).

In this paper, we show that a discriminator set is guaranteed to be discriminative whenever its linear span is dense in the set of bounded continuous functions.

This is a very mild condition satisfied even by neural networks with a single neuron.

Further, we develop generalization bounds between the learned distribution and true distribution under different evaluation metrics.

When evaluated with neural distance, our bounds show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set.

When evaluated with KL divergence, our bound provides an explanation on the counter-intuitive behaviors of testing likelihood in GAN training.

Our analysis sheds lights on understanding the practical performance of GANs.

This paper studies the discrimination and generalization properties of GANs when the discriminator set is a restricted function class like neural networks.

Balances capacities of generator and discriminator classes in GANs by guaranteeing that induced IPMs are metrics and not pseudo metrics

This paper provides a mathematical analysis of the role of the size of the adversary/discriminator set in GANs

On the Discrimination-Generalization Tradeoff in GANs


Normalization layers are a staple in state-of-the-art deep neural network architectures.

They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic.

In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization.

Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization.

We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers.

Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation.

All you need to train deep residual networks is a good initialization; normalization layers are not necessary.

A method is presented for initialization and normalization of deep residual networks. This is based on observations of forward and backward explosion in such networks. The method performance is on par with the best results obtained by other networks with more explicit normalization.

The authors propose a novel way to initialize residual networks, which is motivated by the need to avoid exploding/vanishing gradients.

Proposes a new initialization method used to train very deep RedNets without using batch-norm.

Fixup Initialization: Residual Learning Without Normalization

deep residual network
need
good
method
train
initialization
normalization
without


Designing a metric manually for unsupervised sequence generation tasks, such as text generation, is essentially difficult.

In a such situation, learning a metric of a sequence from data is one possible solution.

The previous study, SeqGAN, proposed the framework for unsupervised sequence generation, in which a metric is learned from data, and a generator is optimized with regard to the learned metric with policy gradient, inspired by generative adversarial nets (GANs) and reinforcement learning.

In this paper, we make two proposals to learn better metric than SeqGAN's: partial reward function and expert-based reward function training.

The partial reward function is a reward function for a partial sequence of a certain length.

SeqGAN employs a reward function for completed sequence only.

By combining long-scale and short-scale partial reward functions, we expect a learned metric to be able to evaluate a partial correctness as well as a coherence of a sequence, as a whole.

In expert-based reward function training, a reward function is trained to discriminate between an expert (or true) sequence and a fake sequence that is produced by editing an expert sequence.

Expert-based reward function training is not a kind of GAN frameworks.

This makes the optimization of the generator easier.

We examine the effect of the partial reward function and expert-based reward function training on synthetic data and real text data, and show improvements over SeqGAN and the model trained with MLE.

Specifically, whereas SeqGAN gains 0.42 improvement of NLL over MLE on synthetic data, our best model gains 3.02 improvement, and whereas SeqGAN gains 0.029 improvement of BLEU over MLE, our best model gains 0.250 improvement.

This paper aims to learn a better metric for unsupervised learning, such as text generation, and shows a significant improvement over SeqGAN.

Describes an approach to generating time sequences by learning state-action values, where the state is the sequence generated so far, and the action is the choice of the next value.

This paper considers the problem of improving sequence generation by learning better metrics, specifically the exposure bias problem

Toward learning better metrics for sequence generation training with policy gradient


One of the most successful techniques in generative models has been decomposing a complicated generation task into a series of simpler generation tasks.

For example, generating an image at a low resolution and then learning to refine that into a high resolution image often improves results substantially.

Here we explore a novel strategy for decomposing generation for complicated objects in which we first generate latent variables which describe a subset of the observed variables, and then map from these latent variables to the observed space.

We show that this allows us to achieve decoupled training of complicated generative models and present both theoretical and experimental results supporting the benefit of such an approach.

Decompose the task of learning a generative model into learning disentangled latent factors for subsets of the data and then learning the joint over those latent factors.

Locally Disentangled Factors for hierarchical latent variable generative model, which can be seen as a hierarchical variant of Adversarially Learned Inference

The paper investigates the potential of hierarchical latent variable models for generating images and image sequences and proposes to train several ALI models stacked on top of each other to create a hierarchical representation of the data.

The paper aims to learn the hierarchies for training GAN in a hierarchical optimization schedule directly instead of being designed by a human

Learning Generative Models with Locally Disentangled Latent Factors

learn generative model
datum
hierarchical latent variable
factor
locally disentangle


Visual grounding of language is an active research field aiming at enriching text-based representations with visual information.

In this paper, we propose a new way to leverage visual knowledge for sentence representations.

Our approach transfers the structure of a visual representation space to the textual space by using two complementary sources of information: (1) the cluster information: the implicit knowledge that two sentences associated with the same visual content describe the same underlying reality and (2) the perceptual information contained within the structure of the visual space.

We use a joint approach to encourage beneficial interactions during training between textual, perceptual, and cluster information.

We demonstrate the quality of the learned representations on semantic relatedness, classification, and cross-modal retrieval tasks.

We propose a joint model to incorporate visual knowledge in sentence representations

The paper proposes a method to use videos paired with captions to improve sentence embeddings

This submission proposes a model for sentence learning sentence representations that are grounded, based on associated video data.

Proposes a method for improving text-based sentence embeddings through a joint multimodal framework.

Learning Grounded Sentence Representations by Jointly Using Video and Text Information

method
sentence representation
joint
video
model
improve
embedding
learn
ground