Weight-sharing plays a significant role in the success of many deep neural networks, by increasing memory efficiency and incorporating useful inductive priors about the problem into the network.

But understanding how weight-sharing can be used effectively in general is a topic that has not been studied extensively.

Chen et al. (2015) proposed HashedNets, which augments a multi-layer perceptron with a hash table, as a method for neural network compression.

We generalize this method into a framework (ArbNets) that allows for efficient arbitrary weight-sharing, and use it to study the role of weight-sharing in neural networks.

We show that common neural networks can be expressed as ArbNets with different hash functions.

We also present two novel hash functions, the Dirichlet hash and the Neighborhood hash, and use them to demonstrate experimentally that balanced and deterministic weight-sharing helps with the performance of a neural network.

Studied the role of weight sharing in neural networks using hash functions, found that a balanced and deterministic hash function helps network performance.

Proposing ArbNets to study weight sharing in a more systematic way by defining the weight sharing function as a hash function.

Balanced and Deterministic Weight-sharing Helps Network Performance

weight sharing
hash function
help network performance
balanced deterministic
study


We introduce Neural Markov Logic Networks (NMLNs), a statistical relational learning system that borrows ideas from Markov logic.

Like Markov Logic Networks (MLNs), NMLNs are an exponential-family model for modelling distributions over possible worlds, but unlike MLNs, they do not rely on explicitly specified first-order logic rules.

Instead, NMLNs learn an implicit representation of such rules as a neural network that acts as a potential function on fragments of the relational structure.

Interestingly, any MLN can be represented as an NMLN.

Similarly to recently proposed Neural theorem provers (NTPs) (Rocktaschel at al. 2017), NMLNs can exploit embeddings of constants but, unlike NTPs, NMLNs work well also in their absence.

This is extremely important for predicting in settings other than the transductive one.

We showcase the potential of NMLNs on knowledge-base completion tasks and on generation of molecular (graph) data.

We introduce a statistical relational learning system that borrows ideas from Markov logic but learns an implicit representation of rules as a neural network.

The paper provides an extension to Markov Logic Networks by removing their dependency on pre-defined first-order logic rules to model more domains in knowledge-base completion tasks.

Neural Markov Logic Networks


Using variational Bayes neural networks, we develop an algorithm capable of accumulating knowledge into a prior from multiple different tasks.

This results in a rich prior capable of few-shot learning on new tasks.

The posterior can go beyond the mean field approximation and yields good uncertainty on the performed experiments.

Analysis on toy tasks show that it can learn from significantly different tasks while finding similarities among them.

Experiments on Mini-Imagenet reach state of the art with 74.5% accuracy on 5 shot learning.

Finally, we provide two new benchmarks, each showing a failure mode of existing meta learning algorithms such as MAML and prototypical Networks.

A scalable method for learning an expressive prior over neural networks across multiple tasks.

The paper presents a method for training a probabilistic model for Multitasks Transfer Learning by introducing a latent variable per task to capture the commonality in the task instances.

The work proposes a variational approach to meta-learning that employs latent variables corresponding to task-specific datasets.

Aims to learn a prior over neural networks for multiple tasks.

Uncertainty in Multitask Transfer Learning

prior neural network
multitask transfer learn
multiple task
method
latent variable


Sequential data often originates from diverse environments.

Across them exist both shared regularities and environment specifics.

To learn robust cross-environment descriptions of sequences we introduce disentangled state space models (DSSM).

In the latent space of DSSM environment-invariant state dynamics is explicitly disentangled from environment-specific information governing that dynamics.

We empirically show that such separation enables robust prediction, sequence manipulation and environment characterization.

We also propose an unsupervised VAE-based training procedure to learn DSSM as Bayesian filters.

In our experiments, we demonstrate state-of-the-art performance in controlled generation and prediction of bouncing ball video sequences across varying gravitational influences.

The paper presents a generative state space model using a global latent variable E to capture environment-specific information.

DISENTANGLED STATE SPACE MODELS: UNSUPERVISED LEARNING OF DYNAMICS ACROSS HETEROGENEOUS ENVIRONMENTS


In this work, we approach one-shot and few-shot learning problems as methods for finding good prototypes for each class, where these prototypes are generalizable to new data samples and classes.

We propose a metric learner that learns a Bregman divergence by learning its underlying convex function.

Bregman divergences are a good candidate for this framework given they are the only class of divergences with the property that the best representative of a set of points is given by its mean.

We propose a flexible extension to prototypical networks to enable joint learning of the embedding and the divergence, while preserving computational efficiency.

Our preliminary results are comparable with the prior work on the Omniglot and Mini-ImageNet datasets, two standard benchmarks for one-shot and few-shot learning.

We argue that our model can be used for other tasks that involve metric learning or tasks that require approximate convexity such as structured prediction and data completion.

Bregman divergence learning for few-shot learning.

Prototypical Bregman Networks


Motivated by the flexibility of biological neural networks whose connectivity structure changes significantly during their lifetime,we introduce the Unrestricted Recursive Network (URN) and demonstrate that it can exhibit similar flexibility during training via gradient descent.

We show empirically that many of the different neural network structures commonly used in practice today (including fully connected, locally connected and residual networks of differ-ent depths and widths) can emerge dynamically from the same URN.These different structures can be derived using gradient descent on a single general loss function where the structure of the data and the relative strengths of various regulator terms determine the structure of the emergent network.

We show that this loss function and the regulators arise naturally when considering the symmetries of the network as well as the geometric properties of the input data.

We introduce a network framework which can modify its structure during training and show that it can converge to various ML network archetypes such as MLPs and LCNs.

Emergent Structures and Lifetime Structure Evolution in Artificial Neural Networks


We present CROSSGRAD , a method to use multi-domain training data to learn a classifier that generalizes to new domains.

CROSSGRAD does not need an adaptation phase via labeled or unlabeled data, or domain features in the new domain.

Most existing domain adaptation methods attempt to erase domain signals using techniques like domain adversarial training.

In contrast, CROSSGRAD is free to use domain signals for predicting labels, if it can prevent overfitting on training domains.

We conceptualize the task in a Bayesian setting, in which a sampling step is implemented as data augmentation, based on domain-guided perturbations of input instances.

CROSSGRAD jointly trains a label and a domain classifier on examples perturbed by loss gradients of each other’s objectives.

This enables us to directly perturb inputs, without separating and re-mixing domain signals while making various distributional assumptions.

Empirical evaluation on three different applications where this setting is natural establishes that

(1) domain-guided perturbation provides consistently better generalization to unseen domains, compared to generic instance perturbation methods, and

(2) data augmentation is a more stable and accurate method than domain adversarial training.

Domain guided augmentation of data provides a robust and stable method of domain generalization

This paper proposes a domain generalization approach by domain-dependent data augmentation

The authors introduce the CrossGrad method, which trains both a label classification task and a domain classification task.

Generalizing Across Domains via Cross-Gradient Training

domain generalization
method
augmentation
datum


We present sketch-rnn, a recurrent neural network able to construct stroke-based drawings of common objects.

The model is trained on a dataset of human-drawn images representing many different classes.

We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.

We investigate alternative to traditional pixel image modelling approaches, and propose a generative model for vector images.

This paper introduces a neural network architecture for generating sketch drawings inspired by the variational autoencoder.

A Neural Representation of Sketch Drawings


Wilson et al. (2017) showed that, when the stepsize schedule is properly designed, stochastic gradient generalizes better than ADAM (Kingma & Ba, 2014).

In light of recent work on hypergradient methods (Baydin et al., 2018), we revisit these claims to see if such methods close the gap between the most popular optimizers.

As a byproduct, we analyze the true benefit of these hypergradient methods compared to more classical schedules, such as the fixed decay of Wilson et al. (2017).

In particular, we observe they are of marginal help since their performance varies significantly when tuning their hyperparameters.

Finally, as robustness is a critical quality of an optimizer, we provide a sensitivity analysis of these gradient based optimizers to assess how challenging their tuning is.

We provide a study trying to see how the recent online learning rate adaptation extends the conclusion made by Wilson et al. 2018 about adaptive gradient methods, along with comparison and sensitivity analysis.

Reports the results of testing several stepsize adjustment related methods including vanilla SGD, SGD with Neserov momentum, and ADAM and compares those methods with hypergradient and without.

Combining adaptive algorithms and hypergradient method: a performance and robustness study


Despite an ever growing literature on reinforcement learning algorithms and applications, much less is known about their statistical inference.

In this paper, we investigate the large-sample behaviors of the Q-value estimates with closed-form characterizations of the asymptotic variances.

This allows us to efficiently construct confidence regions for Q-value and optimal value functions, and to develop policies to minimize their estimation errors.

This also leads to a policy exploration strategy that relies on estimating the relative discrepancies among the Q estimates.

Numerical experiments show superior performances of our exploration strategy than other benchmark approaches.

We investigate the large-sample behaviors of the Q-value estimates and proposed an efficient exploration strategy that relies on estimating the relative discrepancies among the Q estimates.

This paper presents a pure-exploration algorithm for reinforcement learning based on an asymptotic analysis of Q-values and their convergence to central limit distribution, outperforming benchmark exploration algorithms.

Efficient Inference and Exploration for Reinforcement Learning

q-value
reinforcement learning
efficient
exploration


We perform completely unsupervised one-sided image to image translation between a source domain $X$ and a target domain $Y$ such that we preserve relevant underlying shared semantics (e.g., class, size, shape, etc).

In particular, we are interested in a more difficult case than those typically addressed in the literature, where the source and target are ``far" enough that reconstruction-style or pixel-wise approaches fail.

We argue that transferring (i.e., \emph{translating}) said relevant information should involve both discarding source domain-specific information while incorporate target domain-specific information, the latter of which we model with a noisy prior distribution.

In order to avoid the degenerate case where the generated samples are only explained by the prior distribution, we propose to minimize an estimate of the mutual information between the generated sample and the sample from the prior distribution.

We discover that the architectural choices are an important factor to consider in order to preserve the shared semantic between $X$ and $Y$.

We show state of the art results on the MNIST to SVHN task for unsupervised image to image translation.

We train an image to image translation network that take as input the source image and a sample from a prior distribution to generate a sample from the target distribution

This paper formalizes the problem of unsupervised translation and proposes an augmented GAN framework which uses the mutual information to avoid the degenerate case

This paper formulates the problem of unsupervised one-to-many image translation and addresses the problem by minimizing the mutual information.

Unsupervised  one-to-many image translation

mutual information
problem unsupervised
image translation
one-to-many


Identifying salient points in images is a crucial component for visual odometry, Structure-from-Motion or SLAM algorithms.

Recently, several learned keypoint methods have demonstrated compelling performance on challenging benchmarks.

However, generating consistent and accurate training data for interest-point detection in natural images still remains challenging, especially for human annotators.

We introduce IO-Net (i.e. InlierOutlierNet), a novel proxy task for the self-supervision of keypoint detection, description and matching.

By making the sampling of inlier-outlier sets from point-pair correspondences fully differentiable within the keypoint learning framework, we show that are able to simultaneously self-supervise keypoint description and improve keypoint matching.

Second, we introduce KeyPointNet, a keypoint-network architecture that is especially amenable to robust keypoint detection and description.

We design the network to allow local keypoint aggregation to avoid artifacts due to spatial discretizations commonly used for this task, and we improve fine-grained keypoint descriptor performance by taking advantage of efficient sub-pixel convolutions to upsample the descriptor feature-maps to a higher operating resolution.

Through extensive experiments and ablative analysis, we show that the proposed self-supervised keypoint learning method greatly improves the quality of feature matching and homography estimation on challenging benchmarks over the state-of-the-art.

Learning to extract distinguishable keypoints from a proxy task, outlier rejection.

This paper is devoted to the self-supervised learning of local features using Neural Guided RANSAC as an additional auxillary loss provider for improving descriptor interpolation.

Neural Outlier Rejection for Self-Supervised Keypoint Learning

rejection self-supervised keypoint neural learning


We study the role of intrinsic motivation as an exploration bias for reinforcement learning in sparse-reward synergistic tasks, which are tasks where multiple agents must work together to achieve a goal they could not individually.

Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own.

Thus, we propose to incentivize agents to take (joint) actions whose effects cannot be predicted via a composition of the predicted effect for each individual agent.

We study two instantiations of this idea, one based on the true states encountered, and another based on a dynamics model trained concurrently with the policy.

While the former is simpler, the latter has the benefit of being analytically differentiable with respect to the action taken.

We validate our approach in robotic bimanual manipulation tasks with sparse rewards; we find that our approach yields more efficient learning than both

1) training with only the sparse reward and

2) using the typical surprise-based formulation of intrinsic motivation, which does not bias toward synergistic behavior.

Videos are available on the project webpage: https://sites.google.com/view/iclr2020-synergistic.

We propose a formulation of intrinsic motivation that is suitable as an exploration bias in multi-agent sparse-reward synergistic tasks, by encouraging agents to affect the world in ways that would not be achieved if they were acting individually.

The paper focuses on using intrinsic motivation to improve the exploration process of reinforcement learning agents in tasks that require multi-agent to achieve.

Intrinsic Motivation for Encouraging Synergistic Behavior

intrinsic motivation
achieve
task
exploration
encourage
multi-agent
agent
synergistic


A general graph-structured neural network architecture operates on graphs through two core components: (1) complex enough message functions; (2) a fixed information aggregation process.

In this paper, we present the Policy Message Passing algorithm, which takes a probabilistic perspective and reformulates the whole information aggregation as stochastic sequential processes.

The algorithm works on a much larger search space, utilizes reasoning history to perform inference, and is robust to noisy edges.

We apply our algorithm to multiple complex graph reasoning and prediction tasks and show that our algorithm consistently outperforms state-of-the-art graph-structured models by a significant margin.

An probabilistic inference algorithm driven by neural network for graph-structured models

This paper introduces policy message passing, a graph neural network with an inference mechanism that assigns messages to edges in a recurrent fashion, indicating competitive performance on visual reasoning tasks.

Policy Message Passing: A New Algorithm for Probabilistic Graph Inference

policy message pass
neural network
probabilistic
algorithm
inference
graph


Deep multitask networks, in which one neural network produces multiple predictive outputs, are more scalable and often better regularized than their single-task counterparts.

Such advantages can potentially lead to gains in both speed and performance, but multitask networks are also difficult to train without finding the right balance between tasks.

We present a novel gradient normalization (GradNorm) technique which automatically balances the multitask loss function by directly tuning the gradients to equalize task training rates.

We show that for various network architectures, for both regression and classification tasks, and on both synthetic and real datasets, GradNorm improves accuracy and reduces overfitting over single networks, static baselines, and other adaptive multitask loss balancing techniques.

GradNorm also matches or surpasses the performance of exhaustive grid search methods, despite only involving a single asymmetry hyperparameter $\alpha$.

Thus, what was once a tedious search process which incurred exponentially more compute for each task added can now be accomplished within a few training runs, irrespective of the number of tasks.

Ultimately, we hope to demonstrate that gradient manipulation affords us great control over the training dynamics of multitask networks and may be one of the keys to unlocking the potential of multitask learning.

We show how you can boost performance in a multitask network by tuning an adaptive multitask loss function that is learned through directly balancing network gradients.

This work proposes a dynamic weight update scheme that updates weights for different task losses during training time by making use of the loss ratios of different tasks.

GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

multitask network
gradient
adaptive
loss
balance


Image segmentation aims at grouping pixels that belong to the same object or region.

At the heart of image segmentation lies the problem of determining whether a pixel is inside or outside a region, which we denote as the "insideness" problem.

Many Deep Neural Networks (DNNs) variants excel in segmentation benchmarks, but regarding insideness, they have not been well visualized or understood: What representations do DNNs use to address the long-range relationships of insideness?

How do architectural choices affect the learning of these representations?

In this paper, we take the reductionist approach by analyzing DNNs solving the insideness problem in isolation, i.e. determining the inside of closed (Jordan) curves.

We demonstrate analytically that state-of-the-art feed-forward and recurrent architectures can implement solutions of the insideness problem for any given curve.

Yet, only recurrent networks could  learn these general solutions when the training enforced a specific "routine" capable of breaking down the long-range relationships.

Our results highlights the need for new training strategies that decompose the learning into appropriate stages, and that lead to the general class of solutions necessary for DNNs to understand insideness.

DNNs for image segmentation can implement solutions for the insideness problem but only some recurrent nets could learn them with a specific type of supervision.

This paper introduces insideness to study semantic segmentation in deep learning era, and the results can help models generalize better.

Do Deep Neural Networks for Segmentation Understand Insideness?


We address the challenging problem of deep representation learning--the efficient adaption of a pre-trained deep network to different tasks.

Specifically, we propose to explore gradient-based features.

These features are gradients of the model parameters with respect to a task-specific loss given an input sample.

Our key innovation is the design of a linear model that incorporates both gradient features and the activation of the network.

We show that our model provides a local linear approximation to a underlying deep model, and discuss important theoretical insight.

Moreover, we present an efficient algorithm for the training and inference of our model without computing the actual gradients.

Our method is evaluated across a number of representation learning tasks on several datasets and using different network architectures.

We demonstrate strong results in all settings.

And our results are well-aligned with our theoretical insight.

Given a pre-trained model, we explored the per-sample gradients of the model parameters relative to a task-specific loss, and constructed a linear model that combines gradients of model parameters and the activation of the model.

This paper proposes to use the gradients of specific layers of convolutional networks as features in a linearized model for transfer learning and fast adaptation.

Gradients as Features for Deep Representation Learning


Recovering 3D geometry shape, albedo and lighting from a single image has wide applications in many areas, which is also a typical ill-posed problem.

In order to eliminate the ambiguity, face prior knowledge like linear 3D morphable models (3DMM) learned from limited scan data are often adopted to the reconstruction process.

However, methods based on linear parametric models cannot generalize well for facial images in the wild with various ages, ethnicity, expressions, poses, and lightings.

Recent methods aim to learn a nonlinear parametric model using convolutional neural networks (CNN) to regress the face shape and texture directly.

However, the models were only trained on a dataset that is generated from a linear 3DMM.

Moreover, the identity and expression representations are entangled in these models, which hurdles many facial editing applications.

In this paper, we train our model with adversarial loss in a semi-supervised manner on hybrid batches of unlabeled and labeled face images to exploit the value of large amounts of unlabeled face images from unconstrained photo collections.

A novel center loss is introduced to make sure that different facial images from the same person have the same identity shape and albedo.

Besides, our proposed model disentangles identity, expression, pose, and lighting representations, which improves the overall reconstruction performance and facilitates facial editing applications, e.g., expression transfer.

Comprehensive experiments demonstrate that our model produces high-quality reconstruction compared to state-of-the-art methods and is robust to various expression, pose, and lighting conditions.

We train our face reconstruction model with adversarial loss in semi-supervised manner on hybrid batches of unlabeled and labeled face images to exploit the value of large amounts of unlabeled face images from unconstrained photo collections.

This paper proposes a semi-supervised and adversarial training process to exact nonlinear disentangled representations from a face image with loss functions, achieving state-of-the-art performance in face reconstruction.

Semi-supervised 3D Face Reconstruction with Nonlinear Disentangled Representations

nonlinear disentangle representation
face image
adversarial
loss
semi-supervised
reconstruction


Human conversations naturally evolve around related entities and connected concepts, while may also shift from topic to topic.

This paper presents ConceptFlow, which leverages commonsense knowledge graphs to explicitly model such conversation flows for better conversation response generation.

ConceptFlow grounds the conversation inputs to the latent concept space and represents the potential conversation flow as a concept flow along the commonsense relations.

The concept is guided by a graph attention mechanism that models the possibility of the conversation evolving towards different concepts.

The conversation response is then decoded using the encodings of both utterance texts and concept flows, integrating the learned conversation structure in the concept space.

Our experiments on Reddit conversations demonstrate the advantage of ConceptFlow over previous commonsense aware dialog models and fine-tuned GPT-2 models, while using much fewer parameters but with explicit modeling of conversation structures.

This paper presents ConceptFlow that explicitly models the conversation flow in commonsense knowledge graph for better conversation generation.

The paper proposes a system for generating a single-turn response to a posted utterance in an open-domain dialog setting using the diffiusion into the neighbors of the grounded concepts.

Conversation Generation with Concept Flow


Biological neural networks face homeostatic and resource constraints that restrict the allowed configurations of connection weights.

If a constraint is tight it defines a very small solution space, and the size of these constraint spaces determines their potential overlap with the solutions for computational tasks.

We study the geometry of the solution spaces for constraints on neurons' total synaptic weight and on individual synaptic weights, characterizing the connection degrees (numbers of partners) that maximize the size of these solution spaces.

We then hypothesize that the size of constraints' solution spaces could serve as a cost function governing neural circuit development.

We develop analytical approximations and bounds for the model evidence of the maximum entropy degree distributions under these cost functions.

We test these on a published electron microscopic connectome of an associative learning center in the fly brain, finding evidence for a developmental progression in circuit structure.

We examine the hypothesis that the entropy of solution spaces for constraints on synaptic weights (the "flexibility" of the constraint) could serve as a cost function for neural circuit development.

Flexible degrees of connectivity under synaptic weight constraints


In  this  preliminary  work,  we  study  the  generalization  properties  of  infinite  ensembles  of infinitely-wide neural networks.

Amazingly, this model family admits tractable calculations for many information-theoretic quantities.

We report analytical and empirical investigations in the search for signals that correlate with generalization.

Infinite ensembles of infinitely wide neural networks are an interesting model family from an information theoretic perspective.

Information in Infinite Ensembles of Infinitely-Wide Neural Networks

infinite ensemble
information
neural network


Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks.

There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns unified multilingual representations using monolingual and cross-lingual objectives jointly.

In this paper, we first conduct direct comparisons of representations learned using both of these methods across diverse cross-lingual tasks.

Our empirical results reveal a set of pros and cons for both methods, and show that the relative performance of alignment versus joint training is task-dependent.

Stemming from this analysis, we propose a simple and novel framework that combines these two previously mutually-exclusive approaches.

Extensive experiments on various tasks demonstrate that our proposed framework alleviates limitations of both approaches, and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark.

We further show that our proposed framework can generalize to contextualized representations and achieves state-of-the-art results on the CoNLL cross-lingual NER benchmark.

We conduct a comparative study of cross-lingual alignment vs joint training methods and unify these two previously exclusive paradigms in a new framework.

This paper compares approaches to bilingual lexicon induction and shows which method performs better on lexicon, induction, and NER and MT tasks.

Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework


Large number of weights in deep neural networks make the models difficult to be deployed in low memory environments such as, mobile phones, IOT edge devices as well as "inferencing as a service" environments on the cloud.

Prior work has considered reduction in the size of the models, through compression techniques like weight pruning, filter pruning, etc. or through low-rank decomposition of the convolution layers.

In this paper, we demonstrate the use of multiple techniques to achieve not only higher model compression but also reduce the compute resources required during inferencing.

We do filter pruning followed by low-rank decomposition using Tucker decomposition for model compression.

We show that our approach achieves upto 57\% higher model compression when compared to either Tucker Decomposition or Filter pruning alone  at similar accuracy for GoogleNet.

Also, it reduces the Flops by upto 48\% thereby making the inferencing faster.

Combining orthogonal model compression techniques to get significant reduction in model size and number of flops required during inferencing.

This paper proposes combining Tucker Decomposition with Filter pruning.

Compression of Deep Neural Networks by combining pruning and low rank decomposition

pruning combine compression decomposition


We review the limitations of BLEU and ROUGE -- the most popular metrics used to assess reference summaries against hypothesis summaries, and introduce JAUNE:  a set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries against hypothesis summaries.

Introduces JAUNE: a methodology to replace BLEU and ROUGE score with multidimensional, model-based evaluators for assessing summaries

This paper proposes a new JAUNE metric for the evaluation of machine translation and text summarization systems, showing that their model corresponds better to ground truth similarity labels than BLEU.

JAUNE: Justified And Unified Neural language Evaluation


This paper presents a new Graph Neural Network (GNN) type using feature-wise linear modulation (FiLM).

Many standard GNN variants propagate information along the edges of a graph by computing ``messages'' based only on the representation of the source of each edge.

In GNN-FiLM, the representation of the target node of an edge is additionally used to compute a transformation that can be applied to all incoming messages, allowing feature-wise modulation of the passed information.

Results of experiments comparing different GNN architectures on three tasks from the literature are presented, based on re-implementations of baseline methods.

Hyperparameters for all methods were found using extensive search, yielding somewhat surprising results: differences between baseline models are smaller than reported in the literature.

Nonetheless, GNN-FiLM outperforms baseline methods on a regression task on molecular graphs and performs competitively on other tasks.

new GNN formalism + extensive experiments; showing differences between GGNN/GCN/GAT are smaller than thought

The paper proposes a new Graph Neural Network architecture that uses Feature-wise Linear Modulation to condition the source-to-target node message-passing based on the target node representation.

GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation

feature-wise linear modulation
graph neural network
new


To deal simultaneously with both, the attributed network embedding and clustering, we propose a new model.

It exploits both content and structure information, capitalising on their simultaneous use.

The proposed model relies on the approximation of the relaxed continuous embedding solution by the true discrete clustering one.

Thereby, we show that incorporating an embedding representation provides simpler and more interpretable solutions.

Experiment results demonstrate that the proposed algorithm performs better, in terms of clustering and embedding, than the state-of-art algorithms, including deep learning methods devoted to similar tasks for attributed network datasets with different proprieties.

This paper propose a novel matrix decomposition framework for simultaneous attributed network data embedding and clustering.

This paper proposes an algorithm to perform jointly attribute network embedding and clustering together.

SIMULTANEOUS ATTRIBUTED NETWORK EMBEDDING AND CLUSTERING

attribute network embed cluster
simultaneous


We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis.

The goal of our method is to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications (e.g., virtual showrooms, virtual tours and sightseeing, the digital inspection of historical artifacts).

A core component of our work is the handling of view-dependent effects.

Specifically, we directly train an object-specific deep neural network to synthesize the view-dependent appearance of an object.

As input data we are using an RGB video of the object.

This video is used to reconstruct a proxy geometry of the object via multi-view stereo.

Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.

This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts.

To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects.

Based on these estimations, we are able to convert observed images to diffuse images.

These diffuse images can be projected into other views.

In the target view, our pipeline reinserts the new view-dependent effects.

To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realistic results.

Using this image-guided approach, the network does not have to allocate capacity on ``remembering'' object appearance, instead it learns how to combine the appearance of captured images.

We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data.

We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis while considering view-dependent effects.

This submission proposes a method to handle view-dependent effects in neural rendering, which improves the robustness of existing neural rendering methods.

Image-guided Neural Object Rendering


We evaluate the distribution learning capabilities of generative adversarial networks by testing them on synthetic datasets.

The datasets include common distributions of points in $R^n$ space and images containing polygons of various shapes and sizes.

We find that by and large GANs fail to faithfully recreate point datasets which contain discontinous support or sharp bends with noise.

Additionally, on image datasets, we find that GANs do not seem to learn to count the number of objects of the same kind in an image.

We also highlight the apparent tension between generalization and learning in GANs.

GANs are evaluated on synthetic datasets

Evaluating the distribution learning capabilities of GANs


This paper proposes a new approach for step size adaptation in gradient methods.

The proposed method called step size optimization (SSO) formulates the step size adaptation as an optimization problem which minimizes the loss function with respect to the step size for the given model parameters and gradients.

Then, the step size is optimized based on alternating direction method of multipliers (ADMM).

SSO does not require the second-order information or any probabilistic models for adapting the step size, so it is efficient and easy to implement.

Furthermore, we also introduce stochastic SSO for stochastic learning environments.

In the experiments, we integrated SSO to vanilla SGD and Adam, and they outperformed state-of-the-art adaptive gradient methods including RMSProp, Adam, L4-Adam, and AdaBound on extensive benchmark datasets.

We propose an efficient and effective step size adaptation method for the gradient methods.

A new step size adaptation in first-order gradient methods that establishes a new optimization problem with the first-order expansion of the loss function and regularization, where step size is treated as variable.

Step Size Optimization

step size adaptation
gradient method
optimization


Despite the fact that generative models are extremely successful in practice, the theory underlying this phenomenon is only starting to catch up with practice.

In this work we address the question of the universality of generative models: is it true that neural networks can approximate any data manifold arbitrarily well?

We provide a positive answer to this question and show that under mild assumptions on the activation function one can always find a feedforward neural network that maps the latent space onto a set located within the specified Hausdorff distance from the desired data manifold.

We also prove similar theorems for the case of multiclass generative models and cycle generative models, trained to map samples from one manifold to another and vice versa.

We shot that a wide class of manifolds can be generated by ReLU and sigmoid networks with arbitrary precision.

This paper provides certain basic guarantees on when manifolds can be written as the image of a map approximated by a neural net, and stitches together theorems from manifold geometry and standard universal approximation results.

This paper theoretically shows that neural-network-based generative models can approximate data manifolds, and proves that under mild assumptions neural networks can map a latent space onto a set close to the given data manifold within a small Hausdorff distance.

Universality Theorems for Generative Models

manifold
map
generative model
network
approximate
theorem
neural


Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL.

However, the theoretical understanding of such methods has been rather limited.

This paper introduces a novel algorithmic framework for designing and analyzing model-based RL algorithms with theoretical guarantees.

We design a meta-algorithm with a theoretical guarantee of monotone improvement to a local maximum of the expected reward.

The meta-algorithm iteratively builds a lower bound of the expected reward based on the estimated dynamical model and sample trajectories, and then maximizes the lower bound jointly over the policy and the model.

The framework extends the optimism-in-face-of-uncertainty principle to non-linear dynamical models in a way that requires no explicit uncertainty quantification.

Instantiating our framework with simplification gives a  variant of model-based RL algorithms Stochastic Lower Bounds Optimization (SLBO).

Experiments demonstrate that SLBO achieves the state-of-the-art performance when only 1M or fewer samples are permitted on a range of continuous control benchmark tasks.

We design model-based reinforcement learning algorithms with theoretical guarantees and achieve state-of-the-art results on Mujuco benchmark tasks when one million or fewer samples are permitted.

The paper proposed a framework to design model-based RL algorithms based on OFU that achieves SOTA performance on MuJoCo tasks.

Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees

design model-based
theoretical guarantee
achieve
task
reinforcement learning
framework
algorithm


We study the use of knowledge distillation to compress the U-net architecture.

We show that, while standard distillation is not sufficient to reliably train a compressed U-net, introducing other regularization methods, such as batch normalization and class re-weighting, in knowledge distillation significantly improves the training process.

This allows us to compress a U-net by over 1000x, i.e., to 0.1% of its original number of parameters, at a negligible decrease in performance.

We present additional techniques to use knowledge distillation to compress U-net by over 1000x.

The authors introduced a modified distillation strategy to compress a U-net architecture by over 1000x while retaining an accuracy close to the original U-net.

On Compressing U-net Using Knowledge Distillation

compress u-net
1000x
knowledge distillation


Learning neural networks with gradient descent over a long sequence of tasks is problematic as their fine-tuning to new tasks overwrites the network weights that are important for previous tasks.

This leads to a poor performance on old tasks – a phenomenon framed as catastrophic forgetting.

While early approaches use task rehearsal and growing networks that both limit the scalability of the task sequence orthogonal approaches build on regularization.

Based on the Fisher information matrix (FIM) changes to parameters that are relevant to old tasks are penalized, which forces the task to be mapped into the available remaining capacity of the network.

This requires to calculate the Hessian around a mode, which makes learning tractable.

In this paper, we introduce Hessian-free curvature estimates as an alternative method to actually calculating the Hessian.

In contrast to previous work, we exploit the fact that most regions in the loss surface are flat and hence only calculate a Hessian-vector-product around the surface that is relevant for the current task.

Our experiments show that on a variety of well-known task sequences we either significantly outperform or are en par with previous work.

This paper provides an approach to address catastrophic forgetting via Hessian-free curvature estimates

The paper proposes an approximate Laplace's method in neural network training in the continual learning setting with a low space complexity.

Overcoming Catastrophic Forgetting via Hessian-free Curvature Estimates

catastrophic forgetting via hessian-free curvature estimate


There has been recent interest in improving performance of simple models for multiple reasons such as interpretability, robust learning from small data, deployment in memory constrained settings as well as environmental considerations.

In this paper, we propose a novel method SRatio that can utilize information from high performing complex models (viz. deep neural networks, boosted trees, random forests) to reweight a training dataset for a potentially low performing simple model such as a decision tree or a shallow network enhancing its performance.

Our method also leverages the per sample hardness estimate of the simple model which is not the case with the prior works which primarily consider the complex model's confidences/predictions and is thus conceptually novel.

Moreover, we generalize and formalize the concept of attaching probes to intermediate layers of a neural network, which was one of the main ideas in previous work \citep{profweight}, to other commonly used classifiers and incorporate this into our method.

The benefit of these contributions is witnessed in the experiments where on 6 UCI datasets and CIFAR-10 we outperform competitors in a majority (16 out of 27) of the cases and tie for best performance in the remaining cases.

In fact, in a couple of cases, we even approach the complex model's performance.

We also conduct further experiments to validate assertions and intuitively understand why our method works.

Theoretically, we motivate our approach by showing that the weighted loss minimized by simple models using our weighting upper bounds the loss of the complex model.

Method to improve simple models performance given a (accurate) complex model.

The paper proposes a means of improving the predictions of a low-capacity model which shows benefits over existing approaches.

Leveraging Simple Model Predictions for Enhancing its Performance

model improve prediction performance simple


We propose a principled method for kernel learning, which relies on a Fourier-analytic characterization of translation-invariant or rotation-invariant kernels.

Our method produces a sequence of feature maps, iteratively refining the SVM margin.

We provide rigorous guarantees for optimality and generalization, interpreting our algorithm as online equilibrium-finding dynamics in a certain two-player min-max game.

Evaluations on synthetic and real-world datasets demonstrate scalability and consistent improvements over related random features-based methods.

A simple and practical algorithm for learning a margin-maximizing translation-invariant or spherically symmetric kernel from training data, using tools from Fourier analysis and regret minimization.

The paper proposes to learn a custom translation or rotation invariant kernal in the Fourier representation to maximize the margin of SVM.

The authors propose an interesting algorithm for learning the l1-SVM and the Fourier represented kernel together

The authors consider learning directly Fourier representations of shift/translation invariant kernels for machine learning applications with the alignment of the kernel to data as the objective function to optimize.

Not-So-Random Features

algorithm learn
datum
fouri representation
kernel
author
invariant
translation


We elaborate on using importance sampling for causal reasoning, in particular for counterfactual inference.

We show how this can be implemented natively in probabilistic programming.

By considering the structure of the counterfactual query, one can significantly optimise the inference process.

We also consider design choices to enable further optimisations.

We introduce MultiVerse, a probabilistic programming prototype engine for approximate causal reasoning.

We provide experimental results and compare with Pyro, an existing probabilistic programming framework with some of causal reasoning tools.

Probabilistic Programming that Natively Supports Causal, Counterfactual Inference

MultiVerse: Causal Reasoning using Importance Sampling in Probabilistic Programming


We consider the problem of representing collective behavior of large populations and predicting the evolution of a population distribution over a discrete state space.

A discrete time mean field game (MFG) is motivated as an interpretable model founded on game theory for understanding the aggregate effect of individual actions and predicting the temporal evolution of population distributions.

We achieve a synthesis of MFG and Markov decision processes (MDP) by showing that a special MFG is reducible to an MDP.

This enables us to broaden the scope of mean field game theory and infer MFG models of large real-world systems via deep inverse reinforcement learning.

Our method learns both the reward function and forward dynamics of an MFG from real data, and we report the first empirical test of a mean field game model of a real-world social media population.

Inference of a mean field game (MFG) model of large population behavior via a synthesis of MFG and Markov decision processes.

The authors deal with inference in models of collective behavior by using inverse reinforcement learning to learn the reward functions of agents in the model.

Learning Deep Mean Field Games for Modeling Large Population Behavior

model large population behavior
mean field game
inference
learn


We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as  offensive basketball gameplay.

When modeling such settings, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables.

Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulable way.

We present a hierarchical framework that can effectively learn such sequential generative models.

Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime.

In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods.

We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts.

We blend deep generative models with programmatic weak supervision to generate coordinated multi-agent trajectories of significantly higher quality than previous baselines.

Proposes multi-agent sequential generative models.

The paper proposes training generative models that produce multi-agent trajectories using heuristic functions that label variables that would otherwise be latent in training data

Generating Multi-Agent Trajectories using Programmatic Weak Supervision

programmatic weak supervision
multi-agent trajectory
generative model
generate


Many automated machine learning methods, such as those for hyperparameter and neural architecture optimization, are computationally expensive because they involve training many different model configurations.

In this work, we present a new method that saves computational budget by terminating poor configurations early on in the training.

In contrast to existing methods, we consider this task as a ranking and transfer learning problem.

We qualitatively show that by optimizing a pairwise ranking loss and leveraging learning curves from other data sets, our model is able to effectively rank learning curves without having to observe many or very long learning curves.

We further demonstrate that our method can be used to accelerate a neural architecture search by a factor of up to 100 without a significant performance degradation of the discovered architecture.

In further experiments we analyze the quality of ranking, the influence of different model components as well as the predictive behavior of the model.

Learn to rank learning curves in order to stop unpromising training jobs early. Novelty: use of pairwise ranking loss to directly model the probability of improving and transfer learning across data sets to reduce required training data.

The paper proposes a method to rank learning curves of neural networks that can model learning curves across different datasets, achieving higher speed-ups on image classification tasks.

Learning to Rank Learning Curves


Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster.

Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distributions.

This phenomenon, referred to as catastrophic forgetting, is considered a major hurdle to learning with non-stationary data or sequences of new tasks, and prevents networks from continually accumulating knowledge and skills.

We examine this issue in the context of reinforcement learning, in a setting where an agent is exposed to tasks in a sequence.

Unlike most other work, we do not provide an explicit indication to the model of task boundaries, which is the most general circumstance for a learning agent exposed to continuous experience.

While various methods to counteract catastrophic forgetting have recently been proposed, we explore a straightforward, general, and seemingly overlooked solution - that of using experience replay buffers for all past events - with a mixture of on- and off-policy learning, leveraging behavioral cloning.

We show that this strategy can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities.

When buffer storage is constrained, we confirm that a simple mechanism for randomly discarding data allows a limited size buffer to perform almost as well as an unbounded one.

We show that, in continual learning settings, catastrophic forgetting can be avoided by applying off-policy RL to a mixture of new and replay experience, with a behavioral cloning loss.

Proposes a particular variant of experience replay with behavior cloning as a method for continual learning.

Experience replay for continual learning

continual learning
experience replay
cloning


We present a method which learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents.

Our method is based on a graph-structured variational recurrent neural network, which is trained end-to-end to infer the current state of the (partially observed) world, as well as to forecast future states.

We show that our method outperforms various baselines on two sports datasets, one based on real basketball trajectories, and one generated by a soccer game engine.

We present a method which learns to integrate temporal information and ambiguous visual information in the context of interacting agents.

The authors propose Graph VRNN which models the interaction of multiple agents by deploying a VRNN for each agent

This paper presents a graph neural network based architecture that is trained to locate and model the interactions of agents in an environment directly from pixels and show advantage of model for tracking tasks and forecasting agent locations.

Stochastic Prediction of Multi-Agent Interactions from Partial Observations


In this paper we study the problem of learning the weights of a deep convolutional neural network.

We consider a network where convolutions are carried out over non-overlapping patches with a single kernel in each layer.

We develop an algorithm for simultaneously learning all the kernels from the training data.

Our approach dubbed Deep Tensor Decomposition (DeepTD) is based on a rank-1 tensor decomposition.

We theoretically investigate DeepTD under a realizable model for the training data where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to planted convolutional kernels.

We show that DeepTD is data-efficient and provably works as soon as the sample size exceeds the total number of convolutional weights in the network.

Our numerical experiments demonstrate the effectiveness of DeepTD and verify our theoretical findings.

We consider a simplified deep convolutional neural network model. We show that all layers of this network can be approximately learned with a proper application of tensor decomposition.

Provides theoretical guarantees for learning deep convolutional neural networks using rank-one tensor decomposition.

This paper proposes a learning method for a restricted case of deep convolutional networks, where the layers are limited to the non-overlapping case and have only one output channel per layer

Analyzes the problem of learning a very special class of CNNs: each layers consists of a single filter, applied to non-overlapping patches of the input.

End-to-end Learning of a Convolutional Neural Network via Deep Tensor Decomposition

layer
deep convolutional neural network
learn
tensor decomposition
learning
non-overlapping


Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.

However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.

We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively.

Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations.

The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.

We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations.

We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10.

Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

Feedforward neural networks that can have weights pruned after training could have had the same weights pruned before training

Shows that there exists sparse subnetworks that can be trained from scratch with good generalization performance and proposes a unpruned, randomly initialized NNs contain subnetworks that can be trained from scratch with similar generalization accuracy.

The paper examines the hypothesis that randomly initialized neural networks contain sub-networks that converge equally fast or faster and can reach the same or better classification accuracy

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

randomly initialize
neural network
train
hypothesis
accuracy
sparse
contain


We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime.

Recent work of \citep{Gale2019, Liu2018} has shown that sparse ResNet-50 architectures trained on ImageNet-2012 dataset converge to solutions that are significantly worse than those found by pruning.

We show that, despite the failure of optimizers, there is a linear path with a monotonically decreasing objective from the initialization to the ``good'' solution.

Additionally, our attempts to find a decreasing objective path from ``bad'' solutions to the ``good'' ones in the sparse subspace fail.

However, if we allow the path to traverse the dense subspace, then we consistently find a path between two solutions.

These findings suggest traversing extra dimensions may be needed to escape stationary points found in the sparse subspace.

In this paper we highlight  the difficulty of training sparse neural networks by doing interpolation experiments in the energy landscape

The Difficulty of Training Sparse Neural Networks


Neural network training depends on the structure of the underlying loss landscape, i.e. local minima, saddle points, flat plateaus, and loss barriers.

In relation to the structure of the landscape, we study the permutation symmetry of neurons in each layer of a deep neural network, which gives rise not only to multiple equivalent global minima of the loss function but also to critical points in between partner minima.

In a network of $d-1$ hidden layers with $n_k$ neurons in layers $k = 1, \ldots, d$, we construct continuous paths between equivalent global minima that lead through a `permutation point' where the input and output weight vectors of two neurons in the same hidden layer $k$ collide and interchange.

We show that such permutation points are critical points which lie inside high-dimensional subspaces of equal loss, contributing to the global flatness of the landscape.

We also find that a permutation point for the exchange of neurons $i$ and $j$ transits into a flat high-dimensional plateau that enables all $n_k!$

permutations of neurons in a given layer $k$ at the same loss value. Moreover

, we introduce higher-order permutation points by exploiting the hierarchical structure in the loss landscapes of neural networks, and find that the number of $K$-th order permutation points is much larger than the (already huge) number of equivalent global minima -- at least by a polynomial factor of order $K$. In two

tasks, we demonstrate numerically with our path finding method that continuous paths between partner minima exist: first, in a toy network with a single hidden layer on a function approximation task and, second, in a multilayer network on the MNIST task. Our geometric

approach yields a lower bound on the number of critical points generated by weight-space symmetries and provides a simple intuitive link between previous theoretical results and numerical observations.

Weight-space symmetry in neural network landscapes gives rise to numerous number of saddles and flat high-dimensional subspaces.

The paper presented a low-loss method for studying the loss function with respect to parameters in a neural network from the perspective of weight-space symmetry.

Weight-space symmetry in neural network loss landscapes revisited

weight-space symmetry neural network
landscape
loss


The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated.

We derive, using mean field theory, a set of scalar equations describing how input signals propagate through surrogate networks.

The equations reveal that depending on the choice of surrogate model, the networks may or may not exhibit an order to chaos transition, and the presence of depth scales that limit the maximum trainable depth.

Specifically, in solving the equations for edge of chaos conditions, we show that surrogates derived using the Gaussian local reparameterisation trick have no critical initialisation, whereas a deterministic surrogates based on analytic Gaussian integration do.

The theory is applied to a range of binary neuron and weight design choices, such as different neuron noise models, allowing the categorisation of algorithms in terms of their behaviour at initialisation.

Moreover, we predict theoretically and confirm numerically, that common weight initialization schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance.

This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to close to $\pm 1$ for deeper networks to be trainable.

signal propagation theory applied to continuous surrogates of binary nets;  counter intuitive initialisation; reparameterisation trick not helpful

The authors investigate the training dynamics of binary neural networks when using continuous surrogates, study what properties networks should have at initialization to best train, and provide concrete advice about stochastic weights at initialization.

An in-depth exploration of stochastic binary networks, continuous surrogates, and their training dynamics, with insights on how to initialize weights for best performance.

Critical initialisation in continuous approximations of binary neural networks

network continuous surrogate
training dynamic
binary neural
good
weight
stochastic
initialisation


Semantic dependency parsing, which aims to find rich bi-lexical relationships, allows words to have multiple dependency heads, resulting in graph-structured representations.

We propose an approach to semi-supervised learning of semantic dependency parsers based on the CRF autoencoder framework.

Our encoder is a discriminative neural semantic dependency parser that predicts the latent parse graph of the input sentence.

Our decoder is a generative neural model that reconstructs the input sentence conditioned on the latent parse graph.

Our model is arc-factored and therefore parsing and learning are both tractable.

Experiments show our model achieves significant and consistent improvement over the supervised baseline.

This paper focuses on semi-supervised semantic dependency parsing using the CRF-autoencoder to train the model in a semi-supervised style, indicating effectiveness on low resource labeled data tasks.

Semi-Supervised Semantic Dependency Parsing Using CRF Autoencoders

semi-supervised semantic dependency parse


For sequence models with large word-level vocabularies, a majority of network parameters lie in the input and output layers.

In this work, we describe a new method, DeFINE, for learning deep word-level representations efficiently.

Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods.

DeFINE can be incorporated easily in new or existing sequence models.

Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity.

On WikiText-103, DeFINE reduces total parameters of Transformer-XL by half with minimal impact on performance.

On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters,  achieving comparable performance to state-of-the-art methods with fewer parameters.

For machine translation, DeFINE improves a Transformer model by 2% while simultaneously reducing total parameters by 26%

DeFINE uses a deep, hierarchical, sparse network with new skip connections to learn better word embeddings efficiently.

This paper describes a new method for learning deep word-level representations efficiently by using a hierarchical structure with skip-connections for the use of low dimensional input and output layers.

DeFINE: Deep Factorized Input Word Embeddings for Neural Sequence Modeling

hierarchical
deep
learn
define
new
efficiently
word embedding
input


In this paper, we present a reproduction of the paper of Bertinetto et al. [2019] "Meta-learning with differentiable closed-form solvers" as part of the ICLR 2019 Reproducibility Challenge.

In successfully reproducing the most crucial part of the paper, we reach a performance that is comparable with or superior to the original paper on two benchmarks for several settings.

We evaluate new baseline results, using a new dataset presented in the paper.

Yet, we also provide multiple remarks and recommendations about reproducibility and comparability.

After we brought our reproducibility work to the authors’ attention, they have updated the original paper on which this work is based and released code as well.

Our contributions mainly consist in reproducing the most important results of their original paper, in giving insight in the reproducibility and in providing a first open-source implementation.

We successfully reproduce and give remarks on the comparison with baselines of a meta-learning approach for few-shot classification that works by backpropagating through the solution of a closed-form solver.

Reproducing Meta-learning with differentiable closed-form solvers


Network pruning has emerged as a powerful technique for reducing the size of deep neural networks.

Pruning uncovers high-performance subnetworks by taking a trained dense network and gradually removing unimportant connections.

Recently, alternative techniques have emerged for training sparse networks directly without having to train a large dense model beforehand, thereby achieving small memory footprints during both training and inference.These techniques are based on dynamic reallocation of non-zero parameters during training.

Thus, they are in effect executing a training-time search for the optimal subnetwork.

We investigate a most recent one of these techniques and conduct additional experiments to elucidate its behavior in training sparse deep convolutional networks.

Dynamic parameter reallocation converges early during training to a highly trainable subnetwork.

We show that neither the structure, nor the initialization of the discovered high-performance subnetwork is sufficient to explain its good performance.

Rather, it is the dynamics of parameter reallocation that are responsible for successful learning.

Dynamic parameter reallocation thus improves the trainability of deep convolutional networks, playing a similar role as overparameterization, without incurring the memory and computational cost of the latter.

Dynamic parameter-reallocation enables the successful direct training of compact sparse networks, and it plays an indispensable role even when we know the optimal sparse network a-priori

Dynamic parameter reallocation improves trainability of deep convolutional networks


n this paper we present a thrust in three directions of visual development us- ing supervised and semi-supervised techniques.

The first is an implementation of semi-supervised object detection and recognition using the principles of Soft At- tention and Generative Adversarial Networks (GANs).

The second and the third are supervised networks that learn basic concepts of spatial locality and quantity respectively using Convolutional Neural Networks (CNNs).

The three thrusts to- gether are based on the approach of Experiential Robot Learning, introduced in previous publication.

While the results are unripe for implementation, we believe they constitute a stepping stone towards autonomous development of robotic vi- sual modules.

3 thrusts serving as stepping stones for robot experiential learning of vision module

Investigates is performance of existing image classifiers and object detectors.

TOWARDS ROBOT VISION MODULE DEVELOPMENT WITH EXPERIENTIAL ROBOT LEARNING

vision module
robot
experiential
learning


Characterization of the representations learned in intermediate layers of deep networks can provide valuable insight into the nature of a task and can guide the development of well-tailored learning strategies.

Here we study convolutional neural network-based acoustic models in the context of automatic speech recognition.

Adapting a method proposed by Yosinski et al. [2014], we measure the transferability of each layer between German and English to assess the their language-specifity.

We observe three distinct regions of transferability: (1) the first two layers are entirely transferable between languages, (2) layers 2–8 are also highly transferable but we find evidence of some language specificity, (3) the subsequent fully connected layers are more language specific but can be successfully finetuned to the target language.

To further probe the effect of weight freezing, we performed follow-up experiments using freeze-training [Raghu et al., 2017].

Our results are consistent with the observation that CCNs converge 'bottom up' during training and demonstrate the benefit of freeze training, especially for transfer learning.

All but the first two layers of our CNNs based acoustic models demonstrated some degree of language-specificity but freeze training enabled successful transfer between languages.

The paper measures the transferability of features for each layer in CNN-based acoustic models across languages, concluding that AMs trained with 'the freeze training' technique outperformed other transferred models.

How transferable are features in convolutional neural network acoustic models across languages?

acoustic model across language
layer
freeze training
feature
transfer


Policy gradients methods often achieve better performance when the change in policy is limited to a small Kullback-Leibler divergence.

We derive policy gradients where the change in policy is limited to a small Wasserstein distance (or trust region).

This is done in the discrete and continuous multi-armed bandit settings with entropy regularisation.

We show that in the small steps limit with respect to the Wasserstein distance $W_2$, policy dynamics are governed by the heat equation, following the Jordan-Kinderlehrer-Otto result.

This means that policies undergo diffusion and advection, concentrating near actions with high reward.

This helps elucidate the nature of convergence in the probability matching setup, and provides justification for empirical practices such as Gaussian policy priors and additive gradient noise.

Linking Wasserstein-trust region entropic policy gradients, and the heat equation.

The paper explores the connections between reinforcement learning and the theory of quadratic optimal transport

The authors studied policy gradient with change of policies limited by a trust region of Wasserstein distance in the multi-armed bandit setting, showing that in the small steps limit, the policy dynamics are governed by the heat equation (Fokker-Planck equation).

Diffusing Policies : Towards Wasserstein Policy Gradient Flows

heat equation
policy gradient
region
wasserstein


The softmax function is widely used to train deep neural networks for multi-class classification.

Despite its outstanding performance in classification tasks, the features derived from the supervision of softmax are usually sub-optimal in some scenarios where Euclidean distances apply in feature spaces.

To address this issue, we propose a new loss, dubbed the isotropic loss, in the sense that the overall distribution of data points is regularized to approach the isotropic normal one.

Combined with the vanilla softmax, we formalize a novel criterion called the isotropic softmax, or isomax for short, for supervised learning of deep neural networks.

By virtue of the isomax, the intra-class features are penalized by the isotropic loss while inter-class distances are well kept by the original softmax loss.

Moreover, the isomax loss does not require any additional modifications to the network, mini-batches or the training process.

Extensive experiments on classification and clustering are performed to demonstrate the superiority and robustness of the isomax loss.

The discriminative capability of softmax for learning feature vectors of objects is effectively enhanced by virture of isotropic normalization on global distribution of data points.

Softmax Supervision with Isotropic Normalization


A fundamental question in reinforcement learning is whether model-free algorithms are sample efficient.

Recently,  Jin et al. (2018) proposed a Q-learning algorithm with UCB exploration policy, and proved it has nearly optimal regret bound for finite-horizon episodic MDP.

In this paper, we adapt Q-learning with UCB-exploration bonus to infinite-horizon MDP with discounted rewards \emph{without} accessing a generative model.

We show that the \textit{sample complexity of exploration} of our algorithm is bounded by $\tilde{O}({\frac{SA}{\epsilon^2(1-\gamma)^7}})$.

This improves the previously best known result of $\tilde{O}({\frac{SA}{\epsilon^4(1-\gamma)^8}})$ in this setting achieved by delayed Q-learning (Strehlet al., 2006),, and matches the lower bound in terms of $\epsilon$ as well as $S$ and $A$ up to logarithmic factors.

We adapt Q-learning with UCB-exploration bonus to infinite-horizon MDP with discounted rewards without accessing a generative model, and improves the previously best known result.

This paper considered a Q-learning algorithm with UCB exploration policy for infinite-horizon MDP.

Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDP

infinite-horizon mdp
ucb exploration
q-learne


Backpropagation is driving today's artificial neural networks (ANNs).

However, despite extensive research, it remains unclear if the brain implements this algorithm.

Among neuroscientists, reinforcement learning (RL) algorithms are often seen as a realistic alternative: neurons can randomly introduce change, and use unspecific feedback signals to observe their effect on the cost and thus approximate their gradient.

However, the convergence rate of such learning scales poorly with the number of involved neurons.

Here we propose a hybrid learning approach.

Each neuron uses an RL-type strategy to learn how to approximate the gradients that backpropagation would provide.

We provide proof that our approach converges to the true gradient for certain classes of networks.

In both feedforward and convolutional networks, we empirically show that our approach learns to approximate the gradient, and can match the performance of gradient-based learning.

Learning feedback weights provides a biologically plausible mechanism of achieving good performance, without the need for precise, pre-specified learning rules.

Perturbations can be used to train feedback weights to learn in fully connected and convolutional neural networks

This paper proposes a method that addresses the "weight transport" problem by estimating the weights for the backward pass using a noise-based estimator

Learning to solve the credit assignment problem


This paper proposes and demonstrates a surprising pattern in the training of neural networks: there is a one to one relation between the values of any pair of losses (such as cross entropy, mean squared error, 0/1 error etc.) evaluated for a model arising at (any point of) a training run.

This pattern is universal in the sense that this one to one relationship is identical across architectures (such as VGG, Resnet, Densenet etc.), algorithms (SGD and SGD with momentum) and training loss functions (cross entropy and mean squared error).

We identify some universal patterns (i.e., holding across architectures) in the behavior of different surrogate losses (CE, MSE, 0-1 loss) while training neural networks and present supporting empirical evidence.

Universality Patterns in the Training of Neural Networks