This paper presents two methods to disentangle and interpret contextual effects that are encoded in a pre-trained deep neural network.

Unlike convolutional studies that visualize image appearances corresponding to the network output or a neural activation from a global perspective, our research aims to clarify how a certain input unit (dimension) collaborates with other units (dimensions) to constitute inference patterns of the neural network and thus contribute to the network output.

The analysis of local contextual effects w.r.t. certain input units is of special values in real applications.

In particular, we used our methods to explain the gaming strategy of the alphaGo Zero model in experiments, and our method successfully disentangled the rationale of each move during the game.

This paper presents methods to disentangle and interpret contextual effects that are encoded in a deep neural network.

Explaining AlphaGo: Interpreting Contextual Effects in Neural Networks

interpret contextual effect
neural network


The main goal of network pruning is imposing sparsity on the neural network by increasing the number of parameters with zero value in order to reduce the architecture size and the computational speedup.

Proposing a novel method based on the guided attention to enforce the sparisty in deep neural networks.

Sparsity Learning in Deep Neural Networks


Giving provable guarantees for learning neural networks is a core challenge of machine learning theory.

Most prior work gives parameter recovery guarantees for one hidden layer networks, however, the networks used in practice have multiple non-linear layers.

In this work, we show how we can strengthen such results to deeper networks -- we address the problem of uncovering the lowest layer in a deep neural network under the assumption that the lowest layer uses a high threshold before applying the activation, the upper network can be modeled as a well-behaved polynomial and the input distribution is gaussian.

We provably recover the lowest layer in a deep neural network assuming that the lowest layer uses a "high threshold" activation and the above network is a "well-behaved" polynomial.

Recovering the Lowest Layer of Deep Networks with High Threshold Activations

recover low layer
high threshold
deep
network
activation


Federated learning involves training and effectively combining machine learning models from distributed partitions of data (i.e., tasks) on edge devices, and be naturally viewed as a multi- task learning problem.

While Federated Averaging (FedAvg) is the leading optimization method for training non-convex models in this setting, its behavior is not well understood in realistic federated settings when the devices/tasks are statistically heterogeneous, i.e., where each device collects data in a non-identical fashion.

In this work, we introduce a framework, called FedProx, to tackle statistical heterogeneity.

FedProx encompasses FedAvg as a special case.

We provide convergence guarantees for FedProx through a device dissimilarity assumption.

Our empirical evaluation validates our theoretical analysis and demonstrates the improved robustness and stability of FedProx for learning in heterogeneous networks.

We introduce FedProx, a framework to tackle statistical heterogeneity in federated settings with convergence guarantees and improved robustness and stability.

Federated Optimization for Heterogeneous Networks


Referential games offer a grounded learning environment for neural agents which accounts for the fact that language is functionally used to communicate.

However, they do not take into account a second constraint considered to be fundamental for the shape of human language: that it must be learnable by new language learners and thus has to overcome a transmission bottleneck.

In this work, we insert such a bottleneck in a referential game, by introducing a changing population of agents in which new agents learn by playing with more experienced agents.

We show that mere cultural transmission results in a substantial improvement in language efficiency and communicative success, measured in convergence speed, degree of structure in the emerged languages and within-population consistency of the language.

However, as our core contribution, we show that the optimal situation is to co-evolve language and agents.

When we allow the agent population to evolve through genotypical evolution, we achieve across the board improvements on all considered metrics.

These results stress that for language emergence studies cultural evolution is important, but also the suitability of the architecture itself should be considered.

We enable both the cultural evolution of language and the genetic evolution of agents in a referential game, using a new Language Transmission Engine.

Co-evolution of language and agents in referential games


The need for large amounts of training image data with clearly defined features is a major obstacle to applying generative adversarial networks(GAN) on image generation where training data is limited but diverse, since insufficient latent feature representation in the already scarce data often leads to instability and mode collapse during GAN training.

To overcome the hurdle of limited data when applying GAN to limited datasets, we propose in this paper the strategy of \textit{parallel recurrent data augmentation}, where the GAN model progressively enriches its training set with sample images constructed from GANs trained in parallel at consecutive training epochs.

Experiments on a variety of small yet diverse datasets demonstrate that our method, with little model-specific considerations, produces images of better quality as compared to the images generated  without such strategy.

The source code and generated images of this paper will be made public after review.

We introduced a novel, simple, and efficient data augmentation method that boosts the performances of existing GANs when training data is limited and diverse.

Parallel Recurrent Data Augmentation for GAN training with Limited and Diverse Data

limited diverse
gan training
datum augmentation


We develop a stochastic whole-brain and body simulator of the nematode roundworm Caenorhabditis elegans (C. elegans) and show that it is sufficiently regularizing to allow imputation of latent membrane potentials from partial calcium fluorescence imaging observations.

This is the first attempt we know of to ``complete the circle,'' where an anatomically grounded whole-connectome simulator is used to impute a time-varying ``brain'' state at single-cell fidelity from covariates that are measurable in practice.

Using state of the art Bayesian machine learning methods to condition on readily obtainable data, our method paves the way for neuroscientists to recover interpretable connectome-wide state representations, automatically estimate physiologically relevant parameter values from data, and perform simulations investigating intelligent lifeforms in silico.

We develop a whole-connectome and body simulator for C. elegans and demonstrate joint state-space and parameter inference in the simulator.

The Virtual Patch Clamp: Imputing C. elegans Membrane Potentials from Calcium Imaging


The high-quality node embeddings learned from the Graph Neural Networks (GNNs) have been applied to a wide range of node-based applications and some of them have achieved state-of-the-art (SOTA) performance.

However, when applying node embeddings learned from GNNs to generate graph embeddings, the scalar node representation may not suffice to preserve the node/graph properties efficiently, resulting in sub-optimal graph embeddings.

Inspired by the Capsule Neural Network (CapsNet), we propose the Capsule Graph Neural Network (CapsGNN), which adopts the concept of capsules to address the weakness in existing GNN-based graph embeddings algorithms.

By extracting node features in the form of capsules, routing mechanism can be utilized to capture important information at the graph level.

As a result, our model generates multiple embeddings for each graph to capture graph properties from different aspects.

The attention module incorporated in CapsGNN is used to tackle graphs with various sizes which also enables the model to focus on critical parts of the graphs.

Our extensive evaluations with 10 graph-structured datasets demonstrate that CapsGNN has a powerful mechanism that operates to capture macroscopic properties of the whole graph by data-driven.

It outperforms other SOTA techniques on several graph classification tasks, by virtue of the new instrument.

Inspired by CapsNet, we propose a novel architecture for graph embeddings on the basis of node features extracted from GNN.

Capsule Graph Neural Network


We introduce a novel framework for generative models based on Restricted Kernel Machines (RKMs) with multi-view generation and uncorrelated feature learning capabilities, called Gen-RKM.

To incorporate multi-view generation, this mechanism uses a shared representation of data from various views.

The mechanism is flexible to incorporate both kernel-based, (deep) neural network and convolutional based models within the same setting.

To update the parameters of the network, we propose a novel training procedure which jointly learns the features and shared representation.

Experiments demonstrate the potential of the framework through qualitative evaluation of generated samples.

Gen-RKM: a novel framework for generative models using Restricted Kernel Machines with multi-view generation and uncorrelated feature learning.

Generative Restricted Kernel Machines


Work on the problem of contextualized word representation—the development of reusable neural network components for sentence understanding—has recently seen a  surge of progress centered on the unsupervised pretraining task of language modeling with methods like ELMo (Peters et al., 2018).

This paper contributes the first large-scale systematic study comparing different pretraining tasks in this context, both as complements to language modeling and as potential alternatives.

The primary results of the study support the use of language modeling as a pretraining task and set a new state of the art among comparable models using multitask learning with language models.

However, a closer look at these results reveals worryingly strong baselines and strikingly varied results across target tasks, suggesting that the widely-used paradigm of pretraining and freezing sentence encoders may not be an ideal platform for further work.

We compare many tasks and task combinations for pretraining sentence-level BiLSTMs for NLP tasks. Language modeling is the best single pretraining task, but simple baselines also do well.

Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling

language modeling
pretraine
sentence-level


In this paper, we study the adversarial attack and defence problem in deep learning from the perspective of Fourier analysis.

We first explicitly compute the Fourier transform of deep ReLU neural networks and show that there exist decaying but non-zero high frequency components in the Fourier spectrum of neural networks.

We then demonstrate that the vulnerability of neural networks towards adversarial samples can be attributed to these insignificant but non-zero high frequency components.

Based on this analysis, we propose to use a simple post-averaging technique to smooth out these high frequency components to improve the robustness of neural networks against adversarial attacks.

Experimental results on the ImageNet and the CIFAR-10 datasets have shown that our proposed method is universally effective to defend many existing adversarial attacking methods proposed in the literature, including FGSM, PGD, DeepFool and C&W attacks.

Our post-averaging method is simple since it does not require any re-training, and meanwhile it can successfully defend over 80-96% of the adversarial samples generated by these methods without introducing significant performance degradation (less than 2%) on the original clean images.

An insight into the reason of adversarial vulnerability, an effective defense method against adversarial attacks.

Bandlimiting Neural Networks Against Adversarial Attacks


Reinforcement learning methods that continuously learn neural networks by episode generation with game tree search have been successful in two-person complete information deterministic games such as chess, shogi, and Go.

However, there are only reports of practical cases and there are little evidence to guarantee the stability and the final performance of learning process.

In this research, the coordination of episode generation was focused on.

By means of regarding the entire system as game tree search, the new method can handle the trade-off between exploitation and exploration during episode generation.

The experiments with a small problem showed that it had robust performance compared to the existing method, Alpha Zero.

Apply Monte Carlo Tree Search to episode generation in Alpha Zero

REFINING MONTE CARLO TREE SEARCH AGENTS BY MONTE CARLO TREE SEARCH


Message-passing neural networks (MPNNs) have been successfully applied in a wide variety of applications in the real world.

However, two fundamental weaknesses of MPNNs' aggregators limit their ability to represent graph-structured data: losing the structural information of nodes in neighborhoods and lacking the ability to capture long-range dependencies in disassortative graphs.

Few studies have noticed the weaknesses from different perspectives.

From the observations on classical neural network and network geometry, we propose a novel geometric aggregation scheme for graph neural networks to overcome the two weaknesses.

The behind basic idea is the aggregation on a graph can benefit from a continuous space underlying the graph.

The proposed aggregation scheme is permutation-invariant and consists of three modules, node embedding, structural neighborhood, and bi-level aggregation.

We also present an implementation of the scheme in graph convolutional networks, termed Geom-GCN, to perform transductive learning on graphs.

Experimental results show the proposed Geom-GCN achieved state-of-the-art performance on a wide range of open datasets of graphs.

For graph neural networks, the aggregation on a graph can benefit from a continuous space underlying the graph.

Geom-GCN: Geometric Graph Convolutional Networks


We consider the task of program synthesis in the presence of a reward function over the output of programs, where the goal is to find programs with maximal rewards.

We introduce a novel iterative optimization scheme, where we train an RNN on a dataset of K best programs from a priority queue of the generated programs so far.

Then, we synthesize new programs and add them to the priority queue by sampling from the RNN.

We benchmark our algorithm called priority queue training (PQT) against genetic algorithm and reinforcement learning baselines on a simple but expressive Turing complete programming language called BF.

Our experimental results show that our deceptively simple PQT algorithm significantly outperforms the baselines.

By adding a program length penalty to the reward function, we are able to synthesize short, human readable programs.

We use a simple search algorithm involving an RNN and priority queue to find solutions to coding tasks.

Code Synthesis with Priority Queue Training


We present graph wavelet neural network (GWNN), a novel graph convolutional neural network (CNN), leveraging graph wavelet transform to address the shortcomings of previous spectral graph CNN methods that depend on graph Fourier transform.

Different from graph Fourier transform, graph wavelet transform can be obtained via a fast algorithm without requiring matrix eigendecomposition with high computational cost.

Moreover, graph wavelets are sparse and localized in vertex domain, offering high efficiency and good interpretability for graph convolution.

The proposed GWNN significantly outperforms previous spectral graph CNNs in the task of graph-based semi-supervised classification on three benchmark datasets: Cora, Citeseer and Pubmed.

We present graph wavelet neural network (GWNN), a novel graph convolutional neural network (CNN), leveraging graph wavelet transform to address the shortcoming of previous spectral graph CNN methods that depend on graph Fourier transform.

Graph Wavelet Neural Network


Learning rate decay (lrDecay) is a \emph{de facto} technique for training modern neural networks.

It starts with a large learning rate and then decays it multiple times.

It is empirically observed to help both optimization and generalization.

Common beliefs in how lrDecay works come from the optimization analysis of (Stochastic) Gradient Descent:

1) an initially large learning rate accelerates training or helps the network escape spurious local minima;

2) decaying the learning rate helps the network converge to a local minimum and avoid oscillation.

Despite the popularity of these common beliefs, experiments suggest that they are insufficient in explaining the general effectiveness of lrDecay in training modern neural networks that are deep, wide, and nonconvex.

We provide another novel explanation: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns.

The proposed explanation is validated on a carefully-constructed dataset with tractable pattern complexity.

And its implication, that additional patterns learned in later stages of lrDecay are more complex and thus less transferable, is justified in real-world datasets.

We believe that this alternative explanation will shed light into the design of better training strategies for modern neural networks.

We provide another novel explanation of learning rate decay: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns.

How Does Learning Rate Decay Help Modern Neural Networks?


We show how an ensemble of $Q^*$-functions can be leveraged for more effective exploration in deep reinforcement learning.

We build on well established algorithms from the bandit setting, and adapt them to the $Q$-learning setting.

We propose an exploration strategy based on upper-confidence bounds (UCB).

Our experiments show significant gains on the Atari benchmark.

Adapting UCB exploration to ensemble Q-learning improves over prior methods such as Double DQN, A3C+ on Atari benchmark

UCB EXPLORATION VIA Q-ENSEMBLES


We focus on the problem of learning a single motor module that can flexibly express a range of behaviors for the control of high-dimensional physically simulated humanoids.

To do this, we propose a motor architecture that has the general structure of an inverse model with a latent-variable bottleneck.

We show that it is possible to train this model entirely offline to compress thousands of expert policies and learn a motor primitive embedding space.

The trained neural probabilistic motor primitive system can perform one-shot imitation of whole-body humanoid behaviors, robustly mimicking unseen trajectories.

Additionally, we demonstrate that it is also straightforward to train controllers to reuse the learned motor primitive space to solve tasks, and the resulting movements are relatively naturalistic.

To support the training of our model, we compare two approaches for offline policy cloning, including an experience efficient method which we call linear feedback policy cloning.

We encourage readers to view a supplementary video (https://youtu.be/CaDEf-QcKwA ) summarizing our results.

Neural Probabilistic Motor Primitives compress motion capture tracking policies into one flexible model capable of one-shot imitation and reuse as a low-level controller.

Neural Probabilistic Motor Primitives for Humanoid Control


Data augmentation is a useful technique to enlarge the size of the training set and prevent overfitting for different machine learning tasks when training data is scarce.

However, current data augmentation techniques rely heavily on human design and domain knowledge, and existing automated approaches are yet to fully exploit the latent features in the training dataset.

In this paper we propose  \textit{Parallel Adaptive GAN Data Augmentation}(PAGANDA), where the training set adaptively enriches itself with sample images automatically constructed from Generative Adversarial Networks (GANs) trained in parallel.

We demonstrate by experiments that our data augmentation strategy, with little model-specific considerations, can be easily adapted to cross-domain deep learning/machine learning tasks such as image classification and image inpainting, while significantly improving model performance in both tasks.

Our source code and experimental details are available at \url{https://github.com/miaojiang1987/k-folder-data-augmentation-gan/}.

We present an automated adaptive data augmentation that works for multiple different tasks.

PAGANDA: An Adaptive Task-Independent Automatic Data Augmentation


Deep neural networks with millions of parameters may suffer from poor generalizations due to overfitting.

To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples.

In particular, we distill the predictive distribution between different samples of the same label and augmented samples of the same source during training.

In other words, we regularize the dark knowledge (i.e., the knowledge on wrong predictions) of a single network, i.e., a self-knowledge distillation technique, to force it output more meaningful predictions.

We demonstrate the effectiveness of the proposed method  via  experiments  on  various  image  classification  tasks:  it  improves  not only the generalization ability, but also the calibration accuracy of modern neural networks.

We propose a new regularization technique based on the knowledge distillation.

Regularizing Predictions via Class-wise Self-knowledge Distillation


In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models.

We propose the weight-dropped LSTM, which uses DropConnect on hidden-to-hidden weights, as a form of recurrent regularization.

Further, we introduce NT-ASGD, a non-monotonically triggered  (NT) variant of the averaged stochastic gradient method (ASGD), wherein the averaging trigger is determined using a NT condition as opposed to being tuned by the user.

Using these and other regularization strategies, our ASGD Weight-Dropped LSTM (AWD-LSTM) achieves state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2.

In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.

We also explore the viability of the proposed regularization and optimization strategies in the context of the quasi-recurrent neural network (QRNN) and demonstrate comparable performance to the AWD-LSTM counterpart.

The code for reproducing the results is open sourced and is available at https://github.com/salesforce/awd-lstm-lm.

Effective regularization and optimization strategies for LSTM-based language models achieves SOTA on PTB and WT2.

Regularizing and Optimizing LSTM Language Models


Various methods of measuring unit selectivity have been developed with the aim of better understanding how neural networks work.

But the different measures provide divergent estimates of selectivity, and this has led to different conclusions regarding the conditions in which selective object representations are learned and the functional relevance of these representations.

In an attempt to better characterize object selectivity, we undertake a comparison of various selectivity measures on a large set of units in AlexNet, including localist selectivity, precision, class-conditional mean activity selectivity (CCMAS), network dissection, the human interpretation of activation maximization (AM) images, and standard signal-detection measures.

We find that the different measures provide different estimates of object selectivity, with precision and CCMAS measures providing misleadingly high estimates.

Indeed, the most selective units had a poor hit-rate or a high false-alarm rate (or both) in object classification, making them poor object detectors.

We fail to find any units that are even remotely as selective as the 'grandmother cell' units reported in recurrent neural networks.

In order to generalize these results, we compared selectivity measures on a few units in VGG-16 and GoogLeNet trained on the ImageNet or Places-365 datasets that have been described as 'object detectors'.

Again, we find poor hit-rates and high false-alarm rates for object classification.

Looking for object detectors using many different selectivity measures; CNNs are slightly selective , but not enough to be termed object detectors.

Are there any 'object detectors' in the hidden layers of CNNs trained to identify objects or scenes?


The folding structure of the DNA molecule combined with helper molecules, also referred to as the chromatin, is highly relevant for the functional properties of DNA.

The chromatin structure is largely determined by the underlying primary DNA sequence, though the interaction is not yet fully understood.

In this paper we develop a convolutional neural network that takes an image-representation of primary DNA sequence as its input, and predicts key determinants of chromatin structure.

The method is developed such that it is capable of detecting interactions between distal elements in the DNA sequence, which are known to be highly relevant.

Our experiments show that the method outperforms several existing methods both in terms of prediction accuracy and training time.

A method to transform DNA sequences into 2D images using space-filling Hilbert Curves to enhance the strengths of CNNs

An image representation based convolutional network for DNA classification


This paper introduces a novel method to perform transfer learning across domains and tasks, formulating it as a problem of learning to cluster.

The key insight is that, in addition to features, we can transfer similarity information and this is sufficient to learn a similarity function and clustering network to perform both domain adaptation and cross-task transfer learning.

We begin by reducing categorical information to pairwise constraints, which only considers whether two instances belong to the same class or not (pairwise semantic similarity).

This similarity is category-agnostic and can be learned from data in the source domain using a similarity network.

We then present two novel approaches for performing transfer learning using this similarity function.

First, for unsupervised domain adaptation, we design a new loss function to regularize classification with a constrained clustering loss, hence learning a clustering network with the transferred similarity metric generating the training inputs.

Second, for cross-task learning (i.e., unsupervised clustering with unseen categories), we propose a framework to reconstruct and estimate the number of semantic clusters, again using the clustering network.

Since the similarity network is noisy, the key is to use a robust clustering algorithm, and we show that our formulation is more robust than the alternative constrained and unconstrained clustering approaches.

Using this method, we first show state of the art results for the challenging cross-task problem, applied on Omniglot and ImageNet.

Our results show that we can reconstruct semantic clusters with high accuracy.

We then evaluate the performance of cross-domain transfer using images from the Office-31 and SVHN-MNIST tasks and present top accuracy on both datasets.

Our approach doesn't explicitly deal with domain discrepancy.

If we combine with a domain adaptation loss, it shows further improvement.

A learnable clustering objective to facilitate transfer learning across domains and tasks

Learning to cluster in order to transfer across domains and tasks


Recent advances in cross-lingual word embeddings have primarily relied on mapping-based methods, which project pretrained word embeddings from different languages into a shared space through a linear transformation.

However, these approaches assume word embedding spaces are isomorphic between different languages, which has been shown not to  hold in practice (Søgaard et al., 2018), and fundamentally limits their performance.

This motivates investigating joint learning methods which can overcome this impediment, by simultaneously learning embeddings across languages via a cross-lingual term in the training objective.

Given the abundance of parallel data available (Tiedemann, 2012), we propose a bilingual extension of the CBOW method which leverages sentence-aligned corpora to obtain robust cross-lingual word and sentence representations.

Our approach significantly improves cross-lingual sentence retrieval performance over all other approaches, as well as convincingly outscores mapping methods while maintaining parity with jointly trained methods on word-translation.

It also achieves parity with a deep RNN method on a zero-shot cross-lingual document classification task, requiring far fewer computational resources for training and inference.

As an additional advantage, our bilingual method also improves the quality of monolingual word vectors despite training on much smaller datasets.

We make our code and models publicly available.

Joint method for learning cross-lingual embeddings with state-of-art performance for cross-lingual tasks and mono-lingual quality

Robust Cross-lingual Embeddings from Parallel Sentences


To act and plan in complex environments, we posit that agents should have a mental simulator of the world with three characteristics:

(a) it should build an abstract state representing the condition of the world;

(b) it should form a belief which represents uncertainty on the world;

(c) it should go beyond simple step-by-step simulation, and exhibit temporal abstraction.

Motivated by the absence of a model satisfying all these requirements, we propose TD-VAE, a generative sequence model that learns representations containing explicit beliefs about states several steps into the future, and that can be rolled out directly without single-step transitions.

TD-VAE is trained on pairs of temporally separated time points, using an analogue of temporal difference learning used in reinforcement learning.

Generative model of temporal data, that builds online belief state, operates in latent space, does jumpy predictions and rollouts of states.

Temporal Difference Variational Auto-Encoder


This paper introduces an information theoretic co-training objective for unsupervised learning.

We consider the problem of predicting the future

. Rather than predict future sensations (image pixels or sound waves) we predict ``hypotheses'' to be confirmed by future sensations

. More formally, we assume a population distribution on pairs $(x,y)$ where we can think of $x$ as a past sensation and $y$ as a future sensation

. We train both a predictor model $P_\Phi(z|x)$ and a confirmation model $P_\Psi(z|y)$ where we view $z$ as hypotheses (when predicted) or facts (when confirmed

). For a population distribution on pairs $(x,y)$ we focus on the problem of measuring the mutual information between $x$ and $

y$. By the data processing inequality this mutual information is at least as large as the mutual information between $x$ and $z$ under the distribution on triples $(x,z,y)$ defined by the confirmation model $P_\Psi(z|y)$.

The information theoretic training objective for $P_\Phi(z|x)$ and $P_\Psi(z|y)$ can be viewed as a form of co-training where we want the prediction from $x$ to match the confirmation from $y$.

We give experiments on applications to learning phonetics on the TIMIT dataset.

Presents an information theoretic training objective for co-training and demonstrates its power in unsupervised learning of phonetics.

Information Theoretic Co-Training


Generative models for singing voice have been mostly concerned with the task of "singing voice synthesis," i.e., to produce singing voice waveforms given musical scores and text lyrics.

In this work, we explore a novel yet challenging alternative: singing voice generation without pre-assigned scores and lyrics, in both training and inference time.

In particular, we experiment with three different schemes:

1) free singer, where the model generates singing voices without taking any conditions;

2) accompanied singer, where the model generates singing voices over a waveform of instrumental music; and

3) solo singer, where the model improvises a chord sequence first and then uses that to generate voices.

We outline the associated challenges and propose a pipeline to tackle these new tasks.

This involves the development of source separation and transcription models for data preparation, adversarial networks for audio generation, and customized metrics for evaluation.

Our models generate singing voices without lyrics and scores. They take accompaniment as input and output singing voices.

Score and Lyrics-Free Singing Voice Generation


The carbon footprint of natural language processing (NLP) research has been increasing in recent years due to its reliance on large and inefficient neural network implementations.

Distillation is a network compression technique which attempts to impart knowledge from a large model to a smaller one.

We use teacher-student distillation  to  improve  the  efficiency  of  the  Biaffine  dependency  parser  which obtains state-of-the-art performance with respect to accuracy and parsing speed (Dozat & Manning, 2016).

When distilling to 20% of the original model’s trainable parameters, we only observe an average decrease of ∼1 point for both UAS and LAS across a number of diverse Universal Dependency treebanks while being 2.26x (1.21x) faster than the baseline model on CPU (GPU) at inference time.

We also observe a small increase in performance when compressing to 80% for some treebanks.

Finally, through distillation we attain a parser which is not only faster but also more accurate than the fastest modern parser on the Penn Treebank.

We increase the efficiency of neural network dependency parsers with teacher-student distillation.

Distilling Neural Networks for Faster and Greener Dependency Parsing


While autoencoders are a key technique in representation learning for continuous structures, such as images or wave forms, developing general-purpose autoencoders for discrete structures, such as text sequence or discretized images, has proven to be more challenging.

In particular, discrete inputs make it more difficult to learn a smooth encoder that preserves the complex local relationships in the input space.

In this work, we propose an adversarially regularized autoencoder (ARAE) with the goal of learning more robust discrete-space representations.

ARAE jointly trains both a rich discrete-space encoder, such as an RNN, and a simpler continuous space generator function, while using generative adversarial network (GAN) training to constrain the distributions to be similar.

This method yields a smoother contracted code space that maps similar inputs to nearby codes, and also an implicit latent variable GAN model for generation.

Experiments on text and discretized images demonstrate that the GAN model produces clean interpolations and captures the multimodality of the original space, and that the autoencoder produces improvements in semi-supervised learning as well as state-of-the-art results in unaligned text style transfer task using only a shared continuous-space representation.

Adversarially Regularized Autoencoders learn smooth representations of discrete structures allowing for interesting results in text generation, such as unaligned style transfer, semi-supervised learning, and latent space interpolation and arithmetic.

Adversarially Regularized Autoencoders


When data arise from multiple latent subpopulations, machine learning frameworks typically estimate parameter values independently for each sub-population.

to overcome these limits by considering samples as tasks in a multitask learning framework.

We present a method to estimate collections of regression models in which each model is personalized to a single sample.

Every Sample a Task: Pushing the Limits of Heterogeneous Models with Personalized Regression


In this work, we first conduct mathematical analysis on the memory, which is

defined as a function that maps an element in a sequence to the current output,

of three RNN cells; namely, the simple recurrent neural network (SRN), the long

short-term memory (LSTM) and the gated recurrent unit (GRU).

analysis, we propose a new design, called the extended-long short-term memory

(ELSTM), to extend the memory length of a cell.

RNN model that is robust to previous erroneous predictions, called the dependent

bidirectional recurrent neural network (DBRNN), for the sequence-in-sequenceout

Finally, the performance of the DBRNN model with the

ELSTM cell is demonstrated by experimental results.

A recurrent neural network cell with extended-long short-term memory and a multi-task RNN model for sequence-in-sequence-out problems

Dependent Bidirectional RNN with Extended-long Short-term Memory


Many recently trained neural networks employ large numbers of parameters to achieve good performance.

One may intuitively use the number of parameters required as a rough gauge of the difficulty of a problem.

In this paper we attempt to answer this question by training networks not in their native parameter space, but instead in a smaller, randomly oriented subspace.

We slowly increase the dimension of this subspace, note at which dimension solutions first appear, and define this to be the intrinsic dimension of the objective landscape.

The approach is simple to implement, computationally tractable, and produces several suggestive conclusions.

Many problems have smaller intrinsic dimensions than one might suspect, and the intrinsic dimension for a given dataset varies little across a family of models with vastly different sizes.

This latter result has the profound implication that once a parameter space is large enough to solve a problem, extra parameters serve directly to increase the dimensionality of the solution manifold.

Intrinsic dimension allows some quantitative comparison of problem difficulty across supervised, reinforcement, and other types of learning where we conclude, for example, that solving the inverted pendulum problem is 100 times easier than classifying digits from MNIST, and playing Atari Pong from pixels is about as hard as classifying CIFAR-10.

In addition to providing new cartography of the objective landscapes wandered by parameterized models, the method is a simple technique for constructively obtaining an upper bound on the minimum description length of a solution.

A byproduct of this construction is a simple approach for compressing networks, in some cases by more than 100 times.

We train in random subspaces of parameter space to measure how many dimensions are really needed to find a solution.

Measuring the Intrinsic Dimension of Objective Landscapes


Graph Neural Networks as a combination of Graph Signal Processing and Deep Convolutional Networks shows great power in pattern recognition in non-Euclidean domains.

In this paper, we propose a new method to deploy two pipelines based on the duality of a graph to improve accuracy.

By exploring the primal graph and its dual graph where nodes and edges can be treated as one another, we have exploited the benefits of both vertex features and edge features.

As a result, we have arrived at a framework that has great potential in both semisupervised and unsupervised learning.

A primal dual graph neural network model for semi-supervised learning

TWIN GRAPH CONVOLUTIONAL NETWORKS: GCN WITH DUAL GRAPH SUPPORT FOR SEMI-SUPERVISED LEARNING

semi-supervised learning
dual graph
network


We describe two end-to-end autoencoding models for semi-supervised graph-based dependency parsing.

The first model is a Local Autoencoding Parser (LAP) encoding the input using continuous latent variables in a sequential manner; The second model is a Global Autoencoding Parser (GAP) encoding the input into dependency trees as latent variables, with exact inference.

Both models consist of two parts: an encoder enhanced by deep neural networks (DNN) that can utilize the contextual information to encode the input into latent variables, and a decoder which is a generative model able to reconstruct the input.

Both LAP and GAP admit a unified structure with different loss functions for labeled and unlabeled data with shared parameters.

We conducted experiments on WSJ and UD dependency parsing data sets, showing that our models can exploit the unlabeled data to boost the performance given a limited amount of labeled data.

We describe two end-to-end autoencoding parsers for semi-supervised graph-based dependency parsing.

Semi-supervised Autoencoding Projective Dependency Parsing


We improve previous end-to-end differentiable neural networks (NNs) with fast

A gate mechanism updates fast weights at every time step of

a sequence through two separate outer-product-based matrices generated by slow

The system is trained on a complex sequence to sequence variation

of the Associative Retrieval Problem with roughly 70 times more temporal

memory (i.e. time-varying variables) than similar-sized standard recurrent NNs

In terms of accuracy and number of parameters, our architecture outperforms

a variety of RNNs, including Long Short-Term Memory, Hypernetworks,

An improved Fast Weight network which shows better results on a general toy task.

GATED FAST WEIGHTS FOR ASSOCIATIVE RETRIEVAL


The field of deep learning has been craving for an optimization method that shows outstanding property for both optimization and generalization.

We propose a method for mathematical optimization based on flows along geodesics, that is, the shortest paths between two points, with respect to the Riemannian metric induced by a non-linear function.

In our method, the flows refer to Exponentially Decaying Flows (EDF), as they can be designed to converge on the local solutions exponentially.

In this paper, we conduct experiments to show its high performance on optimization benchmarks (i.e., convergence properties), as well as its potential for producing good machine learning benchmarks (i.e., generalization properties).

Introduction of a new optimization method and its application to deep learning.

Exponentially Decaying Flows for Optimization in Deep Learning


We introduce Quantum Graph Neural Networks (QGNN), a new class of quantum neural network ansatze which are tailored to represent quantum processes which have a graph structure, and are particularly suitable to be executed on distributed quantum systems over a quantum network.

Along with this general class of ansatze, we introduce further specialized architectures, namely, Quantum Graph Recurrent Neural Networks (QGRNN) and Quantum Graph Convolutional Neural Networks (QGCNN).

We provide four example applications of QGNN's: learning Hamiltonian dynamics of quantum systems, learning how to create multipartite entanglement in a quantum network, unsupervised learning for spectral clustering, and supervised learning for graph isomorphism classification.

Introducing a new class of quantum neural networks for learning graph-based representations on quantum computers.

Quantum Graph Neural Networks


Data breaches involve information being accessed by unauthorized parties.

Our research concerns user perception of data breaches, especially issues relating to accountability.

A preliminary study indicated many people had weak understanding of the issues, and felt they themselves were somehow responsible.

We speculated that this impression might stem from organizational communication strategies.

We therefore compared texts from organizations with external sources, such as the news media.

This suggested that organizations use well-known crisis communication methods to reduce their reputational damage, and that these strategies align with repositioning of the narrative elements involved in the story.

We then conducted a quantitative study, asking participants to rate either organizational texts or news texts about breaches.

The findings of this study were in line with our document analysis, and suggest that organizational communication affects the users' perception of victimization, attitudes in data protection, and accountability.

Our study suggests some software design and legal implications supporting users to protect themselves and develop better mental models of security breaches.

"In this paper, we tested communication strategies' influence on users mental models of a data breach."

We're Here to Help: Crisis Communication and User Perception of Data Breaches


Goal recognition is the problem of inferring the correct goal towards which an agent executes a plan, given a set of goal hypotheses, a domain model, and a (possibly noisy) sample of the plan being executed.

This is a key problem in both cooperative and competitive agent interactions and recent approaches have produced fast and accurate goal recognition algorithms.

In this paper, we leverage advances in operator-counting heuristics computed using linear programs over constraints derived from classical planning problems to solve goal recognition problems.

Our approach uses additional operator-counting constraints derived from the observations to efficiently infer the correct goal, and serves as basis for a number of further methods with additional constraints.

A goal recognition approach based on operator counting heuristics used to account for noise in the dataset.

Robust Goal Recognition with Operator-Counting Heuristics


It can be challenging to train multi-task neural networks that outperform or even match their single-task counterparts.

To help address this, we propose using knowledge distillation where single-task models teach a multi-task model.

We enhance this training with teacher annealing, a novel method that gradually transitions the model from distillation to supervised learning, helping the multi-task model surpass its single-task teachers.

We evaluate our approach by multi-task fine-tuning BERT on the GLUE benchmark.

Our method consistently improves over standard single-task and multi-task training.

distilling single-task models into a multi-task model improves natural language understanding performance

BAM! Born-Again Multi-Task Networks for Natural Language Understanding


The detection of out of distribution samples for image classification has been widely researched.

Safety critical applications, such as autonomous driving, would benefit from the ability to localise the unusual objects causing the image to be out of distribution.

This paper adapts state-of-the-art methods for detecting out of distribution images for image classification to the new task of detecting out of distribution pixels, which can localise the unusual objects.

It further experimentally compares the adapted methods on two new datasets derived from existing semantic segmentation datasets using PSPNet and DeeplabV3+ architectures, as well as proposing a new metric for the task.

The evaluation shows that the performance ranking of the compared methods does not transfer to the new task and every method performs significantly worse than their image-level counterparts.

Evaluating pixel-level out-of-distribution detection methods on two new real world datasets using PSPNet and DeeplabV3+.

Efficacy of Pixel-Level OOD Detection for Semantic Segmentation


We present a domain adaptation method for transferring neural representations from label-rich source domains to unlabeled target domains.

Recent adversarial methods proposed for this task learn to align features across domains by ``fooling'' a special domain classifier network.

However, a drawback of this approach is that the domain classifier simply labels the generated features as in-domain or not, without considering the boundaries between classes.

This means that ambiguous target features can be generated near class boundaries, reducing target classification accuracy.

We propose a novel approach, Adversarial Dropout Regularization (ADR), which encourages the generator to output more discriminative features for the target domain.

Our key idea is to replace the traditional domain critic with a critic that detects non-discriminative features by using dropout on the classifier network.

The generator then learns to avoid these areas of the feature space and thus creates better features.

We apply our ADR approach to the problem of unsupervised domain adaptation for image classification and semantic segmentation tasks, and demonstrate significant improvements over the state of the art.

We present a new adversarial method for adapting neural representations based on a critic that detects non-discriminative features.

Adversarial Dropout Regularization


The use of deep learning for a wide range of data problems has increased the need for understanding and diagnosing these models, and deep learning interpretation techniques have become an essential tool for data analysts.

Although numerous model interpretation methods have been proposed in recent years, most of these procedures are based on heuristics with little or no theoretical guarantees.

In this work, we propose a statistical framework for saliency estimation for black box computer vision models.

We build a model-agnostic estimation procedure that is statistically consistent and passes the saliency checks of Adebayo et al. (2018).

Our method requires solving a linear program, whose solution can be efficiently computed in polynomial time.

Through our theoretical analysis, we establish an upper bound on the number of model evaluations needed to recover the region of importance with high probability, and build a new perturbation scheme for estimation of local gradients that is shown to be more efficient than the commonly used random perturbation schemes.

Validity of the new method is demonstrated through sensitivity analysis.

We propose a statistical framework and a theoretically consistent procedure for saliency estimation.

Statistically Consistent Saliency Estimation


We explore the idea of compositional set embeddings that can be used to infer not

just a single class, but the set of classes associated with the input data (e.g., image,

This can be useful, for example, in multi-object detection in

images, or multi-speaker diarization (one-shot learning) in audio.

devise and implement two novel models consisting of (1) an embedding function

f trained jointly with a “composite” function g that computes set union opera-

tions between the classes encoded in two embedding vectors; and (2) embedding

f trained jointly with a “query” function h that computes whether the classes en-

coded in one embedding subsume the classes encoded in another embedding.

contrast to prior work, these models must both perceive the classes associated

with the input examples, and also encode the relationships between different class

In experiments conducted on simulated data, OmniGlot, and COCO

datasets, the proposed composite embedding models outperform baselines based

We explored how a novel method of compositional set embeddings can both perceive and represent not just a single class but an entire set of classes that is associated with the input data.

Compositional Embeddings: Joint Perception and Comparison of Class Label Sets


In this work, we propose a goal-driven collaborative task that contains language, vision, and action in a virtual environment as its core components.

Specifically, we develop a Collaborative image-Drawing game between two agents, called CoDraw.

Our game is grounded in a virtual world that contains movable clip art objects.

The game involves two players: a Teller and a Drawer.

The Teller sees an abstract scene containing multiple clip art pieces in a semantically meaningful configuration, while the Drawer tries to reconstruct the scene on an empty canvas using available clip art pieces.

The two players communicate via two-way communication using natural language.

We collect the CoDraw dataset of ~10K dialogs consisting of ~138K messages exchanged between human agents.

We define protocols and metrics to evaluate the effectiveness of learned agents on this testbed, highlighting the need for a novel "crosstalk" condition which pairs agents trained independently on disjoint subsets of the training data for evaluation.

We present models for our task, including simple but effective baselines and neural network approaches trained using a combination of imitation learning and goal-driven training.

All models are benchmarked using both fully automated evaluation and by playing the game with live human agents.

We introduce a dataset, models, and training + evaluation protocols for a collaborative drawing task that allows studying goal-driven and perceptually + actionably grounded language generation and understanding.

CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication

collaborative drawing
goal-driven
ground


Presence of bias and confounding effects is inarguably one of the most critical challenges in machine learning applications that has alluded to pivotal debates in the recent years.

Such challenges range from spurious associations of confounding variables in medical studies to the bias of race in gender or face recognition systems.

One solution is to enhance datasets and organize them such that they do not reflect biases, which is a cumbersome and intensive task.

The alternative is to make use of available data and build models considering these biases.

Traditional statistical methods apply straightforward techniques such as residualization or stratification to precomputed features to account for confounding variables.

However, these techniques are not in general applicable to end-to-end deep learning methods.

In this paper, we propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s).

This is enabled by incorporating a new adversarial loss function that encourages a vanished correlation between the bias and learned features.

We apply our method to a synthetic, a medical diagnosis, and a gender classification (Gender Shades) dataset.

Our results show that the learned features by our method not only result in superior prediction performance but also are uncorrelated with the bias or confounder variables.

The code is available at http://blinded_for_review/.

We propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s) by incorporating a loss function that encourages a vanished correlation between the bias and learned features.

Bias-Resilient Neural Network


Existing neural question answering (QA) models are required to reason over and draw complicated inferences from a long context for most large-scale QA datasets.

However, if we view QA as a combined retrieval and reasoning task, we can assume the existence of a minimal context which is necessary and sufficient to answer a given question.

Recent work has shown that a sentence selector module that selects a shorter context and feeds it to the downstream QA model achieves performance comparable to a QA model trained on full context, while also being more interpretable.

Recent work has also shown that most state-of-the-art QA models break when adversarially generated sentences are appended to the context.

While humans are immune to such distractor sentences, QA models get easily misled into selecting answers from these sentences.

We hypothesize that the sentence selector module can filter out extraneous context, thereby allowing the downstream QA model to focus and reason over the parts of the context that are relevant to the question.

In this paper, we show that the sentence selector itself is susceptible to adversarial inputs.

However, we demonstrate that a pipeline consisting of a sentence selector module followed by the QA model can be made more robust to adversarial attacks in comparison to a QA model trained on full context.

Thus, we provide evidence towards a modular approach for question answering that is more robust and interpretable.

A modular approach consisting of a sentence selector module followed by the QA model can be made more robust to adversarial attacks in comparison to a QA model trained on full context.

Making question answering more robust through relevant context selection


Multi-relational graph embedding which aims at achieving effective representations with reduced low-dimensional parameters, has been widely used in knowledge base completion.

Although knowledge base data usually contains tree-like or cyclic structure, none of existing approaches can embed these data into a compatible space that in line with the structure.

To overcome this problem, a novel framework, called Riemannian TransE, is proposed in this paper to embed the entities in a Riemannian manifold.

Riemannian TransE models each relation as a move to a point and defines specific novel distance dissimilarity for each relation, so that all the relations are naturally embedded in correspondence to the structure of data.

Experiments on several knowledge base completion tasks have shown that, based on an appropriate choice of manifold, Riemannian TransE achieves good performance even with a significantly reduced parameters.

Multi-relational graph embedding with Riemannian manifolds and TransE-like loss function.

Riemannian TransE: Multi-relational Graph Embedding in Non-Euclidean Space


Catastrophic forgetting in neural networks is one of the most well-known problems in continual learning.

Previous attempts on addressing the problem focus on preventing important weights from changing.

Such methods often require task boundaries to learn effectively and do not support backward transfer learning.

In this paper, we propose a meta-learning algorithm which learns to reconstruct the gradients of old tasks w.r.t. the current parameters and combines these reconstructed gradients with the current gradient to enable continual learning and backward transfer learning from the current task to previous tasks.

Experiments on standard continual learning benchmarks show that our algorithm can effectively prevent catastrophic forgetting and supports backward transfer learning.

We propose a meta learning algorithm for continual learning which can effectively prevent catastrophic forgetting problem and support backward transfer learning.

Enabling Continual Learning in Neural Networks with Meta Learning


 We give a formal procedure for computing preimages of convolutional
  network outputs using the dual basis defined from the set of
  hyperplanes associated with the layers of the network.

We point out
  the special symmetry associated with arrangements of hyperplanes of
  convolutional networks that take the form of regular
  multidimensional polyhedral cones.

We discuss  the efficiency of of
  large number of layers of nested cones that result from incremental
  small size convolutions in order to give a good compromise between
  efficient contraction of data to low dimensions and shaping of
  preimage manifolds.

We demonstrate how a specific network flattens a
  non linear input manifold to an affine output manifold and discuss
  it's relevance to understanding classification properties of deep
  networks.

Analysis of deep convolutional networks in terms of associated arrangement of hyperplanes

Geometry of Deep Convolutional Networks


Meta learning has been making impressive progress for fast model adaptation.

However, limited work has been done on learning fast uncertainty adaption for Bayesian modeling.

In this paper, we propose to achieve the goal by placing meta learning on the space of probability measures, inducing the concept of meta sampling for fast uncertainty adaption.

Specifically, we propose a Bayesian meta sampling framework consisting of two main components: a meta sampler and a sample adapter.

The meta sampler is constructed by adopting a neural-inverse-autoregressive-flow (NIAF) structure, a variant of the recently proposed neural autoregressive flows, to efficiently generate meta samples to be adapted.

The sample adapter moves meta samples to task-specific samples, based on a newly proposed and general Bayesian sampling technique, called optimal-transport Bayesian sampling.

The combination of the two components allows a simple learning procedure for the

meta sampler to be developed, which can be efficiently optimized via standard back-propagation.

Extensive experimental results demonstrate the efficiency and effectiveness of the proposed framework, obtaining better sample quality and faster

uncertainty adaption compared to related methods.

We proposed a Bayesian meta sampling method for adapting the model uncertainty in meta learning

Bayesian Meta Sampling for Fast Uncertainty Adaptation


We propose a context-adaptive entropy model for use in end-to-end optimized image compression.

Our model exploits two types of contexts, bit-consuming contexts and bit-free contexts, distinguished based upon whether additional bit

Based on these contexts, we allow the model to more accurately estimate the distribution of each latent representation with a more generalized form of the approximation models, which accordingly leads to an

Based on the experimental results, the proposed method outperforms the traditional image codecs, such as BPG and JPEG2000, as well as other previous artificial-neural-network (ANN) based approaches, in terms of the peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) index.

The test code is publicly available at https://github.com/JooyoungLeeETRI/CA_Entropy_Model.

Context-adaptive entropy model for use in end-to-end optimized image compression, which significantly improves compression performance

Context-adaptive Entropy Model for End-to-end Optimized Image Compression

end-to-end optimize image compression
context-adaptive entropy model


Deep networks often perform well on the data distribution on which they are trained, yet give incorrect (and often very confident) answers when evaluated on points from off of the training distribution.

This is exemplified by the adversarial examples phenomenon but can also be seen in terms of model generalization and domain shift.

Ideally, a model would assign lower confidence to points unlike those from the training distribution.

We propose a regularizer which addresses this issue by training with interpolated hidden states and encouraging the classifier to be less confident at these points.

Because the hidden states are learned, this has an important effect of encouraging the hidden states for a class to be concentrated in such a way so that interpolations within the same class or between two different classes do not intersect with the real data points from other classes.

This has a major advantage in that it avoids the underfitting which can result from interpolating in the input space.

We prove that the exact condition for this problem of underfitting to be avoided by Manifold Mixup is that the dimensionality of the hidden states exceeds the number of classes, which is often the case in practice.

Additionally, this concentration can be seen as making the features in earlier layers more discriminative.

We show that despite requiring no significant additional computation, Manifold Mixup achieves large improvements over strong baselines in supervised learning, robustness to single-step adversarial attacks, semi-supervised learning, and Negative Log-Likelihood on held out samples.

A method for learning better representations, that acts as a regularizer and despite its no significant additional computation cost , achieves improvements over strong baselines on Supervised and Semi-supervised Learning tasks.

Manifold Mixup: Learning Better Representations by Interpolating Hidden States


Sometimes SRS (Stereotactic Radio Surgery) requires using sphere packing on a Region of Interest (ROI) such as cancer to determine a treatment plan.

We have developed a sphere packing algorithm which packs non-intersecting spheres inside the ROI.

The region of interest in our case are those voxels which are identified as cancer tissues.

In this paper, we analyze the rotational invariant properties of  our sphere-packing algorithm which is based on distance transformations.

Epsilon-Rotation invariant means the ability to arbitrary rotate the 3D ROI while keeping the volume properties remaining (almost) same within some limit of epsilon.

The applied rotations produce spherical packing which remains highly correlated as we analyze the geometrically properties of sphere packing before and after the rotation of the volume data for the ROI.

Our novel sphere packing algorithm has high degree of rotation invariance within the range of +/- epsilon.

Our method used a shape descriptor derived from the values of the disjoint set of spheres form the distance-based sphere packing algorithm to extract the invariant descriptor from the ROI.

We demonstrated by implementing these ideas using Slicer3D platform available for our research.

The data is based on sing MRI Stereotactic images.

We presented several performance results on different benchmarks data of over 30 patients in Slicer3D platform.

Packing region of Interest (ROI) such as cancerous regions identified in 3D Volume Data, Packing spheres inside the ROI, rotating the ROI , measures of difference in sphere packing before and after the rotation.

epsilon-Rotation Invariant Euclidean Spheres Packing in Slicer3D


We show implicit filter level sparsity manifests in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained using adaptive gradient descent techniques with L2 regularization or weight decay.

Through an extensive empirical study (Anonymous, 2019) we hypothesize the mechanism be hind the sparsification process.

We find that the interplay  of  various  phenomena  influences  the strength of L2 and weight decay regularizers, leading the supposedly non sparsity inducing regularizers to induce filter sparsity.

In this workshop article we summarize some of our key findings and experiments, and present additional results on modern network architectures such as ResNet-50.

Filter level sparsity emerges implicitly in CNNs trained with adaptive gradient descent approaches due to various phenomena, and the extent of sparsity can be inadvertently affected by different seemingly unrelated hyperparameters.

Emergence of Implicit Filter Sparsity in Convolutional Neural Networks


Humans can learn a variety of concepts and skills incrementally over the course of their lives while exhibiting an array of desirable properties, such as non-forgetting, concept rehearsal, forward transfer and backward transfer of knowledge, few-shot learning, and selective forgetting.

Previous approaches to lifelong machine learning can only demonstrate subsets of these properties, often by combining multiple complex mechanisms.

In this Perspective, we propose a powerful unified framework that can demonstrate all of the properties by utilizing a small number of weight consolidation parameters in deep neural networks.

In addition, we are able to draw many parallels between the behaviours and mechanisms of our proposed framework and those surrounding human learning, such as memory loss or sleep deprivation.

This Perspective serves as a conduit for two-way inspiration to further understand lifelong learning in machines and humans.

Drawing parallels with human learning, we propose a unified framework to exhibit many lifelong learning abilities in neural networks by utilizing a small number of weight consolidation parameters.

A Unified Framework for Lifelong Learning in Deep Neural Networks

unified framework
neural network
learning
lifelong


Limited angle CT reconstruction is an under-determined linear inverse problem that requires appropriate regularization techniques to be solved.

In this work we study how pre-trained generative adversarial networks (GANs) can be used to clean noisy, highly artifact laden reconstructions from conventional techniques, by effectively projecting onto the inferred image manifold.

In particular, we use a robust version of the popularly used GAN prior for inverse problems, based on a recent technique called corruption mimicking, that significantly improves the reconstruction quality.

The proposed approach operates in the image space directly, as a result of which it does not need to be trained or require access to the measurement model, is scanner agnostic, and can work over a wide range of sensing scenarios.

We show that robust GAN priors work better than GAN priors for limited angle CT reconstruction which is a highly under-determined inverse problem.

Improving Limited Angle CT Reconstruction with a Robust GAN Prior

limited angle ct reconstruction
robust gan prior


We propose an effective multitask learning setup for reducing distant supervision noise by leveraging sentence-level supervision.

We show how sentence-level supervision can be used to improve the encoding of individual sentences, and to learn which input sentences are more likely to express the relationship between a pair of entities.

We also introduce a novel neural architecture for collecting signals from multiple input sentences, which combines the benefits of attention and maxpooling.

The proposed method increases AUC by 10% (from 0.261 to 0.284), and outperforms recently published results on the FB-NYT dataset.

A new form of attention that works well for the distant supervision setting, and a multitask learning approach to add sentence-level annotations.

Reducing Distant Supervision Noise with Maxpooled Attention and Sentence-Level Supervision

attention
distant supervision
sentence-level


The attention layer in a neural network model provides insights into the model’s reasoning behind its prediction, which are usually criticized for being opaque.

Recently, seemingly contradictory viewpoints have emerged about the interpretability of attention weights (Jain & Wallace, 2019; Vig & Belinkov, 2019).

Amid such confusion arises the need to understand attention mechanism more systematically.

In this work, we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not).

Through a series of experiments on diverse NLP tasks, we validate our observations and reinforce our claim of interpretability of attention through manual evaluation.

Analysis of attention mechanism across diverse NLP tasks.

Attention Interpretability Across NLP Tasks


Generative models forsource code are an interesting structured prediction problem, requiring to reason about both hard syntactic and semantic constraints as well as about natural, likely programs.

We present a novel model for this problem that uses a graph to represent the intermediate state of the generated output.

Our model generates code by interleaving grammar-driven expansion steps with graph augmentation and neural message passing steps.

An experimental evaluation shows that our new model can generate semantically meaningful expressions, outperforming a range of strong baselines.

Representing programs as graphs including semantics helps when generating programs

Generative Code Modeling with Graphs


The inference of models, prediction of future symbols, and entropy rate estimation of discrete-time, discrete-event processes is well-worn ground.

However, many time series are better conceptualized as continuous-time, discrete-event processes.

Here, we provide new methods for inferring models, predicting future symbols, and estimating the entropy rate of continuous-time, discrete-event processes.

The methods rely on an extension of Bayesian structural inference that takes advantage of neural network’s universal approximation power.

Based on experiments with simple synthetic data, these new methods seem to be competitive with state-of- the-art methods for prediction and entropy rate estimation as long as the correct model is inferred.

A new method for inferring a model of, estimating the entropy rate of, and predicting continuous-time, discrete-event processes.

INFERENCE, PREDICTION, AND ENTROPY RATE OF CONTINUOUS-TIME, DISCRETE-EVENT PROCESSES

entropy rate
continuous-time discrete-event process


Recent studies show that convolutional neural networks (CNNs) are vulnerable under various settings, including adversarial examples, backdoor attacks, and distribution shifting.

Motivated by the findings that human visual system pays more attention to global structure (e.g., shape) for recognition while CNNs are biased towards local texture features in images, we propose a unified framework EdgeGANRob based on robust edge features to improve the robustness of CNNs in general, which first explicitly extracts shape/structure features from a given image and then reconstructs a new image by refilling the texture information with a trained generative adversarial network (GAN).

In addition, to reduce the sensitivity of edge detection algorithm to adversarial perturbation, we propose a robust edge detection approach Robust Canny based on the vanilla Canny algorithm.

To gain more insights, we also compare EdgeGANRob with its simplified backbone procedure EdgeNetRob, which performs learning tasks directly on the extracted robust edge features.

We find that EdgeNetRob can help boost model robustness significantly but at the cost of the clean model accuracy.

EdgeGANRob, on the other hand, is able to improve clean model accuracy compared with EdgeNetRob and without losing the robustness benefits introduced by EdgeNetRob.

Extensive experiments show that EdgeGANRob is resilient in different learning tasks under diverse settings.

A unified  model  to improve model robustness against multiple tasks

Shape Features Improve General Model Robustness


Learning rich representation from data is an important task for deep generative models such as variational auto-encoder (VAE).

However, by extracting high-level abstractions in the bottom-up inference process, the goal of preserving all factors of variations for top-down generation is compromised.

Motivated by the concept of “starting small”, we present a strategy to progressively learn independent hierarchical representations from high- to low-levels of abstractions.

The model starts with learning the most abstract representation, and then progressively grow the network architecture to introduce new  representations at different levels of abstraction.

We quantitatively demonstrate the ability of the presented model to improve disentanglement in comparison to existing works on two benchmark datasets using three disentanglement metrics, including a new metric we proposed to complement the previously-presented metric of mutual information gap.

We further present both qualitative and quantitative evidence on how the progression of learning improves disentangling of hierarchical representations.

By drawing on the respective advantage of hierarchical representation learning and progressive learning, this is to our knowledge the first attempt to improve disentanglement by progressively growing the capacity of VAE to learn hierarchical representations.

We proposed a progressive learning method to improve learning and disentangling latent representations at different levels of abstraction.

PROGRESSIVE LEARNING AND DISENTANGLEMENT OF HIERARCHICAL REPRESENTATIONS


Deep latent variable models are powerful tools for representation learning.

In this paper, we adopt the deep information bottleneck model, identify its shortcomings and propose a model that circumvents them.

To this end, we apply a copula transformation which, by restoring the invariance properties of the information bottleneck method, leads to disentanglement of the features in the latent space.

Building on that, we show how this transformation translates to sparsity of the latent space in the new model.

We evaluate our method on artificial and real data.

We apply the copula transformation to the Deep Information Bottleneck which leads to restored invariance properties and a disentangled latent space with superior predictive capabilities.

Learning Sparse Latent Representations with the Deep Copula Information Bottleneck

deep
information bottleneck
copula
latent


State-of-the-art machine learning methods exhibit limited compositional generalization.

At the same time, there is a lack of realistic benchmarks that comprehensively measure this ability, which makes it challenging to find and evaluate improvements.

We introduce a novel method to systematically construct such benchmarks by maximizing compound divergence while guaranteeing a small atom divergence between train and test sets, and we quantitatively compare this method to other approaches for creating compositional generalization benchmarks.

We present a large and realistic natural language question answering dataset that is constructed according to this method, and we use it to analyze the compositional generalization ability of three machine learning architectures.

We find that they fail to generalize compositionally and that there is a surprisingly strong negative correlation between compound divergence and accuracy.

We also demonstrate how our method can be used to create new compositionality benchmarks on top of the existing SCAN dataset, which confirms these findings.

Benchmark and method to measure compositional generalization by maximizing divergence of compound frequency at small divergence of atom frequency.

Measuring Compositional Generalization: A Comprehensive Method on Realistic Data

measure compositional generalization
method


To understand how object vision develops in infancy and childhood, it will be necessary to develop testable computational models.

Deep neural networks (DNNs) have proven valuable as models of adult vision, but it is not yet clear if they have any value as models of development.

As a first model, we measured learning in a DNN designed to mimic the architecture and representational geometry of the visual system (CORnet).

We quantified the development of explicit object representations at each level of this network through training by freezing the convolutional layers and training an additional linear decoding layer.

We evaluate decoding accuracy on the whole ImageNet validation set, and also for individual visual classes.

CORnet, however, uses supervised training and because infants have only extremely impoverished access to labels they must instead learn in an unsupervised manner.

We therefore also measured learning in a state-of-the-art unsupervised network (DeepCluster).

CORnet and DeepCluster differ in both supervision and in the convolutional networks at their heart, thus to isolate the effect of supervision, we ran a control experiment in which we trained the convolutional network from DeepCluster (an AlexNet variant) in a supervised manner.

We make predictions on how learning should develop across brain regions in infants.

In all three networks, we also tested for a relationship in the order in which infants and machines acquire visual classes, and found only evidence for a counter-intuitive relationship.

We discuss the potential reasons for this.

Unsupervised networks learn from bottom up; machines and infants acquire visual classes in different orders

Layerwise Learning Rates for Object Features in Unsupervised and Supervised Neural Networks And Consequent Predictions for the Infant Visual System


We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training.

The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training.

A simple argument then suggests that gradient descent may happen mostly in this subspace.

We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.

For classification problems with k classes, we show that the gradient tends to live in a tiny, slowly-evolving subspace spanned by the eigenvectors corresponding to the k-largest eigenvalues of the Hessian.

Gradient Descent Happens in a Tiny Subspace


Answering questions that require multi-hop reasoning at web-scale necessitates retrieving multiple evidence documents, one of which often has little lexical or semantic relationship to the question.

This paper introduces a new graph-based recurrent retrieval approach that learns to retrieve reasoning paths over the Wikipedia graph to answer multi-hop open-domain questions.

Our retriever model trains a recurrent neural network that learns to sequentially retrieve evidence paragraphs in the reasoning path by conditioning on the previously retrieved documents.

Our reader model ranks the reasoning paths and extracts the answer span included in the best reasoning path.

Experimental results show state-of-the-art results in three open-domain QA datasets, showcasing the effectiveness and robustness of our method.

Notably, our method achieves significant improvement in HotpotQA, outperforming the previous best model by more than 14 points.

Graph-based recurrent retriever that learns to retrieve reasoning paths over Wikipedia Graph outperforms the most recent state of the art on HotpotQA by more than 14 points.

Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering

learn retrieve reasoning path wikipedia graph


Stereo matching is one of the important basic tasks in the computer vision field.

In recent years, stereo matching algorithms based on deep learning have achieved excellent performance and become the mainstream research direction.

Existing algorithms generally use deep convolutional neural networks (DCNNs) to extract more abstract semantic information, but we believe that the detailed information of the spatial structure is more important for stereo matching tasks.

Based on this point of view, this paper proposes a shallow feature extraction network with a large receptive field.

The network consists of three parts: a primary feature extraction module, an atrous spatial pyramid pooling (ASPP) module and a feature fusion module.

The primary feature extraction network contains only three convolution layers.

This network utilizes the basic feature extraction ability of the shallow network to extract and retain the detailed information of the spatial structure.

In this paper, the dilated convolution and atrous spatial pyramid pooling (ASPP) module is introduced to increase the size of receptive field.

In addition, a feature fusion module is designed, which integrates the feature maps with multiscale receptive fields and mutually complements the feature information of different scales.

We replaced the feature extraction part of the existing stereo matching algorithms with our shallow feature extraction network, and achieved state-of-the-art performance on the KITTI 2015 dataset.

Compared with the reference network, the number of parameters is reduced by 42%, and the matching accuracy is improved by 1.9%.

We introduced a shallow featrue extraction network with a large receptive field for stereo matching tasks, which uses a simple structure to get better performance.

A shallow feature extraction network with a large receptive field for stereo matching tasks

extraction network large receptive field stereo matching


We propose a new output layer for deep neural networks that permits the use of logged contextual bandit feedback for training.

Such contextual bandit feedback can be available in huge quantities (e.g., logs of search engines, recommender systems) at little cost, opening up a path for training deep networks on orders of magnitude more data.

To this effect, we propose a Counterfactual Risk Minimization (CRM) approach for training deep networks using an equivariant empirical risk estimator with variance regularization, BanditNet, and show how the resulting objective can be decomposed in a way that allows Stochastic Gradient Descent (SGD) training.

We empirically demonstrate the effectiveness of the method by showing how deep networks -- ResNets in particular -- can be trained for object recognition without conventionally labeled images.

The paper proposes a new output layer for deep networks that permits the use of logged contextual bandit feedback for training.

Deep Learning with Logged Bandit Feedback


Protein classification is responsible for the biological sequence, we came up with an idea whichdeals with the classification of proteomics using deep learning algorithm.

This algorithm focusesmainly to classify sequences of protein-vector which is used for the representation of proteomics.Selection  of  the  type  protein  representation  is  challenging  based  on  which  output  in  terms  ofaccuracy is depended on, The protein representation used here is n-gram i.e.  3-gram and Kerasembedding used for biological sequences like protein.

In this paper we are working on the Proteinclassification to show the strength and representation of biological sequence of the proteins

Protein Family Classification using Deep Learning

protein family classification deep learning


In this work, we attempt to answer a critical question: whether there exists some input sequence that will cause a well-trained discrete-space neural network sequence-to-sequence (seq2seq)  model to generate egregious outputs (aggressive, malicious, attacking, etc.).

And if such inputs exist, how to find them efficiently.

We adopt an empirical methodology, in which we first create lists of egregious output sequences, and then design a discrete optimization algorithm to find input sequences that will cause the model to generate them.

Moreover, the optimization algorithm is enhanced for large vocabulary search and constrained to search for input sequences that are likely to be input by real-world users.

In our experiments, we apply this approach to  dialogue response generation models trained on three real-world dialogue data-sets: Ubuntu, Switchboard and OpenSubtitles, testing whether the model can generate malicious responses.

We demonstrate that given the trigger inputs our algorithm finds, a significant number of malicious sentences are assigned large probability by the model, which reveals an undesirable consequence of standard seq2seq training.

This paper aims to provide an empirical answer to the question of whether well-trained dialogue response model can output malicious responses.

Detecting Egregious Responses in Neural Sequence-to-sequence Models


In the problem of unsupervised learning of disentangled representations, one of the promising methods is to penalize the total correlation of sampled latent vari-ables.

Unfortunately, this well-motivated strategy often fail to achieve disentanglement due to a problematic difference between the sampled latent representation and its corresponding mean representation.

We provide a theoretical explanation that low total correlation of sample distribution cannot guarantee low total correlation of the  mean representation.

We prove that for the mean representation of arbitrarily high total correlation, there exist distributions of latent variables of abounded total correlation.

However, we still believe that total correlation could be a key to the disentanglement of unsupervised representative learning, and we propose a remedy,  RTC-VAE, which rectifies the total correlation penalty.

Experiments show that our model has a more reasonable distribution of the mean representation compared with baseline models, e.g.,β-TCVAE and FactorVAE.

diagnosed all the problem of STOA VAEs theoretically and qualitatively

RTC-VAE: HARNESSING THE PECULIARITY OF TOTAL CORRELATION  IN LEARNING DISENTANGLED REPRESENTATIONS


Visual attention mechanisms have been widely used in image captioning models.

In this paper, to better link the image structure with the generated text, we replace the traditional softmax attention mechanism by two alternative sparsity-promoting transformations: sparsemax and Total-Variation Sparse Attention (TVmax).

With sparsemax, we obtain sparse attention weights, selecting relevant features.

In order to promote sparsity and encourage fusing of the related adjacent spatial locations, we propose TVmax.

By selecting relevant groups of features, the TVmax transformation improves interpretability.

We present results in the Microsoft COCO and Flickr30k datasets, obtaining gains in comparison to softmax.

TVmax outperforms the other compared attention mechanisms in terms of human-rated caption quality and attention relevance.

We propose a new sparse and structured attention mechanism, TVmax, which promotes sparsity and encourages the weight of related adjacent locations to be the same.

Sparse and Structured Visual Attention


Although there are more than 65,000 languages in the world, the pronunciations of many phonemes sound similar across the languages.

When people learn a foreign language, their pronunciation often reflect their native language's characteristics.

That motivates us to investigate how the speech synthesis network learns the pronunciation when multi-lingual dataset is given.

In this study, we train the speech synthesis network bilingually in English and Korean, and analyze how the network learns the relations of phoneme pronunciation between the languages.

Our experimental result shows that the learned phoneme embedding vectors are located closer if their pronunciations are similar across the languages.

Based on the result, we also show that it is possible to train networks that synthesize English speaker's Korean speech and vice versa.

In another experiment, we train the network with limited amount of English dataset and large Korean dataset, and analyze the required amount of dataset to train a resource-poor language with the help of resource-rich languages.

Learned phoneme embeddings of multilingual neural speech synthesis network could represent relations of phoneme pronunciation between the languages.

Learning pronunciation from a foreign language in speech synthesis networks

speech synthesis network
pronunciation
language
learn


Generative Adversarial Networks (GANs) have recently achieved impressive results for many real-world applications, and many GAN variants have emerged with improvements in sample quality and training stability.

However, visualization and understanding of GANs is largely missing.

How does a GAN represent our visual world internally?

What causes the artifacts in GAN results?

How do architectural choices affect GAN learning?

Answering such questions could enable us to develop new insights and better models.

In this work, we present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level.

We first identify a group of interpretable units that are closely related to object concepts with a segmentation-based network dissection method.

Then, we quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.

Finally, we examine the contextual relationship between these units and their surrounding by inserting the discovered object concepts into new images.

We show several practical applications enabled by our framework, from comparing internal representations across different layers, models, and datasets, to improving GANs by locating and removing artifact-causing units, to interactively manipulating objects in the scene.

We provide open source interpretation tools to help peer researchers and practitioners better understand their GAN models.

GAN representations are examined in detail, and sets of representation units are found that control the generation of semantic concepts in the output.

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks


Historically, the pursuit of efficient inference has been one of the driving forces be-hind the research into new deep learning architectures and building blocks.

Some of the recent examples include:  the squeeze-and-excitation module of (Hu et al.,2018), depthwise separable convolutions in Xception (Chollet, 2017), and the inverted bottleneck in MobileNet v2 (Sandler et al., 2018).

Notably, in all of these cases, the resulting building blocks enabled not only higher efficiency, but also higher accuracy, and found wide adoption in the field.

In this work, we further expand the arsenal of efficient building blocks for neural network architectures; but instead of combining standard primitives (such as convolution), we advocate for the replacement of these dense primitives with their sparse counterparts.

While the idea of using sparsity to decrease the parameter count is not new (Mozer & Smolensky, 1989), the conventional wisdom is that this reduction in theoretical FLOPs does not translate into real-world efficiency gains.

We aim to correct this misconception by introducing a family of efficient sparse kernels for several hardware platforms, which we plan to open-source for the benefit of the community.

Equipped with our efficient implementation of sparse primitives,  we show that sparse versions of MobileNet  v1 and MobileNet  v2  architectures  substantially outperform strong dense baselines on the efficiency-accuracy curve.

On Snapdragon 835 our sparse networks outperform their dense equivalents by 1.1−2.2x – equivalent to approximately one entire generation of improvement.

We hope that our findings will facilitate wider adoption of sparsity as a tool for creating efficient and accurate deep learning architectures.

Sparse MobileNets are faster than Dense ones with the appropriate kernels.

Fast Sparse ConvNets


In this work, we address the semi-supervised classification of graph data, where the categories of those unlabeled nodes are inferred from labeled nodes as well as graph structures.

Recent works often solve this problem with the advanced graph convolution in a conventional supervised manner, but the performance could be heavily affected when labeled data is scarce.

Here we propose a Graph Inference Learning (GIL) framework to boost the performance of node classification by learning the inference of node labels on graph topology.

To bridge the connection of two nodes, we formally define a structure relation by encapsulating node attributes, between-node paths and local topological structures together, which can make inference conveniently deduced from one node to another node.

For learning the inference process, we further introduce meta-optimization on structure relations from training nodes to validation nodes, such that the learnt graph inference capability can be better self-adapted into test nodes.

Comprehensive evaluations on four benchmark datasets (including Cora, Citeseer, Pubmed and NELL) demonstrate the superiority of our GIL when compared with other state-of-the-art methods in the semi-supervised node classification task.

We propose a novel graph inference learning framework by building structure relations to infer unknown node labels from those labeled nodes in an end-to-end way.

Graph inference learning for semi-supervised classification


This paper proposes a method for efficient training of Q-function for continuous-state Markov Decision Processes (MDP), such that the traces of the resulting policies satisfy a Linear Temporal Logic (LTL) property.

LTL, a modal logic, can express a wide range of time-dependent logical properties including safety and liveness.

We convert the LTL property into a limit deterministic Buchi automaton with which a synchronized product MDP is constructed.

The control policy is then synthesised by a reinforcement learning algorithm assuming that no prior knowledge is available from the MDP.

The proposed method is evaluated in a numerical study to test the quality of the generated control policy and is compared against conventional methods for policy synthesis such as MDP abstraction (Voronoi quantizer) and approximate dynamic programming (fitted value iteration).

As safety is becoming a critical notion in machine learning we believe that this work can act as a foundation for a number of research directions such as safety-aware learning algorithms.

Logically-Constrained Neural Fitted Q-iteration


We introduce two approaches for conducting efficient Bayesian inference in stochastic simulators containing nested stochastic sub-procedures, i.e., internal procedures for which the density cannot be calculated directly such as rejection sampling loops.

The resulting class of simulators are used extensively throughout the sciences and can be interpreted as probabilistic generative models.

However, drawing inferences from them poses a substantial challenge due to the inability to evaluate even their unnormalised density, preventing the use of many standard inference procedures like Markov Chain Monte Carlo (MCMC).

To address this, we introduce inference algorithms based on a two-step approach that first approximates the conditional densities of the individual sub-procedures, before using these approximations to run MCMC methods on the full program.

Because the sub-procedures can be dealt with separately and are lower-dimensional than that of the overall problem, this two-step process allows them to be isolated and thus be tractably dealt with, without placing restrictions on the overall dimensionality of the problem.

We demonstrate the utility of our approach on a simple, artificially constructed simulator.

We introduce two approaches for efficient and scalable inference in stochastic simulators for which the density cannot be evaluated directly due to, for example, rejection sampling loops.

Efficient Bayesian Inference for Nested Simulators