We develop end-to-end learned reconstructions for lensless mask-based cameras, including an experimental system for capturing aligned lensless and lensed images for training.

Various reconstruction methods are explored, on a scale from classic iterative approaches (based on the physical imaging model) to deep learned methods with many learned parameters.

In the middle ground, we present several variations of unrolled alternating direction method of multipliers (ADMM) with varying numbers of learned parameters.

The network structure combines knowledge of the physical imaging model with learned parameters updated from the data, which compensate for artifacts caused by physical approximations.

Our unrolled approach is 20X faster than classic methods and produces better reconstruction quality than both the classic and deep methods on our experimental system.

We improve the reconstruction time and quality on an experimental mask-based lensless imager using an end-to-end learning approach which incorporates knowledge of the imaging model.

Unrolled, model-based networks for lensless imaging


Deep learning, a rebranding of deep neural network research works, has achieved a remarkable success in recent years.

With multiple hidden layers, deep learning models aim at computing the hierarchical feature representations of the observational data.

Meanwhile, due to its severe disadvantages in data consumption, computational resources, parameter tuning costs and the lack of result explainability, deep learning has also suffered from lots of criticism.

In this paper, we will introduce a new representation learning model, namely “Sample-Ensemble Genetic Evolutionary Network” (SEGEN), which can serve as an alternative approach to deep learning models.

Instead of building one single deep model, based on a set of sampled sub-instances, SEGEN adopts a genetic-evolutionary learning strategy to build a group of unit models generations by generations.

The unit models incorporated in SEGEN can be either traditional machine learning models or the recent deep learning models with a much “narrower” and “shallower” architecture.

The learning results of each instance at the final generation will be effectively combined from each unit model via diffusive propagation and ensemble learning strategies.

From the computational perspective, SEGEN requires far less data, fewer computational resources and parameter tuning efforts, but has sound theoretic interpretability of the learning process and results.

Extensive experiments have been done on several different real-world benchmark datasets, and the experimental results obtained by SEGEN have demonstrated its advantages over the state-of-the-art representation learning models.

We introduce a new representation learning model, namely “Sample-Ensemble Genetic Evolutionary Network” (SEGEN), which can serve as an alternative approach to deep learning models.

SEGEN: SAMPLE-ENSEMBLE GENETIC EVOLUTIONARY NETWORK MODEL

sample-ensemble genetic evolutionary network


How can we teach artificial agents to use human language flexibly to solve problems in a real-world environment?

We have one example in nature of agents being able to solve this problem: human babies eventually learn to use human language to solve problems, and they are taught with an adult human-in-the-loop.

Unfortunately, current machine learning methods (e.g. from deep reinforcement learning) are too data inefficient to learn a language in this way (3).

An outstanding goal is finding an algorithm with a suitable ‘language learning prior’ that allows it to learn human language, while minimizing the number of required human interactions.

In this paper, we propose to learn such a prior in simulation, leveraging the increasing amount of available compute for machine learning experiments (1).

We call our approach Learning to Learn to Communicate (L2C).

Specifically, in L2C we train a meta-learning agent in simulation to interact with populations of pre-trained agents, each with their own distinct communication protocol.

Once the meta-learning agent is able to quickly adapt to each population of agents, it can be deployed in new populations unseen during training, including populations of humans.

To show the promise of the L2C framework, we conduct some preliminary experiments in a Lewis signaling game (4), where we show that agents

trained with L2C are able to learn a simple form of human language (represented by a hand-coded compositional language) in fewer iterations than randomly initialized agents.

We propose to use meta-learning for more efficient language learning, via a kind of 'domain randomization'.

Learning to learn to communicate


We study the problem of fitting task-specific learning rate schedules from the perspective of hyperparameter optimization.

This allows us to explicitly search for schedules that achieve good generalization.

We describe the structure of the gradient of a validation error w.r.t.

the learning rates, the hypergradient, and based on this we introduce a novel online algorithm.

Our method adaptively interpolates between two recently proposed techniques (Franceschi et al., 2017; Baydin et al.,2018), featuring increased stability and faster convergence.

We show empirically that the proposed technique compares favorably with baselines and related methodsin terms of final test accuracy.

MARTHE: a new method to fit task-specific learning rate schedules from the perspective of hyperparameter optimization

Scheduling the Learning Rate Via Hypergradients: New Insights and a New Algorithm


Recent years have witnessed some exciting developments in the domain of generating images from scene-based text descriptions.

These approaches have primarily focused on generating images from a static text description and are limited to generating images in a single pass.

They are unable to generate an image interactively based on an incrementally additive text description (something that is more intuitive and similar to the way we describe an image).

We propose a method to generate an image incrementally based on a sequence of graphs of scene descriptions (scene-graphs).

We propose a recurrent network architecture that preserves the image content generated in previous steps and modifies the cumulative image as per the newly provided scene information.

Our model utilizes Graph Convolutional Networks (GCN) to cater to variable-sized scene graphs along with Generative Adversarial image translation networks to generate realistic multi-object images without needing any intermediate supervision during training.

We experiment with Coco-Stuff dataset which has multi-object images along with annotations describing the visual scene and show that our model significantly outperforms other approaches on the same dataset in generating visually consistent images for incrementally growing scene graphs.

Interactively generating image from incrementally growing scene graphs in multiple steps using GANs while preserving the contents of image generated in previous steps

Interactive Image Generation Using Scene Graphs


In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images.

However, most Convolutional Neural Networks (CNNs) for image classification were developed using biased datasets that contain large objects, in mostly central image positions.

To assess whether classical CNN architectures work well for tiny object classification we build a comprehensive testbed containing two datasets: one derived from MNIST digits and one from histopathology images.

This testbed allows controlled experiments to stress-test CNN architectures with a broad spectrum of signal-to-noise ratios.

Our observations indicate that: (1) There exists a limit to signal-to-noise below which CNNs fail to generalize and that this limit is affected by dataset size - more data leading to better performances; however, the amount of training data required for the model to generalize scales rapidly with the inverse of the object-to-image ratio (2) in general, higher capacity models exhibit better generalization; (3) when knowing the approximate object sizes, adapting receptive field is beneficial; and (4) for very small signal-to-noise ratio the choice of global pooling operation affects optimization, whereas for relatively large signal-to-noise values, all tested global pooling operations exhibit similar performance.

We study low- and very-low-signal-to-noise classification scenarios, where objects that correlate with class label occupy tiny proportion of the entire image (e.g. medical or hyperspectral imaging).

Needles in Haystacks: On Classifying Tiny Objects in Large Images


Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block.

Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019) showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks.

This raises the question: do learned attention layers operate similarly to convolutional layers?

This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice.

Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer.

Our numerical experiments then show that self-attention layers attend to pixel-grid patterns similarly to CNN layers, corroborating our analysis.

A self-attention layer can perform convolution and often learns to do so in practice.

On the Relationship between Self-Attention and Convolutional Layers


We introduce a “learning-based” algorithm for the low-rank decomposition problem: given an $n \times d$ matrix $A$, and a parameter $k$, compute a rank-$k$ matrix $A'$ that minimizes the approximation loss $||A- A'||_F$.

The algorithm uses a training set of input matrices in order to optimize its performance.

Specifically, some of the most efficient approximate algorithms for computing low-rank approximations proceed by computing a projection $SA$, where $S$ is a sparse random $m \times n$ “sketching matrix”, and then performing the singular value decomposition of $SA$.

We show how to replace the random matrix $S$ with a “learned” matrix of the same sparsity to reduce the error.

Our experiments show that, for multiple types of data sets, a learned sketch matrix can substantially reduce the approximation loss compared to a random matrix $S$, sometimes by one order of magnitude.

We also study mixed matrices where only some of the rows are trained and the remaining ones are random, and show that matrices still offer improved performance while retaining worst-case guarantees.

Learning-based algorithms can improve upon the performance of classical algorithms for the low-rank approximation problem while retaining the worst-case guarantee.

Learning-Based Low-Rank Approximations


Neural conversational models are widely used in applications like personal assistants and chat bots.

These models seem to give better performance when operating on word level.

However, for fusion languages like French, Russian and Polish vocabulary size sometimes become infeasible since most of the words have lots of word forms.

We propose a neural network architecture for transforming normalized text into a grammatically correct one.

Our model efficiently employs correspondence between normalized and target words and significantly outperforms character-level models while being 2x faster in training and 20\% faster at evaluation.

We also propose a new pipeline for building conversational models: first generate a normalized answer and then transform it into a grammatically correct one using our network.

The proposed pipeline gives better performance than character-level conversational models according to assessor testing.

Proposed architecture to solve morphological agreement task

Achieving morphological agreement with Concorde


This paper proposes the use of spectral element methods \citep{canuto_spectral_1988} for fast and accurate training of Neural Ordinary Differential Equations (ODE-Nets; \citealp{Chen2018NeuralOD}) for system identification.

This is achieved by expressing their dynamics as a truncated series of Legendre polynomials.

The series coefficients, as well as the network weights, are computed by minimizing the weighted sum of the loss function and the violation of the ODE-Net dynamics.

The problem is solved by coordinate descent that alternately minimizes, with respect to the coefficients and the weights, two unconstrained sub-problems using standard backpropagation and gradient methods.

The resulting optimization scheme is fully time-parallel and results in a low memory footprint.

Experimental comparison to standard methods, such as backpropagation through explicit solvers and the adjoint technique \citep{Chen2018NeuralOD}, on training surrogate models of small and medium-scale dynamical systems shows that it is at least one order of magnitude faster at reaching a comparable value of the loss function.

The corresponding testing MSE is one order of magnitude smaller as well, suggesting generalization capabilities increase.

This paper proposes the use of spectral element methods for fast and accurate training of Neural Ordinary Differential Equations for system identification.

SNODE: Spectral Discretization of Neural ODEs for System Identification


Exploration in sparse reward reinforcement learning remains an open challenge.

Many state-of-the-art methods use intrinsic motivation to complement the sparse extrinsic reward signal, giving the agent more opportunities to receive feedback during exploration.

Commonly these signals are added as bonus rewards, which results in a mixture policy that neither conducts exploration nor task fulfillment resolutely.

In this paper, we instead learn separate intrinsic and extrinsic task policies and schedule between these different drives to accelerate exploration and stabilize learning.

Moreover, we introduce a new type of intrinsic reward denoted as successor feature control (SFC), which is general and not task-specific.

It takes into account statistics over complete trajectories and thus differs from previous methods that only use local information to evaluate intrinsic motivation.

We evaluate our proposed scheduled intrinsic drive (SID) agent using three different environments with pure visual inputs: VizDoom, DeepMind Lab and DeepMind Control Suite.

The results show a substantially improved exploration efficiency with SFC and the hierarchical usage of the intrinsic drives.

A video of our experimental results can be found at https://gofile.io/?c=HpEwTd

A new intrinsic reward signal based on successor features and a novel way to combine extrinsic and intrinsic reward.

Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration


Meta-Reinforcement learning approaches aim to develop learning procedures that can adapt quickly to a distribution of tasks with the help of a few examples.

Developing efficient exploration strategies capable of finding the most useful samples becomes critical in such settings.

Existing approaches to finding efficient exploration strategies add auxiliary objectives to promote exploration by the pre-update policy, however, this makes the adaptation using a few gradient steps difficult as the pre-update (exploration) and post-update (exploitation) policies are quite different.

Instead, we propose to explicitly model a separate exploration policy for the task distribution.

Having two different policies gives more flexibility in training the exploration policy and also makes adaptation to any specific task easier.

We show that using self-supervised or supervised learning objectives for adaptation stabilizes the training process and also demonstrate the superior performance of our model compared to prior works in this domain.

We propose to use a separate exploration policy to collect the pre-adaptation trajectories in MAML. We also show that using a self-supervised objective in the inner loop leads to more stable training and much better performance.

Learning Exploration Policies for Model-Agnostic Meta-Reinforcement Learning


The “Supersymmetric Artificial Neural Network” in deep learning (denoted (x; θ, bar{θ})Tw), espouses the importance of considering biological constraints in the aim of further generalizing backward propagation.

Looking at the progression of ‘solution geometries’; going from SO(n) representation (such as Perceptron like models) to SU(n) representation (such as UnitaryRNNs) has guaranteed richer and richer representations in weight space of the artificial neural network, and hence better and better hypotheses were generatable.

The Supersymmetric Artificial Neural Network explores a natural step forward, namely SU(m|n) representation.

These supersymmetric biological brain representations (Perez et al.) can be represented by supercharge compatible special unitary notation SU(m|n), or (x; θ, bar{θ})Tw parameterized by θ, bar{θ}, which are supersymmetric directions, unlike θ seen in the typical non-supersymmetric deep learning model.

Notably, Supersymmetric values can encode or represent more information than the typical deep learning model, in terms of “partner potential” signals for example.

Generalizing backward propagation, using formal methods from supersymmetry.


Regularization-based continual learning approaches generally prevent catastrophic forgetting by augmenting the training loss with an auxiliary objective.

However in most practical optimization scenarios with noisy data and/or gradients, it is possible that stochastic gradient descent can inadvertently change critical parameters.

In this paper, we argue for the importance of regularizing optimization trajectories directly.

We derive a new co-natural gradient update rule for continual learning whereby the new task gradients are preconditioned with the empirical Fisher information of previously learnt tasks.

We show that using the co-natural gradient systematically reduces forgetting in continual learning.

Moreover, it helps combat overfitting when learning a new task in a low resource scenario.

Regularizing the optimization trajectory with the Fisher information of old tasks reduces catastrophic forgetting greatly

Regularizing Trajectories to Mitigate Catastrophic Forgetting

catastrophic forgetting
trajectory
regularize


We study the problem of generating source code in a strongly typed,

Java-like programming language, given a label (for example a set of

API calls or types) carrying a small amount of information about the

The generated programs are expected to respect a

`"realistic" relationship between programs and labels, as exemplified

by a corpus of labeled programs available during training.

Two challenges in such *conditional program generation* are that

the generated programs must satisfy a rich set of syntactic and

semantic constraints, and that source code contains many low-level

a neural generator not on code but on *program sketches*, or

models of program syntax that abstract out names and operations that

posterior distribution over sketches, then concretize samples from

this distribution into type-safe programs using combinatorial

We implement our ideas in a system for generating

API-heavy Java code, and show that it can often predict the entire

body of a method given just a few API calls or data types that appear

We give a method for generating type-safe programs in a Java-like language, given a small amount of syntactic information about the desired code.

Neural Sketch Learning for Conditional Program Generation


We propose an approach for sequence modeling based on autoregressive normalizing flows.

Each autoregressive transform, acting across time, serves as a moving reference frame for modeling higher-level dynamics.

This technique provides a simple, general-purpose method for improving sequence modeling, with connections to existing and classical techniques.

We demonstrate the proposed approach both with standalone models, as well as a part of larger sequential latent variable models.

Results are presented on three benchmark video datasets, where flow-based dynamics improve log-likelihood performance over baseline models.

We show how autoregressive flows can be used to improve sequential latent variable models.

Improving Sequential Latent Variable Models with Autoregressive Flows


It is well-known that many machine learning models are susceptible to adversarial attacks, in which an attacker evades a classifier by making small perturbations to inputs.

This paper discusses how industrial copyright detection tools, which serve a central role on the web, are susceptible to adversarial attacks.

We discuss a range of copyright detection systems, and why they are particularly vulnerable to attacks.

These vulnerabilities are especially apparent for neural network based systems.

As proof of concept, we describe a well-known music identification method and implement this system in the form of a neural net.

We then attack this system using simple gradient methods.

Adversarial music created this way successfully fools industrial systems, including the AudioTag copyright detector and YouTube's Content ID system.

Our goal is to raise awareness of the threats posed by adversarial examples in this space and to highlight the importance of hardening copyright detection systems to attacks.

Adversarial examples can fool YouTube's copyright detection system

Adversarial Attacks on Copyright Detection Systems


Equilibrium Propagation (EP) is a learning algorithm that bridges Machine Learning and Neuroscience, by computing gradients closely matching those of Backpropagation Through Time (BPTT), but with a learning rule local in space.

Given an input x and associated target y, EP proceeds in two phases: in the first phase neurons evolve freely towards a first steady state; in the second phase output neurons are nudged towards y until they reach a second steady state.

However, in existing implementations of EP, the learning rule is not local in time:

the weight update is performed after the dynamics of the second phase have converged and requires information of the first phase that is no longer available physically.

This is a major impediment to the biological plausibility of EP and its efficient hardware implementation.

In this work, we propose a version of EP named Continual Equilibrium Propagation (C-EP) where neuron and synapse dynamics occur simultaneously throughout the second phase, so that the weight update becomes local in time.

We prove theoretically that, provided the learning rates are sufficiently small, at each time step of the second phase the dynamics of neurons and synapses follow the gradients of the loss given by BPTT (Theorem 1).

We demonstrate training with C-EP on MNIST and generalize C-EP to neural networks where neurons are connected by asymmetric connections.

We show through experiments that the more the network updates follows the gradients of BPTT, the best it performs in terms of training.

These results bring EP a step closer to biology while maintaining its intimate link with backpropagation.

We propose a continual version of Equilibrium Propagation, where neuron and synapse dynamics occur simultaneously throughout the second phase, with theoretical guarantees and numerical simulations.

Equilibrium Propagation with Continual Weight Updates


There are two main lines of research on visual reasoning: neural module network (NMN) with explicit multi-hop reasoning through handcrafted neural modules, and monolithic network with implicit reasoning in the latent feature space.

The former excels in interpretability and compositionality, while the latter usually achieves better performance due to model flexibility and parameter efficiency.

In order to bridge the gap of the two, we present Meta Module Network (MMN), a novel hybrid approach that can efficiently utilize a Meta Module to perform versatile functionalities, while preserving compositionality and interpretability through modularized design.

The proposed model first parses an input question into a functional program through a Program Generator.

Instead of handcrafting a task-specific network to represent each function like traditional NMN, we use Recipe Encoder to translate the functions into their corresponding recipes (specifications), which are used to dynamically instantiate the Meta Module into Instance Modules.

To endow different instance modules with designated functionality, a Teacher-Student framework is proposed, where a symbolic teacher pre-executes against the scene graphs to provide guidelines for the instantiated modules (student) to follow.

In a nutshell, MMN adopts the meta module to increase its parameterization efficiency, and uses recipe encoding to improve its generalization ability over NMN.

Experiments conducted on the GQA benchmark demonstrates that: (1) MMN achieves significant improvement over both NMN and monolithic network baselines; (2) MMN is able to generalize to unseen but related functions.

We propose a new Meta Module Network to resolve some of the restrictions of previous Neural Module Network to achieve strong performance on realistic visual reasoning dataset.

Meta Module Network for Compositional Visual Reasoning


We propose a new perspective on adversarial attacks against deep reinforcement learning agents.

Our main contribution is CopyCAT, a targeted attack able to consistently lure an agent into following an outsider's policy.

It is pre-computed, therefore fast inferred, and could thus be usable in a real-time scenario.

We show its effectiveness on Atari 2600 games in the novel read-only setting.

In the latter, the adversary cannot directly modify the agent's state -its representation of the environment- but can only attack the agent's observation -its perception of the environment.

Directly modifying the agent's state would require a write-access to the agent's inner workings and we argue that this assumption is too strong in realistic settings.

We propose a new attack for taking full control of neural policies in realistic settings.

CopyCAT: Taking Control of Neural Policies with Constant Attacks


Cold-start and efficiency issues of the Top-k recommendation are critical to large-scale recommender systems.

Previous hybrid recommendation methods are effective to deal with the cold-start issues by extracting real latent factors of cold-start items(users) from side information, but they still suffer low efficiency in online recommendation caused by the expensive similarity search in real latent space.

This paper presents a collaborative generated hashing (CGH) to improve the efficiency by denoting users and items as binary codes, which applies to various settings: cold-start users, cold-start items and warm-start ones.

Specifically, CGH is designed to learn hash functions of users and items through the Minimum Description Length (MDL) principle; thus, it can deal with various recommendation settings.

In addition, CGH initiates a new marketing strategy through mining potential users by a generative step.

To reconstruct effective users, the MDL principle is used to learn compact and informative binary codes from the content data.

Extensive experiments on two public datasets show the advantages for recommendations in various settings over competing baselines and analyze the feasibility of the application in marketing.

It can generate effective hash codes for efficient cold-start recommendation and meanwhile provide a feasible marketing strategy.

Collaborative Generated Hashing for Market Analysis and Fast Cold-start Recommendation


Recent efforts to combine Representation Learning with Formal Methods, commonly known as the Neuro-Symbolic Methods, have given rise to a new trend of applying rich neural architectures to solve classical combinatorial optimization problems.

In this paper, we propose a neural framework that can learn to solve the Circuit Satisfiability problem.

Our framework is built upon two fundamental contributions: a rich embedding architecture that encodes the problem structure and an end-to-end differentiable training procedure that mimics Reinforcement Learning and trains the model directly toward solving the SAT problem.

The experimental results show the superior out-of-sample generalization performance of our framework compared to the recently developed NeuroSAT method.

We propose a neural framework that can learn to solve the Circuit Satisfiability problem from (unlabeled) circuit instances.

Learning To Solve Circuit-SAT: An Unsupervised Differentiable Approach


Sequence generation models such as recurrent networks can be trained with a diverse set of learning algorithms.

For example, maximum likelihood learning is simple and efficient, yet suffers from the exposure bias problem.

Reinforcement learning like policy gradient addresses the problem but can have prohibitively poor exploration efficiency.

A variety of other algorithms such as RAML, SPG, and data noising, have also been developed in different perspectives.

This paper establishes a formal connection between these algorithms.

We present a generalized entropy regularized policy optimization formulation, and show that the apparently divergent algorithms can all be reformulated as special instances of the framework, with the only difference being the configurations of reward function and a couple of hyperparameters.

The unified interpretation offers a systematic view of the varying properties of exploration and learning efficiency.

Besides, based on the framework, we present a new algorithm that dynamically interpolates among the existing algorithms for improved learning.

Experiments on machine translation and text summarization demonstrate the superiority of the proposed algorithm.

A unified perspective of various learning algorithms for sequence generation, such as MLE, RL, RAML, data noising, etc.

Connecting the Dots Between MLE and RL for Sequence Generation


We are reporting the SHINRA project, a project for structuring Wikipedia with collaborative construction scheme.

The goal of the project is to create a huge and well-structured knowledge base to be used in NLP applications, such as QA, Dialogue systems and explainable NLP systems.

It is created based on a scheme of ”Resource by Collaborative Contribution (RbCC)”.

We conducted a shared task of structuring Wikipedia, and at the same, submitted results are used to construct a knowledge base.

There are machine readable knowledge bases such as CYC, DBpedia, YAGO, Freebase Wikidata and so on, but each of them has problems to be solved.

CYC has a coverage problem, and others have a coherence problem due to the fact that these are based on Wikipedia and/or created by many but inherently incoherent crowd workers.

In order to solve the later problem, we started a project for structuring Wikipedia using automatic knowledge base construction shared-task.

The automatic knowledge base construction shared-tasks have been popular and well studied for decades.

However, these tasks are designed only to compare the performances of different systems, and to find which system ranks the best on limited test data.

The results of the participated systems are not shared and the systems may be abandoned once the task is over.

We believe this situation can be improved by the following changes:

1. designing the shared-task to construct knowledge base rather than evaluating only limited test data

2. making the outputs of all the systems open to public so that we can run ensemble learning to create the better results than the best systems

3. repeating the task so that we can run the task with the larger and better training data from the output of the previous task (bootstrapping and active learning)

We conducted “SHINRA2018” with the above mentioned scheme and in this paper

we report the results and the future directions of the project.

The task is to extract the values of the pre-defined attributes from Wikipedia pages.

We have categorized most of the entities in Japanese Wikipedia (namely 730 thousand entities) into the 200 ENE categories.

Based on this data, the shared-task is to extract the values of the attributes from Wikipedia pages.

We gave out the 600 training data and the participants are required to submit the attribute-values for all remaining entities of the same category type.

Then 100 data out of them for each category are used to evaluate the system output in the shared-task.

We conducted a preliminary ensemble learning on the outputs and found 15 F1 score improvement on a category and the average of 8 F1 score improvements on all 5 categories we tested over a strong baseline.

Based on this promising results, we decided to conduct three tasks in 2019; multi-lingual categorization task (ML), extraction for the same 5 categories in Japanese with a larger training data (JP-5) and extraction for 34 new categories in Japanese (JP-34).

We introduce a "Resource by Collaborative Construction" scheme to create KB, structured Wikipedia

SHINRA: Structuring Wikipedia by Collaborative Contribution


Recent image super-resolution(SR) studies leverage very deep convolutional neural networks and the rich hierarchical features they offered, which leads to better reconstruction performance than conventional methods.

However, the small receptive fields in the up-sampling and reconstruction process of those models stop them to take full advantage of global contextual information.

This causes problems for further performance improvement.

In this paper, inspired by image reconstruction principles of human visual system, we propose an image super-resolution global reasoning network (SRGRN) to effectively learn the correlations between different regions of an image, through global reasoning.

Specifically, we propose global reasoning up-sampling module (GRUM) and global reasoning reconstruction block (GRRB).

They construct a graph model to perform relation reasoning on regions of low resolution (LR) images.They aim to reason the interactions between different regions in the up-sampling and reconstruction process and thus leverage more contextual information to generate accurate details.

Our proposed SRGRN are more robust and can handle low resolution images that are corrupted by multiple types of degradation.

Extensive experiments on different benchmark data-sets show that our model outperforms other state-of-the-art methods.

Also our model is lightweight and consumes less computing power, which makes it very suitable for real life deployment.

A state-of-the-art model based on global reasoning for image super-resolution

Global reasoning network for image super-resolution


Graph Neural Networks (GNNs) have received tremendous attention recently due to their power in handling graph data for different downstream tasks across different application domains.

The key of GNN is its graph convolutional filters, and recently various kinds of filters are designed.

However, there still lacks in-depth analysis on (1) Whether there exists a best filter that can perform best on all graph data; (2) Which graph properties will influence the optimal choice of graph filter; (3) How to design appropriate filter adaptive to the graph data.

In this paper, we focus on addressing the above three questions.

We first propose a novel assessment tool to evaluate the effectiveness of graph convolutional filters for a given graph.

Using the assessment tool, we find out that there is no single filter as a `silver bullet' that perform the best on all possible graphs.

In addition, different graph structure properties will influence the optimal graph convolutional filter's design choice.

Based on these findings, we develop Adaptive Filter Graph Neural Network (AFGNN), a simple but powerful model that can adaptively learn task-specific filter.

For a given graph, it leverages graph filter assessment as regularization and learns to combine from a set of base filters.

Experiments on both synthetic and real-world benchmark datasets demonstrate that our proposed model can indeed learn an appropriate filter and perform well on graph tasks.

Propose an assessment framework to analyze and learn graph convolutional filter

Demystifying Graph Neural Network Via Graph Filter Assessment


The advance of node pooling operations in Graph Neural Networks (GNNs) has lagged behind the feverish design of new message-passing techniques, and pooling remains an important and challenging endeavor for the design of deep architectures.

In this paper, we propose a pooling operation for GNNs that leverages a differentiable unsupervised loss based on the minCut optimization objective.

For each node, our method learns a soft cluster assignment vector that depends on the node features, the target inference task (e.g., a graph classification loss), and, thanks to the minCut objective, also on the connectivity structure of the graph.

Graph pooling is obtained by applying the matrix of assignment vectors to the adjacency matrix and the node features.

We validate the effectiveness of the proposed pooling method on a variety of supervised and unsupervised tasks.

A new pooling layer for GNNs that learns how to pool nodes, according to their features, the graph connectivity, and the dowstream task objective.

Mincut Pooling in Graph Neural Networks


Knowledge graphs are structured representations of real world facts.

However, they typically contain only a small subset of all possible facts.

Link prediction is the task of inferring missing facts based on existing ones.

We propose TuckER, a relatively simple yet powerful linear model based on Tucker decomposition of the binary tensor representation of knowledge graph triples.

By using this particular decomposition, parameters are shared between relations, enabling multi-task learning.

TuckER outperforms previous state-of-the-art models across several standard link prediction datasets.

We propose TuckER, a relatively simple but powerful linear model for link prediction in knowledge graphs, based on Tucker decomposition of the binary tensor representation of knowledge graph triples.

TuckER: Tensor Factorization for Knowledge Graph Completion


With innovations in architecture design, deeper and wider neural network models deliver improved performance on a diverse variety of tasks.

But the increased memory footprint of these models presents a challenge during training, when all intermediate layer activations need to be stored for back-propagation.

Limited GPU memory forces practitioners to make sub-optimal choices: either train inefficiently with smaller batches of examples; or limit the architecture to have lower depth and width, and fewer layers at higher spatial resolutions.

This work introduces an approximation strategy that significantly reduces a network's memory footprint during training, but has negligible effect on training performance and computational expense.

During the forward pass, we replace activations with lower-precision approximations immediately after they have been used by subsequent layers, thus freeing up memory.

The approximate activations are then used during the backward pass.

This approach limits the accumulation of errors across the forward and backward pass---because the forward computation across the network still happens at full precision, and the approximation has a limited effect when computing gradients to a layer's input.

Experiments, on CIFAR and ImageNet, show that using our approach with 8- and even 4-bit fixed-point approximations of 32-bit floating-point activations has only a minor effect on training and validation performance, while affording significant savings in memory usage.

An algorithm to reduce the amount of memory required for training deep networks, based on an approximation strategy.

Backprop with Approximate Activations for Memory-efficient Network Training


Estimating the frequencies of elements in a data stream is a fundamental task in data analysis and machine learning.

The problem is typically addressed using streaming algorithms which can process very large data using limited storage.

Today's streaming algorithms, however, cannot exploit patterns in their input to improve performance.

We propose a new class of algorithms that automatically learn relevant patterns in the input data and use them to improve its frequency estimates.

The proposed algorithms combine the benefits of machine learning with the formal guarantees available through algorithm theory.

We prove that our learning-based algorithms have lower estimation errors than their non-learning counterparts.

We also evaluate our algorithms on two real-world datasets and demonstrate empirically their performance gains.

Data stream algorithms can be improved using deep learning, while retaining performance guarantees.

Learning-Based Frequency Estimation Algorithms


Link prediction in simple graphs is a fundamental problem in which new links between nodes are predicted based on the observed structure of the graph.

However, in many real-world applications, there is a need to model relationships among nodes which go beyond pairwise associations.

For example, in a chemical reaction, relationship among the reactants and products is inherently higher-order.

Additionally, there is need to represent the direction from reactants to products.

Hypergraphs provide a natural way to represent such complex higher-order relationships.

Even though Graph Convolutional Networks (GCN) have recently emerged as a powerful deep learning-based approach for link prediction over simple graphs, their suitability for link prediction in hypergraphs is unexplored -- we fill this gap in this paper and propose Neural Hyperlink Predictor (NHP). NHP adapts GCNs for link prediction in hypergraphs. We propose two variants of NHP --NHP-U and NHP-D -- for link prediction over undirected and directed hypergraphs, respectively.

To the best of our knowledge, NHP-D is the first method for link prediction over directed hypergraphs.

Through extensive experiments on multiple real-world datasets, we show NHP's effectiveness.

We propose Neural Hyperlink Predictor (NHP). NHP adapts graph convolutional networks for link prediction in hypergraphs

Link Prediction in Hypergraphs using Graph Convolutional Networks

link prediction hypergraph
graph convolutional network


In this paper, we present a method for adversarial decomposition of text representation.

This method can be used to decompose a representation of an input sentence into several independent vectors, where each vector is responsible for a specific aspect of the input sentence.

We evaluate the proposed method on two case studies: the conversion between different social registers and diachronic language change.

We show that the proposed method is capable of fine-grained con- trolled change of these aspects of the input sentence.

For example, our model is capable of learning a continuous (rather than categorical) representation of the style of the sentence, in line with the reality of language use.

The model uses adversarial-motivational training and includes a special motivational loss, which acts opposite to the discriminator and encourages a better decomposition.

Finally, we evaluate the obtained meaning embeddings on a downstream task of para- phrase detection and show that they are significantly better than embeddings of a regular autoencoder.

A method which learns separate representations for the meaning and the form of a sentence

Adversarial Decomposition of Text Representation


We were approached by a group of healthcare providers who are involved in the care of chronic patients looking for potential technologies to facilitate the process of reviewing patient-generated data during clinical visits.

Aiming at understanding the healthcare providers' attitudes towards reviewing patient-generated data, we (1) conducted a focus group with a mixed group of healthcare providers.

Next, to gain the patients' perspectives, we (2) interviewed eight chronic patients, collected a sample of their data and designed a series of visualizations representing patient data we collected.

Last, we (3) sought feedback on the visualization designs from healthcare providers who requested this exploration.

We found four factors shaping patient-generated data: data & context, patient's motivation, patient's time commitment, and patient's support circle.

Informed by the results of our studies, we discussed the importance of designing patient-generated visualizations for individuals by considering both patient and healthcare provider rather than designing with the purpose of generalization and provided guidelines for designing future patient-generated data visualizations.

We explored the visualization designs that can support chronic patients to present and review their health data with healthcare providers during clinical visits.

Exploring the Design of Patient-Generated Data Visualizations


Recently,  neural-network  based  forward  dynamics  models  have  been  proposed that  attempt  to  learn  the  dynamics  of  physical  systems  in  a  deterministic  way.

While near-term motion can be predicted accurately, long-term predictions suffer from accumulating input and prediction errors which can lead to plausible but different trajectories that diverge from the ground truth.

A system that predicts distributions of the future physical states for long time horizons based on its uncertainty is thus a promising solution.

In this work, we introduce a novel robust Monte Carlo sampling based graph-convolutional dropout method that allows us to sample multiple plausible trajectories for an initial state given a neural-network based forward dynamics predictor.

By introducing a new shape preservation loss and training our dynamics model recurrently, we stabilize long-term predictions.

We show that our model’s long-term forward dynamics prediction errors on complicated physical interactions of rigid and deformable objects of various shapes are significantly lower than existing strong baselines.

Lastly, we demonstrate how generating multiple trajectories with our Monte Carlo dropout method can be used to train model-free reinforcement learning agents faster and to better solutions on simple manipulation tasks.

We propose a stochastic differentiable forward dynamics predictor that is able to sample multiple physically plausible trajectories under the same initial input state and show that it can be used to train model-free policies more efficiently.

Stochastic Neural Physics Predictor


There has been a large amount of interest, both in the past and particularly recently, into the relative advantage of different families of universal function approximators, for instance neural networks, polynomials, rational functions, etc.

However, current research has focused almost exclusively on understanding this problem in a worst case setting: e.g. characterizing the best L1 or L_{infty} approximation in a box (or sometimes, even under an adversarially constructed data distribution.)

In this setting many classical tools from approximation theory can be effectively used.

However, in typical applications we expect data to be high dimensional, but structured -- so, it would only be important to approximate the desired function well on the relevant part of its domain, e.g. a small manifold on which real input data actually lies.

Moreover, even within this domain the desired quality of approximation may not be uniform; for instance in classification problems, the approximation needs to be more accurate near the decision boundary.

These issues, to the best of our knowledge, have remain unexplored until now.

With this in mind, we analyze the performance of neural networks and polynomial kernels in a natural regression setting where the data enjoys sparse latent structure, and the labels depend in a simple way on the latent variables.

We give an almost-tight theoretical analysis of the performance of both neural networks and polynomials for this problem, as well as verify our theory with simulations.

Our results both involve new (complex-analytic) techniques, which may be of independent interest, and show substantial qualitative differences with what is known in the worst-case setting.

Beyond-worst-case analysis of the representational power of  ReLU nets & polynomial kernels  -- in particular in the presence of sparse latent structure.

The Comparative Power of ReLU Networks and Polynomial Kernels in the Presence of Sparse Latent Structure

presence sparse latent structure
power
polynomial kernel
relu


Recent deep generative models can provide photo-realistic images as well as visual or textual content embeddings useful to address various tasks of computer vision and natural language processing.

Their usefulness is nevertheless often limited by the lack of control over the generative process or the poor understanding of the learned representation.

To overcome these major issues, very recent works have shown the interest of studying the semantics of the latent space of generative models.

In this paper, we propose to advance on the interpretability of the latent space of generative models by introducing a new method to find meaningful directions in the latent space of any generative model along which we can move to control precisely specific properties of the generated image like position or scale of the object in the image.

Our method is weakly supervised and particularly well suited for the search of directions encoding simple transformations of the generated image, such as translation, zoom or color variations.

We demonstrate the effectiveness of our method qualitatively and quantitatively, both for GANs and variational auto-encoders.

A model to control the generation of images with GAN and beta-VAE with regard to scale and position of the objects

Controlling generative models with continuous factors of variations


Modern neural network architectures use structured linear transformations, such as low-rank matrices, sparse matrices, permutations, and the Fourier transform, to improve inference speed and reduce memory usage compared to general linear maps.

However, choosing which of the myriad structured transformations to use (and its associated parameterization) is a laborious task that requires trading off speed, space, and accuracy.

We consider a different approach: we introduce a family of matrices called kaleidoscope matrices (K-matrices) that provably capture any structured matrix with near-optimal space (parameter) and time (arithmetic operation) complexity.

We empirically validate that K-matrices can be automatically learned within end-to-end pipelines to replace hand-crafted procedures, in order to improve model quality.

For example, replacing channel shuffles in ShuffleNet improves classification accuracy on ImageNet by up to 5%.

Learnable K-matrices can also simplify hand-engineered pipelines---we replace filter bank feature computation in speech data preprocessing with a kaleidoscope layer, resulting in only 0.4% loss in accuracy on the TIMIT speech recognition task.

K-matrices can also capture latent structure in models: for a challenging permuted image classification task, adding a K-matrix to a standard convolutional architecture can enable learning the latent permutation and improve accuracy by over 8 points.

We provide a practically efficient implementation of our approach, and use K-matrices in a Transformer network to attain 36% faster end-to-end inference speed on a language translation task.

We propose a differentiable family of "kaleidoscope matrices," prove that all structured matrices can be represented in this form, and use them to replace hand-crafted linear maps in deep learning models.

Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps


We propose a method, called Label Embedding Network, which can learn label representation (label embedding) during the training process of deep networks.

With the proposed method, the label embedding is adaptively and automatically learned through back propagation.

The original one-hot represented loss function is converted into a new loss function with soft distributions, such that the originally unrelated labels have continuous interactions with each other during the training process.

As a result, the trained model can achieve substantially higher accuracy and with faster convergence speed.

Experimental results based on competitive tasks demonstrate the effectiveness of the proposed method, and the learned label embedding is reasonable and interpretable.

The proposed method achieves comparable or even better results than the state-of-the-art systems.

Learning Label Representation for Deep Networks

Label Embedding Network: Learning Label Representation for Soft Training of Deep Networks


Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters.

As current approaches are limited by network bandwidth, we propose the use of communication compression in the decentralized training context.

We show that Choco-SGD achieves linear speedup in the number of workers for arbitrary high compression ratios on general non-convex functions, and non-IID training data.

We demonstrate the practical performance of the algorithm in two key scenarios: the training of deep learning models

(i) over decentralized user devices, connected by a peer-to-peer network and

We propose Choco-SGD---decentralized SGD with compressed communication---for non-convex objectives and show its strong performance in various deep learning applications (on-device learning, datacenter case).

Decentralized Deep Learning with Arbitrary Communication Compression


We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier obtained by a risk-sensitive perturbation of the weights of a learned classifier.

Entropy-SGD works by optimizing the bound’s prior, violating the hypothesis of the PAC-Bayes theorem that the prior is chosen independently of the data.

Indeed, available implementations of Entropy-SGD rapidly obtain zero training error on random labels and the same holds of the Gibbs posterior.

In order to obtain a valid generalization bound, we show that an ε-differentially private prior yields a valid PAC-Bayes bound, a straightforward consequence of results connecting generalization with differential privacy.

Using stochastic gradient Langevin dynamics (SGLD) to approximate the well-known exponential release mechanism, we observe that generalization error on MNIST (measured on held out data) falls within the (empirically nonvacuous) bounds computed under the assumption that SGLD produces perfect samples.

In particular, Entropy-SGLD can be configured to yield relatively tight generalization bounds and still fit real labels, although these same settings do not obtain state-of-the-art performance.

We show that Entropy-SGD optimizes the prior of a PAC-Bayes bound, violating the requirement that the prior be independent of data; we use differential privacy to resolve this and improve generalization.

Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy

entropy-sgd optimize prior pac-bayes bind


In this paper, we investigate learning the deep neural networks for automated optical inspection in industrial manufacturing.

Our preliminary result has shown the stunning performance improvement by transfer learning from the completely dissimilar source domain: ImageNet.

Further study for demystifying this improvement shows that the transfer learning produces a highly compressible network, which was not the case for the network learned from scratch.

The experimental result shows that there is a negligible accuracy drop in the network learned by transfer learning until it is compressed to 1/128 reduction of the number of convolution ﬁlters.

This result is contrary to the compression without transfer learning which loses more than 5% accuracy at the same compression rate.

We experimentally show that transfer learning makes sparse features in the network and thereby produces a more compressible network.

Efﬁcient Neural Network Compression via Transfer Learning for Industrial Optical Inspection


The key challenge in semi-supervised learning is how to effectively leverage unlabeled data to improve learning performance.

The classical label propagation method, despite its popularity, has limited modeling capability in that it only exploits graph information for making predictions.

In this paper, we consider label propagation from a graph signal processing perspective and decompose it into three components: signal, filter, and classifier.

By extending the three components, we propose a simple generalized label propagation (GLP) framework for semi-supervised learning.

GLP naturally integrates graph and data feature information, and offers the flexibility of selecting appropriate filters and domain-specific classifiers for different applications.

Interestingly, GLP also provides new insight into the popular graph convolutional network and elucidates its working mechanisms.

Extensive experiments on three citation networks, one knowledge graph, and one image dataset demonstrate the efficiency and effectiveness of GLP.

We extend the classical label propation methods to jointly model graph and feature information from a graph filtering perspective, and show connections to the graph convlutional networks.

Generalized Label Propagation Methods for Semi-Supervised Learning


Because the choice and tuning of the optimizer affects the speed, and ultimately the performance of deep learning, there is significant past and recent research in this area.

Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of optimization strategies for deep learning.

We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization.

As the primary contribution, we present DeepOBS, a Python package of deep learning optimization benchmarks.

The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking.

The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for image classification on ImageNet or character-level language prediction models, as well as popular classics like MNIST and CIFAR-10.

The package also provides realistic baseline results for the most popular optimizers on these test problems, ensuring a fair comparison to the competition when benchmarking new optimizers, and without having to run costly experiments.

It comes with output back-ends that directly produce LaTeX code for inclusion in academic publications.

It supports TensorFlow and is available open source.

We provide a software package that drastically simplifies, automates, and improves the evaluation of deep learning optimizers.

DeepOBS: A Deep Learning Optimizer Benchmark Suite


Pre-trained word embeddings are the primary

method for transfer learning in several Natural Language Processing (NLP) tasks.

works have focused on using unsupervised

techniques such as language modeling to obtain these embeddings.

focuses on extracting representations from

multiple pre-trained supervised models, which

enriches word embeddings with task and domain specific knowledge.

Experiments performed in cross-task, cross-domain and crosslingual settings indicate that such supervised

embeddings are helpful, especially in the lowresource setting, but the extent of gains is dependent on the nature of the task and domain.

extract contextual embeddings from off-the-shelf supervised model. Helps downstream NLP models in low-resource settings

Supervised Contextual Embeddings for Transfer Learning in Natural Language Processing Tasks


We build a theoretical framework for understanding practical meta-learning methods that enables the integration of sophisticated formalizations of task-similarity with the extensive literature on online convex optimization and sequential prediction algorithms in order to provide within-task performance guarantees.

Our approach improves upon recent analyses of parameter-transfer by enabling the task-similarity to be learned adaptively and by improving transfer-risk bounds in the setting of statistical learning-to-learn.

It also leads to straightforward derivations of average-case regret bounds for efficient algorithms in settings where the task-environment changes dynamically or the tasks share a certain geometric structure.

Practical adaptive algorithms for gradient-based meta-learning with provable guarantees.

Adaptive Gradient-Based Meta-Learning Methods


In this work, we propose a self-supervised method to learn sentence representations with an injection of linguistic knowledge.

Multiple linguistic frameworks propose diverse sentence structures from which semantic meaning might be expressed out of compositional words operations.

We aim to take advantage of this linguist diversity and learn to represent sentences by contrasting these diverse views.

Formally, multiple views of the same sentence are mapped to close representations.

On the contrary, views from other sentences are mapped further.

By contrasting different linguistic views, we aim at building embeddings which better capture semantic and which are less sensitive to the sentence outward form.

We aim to exploit the diversity of linguistic structures to build sentence representations.

Sentence embedding with contrastive multi-views learning


The peripheral nervous system represents the input/output system for the brain.

Cuff electrodes implanted on the peripheral nervous system allow observation and control over this system, however, the data produced by these electrodes have a low signal-to-noise ratio and a complex signal content.

In this paper, we consider the analysis of neural data recorded from the vagus nerve in animal models, and develop an unsupervised learner based on convolutional neural networks that is able to simultaneously de-noise and cluster regions of the data by signal content.

Unsupervised analysis of data recorded from the peripheral nervous system denoises and categorises signals.

Coordinate-VAE: Unsupervised clustering and de-noising of peripheral nervous system data

peripheral nervous system
unsupervise
datum


Adversarial attacks on convolutional neural networks (CNN) have gained significant attention and there have been active research efforts on defense mechanisms.

Stochastic input transformation methods have been proposed, where the idea is to recover the image from adversarial attack by random transformation, and to take the majority vote as consensus among the random samples.

However, the transformation improves the accuracy on adversarial images at the expense of the accuracy on clean images.

While it is intuitive that the accuracy on clean images would deteriorate, the exact mechanism in which how this occurs is unclear.

In this paper, we study the distribution of softmax induced by stochastic transformations.

We observe that with random transformations on the clean images, although the mass of the softmax distribution could shift to the wrong class, the resulting distribution of softmax could be used to correct the prediction.

Furthermore, on the adversarial counterparts, with the image transformation, the resulting shapes of the distribution of softmax are similar to the distributions from the clean images.

With these observations, we propose a method to improve existing transformation-based defenses.

We train a separate lightweight distribution classifier to recognize distinct features in the distributions of softmax outputs of transformed images.

Our empirical studies show that our distribution classifier, by training on distributions obtained from clean images only, outperforms majority voting for both clean and adversarial images.

Our method is generic and can be integrated with existing transformation-based defenses.

We enhance existing transformation-based defenses by using a distribution classifier on the distribution of softmax obtained from transformed images.

Enhancing Transformation-Based Defenses Against Adversarial Attacks with a Distribution Classifier

distribution classifier
transformation-based defense
enhance


Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings.

We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep.

This attention mechanism enables more effective and scalable learning in complex multi-agent environments, when compared to recent approaches.

Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, and it makes no assumptions about the action spaces of the agents.

As such, it is flexible enough to be applied to most multi-agent learning problems

We propose an approach to learn decentralized policies in multi-agent settings using attention-based critics and demonstrate promising results in environments with complex interactions.

Actor-Attention-Critic for Multi-Agent Reinforcement Learning


The recent expansion of machine learning applications to molecular biology proved to have a significant contribution to our understanding of biological systems, and genome functioning in particular.

Technological advances enabled the collection of large epigenetic datasets, including information about various DNA binding factors (ChIP-Seq) and DNA spatial structure (Hi-C).

Several studies have confirmed the correlation between DNA binding factors and Topologically Associating Domains (TADs) in DNA structure.

However, the information about physical proximity represented by genomic coordinate was not yet used for the improvement of the prediction models.

In this research, we focus on Machine Learning methods for prediction of folding patterns of DNA in a classical model organism Drosophila melanogaster.

The paper considers linear models with four types of regularization, Gradient Boosting and Recurrent Neural Networks for the prediction of chromatin folding patterns from epigenetic marks.

The bidirectional LSTM RNN model outperformed all the models and gained the best prediction scores.

This demonstrates the utilization of complex models and the importance of memory of sequential DNA states for the chromatin folding.

We identify informative epigenetic features that lead to the further conclusion of their biological significance.

We apply RNN to solve the biological problem of chromatin folding patterns prediction from epigenetic marks and demonstrate for the first time that utilization of memory of sequential states on DNA molecule is significant for the best performance.

Learning DNA folding patterns with Recurrent Neural Networks


Previous work showed empirically that large neural networks can be significantly reduced in size while preserving their accuracy.

Model compression became a central research topic, as it is crucial for deployment of neural networks on devices with limited computational and memory resources.

The majority of the compression methods are based on heuristics and offer no worst-case guarantees on the trade-off between the compression rate and the approximation error for an arbitrarily new sample.

We propose the first efficient, data-independent neural pruning algorithm with a provable trade-off between its compression rate and the approximation error for any future test sample.

Our method is based on the coreset framework, which finds a small weighted subset of points that provably approximates the original inputs.

Specifically, we approximate the output of a layer of neurons by a coreset of neurons in the previous layer and discard the rest.

We apply this framework in a layer-by-layer fashion from the top to the bottom.

Unlike previous works, our coreset is data independent, meaning that it provably guarantees the accuracy of the function for any input $x\in \mathbb{R}^d$, including an adversarial one.

We demonstrate the effectiveness of our method on popular network architectures.

In particular, our coresets yield 90% compression of the LeNet-300-100 architecture on MNIST while improving the accuracy.

We propose an efficient, provable and data independent method for network compression via neural pruning using coresets of neurons -- a novel construction proposed in this paper.

Data-Independent Neural Pruning via Coresets


Planning problems in partially observable environments cannot be solved directly with convolutional networks and require some form of memory.

But, even memory networks with sophisticated addressing schemes are unable to learn intelligent reasoning satisfactorily due to the complexity of simultaneously learning to access memory and plan.

To mitigate these challenges we propose the Memory Augmented Control Network (MACN).

The network splits planning into a hierarchical process.

At a lower level, it learns to plan in a locally observed space.

At a higher level, it uses a collection of policies computed on locally observed spaces to learn an optimal plan in the global environment it is operating in.

The performance of the network is evaluated on path planning tasks in environments in the presence of simple and complex obstacles and in addition, is tested for its ability to generalize to new environments not seen in the training set.

Memory Augmented Network to plan in partially observable environments.

Memory Augmented Control Networks


Contextualized word representations such as ELMo and BERT have become the de facto starting point for incorporating pretrained representations for downstream NLP tasks.

In these settings, contextual representations have largely made obsolete their static embedding predecessors such as Word2Vec and GloVe.

However, static embeddings do have their advantages in that they are straightforward to understand and faster to use.

Additionally, embedding analysis methods for static embeddings are far more diverse and mature than those available for their dynamic counterparts.

In this work, we introduce simple methods for generating static lookup table embeddings from existing pretrained contextual representations and demonstrate they outperform Word2Vec and GloVe embeddings on a variety of word similarity and word relatedness tasks.

In doing so, our results also reveal insights that may be useful for subsequent downstream tasks using our embeddings or the original contextual models.

Further, we demonstrate the increased potential for analysis by applying existing approaches for estimating social bias in word embeddings.

Our analysis constitutes the most comprehensive study of social bias in contextual word representations (via the proxy of our distilled embeddings) and reveals a number of inconsistencies in current techniques for quantifying social bias in word embeddings.

We publicly release our code and distilled word embeddings to support reproducible research and the broader NLP community.

A procedure for distilling contextual models into static embeddings; we apply our method to 9 popular models and demonstrate clear gains in representation quality wrt Word2Vec/GloVe and improved analysis potential by thoroughly studying social bias.

BERT Wears GloVes: Distilling Static Embeddings from Pretrained Contextual Representations

static embedding
distil
glove
representation
contextual


The brain performs unsupervised learning and (perhaps) simultaneous supervised learning.

This raises the question as to whether a hybrid of supervised and unsupervised methods will produce better learning.

Inspired by the rich space of Hebbian learning rules, we set out to directly learn the unsupervised learning rule on local information that best augments a supervised signal.

We present the Hebbian-augmented training algorithm (HAT) for combining gradient-based learning with an unsupervised rule on pre-synpatic activity, post-synaptic activities, and current weights.

We test HAT's effect on a simple problem (Fashion-MNIST) and find consistently higher performance than supervised learning alone.

This finding provides empirical evidence that unsupervised learning on synaptic activities provides a strong signal that can be used to augment gradient-based methods.

We further find that the meta-learned update rule is a time-varying function; thus, it is difficult to pinpoint an interpretable Hebbian update rule that aids in training.

We do find that the meta-learner eventually degenerates into a non-Hebbian rule that preserves important weights so as not to disturb the learner's convergence.

Metalearning unsupervised update rules for neural networks improves performance and potentially demonstrates how neurons in the brain learn without access to global labels.

Augmenting Supervised Learning by Meta-learning Unsupervised Local Rules


Deep convolutional networks often append additive constant ("bias") terms to their convolution operations, enabling a richer repertoire of functional mappings.

Biases are also used to facilitate training, by subtracting mean response over batches of training images (a component of "batch normalization").

Recent state-of-the-art blind denoising methods seem to require these terms for their success.

Here, however, we show that bias terms used in most CNNs (additive constants, including those used for batch normalization) interfere with the interpretability of these networks, do not help performance, and in fact prevent generalization of performance to noise levels not including in the training data.

In particular, bias-free CNNs (BF-CNNs) are locally linear, and hence amenable to direct analysis with linear-algebraic tools.

These analyses provide interpretations of network functionality in terms of projection onto a union of low-dimensional subspaces, connecting the learning-based method to more traditional denoising methodology.

Additionally, BF-CNNs generalize robustly, achieving near-state-of-the-art performance at noise levels well beyond the range over which they have been trained.

We show that removing constant terms from CNN architectures provides interpretability of the denoising method via linear-algebra techniques and also boosts generalization performance across noise levels.

Interpretable and robust blind image denoising with bias-free convolutional neural networks


The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance.

Yet despite significant empirical and theoretical analysis, relatively little has been proved about the concrete effects of different initialization schemes.

In this work, we analyze the effect of initialization in deep linear networks, and provide for the first time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights.

We show that for deep networks, the width needed for efficient convergence to a global minimum with orthogonal initializations is independent of the depth, whereas the width needed for efficient convergence with Gaussian initializations scales linearly in the depth.

Our results demonstrate how the benefits of a good initialization can persist throughout learning, suggesting an explanation for the recent empirical successes found by initializing very deep non-linear networks according to the principle of dynamical isometry.

We provide for the first time a rigorous proof that orthogonal initialization speeds up convergence relative to Gaussian initialization, for deep linear networks.

Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks

orthogonal initialization
deep linear network


Survival function estimation is used in many disciplines, but it is most common in medical analytics in the form of the Kaplan-Meier estimator.

Sensitive data (patient records) is used in the estimation without any explicit control on the information leakage, which is a significant privacy concern.

We propose a first differentially private estimator of the survival function and show that it can be easily extended to provide differentially private confidence intervals and test statistics without spending any extra privacy budget.

We further provide extensions for differentially private estimation of the competing risk cumulative incidence function.

Using nine real-life clinical datasets, we provide empirical evidence that our proposed method provides good utility while simultaneously providing strong privacy guarantees.

A first differentially private estimate of the survival function

Differentially Private Survival Function Estimation

survival function
differentially private


Neural networks can learn to extract statistical properties from data, but they seldom make use of structured information from the label space to help representation learning.

Although some label structure can implicitly be obtained when training on huge amounts of data, in a few-shot learning context where little data is available, making explicit use of the label structure can inform the model to reshape the representation space to reflect a global sense of class dependencies.

We propose a meta-learning framework, Conditional class-Aware Meta-Learning (CAML), that conditionally transforms feature representations based on a metric space that is trained to capture inter-class dependencies.

This enables a conditional modulation of the feature representations of the base-learner to impose regularities informed by the label space.

Experiments show that the conditional transformation in CAML leads to more disentangled representations and achieves competitive results on the miniImageNet benchmark.

CAML is an instance of MAML with conditional class dependencies.

Learning to Learn with Conditional Class Dependencies


We study the problem of multiset prediction.

The goal of multiset prediction is to train a predictor that maps an input to a multiset consisting of multiple items.

Unlike existing problems in supervised learning, such as classification, ranking and sequence generation, there is no known order among items in a target multiset, and each item in the multiset may appear more than once, making this problem extremely challenging.

In this paper, we propose a novel multiset loss function by viewing this problem from the perspective of sequential decision making.

The proposed multiset loss function is empirically evaluated on two families of datasets, one synthetic and the other real, with varying levels of difficulty, against various baseline loss functions including reinforcement learning, sequence, and aggregated distribution matching loss functions.

The experiments reveal the effectiveness of the proposed loss function over the others.

We study the problem of multiset prediction and propose a novel multiset loss function, providing analysis and empirical evidence that demonstrates its effectiveness.

Loss Functions for Multiset Prediction


Understanding theoretical properties of deep and locally connected nonlinear network, such as deep convolutional neural network (DCNN), is still a hard problem despite its empirical success.

In this paper, we propose a novel theoretical framework for such networks with ReLU nonlinearity.

The framework bridges data distribution with gradient descent rules, favors disentangled representations and is compatible with common regularization techniques such as Batch Norm, after a novel discovery of its projection nature.

The framework is built upon teacher-student setting, by projecting the student's forward/backward pass onto the teacher's computational graph.

We do not impose unrealistic assumptions (e.g., Gaussian inputs, independence of activation, etc).

Our framework could help facilitate theoretical analysis of many practical issues, e.g. disentangled representations in deep networks.

This paper presents a theoretical framework that models data distribution explicitly for deep and locally connected ReLU network

A theoretical framework for deep and locally connected ReLU network

deep locally connect relu network
theoretical framework


Multi-agent cooperation is an important feature of the natural world.

Many tasks involve individual incentives that are misaligned with the common good, yet a wide range of organisms from bacteria to insects and humans are able to overcome their differences and collaborate.

Therefore, the emergence of cooperative behavior amongst self-interested individuals is an important question for the fields of multi-agent reinforcement learning (MARL) and evolutionary theory.

Here, we study a particular class of multi-agent problems called intertemporal social dilemmas (ISDs), where the conflict between the individual and the group is particularly sharp.

By combining MARL with appropriately structured natural selection, we demonstrate that individual inductive biases for cooperation can be learned in a model-free way.

To achieve this, we introduce an innovative modular architecture for deep reinforcement learning agents which supports multi-level selection.

We present results in two challenging environments, and interpret these in the context of cultural and ecological evolution.

We introduce a biologically-inspired modular evolutionary algorithm in which deep RL agents learn to cooperate in a difficult multi-agent social game, which could help to explain the evolution of altruism.

Evolving intrinsic motivations for altruistic behavior


In adversarial attacks to machine-learning classifiers, small perturbations are added to input that is correctly classified.

The perturbations yield adversarial examples, which are virtually indistinguishable from the unperturbed input, and yet are misclassified.

In standard neural networks used for deep learning, attackers can craft adversarial examples from most input to cause a misclassification of their choice.

We introduce a new type of network units, called RBFI units, whose non-linear structure makes them inherently resistant to adversarial attacks.

On permutation-invariant MNIST, in absence of adversarial attacks, networks using RBFI units match the performance of networks using sigmoid units, and are slightly below the accuracy of networks with ReLU units.

When subjected to adversarial attacks based on projected gradient descent or fast gradient-sign methods, networks with RBFI units retain accuracies above 75%, while ReLU or Sigmoid see their accuracies reduced to below 1%.

Further, RBFI networks trained on regular input either exceed or closely match the accuracy of sigmoid and ReLU network trained with the help of adversarial examples.

The non-linear structure of RBFI units makes them difficult to train using standard gradient descent.

We show that RBFI networks of RBFI units can be efficiently trained to high accuracies using pseudogradients, computed using functions especially crafted to facilitate learning instead of their true derivatives.

We introduce a type of neural network that is structurally resistant to adversarial attacks, even when trained on unaugmented training sets.  The resistance is due to the stability of network units wrt input perturbations.

Neural Networks with Structural Resistance to Adversarial Attacks

neural network
adversarial attack
resistance


We show that there exists an inherent tension between the goal of adversarial robustness and that of standard generalization.

Specifically, training robust models may not only be more resource-consuming, but also lead to a reduction of standard accuracy.

We demonstrate that this trade-off between the standard accuracy of a model and its robustness to adversarial perturbations provably exists even in a fairly simple and natural setting.

These findings also corroborate a similar phenomenon observed in practice.

Further, we argue that this phenomenon is a consequence of robust classifiers learning fundamentally different feature representations than  standard classifiers.

These differences, in particular, seem to result in unexpected benefits: the features learned by robust models tend to align better with salient data characteristics and human perception.

We show that adversarial robustness might come at the cost of standard classification performance, but also yields unexpected benefits.

Robustness May Be at Odds with Accuracy


Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples.

In this work, we propose mixup, a simple learning principle to alleviate these issues.

In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels.

By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples.

Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures.

We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.

Training on convex combinations between random training examples and their labels improves generalization in deep neural networks

mixup: Beyond Empirical Risk Minimization


We present a novel approach to spike sorting for high-density multielectrode probes using the Neural Clustering Process (NCP), a recently introduced neural architecture that performs scalable amortized approximate Bayesian inference for efficient probabilistic clustering.

To optimally encode spike waveforms for clustering, we extended NCP by adding a convolutional spike encoder, which is learned end-to-end with the NCP network.

Trained purely on labeled synthetic spikes from a simple generative model, the NCP spike sorting model shows promising performance for clustering multi-channel spike waveforms.

The model provides higher clustering quality than an alternative Bayesian algorithm, finds more spike templates with clear receptive fields on real data and recovers more ground truth neurons on hybrid test data compared to a recent spike sorting algorithm.

Furthermore, NCP is able to handle the clustering uncertainty of ambiguous small spikes by GPU-parallelized posterior sampling.

We present a novel approach to spike sorting using the Neural Clustering Process (NCP), a recently introduced neural architecture that performs scalable amortized approximate Bayesian inference for efficient probabilistic clustering.

Spike Sorting using the Neural Clustering Process


The goal of the paper is to propose an algorithm for learning the most generalizable solution from given training data.

It is shown that Bayesian approach leads to a solution that dependent on statistics of training data and not on particular

The solution is stable under perturbations of training data because it is defined by an integral contribution of multiple maxima of the likelihood and not by a single global maximum.

Specifically, the Bayesian probability distribution

of parameters (weights) of a probabilistic model given by a neural network is estimated via recurrent variational approximations.

Derived recurrent update rules correspond to SGD-type rules for finding a minimum of an effective loss that is an average of an original negative log-likelihood over the Gaussian distributions of weights, which makes it a function of means and variances.

The effective loss is convex for large variances and non-convex in the limit of small variances.

Among stationary solutions of the update rules there are trivial solutions with zero variances at local minima of the original loss and a single non-trivial solution with finite variances that is a critical point at the end of convexity of the effective loss

At the critical point both first- and second-order gradients of the effective loss w.r.t. means are zero.

The empirical study confirms that the critical point represents the most generalizable solution.

the critical point in the weight space depends on specifics of the used probabilistic model some properties at the critical point are universal and model independent.

Proposed method for finding the most generalizable solution that is stable w.r.t. perturbations of trainig data.

VARIATIONAL SGD: DROPOUT , GENERALIZATION AND CRITICAL POINT AT THE END OF CONVEXITY


Humans rely on episodic memory constantly, in remembering the name of someone they met 10 minutes ago, the plot of a movie as it unfolds, or where they parked the car.

Endowing reinforcement learning agents with episodic memory is a key step on the path toward replicating human-like general intelligence.

We analyze why standard RL agents lack episodic memory today, and why existing RL tasks don't require it.

We design a new form of external memory called Masked Experience Memory, or MEM, modeled after key features of human episodic memory.

To evaluate episodic memory we define an RL task based on the common children's game of Concentration.

We find that a MEM RL agent leverages episodic memory effectively to master Concentration, unlike the baseline agents we tested.

Implementing and evaluating episodic memory for RL.

Now I Remember! Episodic Memory For Reinforcement Learning


Parameters are one of the most critical components of machine learning models.

As datasets and learning domains change, it is often necessary and time-consuming to re-learn entire models.

Rather than re-learning the parameters from scratch, replacing learning with optimization, we propose a framework building upon the theory of \emph{optimal transport} to adapt model parameters by discovering correspondences between models and data, significantly amortizing the training cost.

We demonstrate our idea on the challenging problem of creating probabilistic spatial representations for autonomous robots.

Although recent mapping techniques have facilitated robust occupancy mapping, learning all spatially-diverse parameters in such approximate Bayesian models demand considerable computational time, discouraging them to be used in real-world robotic mapping.

Considering the fact that the geometric features a robot would observe with its sensors are similar across various environments, in this paper, we demonstrate how to re-use parameters and hyperparameters learned in different domains.

This adaptation is computationally more efficient than variational inference and Monte Carlo techniques.

A series of experiments conducted on realistic settings verified the possibility of transferring thousands of such parameters with a negligible time and memory cost, enabling large-scale mapping in urban environments.

We present a method of adapting hyperparameters of probabilistic models using optimal transport with applications in robotics

Optimal Transport for Distribution Adaptation in Bayesian Hilbert Maps


An algorithm is introduced for learning a predictive state representation with off-policy temporal difference (TD) learning that is then used to learn to steer a vehicle with reinforcement learning.

There are three components being learned simultaneously:  (1) the off-policy predictions as a compact representation of state, (2) the behavior policy distribution for estimating the off-policy predictions, and (3) the deterministic policy gradient for learning to act.

A behavior policy discriminator is learned and used for estimating the important sampling ratios needed to learn the predictive representation off-policy with general value functions (GVFs).

A linear deterministic policy gradient method is used to train the agent with only the predictive representations while the predictions are being learned.

All three components are combined, demonstrated and evaluated on the problem of steering the vehicle from images in the TORCS racing simulator environment.

Steering from only images is a challenging problem where evaluation is completed on a held-out set of tracks that were never seen during training in order to measure the generalization of the predictions and controller.

Experiments show the proposed method is able to steer smoothly and navigate many but not all of the tracks available in TORCS with performance that exceeds DDPG using only images as input and approaches the performance of an ideal non-vision based kinematics model.

An algorithm to learn a predictive state representation with general value functions and off-policy learning is applied to the problem of vision-based steering in autonomous driving.

Learning an off-policy predictive state representation for deep reinforcement learning for vision-based steering in autonomous driving

vision-based steering autonomous driving


For numerous domains, including for instance earth observation, medical imaging, astrophysics,..., available image and signal datasets often irregular space-time sampling patterns and large missing data rates.

These sampling properties is a critical issue to apply state-of-the-art learning-based (e.g., auto-encoders, CNNs,...) to fully benefit from the available large-scale observations and reach breakthroughs in the reconstruction and identification of processes of interest.

In this paper, we address the end-to-end learning of representations of signals, images and image sequences from irregularly-sampled data, {\em i.e.} when the training data involved missing data.

From an analogy to Bayesian formulation, we consider energy-based representations.

Two energy forms are investigated: one derived from auto-encoders and one relating to Gibbs energies.

The learning stage of these energy-based representations (or priors) involve a joint interpolation issue, which resorts to solving an energy minimization problem under observation constraints.

Using a neural-network-based implementation of the considered energy forms, we can state an end-to-end learning scheme from irregularly-sampled data.

We demonstrate the relevance of the proposed representations for different case-studies: namely, multivariate time series, 2{\sc } images and image sequences.

We address the end-to-end learning of energy-based representations for signal and image observation dataset with irregular sampling patterns.

End-to-end learning of energy-based representations for irregularly-sampled signals and images

end-to-end learning energy-based representation


Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood.

Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively.

This leads to poor sample-efficiency during meta-training as well as ineffective task identification strategies.

This paper provides a theoretical analysis of credit assignment in gradient-based Meta-RL.

Building on the gained insights we develop a novel meta-learning algorithm that overcomes both the issue of poor credit assignment and previous difficulties in estimating meta-policy gradients.

By controlling the statistical distance of both pre-adaptation and adapted policies during meta-policy search, the proposed algorithm endows efficient and stable meta-learning.

Our approach leads to superior pre-adaptation policy behavior and consistently outperforms previous Meta-RL algorithms in sample-efficiency, wall-clock time, and asymptotic performance.

A novel and theoretically grounded meta-reinforcement learning algorithm

ProMP: Proximal Meta-Policy Search


When training a neural network for a desired task, one may prefer to adapt a pretrained network rather than start with a randomly initialized one -- due to lacking enough training data, performing lifelong learning where the system has to learn a new task while being previously trained for other tasks, or wishing to encode priors in the network via preset weights.

The most commonly employed approaches for network adaptation are fine-tuning and using the pre-trained network as a fixed feature extractor, among others.

In this paper we propose a straightforward alternative: Side-Tuning.

Side-tuning adapts a pretrained network by training a lightweight "side" network that is fused with the (unchanged) pre-rained network using a simple additive process.

This simple method works as well as or better than existing solutions while it resolves some of the basic issues with fine-tuning, fixed features, and several other common baselines.

In particular, side-tuning is less prone to overfitting when little training data is available, yields better results than using a fixed feature extractor, and doesn't suffer from catastrophic forgetting in lifelong learning.

We demonstrate the performance of side-tuning under a diverse set of scenarios, including lifelong learning (iCIFAR, Taskonomy), reinforcement learning, imitation learning (visual navigation in Habitat), NLP question-answering (SQuAD v2), and single-task transfer learning (Taskonomy), with consistently promising results.

Side-tuning adapts a pre-trained network by training a lightweight "side" network that is fused with the (unchanged) pre-trained network using a simple additive process.

Side-Tuning: Network Adaptation via Additive Side Networks


In this paper, we present a technique for generating artificial datasets that retain statistical properties of the real data while providing differential privacy guarantees with respect to this data.

We include a Gaussian noise layer in the discriminator of a generative adversarial network to make the output and the gradients differentially private with respect to the training data, and then use the generator component to synthesise privacy-preserving artificial dataset.

Our experiments show that under a reasonably small privacy budget we are able to generate data of high quality and successfully train machine learning models on this artificial data.

Train GANs with differential privacy to generate artificial privacy-preserving datasets.

Generating Differentially Private Datasets Using GANs