In most real-world scenarios, training datasets are highly class-imbalanced, where deep neural networks suffer from generalizing to a balanced testing criterion.

In this paper, we explore a novel yet simple way to alleviate this issue via synthesizing less-frequent classes with adversarial examples of other classes.

Surprisingly, we found this counter-intuitive method can effectively learn generalizable features of minority classes by transferring and leveraging the diversity of the majority information.

Our experimental results on various types of class-imbalanced datasets in image classification and natural language processing show that the proposed method not only improves the generalization of minority classes significantly compared to other re-sampling or re-weighting methods, but also surpasses other methods of state-of-art level for the class-imbalanced classification.

We develop a new method for imbalanced classification using adversarial examples

Proposes a new optimization objective that generates synthetic samples by over-sampling the majority classes instead of minority classes, solving the problem of overfitting minority classes.

The authors propose to tackle imbalance classification using re-sampling methods, showing that adversarial examples in the minority class would help to train a new model that generalizes better.

Imbalanced Classification via Adversarial Minority Over-sampling

minority class
adversarial example
method
classification
new
imbalance


Active matter consists of active agents which transform energy extracted from surroundings into momentum, producing a variety of collective phenomena.

A model, synthetic active system composed of microtubule polymers driven by protein motors spontaneously forms a liquid-crystalline nematic phase.

Extensile stress created by the protein motors precipitates continuous buckling and folding of the microtubules creating motile topological defects and turbulent fluid flows.

Defect motion is determined by the rheological properties of the material; however, these remain largely unquantified.

Measuring defects dynamics can yield fundamental insights into active nematics, a class of materials that include bacterial films and animal cells.

Current methods for defect detection lack robustness and precision, and require fine-tuning for datasets with different visual quality.

In this study, we applied Deep Learning to train a defect detector to automatically analyze microscopy videos of the microtubule active nematic.

Experimental results indicate that our method is robust and accurate.

It is expected to significantly increase the amount of video data that can be processed.

An interesting application of CNN in soft condensed matter physics experiments.

The authors demonstrate that a deep learning approach offers improvement to both the identification accuracy and rate at which defects can be identified of nematic liquid crystals.

Apply a well known neural model (YOLO) to detect bounding boxes of objects in images.

Detecting Topological Defects in 2D Active Nematics Using Convolutional Neural Networks


In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL).

In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed.

The results of our experiment show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning.

An analysis of the effects of compositionality and locality on representation learning for zero-shot learning.

Proposes evaluation framework for ZSL where the model is not allowed to be pretrained and instead, model parameters are randomly initialized for better understanding of what's happening in ZSL.

Locality and Compositionality in Zero-Shot Learning

zero-shot learning
compositionality
locality


It is becoming increasingly clear that many machine learning classifiers are vulnerable to adversarial examples.

In attempting to explain the origin of adversarial examples, previous studies have typically focused on the fact that neural networks operate on high dimensional data, they overfit, or they are too linear.

Here we show that distributions of logit differences have a universal functional form.

This functional form is independent of architecture, dataset, and training protocol; nor does it change during training.

This leads to adversarial error having a universal scaling, as a power-law, with respect to the size of the adversarial perturbation.

We show that this universality holds for a broad range of datasets (MNIST, CIFAR10, ImageNet, and random data), models (including state-of-the-art deep networks, linear models, adversarially trained networks, and networks trained on randomly shuffled labels), and attacks (FGSM, step l.l., PGD).

Motivated by these results, we study the effects of reducing prediction entropy on adversarial robustness.

Finally, we study the effect of network architectures on adversarial sensitivity.

To do this, we use neural architecture search with reinforcement learning to find adversarially robust architectures on CIFAR10.

Our resulting architecture is more robust to white \emph{and} black box attacks compared to previous attempts.

Adversarial error has similar power-law form for all datasets and models studied, and architecture matters.

Intriguing Properties of Adversarial Examples


Reinforcement learning (RL) has led to increasingly complex looking behavior in recent years.

However, such complexity can be misleading and hides over-fitting.

We find that visual representations may be a useful metric of complexity, and both correlates well objective optimization and causally effects reward optimization.

We then propose curious representation learning (CRL) which allows us to use better visual representation learning algorithms to correspondingly increase visual representation in policy through an intrinsic objective on both simulated environments and transfer to real images.

Finally, we show better visual representations induced by CRL allows us to obtain better performance on Atari without any reward than other curiosity objectives.

We present a formulation of curiosity as a visual representation learning problem and show that it allows good visual representations in agents.

This paper formulates curiosity based RL training as learning a visual representation model, arguing that focusing on better LR and maximising model loss for novel scenes will get better overall performance.

Learning Good Policies By Learning Good Perceptual Models

visual representation
learn
good
curiosity
model


This paper introduces the task of semantic instance completion: from an incomplete RGB-D scan of a scene, we aim to detect the individual object instances comprising the scene and infer their complete object geometry.

This enables a semantically meaningful decomposition of a scanned scene into individual, complete 3D objects, including hidden and unobserved object parts.

This will open up new possibilities for interactions with object in a scene, for instance for virtual or robotic agents.

To address this task, we propose 3D-SIC, a new data-driven approach that jointly detects object instances and predicts their completed geometry.

The core idea of 3D-SIC is a novel end-to-end 3D neural network architecture that leverages joint color and geometry feature learning.

The fully-convolutional nature of our 3D network enables efficient inference of semantic instance completion for 3D scans at scale of large indoor environments in a single forward pass.

In a series evaluation, we evaluate on both real and synthetic scan benchmark data, where we outperform state-of-the-art approaches by over 15 in mAP@0.5 on ScanNet, and over 18 in mAP@0.5 on SUNCG.

From an incomplete RGB-D scan of a scene, we aim to detect the individual object instances comprising the scene and infer their complete object geometry.

Proposes an end-to-end 3D CNN structure which combines color features and 3D features to predict the missing 3D structure of a scene from RGB-D scans.

The authors propose a novel end-to-end 3D convolutional network which predicts 3D semantic instance completion as object bounding boxes, class labels and complete object geometry.

3D-SIC: 3D Semantic Instance Completion for RGB-D Scans

complete object geometry
3d semantic instance completion
rgb-d scan
scene
predict
end-to-end


Style transfer usually refers to the task of applying color and texture information from a specific style image to a given content image while preserving the structure of the latter.

Here we tackle the more generic problem of semantic style transfer: given two unpaired collections of images, we aim to learn a mapping between the corpus-level style of each collection, while preserving semantic content shared across the two domains.

We introduce XGAN ("Cross-GAN"), a dual adversarial autoencoder, which captures a shared representation of the common domain semantic content in an unsupervised way, while jointly learning the domain-to-domain image translations in both directions.

We exploit ideas from the domain adaptation literature and define a semantic consistency loss which encourages the model to preserve semantics in the learned embedding space.

We report promising qualitative results for the task of face-to-cartoon translation.

The cartoon dataset we collected for this purpose will also be released as a new benchmark for semantic style transfer.

XGAN is an unsupervised model for feature-level image-to-image translation applied to semantic style transfer problems such as the face-to-cartoon task, for which we introduce a new dataset.

This paper proposes a new GAN-based model for unpaired image-to-image translation similar to DTN

XGAN: Unsupervised Image-to-Image Translation for many-to-many Mappings

image-to-image translation
xgan
model
new


Training neural networks on large datasets can be accelerated by distributing the workload over a network of machines.

As datasets grow ever larger, networks of hundreds or thousands of machines become economically viable.

The time cost of communicating gradients limits the effectiveness of using such large machine counts, as may the increased chance of network faults.

We explore a particularly simple algorithm for robust, communication-efficient learning---signSGD.

Workers transmit only the sign of their gradient vector to a server, and the overall update is decided by a majority vote.

This algorithm uses 32x less communication per iteration than full-precision, distributed SGD.

Under natural conditions verified by experiment, we prove that signSGD converges in the large and mini-batch settings, establishing convergence for a parameter regime of Adam as a byproduct.

Aggregating sign gradients by majority vote means that no individual worker has too much power.

We prove that unlike SGD, majority vote is robust when up to 50% of workers behave adversarially.

The class of adversaries we consider includes as special cases those that invert or randomise their gradient estimate.

On the practical side, we built our distributed training system in Pytorch.

Benchmarking against the state of the art collective communications library (NCCL), our framework---with the parameter server housed entirely on one machine---led to a 25% reduction in time for training resnet50 on Imagenet when using 15 AWS p3.2xlarge machines.

Workers send gradient signs to the server, and the update is decided by majority vote. We show that this algorithm is convergent, communication efficient and fault tolerant, both in theory and in practice.

Presents a distributed implementation of signSGD with majority vote as aggregation.

signSGD with Majority Vote is Communication Efficient and Fault Tolerant

communication efficient fault tolerant
signsgd majority vote


Profiling cellular phenotypes from microscopic imaging can provide meaningful biological information resulting from various factors affecting the cells.

One motivating application is drug development: morphological cell features can be captured from images, from which similarities between different drugs applied at different dosages can be quantified.

The general approach is to find a function mapping the images to an embedding space of manageable dimensionality whose geometry captures relevant features of the input images.

An important known issue for such methods is separating relevant biological signal from nuisance variation.

For example, the embedding vectors tend to be more correlated for cells that were cultured and imaged during the same week than for cells from a different week, despite having identical drug compounds applied in both cases.

In this case, the particular batch a set of experiments were conducted in constitutes the domain of the data; an ideal set of image embeddings should contain only the relevant biological information (e.g. drug effects).

We develop a general framework for adjusting the image embeddings in order to `forget' domain-specific information while preserving relevant biological information.

To do this, we minimize a loss function based on distances between marginal distributions (such as the Wasserstein distance) of embeddings across domains for each replicated treatment.

For the dataset presented, the replicated treatment is the negative control.

We find that for our transformed embeddings (1) the underlying geometric structure is not only preserved but the embeddings also carry improved biological signal (2) less domain-specific information is present.

We correct nuisance variation for image embeddings across different domains, preserving only relevant information.

Discusses a method for adjusting image embeddings in order tease apart technical variation from biological signal.

The authors present a method to remove domain-specific information while preserving the relevant biological information by training a network that minimizes the Wasserstein distance between distrbutions.

Correcting Nuisance Variation using Wasserstein Distance

correct nuisance variation
preserve relevant
image embedding
method
wasserstein distance
information
biological


This paper presents a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size.

MINE is  back-propable and we prove that it is strongly consistent.

We illustrate a handful of applications in which MINE is succesfully applied  to enhance the property of generative models in both unsupervised and supervised settings.

We apply our framework to estimate the information bottleneck, and apply it in tasks related to supervised classification problems.

Our results  demonstrate substantial added flexibility and improvement in these settings.

A scalable in sample size and dimensions mutual information estimator.

MINE: Mutual Information Neural Estimation


Reinforcement learning methods have recently achieved impressive results on a wide range of control problems.

However, especially with complex inputs, they still require an extensive amount of training data in order to converge to a meaningful solution.

This limitation largely prohibits  their usage for complex input spaces such as video signals, and it is still impossible to use it for a number of complex problems in a real world environments, including many of those for video based control.

Supervised learning, on the contrary, is capable of learning on a relatively small number of samples, however it does not take into account reward-based control policies and is not capable to provide independent control policies.

In this article we propose a model-free control method, which uses a combination of reinforcement and supervised learning for autonomous control and paves the way towards policy based control in real world environments.

We use SpeedDreams/TORCS video game to demonstrate that our approach requires much less samples (hundreds of thousands against millions or tens of millions) comparing to the state-of-the-art reinforcement learning techniques on similar data, and at the same time overcomes both supervised and reinforcement learning approaches in terms of quality.

Additionally, we demonstrate the applicability of the method to MuJoCo control problems.

The new combination of reinforcement and supervised learning, dramatically decreasing the number of required samples for training on video

This paper proposes leveraging labelled controlled data to accelerate reinforcement-based learning of a control policy

Combination of Supervised and Reinforcement Learning For Vision-Based Autonomous Control

reinforcement learning combination supervised control


A typical experiment to study cognitive function is to train animals to perform tasks, while the researcher records the electrical activity of the animals neurons.

The main obstacle faced, when using this type of electrophysiological experiment to uncover the circuit mechanisms underlying complex behaviors, is our incomplete access to relevant circuits in the brain.

One promising approach is to model neural circuits using an artificial neural network (ANN), which can provide complete access to the “neural circuits” responsible for a behavior.

More recently, reinforcement learning models have been adopted to understand the functions of cortico-basal ganglia circuits as reward-based learning has been found in mammalian brain.

In this paper, we propose a Biologically-plausible Actor-Critic with Episodic Memory (B-ACEM) framework to model a prefrontal cortex-basal ganglia-hippocampus (PFC-BG) circuit, which is verified to capture the behavioral findings from a well-known perceptual decision-making task, i.e., random dots motion discrimination.

This B-ACEM framework links neural computation to behaviors, on which we can explore how episodic memory should be considered to govern future decision.

Experiments are conducted using different settings of the episodic memory and results show that all patterns of episodic memories can speed up learning.

In particular, salient events are prioritized to propagate reward information and guide decisions.

Our B-ACEM framework and the built-on experiments give inspirations to both designs for more standard decision-making models in biological system and a more biologically-plausible ANN.

Fast learning via episodic memory verified by a biologically plausible framework for prefrontal cortex-basal ganglia-hippocampus (PFC-BG) circuit

FAST LEARNING VIA EPISODIC MEMORY: A PERSPECTIVE FROM ANIMAL DECISION-MAKING


Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet challenging question in deep learning and approximation theory.

In a seminal paper, Telgarsky high- lighted the benefits of depth by presenting a family of functions (based on sim- ple triangular waves) for which DNNs achieve zero classification error, whereas shallow networks with fewer than exponentially many nodes incur constant error.

Even though Telgarsky’s work reveals the limitations of shallow neural networks, it doesn’t inform us on why these functions are difficult to represent and in fact he states it as a tantalizing open question to characterize those functions that cannot be well-approximated by smaller depths.

In this work, we point to a new connection between DNNs expressivity and Sharkovsky’s Theorem from dynamical systems, that enables us to characterize the depth-width trade-offs of ReLU networks for representing functions based on the presence of a generalized notion of fixed points, called periodic points (a fixed point is a point of period 1).

Motivated by our observation that the triangle waves used in Telgarsky’s work contain points of period 3 – a period that is special in that it implies chaotic behaviour based on the celebrated result by Li-Yorke – we proceed to give general lower bounds for the width needed to represent periodic functions as a function of the depth.

Technically, the crux of our approach is based on an eigenvalue analysis of the dynamical systems associated with such functions.

Shows how the expressive power of NN depends on its depth and width, furthering the understanding of the benefit of deep nets for representing certain function classes.

The authors derive depth-width tradeoff conditions for when relu networks are able to represent periodic functions using dynamical systems analysis.

Depth-Width Trade-offs for ReLU Networks via Sharkovsky's Theorem

depth-width trade-off relu network
dynamical system
represent
sharkovsky
theorem
function


We investigate low-bit quantization to reduce computational cost of deep neural network (DNN) based keyword spotting (KWS).

We propose approaches to further reduce quantization bits via integrating quantization into keyword spotting model training, which we refer to as quantization-aware training.

Our experimental results on large dataset indicate that quantization-aware training can recover performance models quantized to lower bits representations.

By combining quantization-aware training and weight matrix factorization, we are able to significantly reduce model size and computation for small-footprint keyword spotting, while maintaining performance.

We investigate quantization-aware training in very low-bit quantized keyword spotters to reduce the cost of on-device keyword spotting.

This submission proposes a combination of low-rank decomposition and quanitization approach to compress DNN models for keyword spotting.

Low-bit quantization and quantization-aware training for small-footprint keyword spotting

quantization-aware training
keyword spotting
low-bit


Single-cell RNA-sequencing (scRNA-seq) is a powerful tool for analyzing biological systems.

However, due to biological and technical noise, quantifying the effects of multiple experimental conditions presents an analytical challenge.

To overcome this challenge, we developed MELD: Manifold Enhancement of Latent Dimensions.

MELD leverages tools from graph signal processing to learn a latent dimension within the data scoring the prototypicality of each datapoint with respect to experimental or control conditions.

We call this dimension the Enhanced Experimental Signal (EES).

MELD learns the EES by filtering the noisy categorical experimental label in the graph frequency domain to recover a smooth signal with continuous values.

This method can be used to identify signature genes that vary between conditions and identify which cell types are most affected by a given perturbation.

We demonstrate the advantages of MELD analysis in two biological datasets, including T-cell activation in response to antibody-coated beads and treatment of human pancreatic islet cells with interferon gamma.

A novel graph signal processing framework for quantifying the effects of experimental perturbations in single cell biomedical data.

This paper introduces several methods to process experimental results on biological cells and proposes a MELD algorithm mapping hard group assignments to soft assignments, allowing relevant groups of cells to be clustered.

Enhancing experimental signals in single-cell RNA-sequencing data using graph signal processing

graph signal processing
datum
cell
experimental


Models of user behavior are critical inputs in many prescriptive settings and can be viewed as decision rules that transform state information available to the user into actions.

Gaussian processes (GPs), as well as nonlinear extensions thereof, provide a flexible framework to learn user models in conjunction with approximate Bayesian inference.

However, the resulting models may not be interpretable in general.

We propose decision-rule GPs (DRGPs) that apply GPs in a transformed space defined by decision rules that have immediate interpretability to practitioners.

We illustrate this modeling tool on a real application and show that structural variational inference techniques can be used with DRGPs.

We find that DRGPs outperform the direct use of GPs in terms of out-of-sample performance.

We propose a class of user models based on using Gaussian processes applied to a transformed space defined by decision rules

Interpretable User Models via Decision-rule Gaussian Processes: Preliminary Results on Energy Storage


While Bayesian optimization (BO) has achieved great success in optimizing expensive-to-evaluate black-box functions, especially tuning hyperparameters of neural networks, methods such as random search (Li et al., 2016) and multi-fidelity BO (e.g. Klein et al. (2017)) that exploit cheap approximations, e.g. training on a smaller training data or with fewer iterations, can outperform standard BO approaches that use only full-fidelity observations.

In this paper, we propose a novel Bayesian optimization algorithm, the continuous-fidelity knowledge gradient (cfKG) method, that can be used when fidelity is controlled by one or more continuous settings such as training data size and the number of training iterations.

cfKG characterizes the value of the information gained by sampling a point at a given fidelity, choosing to sample at the point and fidelity with the largest value per unit cost.

Furthermore, cfKG can be generalized, following Wu et al. (2017), to settings where derivatives are available in the optimization process, e.g. large-scale kernel learning, and where more than one point can be evaluated simultaneously.

Numerical experiments show that cfKG outperforms state-of-art algorithms when optimizing synthetic functions, tuning convolutional neural networks (CNNs) on CIFAR-10 and SVHN, and in large-scale kernel learning.

We propose a Bayes-optimal Bayesian optimization algorithm for hyperparameter tuning by exploiting cheap approximations.

Studies hyperparameter-optimization by Bayesian optimization, using the Knowledge Gradient framework and allowing the Bayesian optimizer to tune fidelity against cost.

Continuous-fidelity Bayesian Optimization with Knowledge Gradient

knowledge gradient
bayesian optimization
tune


Neural networks trained only to optimize for training accuracy can often be fooled by adversarial examples --- slightly perturbed inputs misclassified with high confidence.

Verification of networks enables us to gauge their vulnerability to such adversarial examples.

We formulate verification of piecewise-linear neural networks as a mixed integer program.

On a representative task of finding minimum adversarial distortions, our verifier is two to three orders of magnitude quicker than the state-of-the-art.

We achieve this computational speedup via tight formulations for non-linearities, as well as a novel presolve algorithm that makes full use of all information available.

The computational speedup allows us to verify properties on convolutional and residual networks with over 100,000 ReLUs --- several orders of magnitude more than networks previously verified by any complete verifier.

In particular, we determine for the first time the exact adversarial accuracy of an MNIST classifier to perturbations with bounded l-∞ norm ε=0.1: for this classifier, we find an adversarial example for 4.38% of samples, and a certificate of robustness to norm-bounded perturbations for the remainder.

Across all robust training procedures and network architectures considered, and for both the MNIST and CIFAR-10 datasets, we are able to certify more samples than the state-of-the-art and find more adversarial examples than a strong first-order attack.

We efficiently verify the robustness of deep neural models with over 100,000 ReLUs, certifying more samples than the state-of-the-art and finding more adversarial examples than a strong first-order attack.

Performs a careful study of mixed integer linear programming approaches for verifying robustness of neural networks to adversarial perturbations and proposes three enhancements to MILP formulations of neural network verification.

Evaluating Robustness of Neural Networks with Mixed Integer Programming

robustness neural network
mixed integer
adversarial
verify
programming


Uncertainty estimation is an essential step in the evaluation of the robustness for deep learning models in computer vision, especially when applied in risk-sensitive areas.

However, most state-of-the-art deep learning models either fail to obtain uncertainty estimation or need significant modification (e.g., formulating a proper Bayesian treatment) to obtain it.

None of the previous methods are able to take an arbitrary model off the shelf and generate uncertainty estimation without retraining or redesigning it.

To address this gap, we perform the first systematic exploration into training-free uncertainty estimation.

We propose three simple and scalable methods to analyze the variance of output from a trained network under tolerable perturbations: infer-transformation, infer-noise, and infer-dropout.

They operate solely during inference, without the need to re-train, re-design, or fine-tune the model, as typically required by other state-of-the-art uncertainty estimation methods.

Surprisingly, even without involving such perturbations in training, our methods produce comparable or even better uncertainty estimation when compared to other training-required state-of-the-art methods.

Last but not least, we demonstrate that the uncertainty from our proposed methods can be used to improve the neural network training.

A set of methods to obtain uncertainty estimation of any given model without re-designing, re-training, or to fine-tuning it.

Describes several approaches for measuring uncertainty in arbitrary neural networks when there is an absence of distortion during training.

Training-Free Uncertainty Estimation for Neural Networks


Capturing spatiotemporal dynamics is an essential topic in video recognition.

In this paper, we present learnable higher-order operation as a generic family of building blocks for capturing higher-order correlations from high dimensional input video space.

We prove that several successful architectures for visual classification tasks are in the family of higher-order neural networks, theoretical and experimental analysis demonstrates their underlying mechanism is higher-order.

On the task of video recognition, even using RGB only without fine-tuning with other video datasets, our higher-order models can achieve results on par with or better than the existing state-of-the-art methods on both Something-Something (V1 and V2) and Charades datasets.

Proposed higher order operation for context learning

Proposes a new 3D convolutional block which convolves video input with its context, based on the assumpton that relevant context is present around the image's object.

Learnable Higher-order Representation for Action Recognition


Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters.

To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedures.

The consistency loss dramatically improves generalization performance over supervised-only training; however, we show that SGD struggles to converge on the consistency loss and continues to make large steps that lead to changes in predictions on the test data.

Motivated by these observations, we propose to train consistency-based methods with Stochastic Weight Averaging (SWA), a recent approach which averages weights along the trajectory of SGD with a modified learning rate schedule.

We also propose fast-SWA, which further accelerates convergence by averaging multiple points within each cycle of a cyclical learning rate schedule.

With weight averaging, we achieve the best known semi-supervised results on CIFAR-10 and CIFAR-100, over many different quantities of labeled training data.

For example, we achieve 5.0% error on CIFAR-10 with only 4000 labels, compared to the previous best result in the literature of 6.3%.

Consistency-based models for semi-supervised learning do not converge to a single point but continue to explore a diverse set of plausible solutions on the perimeter of a flat region. Weight averaging helps improve generalization performance.

The paper proposes to apply Stochastic Weight Averaging to the semi-supervised learning context, arguing that the semi-supervised MT/Pi models are especially amenable to SWA and propose fast SWA to speed up training.

There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average

semi-supervised learning
weight averaging
model


In this paper, we find that by designing a novel loss function entitled, ''tracking loss'', Convolutional Neural Network (CNN) based object detectors can be successfully converted to well-performed visual trackers without any extra computational cost.

This property is preferable to visual tracking where annotated video sequences for training are always absent, because rich features learned by detectors from still images could be utilized by dynamic trackers.

It also avoids extra machinery such as feature engineering and feature aggregation proposed in previous studies.

Tracking loss achieves this property by exploiting the internal structure of feature maps within the detection network and treating different feature points discriminatively.

Such structure allows us to simultaneously consider discrimination quality and bounding box accuracy which is found to be crucial to the success.

We also propose a network compression method to accelerate tracking speed without performance reduction.

That also verifies tracking loss will remain highly effective even if the network is drastically compressed.

Furthermore, if we employ a carefully designed tracking loss ensemble, the tracker would be much more robust and accurate.

Evaluation results show that our trackers (including the ensemble tracker and two baseline trackers), outperform all state-of-the-art methods on VOT 2016 Challenge in terms of Expected Average Overlap (EAO) and robustness.

We will make the code publicly available.

We successfully convert a popular detector RPN to a well-performed tracker from the viewpoint of loss function.

Tracking Loss: Converting Object Detector to Robust Visual Tracker


We study the problem of semantic code repair, which can be broadly defined as automatically fixing non-syntactic bugs in source code.

The majority of past work in semantic code repair assumed access to unit tests against which candidate repairs could be validated.

In contrast, the goal here is to develop a strong statistical model to accurately predict both bug locations and exact fixes without access to information about the intended correct behavior of the program.

Achieving such a goal requires a robust contextual repair model, which we train on a large corpus of real-world source code that has been augmented with synthetically injected bugs.

Our framework adopts a two-stage approach where first a large set of repair candidates are generated by rule-based processors, and then these candidates are scored by a statistical model using a novel neural network architecture which we refer to as Share, Specialize, and Compete.

Specifically, the architecture (1) generates a  shared encoding of the source code using an RNN over the abstract syntax tree, (2) scores each candidate repair using specialized network modules, and (3) then normalizes these scores together so they can compete against one another in comparable probability space.

We evaluate our model on a real-world test set gathered from GitHub containing four common categories of bugs.

Our model is able to predict the exact correct repair 41% of the time with a single guess, compared to 13% accuracy for an attentional sequence-to-sequence model.

A neural architecture for scoring and ranking program repair candidates to perform semantic program repair statically without access to unit tests.

Presents a neural network architecture consisting of the share, specialize and compete parts for repairing code in four cases.

Semantic Code Repair using Neuro-Symbolic Transformation Networks

neural repair architecture semantic network code


Deep networks were recently suggested to face the odds between accuracy (on clean natural images) and robustness (on adversarially perturbed images) (Tsipras et al., 2019).

Such a dilemma is shown to be rooted in the inherently higher sample complexity (Schmidt et al., 2018) and/or model capacity (Nakkiran, 2019), for learning a high-accuracy and robust classifier.

In view of that, give a classification task, growing the model capacity appears to help draw a win-win between accuracy and robustness, yet at the expense of model size and latency, therefore posing challenges for resource-constrained applications.

This paper studies multi-exit networks associated with input-adaptive efficient inference, showing their strong promise in achieving a “sweet point" in co-optimizing model accuracy, robustness, and efficiency.

Our proposed solution, dubbed Robust Dynamic Inference Networks (RDI-Nets), allows for each input (either clean or adversarial) to adaptively choose one of the multiple output layers (early branches or the final one) to output its prediction.

That multi-loss adaptivity adds new variations and flexibility to adversarial attacks and defenses, on which we present a systematical investigation.

We show experimentally that by equipping existing backbones with such robust adaptive inference, the resulting RDI-Nets can achieve better accuracy and robustness, yet with over 30% computational savings, compared to the defended original models.

Is it possible to co-design model accuracy, robustness and efficiency to achieve their triple wins? Yes!

Exploits input-adaptive multiple early-exits for the field of adversarial attack and defense, reducing the average inference complexity without conflicting the larger capacity assumption.

Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference

accuracy robustness efficiency
triple win
input-adaptive
inference


Although deep convolutional networks have achieved improved performance in many natural language tasks, they have been treated as black boxes because they are difficult to interpret.

Especially, little is known about how they represent language in their intermediate layers.

In an attempt to understand the representations of deep convolutional networks trained on language tasks, we show that individual units are selectively responsive to specific morphemes, words, and phrases, rather than responding to arbitrary and uninterpretable patterns.

In order to quantitatively analyze such intriguing phenomenon, we propose a concept alignment method based on how units respond to replicated text.

We conduct analyses with different architectures on multiple datasets for classification and translation tasks and provide new insights into how deep models understand natural language.

We show that individual units in CNN representations learned in NLP tasks are selectively responsive to specific natural language concepts.

Uses grammatical units of natural language that preserve meanings to show that the units of deep CNNs learned in NLP tasks could act as a natural language concept detector.

Natural Language Detectors Emerge in Individual Neurons

natural language concept
learn nlp task
show
unit
individual
cnn
detector


We study the problem of building models that disentangle independent factors of variation.

Such models encode features that can efficiently be used for classification and to transfer attributes between different images in image synthesis.

As data we use a weakly labeled training set, where labels indicate what single factor has changed between two data samples, although the relative value of the change is unknown.

This labeling is of particular interest as it may be readily available without annotation costs.

We introduce an autoencoder model and train it through constraints on image pairs and triplets.

We show the role of feature dimensionality and adversarial training theoretically and experimentally.

We formally prove the existence of the reference ambiguity, which is inherently present in the disentangling task when weakly labeled data is used.

The numerical value of a factor has different meaning in different reference frames.

When the reference depends on other factors, transferring that factor becomes ambiguous.

We demonstrate experimentally that the proposed model can successfully transfer attributes on several datasets, but show also cases when the reference ambiguity occurs.

It is a mostly theoretical paper that describes the challenges in disentangling factors of variation, using autoencoders and GAN.

This paper considers disentangling factors of variation in images, shows that in general, without further assumptions, one cannot tell apart two different variation factors, and suggests a novel AE+GAN architecture to try and disentangle variation factors.

This paper studies the challenges of disentangling independent factors of variation under weakly labeled data and introduces the term reference ambiguity for data point mapping.

Challenges in Disentangling Independent Factors of Variation

challenge disentangle independent factor variation


In information retrieval, learning to rank constructs a machine-based ranking model which given a query, sorts the search results by their degree of relevance or importance to the query.

Neural networks have been successfully applied to this problem, and in this paper, we propose an attention-based deep neural network which better incorporates different embeddings of the queries and search results with an attention-based mechanism.

This model also applies a decoder mechanism to learn the ranks of the search results in a listwise fashion.

The embeddings are trained with convolutional neural networks or the word2vec model.

We demonstrate the performance of this model with image retrieval and text querying data sets.

learning to rank with several embeddings and attentions

Proposes to use attention to combine multiple input representations for both query and search results in the learning to rank task.

AN ATTENTION-BASED DEEP NET FOR LEARNING TO RANK


Computational neuroscience aims to fit reliable models of in vivo neural activity and interpret them as abstract computations.

Recent work has shown that functional diversity of neurons may be limited to that of relatively few cell types; other work has shown that incorporating constraints into artificial neural networks (ANNs) can improve their ability to mimic neural data.

Here we develop an algorithm that takes as input recordings of neural activity and returns clusters of neurons by cell type and models of neural activity constrained by these clusters.

The resulting models are both more predictive and more interpretable, revealing the contributions of functional cell types to neural computation and ultimately informing the design of future ANNs.

We developed an algorithm that takes as input recordings of neural activity and returns clusters of neurons by cell type and models of neural activity constrained by these clusters.

Data-Driven Discovery of Functional Cell Types that Improve Models of Neural Activity


Graph Neural Networks (GNNs) are a powerful representational tool for solving problems on graph-structured inputs.

In almost all cases so far, however, they have been applied to directly recovering a final solution from raw inputs, without explicit guidance on how to structure their problem-solving.

Here, instead, we focus on learning in the space of algorithms: we train several state-of-the-art GNN architectures to imitate individual steps of classical graph algorithms, parallel (breadth-first search, Bellman-Ford) as well as sequential (Prim's algorithm).

As graph algorithms usually rely on making discrete decisions within neighbourhoods, we hypothesise that maximisation-based message passing neural networks are best-suited for such objectives, and validate this claim empirically.

We also demonstrate how learning in the space of algorithms can yield new opportunities for positive transfer between tasks---showing how learning a shortest-path algorithm can be substantially improved when simultaneously learning a reachability algorithm.

We supervise graph neural networks to imitate intermediate and step-wise outputs of classical graph algorithms, recovering highly favourable insights.

Suggests training neural networks to imitate graph algorithms by learning primitives and subroutines rather than the final output.

Neural Execution of Graph Algorithms

neural network imitate
graph algorithm
output


Prospection is an important part of how humans come up with new task plans, but has not been explored in depth in robotics.

Predicting multiple task-level is a challenging problem that involves capturing both task semantics and continuous variability over the state of the world.

Ideally, we would combine the ability of machine learning to leverage big data for learning the semantics of a task, while using techniques from task planning to reliably generalize to new environment.

In this work, we propose a method for learning a model encoding just such a representation for task planning.

We learn a neural net that encodes the k most likely outcomes from high level actions from a given world.

Our approach creates comprehensible task plans that allow us to predict changes to the environment many time steps into the future.

We demonstrate this approach via application to a stacking task in a cluttered environment, where the robot must select between different colored blocks while avoiding obstacles, in order to perform a task.

We also show results on a simple navigation task.

Our algorithm generates realistic image and pose predictions at multiple points in a given task.

We describe an architecture for generating diverse hypotheses for intermediate goals during robotic manipulation tasks.

Evaluates the quality of a proposed generative predictive model to generate plans for robot execution.

This paper proposes a method for learning a high-level transition function that is useful for task planning.

Learning to Imagine Manipulation Goals for Robot Task Planning

generate goal task robot learn manipulation plan


Adaptive gradient algorithms perform gradient-based updates using the history of gradients and are ubiquitous in training deep neural networks.

While adaptive gradient methods theory is well understood for minimization problems, the underlying factors driving their empirical success in min-max problems such as GANs remain unclear.

In this paper, we aim at bridging  this gap from both theoretical and empirical perspectives.

First, we analyze a variant of Optimistic Stochastic Gradient (OSG) proposed in~\citep{daskalakis2017training} for solving a class of non-convex non-concave min-max problem and establish $O(\epsilon^{-4})$ complexity for finding $\epsilon$-first-order stationary point, in which the algorithm only requires invoking one stochastic first-order oracle while enjoying state-of-the-art iteration complexity achieved by stochastic extragradient method by~\citep{iusem2017extragradient}.

Then we propose an adaptive variant of OSG named Optimistic Adagrad (OAdagrad) and reveal an \emph{improved} adaptive complexity $\widetilde{O}\left(\epsilon^{-\frac{2}{1-\alpha}}\right)$~\footnote{Here $\widetilde{O}(\cdot)$ compresses a logarithmic factor of $\epsilon$.

}, where $\alpha$ characterizes the growth rate of the cumulative stochastic gradient and $0\leq \alpha\leq 1/2$.

To the best of our knowledge, this is the first work for establishing adaptive complexity in non-convex non-concave min-max optimization.

Empirically, our experiments show that indeed adaptive gradient algorithms outperform their non-adaptive counterparts in GAN training.

Moreover, this observation can be explained by the slow growth rate of the cumulative stochastic gradient, as observed empirically.

This paper provides novel analysis of adaptive gradient algorithms for solving non-convex non-concave min-max problems as GANs, and explains the reason why adaptive gradient methods outperform its non-adaptive counterparts by empirical studies.

Develops algorithms for the solution of variational inequalities in the stochastic setting, proposing a variation of the extragradient method.

Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets


We consider the problem of unsupervised learning of a low dimensional, interpretable, latent state of a video containing a moving object.

The problem of distilling dynamics from pixels has been extensively considered through the lens of graphical/state space models that exploit Markov structure for cheap computation and structured graphical model priors for enforcing interpretability on latent representations.

We take a step towards extending these approaches by discarding the Markov structure; instead, repurposing the recently proposed Gaussian Process Prior Variational Autoencoder for learning sophisticated latent trajectories.

We describe the model and perform experiments on a synthetic dataset and see that the model reliably reconstructs smooth dynamics exhibiting U-turns and loops.

We also observe that this model may be trained without any beta-annealing or freeze-thaw of training parameters.

Training is performed purely end-to-end on the unmodified evidence lower bound objective.

This is in contrast to previous works, albeit for slightly different use cases, where application specific training tricks are often required.

We learn sohpisticated trajectories of an object purely from pixels with a toy video dataset by using a VAE structure with a Gaussian process prior.

The Gaussian Process Prior VAE for Interpretable Latent Dynamics from Pixels


Dreams and our ability to recall them are among the most puzzling questions in sleep research.

Specifically, putative differences in brain network dynamics between individuals with high versus low dream recall rates, are still poorly understood.

In this study, we addressed this question as a classification problem where we applied deep convolutional networks (CNN) to sleep EEG recordings to predict whether subjects belonged to the high or low dream recall group (HDR and LDR resp.).

Our model achieves significant accuracy levels across all the sleep stages, thereby indicating subtle signatures of dream recall in the sleep microstructure.

We also visualized the feature space to inspect the subject-specificity of the learned features, thus ensuring that the network captured population level differences.

Beyond being the first study to apply deep learning to sleep EEG in order to classify HDR and LDR, guided backpropagation allowed us to visualize the most discriminant features in each sleep stage.

The significance of these findings and future directions are discussed.

We investigate the neural basis of dream recall using convolutional neural network and feature visualization techniques, like tSNE and guided-backpropagation.

Unravelling the neural signatures of dream recall in EEG: a deep learning approach


This paper considers multi-agent reinforcement learning (MARL) in networked system control.

Specifically, each agent learns a decentralized control policy based on local observations and messages from connected neighbors.

We formulate such a networked MARL (NMARL) problem as a spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent.

Further, we propose a new differentiable communication protocol, called NeurComm, to reduce information loss and non-stationarity in NMARL.

Based on experiments in realistic NMARL scenarios of adaptive traffic signal control and cooperative adaptive cruise control, an appropriate spatial discount factor effectively enhances the learning curves of non-communicative MARL algorithms, while NeurComm outperforms existing communication protocols in both learning efficiency and control performance.

This paper proposes a new formulation and a new communication protocol for networked multi-agent control problems

Concerned with N-MARL's where agents update their policy based only on messages from neighboring nodes, showing that introducing a spatial discount factor stabilizes learning.

Multi-agent Reinforcement Learning for Networked System Control


Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights.

Recent work developing this class of methods has explored ever richer parameterizations of the approximate posterior in the hope of improving performance.

In contrast, here we share a curious experimental finding that suggests instead restricting the variational distribution to a more compact parameterization.

For a variety of deep Bayesian neural networks trained using Gaussian mean-field variational inference, we find that the posterior standard deviations consistently exhibit strong low-rank structure after convergence.

This means that by decomposing these variational parameters into a low-rank factorization, we can make our variational approximation more compact without decreasing the models' performance.

Furthermore, we find that such factorized parameterizations improve the signal-to-noise ratio of stochastic gradient estimates of the variational lower bound, resulting in faster convergence.

Mean field VB uses twice as many parameters; we tie variance parameters in mean field VB without any loss in ELBO, gaining speed and lower variance gradients.

The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks


Aspect extraction in online product reviews is a key task in sentiment analysis and opinion mining.

Training supervised neural networks for aspect extraction is not possible when ground truth aspect labels are not available, while the unsupervised neural topic models fail to capture the particular aspects of interest.

In this work, we propose a weakly supervised approach for training neural networks for aspect extraction in cases where only a small set of seed words, i.e., keywords that describe an aspect, are available.

First, we show that current weakly supervised networks fail to leverage the predictive power of the available seed words by comparing them to a simple bag-of-words classifier.

Second, we propose a distillation approach for aspect extraction where the seed words are considered by the bag-of-words classifier (teacher) and distilled to the parameters of a neural network (student).

Third, we show that regularization encourages the student to consider non-seed words for classification and, as a result, the student outperforms the teacher, which only considers the seed words.

Finally, we empirically show that our proposed distillation approach outperforms (by up to 34.4% in F1 score) previous weakly supervised approaches for aspect extraction in six domains of Amazon product reviews.

We effectively leverage a few keywords as weak supervision for training neural networks for aspect extraction.

Discusses a variant of knowledge distillation which uses a "teacher" based on a bag-of-words classifier with seed words and a "student" which is an embedding-based neural network.

Training Neural Networks for Aspect Extraction Using Descriptive Keywords Only

train neural network aspect extraction
keyword


Forming perceptual groups and individuating objects in visual scenes is an essential step towards visual intelligence.

This ability is thought to arise in the brain from computations implemented by bottom-up, horizontal, and top-down connections between neurons.

However, the relative contributions of these connections to perceptual grouping are poorly understood.

We address this question by systematically evaluating neural network architectures featuring combinations of these connections on two synthetic visual tasks, which stress low-level "Gestalt" vs. high-level object cues for perceptual grouping.

We show that increasing the difficulty of either task strains learning for networks that rely solely on bottom-up processing.

Horizontal connections resolve this limitation on tasks with Gestalt cues by supporting incremental spatial propagation of activities, whereas top-down connections rescue learning on tasks with high-level object cues by modifying coarse predictions about the position of the target object.

Our findings dissociate the computational roles of bottom-up, horizontal and top-down connectivity, and demonstrate how a model featuring all of these interactions can more flexibly learn to form perceptual groups.

Horizontal and top-down feedback connections are responsible for complementary perceptual grouping strategies in biological and recurrent vision systems.

Using neural networks as a computational model of the brain, examines the efficiency of different strategies for solving two visual challenges.

Disentangling neural mechanisms for perceptual grouping


Generative adversarial networks have seen rapid development in recent years and have led to remarkable improvements in generative modelling of images.

However, their application in the audio domain has received limited attention,
and autoregressive models, such as WaveNet, remain the state of the art in generative modelling of audio signals such as human speech.

To address this paucity, we introduce GAN-TTS, a Generative Adversarial Network for Text-to-Speech.

Our architecture is composed of a conditional feed-forward generator producing raw speech audio, and an ensemble of discriminators which operate on random windows of different sizes.

The discriminators analyse the audio both in terms of general realism, as well as how well the audio corresponds to the utterance that should be pronounced.

To measure the performance of GAN-TTS, we employ both subjective human evaluation (MOS - Mean Opinion Score), as well as novel quantitative metrics (Fréchet DeepSpeech Distance and Kernel DeepSpeech Distance), which we find to be well correlated with MOS.

We show that GAN-TTS is capable of generating high-fidelity speech with naturalness comparable to the state-of-the-art models, and unlike autoregressive models, it is highly parallelisable thanks to an efficient feed-forward generator.

Listen to GAN-TTS reading this abstract at http://tiny.cc/gantts.

We introduce GAN-TTS, a Generative Adversarial Network for Text-to-Speech, which achieves Mean Opinion Score (MOS) 4.2.

Solves the GAN challenge in raw waveform synthesis and begins to close the existing performance gap between autoregressive models and GANs for raw audios.

High Fidelity Speech Synthesis with Adversarial Networks


This paper proposes a Pruning in Training (PiT) framework of learning to reduce the parameter size of networks.

Different from existing works, our PiT framework employs the sparse penalties to train networks and thus help rank the importance of weights and filters.

Our PiT algorithms can directly prune the network without any fine-tuning.

The pruned networks can still achieve comparable performance to the original networks.

In particular, we introduce the (Group) Lasso-type Penalty (L-P /GL-P), and (Group) Split LBI Penalty (S-P / GS-P) to regularize the networks, and a pruning strategy proposed  is used in help prune the network.

We conduct the extensive experiments on MNIST, Cifar-10, and miniImageNet.

The results validate the efficacy of our proposed methods.

Remarkably, on MNIST dataset, our PiT framework can save 17.5% parameter size of LeNet-5, which achieves the 98.47% recognition accuracy.

we propose an algorithm of learning to prune network by enforcing structure sparsity penalties

This paper introduces an approach to pruning while training a network using lasso and split LBI penalties

PRUNING IN TRAINING: LEARNING AND RANKING SPARSE CONNECTIONS IN DEEP CONVOLUTIONAL NETWORKS


We first pose the Unsupervised Continual Learning (UCL) problem: learning salient representations from a non-stationary stream of unlabeled data in which the number of object classes varies with time.

Given limited labeled data just before inference, those representations can also be associated with specific object types to perform classification.

To solve the UCL problem, we propose an architecture that involves a single module, called Self-Taught Associative Memory (STAM), which loosely models the function of a cortical column in the mammalian brain.

Hierarchies of STAM modules learn based on a combination of Hebbian learning, online clustering, detection of novel patterns and forgetting outliers, and top-down predictions.

We illustrate the operation of STAMs in the context of learning handwritten digits in a continual manner with only 3-12 labeled examples per class.

STAMs suggest a promising direction to solve the UCL problem without catastrophic forgetting.

We introduce unsupervised continual learning (UCL) and a neuro-inspired architecture that solves the UCL problem.

Proposes using hierachies of STAM modules to solve the UCL problem, providing evidence that the representations the modules learn are well-suited for few-shot classification.

Unsupervised Continual Learning and Self-Taught Associative Memory Hierarchies


Recent advances have made it possible to create deep complex-valued neural networks.

Despite this progress, the potential power of fully complex intermediate computations and representations has not yet been explored for many challenging learning problems.

Building on recent advances, we propose a novel mechanism for extracting signals in the frequency domain.

As a case study, we perform audio source separation in the Fourier domain.

Our extraction mechanism could be regarded as a local ensembling method that combines a complex-valued convolutional version of Feature-Wise Linear Modulation (FiLM) and a signal averaging operation.

We also introduce a new explicit amplitude and phase-aware loss, which is scale and time invariant, taking into account the complex-valued components of the spectrogram.

Using the Wall Street Journal Dataset, we compare our phase-aware loss to several others that operate both in the time and frequency domains and demonstrate the effectiveness of our proposed signal extraction method and proposed loss.

When operating in the complex-valued frequency domain, our deep complex-valued network substantially outperforms its real-valued counterparts even with half the depth and a third of the parameters.

Our proposed mechanism improves significantly deep complex-valued networks' performance and we demonstrate the usefulness of its regularizing effect.

New Signal Extraction Method in the Fourier Domain

Contributes a complex-valued convolutional version of the Feature-Wise Linear Modulation which allows parameter optimization and designs a loss which takes into account magnitude and phase.

Retrieving Signals in the Frequency Domain with Deep Complex Extractors


It is challenging to disentangle an object into two orthogonal spaces of content and style since each can influence the visual observation in a different and unpredictable way.

It is rare for one to have access to a large number of data to help separate the influences.

In this paper, we present a novel framework to learn this disentangled representation in a completely unsupervised manner.

We address this problem in a two-branch Autoencoder framework.

For the structural content branch, we project the latent factor into a soft structured point tensor and constrain it with losses derived from prior knowledge.

This encourages the branch to distill geometry information.

Another branch learns the complementary style information.

The two branches form an effective framework that can disentangle object's content-style representation without any human annotation.

We evaluate our approach on four image datasets, on which we demonstrate the superior disentanglement and visual analogy quality both in synthesized and real-world data.

We are able to generate photo-realistic images with 256x256 resolution that are clearly disentangled in content and style.

We present a novel framework to learn the disentangled representation of content and style in a completely unsupervised manner.

Propose model based on autoencoder framework to disentangle an object's representation, results show that model can produce representations capturing content and style.

Disentangling Content and Style via Unsupervised Geometry Distillation

content style
framework
disentangle
representation
unsupervised


We develop the Y-learner for estimating heterogeneous treatment effects in experimental and observational studies.

The Y-learner is designed to leverage the abilities of neural networks to optimize multiple objectives and continually update, which allows for better pooling of underlying feature information between treatment and control groups.

We evaluate the Y-learner on three test problems: (1) A set of six simulated data benchmarks from the literature.

(2) A real-world large-scale experiment on voter persuasion.

(3) A task from the literature that estimates artificially generated treatment effects on MNIST didgits.

The Y-learner achieves state of the art results on two of the three tasks.

On the MNIST task, it gets the second best results.

We develop a CATE estimation strategy that takes advantage some of the intriguing properties of neural networks.

Shows improvements to X-learner by modeling the treatment response function, the control response function, and the mapping from imputed treatment effect to the conditional average treatment effect, as neural networks.

The authors propose the Y-learner to estimate conditional average treatment effect(CATE), which simultaneously updates the parameters of the outcome functions and the CATE estimator.

Estimating Heterogeneous Treatment Effects Using Neural Networks With The Y-Learner

treatment effect neural network
function
y-learner
cate
conditional average
estimate


With the rapid proliferation of IoT devices, our cyberspace is nowadays dominated by billions of low-cost computing nodes, which expose an unprecedented heterogeneity to our computing systems.

Dynamic analysis, one of the most effective approaches to finding software bugs, has become paralyzed due to the lack of a generic emulator capable of running diverse previously-unseen firmware.

In recent years, we have witnessed devastating security breaches targeting IoT devices.

These security concerns have significantly hamstrung further evolution of IoT technology.

In this work, we present Laelaps, a device emulator specifically designed to run diverse software on low-cost IoT devices.

We do not encode into our emulator any specific information about a device.

Instead, Laelaps infers the expected behavior of firmware via symbolic-execution-assisted peripheral emulation and generates proper inputs to steer concrete execution on the fly.

This unique design feature makes Laelaps the first generic device emulator capable of running diverse firmware with no a priori knowledge about the target device.

To demonstrate the capabilities of Laelaps, we deployed two popular dynamic analysis techniques---fuzzing testing and dynamic symbolic execution---on top of our emulator.

We successfully identified both self-injected and real-world vulnerabilities.

Device-agnostic Firmware Execution is Possible


Deep neural models, such as convolutional and recurrent networks, achieve phenomenal results over spatial data such as images and text.

However, when considering tabular data, gradient boosting of decision trees (GBDT) remains the method of choice.

Aiming to bridge this gap, we propose \emph{deep neural forests} (DNF)

--  a novel architecture that combines elements from decision trees as well as dense residual connections.

We present the results of extensive empirical study in which we examine the performance of GBDTs, DNFs and (deep) fully-connected networks.

These results indicate that DNFs achieve comparable results to GBDTs on tabular data, and open the door to end-to-end neural modeling of multi-modal data.

To this end, we present a successful application of DNFs as part of a hybrid architecture for a multi-modal driving scene understanding classification task.

An architecture for tabular data, which emulates branches of decision trees and uses dense residual connectivity

This paper proposes deep neural forest, an algorithm which targets tabular data and integrates strong points of gradient boosting of decision trees.

A novel neural network architecture mimicking how decision forests work to tackle the general problem of training deep models for tabular data and showcasing effectiveness on par with GBDT.

Deep Neural Forests: An Architecture for Tabular Data

architecture tabular datum
deep neural forest
decision tree


Hyperparameter tuning is one of the most time-consuming workloads in deep learning.

State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable.

Recently researchers have shown renewed interest in simpler methods like momentum SGD as they may yield better results.

Motivated by this trend, we ask: can simple adaptive methods, based on SGD perform as well or better?

We revisit the momentum SGD algorithm and show that hand-tuning a single learning rate and momentum makes it competitive with Adam.

We then analyze its robustness to learning rate misspecification and objective curvature variation.

Based on these insights, we design YellowFin, an automatic tuner for momentum and learning rate in SGD.

YellowFin optionally uses a negative-feedback loop to compensate for the momentum dynamics in asynchronous settings on the fly.

We empirically show YellowFin can converge in fewer iterations than Adam on ResNets and LSTMs for image recognition, language modeling and constituency parsing, with a speedup of up to $3.28$x in synchronous and up to $2.69$x in asynchronous settings.

YellowFin is an SGD based optimizer with both momentum and learning rate adaptivity.

Proposes a method to automatically tuning the momentum parameter in momentum SGD methods, which achieves better results and fast convergence speed that state-of-the-art Adam algorithm.

YellowFin and the Art of Momentum Tuning


Robustness and security of machine learning (ML) systems are intertwined, wherein a non-robust ML system (classifiers, regressors, etc.) can be subject to attacks using a wide variety of exploits.

With the advent of scalable deep learning methodologies, a lot of emphasis has been put on the robustness of supervised, unsupervised and reinforcement learning algorithms.

Here, we study the robustness of the latent space of a deep variational autoencoder (dVAE), an unsupervised generative framework, to show that it is indeed possible to perturb the latent space, flip the class predictions and keep the classification probability approximately equal before and after an attack.

This means that an agent that looks at the outputs of a decoder would remain oblivious to an attack.

Adversarial attacks on the latent space of variational autoencoders to change the semantic meaning of inputs

This paper concerns security and machine learning and proposes a man-in-middle attack that alters the VAE encoding of input data so that decoded output will be misclassified.

LatentPoison -- Adversarial Attacks On The Latent Space


Graph-based dependency parsing consists of two steps: first, an encoder produces a feature representation for each parsing substructure of the input sentence, which is then used to compute a score for the substructure; and second, a decoder} finds the parse tree whose substructures have the largest total score.

Over the past few years, powerful neural techniques have been introduced into the encoding step which substantially increases parsing accuracies.

However, advanced decoding techniques, in particular high-order decoding, have seen a decline in usage.

It is widely believed that contextualized features produced by neural encoders can help capture high-order decoding information and hence diminish the need for a high-order decoder.

In this paper, we empirically evaluate the combinations of different neural and non-neural encoders with first- and second-order decoders and provide a comprehensive analysis about the effectiveness of these combinations with varied training data sizes.

We find that: first, when there is large training data, a strong neural encoder with first-order decoding is sufficient to achieve high parsing accuracy and only slightly lags behind the combination of neural encoding and second-order decoding; second, with small training data, a non-neural encoder with a second-order decoder outperforms the other combinations in most cases.

An empirical study that examines the effectiveness of different encoder-decoder combinations for the task of dependency parsing

Empirically analyzes various encoders, decoders, and their dependencies for graph-based dependency parsing.

An Empirical Study of Encoders and Decoders in Graph-Based Dependency Parsing

encoder decoder
empirical study
graph-based dependency parsing


Meta-learning will be crucial to creating lifelong, generalizable AI.

In practice, however, it is hard to define the meta-training task distribution that is used to train meta-learners.

If made too small, tasks are too similar for a model to meaningfully generalize.

If made too large, generalization becomes incredibly difficult.

We argue that both problems can be alleviated by introducing a teacher model that controls the sequence of tasks that a meta-learner is trained on.

This teacher model is incentivized to start the student meta-learner on simple tasks then adaptively increase task difficulty in response to student progress.

While this approach has been previously studied in curriculum generation, our main contribution is in extending it to meta-learning.

Teacher that trains meta-learners like humans

Meta-Teaching: Curriculum Generation for Lifelong Learning


Using higher order knowledge to reduce training data has become a popular research topic.

However, the ability for available methods to draw effective decision boundaries is still limited: when training set is small, neural networks will be biased to certain labels.

Based on this observation, we consider constraining output probability distribution as higher order domain knowledge.

We design a novel algorithm that jointly optimizes output probability distribution on a clustered embedding space to make neural networks draw effective decision boundaries.

While directly applying probability constraint is not effective, users need to provide additional very weak supervisions: mark some batches that have output distribution greatly differ from target probability distribution.

We use experiments to empirically prove that our model can converge to an accuracy higher than other state-of-art semi-supervised learning models with less high quality labeled training examples.

We introduce an embedding space approach to constrain neural network output probability distribution.

This paper introduces a method to perform semi-supervised learning with deep neural networks, and the model achieves relatively high accuracy, given a small training size.

This paper incorporates label distribution into model learning when a limited number of training instances is available, and proposes two techniques for handling the problem of output label distribution being wrongly biased.

Train Neural Network by Embedding Space Probabilistic Constraint

embed space
introduce
neural network
distribution
model
learning
output
training


We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy).

Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pretrained on a large text corpus.

We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis.

We also present an analysis showing that exposing the deep internals of the pretrained network is crucial, allowing downstream models to mix different types of semi-supervision signals.

We introduce a new type of deep contextualized word representation that significantly improves the state of the art for a range of challenging NLP tasks.

Deep contextualized word representations


This work addresses the long-standing problem of robust event localization in the presence of temporally of misaligned labels in the training data.

We propose a novel versatile loss function that generalizes a number of training regimes from standard fully-supervised cross-entropy to count-based weakly-supervised learning.

Unlike classical models which are constrained to strictly fit the annotations during training, our soft localization learning approach relaxes the reliance on the exact position of labels instead.

Training with this new loss function exhibits strong robustness to temporal misalignment of labels, thus alleviating the burden of precise annotation of temporal sequences.

We demonstrate state-of-the-art performance against standard benchmarks in a number of challenging experiments and further show that robustness to label noise is not achieved at the expense of raw performance.

This work introduces a novel loss function for the robust training of temporal localization DNN in the presence of misaligned labels.

A new loss for training models that predict where events occur in a training sequence with noisy labels by comparing smoothed label and prediction sequence.

SoftLoc: Robust Temporal Localization under Label Misalignment

temporal localization
robust
training
label
loss


The driving force behind deep networks is their ability to compactly represent rich classes of functions.

The primary notion for formally reasoning about this phenomenon is expressive efficiency, which refers to a situation where one network must grow unfeasibly large in order to replicate functions of another.

To date, expressive efficiency analyses focused on the architectural feature of depth, showing that deep networks are representationally superior to shallow ones.

In this paper we study the expressive efficiency brought forth by connectivity, motivated by the observation that modern networks interconnect their layers in elaborate ways.

We focus on dilated convolutional networks, a family of deep models delivering state of the art performance in sequence processing tasks.

By introducing and analyzing the concept of mixed tensor decompositions, we prove that interconnecting dilated convolutional networks can lead to expressive efficiency.

In particular, we show that even a single connection between intermediate layers can already lead to an almost quadratic gap, which in large-scale settings typically makes the difference between a model that is practical and one that is not.

Empirical evaluation demonstrates how the expressive efficiency of connectivity, similarly to that of depth, translates into gains in accuracy.

This leads us to believe that expressive efficiency may serve a key role in developing new tools for deep network design.

We introduce the notion of mixed tensor decompositions, and use it to prove that interconnecting dilated convolutional networks boosts their expressive power.

This paper theoretically validates that interconnecting networks with different dilations can lead to expressive efficiency using mixed tensor decomposition.

The authors study dilated convolutional networks and show that intertwining two dilated convolutional networks A and B at various stages is more expressively efficient than not intertwining.

Shows that the WaveNet's structural assumption of a single perfect binary tree is hindering its performance and that WaveNet-like architectures with more complex mixed tree structures perform better.

Boosting Dilated Convolutional Networks with Mixed Tensor Decompositions

mixed tensor decomposition
dilate convolutional network
show
interconnect
expressive
boost


We apply multi-task learning to image classification tasks on MNIST-like datasets.

MNIST dataset has been referred to as the {\em drosophila} of machine learning and has been the testbed of many learning theories.

The NotMNIST dataset and the FashionMNIST dataset have been created with the MNIST dataset as reference.

In this work, we exploit these MNIST-like datasets for multi-task learning.

The datasets are pooled together for learning the parameters of joint classification networks.

Then the learned parameters are used as the initial parameters to retrain disjoint classification networks.

The baseline recognition model are all-convolution neural networks.

Without multi-task learning, the recognition accuracies for MNIST, NotMNIST and FashionMNIST are 99.56\%, 97.22\% and 94.32\% respectively.

With multi-task learning to pre-train the networks, the recognition accuracies are respectively 99.70\%, 97.46\% and 95.25\%.

The results re-affirm that multi-task learning framework, even with data with different genres, does lead to significant improvement.

This paper presents a multi-task neural network for classification on MNIST-like datasets

Multi-task Learning on MNIST Image Datasets


Recent work has demonstrated that neural networks are vulnerable to adversarial examples, i.e., inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network.

To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization.

This approach provides us with a broad and unifying view on much prior work on this topic.

Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal.

In particular, they specify a concrete security guarantee that would protect against a well-defined class of adversaries.

These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks.

They also suggest robustness against a first-order adversary as a natural security guarantee.

We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models.

We provide a principled, optimization-based re-look at the notion of adversarial examples, and develop methods that produce models that are adversarially robust against a wide range of adversaries.

Investigates a minimax formulation of deep network learning to increase their robustness, using projected gradient descent as the main adversary.

This paper proposes to look at making neural networks resistant to adversarial loss through the framework of saddle-point problems.

Towards Deep Learning Models Resistant to Adversarial Attacks

resistant adversarial
adversary
model
deep
network


In recent years there has been a rapid increase in classification methods on graph structured data.

Both in graph kernels and graph neural networks, one of the implicit assumptions of successful state-of-the-art models was that incorporating graph isomorphism features into the architecture leads to better empirical performance.

However, as we discover in this work, commonly used data sets for graph classification have repeating instances which cause the problem of isomorphism bias, i.e. artificially increasing the accuracy of the models by memorizing target information from the training set.

This prevents fair competition of the algorithms and raises a question of the validity of the obtained results.

We analyze 54 data sets, previously extensively used for graph-related tasks, on the existence of isomorphism bias, give a set of recommendations to machine learning practitioners to properly set up their models, and open source new data sets for the future experiments.

Many graph classification data sets have duplicates, thus raising questions about generalization abilities and fair comparison of the models.

The authors discuss isomorphism bias in graph datasets, the overfitting effect in learning networks whenever graph isomorphism features are incorporated within the model, theoretically analogous to data leakage effects.

Understanding Isomorphism Bias in Graph Data Sets


Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently.

However, learning from demonstrations often suffers from the covariate shift problem, which results

in cascading errors of the learned policy.

We introduce a notion of conservatively extrapolated value functions, which provably lead to policies with self-correction.

We design an algorithm Value Iteration with Negative Sampling (VINS) that practically learns such value functions with conservative extrapolation.

We show that VINS can correct mistakes of the behavioral cloning policy on simulated robotics benchmark tasks.

We also propose the algorithm of using VINS to initialize a reinforcement learning algorithm, which is shown to outperform prior works in sample efficiency.

We introduce a notion of conservatively-extrapolated value functions, which provably lead to policies that can self-correct to stay close to the demonstration states, and learn them with a novel negative sampling technique.

An algorithm called value iteration with negative sampling to address the covariate shift problem in imitation learning.

Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling

value function
demonstration
negative sampling
policy
learn


A structured understanding of our world in terms of objects, relations, and hierarchies is an important component of human cognition.

Learning such a structured world model from raw sensory data remains a challenge.

As a step towards this goal, we introduce Contrastively-trained Structured World Models (C-SWMs).

C-SWMs utilize a contrastive approach for representation learning in environments with compositional structure.

We structure each state embedding as a set of object representations and their relations, modeled by a graph neural network.

This allows objects to be discovered from raw pixel observations without direct supervision as part of the learning process.

We evaluate C-SWMs on compositional environments involving multiple interacting objects that can be manipulated independently by an agent, simple Atari games, and a multi-object physics simulation.

Our experiments demonstrate that C-SWMs can overcome limitations of models based on pixel reconstruction and outperform typical representatives of this model class in highly structured environments, while learning interpretable object-based representations.

Contrastively-trained Structured World Models (C-SWMs) learn object-oriented state representations and a relational model of an environment from raw pixel input.

The authors overcome the problem of using pixel-based losses in the construction and learning of structured world models by using a contrastive latent space.

Contrastive Learning of Structured World Models

learning structured world model
contrastive


Neural machine translation (NMT) models learn representations containing substantial linguistic information.

However, it is not clear if such information is fully distributed or if some of it can be attributed to individual neurons.

We develop unsupervised methods for discovering important neurons in NMT models.

Our methods rely on the intuition that different models learn similar properties, and do not require any costly external supervision.

We show experimentally that translation quality depends on the discovered neurons, and find that many of them capture common linguistic phenomena.

Finally, we show how to control NMT translations in predictable ways, by modifying activations of individual neurons.

Unsupervised methods for finding, analyzing, and controlling important neurons in NMT

This work proposes finding "meaningful" neurons in Neural Machine Translation models by ranking based on correlation between pairs of models, different epochs, or different datasets, and proposes a controlling mechanism for the models.

Identifying and Controlling Important Neurons in Neural Machine Translation

control important neuron
neural machine translation
find


Computations for the softmax function in neural network models are expensive when the number of output classes is large.

This can become a significant issue in both training and inference for such models.

In this paper, we present Doubly Sparse Softmax (DS-Softmax), Sparse Mixture of Sparse of Sparse Experts, to improve the efficiency for softmax inference.

During training, our method learns a two-level class hierarchy by dividing entire output class space into several partially overlapping experts.

Each expert is responsible for a learned subset of the output class space and each output class only belongs to a small number of those experts.

During inference, our method quickly locates the most probable expert to compute small-scale softmax.

Our method is learning-based and requires no knowledge of the output class partition space a priori.

We empirically evaluate our method on several real-world tasks and demonstrate that we can achieve significant computation reductions without loss of

We present doubly sparse softmax, the sparse mixture of sparse of sparse experts, to improve the efficiency for softmax inference through exploiting the two-level overlapping hierarchy.

The paper proposes the new Softmax algorithm implementation with two hierarchical levels of sparsity which speeds up the operation in language modeling.

Doubly Sparse: Sparse Mixture of Sparse Experts for Efficient Softmax Inference

sparse mixture sparse
expert
softmax inference
doubly


Our work presents empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance.

Compared to previously studied indicators of generalization, we show that layer rotation has the additional benefit of being easily monitored and controlled, as well as having a network-independent optimum: the training procedures during which all layers' weights reach a cosine distance of 1 from their initialization consistently outperform other configurations -by up to 20% test accuracy.

Finally, our results also suggest that the study of layer rotation can provide a unified framework to explain the impact of weight decay and adaptive gradient methods on generalization.

This paper presents empirical evidence supporting the discovery of an indicator of generalization: the evolution across training of the cosine distance between each layer's weight vector and its initialization.

Layer rotation: a surprisingly simple indicator of generalization in deep networks?


Models of code can learn distributed representations of a program's syntax and semantics to predict many non-trivial properties of a program.

Recent state-of-the-art models leverage highly structured representations of programs, such as trees, graphs and paths therein (e.g. data-flow relations), which are precise and abundantly available for code.

This provides a strong inductive bias towards semantically meaningful relations, yielding more generalizable representations than classical sequence-based models.

Unfortunately, these models primarily rely on graph-based message passing to represent relations in code, which makes them de facto local due to the high cost of message-passing steps, quite in contrast to modern, global sequence-based models, such as the Transformer.

In this work, we bridge this divide between global and structured models by introducing two new hybrid model families that are both global and incorporate structural bias: Graph Sandwiches, which wrap traditional (gated) graph message-passing layers in sequential message-passing layers; and Graph Relational Embedding Attention Transformers (GREAT for short), which bias traditional Transformers with relational information from graph edge types.

By studying a popular, non-trivial program repair task, variable-misuse identification, we explore the relative merits of traditional and hybrid model families for code representation.

Starting with a  graph-based model that already improves upon the prior state-of-the-art for this task by 20%, we show that our proposed hybrid models improve an additional 10-15%, while training both faster and using fewer parameters.

Models of source code that combine global and structural features learn more powerful representations of programs.

A new method to model the source code for the bug repairing task using a sandwich model like [RNN GNN RNN] which significantly improves localization and repair accuracy.

Global Relational Models of Source Code


Recurrent neural networks (RNNs) are particularly well-suited for modeling long-term dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate.

While a number of works attempt to mitigate this effect through gated recurrent units, skip-connections, parametric constraints and design choices, we propose a novel incremental RNN (iRNN), where hidden state vectors keep track of incremental changes, and as such approximate state-vector increments of Rosenblatt's (1962) continuous-time RNNs.

iRNN exhibits identity gradients and is able to account for long-term dependencies (LTD).

We show that our method is computationally efficient overcoming overheads of many existing methods that attempt to improve RNN training, while suffering no performance degradation.

We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks.

Incremental-RNNs resolves exploding/vanishing gradient problem by updating state vectors based on difference between previous state and that predicted by an ODE.

The authors address the problem of signal propagation in recurrent neural networks by building an attractor system for the signal transition and checking whether it converges to an equilibrium.

RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?

equilibrium problem gradient vanish explode


Recent empirical results on over-parameterized deep networks are marked by a striking absence of the classic U-shaped test error curve: test error keeps decreasing in wider networks.

Researchers are actively working on bridging this discrepancy by proposing better complexity measures.

Instead, we directly measure prediction bias and variance for four classification and regression tasks on modern deep networks.

We find that both bias and variance can decrease as the number of parameters grows.

Qualitatively, the phenomenon persists over a number of gradient-based optimizers.

To better understand the role of optimization, we decompose the total variance into variance due to training set sampling and variance due to initialization.

Variance due to initialization is significant in the under-parameterized regime.

In the over-parameterized regime, total variance is much lower and dominated by variance due to sampling.

We provide theoretical analysis in a simplified setting that is consistent with our empirical findings.

We provide evidence against classical claims about the bias-variance tradeoff and propose a novel decomposition for variance.

A Modern Take on the Bias-Variance Tradeoff in Neural Networks


Real world images often contain large amounts of private / sensitive information that should be carefully protected without reducing their utilities.

In this paper, we propose a privacy-preserving deep learning framework with a learnable ob- fuscator for the image classification task.

Our framework consists of three mod- els: learnable obfuscator, classifier and reconstructor.

The learnable obfuscator is used to remove the sensitive information in the images and extract the feature maps from them.

The reconstructor plays the role as an attacker, which tries to recover the image from the feature maps extracted by the obfuscator.

In order to best protect users’ privacy in images, we design an adversarial training methodol- ogy for our framework to optimize the obfuscator.

Through extensive evaluations on real world datasets, both the numerical metrics and the visualization results demonstrate that our framework is qualified to protect users’ privacy and achieve a relatively high accuracy on the image classification task.

We proposed a novel deep learning image classification framework that can both accurately classify images and protect users' privacy.

This paper proposes a framework which preserves the private information in the image and doesn’t compromise the usability of the image.

This current work suggests using adversarial networks to obfuscate images and thus allow collecting them without privacy concerns to use them for training machine learning models.

A PRIVACY-PRESERVING IMAGE CLASSIFICATION FRAMEWORK WITH A LEARNABLE OBFUSCATOR


Bitcoin is a virtual coinage system that enables users to trade virtually free of a central trusted authority.

All transactions on the Bitcoin blockchain are publicly available for viewing, yet as Bitcoin is built mainly for security it’s original structure does not allow for direct analysis of address transactions.

Existing analysis methods of the Bitcoin blockchain can be complicated, computationally expensive or inaccurate.

We propose a computationally efficient model to analyze bitcoin blockchain addresses and allow for their use with existing machine learning algorithms.

We compare our approach against Multi Level Sequence Learners (MLSLs), one of the best performing models on bitcoin address data.

a 2vec model for cryptocurrency transaction graphs

The paper proposes to use an autoencoder, networkX, and node2Vec to predict whether a Bitcoin address will become empty after a year, but the results are worse than an existing baseline.

Address2vec: Generating vector embeddings for blockchain analytics