Targeted clean-label poisoning is a type of adversarial attack on machine learning systems where the adversary injects a few correctly-labeled, minimally-perturbed samples into the training data thus causing the deployed model to misclassify a particular test sample during inference.

Although defenses have been proposed for general poisoning attacks (those which aim to reduce overall test accuracy), no reliable defense for clean-label attacks has been demonstrated, despite the attacks' effectiveness and their realistic use cases.

In this work, we propose a set of simple, yet highly-effective defenses against these attacks.

We test our proposed approach against two recently published clean-label poisoning attacks, both of which use the CIFAR-10 dataset.

After reproducing their experiments, we demonstrate that our defenses are able to detect over 99% of poisoning examples in both attacks and remove them without any compromise on model performance.

Our simple defenses show that current clean-label poisoning attack strategies can be annulled, and serve as strong but simple-to-implement baseline defense for which to test future clean-label poisoning attacks.

We present effective defenses to clean-label poisoning attacks.

Strong Baseline Defenses Against Clean-Label Poisoning Attacks


Abstract reasoning, particularly in the visual domain, is a complex human ability, but it remains a challenging problem for artificial neural learning systems.

In this work we propose MXGNet, a multilayer graph neural network for multi-panel diagrammatic reasoning tasks.

MXGNet combines three powerful concepts, namely, object-level representation, graph neural networks and multiplex graphs, for solving visual reasoning tasks.

MXGNet first extracts object-level representations for each element in all panels of the diagrams, and then forms a multi-layer multiplex graph capturing multiple relations between objects across different diagram panels.

MXGNet summarises the multiple graphs extracted from the diagrams of the task, and uses this summarisation to pick the most probable answer from the given candidates.

We have tested MXGNet on two types of diagrammatic reasoning tasks, namely Diagram Syllogisms and Raven Progressive Matrices (RPM).

For an Euler Diagram Syllogism task MXGNet achieves state-of-the-art accuracy of 99.8%.

For PGM and RAVEN, two comprehensive datasets for RPM reasoning, MXGNet outperforms the state-of-the-art models by a considerable margin.

MXGNet is a multilayer, multiplex graph based architecture which achieves good performance on various diagrammatic reasoning tasks.

Abstract Diagrammatic Reasoning with Multiplex Graph Networks


Semantic structure extraction for spreadsheets includes detecting table regions, recognizing structural components and classifying cell types.

Automatic semantic structure extraction is key to automatic data transformation from various table structures into canonical schema so as to enable data analysis and knowledge discovery.

However, they are challenged by the diverse table structures and the spatial-correlated semantics on cell grids.

To learn spatial correlations and capture semantics on spreadsheets, we have developed a novel learning-based framework for spreadsheet semantic structure extraction.

First, we propose a multi-task framework that learns table region, structural components and cell types jointly; second, we leverage the advances of the recent language model to capture semantics in each cell value; third, we build a large human-labeled dataset with broad coverage of table structures.

Our evaluation shows that our proposed multi-task framework is highly effective that outperforms the results of training each task separately.

We propose a novel multi-task framework that learns table detection, semantic component recognition and cell type classification for spreadsheet tables with promising results.

Semantic Structure Extraction for Spreadsheet Tables with a Multi-task Learning Architecture


Open-domain dialogue generation has gained increasing attention in Natural Language Processing.

Comparing these methods requires a holistic means of dialogue evaluation.

Human ratings are deemed as the gold standard.

As human evaluation is inefficient and costly, an automated substitute is desirable.

In this paper, we propose holistic evaluation metrics which capture both the quality and diversity of dialogues.

Our metrics consists of (1) GPT-2 based context coherence between sentences in a dialogue, (2) GPT-2 based fluency in phrasing, and, (3) $n$-gram based diversity in responses to augmented queries.

The empirical validity of our metrics is demonstrated by strong correlation with human judgments.

We provide the associated code, datasets and human ratings.

We propose automatic metrics to holistically evaluate open-dialogue generation and they strongly correlate with human evaluation.

Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation


Convolution Neural Network (CNN) has gained tremendous success in computer vision tasks with its outstanding ability to capture the local latent features.

Recently, there has been an increasing interest in extending CNNs to the general spatial domain.

Although various types of graph convolution and geometric convolution methods have been proposed, their connections to traditional 2D-convolution are not well-understood.

In this paper, we show that depthwise separable convolution is a path to unify the two kinds of convolution methods in one mathematical view, based on which we derive a novel Depthwise Separable Graph Convolution that subsumes existing graph convolution methods as special cases of our formulation.

Experiments show that the proposed approach consistently outperforms other graph convolution and geometric convolution baselines on benchmark datasets in multiple domains.

We devise a novel Depthwise Separable Graph Convolution (DSGC) for the generic spatial domain data, which is highly compatible with depthwise separable convolution.

Learning Graph Convolution Filters from Data Manifold


Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales.

Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments.

Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave.

This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (~3 ms) between note labels and audio waveforms.

The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music.

We train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure, enabled by the new MAESTRO dataset.

Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset


Variational inference based on chi-square divergence minimization (CHIVI) provides a way to approximate a model's posterior while obtaining an upper bound on the marginal likelihood.

However, in practice CHIVI relies on Monte Carlo (MC) estimates of an upper bound objective that at modest sample sizes are not guaranteed to be true bounds on the marginal likelihood.

This paper provides an empirical study of CHIVI performance on a series of synthetic inference tasks.

We show that CHIVI is far more sensitive to initialization than classic VI based on KL minimization, often needs a very large number of samples (over a million), and may not be a reliable upper bound.

We also suggest possible ways to detect and alleviate some of these pathologies, including diagnostic bounds and initialization strategies.

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO

Challenges in Computing and Optimizing Upper Bounds of Marginal Likelihood based on Chi-Square Divergences


It has been widely recognized that adversarial examples can be easily crafted to fool deep networks, which mainly root from the locally non-linear behavior nearby input examples.

Applying mixup in training provides an effective mechanism to improve generalization performance and model robustness against adversarial perturbations, which introduces the globally linear behavior in-between training examples.

However, in previous work, the mixup-trained models only passively defend adversarial attacks in inference by directly classifying the inputs, where the induced global linearity is not well exploited.

Namely, since the locality of the adversarial perturbations, it would be more efficient to actively break the locality via the globality of the model predictions.

Inspired by simple geometric intuition, we develop an inference principle, named mixup inference (MI), for mixup-trained models.

MI mixups the input with other random clean samples, which can shrink and transfer the equivalent perturbation if the input is adversarial.

Our experiments on CIFAR-10 and CIFAR-100 demonstrate that MI can further improve the adversarial robustness for the models trained by mixup and its variants.

We exploit the global linearity of the mixup-trained models in inference to break the locality of the adversarial perturbations.

Mixup Inference: Better Exploiting Mixup to Defend Adversarial Attacks


Fine-tuning language models, such as BERT, on domain specific corpora has proven to be valuable in domains like scientific papers and biomedical text.

In this paper, we show that fine-tuning BERT on legal documents similarly provides valuable improvements on NLP tasks in the legal domain.

Demonstrating this outcome is significant for analyzing commercial agreements, because obtaining large legal corpora is challenging due to their confidential nature.

As such, we show that having access to large legal corpora is a competitive advantage for commercial applications, and academic research on analyzing contracts.

Fine-tuning BERT on legal corpora provides marginal, but valuable, improvements on NLP tasks in the legal domain.

BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding


We present a deep generative model for unsupervised text style transfer that unifies previously proposed non-generative techniques.

Our probabilistic approach models non-parallel data from two domains as a partially observed parallel corpus.

By hypothesizing a parallel latent sequence that generates each observed sequence, our model learns to transform sequences from one domain to another in a completely unsupervised fashion.

In contrast with traditional generative sequence models (e.g. the HMM), our model makes few assumptions about the data it generates: it uses a recurrent language model as a prior and an encoder-decoder as a transduction distribution.

While computation of marginal data likelihood is intractable in this model class, we show that amortized variational inference admits a practical surrogate.

Further, by drawing connections between our variational objective and other recent unsupervised style transfer and machine translation techniques, we show how our probabilistic view can unify some known non-generative objectives such as backtranslation and adversarial loss.

Finally, we demonstrate the effectiveness of our method on a wide range of unsupervised style transfer tasks, including sentiment transfer, formality transfer, word decipherment, author imitation, and related language translation.

Across all style transfer tasks, our approach yields substantial gains over state-of-the-art non-generative baselines, including the state-of-the-art unsupervised machine translation techniques that our approach generalizes.

Further, we conduct experiments on a standard unsupervised machine translation task and find that our unified approach matches the current state-of-the-art.

We formulate a probabilistic latent sequence model to tackle unsupervised text style transfer, and show its effectiveness across a suite of unsupervised text style transfer tasks.

A Probabilistic Formulation of Unsupervised Text Style Transfer

unsupervised text style transfer
probabilistic


Current practice in machine learning is to employ deep nets in an overparametrized limit, with the nominal number of parameters typically exceeding the number of measurements.

This resembles the situation in compressed sensing, or in sparse regression with $l_1$ penalty terms, and provides a theoretical avenue for understanding phenomena that arise in the context of deep nets.

One such phenonemon is the success of deep nets in providing good generalization in an interpolating regime with zero training error.

Traditional statistical practice calls for regularization or smoothing to prevent "overfitting" (poor generalization performance).

However, recent work shows that there exist data interpolation procedures which are statistically consistent and provide good generalization performance\cite{belkin2018overfitting} ("perfect fitting").

In this context, it has been suggested that "classical" and "modern" regimes for machine learning are separated by a peak in the generalization error ("risk") curve, a phenomenon dubbed "double descent"\cite{belkin2019reconciling}.

While such overfitting peaks do exist and arise from ill-conditioned design matrices, here we challenge the interpretation of the overfitting peak as demarcating the regime where good generalization occurs under overparametrization.

We propose a model of Misparamatrized Sparse Regression (MiSpaR) and analytically compute the GE curves for $l_2$ and $l_1$ penalties.

We show that the overfitting peak arising in the interpolation limit is dissociated from the regime of good generalization.

The analytical expressions are obtained in the so called "thermodynamic" limit.

We find an additional interesting phenomenon: increasing overparametrization in the fitting model increases sparsity, which should intuitively improve performance of $l_1$ penalized regression.

However, at the same time, the relative number of measurements decrease compared to the number of fitting parameters, and eventually overparametrization does lead to poor generalization.

Nevertheless, $l_1$ penalized regression can show good generalization performance under conditions of data interpolation even with a large amount of overparametrization.

These results provide a theoretical avenue into studying inverse problems in the interpolating regime using overparametrized fitting functions such as deep nets.

Proposes an analytically tractable model and inference procedure (misparametrized sparse regression, inferred using L_1 penalty and studied in the data-interpolation limit) to study deep-net related phenomena in the context of inverse problems.

Compressed Sensing and Overparametrized Networks: Overfitting Peaks in a Model of Misparametrized Sparse Regression in the Interpolation Limit

misparametrize sparse regression
model
limit


Hashing-based collaborative filtering learns binary vector representations (hash codes) of users and items, such that recommendations can be computed very efficiently using the Hamming distance, which is simply the sum of differing bits between two hash codes.

A problem with hashing-based collaborative filtering using the Hamming distance, is that each bit is equally weighted in the distance computation, but in practice some bits might encode more important properties than other bits, where the importance depends on the user.

To this end, we propose an end-to-end trainable variational hashing-based collaborative filtering approach that uses the novel concept of self-masking: the user hash code acts as a mask on the items (using the Boolean AND operation), such that it learns to encode which bits are important to the user, rather than the user's preference towards the underlying item property that the bits represent.

This allows a binary user-level importance weighting of each item without the need to store additional weights for each user.

We experimentally evaluate our approach against state-of-the-art baselines on 4 datasets, and obtain significant gains of up to 12% in NDCG.

We also make available an efficient implementation of self-masking, which experimentally yields <4% runtime overhead compared to the standard Hamming distance.

We propose a new variational hashing-based collaborative filtering approach optimized for a novel self-mask variant of the Hamming distance, which outperforms state-of-the-art by up to 12% on NDCG.

Variational Hashing-based Collaborative Filtering with Self-Masking


Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search.

This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit that achieves performance equivalent to that of best fixed batch-size.

At each epoch, the RMGD samples a batch size according to a certain probability distribution proportional to a batch being successful in reducing the loss function.

Sampling from this probability provides a mechanism for exploring different batch size and exploiting batch sizes with history of success.

After obtaining the validation loss at each epoch with the sampled batch size, the probability distribution is updated to incorporate the effectiveness of the sampled batch size.

Experimental results show that the RMGD achieves performance better than the best performing single batch size.

It is surprising that the RMGD achieves better performance than grid search.

Furthermore, it attains this performance in a shorter amount of time than grid search.

An optimization algorithm that explores various batch sizes based on probability and automatically exploits successful batch size which minimizes validation loss.

A Resizable Mini-batch Gradient Descent based on a Multi-Armed Bandit


Knowledge graph has gained increasing attention in recent years for its successful applications of numerous tasks.

Despite the rapid growth of knowledge construction, knowledge graphs still suffer from severe incompletion and inevitably involve various kinds of errors.

Several attempts have been made to complete knowledge graph as well as to detect noise.

However, none of them considers unifying these two tasks even though they are inter-dependent and can mutually boost the performance of each other.

In this paper, we proposed to jointly combine these two tasks with a unified Generative Adversarial Networks (GAN) framework to learn noise-aware knowledge graph embedding.

Extensive experiments have demonstrated that our approach is superior to existing state-of-the-art algorithms both in regard to knowledge graph completion and error detection.

We proposed a unified Generative Adversarial Networks (GAN) framework to learn noise-aware knowledge graph embedding.

NoiGAN: NOISE AWARE KNOWLEDGE GRAPH EMBEDDING WITH GAN


un-normalized models, have had recent successes in continuous spaces.

However, they have not been successfully applied to model text sequences.

While decreasing the energy at training samples is straightforward, mining (negative) samples where the energy should be increased is difficult.

In part, this is because standard gradient-based methods are not readily applicable when the input is high-dimensional and discrete.

Here, we side-step this issue by generating negatives using pre-trained auto-regressive language models.

The EBM then works
in the {\em residual} of the language model; and is trained to discriminate real text from text generated by the auto-regressive models.

We  investigate the generalization ability of residual EBMs, a pre-requisite for using them in other applications.

We extensively analyze generalization for the task of classifying whether an input is machine or human generated, a natural task given the training loss and how we mine negatives.

Overall, we observe that EBMs can generalize remarkably well to changes in the architecture of the generators producing negatives.

However, EBMs exhibit more sensitivity to the training set used by such generators.

A residual EBM for text whose formulation is equivalent to discriminating between human and machine generated text. We study its generalization behavior.

Residual EBMs: Does Real vs. Fake Text Discrimination Generalize?


Semi-supervised learning, i.e. jointly learning from labeled an unlabeled samples, is an active research topic due to its key role on relaxing human annotation constraints.

In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different  perturbations of unlabeled samples.

We, conversely, propose to learn from unlabeled data by generating soft pseudo-labels using the network predictions.

We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it.

The proposed approach achieves state-of-the-art results in CIFAR-10/100 and Mini-ImageNet despite being much simpler than other state-of-the-art.

These results demonstrate that pseudo-labeling can outperform consistency regularization methods, while the opposite was supposed in previous work.

Pseudo-labeling has shown to be a weak alternative for semi-supervised learning. We, conversely, demonstrate that dealing with confirmation bias with several regularizations makes pseudo-labeling a suitable approach.

Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning

confirmation bias
pseudo-labele
semi-supervised learning


Model-free reinforcement learning (RL) has been proven to be a powerful, general tool for learning complex behaviors.

However, its sample efficiency is often impractically large for solving challenging real-world problems, even for off-policy algorithms such as Q-learning.

A limiting factor in classic model-free RL is that the learning signal consists only of scalar rewards, ignoring much of the rich information contained in state transition tuples.

Model-based RL uses this information, by training a predictive model, but often does not achieve the same asymptotic performance as model-free RL due to model bias.

We introduce temporal difference models (TDMs), a family of goal-conditioned value functions that can be trained with model-free learning and used for model-based control.

TDMs combine the benefits of model-free and model-based RL: they leverage the rich information in state transitions to learn very efficiently, while still attaining asymptotic performance that exceeds that of direct model-based RL methods.

Our experimental results show that, on a range of continuous control tasks, TDMs provide a substantial improvement in efficiency compared to state-of-the-art model-based and model-free methods.

We show that a special goal-condition value function trained with model free methods can be used within model-based control, resulting in substantially better sample efficiency and performance.

Temporal Difference Models: Model-Free Deep RL for Model-Based Control


We introduce a neural architecture to perform amortized approximate Bayesian inference over latent random permutations of two sets of objects.

The method involves approximating permanents of matrices of pairwise probabilities using recent ideas on functions defined over sets.

Each sampled permutation comes with a probability estimate, a quantity unavailable in MCMC approaches.

We illustrate the method in sets of 2D points and MNIST images.

A novel neural architecture for efficient amortized inference over latent permutations

Neural Permutation Processes


Machine learned large-scale retrieval systems require a large amount of training data representing query-item relevance.

However, collecting users' explicit feedback is costly.

In this paper, we propose to leverage user logs and implicit feedback as auxiliary objectives to improve relevance modeling in retrieval systems.

Specifically, we adopt a two-tower neural net architecture to model query-item relevance given both collaborative and content information.

By introducing auxiliary tasks trained with much richer implicit user feedback data, we improve the quality and resolution for the learned representations of queries and items.

Applying these learned representations to an industrial retrieval system has delivered significant improvements.

We propose a novel two-tower shared-bottom model architecture for transferring knowledge from rich implicit feedbacks to predict relevance for large-scale retrieval systems.

Improving Relevance Prediction with Transfer Learning in Large-scale Retrieval Systems

large-scale retrieval system
transfer
relevance


The ability to autonomously explore and navigate a physical space is a fundamental requirement for virtually any mobile autonomous agent, from household robotic vacuums to autonomous vehicles.

Traditional SLAM-based approaches for exploration and navigation largely focus on leveraging scene geometry, but fail to model dynamic objects (such as other agents) or semantic constraints (such as wet floors or doorways).

Learning-based RL agents are an attractive alternative because they can incorporate both semantic and geometric information, but are notoriously sample inefficient, difficult to generalize to novel settings, and are difficult to interpret.

In this paper, we combine the best of both worlds with a modular approach that {\em learns} a spatial representation of a scene that is trained to be effective when coupled with traditional geometric planners.

Specifically, we design an agent that learns to predict a spatial affordance map that elucidates what parts of a scene are navigable through active self-supervised experience gathering.

In contrast to most simulation environments that assume a static world, we evaluate our approach in the VizDoom simulator, using large-scale randomly-generated maps containing a variety of dynamic actors and hazards.

We show that learned affordance maps can be used to augment traditional approaches for both exploration and navigation, providing significant improvements in performance.

We address the task of autonomous exploration and navigation using spatial affordance maps that can be learned in a self-supervised manner, these outperform classic geometric baselines while being more sample efficient than contemporary RL algorithms

Learning to Move with Affordance Maps