Deep approaches to anomaly detection have recently shown promising results over shallow methods on large and complex datasets.Typically anomaly detection is treated as an unsupervised learning problem.In practice however, one may have---in addition to a large set of unlabeled samples---access to a small pool of labeled samples, e.g. a subset verified by some domain expert as being normal or anomalous.Semi-supervised approaches to anomaly detection aim to utilize such labeled samples, but most proposed methods are limited to merely including labeled normal samples.Only a few methods take advantage of labeled anomalies, with existing deep approaches being domain-specific.In this work we present Deep SAD, an end-to-end deep methodology for general semi-supervised anomaly detection.Using an information-theoretic perspective on anomaly detection, we derive a loss motivated by the idea that the entropy of the latent distribution for normal data should be lower than the entropy of the anomalous distribution.We demonstrate in extensive experiments on MNIST, Fashion-MNIST, and CIFAR-10, along with other anomaly detection benchmark datasets, that our method is on par or outperforms shallow, hybrid, and deep competitors, yielding appreciable performance improvements even when provided with only little labeled data.

我们介绍了Deep SAD，这是一种用于一般半监督性异常检测的深度方法，特别是利用了标记的异常情况。

一种新的方法来寻找异常数据，当给定一些标记的异常数据时，应用信息理论衍生的损失，基于正常数据通常比异常数据的熵低。

提出了在无标签数据、有标签的阳性数据和有标签的阴性数据的情况下的异常检测框架，并建议从信息论的角度来处理半监督的AD。


To analyze deep ReLU network, we adopt a student-teacher setting in which an over-parameterized student network learns from the output of a fixed teacher network of the same depth, with Stochastic Gradient Descent (SGD).Our contributions are two-fold.First, we prove that when the gradient is zero (or bounded above by a small constant) at every data point in training, a situation called  \emph{interpolation setting}, there exists many-to-one \emph{alignment} between student and teacher nodes in the lowest layer under mild conditions.This suggests that generalization in unseen dataset is achievable, even the same condition often leads to zero training error.Second, analysis of noisy recovery and training dynamics in 2-layer network shows that strong teacher nodes (with large fan-out weights) are learned first and subtle teacher nodes are left unlearned until late stage of training.As a result, it could take a long time to converge into these small-gradient critical points.Our analysis shows that over-parameterization plays two roles: (1) it is a necessary condition for alignment to happen at the critical points, and (2) in training dynamics, it helps student nodes cover more teacher nodes with fewer iterations.Both improve generalization.Experiments justify our finding.

本文分析了在师生环境下通过SGD训练深度ReLU网络的训练动态和关键点。

研究学生-教师多层ReLU网络中的过度参数化，理论部分是关于教师-学生设置的SGD临界点，启发式和经验式部分是关于SDG算法作为教师网络的函数的动态性。


We study the convergence of gradient descent (GD) and stochastic gradient descent (SGD) for training $L$-hidden-layer linear residual networks (ResNets).We prove that for training deep residual networks with certain linear transformations at input and output layers, which are fixed throughout training, both GD and SGD with zero initialization on all hidden weights can converge to the global minimum of the training loss.Moreover, when specializing to appropriate Gaussian random linear transformations, GD and SGD provably optimize wide enough deep linear ResNets.Compared with the global convergence result of GD for training standard deep linear networks \citep{du2019width}, our condition on the neural network width is sharper by a factor of $O(\kappa L)$, where $\kappa$ denotes the condition number of the covariance matrix of the training data.In addition, for the first time we establish the global convergence of SGD for training deep linear ResNets and prove a linear convergence rate when the global minimum is $0$.

在输入和输出线性变换的某些条件下，GD和SGD都可以实现全域收敛，用于训练深度线性ResNets。

作者研究了梯度下降在训练深度线性残差网络中的收敛性，并建立了GD/SGD的全域收敛和SG/SGD的线性收敛率。

研究GD和SGD在深层线性网络上的收敛特性，并证明在输入和输出变换的某些条件下，在初始化为零的情况下，GD和SGD会收敛到全域最小值。


In this paper, we empirically investigate the training journey of deep neural networks relative to fully trained shallow machine learning models.We observe that the deep neural networks (DNNs) train by learning to correctly classify shallow-learnable examples in the early epochs before learning the harder examples.We build on this observation this to suggest a way for partitioning the dataset into hard and easy subsets that can be used for improving the overall training process.Incidentally, we also found evidence of a subset of intriguing examples across all the datasets we considered, that were shallow learnable but not deep-learnable.In order to aid reproducibility, we also duly release our code for this work at https://github.com/karttikeya/Shallow_to_Deep/

我们分析了深度网络的训练过程，并表明它们从快速学习浅层可分类的例子开始，慢慢地归纳到更难的数据点。


While much recent work has targeted learning deep discrete latent variable models with variational inference, this setting remains challenging, and it is often necessary to make use of potentially high-variance gradient estimators in optimizing the ELBO.As an alternative, we propose to optimize a non-ELBO objective derived from the Bethe free energy approximation to an MRF's partition function.This objective gives rise to a saddle-point learning problem, which we train inference networks to approximately optimize.The derived objective requires no sampling, and can be efficiently computed for many MRFs of interest.We evaluate the proposed approach in learning high-order neural HMMs on text, and find that it often outperforms other approximate inference schemes in terms of true held-out log likelihood.At the same time, we find that all the approximate inference-based approaches to learning high-order neural HMMs we consider underperform learning with exact inference by a significant margin.

用来自Bethe分区函数近似的鞍点目标学习深层潜变量MRFs。

一种学习深度潜伏变量MRF的方法，其优化目标是利用Bethe自由能，同时也解决了Bethe自由能优化的基本限制。

基于Bethe自由能和摊销推理的潜变量MRF学习目标，与优化标准ELBO不同。


In an explanation generation problem, an agent needs to identify and explain the reasons for its decisions to another agent.Existing work in this area is mostly confined to planning-based systems that use automated planning approaches to solve the problem.In this paper, we approach this problem from a new perspective, where we propose a general logic-based framework for explanation generation.In particular, given a knowledge base $KB_1$ that entails a formula $\phi$ and a second knowledge base $KB_2$ that does not entail $\phi$, we seek to identify an explanation $\epsilon$ that is a subset of $KB_1$ such that the union of $KB_2$ and $\epsilon$ entails $\phi$.We define two types of explanations, model- and proof-theoretic explanations, and use cost functions to reflect preferences between explanations.Further, we present our algorithm implemented for propositional logic that compute such explanations and empirically evaluate it in random knowledge bases and a planning domain.

一个使用逻辑学生成解释的一般框架。

本文从KR的角度研究了解释的生成，并进行了测量随机公式和Blocksworld实例中的公式的解释大小和运行时间的实验。

本文提供了一个关于两个知识库之间解释的视角，并与规划文献中关于模型调和的工作并行。


Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem.This problem can be typically resolved by the rectified linear unit (ReLU) activation.However, here we show that even for such activation, deep and narrow neural networks (NNs) will converge to erroneous mean or median states of the target function depending on the loss with high probability.Deep and narrow NNs are encountered in solving partial differential equations with high-order derivatives.We demonstrate this collapse of such NNs both numerically and theoretically, and provide estimates of the probability of collapse.We also construct a diagram of a safe region for designing NNs that avoid the collapse to erroneous states.Finally, we examine different ways of initialization and normalization that may avoid the collapse problem.Asymmetric initializations may reduce the probability of collapse but do not totally eliminate it.

深度和狭义的神经网络将根据损失的情况，高概率地收敛到目标函数的错误的平均值或中位数状态。

本文研究了深而窄的网络的失败模式，重点是尽可能用小的模型，对于这些模型来说，不希望发生的行为。

本文表明，如果隐层宽度太小，深度ReLU神经网络的训练将以高概率收敛到一个恒定的分类器，而不是随机初始化。


We study adversarial robustness of neural networks from a margin maximization perspective, where margins are defined as the distances from inputs to a classifier's decision boundary.Our study shows that maximizing margins can be achieved by minimizing the adversarial loss on the decision boundary at the "shortest successful perturbation", demonstrating a close connection between adversarial losses and the margins.We propose Max-Margin Adversarial (MMA) training to directly maximize the margins to achieve adversarial robustness. Instead of adversarial training with a fixed $\epsilon$, MMA offers an improvement by enabling adaptive selection of the "correct" $\epsilon$ as the margin individually for each datapoint.In addition, we rigorously analyze adversarial training with the perspective of margin maximization, and provide an alternative interpretation for adversarial training, maximizing either a lower or an upper bound of the margins.Our experiments empirically confirm our theory and demonstrate MMA training's efficacy on the MNIST and CIFAR10 datasets w.r.t. $\ell_\infty$ and $\ell_2$ robustness.

我们提出MMA训练来直接最大化输入空间余量，以便主要通过消除特定固定失真界限的要求来提高对抗性的稳定性。

通过最大化输入到决策边界的最短边际，一种基于自适应边际的对抗性训练方法来训练稳定的DNN，这使得大扰动下的对抗性训练成为可能。

介绍了一种针对对抗性攻击的稳定学习方法，其中输入空间余量被直接最大化，并介绍了最大余量的软性变体。


Many anomaly detection methods exist that perform well on low-dimensional problems however there is a notable lack of effective methods for high-dimensional spaces, such as images.Inspired by recent successes in deep learning we propose a novel approach to anomaly detection using generative adversarial networks.Given a sample under consideration, our method is based on searching for a good representation of that sample in the latent space of the generator; if such a representation is not found, the sample is deemed anomalous.  We achieve state-of-the-art performance on standard image benchmark datasets and visual inspection of the most anomalous samples reveals that our method does indeed return anomalies.

我们提出了一种用GANs进行异常检测的方法，即通过搜索生成器的潜在空间来寻找好的样本表示。

作者提出使用GAN进行异常检测，采用基于梯度下降法来迭代更新潜在表征，并对生成器进行了新的参数更新。

一种基于GAN的方法来进行图像数据的异常检测，其中生成器的潜在空间被探索，以找到测试图像的代表。


Variational inference (VI) and Markov chain Monte Carlo (MCMC) are approximate posterior inference algorithms that are often said to have complementary strengths, with VI being fast but biased and MCMC being slower but asymptotically unbiased.In this paper, we analyze gradient-based MCMC and VI procedures and find theoretical and empirical evidence that these procedures are not as different as one might think.In particular, a close examination of the Fokker-Planck equation that governs the Langevin dynamics (LD) MCMC procedure reveals that LD implicitly follows a gradient flow that corresponds to a variational inference procedure based on optimizing a nonparametric normalizing flow.This result suggests that the transient bias of LD (due to too few warmup steps) may track that of VI (due to too few optimization steps), up to differences due to VIâ€™s parameterization and asymptotic bias.Empirically, we find that the transient biases of these algorithms (and momentum-accelerated versions) do evolve similarly.This suggests that practitioners with a limited time budget may get more accurate results by running an MCMC procedure (even if itâ€™s far from burned in) than a VI procedure, as long as the variance of the MCMC estimator can be dealt with (e.g., by running many parallel chains).

基于梯度的MCMC和变异推理算法的短暂行为比人们想象的更相似，这让人们对变异推理比MCMC更快的说法产生怀疑。


Graph Convolutional Networks (GCNs) have recently been shown to be quite successful in modeling graph-structured data.However, the primary focus has been on handling simple undirected graphs.Multi-relational graphs are a more general and prevalent form of graphs where each edge has a label and direction associated with it.Most of the existing approaches to handle such graphs suffer from over-parameterization and are restricted to learning representations of nodes only.In this paper, we propose CompGCN, a novel Graph Convolutional framework which jointly embeds both nodes and relations in a relational graph.CompGCN leverages a variety of entity-relation composition operations from Knowledge Graph Embedding techniques and scales with the number of relations.It also generalizes several of the existing multi-relational GCN methods.We evaluate our proposed method on multiple tasks such as node classification, link prediction, and graph classification, and achieve demonstrably superior results.We make the source code of CompGCN available to foster reproducible research.

用于多关系图的基于组合的图卷积框架。

作者在多关系图上开发了GCN，并提出了CompGCN，它利用知识图嵌入的洞察力，学习节点和关系表示，以缓解过度参数化的问题。

本文介绍了一个多关系图的GCN框架，并将现有的几种知识图谱嵌入方法概括为一个框架。


State-of-the-art neural machine translation methods employ massive amounts of parameters.Drastically reducing computational costs of such methods without affecting performance has been up to this point unsolved.In this work, we propose a quantization strategy tailored to the Transformer architecture.We evaluate our method on the WMT14 EN-FR and WMT14 EN-DE translation tasks and achieve state-of-the-art quantization results for the Transformer, obtaining no loss in BLEU scores compared to the non-quantized baseline.We further compress the Transformer by showing that, once the model is trained, a good portion of the nodes in the encoder can be removed without causing any loss in BLEU.

我们将变形器完全量化为8位，与全精度模型相比，提高了翻译质量。

一种对机器翻译模型Transformer进行量化的8位元量化方法，提出在推理过程中使用统一的最小-最大量化，并在量化前进行桶状权重以减少量化误差。

一种通过量化技术减少所需内存空间的方法，重点是为Transformer架构减少。


Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems.However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes.We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space.The resulting approach, latent embedding optimization (LEO), decouples the gradient-based adaptation procedure from the underlying high-dimensional space of model parameters.Our evaluation shows that LEO can achieve state-of-the-art performance on the competitive miniImageNet and tieredImageNet few-shot classification tasks.Further analysis indicates LEO is able to capture uncertainty in the data, and can perform adaptation more effectively by optimizing in latent space.

Latent Embedding Optimization（LEO）是一种新型的基于梯度的元学习器，在具有挑战性的5-way 1-shot和5-shot miniImageNet和tieredImageNet分类任务上具有最先进的性能。

一个新的元学习框架，学习与数据相关的潜空间，在潜空间中快速适应，对少样本学习有效，对适应有任务依赖的初始化，对多模态任务分布效果好。

本文提出了一种用于元学习的潜在嵌入优化方法，并声称其贡献在于将基于优化的元学习技术与模型参数的高维空间脱离。


We introduce an approach for augmenting model-free deep reinforcement learning agents with a mechanism for relational reasoning over structured representations, which improves performance, learning efficiency, generalization, and interpretability.Our architecture encodes an image as a set of vectors, and applies an iterative message-passing procedure to discover and reason about relevant entities and relations in a scene.In six of seven StarCraft II Learning Environment mini-games, our agent achieved state-of-the-art performance, and surpassed human grandmaster-level on four.In a novel navigation and planning task, our agent's performance and learning efficiency far exceeded non-relational baselines, it was able to generalize to more complex scenes than it had experienced during training.Moreover, when we examined its learned internal representations, they reflected important structure about the problem and the agent's intentions.The main contribution of this work is to introduce techniques for representing and reasoning about states in model-free deep reinforcement learning agents via relational inductive biases.Our experiments show this approach can offer advantages in efficiency, generalization, and interpretability, and can scale up to meet some of the most challenging test environments in modern artificial intelligence.

关系性归纳偏见提高了无模型强化学习代理中的异常概括能力

一个共享的关系网络架构，用于参数化行为者和批评者网络，专注于分布式优势行为者-批评者算法，用关于环境的关系知识增强无模型的深度强化技术，以便行为者能够学习可解释的状态表示。

在无模型RL的背景下，对与关系网络相结合的自我注意机制进行了定量和定性的分析和评估。


Image translation between two domains is a class of problems aiming to learn mapping from an input image in the source domain to an output image in the target domain.It has been applied to numerous applications, such as data augmentation, domain adaptation, and unsupervised training.When paired training data is not accessible, image translation becomes an ill-posed problem.We constrain the problem with the assumption that the translated image needs to be perceptually similar to the original image and also appears to be drawn from the new domain, and propose a simple yet effective image translation model consisting of a single generator trained with a self-regularization term and an adversarial term.We further notice that existing image translation techniques are agnostic to the subjects of interest and often introduce unwanted changes or artifacts to the input.Thus we propose to add an attention module to predict an attention map to guide the image translation process.The module learns to attend to key parts of the image while keeping everything else unaltered, essentially avoiding undesired artifacts or changes.The predicted attention map also opens door to applications such as unsupervised segmentation and saliency detection.Extensive experiments and evaluations show that our model while being simpler, achieves significantly better performance than existing image translation methods.

我们提出了一个简单的生成模型，用于无监督的图像翻译和显着性检测。


Building deep neural networks to control autonomous agents which have to interact in real-time with the physical world, such as robots or automotive vehicles, requires a seamless integration of time into a networkâ€™s architecture.The central question of this work is, how the temporal nature of reality should be reflected in the execution of a deep neural network and its components.Most artificial deep neural networks are partitioned into a directed graph of connected modules or layers and the layers themselves consist of elemental building blocks, such as single units.For most deep neural networks, all units of a layer are processed synchronously and in parallel, but layers themselves are processed in a sequential manner.In contrast, all elements of a biological neural network are processed in parallel.In this paper, we define a class of networks between these two extreme cases.These networks are executed in a streaming or synchronous layerwise-parallel manner, unlocking the layers of such networks for parallel processing.Compared to the standard layerwise-sequential deep networks, these new layerwise-parallel networks show a fundamentally different temporal behavior and flow of information, especially for networks with skip or recurrent connections.We argue that layerwise-parallel deep networks are better suited for future challenges of deep neural network design, such as large functional modularized and/or recurrent architectures as well as networks allocating different network capacities dependent on current stimulus and/or task complexity.We layout basic properties and discuss major challenges for layerwise-parallel networks.Additionally, we provide a toolbox to design, train, evaluate, and online-interact with layerwise-parallel networks.

我们定义了一个分层模型并行的深度神经网络的概念，对于这些网络来说，各层是平行运行的，并提供了一个工具箱来设计、训练、评估和与这些网络在线互动。

一个用Theano编写的用于并行神经元更新的GPU加速工具箱，支持循环网络和具有跳层连接的网络中的不同更新顺序。

深度神经网络学习和评估的新工具箱，以及从层序网络到层序并行网络的规范转换建议。


Deep neural networks are known to be vulnerable to adversarial perturbations.In this paper, we bridge adversarial robustness of neural nets with Lyapunov stability of dynamical systems.From this viewpoint, training neural nets is equivalent to finding an optimal control of the discrete dynamical system, which allows one to utilize methods of successive approximations, an optimal control algorithm based on Pontryagin's maximum principle, to train neural nets.This decoupled training method allows us to add constraints to the optimization, which makes the deep model more robust.The constrained optimization problem can be formulated as a semi-definite programming problem and hence can be solved efficiently.Experiments show that our method effectively improves deep model's adversarial robustness.

衔接深度神经网络稳定性与Lyapunov稳定性的对抗性防御方法

作者将训练NN表述为寻找离散动态系统的最优控制器，使他们能够使用连续逼近法来训练NN，以使其对对抗性攻击更加强大。

本文使用神经网络作为离散化的ODE的理论观点，发展了一个旨在训练网络同时执行稳定性的稳定控制理论。


In this paper, we propose a method named Dimensional reweighting Graph Convolutional Networks (DrGCNs), to tackle the problem of variance between dimensional information in the node representations of GCNs.We prove that DrGCNs can reduce the variance of the node representations by connecting our problem to the theory of the mean field.However, practically, we find that the degrees DrGCNs help vary severely on different datasets.We revisit the problem and develop a new measure K to quantify the effect.This measure guides when we should use dimensional reweighting in GCNs and how much it can help.Moreover, it offers insights to explain the improvement obtained by the proposed DrGCNs.The dimensional reweighting block is light-weighted and highly flexible to be built on most of the GCN variants.Carefully designed experiments, including several fixes on duplicates, information leaks, and wrong labels of the well-known node classification benchmark datasets, demonstrate the superior performances of DrGCNs over the existing state-of-the-art approaches.Significant improvements can also be observed on a large scale industrial dataset.

我们提出了一个简单而有效的GCN的重新加权方案，在理论上得到了平均场理论的支持。

一种被称为DrGCN的方法，通过减少维度之间的变异，对图卷积网络中节点表示的不同维度进行重新加权。


Knowledge-grounded dialogue is a task of generating an informative response based on both discourse context and external knowledge.As we focus on better modeling the knowledge selection in the multi-turn knowledge-grounded dialogue, we propose a sequential latent variable model as the first approach to this matter.The model named sequential knowledge transformer (SKT) can keep track of the prior and posterior distribution over knowledge; as a result, it can not only reduce the ambiguity caused from the diversity in knowledge selection of conversation but also better leverage the response information for proper choice of knowledge.Our experimental results show that the proposed model improves the knowledge selection accuracy and subsequently the performance of utterance generation.We achieve the new state-of-the-art performance on Wizard of Wikipedia (Dinan et al., 2019) as one of the most large-scale and challenging benchmarks.We further validate the effectiveness of our model over existing conversation methods in another knowledge-based dialogue Holl-E dataset (Moghe et al., 2018).

我们的方法是首次尝试利用顺序潜变量模型在多轮知识基础的对话中进行知识选择。它在Wizard of Wikipedia基准上实现了新的最先进的性能。

一种用于对话生成中的知识选择的顺序潜变量模型，它将后验注意模型扩展到潜知识选择问题上，并取得了比以前最先进的模型更好的性能。

一种选择以知识为基础的多轮对话的新型架构，在相关基准数据集上产生了最先进的技术，并在人类评价中获得了更高的分数。


Meta-learning, or learning-to-learn, has proven to be a successful strategy in attacking problems in supervised learning and reinforcement learning that involve small amounts of data.State-of-the-art solutions involve learning an initialization and/or learning algorithm using a set of training episodes so that the meta learner can generalize to an evaluation episode quickly.These methods perform well but often lack good quantification of uncertainty, which can be vital to real-world applications when data is lacking.We propose a meta-learning method which efficiently amortizes hierarchical variational inference across tasks, learning a prior distribution over neural network weights so that a few steps of Bayes by Backprop will produce a good task-specific approximate posterior.We show that our method produces good uncertainty estimates on contextual bandit and few-shot learning benchmarks.

我们提出了一种元学习方法，该方法在不同的训练事件中有效地摊销了层次化的变异推理。

对MAML型模型的改编，通过在MAML的分层贝叶斯观点中采用任务特定参数的变异推理，说明了任务特定潜变量的后验不确定性。

作者考虑通过元学习来学习神经网络权重的先验，通过摊销的变异推理来完成。


Often we wish to transfer representational knowledge from one neural network to another.Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator.Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network.We demonstrate that this objective ignores important structural knowledge of the teacher network.This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data.We formulate this objective as contrastive learning.Experiments demonstrate that our resulting new objective outperforms knowledge distillation on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer.When combined with knowledge distillation, our method sets a state of the art in many transfer tasks, sometimes even outperforming the teacher network.

通过最大限度地增加教师和学生之间的相互信息来进行表述/知识蒸馏

本文结合了衡量教师和学生网络所学到的表征之间的相互信息的对比性目标来进行模型蒸馏，并提出了一个在蒸馏任务上比现有替代方案有改进的模型。


Developing effective biologically plausible learning rules for deep neural networks is important for advancing connections between deep learning and neuroscience.To date, local synaptic learning rules like those employed by the brain have failed to match the performance of backpropagation in deep networks.In this work, we employ meta-learning to discover networks that learn using feedback connections and local, biologically motivated learning rules.Importantly, the feedback connections are not tied to the feedforward weights, avoiding any biologically implausible weight transport.It can be shown mathematically that this approach has sufficient expressivity to approximate any online learning algorithm.Our experiments show that the meta-trained networks effectively use feedback connections to perform online credit assignment in multi-layer architectures.Moreover, we demonstrate empirically that this model outperforms a state-of-the-art gradient-based meta-learning algorithm for continual learning on regression and classification benchmarks.This approach represents a step toward biologically plausible learning mechanisms that can not only match gradient descent-based learning, but also overcome its limitations.

利用反馈连接和区域可塑性规则进行学习的网络，可以利用元学习进行优化。


In the visual system, neurons respond to a patch of the input known as their classical receptive field (RF), and can be modulated by stimuli in the surround.These interactions are often mediated by lateral connections, giving rise to extra-classical RFs.We use supervised learning via backpropagation to learn feedforward connections, combined with an unsupervised learning rule to learn lateral connections between units within a convolutional neural network.These connections allow each unit to integrate information from its surround, generating extra-classical receptive fields for the units in our new proposed model (CNNEx).We demonstrate that these connections make the network more robust and achieve better performance on noisy versions of the MNIST and CIFAR-10 datasets.Although the image statistics of MNIST and CIFAR-10 differ greatly, the same unsupervised learning rule generalized to both datasets.Our framework can potentially be applied to networks trained on other tasks, with the learned lateral connections aiding the computations implemented by feedforward connections when the input is unreliable.

以非监督方式学习的具有生物启发的横向连接的CNN对嘈杂的输入更加坚固。


Deep learning (DL) has in recent years been widely used in naturallanguage processing (NLP) applications due to its superiorperformance.However, while natural languages are rich ingrammatical structure, DL has not been able to explicitlyrepresent and enforce such structures.This paper proposes a newarchitecture to bridge this gap by exploiting tensor productrepresentations (TPR), a structured neural-symbolic frameworkdeveloped in cognitive science over the past 20 years, with theaim of integrating DL with explicit language structures and rules.We call it the Tensor Product Generation Network(TPGN), and apply it to image captioning.The keyideas of TPGN are:1) unsupervised learning ofrole-unbinding vectors of words via a TPR-based deep neuralnetwork, and2) integration of TPR with typical DL architecturesincluding Long Short-Term Memory (LSTM) models.The novelty of ourapproach lies in its ability to generate a sentence and extractpartial grammatical structure of the sentence by usingrole-unbinding vectors, which are obtained in an unsupervisedmanner.Experimental results demonstrate the effectiveness of theproposed approach.

本文旨在为基于深度学习的自然语言处理应用开发一种张量积表示方法。


It is well-known that  classifiers are vulnerable to adversarial perturbations.To defend against adversarial perturbations, various certified robustness results have been derived.However, existing certified robustnesses are limited to top-1 predictions.In many real-world applications, top-$k$ predictions are more relevant.In this work, we aim to derive certified robustness for top-$k$ predictions.In particular, our certified robustness is based on randomized smoothing, which turns any classifier to a new classifier via adding noise to an input example.We adopt randomized smoothing because it is scalable to large-scale neural networks and applicable to any classifier.We derive a tight robustness in $\ell_2$ norm for top-$k$ predictions  when using randomized smoothing with Gaussian noise.We find that generalizing the certified robustness  from top-1 to top-$k$ predictions faces significant technical challenges.We also empirically evaluate our method on CIFAR10 and ImageNet.For example, our method can obtain an ImageNet classifier with a certified top-5 accuracy of 62.8\% when the $\ell_2$-norms of the adversarial perturbations are less than 0.5 (=127/255).Our code is publicly available at: \url{https://github.com/jjy1994/Certify_Topk}.

我们研究了在高斯噪声下通过随机平滑对top-k预测的认证稳定性，并得出了L_2规范下的严格稳定性约束。

本文扩展了使用随机平滑法推导认证半径的工作，并展示了高斯扰动下的平滑分类器对前k个预测的认证半径。

本文在top-1预测的随机平滑技术的基础上，旨在提供top-k预测的认证。


Recent work has shown increased interest in using the Variational Autoencoder (VAE) framework to discover interpretable representations of data in an unsupervised way.These methods have focussed largely on modifying the variational cost function to achieve this goal.However, we show that methods like beta-VAE simplify the tendency of variational inference to underfit causing pathological over-pruning and over-orthogonalization of learned components.In this paper we take a complementary approach: to modify the probabilistic model to encourage structured latent variable representations to be discovered.Specifically, the standard VAE probabilistic model is unidentifiable: the likelihood of the parameters is invariant under rotations of the latent space.This means there is no pressure to identify each true factor of variation with a latent variable.We therefore employ a rich prior distribution, akin to the ICA model, that breaks the rotational symmetry.Extensive quantitative and qualitative experiments demonstrate that the proposed prior mitigates the trade-off introduced by modified cost functions like beta-VAE and TCVAE between reconstruction loss and disentanglement.The proposed prior allows to improve these approaches with respect to both disentanglement and reconstruction quality significantly over the state of the art.

我们提出了在VAEs中无监督地学习分解表征的结构化先验，大大减轻了分解和重建损失之间的权衡。

使用L^p嵌套分布系列作为VAE的代码向量的先验的一般框架，显示了更高的MIG。

作者指出了目前VAE方法中的问题，并对VAE、β-VAE和β-TCVAE的重建和正交之间的权衡提供了一个新的视角。


Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks.The explanations in the literature for the apparent effectiveness of shortcuts are varied and often contradictory.We hypothesize that shortcuts work primarily because they act as linear counterparts to nonlinear layers.We test this hypothesis by using several variations on the standard residual block, with different types of linear connections, to build small (100k--1.2M parameter) image classification networks.Our experiments show that other kinds of linear connections can be even more effective than the identity shortcuts.Our results also suggest that the best type of linear connection for a given application may depend on both network width and depth.

我们将残余区块泛化为串联区块，使用任意的线性图而不是捷径，并且比ResNets提高了性能。

本文对ResNet-like架构中的捷径连接进行了分析，并提出用一种被称为串联块的替代卷积来替代身份捷径。

本文研究了在ResNet中用可训练的卷积跳过连接取代身份跳过连接的效果，发现性能有所提高。


Adam-typed optimizers, as a class of adaptive moment estimation methods with the exponential moving average scheme, have been successfully used in many applications of deep learning.Such methods are appealing for capability on large-scale sparse datasets.On top of that, they are computationally efficient and insensitive to the hyper-parameter settings.In this paper, we present a new framework for adapting Adam-typed methods, namely AdamT.Instead of applying a simple exponential weighted average, AdamT also includes the trend information when updating the parameters with the adaptive step size and gradients.The newly added term is expected to efficiently capture the non-horizontal moving patterns on the cost surface, and thus converge more rapidly.We show empirically the importance of the trend component, where AdamT outperforms the conventional Adam method constantly in both convex and non-convex settings.

我们提出了一个适应Adam类型方法的新框架，即AdamT，在用自适应步长和梯度更新参数时包括趋势信息。

一种新型的亚当变体，使用霍尔特的线性方法来计算平滑的一阶和二阶动量，而不是使用指数加权平均。


As machine learning methods see greater adoption and implementation in high stakes applications suchas medical image diagnosis, the need for model interpretability and explanation has become morecritical.Classical approaches that assess feature importance (eg saliency maps) do not explain how and why a particular region of an image is relevant to the prediction.We proposea method that explains the outcome of a classification black-box by gradually exaggeratingthe semantic effect of a given class.Given a query input to a classifier, our method produces aprogressive set of plausible variations of that query, which gradually change the posterior probabilityfrom its original class to its negation.These counter-factually generated samples preserve featuresunrelated to the classification decision, such that a user can employ our method as a ``tuning knob'' to traverse a data manifold while crossing the decision boundary.  Our method is model agnostic and only requires the output value and gradient of the predictor with respect to its input.

一种解释分类器的方法，通过夸大或减弱分类器与目标标签相关联的语义特征，产生图像的视觉扰动。

一个模型，当给一个黑盒输入查询时，旨在通过提供合理的和渐进的查询变化来解释结果，这些变化可以导致输出的变化。

一种解释图像分类的黑盒输出的方法，该方法对逐渐扰动的输入查询产生逐渐扰动的输出。


We study the problem of explaining a rich class of behavioral properties of deep neural networks.Our influence-directed explanations approach this problem by peering inside the network to identify neurons with high influence on the property of interest using an axiomatically justified influence measure, and then providing an interpretation for the concepts these neurons represent.We evaluate our approach by training convolutional neural networks on Pubfig, ImageNet, and Diabetic Retinopathy datasets.  Our evaluation demonstrates that influence-directed explanations (1) localize features used by the network, (2) isolate features distinguishing related instances, (3) help extract the essence of what the network learned about the class, and (4) assist in debugging misclassifications.

我们提出了一种以影响为导向的方法来构建对深度卷积网络行为的解释，并展示了它如何被用来回答先前工作无法解决的一系列问题。

一种满足某些公理的衡量影响力的方法，以及一个可以用来识别什么输入部分对深度神经网络中的神经元输出最有影响力的影响力概念。

本文提出测量单个神经元对另一个神经元所代表的感兴趣的数量的影响。


Standard deep learning systems require thousands or millions of examples to learn a concept, and cannot integrate new concepts easily.By contrast, humans have an incredible ability to do one-shot or few-shot learning.For instance, from just hearing a word used in a sentence, humans can infer a great deal about it, by leveraging what the syntax and semantics of the surrounding words tells us.Here, we draw inspiration from this to highlight a simple technique by which deep recurrent networks can similarly exploit their prior knowledge to learn a useful representation for a new word from little data.This could make natural language processing systems much more flexible, by allowing them to learn continually from the new words they encounter.

我们强调了一种技术，通过这种技术，自然语言处理系统可以从上下文中学习一个新词，使其更加灵活。

一种利用先验知识的技术，以最少的数据学习新词的嵌入表示。


Recent research developing neural network architectures with external memory have often used the benchmark bAbI question and answering dataset which provides a challenging number of tasks requiring reasoning.Here we employed a classic associative inference task from the human neuroscience literature in order to more carefully probe the reasoning capacity of existing memory-augmented architectures.This task is thought to capture the essence of reasoning -- the appreciation of distant relationships among elements distributed across multiple facts or memories.Surprisingly, we found that current architectures struggle to reason over long distance associations.Similar results were obtained on a more complex task involving finding the shortest path between nodes in a path.We therefore developed a novel architecture, MEMO, endowed with the capacity to reason over longer distances.This was accomplished with the addition of two novel components.First, it introduces a separation between memories/facts stored in external memory and the items that comprise these facts in external memory.Second, it makes use of an adaptive retrieval mechanism, allowing a variable number of â€˜memory hopsâ€™ before the answer is produced.MEMO is capable of solving our novel reasoning tasks, as well as all 20 tasks in bAbI.

一个支持推理的内存架构。

本文提出了对End2End内存网络架构的改变，引入了一个新的配对关联推理任务，大多数现有模型都难以解决这个问题，并表明他们提出的架构能更好地解决这个任务。

一项来自认知心理学的新任务（成对联想推理），以及对一种新的记忆结构的建议，该结构的特点是允许在成对联想任务上有更好的表现。


Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency.They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures).Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results.In this work, we study how depthwise separable convolutions can be applied to neural machine translation.We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves better results.In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation.We also introduce a new super-separable convolution operation that further reduces the number of parameters and computational cost of the models.

深度可分离卷积改善神经机器翻译：可分离性越强越好。

本文提出在全卷积神经机器翻译模型中使用深度可分离卷积层，并引入了一个新的超可分离卷积层，进一步降低了计算成本。


Interpreting generative adversarial network (GAN) training as approximate divergence minimization has beentheoretically insightful, has spurred discussion, and has lead to theoretically and practically interestingextensions such as f-GANs and Wasserstein GANs.For both classic GANs and f-GANs, there is an original variant of training and a "non-saturating" variant which uses an alternative form of generator gradient.The original variant is theoretically easier to study, but for GANs the alternative variant performs better in practice.The non-saturating scheme is often regarded as a simple modification to deal with optimization issues, but we show that in fact the non-saturating scheme for GANs is effectively optimizing a reverse KL-like f-divergence.We also develop a number of theoretical tools to help compare and classify f-divergences.We hope these results may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.

非饱和的GAN训练有效地减少了类似KL的反向f-散度。

本文提出了f-散度类的有用表达方式，从新开发的工具中研究了流行的f-散度的理论属性，并研究了具有非饱和训练方案的GANs。


We introduce a novel method for converting text data into abstract image representations, which allows image-based processing techniques (e.g. image classification networks) to be applied to text-based comparison problems.We apply the technique to entity disambiguation of inventor names in US patents.The method involves converting text from each pairwise comparison between two inventor name records into a 2D RGB (stacked) image representation.We then train an image classification neural network to discriminate between such pairwise comparison images, and use the trained network to label each pair of records as either matched (same inventor) or non-matched (different inventors), obtaining highly accurate results (F1: 99.09%, precision: 99.41%, recall: 98.76%).Our new text-to-image representation method could potentially be used more broadly for other NLP comparison problems, such as disambiguation of academic publications, or for problems that require simultaneous classification of both text and images.

我们引入了一种新的文本表示方法，使图像分类器能够应用于文本分类问题，并将该方法应用于发明人姓名消歧。

一种将一对文本信息映射到二维RGB图像的方法，可将其输入二维卷积神经网络（图像分类器）。

作者考虑了专利名称发明人的名称消歧问题，并提议建立两个名称字符串的图像页面表示，以进行比较并应用图像分类器。


We propose a novel algorithm, Difference-Seeking Generative Adversarial Network (DSGAN), developed from traditional GAN.DSGAN considers the scenario that the training samples of target distribution, $p_{t}$, are difficult to collect.Suppose there are two distributions  $p_{\bar{d}}$ and $p_{d}$ such that the density of the target distribution can be the differences between the densities of $p_{\bar{d}}$ and $p_{d}$.We show how to learn the target distribution $p_{t}$ only via samples from $p_{d}$ and $p_{\bar{d}}$ (relatively easy to obtain).DSGAN has the flexibility to produce samples from various target distributions (e.g. the out-of-distribution).Two key applications, semi-supervised learning and adversarial training, are taken as examples to validate the effectiveness of DSGAN.We also provide theoretical analyses about the convergence of DSGAN.

我们提出了 "差集生成对抗网络"（DSGAN）模型来学习难以收集训练数据的目标分布。

本文介绍了DS-GAN，其目的是学习样本难以或无法收集的任何两个分布之间的差异，并展示了其在半监督学习和对抗性训练任务上的有效性。

本文考虑了学习GAN来捕捉目标分布的问题，而该分布中只有很少的训练样本可用。


Recently, Generative Adversarial Network (GAN) and numbers of its variants have been widely used to solve the image-to-image translation problem and achieved extraordinary results in both a supervised and unsupervised manner.However, most GAN-based methods suffer from the imbalance problem between the generator and discriminator in practice.Namely, the relative model capacities of the generator and discriminator do not match, leading to mode collapse and/or diminished gradients.To tackle this problem, we propose a GuideGAN based on attention mechanism.More specifically, we arm the discriminator with an attention mechanism so not only it estimates the probability that its input is real, but also does it create an attention map that highlights the critical features for such prediction.This attention map then assists the generator to produce more plausible and realistic images.We extensively evaluate the proposed GuideGAN framework on a  number of image transfer tasks.Both qualitative results and quantitative comparison demonstrate the superiority of our proposed approach.

一种通过使用注意力嵌入判别器提高GAN框架的图像翻译性能的通用方法

GAN框架中的一个反馈机制，在图像与图像之间的转换中提高了生成图像的质量，其判别器输出一张地图，指出生成器应该关注的地方，使其结果更有说服力。

提出了一个具有基于注意力的I2I翻译判别器的GAN，它提供了真/假的概率和反映图像生成的显着性的注意力图。


The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation.However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains unexplored.This paper specifically aims to study the fact verification given semi-structured data as evidence.To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED.TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning.To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA).Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification.LPA parses statements into LISP-like programs and executes them against the tables to obtain the returned binary value for verification.Both methods achieve similar accuracy but still lag far behind human performance.We also perform a comprehensive analysis to demonstrate great future opportunities.

我们提出一个新的数据集来研究半结构化表下的蕴含问题，作为前提条件

本文提出了一个新的数据集，用于基于表的事实验证，并介绍了任务的方法。

作者提出了用表格等半结构化数据源进行事实核查的问题，创建了一个新的数据集，并对有变化的基线模型进行了评估。


This work presents a two-stage neural architecture for learning and refining structural correspondences between graphs.First, we use localized node embeddings computed by a graph neural network to obtain an initial ranking of soft correspondences between nodes.Secondly, we employ synchronous message passing networks to iteratively re-rank the soft correspondences to reach a matching consensus in local neighborhoods between graphs.We show, theoretically and empirically, that our message passing scheme computes a well-founded measure of consensus for corresponding neighborhoods, which is then used to guide the iterative re-ranking process.Our purely local and sparsity-aware architecture scales well to large, real-world inputs while still being able to recover global correspondences consistently.We demonstrate the practical effectiveness of our method on real-world tasks from the fields of computer vision and entity alignment between knowledge graphs, on which we improve upon the current state-of-the-art.

我们开发了一个深度图谱匹配架构，它改进了最初的对应关系，以达成邻里共识。

一个回答图匹配问题的框架，包括本地节点嵌入和消息传递完善步骤。

一个基于GNN的两阶段架构，用于建立两个图之间的对应关系，在图像匹配和知识图实体联合的现实世界任务中表现良好。


This paper extends the proof of density of neural networks in the space of continuous (or even measurable) functions on Euclidean spaces to functions on compact sets of probability measures.By doing so the work parallels a more then a decade old results on mean-map embedding of probability measures in reproducing kernel Hilbert spaces.  The work has wide practical consequences for multi-instance learning, where it theoretically justifies some recently proposed constructions.The result is then extended to Cartesian products, yielding universal approximation theorem for tree-structured domains, which naturally occur in data-exchange formats like JSON, XML, YAML, AVRO, and ProtoBuffer.This has important practical implications, as it enables to automatically create an architecture of neural networks for processing structured data (AutoML paradigms), as demonstrated by an accompanied library for JSON format.

本文将欧几里得空间上的连续（甚至是可测）函数空间中的神经网络密度证明扩展到概率度量的紧凑集合上的函数。

本文研究了为解决多实例学习问题而设计的一系列神经网络的近似特性，并表明标准的单层结构的结果可以扩展到这些模型。

本文将通用近似定理归纳到测量空间上的实函数。


Interactions such as double negation in sentences and scene interactions in images are common forms of complex dependencies captured by state-of-the-art machine learning models.We propose MahÃ©, a novel approach to provide Model-Agnostic Hierarchical Explanations of how powerful machine learning models, such as deep neural networks, capture these interactions as either dependent on or free of the context of data instances.Specifically, MahÃ© provides context-dependent explanations by a novel local interpretation algorithm that effectively captures any-order interactions, and obtains context-free explanations through generalizing context-dependent interactions to explain global behaviors.Experimental results show that MahÃ© obtains improved local interaction interpretations over state-of-the-art methods and successfully provides explanations of interactions that are context-free.

预测的上下文依赖和无上下文解释的新框架

作者扩展了用于解释黑箱模型的线性区域归因方法LIME，并提出了一种辨别上下文依赖性和无上下文互动的方法。

一种能够为模型提供分层解释的方法，包括通过区域解释算法提供上下文相关的解释和无上下文解释。


To realize the promise of ubiquitous embedded deep network inference, it is essential to seek limits of energy and area efficiency.  To this end, low-precision networks offer tremendous promise because both energy and area scale down quadratically with the reduction in precision.  Here, for the first time, we demonstrate ResNet-18, ResNet-34, ResNet-50, ResNet-152, Inception-v3, densenet-161, and VGG-16bn networks on the ImageNet classification benchmark that, at 8-bit precision exceed the accuracy of the full-precision baseline networks after one epoch of finetuning, thereby leveraging the availability of pretrained models.We also demonstrate ResNet-18, ResNet-34, and ResNet-50 4-bit models that match the accuracy of the full-precision baseline networks -- the highest scores to date.Surprisingly, the weights of the low-precision networks are very close (in cosine similarity) to the weights of the corresponding baseline networks, making training from scratch unnecessary.We find that gradient noise due to quantization during training increases with reduced precision, and seek ways to overcome this noise.The number of iterations required by stochastic gradient descent to achieve a given training error is related to the square of (a) the distance of the initial solution from the final plus (b) the maximum variance of the gradient estimates.  By drawing inspiration from this observation, we (a) reduce solution distance by starting with pretrained fp32 precision baseline networks and fine-tuning, and (b) combat noise introduced by quantizing weights and activations during training, by using larger batches along with matched learning rate annealing.  Sensitivity analysis indicates that these techniques, coupled with proper activation function range calibration, offer a promising heuristic to discover low-precision networks, if they exist, close to fp32 precision baseline networks.

量化后的微调与8位元和4位元量化相匹配或超过全精度最先进的网络。

本文提出通过对预训练的模型做量化处理，使用大的批次规模，以及使用适当的学习率退火法，延长训练时间，来提高低精度模型的性能。

一种在高效硬件上进行推理的低位元量化方法，在ResNet50上用4位元权重和激活实现了完全的准确性，其依据是观察到低精度的微调会在梯度上引入噪音。


Analysis methods which enable us to better understand the  representations and functioning of neural models of language are  increasingly needed as deep learning becomes the dominant approach  in NLP.Here we present two methods based on Representational  Similarity Analysis (RSA) and Tree Kernels (TK) which allow us to  directly quantify how strongly the information encoded in neural  activation patterns corresponds to information represented by  symbolic structures such as syntax trees.We first validate our  methods on the case of a simple synthetic language for arithmetic  expressions with clearly defined syntax and semantics, and show that  they exhibit the expected pattern of results.We then apply our methods to  correlate neural representations of English sentences with their  constituency parse trees.

两种基于表征相似性分析（RSA）和树核（TK）的方法，直接量化了神经激活模式中编码的信息与符号结构所代表的信息之间的对应程度。


Supervised deep learning requires a large amount of training samples with annotations (e.g. label class for classification task, pixel- or voxel-wised label map for segmentation tasks), which are expensive and time-consuming to obtain.During the training of a deep neural network, the annotated samples are fed into the network in a mini-batch way, where they are often regarded of equal importance.However, some of the samples may become less informative during training, as the magnitude of the gradient start to vanish for these samples.In the meantime, other samples of higher utility or hardness may be more demanded for the training process to proceed and require more exploitation.To address the challenges of expensive annotations and loss of sample informativeness, here we propose a novel training framework which adaptively selects informative samples that are fed to the training process.The adaptive selection or sampling is performed based on a hardness-aware strategy in the latent space constructed by a generative model.To evaluate the proposed training framework, we perform experiments on three different datasets, including MNIST and CIFAR-10 for image classification task and a medical image dataset IVUS for biophysical simulation task.On all three datasets, the proposed framework outperforms a random sampling method, which demonstrates the effectiveness of our framework.

本文介绍了一个通过潜在空间的自适应抽样来进行数据高效表示学习的框架。

一种顺序和适应性地选择要提交给训练算法的训练实例的方法，其中选择发生在潜空间，基于选择损失梯度方向的样本。

一种在神经网络训练中有效选择硬样本的方法，通过将样本编码到潜空间的变异自动编码器来实现。


Existing methods for AI-generated artworks still struggle with generating high-quality stylized content, where high-level semantics are preserved, or separating fine-grained styles from various artists.We propose a novel Generative Adversarial Disentanglement Network which can disentangle two complementary factors of variations when only one of them is labelled in general, and fully decompose complex anime illustrations into style and content in particular.Training such model is challenging, since given a style, various content data may exist but not the other way round.Our approach is divided into two stages, one that encodes an input image into a style independent content, and one based on a dual-conditional generator.We demonstrate the ability to generate high-fidelity anime portraits with a fixed content and a large variety of styles from over a thousand artists, and vice versa, using a single end-to-end network and with applications in style transfer.We show this unique capability as well as superior output to the current state-of-the-art.

一种基于对抗性训练的方法，用于区分数据集中两组互补的变化，其中只有一个被标记，在动漫插图的风格与内容上进行测试。

一种结合条件GANs和条件VAEs的图像生成方法，可生成具有不同艺术家的各种风格的高保真动漫图像。

建议采用一种方法来学习动漫中分解的风格（艺术家）和内容表征。


Recent research has shown that CNNs are often overly sensitive to high-frequency textural patterns.Inspired by the intuition that humans are more sensitive to the lower-frequency (larger-scale) patterns we design a regularization scheme that penalizes large differences between adjacent components within each convolutional kernel.We apply our regularization onto several popular training methods, demonstrating that the models with the proposed smooth kernels enjoy improved adversarial robustness.Further, building on recent work establishing connections between adversarial robustness and interpretability, we show that our method appears to give more perceptually-aligned gradients.

我们为CNN的卷积核引入了一种平滑度正则化，它可以帮助提高对抗性的稳定性，并导致感知上的连结梯度。

本文提出了一种新的正则化方案，鼓励卷积核更加平滑，认为减少神经网络对高频成分的依赖有助于对对抗性实例的稳定性。

作者提出了一种学习更平滑的卷积核的方法，具体来说，就是用惩罚使用高频输入成分的直觉来惩罚核的连续像素之间的巨大变化的正则器。


Despite an ever growing literature on reinforcement learning algorithms and applications, much less is known about their statistical inference.In this paper, we investigate the large-sample behaviors of the Q-value estimates with closed-form characterizations of the asymptotic variances.This allows us to efficiently construct confidence regions for Q-value and optimal value functions, and to develop policies to minimize their estimation errors.This also leads to a policy exploration strategy that relies on estimating the relative discrepancies among the Q estimates.Numerical experiments show superior performances of our exploration strategy than other benchmark approaches.

我们研究了Q值估计的大样本行为，并提出了一个有效的探索策略，该策略依赖于对Q值估计之间的相对差异的估计。


Entailment vectors are a principled way to encode in a vector what information is known and what is unknown.  They are designed to model relations where one vector should include all the information in another vector, called entailment.  This paper investigates the unsupervised learning of entailment vectors for the semantics of words.  Using simple entailment-based models of the semantics of words in text (distributional semantics), we induce entailment-vector word embeddings which outperform the best previous results for predicting entailment between words, in unsupervised and semi-supervised experiments on hyponymy.

我们基于尾随性而不是相似性来训练词的嵌入，成功地预测了词汇的蕴含。

本文提出了一种词汇嵌入算法，用于词汇的蕴含，该算法遵循Henderson和Popa（ACL，2016）的工作。


We describe a simple scheme that allows an agent to learn about its environment in an unsupervised manner.Our scheme pits two versions of the same agent, Alice and Bob, against one another.Alice proposes a task for Bob to complete; and then Bob attempts to complete the task.  In this work we will focus on two kinds of environments: (nearly) reversible environments and environments that can be reset.Alice will "propose" the task by doing a sequence of actions and then Bob must undo or repeat them, respectively.  Via an appropriate reward structure, Alice and Bob automatically generate a curriculum of exploration, enabling unsupervised training of the agent.When Bob is deployed on an RL task within the environment, this unsupervised training reduces the number of supervised episodes needed to learn, and in some cases converges to a higher reward.

利用自我发挥的自动课程进行强化学习的无监督学习

一种以无监督的方式探索环境的新表述，以帮助以后的特定任务，其中一个代理提出越来越难的任务，学习的代理试图完成这些任务。

一个自我游戏模型，其中一个代理学习提出对他们来说容易但对对手来说困难的任务，创造一个自我游戏目标和学习课程的移动目标。


Many real-world data sets are represented as graphs, such as citation links, social media, and biological interaction.The volatile graph structure makes it non-trivial to employ convolutional neural networks (CNN's) for graph data processing.Recently, graph attention network (GAT) has proven a promising attempt by combining graph neural networks with attention mechanism, so as to achieve massage passing in graphs with arbitrary structures.However, the attention in GAT is computed mainly based on the similarity between the node content, while the structures of the graph remains largely unemployed (except in masking the attention out of one-hop neighbors).In this paper, we propose an `````````````````````````````"ADaptive Structural Fingerprint" (ADSF) model to fully exploit both topological details of the graph and  content features of the nodes.The key idea is to contextualize each node with a weighted, learnable receptive field  encoding rich and diverse local graph structures.By doing this, structural interactions between the nodes can  be inferred accurately, thus improving subsequent attention layer as well as the convergence of learning.Furthermore, our model provides a useful platform  for different subspaces of node features and various scales of graph structures to ``cross-talk'' with each other through the learning of multi-head attention, being particularly useful in handling complex real-world data.  Encouraging performance is observed on a number of benchmark data sets in node classification.

通过自适应的 "结构指纹 "开发图结构数据中丰富的结构细节

一种基于图结构的方法来增强图神经网络的注意机制，其主要想法是探索根节点的本地邻居的不同类型的节点之间的相互作用。

本文扩展了图NN中的自我注意的想法，它通常是基于节点之间的特征相似性，包括结构相似性。


Informed and robust decision making in the face of uncertainty is critical for robots that perform physical tasks alongside people.We formulate this as a Bayesian Reinforcement Learning problem over latent Markov Decision Processes (MDPs).While Bayes-optimality is theoretically the gold standard, existing algorithms do not scale well to continuous state and action spaces.We propose a scalable solution that builds on the following insight: in the absence of uncertainty, each latent MDP is easier to solve.We split the challenge into two simpler components.First, we obtain an ensemble of clairvoyant experts and fuse their advice to compute a baseline policy.Second, we train a Bayesian residual policy to improve upon the ensemble's recommendation and learn to reduce uncertainty.Our algorithm, Bayesian Residual Policy Optimization (BRPO), imports the scalability of policy gradient methods as well as the initialization from prior models.BRPO significantly improves the ensemble of experts and drastically outperforms existing adaptive RL methods.

我们提出了一种可扩展的贝叶斯强化学习算法，该算法在有洞察力专家的集合上学习贝叶斯修正，以解决具有复杂潜在奖励和动态的问题。

本文考虑了贝叶斯强化学习问题，在潜在的马尔科夫决策过程（MDPs）上，透过与专家进行决策。

在本文中，作者激励并提出了一种针对贝叶斯强化学习问题的学习算法，称为贝叶斯剩余策略优化（BRPO）。


One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth.This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural networks.For an $m$ hidden node shallow neural network with ReLU activation and $n$ training data, we show as long as $m$ is large enough and no two inputs are parallel, randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function.Our analysis relies on the following observation: over-parameterization and random initialization jointly restrict every weight vector to be close to its initialization for all iterations, which allows us to exploit a strong convexity-like property to show that gradient descent converges at a global linear rate to the global optimum.We believe these insights are also useful in analyzing deep models and other first order methods.

我们证明梯度下降在过度参数化的神经网络上以线性速率实现了零训练损失。

这项工作考虑优化一个具有平方损失的两层超参数化ReLU网络，并给定一个具有任意标签的数据集。

本文研究了具有平方损失的一隐层神经网络，他们表明在过度参数化的设置中，随机初始化和梯度下降得到的损失为零。


For many applications, in particular in natural science, the task is todetermine hidden system parameters from a set of measurements.Often,the forward process from parameter- to measurement-space is well-defined,whereas the inverse problem is ambiguous: multiple parameter sets canresult in the same measurement.To fully characterize this ambiguity, the fullposterior parameter distribution, conditioned on an observed measurement,has to be determined.We argue that a particular class of neural networksis well suited for this task â€“ so-called Invertible Neural Networks (INNs).Unlike classical neural networks, which attempt to solve the ambiguousinverse problem directly, INNs focus on learning the forward process, usingadditional latent output variables to capture the information otherwiselost.Due to invertibility, a model of the corresponding inverse process islearned implicitly.Given a specific measurement and the distribution ofthe latent variables, the inverse pass of the INN provides the full posteriorover parameter space.We prove theoretically and verify experimentally, onartificial data and real-world problems from medicine and astrophysics, thatINNs are a powerful analysis tool to find multi-modalities in parameter space,uncover parameter correlations, and identify unrecoverable parameters.

用可逆神经网络分析逆向问题

作者提出使用可逆网络来解决模糊的逆向问题，并建议不仅要训练前向模型，还要训练具有MMD批判的逆向模型。

该研究论文提出了一种可逆网络，其观测值为复杂输入分布的后验概率，具有理论上有效的双向训练方案。


Decisions made by machine learning systems have increasing influence on the world.Yet it is common for machine learning algorithms to assume that no such influence exists.An example is the use of the i.i.d.assumption in online learning for applications such as content recommendation, where the (choice of) content displayed can change users' perceptions and preferences, or even drive them away, causing a shift in the distribution of users.Generally speaking, it is possible for an algorithm to change the distribution of its own inputs.We introduce the term self-induced distributional shift (SIDS) to describe this phenomenon.A large body of work in reinforcement learning and causal machine learning aims to deal with distributional shift caused by deploying learning systems previously trained offline.Our goal is similar, but distinct: we point out that changes to the learning algorithm, such as the introduction of meta-learning, can reveal hidden incentives for distributional shift (HIDS), and aim to diagnose and prevent problems associated with hidden incentives.We design a simple Â environment as a "unit test" for HIDS, as well as a content recommendation environment which allows us to disentangle different types of SIDS.Â  We demonstrate the potential for HIDS to cause unexpected or undesirable behavior in these environments, and propose and test a mitigation strategy.Â

绩效指标是不完整的规范；目的并不总是证明方法的正确性。

作者展示了元学习如何揭示了分布式转变的隐藏动机，并提出了一种基于在不同环境中交换学习者以减少自我引入分布式转变的方法。

本文通过在元学习中使任务变得更容易，将学习者获胜的内在动机概括为更多类的问题。


In one-class-learning tasks, only the normal case can be modeled with data, whereas the variation of all possible anomalies is too large to be described sufficiently by samples.Thus, due to the lack of representative data, the wide-spread discriminative approaches cannot cover such learning tasks, and rather generative models, which attempt to learn the input density of the normal cases, are used.However, generative models suffer from a large input dimensionality (as in images) and are typically inefficient learners.We propose to learn the data distribution more efficiently with a multi-hypotheses autoencoder.Moreover, the model is criticized by a discriminator, which prevents artificial data modes not supported by data, and which enforces diversity across hypotheses.This consistency-based anomaly detection (ConAD) framework allows the reliable identification of outof- distribution samples.For anomaly detection on CIFAR-10, it yields up to 3.9% points improvement over previously reported results.On a real anomaly detection task, the approach reduces the error of the baseline models from 6.8% to 1.5%.

我们提出了一种异常检测方法，该方法将通过多个区域密度对前景类进行建模与对抗性训练相结合。

本文提出了一种技术，通过使生成模型与区域密度相一致而使其更加稳定。


Generative Adversarial Networks (GAN) can achieve promising performance on learning complex data distributions on different types of data.In this paper, we first show that a straightforward extension of an existing GAN algorithm is not applicable to point clouds, because the constraint required for discriminators is undefined for set data.We propose a two fold modification to a GAN algorithm to be able to generate point clouds (PC-GAN).First, we combine ideas from hierarchical Bayesian modeling and implicit generative models by learning a hierarchical and interpretable sampling process.A key component of our method is that we train a posterior inference network for the hidden variables.Second, PC-GAN defines a generic framework that can incorporate many existing GAN algorithms.We further propose a sandwiching objective, which results in a tighter Wasserstein distance estimate than the commonly used dual form in WGAN.We validate our claims on the ModelNet40 benchmark dataset and observe that PC- GAN trained by the sandwiching objective achieves better results on test data than existing methods.We also conduct studies on several tasks, including generalization on unseen point clouds, latent space interpolation, classification, and image to point clouds transformation, to demonstrate the versatility of the proposed PC-GAN algorithm.

我们提出了一个GAN变体，它可以学习生成点云。不同的研究已经被探索，包括更紧密的Wasserstein距离估计、条件生成、对未见过的点云的泛化以及图像到点云。

本文提出使用GAN来生成3D点云，并引入了一个夹层目标，平均分配之间的Wasserstein距离的上下限。

本文提出了一个新的无序数据的生成模型，特别适用于点云，其中包括一个推理方法和一个新的目标函数。


Existing attention mechanisms, are mostly item-based in that a model is trained to attend to individual items in a collection (the memory) where each item has a predefined, fixed granularity, e.g., a character or a word.Intuitively, an area in the memory consisting of multiple items can be worth attending to as a whole.We propose area attention: a way to attend to an area of the memory, where each area contains a group of items that are either spatially adjacent when the memory has a 2-dimensional structure, such as images, or temporally adjacent for 1-dimensional memory, such as natural language sentences.Importantly, the size of an area, i.e., the number of items in an area or the level of aggregation, is dynamically determined via learning, which can vary depending on the learned coherence of the adjacent items.By giving the model the option to attend to an area of items, instead of only individual items, a model can attend to information with varying granularity.Area attention can work along multi-head attention for attending to multiple areas in the memory.We evaluate area attention on two tasks: neural machine translation (both character and token-level) and image captioning, and improve upon strong (state-of-the-art) baselines in all the cases.These improvements are obtainable with a basic form of area attention that is parameter free.In addition to proposing the novel concept of area attention, we contribute an efficient way for computing it by leveraging the technique of summed area tables.

本文提出了一种新的注意力机制的方法，可以使一系列的任务受益，如机器翻译和图像标题。

本文将目前的注意力模型从单词层面扩展到相邻单词的组合，将模型应用于由相邻单词合并而成的项目。


We identify a phenomenon, which we refer to as *multi-model forgetting*, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performance of previously-trained models degrades as one optimizes a subsequent one, due to the overwriting of shared parameters.To overcome this, we introduce a statistically-justified weight plasticity loss that regularizes the learning of a model's shared parameters according to their importance for the previous models, and demonstrate its effectiveness when training two models sequentially and for neural architecture search.Adding weight plasticity in neural architecture search preserves the best models to the end of the search and yields improved results in both natural language processing and computer vision tasks.

我们确定了一种现象，即神经洗脑，并引入了统计学上合理的重量可塑性损失来克服这种现象。

本文讨论了 "神经洗脑 "的现象，它指的是一个模型的性能通过另一个模型共享模型参数而受到影响。


Revealing latent structure in data is an active field of research, having introduced exciting technologies such as variational autoencoders and adversarial networks, and is essential to push machine learning towards unsupervised knowledge discovery.However, a major challenge is the lack of suitable benchmarks for an objective and quantitative evaluation of learned representations.To address this issue we introduce Morpho-MNIST, a framework that aims to answer: "to what extent has my model learned to represent specific factors of variation in the data?"We extend the popular MNIST dataset by adding a morphometric analysis enabling quantitative comparison of trained models, identification of the roles of latent variables, and characterisation of sample diversity.We further propose a set of quantifiable perturbations to assess the performance of unsupervised and supervised methods on challenging tasks such as outlier detection and domain adaptation.

本文介绍了Morpho-MNIST，这是一个形状度量和扰动的集合，是对表征学习进行定量评估的一个步骤。

本文讨论了评估和诊断使用生成模型学习的代表的问题。

作者提出了一套对MNISt数据集进行分类的标准和一套有趣的扰动来修改MNIST数据集。


Exploration in environments with sparse rewards is a key challenge for reinforcement learning.How do we design agents with generic inductive biases so that they can explore in a consistent manner instead of just using local exploration schemes like epsilon-greedy?We propose an unsupervised reinforcement learning agent which learns a discrete pixel grouping model that preserves spatial geometry of the sensors and implicitly of the environment as well.We use this representation to derive geometric intrinsic reward functions, like centroid coordinates and area, and learn policies to control each one of them with off-policy learning.These policies form a basis set of behaviors (options) which allows us explore in a consistent way and use them in a hierarchical reinforcement learning setup to solve for extrinsically defined rewards.We show that our approach can scale to a variety of domains with competitive performance, including navigation in 3D environments and Atari games with sparse rewards.

通过无监督的视觉抽象发现和控制进行深度强化学习的结构化探索

本文介绍了用于强化学习的视觉抽象，其中一个算法学习 "控制 "每个抽象，以及选择选项以实现整体任务。


Combinatorial optimization is a common theme in computer science.While in general such problems are NP-Hard, from a practical point of view, locally optimal solutions can be useful.In some combinatorial problems however, it can be hard to define meaningful solution neighborhoods that connect large portions of the search space, thus hindering methods that search this space directly.We suggest to circumvent such cases by utilizing a policy gradient algorithm that transforms the problem to the continuous domain, and to optimize a new surrogate objective that renders the former as generic stochastic optimizer.This is achieved by producing a surrogate objective whose distribution is fixed and predetermined, thus removing the need to fine-tune various hyper-parameters in a case by case manner.Since we are interested in methods which can successfully recover locally optimal solutions, we use the problem of finding locally maximal cliques as a challenging experimental benchmark, and we report results on a large dataset of graphs that is designed to test clique finding algorithms.Notably, we show in this benchmark that fixing the distribution of the surrogate is key to consistently recovering locally optimal solutions, and that our surrogate objective leads to an algorithm that outperforms other methods we have tested in a number of measures.

一种新的策略梯度算法，旨在处理黑盒子组合优化问题。该算法仅依赖于函数评估，并以高概率返回区域最优解。

本文提出了一种构建代理目标的方法，用于将政策梯度方法应用于组合优化，目的是减少对超参数调整的需求。

本文提出用政策梯度算法中的奖励项替换其中心的经验累积分布。


Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust and efficient measures of uncertainty are crucial.While it is possible to train regression networks to output the parameters of a probability distribution by maximizing a Gaussian likelihood function, the resulting model remains oblivious to the underlying confidence of its predictions.In this paper, we propose a novel method for training deterministic NNs to not only estimate the desired target but also the associated evidence in support of that target.We accomplish this by  placing evidential priors over our original Gaussian likelihood function and training our NN to infer the hyperparameters of our evidential distribution.We impose priors during training such that the model is penalized when its predicted evidence is not aligned with the correct output.Thus the model estimates not only the probabilistic mean and variance of our target but also the underlying uncertainty associated with each of those parameters.We observe that our evidential regression method learns well-calibrated measures of uncertainty on various benchmarks, scales to complex computer vision tasks, and is robust to adversarial input perturbations.

无需取样的神经网络的快速、校准的不确定性估计

本文提出了一种新的方法来估计回归环境中预测的信赖度，为具有完全整合不确定性估计的在线应用打开了大门。

本文提出了深度证据回归，这是一种训练神经网络的方法，不仅要估计输出，还要估计支持该输出的相关证据。


The Lottery Ticket Hypothesis from Frankle & Carbin (2019) conjectures that, for typically-sized neural networks, it is possible to find small sub-networks which train faster and yield superior performance than their original counterparts.The proposed algorithm to search for such sub-networks (winning tickets), Iterative Magnitude Pruning (IMP), consistently finds sub-networks with 90-95% less parameters which indeed train faster and better than the overparameterized models they were extracted from, creating potential applications to problems such as transfer learning.In this paper, we propose a new algorithm to search for winning tickets, Continuous Sparsification, which continuously removes parameters from a network during training, and learns the sub-network's structure with gradient-based methods instead of relying on pruning strategies.We show empirically that our method is capable of finding tickets that outperforms the ones learned by Iterative Magnitude Pruning, and at the same time providing up to 5 times faster search, when measured in number of training epochs.

我们提出了一种新的算法，可以快速找到神经网络中的中奖票。

本文提出了一个新的目标函数，可用于联合优化分类目标，同时鼓励网络中的稀疏化，表现出高精确度。

这项工作提出了一种新的迭代修剪方法，即连续稀疏化（Continuous Sparsification），它不断修剪当前权重，直到达到目标比率。


In most practical settings and theoretical analyses, one assumes that a model can be trained until convergence.However, the growing complexity of machine learning datasets and models may violate such assumptions.Indeed, current approaches for hyper-parameter tuning and neural architecture search tend to be limited by practical resource constraints.Therefore, we introduce a formal setting for studying training under the non-asymptotic, resource-constrained regime, i.e., budgeted training.We analyze the following problem: "given a dataset, algorithm, and fixed resource budget, what is the best achievable performance?"We focus on the number of optimization iterations as the representative resource.Under such a setting, we show that it is critical to adjust the learning rate schedule according to the given budget.Among budget-aware learning schedules, we find simple linear decay to be both robust and high-performing.We support our claim through extensive experiments with state-of-the-art models on ImageNet (image classification), Kinetics (video classification), MS COCO (object detection and instance segmentation), and Cityscapes (semantic segmentation).We also analyze our results and find that the key to a good schedule is budgeted convergence, a phenomenon whereby the gradient vanishes at the end of each allowed budget.We also revisit existing approaches for fast convergence and show that budget-aware learning schedules readily outperform such approaches under (the practical but under-explored) budgeted training setting.

引入预算培训的正式设置，并提出一个预算意识的线性学习率时间表

这项工作在固定的时期数下，提出了一种调整神经网络训练的学习率的技术。

本文利用引入的BAS（Budget-Aware Schedule）概念，分析了当迭代次数有限时，应该使用哪种学习率计划。


We present a new approach for efficient exploration which leverages a low-dimensional encoding of the environment learned with a combination of model-based and model-free objectives.Our approach uses intrinsic rewards that are based on a weighted distance of nearest neighbors in the low dimensional representational space to gauge novelty.We then leverage these intrinsic rewards for sample-efficient exploration with planning routines in representational space.One key element of our approach is that we perform more gradient steps in-between every environment step in order to ensure the model accuracy.We test our approach on a number of maze tasks, as well as a control problem and show that our exploration approach is more sample-efficient compared to strong baselines.

我们使用内在的奖励进行探索，这种奖励是基于表征空间中最近的邻居的加权距离。

本文提出了一种在表格MDP中进行有效探索的方法，以及一个简单的控制环境，使用确定性编码器来学习环境动态的低维表示。

本文提出了一种使用基于模型和无模型的方法结合新的度量的RL代理的样本高效探索方法。


Neural networks are vulnerable to small adversarial perturbations.While existing literature largely focused on the vulnerability of learned models, we demonstrate an intriguing phenomenon that adversarial robustness, unlike clean accuracy, is sensitive to the input data distribution.Even a semantics-preserving transformations on the input data distribution can cause a significantly different robustness for the adversarially trained model that is both trained and evaluated on the new distribution.We show this by constructing semantically- identical variants for MNIST and CIFAR10 respectively, and show that standardly trained models achieve similar clean accuracies on them, but adversarially trained models achieve significantly different robustness accuracies.This counter-intuitive phenomenon indicates that input data distribution alone can affect the adversarial robustness of trained neural networks, not necessarily the tasks themselves.Lastly, we discuss the practical implications on evaluating adversarial robustness, and make initial attempts to understand this complex phenomenon.

PGD训练的模型的稳定性表现对图像数据集的语义保存转换很敏感，这意味着在实践中对稳定学习算法的评估很棘手。


Sample inefficiency is a long-lasting problem in reinforcement learning (RL).  The state-of-the-art uses action value function to derive policy while it usually involves an extensive search over the state-action space and unstable optimization.Towards the sample-efficient RL, we propose ranking policy gradient (RPG), a policy gradient method that learns the optimal rank of a set of discrete actions.  To accelerate the learning of policy gradient methods, we establish the equivalence between maximizing the lower bound of return and imitating a near-optimal policy without accessing any oracles.These results lead to a general off-policy learning framework, which preserves the optimality, reduces variance, and improves the sample-efficiency.We conduct extensive experiments showing that when consolidating with the off-policy learning framework, RPG substantially reduces the sample complexity, comparing to the state-of-the-art.

我们提出排名政策梯度，学习行动的最佳排名以实现收益最大化。我们提出了一个通用的非政策学习框架，该框架具有保存最优性、减少方差和样本效率的特性。

本文提出使用一种排名形式对策略进行重新参数化，将RL问题转化为监督学习问题。

本文从排名的角度对政策梯度方法提出了一个新的观点。


We introduce MultiGrain, a neural network architecture that generates compact image embedding vectors that solve multiple tasks of different granularity: class, instance, and copy recognition.MultiGrain is trained jointly for classification by optimizing the cross-entropy loss and for instance/copy recognition by optimizing a self-supervised ranking loss.The self-supervised loss only uses data augmentation and thus does not require additional labels.Remarkably, the unified embeddings are not only much more compact than using several specialized embeddings, but they also have the same or better accuracy.When fed to a linear classifier, MultiGrain using ResNet-50 achieves 79.4% top-1 accuracy on ImageNet, a +1.8% absolute improvement over the the current state-of-the-art AutoAugment method.The same embeddings perform on par with state-of-the-art instance retrieval with images of moderate resolution.An ablation study shows that our approach benefits from the self-supervision, the pooling method and the mini-batches with repeated augmentations of the same image.

将分类和图像检索结合在一个神经网络架构中，我们获得了对这两项任务的改进。 本文提出了一种用于图像分类和实例检索的统一嵌入，以提高这两项任务的性能。 本文提出要联合训练一个深度神经网，用于图像分类、实例和复制识别。


In this paper, we investigate mapping the hyponymy relation of wordnet to feature vectors.  We aim to model lexical knowledge in such a way that it can be used as  input in generic machine-learning models, such as phrase entailment  predictors.  We propose two models.The first one leverages an existing mapping of  words to feature vectors (fasttext), and attempts to classify  such vectors as within or outside of each class.The second model is fully supervised,  using solely wordnet as a ground truth.It maps each concept to an  interval or a disjunction thereof.  On the first model, we approach, but not quite attain state of the  art performance.The second model can achieve near-perfect accuracy.

我们调查了将词网的连词关系映射到特征向量的情况

本文研究词与词之间的连词如何能被映射到特征表征上。

本文探讨了词向量表征中的连词概念，并描述了一种将词网关系组织成树状结构来定义连词的方法。


Recurrent Neural Networks (RNNs) are powerful autoregressive sequence models for learning prevalent patterns in natural language.   Yet language generated by RNNs often shows several degenerate characteristics that are uncommon in human language; while fluent, RNN language production can be overly generic, repetitive, and even self-contradictory.  We postulate that the objective function optimized by RNN language models, which amounts to the overall perplexity of a text, is not expressive enough to capture the abstract qualities of good generation such as Griceâ€™s Maxims.In this paper, we introduce a general learning framework that can construct a decoding objective better suited for generation.Starting with a generatively trained RNN language model, our framework learns to construct a substantially stronger generator by combining several discriminatively trained models that can collectively address the limitations of RNN generation.  Human evaluation demonstrates that text generated by the resulting generator is preferred over  that  of  baselines  by  a  large  margin  and  significantly  enhances  the  overall coherence, style, and information content of the generated text.

我们通过鉴别性地训练评分函数，根据好的写作的各种品质对候选代进行排名，从而建立一个更强大的自然语言产生器。

本文提出将多个希望纠正序列解码中不一致的归纳性偏差汇集在一起，并提出对预先确定的各种子目标组合的参数进行优化。

本文将RNN语言模型与几个经过判别训练的模型结合起来，以改善语言的生成。

本文提出使用受Grice的交流格言启发的增强型目标来改进RNN语言模型的生成。


In recent years, the efficiency and even the feasibility of traditional load-balancing policies are challenged by the rapid growth of cloud infrastructure with increasing levels of server heterogeneity and increasing size of cloud services and applications.In such many software-load-balancers heterogeneous systems, traditional solutions, such as JSQ, incur an increasing communication overhead, whereas low-communication alternatives, such as JSQ(d) and the recently proposed JIQ scheme are either unstable or provide poor performance.We argue that a better low-communication load balancing scheme can be established by allowing each dispatcher to have a different view of the system and keep using JSQ, rather than greedily trying to avoid starvation on a per-decision basis. accordingly, we introduce the Loosely-Shortest -Queue family of load balancing algorithms.Roughly speaking, in Loosely-shortest -Queue, each dispatcher keeps a different approximation of the server queue lengths and routes jobs to the shortest among them.Communication is used only to update the approximations and make sure that they are not too far from the real queue lengths in expectation.We formally establish the strong stability of any Loosely-Shortest -Queue policy and provide an easy-to-verify sufficient condition for verifying that a policy is Loosely-Shortest -Queue.We further demonstrate that the Loosely-Shortest -Queue approach allows constructing throughput optimal policies with an arbitrarily low communication budget.Finally, using extensive simulations that consider homogeneous, heterogeneous and highly skewed heterogeneous systems in scenarios with a single dispatcher as well as with multiple dispatchers, we show that the examined Loosely-Shortest -Queue example policies are always stable as dictated by theory.Moreover, it exhibits an appealing performance and significantly outperforms well-known low-communication policies, such as JSQ(d) and JIQ, while using a similar communication budget.

异质服务器多分配器系统的可扩展和低通信负载平衡解决方案，具有强大的理论保证和有希望的经验结果。


We propose a novel quantitative measure to predict the performance of a deep neural network classifier, where the measure is derived exclusively from the graph structure of the network.We expect that this measure is a fundamental first step in developing a method to evaluate new network architectures and reduce the reliance on the computationally expensive trial and error or "brute force" optimisation processes involved in model selection.The measure is derived in the context of multi-layer perceptrons (MLPs), but the definitions are shown to be useful also in the context of deep convolutional neural networks (CNN), where it is able to estimate and compare the relative performance of different types of neural networks, such as VGG, ResNet, and DenseNet.Our measure is also used to study the effects of some important "hidden" hyper-parameters of the DenseNet architecture, such as number of layers, growth rate and the dimension of 1x1 convolutions in DenseNet-BC.Ultimately, our measure facilitates the optimisation of the DenseNet design, which shows improved results compared to the baseline.

预测深度神经网络模型性能的量化方法。

本文提出了一个新的数量，即计算神经网络中的路径数量，这对具有相同参数数量的神经网络的性能具有预测作用。

本文提出了一种计算深度神经网络中路径的方法，可以说是用来衡量网络的性能。


There is a stark disparity between the learning rate schedules used in the practice of large scale machine learning and what are considered admissible learning rate schedules prescribed in the theory of stochastic approximation.Recent results, such as in the 'super-convergence' methods which use oscillating learning rates, serve to emphasize this point even more.One plausible explanation is that non-convex neural network training procedures are better suited to the use of fundamentally different learning rate  schedules, such as the ``cut the learning rate every constant number of epochs'' method (which more closely resembles an exponentially decaying learning rate schedule); note that this widely used schedule is in stark contrast to the polynomial decay schemes prescribed in the stochastic approximation literature, which are indeed shown to be (worst case) optimal for classes of convex optimization problems.The main contribution of this work shows that the picture is far more nuanced, where we do not even need to move to non-convex optimization to show other learning rate schemes can be far more effective.In fact, even for the simple case of stochastic linear regression with a fixed time horizon, the rate achieved by any polynomial decay scheme is sub-optimal compared to the statistical minimax rate (by a factor of condition number); in contrast the ```''cut the learning rate every constant number of epochs'' provides an exponential improvement (depending only logarithmically on the condition number) compared to any polynomial decay scheme.  Finally, it is important to ask if our theoretical insights are somehow fundamentally tied to quadratic loss minimization (where we have circumvented minimax lower bounds for more general convex optimization problems)?Here, we conjecture that recent results which make the gradient norm small at a near optimal rate, for both convex and non-convex optimization, may also provide more insights into learning rate schedules used in practice.

本文提出了一个严格的研究，即为什么实际使用的学习率计划（对于给定的计算预算）具有显着的优势，尽管这些计划并不被随机逼近的经典理论所提倡。

本文对不同的学习率计划进行了理论研究，得出了多项式和常数和切割方案的统计学最低下限。

本文研究了学习率选择对随机优化的影响，重点是具有衰减步长的最小均方。


We present Value Propagation (VProp), a set of parameter-efficient differentiable planning modules built on Value Iteration which can successfully be trained using reinforcement learning to solve unseen tasks, has the capability to generalize to larger map sizes, and can learn to navigate in dynamic environments.We show that the modules enable learning to plan when the environment also includes stochastic elements, providing a cost-efficient learning system to build low-level size-invariant planners for a variety of interactive navigation problems.We evaluate on static and dynamic configurations of MazeBase grid-worlds, with randomly generated environments of several different sizes, and on a StarCraft navigation scenario, with more complex dynamics, and pixels as input.

我们提出了基于Convnets的规划器，这些规划器具有样本效率，并且可以推广到更大的导航和寻路问题的实例中。

提出了一些方法，这些方法可以看作是对价值迭代网络（VIN）的修改，并进行了一些改进，旨在提高样本效率和对大环境规模的泛化。

本文通过考虑与状态有关的转换函数，提出了对原有价值迭代网络（VIN）的扩展。


Learning high-quality word embeddings is of significant importance in achieving better performance in many down-stream learning tasks.On one hand, traditional word embeddings are trained on a large scale corpus for general-purpose tasks, which are often sub-optimal for many domain-specific tasks.On the other hand, many domain-specific tasks do not have a large enough domain corpus to obtain high-quality embeddings.We observe that domains are not isolated and a small domain corpus can leverage the learned knowledge from many past domains to augment that corpus in order to generate high-quality embeddings.In this paper, we formulate the learning of word embeddings as a lifelong learning process.Given knowledge learned from many previous domains and a small new domain corpus, the proposed method can effectively generate new domain embeddings by leveraging a simple but effective algorithm and a meta-learner, where the meta-learner is able to provide word context similarity information at the domain-level.Experimental results demonstrate that the proposed method can effectively learn new domain embeddings from a small corpus and past domain knowledges\footnote{We will release the code after final revisions.}.Wealso demonstrate that general-purpose embeddings trained from a large scale corpus are sub-optimal in domain-specific tasks.

通过终身学习和元学习来学习更好的领域嵌入

提出了一种学习词嵌入的终身学习方法。

本文提出了一种在新领域中学习嵌入的方法，并在一个方面的提取任务上大大超过了基线。


Parameter pruning is a promising approach for CNN compression and acceleration by eliminating redundant model parameters with tolerable performance loss.Despite its effectiveness, existing regularization-based parameter pruning methods usually drive weights towards zero with large and constant regularization factors, which neglects the fact that the expressiveness of CNNs is fragile and needs a more gentle way of regularization for the networks to adapt during pruning.To solve this problem, we propose a new regularization-based pruning method (named IncReg) to incrementally assign different regularization factors to different weight groups based on their relative importance, whose effectiveness is proved on popular CNNs compared with state-of-the-art methods.

我们提出了一种新的基于正则化的修剪方法（名为IncReg），根据不同权重组的相对重要性，逐步将不同的正则化因子分配给不同的权重组。


Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov's accelerated gradient descent (NAG) method are widely used in practice for training deep networks and other supervised learning models, as they often provide significant improvements over stochastic gradient descent (SGD).Rigorously speaking, fast gradient methods have provable improvements over gradient descent only for the deterministic case, where the gradients are exact.In the stochastic case, the popular explanations for their wide applicability is that when these fast gradient methods are applied in the stochastic case, they partially mimic their exact gradient counterparts, resulting in some practical gain.This work provides a counterpoint to this belief by proving that there exist simple problem instances where these methods cannot outperform SGD despite the best setting of its parameters.These negative problem instances are, in an informal sense, generic; they do not look like carefully constructed pathological instances.These results suggest (along with empirical evidence) that HB or NAG's practical performance gains are a by-product of minibatching.Furthermore, this work provides a viable (and provable) alternative, which, on the same set of problem instances, significantly improves over HB, NAG, and SGD's performance.This algorithm, referred to as Accelerated Stochastic Gradient Descent (ASGD), is a simple to implement stochastic algorithm, based on a relatively less popular variant of Nesterov's Acceleration.Extensive empirical results in this paper show that ASGD has performance gains over HB, NAG, and SGD.The code for implementing the ASGD Algorithm can be found at https://github.com/rahulkidambi/AccSGD.

现有的动量/加速方案，如重球法和Nesterov加速法与随机梯度一起使用，并没有比虚无缥缈的随机梯度下降法有所改善，尤其是在使用小批量的时候。


Oversubscription planning (OSP) is the problem of finding plans that maximize the utility value of their end state while staying within a specified cost bound.Recently, it has been shown that OSP problems can be reformulated as classical planning problems with multiple cost functions but no utilities.  Here we take advantage of this reformulation to show that OSP problems can be solved optimally using the A* search algorithm, in contrast to previous approaches that have used variations on branch-and-bound search.This allows many powerful techniques developed for classical planning to be applied to OSP problems.We also introduce novel bound-sensitive heuristics, which are able to reason about the primary cost of a solution while taking into account secondary cost functions and bounds, to provide superior guidance compared to heuristics that do not take these bounds into account.We implement two such bound-sensitive variants of existing classical planning heuristics, and show experimentally that the resulting search is significantly more informed than comparable heuristics that do not consider bounds.

我们表明，超额认购规划任务可以用A*来解决，并为超额认购规划任务引入了新的边界敏感启发式方法。

提出了一种通过使用多成本函数转换为经典规划的方法来优化解决超额订购规划（OSP）任务。

本文提出了对可接受的启发式方法的修改，使其在多标准的环境下更有依据，其中。


Previous work on adversarially robust neural networks requires large training sets and computationally expensive training procedures.  On the other hand, few-shot learning methods are highly vulnerable to adversarial examples.  The goal of our work is to produce networks which both perform well at few-shot tasks and are simultaneously robust to adversarial examples.  We adapt adversarial training for meta-learning, we adapt robust architectural features to small networks for meta-learning, we test pre-processing defenses as an alternative to adversarial training for meta-learning, and we investigate the advantages of robust meta-learning over robust transfer-learning for few-shot tasks.  This work provides a thorough analysis of adversarially robust methods in the context of meta-learning, and we lay the foundation for future work on defenses for few-shot tasks.

我们为对抗性稳定的少量学习开发了元学习方法。

本文提出了一种方法，通过在元学习算法的内部任务微调阶段引入对抗性查询数据攻击，增强了少量样本学习的稳定性。

本文作者提出了一种新的方法来训练一个健壮的低样本模型。


Many of our core assumptions about how neural networks operate remain empirically untested.One common assumption is that convolutional neural networks need to be stable to small translations and deformations to solve image recognition tasks.For many years, this stability was baked into CNN architectures by incorporating interleaved pooling layers.Recently, however, interleaved pooling has largely been abandoned.This raises a number of questions: Are our intuitions about deformation stability right at all?Is it important?Is pooling necessary for deformation invariance?If not, how is deformation invariance achieved in its absence?In this work, we rigorously test these questions, and find that deformation stability in convolutional networks is more nuanced than it first appears: (1) Deformation invariance is not a binary property, but rather that different tasks require different degrees of deformation stability at different layers.(2) Deformation stability is not a fixed property of a network and is heavily adjusted over the course of training, largely through the smoothness of the convolutional filters.(3) Interleaved pooling layers are neither necessary nor sufficient for achieving the optimal form of deformation stability for natural image classification.(4) Pooling confers \emph{too much} deformation stability for image classification at initialization, and during training, networks have to learn to \emph{counteract} this inductive bias.Together, these findings provide new insights into the role of interleaved pooling and deformation invariance in CNNs, and demonstrate the importance of rigorous empirical testing of even our most basic assumptions about the working of neural networks.

我们发现，仅靠池化并不能决定CNN的变形稳定性，而滤波器的平滑度在决定稳定性方面起着重要作用。