Incremental class learning involves sequentially learning classes in bursts of examples from the same class. This violates the assumptions that underlie  methods for training standard deep neural networks, and will cause them to suffer from catastrophic forgetting. Arguably, the best method for incremental class learning is iCaRL, but it requires storing  training examples for each class, making it challenging to scale. Here, we propose FearNet for incremental class learning. FearNet is a generative model that does not store previous examples, making it memory efficient. FearNet uses a brain-inspired dual-memory system in which new memories are consolidated from a network for recent memories inspired by the mammalian hippocampal complex to a network for long-term storage inspired by medial prefrontal cortex. Memory consolidation is inspired by mechanisms that occur during sleep. FearNet also uses a module inspired by the basolateral amygdala for determining which memory system to use for recall.   FearNet achieves state-of-the-art performance at incremental class learning on image (CIFAR-100, CUB-200) and audio classification (AudioSet) benchmarks.

FearNet是一个记忆效率高的神经网络，其灵感来自于哺乳动物大脑中的记忆形成，能够在没有灾难性遗忘的情况下进行增量类学习。

本文提出了一种基于双存储器系统的增量分类问题的新解决方案。


Multi-view learning can provide self-supervision when different views are available of the same data. Distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we present two multi-view frameworks for learning sentence representations in an unsupervised fashion. One framework uses a generative objective and the other a discriminative one. In both frameworks, the final representation is an ensemble of two views, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model. We show that, after learning, the vectors produced by our multi-view frameworks provide improved representations over their single-view learnt counterparts, and the combination of different views gives representational improvement over each view and demonstrates solid transferability on standard downstream tasks.

多视角学习改进了无监督的句子表征学习。

该方法使用不同的、互补的输入句子编码器和共识最大化。

本文提出了一个多视角框架，使用生成性和鉴别性的目标架构，来改善NLP任务中的句子表征。

本文表明，在学习句子表征方面，多视角框架比使用单个编码器更有效。


We show how discrete objects can be learnt in an unsupervised fashion from pixels, and how to perform reinforcement learning using this object representation.   More precisely, we construct a differentiable mapping from an image to a discrete tabular list of objects, where each object consists of a differentiable position, feature vector, and scalar presence value that allows the representation to be learnt using an attention mechanism.   Applying this mapping to Atari games, together with an interaction net-style architecture for calculating quantities from objects, we construct agents that can play Atari games using objects learnt in an unsupervised fashion. During training, many natural objects emerge, such as the ball and paddles in Pong, and the submarine and fish in Seaquest.   This gives the first reinforcement learning agent for Atari with an interpretable object representation, and opens the avenue for agents that can conduct object-based exploration and generalization.

我们展示了如何以无监督的方式从像素中学习离散的对象，以及如何使用这种对象表征进行强化学习。

一种从像素中学习对象表征以进行强化学习的方法。

本文提出了一个神经架构，利用无监督像素重建损失，将视频流映射到离散的对象集合，而无需人工注释。


Most recent gains in visual recognition have originated from the inclusion of attention mechanisms in deep convolutional networks (DCNs). Because these networks are optimized for object recognition, they learn where to attend using only a weak form of supervision derived from image class labels. Here, we demonstrate the benefit of using stronger supervisory signals by teaching DCNs to attend to image regions that humans deem important for object recognition. We first describe a large-scale online experiment (ClickMe) used to supplement ImageNet with nearly half a million human-derived "top-down" attention maps. Using human psychophysics, we confirm that the identified top-down features from ClickMe are more diagnostic than "bottom-up" saliency features for rapid image categorization. As a proof of concept, we extend a state-of-the-art attention network and demonstrate that adding ClickMe supervision significantly improves its accuracy and yields visual features that are more interpretable and more similar to those used by human observers.

用于训练用于对象识别的注意力模型的大规模数据集可以实现更准确、可解释和类似人类的对象识别。

论证了视觉识别的最新进展源于在深度卷积网络中使用视觉注意力机制，该机制通过基于图像类标签的弱监督形式来学习哪里是重点。

提出了一个关于注意力的新方法，其中收集了大型注意力数据集，并用于以监督的方式训练NN，以利用自我报告的人类注意力。

本文提出了一种新的方法，利用信息量更大的信号，特别是人类认为图像上重要的区域，来改进深度卷积神经网络。


In recent years, deep neural networks have demonstrated outstanding performancein many machine learning tasks. However, researchers have discovered that thesestate-of-the-art models are vulnerable to adversarial examples:  legitimate examples added by small perturbations which are unnoticeable to human eyes. Adversarial training, which augments the training data with adversarial examples duringthe training process,  is a well known defense to improve the robustness of themodel against adversarial attacks.   However, this robustness is only effective tothe same attack method used for adversarial training.   Madry et al. (2017) suggest that effectiveness of iterative multi-step adversarial attacks and particularlythat projected gradient descent (PGD) may be considered the universal first order adversary and applying the adversarial training with PGD implies resistanceagainst many other first order attacks.    However,  the computational cost of theadversarial training with PGD and other multi-step adversarial examples is muchhigher than that of the adversarial training with other simpler attack techniques. In this paper, we show how strong adversarial examples can be generated only ata cost similar to that of two runs of the fast gradient sign method (FGSM), allowing defense against adversarial attacks with a robustness level comparable to thatof the adversarial training with multi-step adversarial examples.   We empiricallydemonstrate the effectiveness of the proposed two-step defense approach againstdifferent attack methods and its improvements over existing defense strategies.

我们提出了一种针对一步式和迭代式对抗性攻击的省时防御方法。

提出一种名为e2SAD的新的、计算效率高的方法，该方法为每个干净的训练样本生成两套训练对抗样本。

本文介绍了一种两步对抗性防御方法，为每个干净的样本生成两个对抗性示例，并将其纳入实际的训练循环，以实现稳健性，并声称它可以胜过更昂贵的迭代方法。

本文提出了一种两步法，与最近的迭代式多步骤对抗性攻击相比，以较少的成本产生强大的对抗性示例。


Recently several different deep learning architectures have been proposed that take a string of characters as the raw input signal and automatically derive features for text classification. Little studies are available that compare the effectiveness of these approaches for character based text classification with each other. In this paper we perform such an empirical comparison for the important cybersecurity problem of DGA detection: classifying domain names as either benign vs. produced by malware (i.e., by a Domain Generation Algorithm). Training and evaluating on a dataset with 2M domain names shows that there is surprisingly little difference between various convolutional neural network (CNN) and recurrent neural network (RNN) based architectures in terms of accuracy, prompting a preference for the simpler architectures, since they are faster to train and less prone to overfitting.

对五种用于检测恶意域名的深度神经网络架构的比较，发现差异令人惊讶的小。

作者提出使用五种深度架构进行网络安全任务的领域生成算法检测。

应用多个NN架构，在良性和恶意软件相关的URL之间对URL进行分类。

本文提出通过使用经过训练的深度网络直接将字符序列分类为恶意或是良性，来自动识别域名是恶意的还是良性的。


Recognizing the relationship between two texts is an important aspect of natural language understanding (NLU), and a variety of neural network models have been proposed for solving NLU tasks. Unfortunately, recent work showed that the datasets these models are trained on often contain biases that allow models to achieve non-trivial performance without possibly learning the relationship between the two texts. We propose a framework for building robust models by using adversarial learning to encourage models to learn latent, bias-free representations. We test our approach in a Natural Language Inference (NLI) scenario, and show that our adversarially-trained models learn robust representations that ignore known dataset-specific biases. Our experiments demonstrate that our models are more robust to new NLI datasets.

对抗性学习方法鼓励NLI模型忽略数据集的特定偏差，并帮助模型跨数据集转移。 本文提出了一种对抗性设置，以减轻自然语言推理数据中的注释artifacts。 本文提出了一种通过对抗性训练目标来消除文本蕴含模型的偏差的方法。


We study the problem of learning representations of entities and relations in knowledge graphs for predicting missing links. The success of such a task heavily relies on the ability of modeling and inferring the patterns of (or between) the relations. In this paper, we present a new approach for knowledge graph embedding called RotatE, which is able to model and infer various relation patterns including: symmetry/antisymmetry, inversion, and composition. Specifically, the RotatE model defines each relation as a rotation from the source entity to the target entity in the complex vector space. In addition, we propose a novel self-adversarial negative sampling technique for efficiently and effectively training the RotatE model. Experimental results on multiple benchmark knowledge graphs show that the proposed RotatE model is not only scalable, but also able to infer and model various relation patterns and significantly outperform existing state-of-the-art models for link prediction.

一种最先进的知识图谱嵌入新方法。

提出了一个可以推断出知识库中关系的对称性、反对称性、反转性和组成模式的神经链接预测评分函数。

本文提出了一种通过将关系建模为复向量空间中的旋转来进行知识图谱嵌入的方法。

提出了一种用于链接预测的图谱嵌入方法。


Deep learning algorithms have been known to be vulnerable to adversarial perturbations in various tasks such as image classification. This problem was addressed by employing several defense methods for detection and rejection of particular types of attacks. However, training and manipulating networks according to particular defense schemes increases computational complexity of the learning algorithms. In this work, we propose a simple yet effective method to improve robustness of convolutional neural networks (CNNs) to adversarial attacks by using data dependent adaptive convolution kernels. To this end, we propose a new type of HyperNetwork in order to employ statistical properties of input data and features for computation of statistical adaptive maps. Then, we filter convolution weights of CNNs with the learned statistical maps to compute dynamic kernels. Thereby, weights and kernels are collectively optimized for learning of image classification models robust to  adversarial attacks without employment of additional target detection and rejection algorithms.  We empirically demonstrate that the proposed method enables CNNs to spontaneously defend against different types of attacks, e.g. attacks generated by Gaussian noise, fast gradient sign methods (Goodfellow et al., 2014) and a black-box attack (Narodytska & Kasiviswanathan, 2016).

我们使用超网络对CNN进行了修改，并观察到对对抗性例子有更好的稳健性。 通过使用依赖数据的卷积核提高深度卷积神经网络的稳健性和可靠性。


Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures.  Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent.  Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently.  In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning.  The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data.  This requires back-propagating errors through the solver steps.  While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage.  We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components.  Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.

我们提出了一种用于小样本分类的元学习方法，该方法通过快速求解器（例如岭回归或逻辑回归）反向传播，从而在高速下实现强大的性能。

本文提出了一种元学习的算法，相当于固定了特征（即深度NN的所有隐藏层），并将每个任务视为有自己的最终层，可以是岭回归或逻辑回归。

本文针对小样本分类问题提出了一种元学习方法，他们使用了一种基于闭式求解器对每个任务的学习者进行参数化的方法。


While many active learning papers assume that the learner can simply ask for a label and receive it, real annotation often presents a mismatch between the form of a label (say, one among many classes), and the form of an annotation (typically yes/no binary feedback). To annotate examples corpora for multiclass classification, we might need to ask multiple yes/no questions, exploiting a label hierarchy if one is available. To address this more realistic setting, we propose active learning with partial feedback (ALPF), where the learner must actively choose both which example to label and which binary question to ask. At each step, the learner selects an example, asking if it belongs to a chosen (possibly composite) class. Each answer eliminates some classes, leaving the learner with a partial label. The learner may then either ask more questions about the same example (until an exact label is uncovered) or move on immediately, leaving the first example partially labeled. Active learning with partial labels requires (i) a sampling strategy to choose (example, class) pairs, and (ii) learning from partial labels between rounds. Experiments on Tiny ImageNet demonstrate that our most effective method improves 26% (relative) in top-1 classification accuracy compared to i.i.d. baselines and standard active learners given 30% of the annotation budget that would be required (naively) to annotate the dataset. Moreover, ALPF-learners fully annotate TinyImageNet at 42% lower cost. Surprisingly, we observe that accounting for per-example annotation costs can alter the conventional wisdom that active learners should solicit labels for hard examples.

我们为在分层标签设置中从头开始训练机器学习模型提供了一个新的视角，即把它看作是人和算法之间的双向交流，并研究我们如何既能衡量和提高效率。

引入了一个新的主动学习设置，其中 oracle 提供部分或弱标签，而不是查询特定示例的标签，从而简化信息检索。

本文提出了一种带有部分反馈的主动学习方法，该方法在有限的预算下优于现有基线。

本文考虑了一个多类分类问题，其中标签被分组在给定数量的M个子集中，这些子集包含所有单个标签的单例。


Despite their prevalence, Euclidean embeddings of data are fundamentally limited in their ability to capture latent semantic structures, which need not conform to Euclidean spatial assumptions. Here we consider an alternative, which embeds data as discrete probability distributions in a Wasserstein space, endowed with an optimal transport metric. Wasserstein spaces are much larger and more flexible than Euclidean spaces, in that they can successfully embed a wider variety of metric structures. We propose to exploit this flexibility by learning an embedding that captures the semantic information in the Wasserstein distance between embedded distributions. We examine empirically the representational capacity of such learned Wasserstein embeddings, showing that they can embed a wide variety of complex metric structures with smaller distortion than an equivalent Euclidean embedding. We also investigate an application to word embedding, demonstrating a unique advantage of Wasserstein embeddings: we can directly visualize the high-dimensional embedding, as it is a probability distribution on a low-dimensional space. This obviates the need for dimensionality reduction techniques such as t-SNE for visualization.

我们表明，Wasserstein空间是嵌入具有复杂语义结构的数据的良好目标。

使用Wasserstein距离的最小化、正规化版本，在离散的概率分布空间中学习嵌入。

本文描述了一种新的嵌入方法，该方法将数据嵌入到具有Wasserstein距离的概率测量空间。

本文提出将数据嵌入到低维的Wasserstein空间，这样可以更准确地捕捉数据的基本结构。


Clustering high-dimensional datasets is hard because interpoint distances become less informative in high-dimensional spaces. We present a clustering algorithm that performs nonlinear dimensionality reduction and clustering jointly. The data is embedded into a lower-dimensional space by a deep autoencoder. The autoencoder is optimized as part of the clustering process. The resulting network produces clustered data. The presented approach does not rely on prior knowledge of the number of ground-truth clusters. Joint nonlinear dimensionality reduction and clustering are formulated as optimization of a global continuous objective. We thus avoid discrete reconfigurations of the objective that characterize prior clustering algorithms. Experiments on datasets from multiple domains demonstrate that the presented algorithm outperforms state-of-the-art clustering schemes, including recent methods that use deep networks.

通过优化全局连续目标，进行联合非线性降维和聚类的一种聚类算法。

提出了一种通过联合解决深度自动编码器和聚类作为全局连续目标的聚类算法，呈现出比最先进的聚类方案更好的结果。

深度连续聚类是一种聚类方法，它将自动编码器目标与聚类目标集成，然后使用 SGD 进行训练。


Deep convolutional neural networks (CNNs) are deployed in various applications but demand immense computational requirements. Pruning techniques and Winograd convolution are two typical methods to reduce the CNN computation. However, they cannot be directly combined because Winograd transformation fills in the sparsity resulting from pruning. Li et al. (2017) propose sparse Winograd convolution in which weights are directly pruned in the Winograd domain, but this technique is not very practical because Winograd-domain retraining requires low learning rates and hence significantly longer training time. Besides, Liu et al. (2018) move the ReLU function into the Winograd domain, which can help increase the weight sparsity but requires changes in the network structure. To achieve a high Winograd-domain weight sparsity without changing network structures, we propose a new pruning method, spatial-Winograd pruning. As the first step, spatial-domain weights are pruned in a structured way, which efficiently transfers the spatial-domain sparsity into the Winograd domain and avoids Winograd-domain retraining. For the next step, we also perform pruning and retraining directly in the Winograd domain but propose to use an importance factor matrix to adjust weight importance and weight gradients. This adjustment makes it possible to effectively retrain the pruned Winograd-domain network without changing the network structure. For the three models on the datasets of CIFAR-10, CIFAR-100, and ImageNet, our proposed method can achieve the Winograd-domain sparsities of 63%, 50%, and 74%, respectively.

为了加速卷积神经网络的计算，我们提出了一种新的两步修剪技术，在不改变网络结构的情况下实现了更高的Winograd域权重稀疏度。

提出了一个空间-Winograd修剪框架，允许在Winograd域中保留空间域的修剪权重，并改善Winograd域的稀疏性。

提出了两种使用Winograd算法的卷积层修剪技术。


In federated learning problems, data is scattered across different servers and exchanging or pooling it is often impractical or prohibited. We develop a Bayesian nonparametric framework for federated learning with neural networks. Each data server is assumed to train local neural network weights, which are modeled through our framework. We then develop an inference approach that allows us to synthesize a more expressive global network without additional supervision or data pooling. We then demonstrate the efficacy of our approach on federated learning problems simulated from two popular image classification datasets.

我们提出了一个用于神经网络联合学习的贝叶斯非参数模型。

使用β过程来做联合神经匹配。

本文考虑了神经网络的联合学习，其中数据分布在多台机器上，数据点的分配可能是不均匀和不平衡的。


We present a general-purpose method to train Markov chain Monte Carlo kernels, parameterized by deep neural networks, that converge and mix quickly to their target distribution. Our method generalizes Hamiltonian Monte Carlo and is trained to maximize expected squared jumped distance, a proxy for mixing speed. We demonstrate large empirical gains on a collection of simple but challenging distributions, for instance achieving a 106x improvement in effective sample size in one case, and mixing when standard HMC makes no measurable progress in a second. Finally, we show quantitative and qualitative gains on a real-world task: latent-variable generative modeling. Python source code will be open-sourced with the camera-ready paper.

训练具有深度神经网络参数的表达性MCMC核的一般方法。给定一个目标分布p，我们的方法提供一个快速混合采样器，能够有效地探索状态空间。

提出了一种一般的HMC，通过使用神经网络修改跃迁积分器，使采样器快速收敛和混合。


This paper addresses the problem of evaluating learning systems in safety critical domains such as autonomous driving, where failures can have catastrophic consequences. We focus on two problems: searching for scenarios when learned agents fail and assessing their probability of failure. The standard method for agent evaluation in reinforcement learning, Vanilla Monte Carlo, can miss failures entirely, leading to the deployment of unsafe agents. We demonstrate this is an issue for current agents, where even matching the compute used for training is sometimes insufficient for evaluation. To address this shortcoming, we draw upon the rare event probability estimation literature and propose an adversarial evaluation approach. Our approach focuses evaluation on adversarially chosen situations, while still providing unbiased estimates of failure probabilities. The key difficulty is in identifying these adversarial situations -- since failures are rare there is little signal to drive optimization. To solve this we propose a continuation approach that learns failure modes in related but less robust agents. Our approach also allows reuse of data already collected for training the agent. We demonstrate the efficacy of adversarial evaluation on two standard domains: humanoid control and simulated driving. Experimental results show that our methods can find catastrophic failures and estimate failures rates of agents multiple orders of magnitude faster than standard evaluation schemes, in minutes to hours rather than days.

我们表明，罕见但灾难性的故障可能完全被随机测试所遗漏，这给安全部署带来了问题。我们提出的对抗性测试方法可以解决这个问题。

提出了一种方法为学习型代理学习失败概率预测器，从而预测哪些初始状态会导致系统失败。

本文提出了一种基于神经网络对代理训练过程中发生的故障进行学习的函数，对RL算法的失败案例进行采样的重要性抽样方法。

本文提出了一种对抗性方法来识别强化学习中的灾难性失败案例。


The variational autoencoder (VAE) is a popular combination of deep latent variable model and accompanying variational learning technique. By using a neural inference network to approximate the model's posterior on latent variables, VAEs efficiently parameterize a lower bound on marginal data likelihood that can be optimized directly via gradient methods. In practice, however, VAE training often results in a degenerate local optimum known as "posterior collapse" where the model learns to ignore the latent variable and the approximate posterior mimics the prior. In this paper, we investigate posterior collapse from the perspective of training dynamics. We find that during the initial stages of training the inference network fails to approximate the model's true posterior, which is a moving target. As a result, the model is encouraged to ignore the latent encoding and posterior collapse occurs. Based on this observation, we propose an extremely simple modification to VAE training to reduce inference lag: depending on the model's current mutual information between latent variable and observation, we aggressively optimize the inference network before performing each model update. Despite introducing neither new model components nor significant complexity over basic VAE, our approach is able to avoid the problem of collapse that has plagued a large amount of previous work. Empirically, our approach outperforms strong autoregressive baselines on text and image benchmarks in terms of held-out likelihood, and is competitive with more complex techniques for avoiding collapse while being substantially faster.

为了解决VAE的后验塌陷问题，我们提出了一个新颖而简单的训练程序，通过更多的更新积极优化推理网络。这种新的训练程序减轻了后验塌陷，得到更好的VAE模型。

研究了后验塌陷的现象，表明增加推理网络的训练可以减少问题并得到更好的优化。

作者提出改变VAE的训练过程，仅作为后验塌陷的解决方案，而模型和目标保持不变。


Online healthcare services can provide the general public with ubiquitous access to medical knowledge and reduce the information access cost for both individuals and societies. To promote these benefits, it is desired to effectively expand the scale of high-quality yet novel relational medical entity pairs that embody rich medical knowledge in a structured form. To fulfill this goal, we introduce a generative model called Conditional Relationship Variational Autoencoder (CRVAE), which can discover meaningful and novel relational medical entity pairs without the requirement of additional external knowledge. Rather than discriminatively identifying the relationship between two given medical entities in a free-text corpus, we directly model and understand medical relationships from diversely expressed medical entity pairs. The proposed model introduces the generative modeling capacity of variational autoencoder to entity pairs, and has the ability to discover new relational medical entity pairs solely based on the existing entity pairs. Beside entity pairs, relationship-enhanced entity representations are obtained as another appealing benefit of the proposed method. Both quantitative and qualitative evaluations on real-world medical datasets demonstrate the effectiveness of the proposed method in generating relational medical entity pairs that are meaningful and novel.

通过纯粹地从现有的有意义的实体对当中学习，生成性地发现有意义的、具有一定医学关系的新实体对，而不需要额外的文本语料库来进行分辨性提取。

提出了一个变分自动编码器，用于在医疗环境中生成给定关系的实体对。

在医学方面，本文仅从结构化数据中描述了"知识库完成"的经典问题。


Although variational autoencoders (VAEs) represent a widely influential deep generative model, many aspects of the underlying energy function remain poorly understood.  In particular, it is commonly believed that Gaussian encoder/decoder assumptions reduce the effectiveness of VAEs in generating realistic samples .  In this regard, we rigorously analyze the VAE objective, differentiating situations where this belief is and is not actually true .  We then leverage the corresponding insights to develop a simple VAE enhancement that requires no additional hyperparameters or sensitive tuning .  Quantitatively, this proposal produces crisp samples and stable FID scores that are actually competitive with a variety of GAN models, all while retaining desirable attributes of the original VAE architecture . The code for our model is available at \url{https://github.com/daib13/TwoStageVAE}.

我们仔细分析了VAE目标函数，并得出了导致简单增强的新结论。

提出了一种两阶段的VAE方法，以产生高质量的样本并避免模糊。

本文提供了一些关于"vanilla"高斯变分自动编码器的理论结果，之后将其用于构建一种称为"2阶段VAE"的新算法。


We give a simple, fast algorithm for hyperparameter optimization inspired by techniques from the analysis of Boolean functions.   We focus on the high-dimensional regime where the canonical example is training a neural network with a large number of hyperparameters. The algorithm --- an iterative application of compressed sensing techniques for orthogonal polynomials --- requires only uniform sampling of the hyperparameters and is thus easily parallelizable.    Experiments for training deep neural networks on Cifar-10 show that compared to state-of-the-art tools (e.g., Hyperband and Spearmint), our algorithm finds significantly improved solutions, in some cases better than what is attainable by hand-tuning.   In terms of overall running time (i.e., time required to sample various settings of hyperparameters plus additional computation time), we are at least an order of magnitude faster than Hyperband and Bayesian Optimization.   We also outperform Random Search $8\times$.     Our method is inspired by provably-efficient algorithms for learning decision trees using the discrete Fourier transform.   We obtain improved sample-complexty bounds for learning decision trees while matching state-of-the-art bounds on running time (polynomial and quasipolynomial, respectively).

一种基于离散傅里叶分析和压缩感知的超参数调谐算法。

研究在未知函数可以被近似的假设下优化超参数的问题，表明近似的最小化可以在布尔超立方体上进行。

本文通过假设将超参数映射到分类精度的未知函数中的结构来探索超参数的优化。


Permutations and matchings are core building blocks in a variety of latent variable models, as they allow us to align, canonicalize, and sort data. Learning in such models is difficult, however, because exact marginalization over these combinatorial objects is intractable. In response, this paper introduces a collection of new methods for end-to-end learning in such models that approximate discrete maximum-weight matching using the continuous Sinkhorn operator.   Sinkhorn iteration is attractive because it functions as a simple, easy-to-implement analog of the softmax operator. With this, we can define the Gumbel-Sinkhorn method, an extension of the Gumbel-Softmax method (Jang et al. 2016, Maddison2016 et al. 2016) to distributions over latent matchings. We demonstrate the effectiveness of our method by outperforming competitive baselines on a range of qualitatively different tasks: sorting numbers, solving jigsaw puzzles, and identifying neural signals in worms.

一种排列梯度下降推理的新方法，应用于潜在匹配推理和神经网络排列的监督学习。

本文利用Sinkhorn算子的有限近似来描述如何构建神经网络，以便从排列值训练数据中学习。

本文提出了一种近似于离散最大权重的新方法，用于学习潜在的排列组合


Recent work in network quantization has substantially reduced the time and space complexity of neural network inference, enabling their deployment on embedded and mobile devices with limited computational and memory resources. However, existing quantization methods often represent all weights and activations with the same precision (bit-width). In this paper, we explore a new dimension of the design space: quantizing different layers with different bit-widths. We formulate this problem as a neural architecture search problem and propose a novel differentiable neural architecture search (DNAS) framework to efficiently explore its exponential search space with gradient-based optimization. Experiments show we surpass the state-of-the-art compression of ResNet on CIFAR-10 and ImageNet. Our quantized models with 21.1x smaller model size or 103.9x lower computational cost can still outperform baseline quantized or even full precision models.

一种用于 ConvNet 混合量化的新型可微分神经架构搜索框架。

作者介绍了一种新的神经结构搜索方法，该方法在每个神经网络层选择权重的精确量化，并将其用于网络压缩的背景下。

本文提出了一种网络量化的新方法，即用不同的位宽对不同的层进行量化，并介绍了一种新的可微分的神经结构搜索框架。


The top-$k$ error is a common measure of performance in machine learning and computer vision. In practice, top-$k$ classification is typically performed with deep neural networks trained with the cross-entropy loss. Theoretical results indeed suggest that cross-entropy is an optimal learning objective for such a task in the limit of infinite data. In the context of limited and noisy data however, the use of a loss function that is specifically designed for top-$k$ classification can bring significant improvements.  Our empirical evidence suggests that the loss function must be smooth and have non-sparse gradients in order to work well with deep neural networks. Consequently, we introduce a family of smoothed loss functions that are suited to top-$k$ optimization via deep learning. The widely used cross-entropy is a special case of our family. Evaluating our smooth loss functions is computationally challenging: a na{\"i}ve algorithm would require $\mathcal{O}(\binom{n}{k})$ operations, where $n$ is the number of classes. Thanks to a connection to polynomial algebra and a divide-and-conquer approach, we provide an algorithm with a time complexity of $\mathcal{O}(k n)$. Furthermore, we present a novel approximation to obtain fast and stable algorithms on GPUs with single floating point precision. We compare the performance of the cross-entropy loss and our margin-based losses in various regimes of noise and data size, for the predominant use case of $k=5$. Our investigation reveals that our loss is more robust to noise and overfitting than cross-entropy.

用于Top-k误差最小化的平滑损失函数

建议使用深度模型的top-k loss来解决训练数据集中存在或不存在的类似类别的类混淆问题。

平滑了top-k的损失。

本文为top-k SVM引入了一个平滑的代用损失函数，目的是将SVM插入深度神经网络。


Designing a molecule with desired properties is one of the biggest challenges in drug development, as it requires optimization of chemical compound structures with respect to many complex properties. To augment the compound design process we introduce Mol-CycleGAN -- a CycleGAN-based model that generates optimized compounds with a chemical scaffold of interest. Namely, given a molecule our model generates a structurally similar one with an optimized value of the considered property. We evaluate the performance of the model on selected optimization objectives related to structural properties (presence of halogen groups, number of aromatic rings) and to a physicochemical property (penalized logP). In the task of optimization of penalized logP of drug-like molecules our model significantly outperforms previous results.

我们介绍了Mol-CycleGAN--一种用于优化分子的新生成模型，以增强药物设计。

本文提出了一种基于CycleGANs应用于分子的变异自动编码器的优化分子特性的方法，并采用了一种领域特定的VAE，称为结点树VAE（JT-VAE）。

本文使用变分自动编码器来学习一个转换函数，从没有相关属性的分子集合到具有该属性的分子集合。


Knowledge distillation is a potential solution for model compression. The idea is to make a small student network imitate the target of a large teacher network, then the student network can be competitive to the teacher one. Most previous studies focus on model distillation in the classification task, where they propose different architectures and initializations for the student network. However, only the classification task is not enough, and other related tasks such as regression and retrieval are barely considered. To solve the problem, in this paper, we take face recognition as a breaking point and propose model distillation with knowledge transfer from face classification to alignment and verification. By selecting appropriate initializations and targets in the knowledge transfer, the distillation can be easier in non-classification tasks. Experiments on the CelebA and CASIA-WebFace datasets demonstrate that the student network can be competitive to the teacher one in alignment and verification, and even surpasses the teacher network under specific compression rates. In addition, to achieve stronger knowledge transfer, we also use a common initialization trick to improve the distillation performance of classification. Evaluations on the CASIA-Webface and large-scale MS-Celeb-1M datasets show the effectiveness of this simple trick.

我们以人脸识别为突破口，提出了从人脸分类到对齐和验证的知识转移的模型提炼。 本文提出将分类器从人脸分类的模型转移到对齐和验证的任务中。 该手稿介绍了如何将知识从面部分类模型提炼到学生模型以进行面部对齐和验证的实验。


RNNs have been shown to be excellent models for sequential data and in particular for session-based user behavior. The use of RNNs provides impressive performance benefits over classical methods in session-based recommendations. In this work we introduce a novel ranking loss function tailored for RNNs in recommendation settings. The better performance of such loss over alternatives, along with further tricks and improvements described in this work, allow to achieve an overall improvement of up to 35% in terms of MRR and Recall@20 over previous session-based RNN solutions and up to 51% over classical collaborative filtering approaches. Unlike data augmentation-based improvements, our method does not increase training times significantly.

使用新设计的损失函数和抽样，将基于会话的RNN（GRU4Rec）推荐提高了35%。

本文分析了现有的基于会话的推荐的损失函数，并提出了两个新的损失函数，在现有的基于排序的损失函数中增加了一个权重

在早期工作的基础上，通过对负面例子的 "相关性 "加权，提出了针对使用RNN的基于会话的推荐的修改。

本文讨论了优化GRU4Rec中损失函数的问题，提出了优化的技巧，并提出了一个加强版。


In representational lifelong learning an agent aims to continually learn to solve novel tasks while updating its representation in light of previous tasks. Under the assumption that future tasks are related to previous tasks, representations should be learned in such a way that they capture the common structure across learned tasks, while allowing the learner sufficient flexibility to adapt to novel aspects of a new task. We develop a framework for lifelong learning in deep neural networks that is based on generalization bounds, developed within the PAC-Bayes framework. Learning takes place through the construction of a distribution over networks based on the tasks seen so far, and its utilization for learning a new task. Thus, prior knowledge is incorporated through setting a history-dependent prior for novel tasks. We develop a gradient-based algorithm implementing these ideas, based on minimizing an objective function motivated by generalization bounds, and demonstrate its effectiveness through numerical examples.

我们开发了一种基于PAC-Bayes理论的终身学习方法，即在遇到新任务时调整预设值，从而促进对新任务的学习。

一种新的PAC-Bayesian风险约束，作为多任务机器学习的目标函数，以及一种使该目标函数的简化版本最小化的算法。

将现有的PAC-Bayes界线扩展到多任务学习中，以允许在不同的任务中调整预设值。


Optimization algorithms for training deep models not only affects the convergence rate and stability of the training process, but are also highly related to the generalization performance of trained models. While adaptive algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in many scenarios, they often lead to worse generalization performance than SGD, when used for training deep neural networks (DNNs). In this work, we identify two problems regarding the direction and step size for updating the weight vectors of hidden units, which may degrade the generalization performance of Adam. As a solution, we propose the normalized direction-preserving Adam (ND-Adam) algorithm, which controls the update direction and step size more precisely, and thus bridges the generalization gap between Adam and SGD. Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits. By bridging the gap between SGD and Adam, we also shed some light on why certain optimization algorithms generalize better than others.

用于训练DNN的Adam的定制版本，它弥补了Adam和SGD之间的泛化差距。

提出了一种ADAM优化算法的变体，该算法使用批量归一化对每个隐藏单元的权重进行归一化处理。

Adam 优化算法的扩展，通过使用梯度向量的 L2 范数将传入权重的学习速率调整为隐藏单位来保留更新方向


Options in reinforcement learning allow agents to hierarchically decompose a task into subtasks, having the potential to speed up learning and planning. However, autonomously learning effective sets of options is still a major challenge in the field. In this paper we focus on the recently introduced idea of using representation learning methods to guide the option discovery process. Specifically, we look at eigenoptions, options obtained from representations that encode diffusive information flow in the environment. We extend the existing algorithms for eigenoption discovery to settings with stochastic transitions and in which handcrafted features are not available.   We propose an algorithm that discovers eigenoptions while learning non-linear state representations from raw pixels. It exploits recent successes in the deep reinforcement learning literature and the equivalence between proto-value functions and the successor representation. We use traditional tabular domains to provide intuition about our approach and Atari 2600 games to demonstrate its potential.

我们展示了如何使用继任者表征来发现随机领域的特征选项，从原始像素开始。特征选择是为导航所学表征的潜在维度而学习的选项。

将特征选择的概念扩展到具有随机过渡的领域，并且在这些领域中，状态特征是可以学习的。

显示了原生价值函数和继任者表征之间的等价性，并推导出特征选项作为选项发现机制的概念。

该论文是Machado等人（2017）先前工作的后续工作，展示了如何使用原型值函数来定义称为"特征选项"的选项。


One form of characterizing the expressiveness of a piecewise linear neural network is by the number of linear regions, or pieces, of the function modeled. We have observed substantial progress in this topic through lower and upper bounds on the maximum number of linear regions and a counting procedure. However, these bounds only account for the dimensions of the network and the exact counting may take a prohibitive amount of time, therefore making it infeasible to benchmark the expressiveness of networks. In this work, we approximate the number of linear regions of specific rectifier networks with an algorithm for probabilistic lower bounds of mixed-integer linear sets. In addition, we present a tighter upper bound that leverages network coefficients. We test both on trained networks. The algorithm for probabilistic lower bounds is several orders of magnitude faster than exact counting and the values reach similar orders of magnitude, hence making our approach a viable method to compare the expressiveness of such networks. The refined upper bound is particularly stronger on networks with narrow layers.

我们为网络表达中使用的线性区域的数量提供了改进的上限，并提供了一种高效的（关于精确计数）算法以获得线性区域实际数量的概率下限。

通过使用近似的概率计数算法和分析，为研究RELU神经网络中的线性区域数量做出贡献

在之前研究深度神经网络中线性区域计数的工作基础上，通过改变维度约束来改进之前提出的上限

本文讨论了分段线性神经网络的表现力，其特征在于建模函数的线性区域的数量，并利用概率算法更快地计算边界，并证明更严格的边界。


The ability to look multiple times through a series of pose-adjusted glimpses is fundamental to human vision. This critical faculty allows us to understand highly complex visual scenes. Short term memory plays an integral role in aggregating the information obtained from these glimpses and informing our interpretation of the scene. Computational models have attempted to address glimpsing and visual attention but have failed to incorporate the notion of memory. We introduce a novel, biologically inspired visual working memory architecture that we term the Hebb-Rosenblatt memory. We subsequently introduce a fully differentiable Short Term Attentive Working Memory model (STAWM) which uses transformational attention to learn a memory over each image it sees. The state of our Hebb-Rosenblatt memory is embedded in STAWM as the weights space of a layer. By projecting different queries through this layer we can obtain goal-oriented latent representations for tasks including classification and visual reconstruction. Our model obtains highly competitive classification performance on MNIST and CIFAR-10. As demonstrated through the CelebA dataset, to perform reconstruction the model learns to make a sequence of updates to a canvas which constitute a parts-based representation. Classification with the self supervised representation obtained from MNIST is shown to be in line with the state of the art models (none of which use a visual attention mechanism). Finally, we show that STAWM can be trained under the dual constraints of classification and reconstruction to provide an interpretable visual sketchpad which helps open the `black-box' of deep learning.

一种受生物启发的工作记忆，可被整合到recurrent视觉注意力模型中，以获得最先进的性能

介绍了一个新的网络架构，其灵感来自于视觉上的工作记忆，并将其应用于分类任务，然后将其作为一个生成模型。

本文利用新的Hebb-Rosenblatt工作记忆模型增强了recurrent注意力模型，并在MNIST上取得了有竞争力的成果。


Generative models have been successfully applied to image style transfer and domain translation. However, there is still a wide gap in the quality of results when learning such tasks on musical audio. Furthermore, most translation models only enable one-to-one or one-to-many transfer by relying on separate encoders or decoders and complex, computationally-heavy models. In this paper, we introduce the Modulated Variational auto-Encoders (MoVE) to perform musical timbre transfer. First, we define timbre transfer as applying parts of the auditory properties of a musical instrument onto another. We show that we can achieve and improve this task by conditioning existing domain translation techniques with Feature-wise Linear Modulation (FiLM). Then, by replacing the usual adversarial translation criterion by a Maximum Mean Discrepancy (MMD) objective, we alleviate the need for an auxiliary pair of discriminative networks. This allows a faster and more stable training, along with a controllable latent space encoder. By further conditioning our system on several different instruments, we can generalize to many-to-many transfer within a single variational architecture able to perform multi-domain transfers. Our models map inputs to 3-dimensional representations, successfully translating timbre from one instrument to another and supporting sound synthesis on a reduced set of control parameters. We evaluate our method in reconstruction and generation tasks while analyzing the auditory descriptor distributions across transferred domains. We show that this architecture incorporates generative controls in multi-domain transfer, yet remaining rather light, fast to train and effective on small datasets.

本文使用变分自动编码和网络调节来实现音乐音色转换，我们开发并推广了我们的架构，用于多对多的乐器转换，并进行了可视化和评估。

提出了一种调制变分自动编码器，通过用最大平均差异代替通常的对抗性翻译标准来进行音乐音色的转换。

描述了一个多对多的音乐音色转换模型，该模型建立在领域和风格转换的最新发展之上。

提出了一个基于VAE的混合模型，在乐器的录音中进行音色转换。


We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena when the depth becomes large. This, in particular, provides quantitative answers and insights to three questions that were yet fully understood in the literature. Firstly, we provide a precise answer on how the random deep weight-tied autoencoder model performs “approximate inference” as posed by Scellier et al. (2018), and its connection to reversibility considered by several theoretical studies. Secondly, we show that deep autoencoders display a higher degree of sensitivity to perturbations in the parameters, distinct from the shallow counterparts. Thirdly, we obtain insights on pitfalls in training initialization practice, and demonstrate experimentally that it is possible to train a deep autoencoder, even with the tanh activation and a depth as large as 200 layers, without resorting to techniques such as layer-wise pre-training or batch normalization. Our analysis is not specific to any depths or any Lipschitz activations, and our analytical techniques may have broader applicability.

我们研究了在随机权重的假设下，权重绑定的多层vanilla自动编码器的行为。通过在大维极限下的精确描述，我们的分析揭示了有趣的相变现象。

通过平均场分析对编码器和解码器之间绑定权重（权重绑定）的自动编码器的理论分析

在高维统计问题分析的最新进展基础上，特别是消息传递算法，分析了加权绑定式自动编码器的性能

本文研究了几个假设下的自动编码器，并指出这种随机自动编码器的模型可以用一维方程进行优雅而严格的分析。


Assessing distance betweeen the true and the sample distribution is a key component of many state of the art generative models, such as Wasserstein Autoencoder (WAE). Inspired by prior work on Sliced-Wasserstein Autoencoders (SWAE) and  kernel smoothing we construct a new generative model – Cramer-Wold AutoEncoder (CWAE). CWAE cost function, based on introduced Cramer-Wold distance between samples, has a simple closed-form in the case of normal prior. As a consequence, while simplifying the optimization procedure (no need of sampling necessary to evaluate the distance function in the training loop), CWAE performance matches quantitatively and qualitatively that of WAE-MMD (WAE using maximum mean discrepancy based distance function) and often improves upon SWAE.

本文提出了一种基于编码数据分布和潜在先验分布之间新的统计距离的WAE变体

介绍了Wasserstein AudoEncoders的一个变体，这是一个新颖的正则化自动编码器架构，提出了发散惩罚的特定选择。


We propose a rejection sampling scheme using the discriminator of a GAN to  approximately correct errors in the GAN generator distribution. We show that  under quite strict assumptions, this will allow us to recover the data distribution  exactly. We then examine where those strict assumptions break down and design a  practical algorithm—called Discriminator Rejection Sampling (DRS)—that can be  used on real data-sets. Finally, we demonstrate the efficacy of DRS on a mixture of  Gaussians and on the state of the art SAGAN model. On ImageNet, we train an  improved baseline that increases the best published Inception Score from 52.52 to  62.36 and reduces the Frechet Inception Distance from 18.65 to 14.79. We then use  DRS to further improve on this baseline, improving the Inception Score to 76.08  and the FID to 13.75.

我们使用一个GAN判别器对GAN发生器的输出进行近似的拒绝采样方案。

提出了一种用于从GAN发生器中取样的拒绝取样算法。

本文提出了一种用于GANs的后处理拒绝采样方案，名为Discriminator Rejection Sampling，以帮助过滤GAN发生器中的 "好 "样本。


The quality of the features used in visual recognition is of fundamental importance for the overall system. For a long time, low-level hand-designed feature algorithms as SIFT and HOG have obtained the best results on image recognition. Visual features have recently been extracted from trained convolutional neural networks. Despite the high-quality results, one of the main drawbacks of this approach, when compared with hand-designed features, is the training time required during the learning process. In this paper, we propose a simple and fast way to train supervised convolutional models to feature extraction while still maintaining its high-quality. This methodology is evaluated on different datasets and compared with state-of-the-art approaches.

从卷积神经网络中提取视觉特征的一种简单快速方法

提出了一种快速学习卷积特征的方法，以后可以通过减少训练epoch的数量和学习率的特定时间表延迟来使用任何分类器。

使用相对于训练中使用的epoch数固定的学习率衰减方案，并提取倒数第二层的输出作为特征来训练常规分类器。


We develop a framework for understanding and improving recurrent neural networks (RNNs) using max-affine spline operators (MASOs). We prove that RNNs using piecewise affine and convex nonlinearities can be written as a simple piecewise affine spline operator. The resulting representation provides several new perspectives for analyzing RNNs, three of which we study in this paper. First, we show that an RNN internally partitions the input space during training and that it builds up the partition through time. Second, we show that the affine slope parameter of an RNN corresponds to an input-specific template, from which we can interpret an RNN as performing a simple template matching (matched filtering) given the input. Third, by carefully examining the MASO RNN affine mapping, we prove that using a random initial hidden state corresponds to an explicit L2 regularization of the affine parameters, which can mollify exploding gradients and improve generalization. Extensive experiments on several datasets of various modalities demonstrate and validate each of the above conclusions. In particular, using a random initial hidden states elevates simple RNNs to near state-of-the-art performers on these datasets.

我们从最大affine spline算子的角度对RNN进行了新的见解和解释。

提供一种使用最大affline spline算子 （MASO） 理解 RNN 的新方法，方法是使用分段affine和凸激活 MASO 重写它们。

作者在最大affine spline算子的基础上，对一大类深度网络进行了解释，重点介绍了使用初始隐藏状态的噪声作为正则化的recurrent神经网络。


Reasoning over text and Knowledge Bases (KBs) is a major challenge for Artificial Intelligence, with applications in machine reading, dialogue, and question answering.  Transducing text to logical forms which can be operated on is a brittle and error-prone process . Operating directly on text by jointly learning representations and transformations thereof by means of neural architectures that lack the ability to learn and exploit general rules can be very data-inefficient and not generalise correctly . These issues are addressed by Neural Theorem Provers (NTPs) (Rocktäschel & Riedel, 2017), neuro-symbolic systems based on a continuous relaxation of Prolog’s backward chaining algorithm, where symbolic unification between atoms is replaced by a differentiable operator computing the similarity between their embedding representations . In this paper, we first propose Neighbourhood-approximated Neural Theorem Provers (NaNTPs) consisting of two extensions toNTPs, namely a) a method for drastically reducing the previously prohibitive time and space complexity during inference and learning, and b) an attention mechanism for improving the rule learning process, deeming them usable on real-world datasets. Then, we propose a novel approach for jointly reasoning over KB facts and textual mentions, by jointly embedding them in a shared embedding space. The proposed method is able to extract rules and provide explanations—involving both textual patterns and KB relations—from large KBs and text corpora. We show that NaNTPs perform on par with NTPs at a fraction of a cost, and can achieve competitive link prediction results on challenging large-scale datasets, including WN18, WN18RR, and FB15k-237 (with and without textual mentions) while being able to provide explanations for each prediction and extract interpretable rules.

我们将神经定理证明器扩展到大型数据集，改进规则学习过程，并将其扩展到文本和知识库的联合推理。

提出神经定理证明系统的扩展，通过降低模型的时间和空间复杂性来解决该模型的主要问题

在统一过程中，通过对事实和规则进行近似的近邻搜索来扩展NTPs，并建议使用对已知谓词的关注来设置谓词的参数

通过使用最近的邻居搜索，在以前提出的神经定理证明方法的基础上进行改进。


We investigate the methods by which a Reservoir Computing Network (RCN) learns concepts such as 'similar' and 'different' between pairs of images using a small training dataset and generalizes these concepts to previously unseen types of data. Specifically, we show that an RCN trained to identify relationships between image-pairs drawn from a subset of digits from the MNIST database or the depth maps of subset of visual scenes from a moving camera generalizes the learned transformations to images of digits unseen during training or depth maps of different visual scenes. We infer, using Principal Component Analysis, that the high dimensional reservoir states generated from an input image pair with a specific transformation converge over time to a unique relationship. Thus, as opposed to training the entire high dimensional reservoir state, the reservoir only needs to train on these unique relationships, allowing the reservoir to perform well with very few training examples. Thus, generalization of learning to unseen images is interpretable in terms of clustering of the reservoir state onto the attractor corresponding to the transformation in reservoir space. We find that RCNs can identify and generalize linear and non-linear transformations, and combinations of transformations, naturally and be a robust and effective image classifier. Additionally, RCNs perform significantly better than state of the art neural network classification techniques such as deep Siamese Neural Networks (SNNs) in generalization tasks both on the MNIST dataset and more complex depth maps of visual scenes from a moving camera. This work helps bridge the gap between explainable machine learning and biological learning through analogies using small datasets, and points to new directions in the investigation of learning processes.

利用可解释的动态系统模型、水库计算和基于类比的生物学上合理的学习技术，将使用少量训练数据学习的成对图像之间的关系推广到以前未见过的图像类型。

通过使用具有标准tanh积分的回声状态网络，在RC背景下声称"组合变换"的结果，其区别在于循环权重未经过训练。

对MNIST数据的不同失真进行分类的新方法

本文使用一个回声状态网络来学习将成对图像之间的图像转换分为五类中的一类。


We present Generative Adversarial Privacy and Fairness (GAPF), a data-driven framework for learning private and fair representations of the data. GAPF leverages recent advances in adversarial learning to allow a data holder to learn "universal" representations that decouple a set of sensitive attributes from the rest of the dataset. Under GAPF, finding the optimal decorrelation scheme is formulated as a constrained minimax game between a generative decorrelator and an adversary. We show that for appropriately chosen adversarial loss functions, GAPF provides privacy guarantees against strong information-theoretic adversaries and enforces demographic parity. We also evaluate the performance of GAPF on multi-dimensional Gaussian mixture models and real datasets, and show how a designer can certify that representations learned under an adversary with a fixed architecture perform well against more complex adversaries.

我们提出了生成对抗性隐私和公平（GAPF），这是一个数据驱动的框架，用于学习具有认证隐私/公平保证的私有和公平表征。

本文使用一个GAN模型来概述与私有/公平表征学习（PRL）相关的工作。

本文提出了一种基于对抗性的方法来进行私人和公平的表示，通过学习性的数据失真，最大限度地减少对敏感变量的依赖，同时失真程度受到限制。

作者描述了一个如何学习可用于训练某些分类器的人口学奇偶校验表示的框架。


Current machine learning algorithms can be easily fooled by adversarial examples. One possible solution path is to make models that use confidence thresholding to avoid making mistakes. Such models refuse to make a prediction when they are not confident of their answer. We propose to evaluate such models in terms of tradeoff curves with the goal of high success rate on clean examples and low failure rate on adversarial examples. Existing untargeted attacks developed for models that do not use confidence thresholding tend to underestimate such models' vulnerability. We propose the MaxConfidence family of attacks, which are optimal in a variety of theoretical settings, including one realistic setting: attacks against linear models. Experiments show the attack attains good results in practice. We show that simple defenses are able to perform well on MNIST but not on CIFAR, contributing further to previous calls that MNIST should be retired as a benchmarking dataset for adversarial robustness research.   We release code for these evaluations as part of the cleverhans (Papernot et al 2018) library  (ICLR reviewers should be careful not to look at who contributed these features to cleverhans to avoid de-anonymizing this submission).

我们提出了评估模型的指标和最佳攻击方式，这些模型利用置信度阈值来抵御对抗性的示例

本文介绍了对置信度阈值算法的攻击系列，主要集中在评估方法上。

提出了一种置信度阈值防御模型的评估方法，以及在使用定向攻击时，通过选择置信度最高的错误类来生成对抗性例子的方法

本文提出了一种评价方法，用于评价对置信度阈值法的攻击，并提出了一种新的攻击。


Deep learning has achieved remarkable successes in solving challenging reinforcement learning (RL) problems when dense reward function is provided. However, in sparse reward environment it still often suffers from the need to carefully shape reward function to guide policy optimization. This limits the applicability of RL in the real world since both reinforcement learning and domain-specific knowledge are required. It is therefore of great practical importance to develop algorithms which can learn from a binary signal indicating successful task completion or other unshaped, sparse reward signals. We propose a novel method called competitive experience replay, which efficiently supplements a sparse reward by placing learning in the context of an exploration competition between a pair of agents. Our method complements the recently proposed hindsight experience replay (HER) by inducing an automatic exploratory curriculum. We evaluate our approach on the tasks of reaching various goal locations in an ant maze and manipulating objects with a robotic arm. Each task provides only binary rewards indicating whether or not the goal is achieved. Our method asymmetrically augments these sparse rewards for a pair of agents each learning the same task, creating a competitive game designed to drive exploration. Extensive experiments demonstrate that this method leads to faster converge and improved task performance.

一种利用对抗性奖励重新标记的稀疏奖励学习的新方法

建议使用竞争性的多代理设置来鼓励探索，并表明CER + HER > HER ~ CER

提出一种在无模型强化学习环境中从稀疏的奖励中学习的新方法，并使奖励密集化

为了解决稀疏的奖励问题并鼓励RL算法的探索，作者提出了一种叫做竞争性经验回复（CER）的重新标记策略。


This paper proposes a neural end-to-end text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions. The model is formulated as a conditional generative model with two levels of hierarchical latent variables. The first level is a categorical variable, which represents attribute groups (e.g. clean/noisy) and provides interpretability. The second level, conditioned on the first, is a multivariate Gaussian variable, which characterizes specific attribute configurations (e.g. noise level, speaking rate) and enables disentangled fine-grained control over these attributes. This amounts to using a Gaussian mixture model (GMM) for the latent distribution. Extensive evaluation demonstrates its ability to control the aforementioned attributes. In particular, it is capable of consistently synthesizing high-quality clean speech regardless of the quality of the training data for the target speaker.

用高斯混合VAE建立TTS模型，可以对说话风格、噪音条件等进行精细的控制。

描述了有条件的GAN模型，通过增强与识别相对应的Z空间来生成说话人的条件Mel谱。

提出一种两层潜变量模型，以获得解缠潜变量表示，从而促进对各种属性的细粒度控制。

本文提出了一个可以控制非注释属性的模型，如说话风格、口音、背景噪音等。


Visual Question Answering (VQA) models have struggled with counting objects in natural images so far. We identify a fundamental problem due to soft attention in these models as a cause. To circumvent this problem, we propose a neural network component that allows robust counting from object proposals. Experiments on a toy task show the effectiveness of this component and we obtain state-of-the-art accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model. On a difficult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.

通过处理重叠的对象建议，使视觉问题回答模型能够计数。

本文提出了一个手工设计的网络架构，在对象建议图上进行软性非最大限度的抑制，以获得对象数量。

关注视觉问题回答中的计数问题，使用注意力机制并提出一个可微分的计数组件，明确地计算物体的数量。

本文解决了视觉问题回答中的物体计数问题，它提出了许多启发式方法来寻找正确的计数。


We propose a simple and robust training-free approach for building sentence representations. Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and its surrounding context in a sentence. We model the semantic meaning of a word in a sentence based on two aspects. One is its relatedness to the word vector subspace already spanned by its contextual words. The other is its novel semantic meaning which shall be introduced as a new basis vector perpendicular to this existing subspace.   Following this motivation, we develop an innovative method based on orthogonal basis to combine pre-trained word embeddings into sentence representation. This approach requires zero training and zero parameters, along with efficient inference performance. We evaluate our approach on 11 downstream NLP tasks. Experimental results show that our model outperforms all existing zero-training alternatives in all the tasks and it is competitive to other approaches relying on either large amounts of labelled data or prolonged training time.

一种简单的、无需训练的句子嵌入方法，与需要大量训练数据或延长训练时间的复杂模型相比，其性能具有竞争力。

通过系统分析，提出了一种新的无训练生成句子嵌入的方法

提出了一种新的基于几何学的方法，通过量化每个词的新颖性、重要性和语料库唯一性，从词的嵌入向量中嵌入句子。

本文探讨了基于单词嵌入的正交分解扫描空间的句子嵌入。


In few-shot classification, we are interested in learning algorithms that train a classifier from only a handful of labeled examples. Recent progress in few-shot classification has featured meta-learning, in which a parameterized model for a learning algorithm is defined and trained on episodes representing different classification problems, each with a small labeled training set and its corresponding test set. In this work, we advance this few-shot classification paradigm towards a scenario where unlabeled examples are also available within each episode. We consider two situations: one where all unlabeled examples are assumed to belong to the same set of classes as the labeled examples of the episode, as well as the more challenging situation where examples from other distractor classes are also provided. To address this paradigm, we propose novel extensions of Prototypical Networks (Snell et al., 2017) that are augmented with the ability to use unlabeled examples when producing prototypes. These models are trained in an end-to-end way on episodes, to learn to leverage the unlabeled examples successfully. We evaluate these methods on versions of the Omniglot and miniImageNet benchmarks, adapted to this new framework augmented with unlabeled examples. We also propose a new split of ImageNet, consisting of a large set of classes, with a hierarchical structure. Our experiments confirm that our Prototypical Networks can learn to improve their predictions due to unlabeled examples, much like a semi-supervised algorithm would.

我们提出了原型网络的新颖扩展，这些扩展通过在生产原型时使用未标记示例的能力进行了增强。

本文是一个原型网络的扩展，考虑采用现有的未标记的例子来帮助训练每个episode

通过将原型网络扩展到半监督学习的设置中，以干扰类为例，研究半监督少数镜头分类的问题。

通过使用分配的伪标签更新原型，处理干扰因素，以及使用与原始原型的距离来衡量样本，将原型网络扩展到半监督环境中。


We investigate the properties of multidimensional probability distributions in the context of latent space prior distributions of implicit generative models. Our work revolves around the phenomena arising while decoding linear interpolations between two random latent vectors -- regions of latent space in close proximity to the origin of the space are oversampled, which restricts the usability of linear interpolations as a tool to analyse the latent space. We show that the distribution mismatch can be eliminated completely by a proper choice of the latent probability distribution or using non-linear interpolations. We prove that there is a trade off between the interpolation being linear, and the latent distribution having even the most basic properties required for stable training, such as finite mean. We use the multidimensional Cauchy distribution as an example of the prior distribution, and also provide a general method of creating non-linear interpolations, that is easily applicable to a large family of commonly used latent distributions.

我们从理论上证明，线性插值不适合用于分析训练好的隐性生成模型。

研究当两个随机变量之间的线性插值遵循相同的分布时，与隐性生成模型的先验分布有关的问题

这项工作问的是如何在给定的潜变量模型中进行插值。


Deep neural networks (DNN) have shown promising performance in computer vision. In medical imaging, encouraging results have been achieved with deep learning for applications such as segmentation, lesion detection and classification. Nearly all of the deep learning based image analysis methods work on reconstructed images, which are obtained from original acquisitions via solving inverse problems (reconstruction). The reconstruction algorithms are designed for human observers, but not necessarily optimized for DNNs which can often observe features that are incomprehensible for human eyes. Hence, it is desirable to train the DNNs directly from the original data which lie in a different domain with the images. In this paper, we proposed an end-to-end DNN for abnormality detection in medical imaging. To align the acquisition with the annotations made by radiologists in the image domain, a DNN was built as the unrolled version of iterative reconstruction algorithms to map the acquisitions to images, and followed by a 3D convolutional neural network (CNN) to detect the abnormality in the reconstructed images. The two networks were trained jointly in order to optimize the entire DNN for the detection task from the original acquisitions. The DNN was implemented for lung nodule detection in low-dose chest computed tomography (CT), where a numerical simulation was done to generate acquisitions from 1,018 chest CT images with radiologists' annotations. The proposed end-to-end DNN demonstrated better sensitivity and accuracy for the task compared to a two-step approach, in which the reconstruction and detection DNNs were trained separately. A significant reduction of false positive rate on suspicious lesions were observed, which is crucial for the known over-diagnosis in low-dose lung CT imaging. The images reconstructed by the proposed end-to-end network also presented enhanced details in the region of interest.

从投影数据而不是图像开始检测肺部结节。

DNNs被用于CT投影数据中基于图像块的肺结节检测。

通过对原始正弦图到检测输出的映射进行端到端训练，对肺部的计算机断层成像重建和病变检测进行联合建模

介绍了一个结合CT图像信号处理和图像分析的CNN架构的端到端训练。


Deep reinforcement learning (DRL) algorithms have demonstrated progress in learning to find a goal in challenging environments. As the title of the paper by Mirowski et al. (2016) suggests, one might assume that DRL-based algorithms are able to “learn to navigate” and are thus ready to replace classical mapping and path-planning algorithms, at least in simulated environments. Yet, from experiments and analysis in this earlier work, it is not clear what strategies are used by these algorithms in navigating the mazes and finding the goal. In this paper, we pose and study this underlying question: are DRL algorithms doing some form of mapping and/or path-planning? Our experiments show that the algorithms are not memorizing the maps of mazes at the testing stage but, rather, at the training stage. Hence, the DRL algorithms fall short of qualifying as mapping or path-planning algorithms with any reasonable definition of mapping. We extend the experiments in Mirowski et al. (2016) by separating the set of training and testing maps and by a more ablative coverage of the space of experiments. Our systematic experiments show that the NavA3C-D1-D2-L algorithm, when trained and tested on the same maps, is able to choose the shorter paths to the goal. However, when tested on unseen maps the algorithm utilizes a wall-following strategy to find the goal without doing any mapping or path planning.

我们在各种条件下对基于深度强化学习的导航方法进行定量和定性评估，以回答它们在多大程度上可以取代经典的路径规划器和绘图算法。

通过测量到达目标的重复延迟和与最短路线的比较，评估基于深度RL的训练迷宫模型。


In many robotic applications, it is crucial to maintain a belief about the state of   a system, like the location of a robot or the pose of an object.  These state estimates serve as input for planning and decision making and   provide feedback during task execution.   Recursive Bayesian Filtering algorithms address the state estimation problem,  but they require a model of the process dynamics and the sensory observations as well as   noise estimates that quantify the accuracy of these models.   Recently, multiple works have demonstrated that the process and sensor models can be   learned by end-to-end training through differentiable versions of Recursive Filtering methods.  However, even if the predictive models are known, finding suitable noise models   remains challenging. Therefore, many practical applications rely on very simplistic noise   models.   Our hypothesis is that end-to-end training through differentiable Bayesian   Filters enables us to learn more complex heteroscedastic noise models for  the system dynamics. We evaluate learning such models with different types of   filtering algorithms and on two different robotic tasks. Our experiments show that especially   for sampling-based filters like the Particle Filter, learning heteroscedastic noise   models can drastically improve the tracking performance in comparison to using   constant noise models.

我们评估了在不同的微分贝叶滤波器中学习异方差噪声模型的情况。

建议通过可微分的贝叶斯滤波器和两个不同版本的无中心卡尔曼滤波器，通过端到端优化预测似然，从数据中学习异方差噪声模型。

重新审视贝叶斯过滤器，并评估在保持所有其他模型固定的情况下训练观察和过程噪声模型的好处。

本文提出了一种在传统贝叶斯滤波算法中学习和使用依赖于状态和观测的噪声的方法。该方法包括构建一个神经网络模型，将原始观测数据作为输入，并产生一个紧凑的表示和相关的对角线协方差。


Graph convolutional neural networks have recently shown great potential for the task of zero-shot learning. These models are highly sample efficient as related concepts in the graph structure share statistical strength allowing generalization to new classes when faced with a lack of data. However, we find that the extensive use of Laplacian smoothing at each layer in current approaches can easily dilute the knowledge from distant nodes and consequently decrease the performance in zero-shot learning. In order to still enjoy the benefit brought by the graph structure while preventing the dilution of knowledge from distant nodes, we propose a Dense Graph Propagation (DGP) module with carefully designed direct links among distant nodes. DGP allows us to exploit the hierarchical graph structure of the knowledge graph through additional connections. These connections are added based on a node's relationship to its ancestors and descendants. A weighting scheme is further used to weigh their contribution depending on the distance to the node. Combined with finetuning of the representations in a two-stage training approach our method outperforms state-of-the-art zero-shot learning approaches.

我们重新思考如何更有效地利用知识图谱中的信息，以提高零点学习任务的性能，并为此提出密集图传播（DGP）模块。

该作者提出了一个解决Graph conv网络中过度平滑问题的方案，即允许所有相关节点之间的密集传播，并以相互距离为权重。

提出了一种新颖的图卷积神经网络，通过使用类之间的关系结构作为图卷积网络的输入来学习未见过的类的分类器，以解决零样本分类问题。


In this paper, we propose a capsule-based neural network model to solve the semantic segmentation problem. By taking advantage of the extractable part-whole dependencies available in capsule layers, we derive the probabilities of the class labels for individual capsules through a recursive, layer-by-layer procedure. We model this procedure as a traceback pipeline and take it as a central piece to build an end-to-end segmentation network. Under the proposed framework, image-level class labels and object boundaries are jointly sought in an explicit manner, which poses a significant advantage over the state-of-the-art fully convolutional network (FCN) solutions. Experiments conducted on modified MNIST and neuroimages demonstrate that our model considerably enhance the segmentation performance compared to the leading FCN variant.

基于胶囊的语义分割，其中类标签的概率是通过胶囊管道追溯的。

作者提出了一个回溯机制，将最低级别的胶囊与它们各自的类别联系起来

提出了胶囊网络的追踪层来做语义分割，并在胶囊层中明确使用部分-整体关系

提出了一种基于Sabour的CapsNet概念的回溯方法，在进行分类的同时进行语义分割。


Studying the evolution of information theoretic quantities during Stochastic Gradient Descent (SGD) learning of Artificial Neural Networks (ANNs) has gained popularity in recent years.   Nevertheless, these type of experiments require estimating mutual information and entropy which becomes intractable for moderately large problems. In this work we propose a framework for understanding SGD learning in the information plane which consists of observing entropy and conditional entropy of the output labels of ANN. Through experimental results and theoretical justifications it is shown that, under some assumptions, the SGD learning trajectories appear to be similar for different ANN architectures. First, the SGD learning is modeled as a Hidden Markov Process (HMP) whose entropy tends to increase to the maximum. Then, it is shown that the SGD learning trajectory appears to move close to the shortest path between the initial and final joint distributions in the space of probability measures equipped with the total variation metric. Furthermore, it is shown that the trajectory of learning in the information plane can provide an alternative for observing the learning process, with potentially richer information about the learning than the trajectories in training and test error.

我们把SGD看作是概率度量空间中的轨迹，显示了它与马尔可夫过程的联系，提出了一个简单的SGD学习的马尔可夫模型，并在实验中用信息论的数量与SGD进行比较。

构建一个马尔科夫链，该链遵循P上TV度量的短路径，并表明SGD和α-SMLC的轨迹具有相似的条件熵

研究H(\hat{y})与H(\hat{y}|y)在训练神经网络的随机梯度下降方法的信息平面上的运行轨迹

从分布p(y',y)的角度描述SGD，其中y是（可能被破坏的）真类标签，y'是模型预测值。


Stochastic gradient Markov chain Monte Carlo (SG-MCMC) has become increasingly popular for simulating posterior samples in large-scale Bayesian modeling. However, existing SG-MCMC schemes are not tailored to any specific probabilistic model, even a simple modification of the underlying dynamical system requires significant physical intuition. This paper presents the first meta-learning algorithm that allows automated design for the underlying continuous dynamics of an SG-MCMC sampler. The learned sampler generalizes Hamiltonian dynamics with state-dependent drift and diffusion, enabling fast traversal and efficient exploration of energy landscapes. Experiments validate the proposed approach on Bayesian fully connected neural network, Bayesian convolutional neural network and Bayesian recurrent neural network tasks, showing that the learned sampler outperforms generic, hand-designed SG-MCMC algorithms, and generalizes to different datasets and larger architectures.

本文提出了一种利用元学习方法自动设计随机梯度MCMC方案的方法。

提出了一种元学习方法，自动设计基于哈密尔顿动力学的MCMC采样器，以便在与训练问题类似的问题上更快地混合。

通过神经网络对扩散和卷曲矩阵进行参数化，并对sg-mcmc算法进行元学习和优化。


We propose a new, multi-component energy function for energy-based Generative Adversarial Networks (GANs) based on methods from the image quality assessment literature. Our approach expands on the Boundary Equilibrium Generative Adversarial Network (BEGAN) by outlining some of the short-comings of the original energy and loss functions. We address these short-comings by incorporating an l1 score, the Gradient Magnitude Similarity score, and a chrominance score into the new energy function. We then provide a set of systematic experiments that explore its hyper-parameters. We show that each of the energy function's components is able to represent a slightly different set of features, which require their own evaluation criteria to assess whether they have been adequately learned. We show that models using the new energy function are able to produce better image representations than the BEGAN model in predicted ways.

图像质量评估技术改善基于能量的生成对抗网络的训练和评估

对BEGAN模式提出了一个基于能量的表述，并对其进行了修改，以包括一个基于图像质量评估的术语。

在BEGAN（边界平衡GAN框架）中提出了一些新的能量函数，包括l_1得分、梯度大小相似度得分和色度得分。


Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions. Its performance depends crucially on a damping coefficient. Largecamping  coefficients can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typically resorts to small values such as 0.5 or 0.9. We propose Aggregated Momentum (AggMo), a variant of momentum which combines multiple velocity vectors with different damping coefficients. AggMo is trivial to implement, but significantly dampens oscillations, enabling it to remain stable even for aggressive damping coefficients such as 0.999. We reinterpret Nesterov's accelerated gradient descent as a special case of AggMo and analyze rates of convergence for quadratic objectives. Empirically, we find that AggMo is a suitable drop-in replacement for other momentum methods, and frequently delivers faster convergence with little to no tuning.

我们引入了一种简单的动量优化变体，它能够在深度学习任务上以最小的超参数调整来超越经典动量、Nesterov和Adam。

引入了一个动量的变体，它聚集了几个具有不同阻尼系数的速度，大大降低了振荡。

提出了一种基于梯度的优化的聚合动量方法，使用具有不同阻尼系数的多个速度矢量而不是单一速度矢量来提高稳定性

作者将几个更新步骤结合在一起，实现了聚合动量，也证明了它比其他动量方法更稳定。


Recurrent Neural Networks architectures excel at processing sequences by  modelling dependencies over different timescales. The recently introduced  Recurrent Weighted Average (RWA) unit captures long term dependencies  far better than an LSTM on several challenging tasks. The RWA achieves  this by applying attention to each input and computing a weighted average  over the full history of its computations. Unfortunately, the RWA cannot  change the attention it has assigned to previous timesteps, and so struggles  with carrying out consecutive tasks or tasks with changing requirements.  We present the Recurrent Discounted Attention (RDA) unit that builds on  the RWA by additionally allowing the discounting of the past.  We empirically compare our model to RWA, LSTM and GRU units on  several challenging tasks. On tasks with a single output the RWA, RDA and  GRU units learn much quicker than the LSTM and with better performance.  On the multiple sequence copy task our RDA unit learns the task three  times as quickly as the LSTM or GRU units while the RWA fails to learn at  all. On the Wikipedia character prediction task the LSTM performs best  but it followed closely by our RDA unit. Overall our RDA unit performs  well and is sample efficient on a large variety of sequence tasks.

我们引入了循环贴现单元，它在线性时间内将注意力应用于任何长度的序列。

本文提出了循环贴现关注（RDA），这是循环加权平均（RWA）的延伸，加入了一个贴现因子。

在保持原有方法优势的基础上，扩展了循环权重平均法，克服了原有方法的局限性，提出了以Elman网为基础的RNN方法


Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging. In this paper, we introduce variance layers, a different kind of stochastic layers. Each weight of a variance layer follows a zero-mean distribution and is only parameterized by its variance. It means that each object is represented by a zero-mean distribution in the space of the activations. We show that such layers can learn surprisingly well, can serve as an efficient exploration tool in reinforcement learning tasks and provide a decent defense against adversarial attacks. We also show that a number of conventional Bayesian neural networks naturally converge to such zero-mean posteriors. We observe that in these cases such zero-mean parameterization leads to a much better training objective than more flexible conventional parameterizations where the mean is being learned.

通过只学习方差，可以在神经网络的权重上学习一个零中心的高斯分布，而且效果出奇地好。

本文研究了变分后验的平均值的影响，并提出了方差层，它只使用方差来存储信息

研究方差神经网络，它近似于具有零均值高斯分布的贝叶斯神经网络的后验。


Graph Convolutional Networks (GCNs) are a recently proposed architecture which has had success in semi-supervised learning on graph-structured data. At the same time, unsupervised learning of graph embeddings has benefited from the information contained in random walks. In this paper we propose a model, Network of GCNs (N-GCN), which marries these two lines of work. At its core, N-GCN trains multiple instances of GCNs over node pairs discovered at different distances in random walks, and learns a combination of the instance outputs which optimizes the classification objective. Our experiments show that our proposed N-GCN model achieves state-of-the-art performance on all of the challenging node classification tasks we consider: Cora, Citeseer, Pubmed, and PPI. In addition, our proposed method has other desirable properties, including generalization to recently proposed semi-supervised learning methods such as GraphSAGE, allowing us to propose N-SAGE, and resilience to adversarial input perturbations.

我们制作了一个图卷积网络网络，为每个邻接矩阵提供不同的幂，将它们的所有表示组合到一个分类子网络中，实现了最先进的半监督节点分类。

提出了一种新的GCN网络，有两种方法：在叠加特征之上的全连接层和每个GCN使用标量权重的注意力机制。

提出了一个图卷积网络的网络，它使用随机行走的统计数据来提取图中近邻和远邻的信息


Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers often with little or no drop in classification accuracy. However most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain classification accuracy. In this paper we propose a cheap pruning algorithm based on difference of convex (DC) optimisation. We also provide theoretical analysis for the growth in the Generalisation Error (GE) of the new pruned network. Our method can be used with any convex regulariser and allows for a controlled degradation in classification accuracy while being orders of magnitude faster than competing approaches. Experiments on common feedforward neural networks show that for sparsity levels above 90% our method achieves 10% higher classification accuracy compared to Hard Thresholding.

一种全连接DNN层的快速修剪算法，具有泛化误差退化的理论分析。

提出了一种廉价的DNN密集层的修剪算法。


Action segmentation as a milestone towards building automatic systems to understand untrimmed videos has received considerable attention in the recent years. It is typically being modeled as a sequence labeling problem but contains intrinsic and sufficient differences than text parsing or speech processing. In this paper, we introduce a novel hybrid temporal convolutional and recurrent network (TricorNet), which has an encoder-decoder architecture: the encoder consists of a hierarchy of temporal convolutional kernels that capture the local motion changes of different actions; the decoder is a hierarchy of recurrent neural networks that are able to learn and memorize long-term action dependencies after the encoding stage. Our model is simple but extremely effective in terms of video sequence labeling. The experimental results on three public action segmentation datasets have shown that the proposed model achieves superior performance over the state of the art.

我们提出了一个新的混合时空网络，在三个公共数据集的视频动作分割上取得了最先进的性能。

讨论了通过使用时间卷积编码器-解码器架构在长达10分钟的长视频中进行动作分割的问题

提出了一种用于视频动作分割的时间卷积和递归网络组合


Convolutional Neural Networks (CNNs) become deeper and deeper in recent years, making the study of model acceleration imperative. It is a common practice to employ a shallow network, called student, to learn from a deep one, which is termed as teacher. Prior work made many attempts to transfer different types of knowledge from teacher to student, however, there are two problems remaining unsolved. Firstly, the knowledge used by existing methods is highly dependent on task and dataset, limiting their applications. Secondly, there lacks an effective training scheme for the transfer process, leading to degradation of performance. In this work, we argue that feature is the most important knowledge from teacher. It is sufficient for student to just learn good features regardless of the target task. From this discovery, we further present an efficient learning strategy to mimic features stage by stage. Extensive experiments demonstrate the importance of features and show that the proposed approach significantly narrows down the gap between student and teacher, outperforming the state-of-the-art methods.

本文提出通过逐级模仿特征将知识从深层模型转移到浅层模型。

通过使用不同的 resnet 结构来解释逐阶段知识转换方法

本文提出将一个网络分为多个部分，并依次对每个部分进行蒸馏，以提高深度教师网络的蒸馏性能


We augment adversarial training (AT) with worst case adversarial training  (WCAT) which improves adversarial robustness by 11% over the current state-  of-the-art result in the `2-norm on CIFAR-10. We interpret adversarial training as  Total Variation Regularization, which is a fundamental tool in mathematical im-  age processing, and WCAT as Lipschitz regularization, which appears in Image  Inpainting. We obtain verifiable worst and average case robustness guarantees,  based on the expected and maximum values of the norm of the gradient of the  loss.

通过用可操作的Lipschitz正则化来增强对抗性训练，可以获得对抗性稳健性的改进，以及可证明的稳健性保证。

探讨用一个额外的梯度正则化项来增加训练损失，以提高模型对敌对例子的稳健性。

使用一个技巧来简化对抗性损失，在其中对抗性扰动以封闭形式出现。


The task of Reading Comprehension with Multiple Choice Questions, requires a human (or machine) to read a given \{\textit{passage, question}\} pair and select one of the $n$ given options. The current state of the art model for this task first computes a query-aware representation for the passage and then \textit{selects} the option which has the maximum similarity with this representation. However, when humans perform this task they do not just focus on option selection but use a combination of \textit{elimination} and \textit{selection}. Specifically, a human would first try to eliminate the most irrelevant option and then read the document again in the light of this new information (and perhaps ignore portions corresponding to the eliminated option). This process could be repeated multiple times till the reader is finally ready to select the correct option. We propose \textit{ElimiNet}, a neural network based model which tries to mimic this process. Specifically, it has gates which decide whether an option can be eliminated given the \{\textit{document, question}\} pair and if so it tries to make the document representation orthogonal to this eliminatedd option (akin to ignoring portions of the document corresponding to the eliminated option). The model makes multiple rounds of partial elimination to refine the document representation and finally uses a selection module to pick the best option. We evaluate our model on the recently released large scale RACE dataset and show that it outperforms the current state of the art model on 7 out of the 13 question types in this dataset. Further we show that taking an ensemble of our \textit{elimination-selection} based method with a \textit{selection} based method gives us an improvement of 7\% (relative) over the best reported performance on this dataset.

一种结合了消除和选择的模式来回答多选题

对 "有门的注意力阅读器 "进行了阐述，在选择阅读理解中根据答案的消除来增加门。

本文提出在阅读理解任务的模型架构中使用消除门，但并没有取得最先进的结果

本文提出了一个新的多选阅读理解模型，该模型基于这样的理念：应该取消一些选项，以推断出更好的段落/问题的表述。


Humans are capable of attributing latent mental contents such as beliefs, or intentions to others. The social skill is critical in everyday life to reason about the potential consequences of their behaviors so as to plan ahead. It is known that humans use this reasoning ability recursively, i.e. considering what others believe about their own beliefs.   In this paper, we start  from level-$1$ recursion and introduce a probabilistic recursive reasoning (PR2) framework for multi-agent reinforcement learning. Our hypothesis is that it is beneficial for each agent to account for how the opponents would react to its future behaviors. Under the PR2 framework, we adopt variational Bayes methods to approximate the opponents' conditional policy, to which each agent finds the  best response and then improve their own policy. We develop  decentralized-training-decentralized-execution  algorithms, PR2-Q and PR2-Actor-Critic, that are proved to converge in the self-play scenario when there is one Nash equilibrium. Our methods are tested on both the matrix game and the differential game, which have a non-trivial equilibrium where common gradient-based methods fail to converge. Our experiments show that it is critical to reason about how the opponents believe about what the agent believes. We expect our work to contribute a new idea of modeling the opponents to the multi-agent reinforcement learning community.

我们提出了一个新颖的概率递归推理（PR2）框架，用于多代理深度强化学习任务。

提出了一种在多代理强化学习中完全去中心化训练的新方法

根据递归推理有利于收敛到非平凡均衡的假设，解决了在多Agent环境下赋予RL代理递归推理能力的问题。

本文介绍了一种用于多代理强化学习的去中心化训练方法，代理推断其他代理的政策并使用推断的模型进行决策。


Due to the substantial computational cost, training state-of-the-art deep neural networks for large-scale datasets often requires distributed training using multiple computation workers. However, by nature, workers need to frequently communicate gradients, causing severe bottlenecks, especially on lower bandwidth connections. A few methods have been proposed to compress gradient for efficient communication, but they either suffer a low compression ratio or significantly harm the resulting model accuracy, particularly when applied to convolutional neural networks. To address these issues, we propose a method to reduce the communication overhead of distributed deep learning. Our key observation is that gradient updates can be delayed until an unambiguous (high amplitude, low variance) gradient has been calculated. We also present an efficient algorithm to compute the variance and prove that it can be obtained with negligible additional cost. We experimentally show that our method can achieve very high compression ratio while maintaining the result model accuracy. We also analyze the efficiency using computation and communication cost models and provide the evidence that this method enables distributed deep learning for many scenarios with commodity environments.

一种新的算法，通过区分 "不明确 "的梯度来减少分布式深度学习的通信开销。
提出了一种基于方差的梯度压缩方法，以减少分布式深度学习的通信开销
提出了一种压缩分布式SGD梯度更新的新方法，以加快整体执行速度
引入了基于方差的梯度压缩方法，用于神经网络的高效分布式训练和测量模糊性。


In this work, we face the problem of unsupervised domain adaptation with a novel deep learning approach which leverages our finding that entropy minimization is induced by the optimal alignment of second order statistics between source and target domains. We formally demonstrate this hypothesis and, aiming at achieving an optimal alignment in practical cases, we adopt a more principled strategy which, differently from the current Euclidean approaches, deploys alignment along geodesics. Our pipeline can be implemented by adding to the standard classification loss (on the labeled source domain), a source-to-target regularizer that is weighted in an unsupervised and data-driven fashion. We provide extensive experiments to assess the superiority of our framework on standard domain and modality adaptation benchmarks.

一种新的无监督深层领域适应技术，有效地统一了相关性排列和熵值最小化。

通过用两个协方差矩阵之间的测地对数欧氏距离取代欧氏距离，并通过目标域上的熵自动选择平衡成本，改进了领域适应的相关排列方法。

提出了最小熵相关排列的建议，这是一种无监督的领域适应性算法，将熵的最小化和相关排列方法联系起来。


Catastrophic interference has been a major roadblock in the research of continual learning. Here we propose a variant of the back-propagation algorithm, "Conceptor-Aided Backprop" (CAB), in which gradients are shielded by conceptors against degradation of previously learned tasks. Conceptors have their origin in reservoir computing, where they have been previously shown to overcome catastrophic forgetting. CAB extends these results to deep feedforward networks. On the disjoint and permuted MNIST tasks, CAB outperforms two other methods for coping with catastrophic interference that have recently been proposed.

我们提出了一种反向传播算法的变体，在这种算法中，梯度被概念器所屏蔽，以防止以前所学任务的退化。

本文应用了概念器的概念，一种正则器的形式，以防止在连续任务的神经网络训练中的遗忘。

介绍了一种使用概念器学习新任务的方法，不干扰以前的任务。


Recent advances in neural Sequence-to-Sequence (Seq2Seq) models reveal a purely data-driven approach to the response generation task. Despite its diverse variants and applications, the existing Seq2Seq models are prone to producing short and generic replies, which blocks such neural network architectures from being utilized in practical open-domain response generation tasks. In this research, we analyze this critical issue from the perspective of the optimization goal of models and the specific characteristics of human-to-human conversational corpora. Our analysis is conducted by decomposing the goal of Neural Response Generation (NRG) into the optimizations of word selection and ordering. It can be derived from the decomposing that Seq2Seq based NRG models naturally tend to select common words to compose responses, and ignore the semantic of queries in word ordering. On the basis of the analysis, we propose a max-marginal ranking regularization term to avoid Seq2Seq models from producing the generic and uninformative responses. The empirical experiments on benchmarks with several metrics have validated our analysis and proposed methodology.

分析神经反应生成模型偏爱通用回答的原因；提出避免这种情况的方法。

研究困扰Seq2Seq神经生成模型的通用回复问题

本文通过修改损失函数和在训练阶段展示共同/普遍的反应来改进神经反应生成任务，不再强调共同反应。


The ability to generate natural language sequences from source code snippets has a variety of applications such as code summarization, documentation, and retrieval. Sequence-to-sequence (seq2seq) models, adopted from neural machine translation (NMT), have achieved state-of-the-art performance on these tasks by treating source code as a sequence of tokens. We present code2seq: an alternative approach that leverages the syntactic structure of programming languages to better encode source code. Our model represents a code snippet as the set of compositional paths in its abstract syntax tree (AST) and uses attention to select the relevant paths while decoding.  We demonstrate the effectiveness of our approach for two tasks, two programming languages, and four datasets of up to 16M examples. Our model significantly outperforms previous models that were specifically designed for programming languages, as well as general state-of-the-art NMT models. An interactive online demo of our model is available at http://code2seq.org. Our code, data and trained models are available at http://github.com/tech-srl/code2seq.

我们利用源代码的句法结构来生成自然语言序列。

介绍了一种通过解析和产生语法树从代码中生成序列的方法

本文介绍了一种基于AST的编程代码编码，并展示了其在极端代码总结和代码字幕任务中的有效性。

本文提出了一个新的代码到序列的模型，利用编程语言的句法结构对源代码片段进行编码，然后将其解码为自然语言。


We propose a novel attention mechanism to enhance Convolutional Neural Networks for fine-grained recognition. The proposed mechanism reuses CNN feature activations to find the most informative parts of the image at different depths with the help of gating mechanisms and without part annotations. Thus, it can be used to augment any layer of a CNN to extract low- and high-level local information to be more discriminative.    Differently, from other approaches, the mechanism we propose just needs a single pass through the input and it can be trained end-to-end through SGD. As a consequence, the proposed mechanism is modular, architecture-independent, easy to implement, and faster than iterative approaches.   Experiments show that, when augmented with our approach, Wide Residual Networks systematically achieve superior performance on each of five different fine-grained recognition datasets: the Adience age and gender recognition benchmark, Caltech-UCSD Birds-200-2011, Stanford Dogs, Stanford Cars, and UEC Food-100, obtaining competitive and state-of-the-art scores.

我们用一种新的注意力机制来增强CNN的细粒度识别。在5个数据集上获得了卓越的性能。

描述了一种适用于细粒度识别的新型注意机制，该机制能持续提高基线的识别精度

本文提出了一种用于细粒度图像分类的前馈注意力机制。

本文提出了一种用于细粒度图像分类的有趣注意力机制。


Most existing GANs architectures that generate images use transposed convolution or resize-convolution as their upsampling algorithm from lower to higher resolution feature maps in the generator. We argue that this kind of fixed operation is problematic for GANs to model objects that have very different visual appearances. We propose a novel adaptive convolution method that learns the upsampling algorithm based on the local context at each location to address this problem. We modify a baseline GANs architecture by replacing normal convolutions with adaptive convolutions in the generator. Experiments on CIFAR-10 dataset show that our modified models improve the baseline model by a large margin. Furthermore, our models achieve state-of-the-art performance on CIFAR-10 and STL-10 datasets in the unsupervised setting.

我们将普通卷积替换为自适应卷积，以改进 GAN 生成器。

建议用自适应卷积块取代生成器中的卷积，自适应卷积块可以学习生成卷积权重和每个像素位置的上采样操作的偏差。

在GANs的背景下使用自适应卷积，用一个叫做AdaConvBlock的块来代替常规的卷积，这给每个内核权重提供了更多的局部背景，因此它可以生成局部灵活的对象。


Techniques such as ensembling and distillation promise model quality improvements when paired with almost any base model. However, due to increased test-time cost (for ensembles) and increased complexity of the training pipeline (for distillation), these techniques are challenging to use in industrial settings. In this paper we explore a variant of distillation which is relatively straightforward to use as it does not require a complicated multi-stage setup or many new hyperparameters. Our first claim is that online distillation enables us to use extra parallelism to fit very large datasets about twice as fast. Crucially, we can still speed up training even after we have already reached the point at which additional parallelism provides no benefit for synchronous or asynchronous stochastic gradient descent. Two neural networks trained on disjoint subsets of the data can share knowledge by encouraging each model to agree with the predictions the other model would have made. These predictions can come from a stale version of the other model so they can be safely computed using weights that only rarely get transmitted. Our second claim is that online distillation is a cost-effective way to make the exact predictions of a model dramatically more reproducible. We support our claims using experiments on the Criteo Display Ad Challenge dataset, ImageNet, and the largest to-date dataset used for neural language modeling, containing $6\times 10^{11}$ tokens and based on the Common Crawl repository of web data.

我们进行了大规模的实验，表明一个简单的在线蒸馏变体可以帮助我们将分布式神经网络训练扩展到更多的机器。

提出了一种超越目前迷你批次随机梯度下降极限的分布式训练方法

提出了一种被称为共同蒸馏的在线蒸馏方法，在规模上的应用，其中训练两个不同的模型以匹配另一个模型的预测，并最大限度地减少其自身损失。

引入在线蒸馏技术，加速传统算法进行大规模分布式神经网络训练


Support Vector Machines (SVMs) are one of the most popular algorithms for classification and regression analysis. Despite their popularity, even efficient implementations have proven to be computationally expensive to train at a large-scale, especially in streaming settings. In this paper, we propose a novel coreset construction algorithm for efficiently generating compact representations of massive data sets to speed up SVM training. A coreset is a weighted subset of the original data points such that SVMs trained on the coreset are provably competitive with those trained on the original (massive) data set. We provide both lower and upper bounds on the number of samples required to obtain accurate approximations to the SVM problem as a function of the complexity of the input data. Our analysis also establishes sufficient conditions on the existence of sufficiently compact and representative coresets for the SVM problem. We empirically evaluate the practical effectiveness of our algorithm against synthetic and real-world data sets.

我们提出了一种算法，通过构建紧凑的表征，在海量数据集上加速SVM的训练，提供高效和可证明的近似推理。

研究SVM的核心集方法，旨在对一小部分加权点进行采样，从而证明这些点的损失函数与整个数据集的损失函数相近。

本文提出了一种基于重要性抽样的Coreset构造来表示SVM的大量训练数据。


The sign stochastic gradient descent method (signSGD) utilizes only the sign of the stochastic gradient in its updates. Since signSGD carries out one-bit quantization of the gradients, it is extremely practical for distributed optimization where gradients need to be aggregated from different processors. For the first time, we establish convergence rates for signSGD on general non-convex functions under transparent conditions. We show that the rate of signSGD to reach first-order critical points matches that of SGD in terms of number of stochastic gradient calls, up to roughly a linear factor in the dimension. We carry out simple experiments to explore the behaviour of sign gradient descent (without the stochasticity) close to saddle points and show that it often helps completely avoid them without using either stochasticity or curvature information.

我们证明了符号随机梯度方法的非凸收敛率。该算法与Adam和Rprop等算法以及分布式机器学习中使用的梯度量化方案有联系。

提供了Sign SGD算法在非凸情况下的收敛性分析

本文探讨了一种使用梯度的符号而不是实际梯度来训练深度模型的算法