Relational databases store a significant amount of the worlds data. However, accessing this data currently requires users to understand a query language such as SQL. We propose Seq2SQL, a deep neural network for translating natural language questions to corresponding SQL queries. Our model uses rewards from in the loop query execution over the database to learn a policy to generate the query, which contains unordered parts that are less suitable for optimization via cross entropy loss. Moreover, Seq2SQL leverages the structure of SQL to prune the space of generated queries and significantly simplify the generation problem. In addition to the model, we release WikiSQL, a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables fromWikipedia that is an order of magnitude larger than comparable datasets. By applying policy based reinforcement learning with a query execution environment to WikiSQL, Seq2SQL outperforms a state-of-the-art semantic parser, improving execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.

我们介绍了Seq2SQL，它利用在线查询执行的奖励将问题翻译成SQL查询，以及WikiSQL，一个比现有数据集大几个数量级的SQL表/问题/查询数据集。

一个新的语义解析数据集，它侧重于使用基于强化学习的模型从自然语言中生成SQL。


We introduce Explainable Adversarial Learning, ExL, an approach for training neural networks that are intrinsically robust to adversarial attacks. We find that the implicit generative modeling of random noise with the same loss function used during posterior maximization, improves a model's understanding of the data manifold furthering adversarial robustness. We prove our approach's efficacy and provide a simplistic visualization tool for understanding adversarial data, using Principal Component Analysis. Our analysis reveals that adversarial robustness, in general, manifests in models with higher variance along the high-ranked principal components. We show that models learnt with our approach perform remarkably well against a wide-range of attacks. Furthermore, combining ExL with state-of-the-art adversarial training extends the robustness of a model, even beyond what it is adversarially trained for, in both white-box and black-box attack scenarios.

鉴别性训练过程中的输入噪声建模提高了对抗性稳健性。提出基于PCA的对抗性稳健性评价指标

本文提出了ExL，一种使用乘法噪声的对抗性训练方法，该方法被证明有助于防御三个数据集的黑箱攻击。

本文在训练数据中包括乘法噪声N，以实现对抗性的稳健性，当对模型参数theta和噪声本身进行训练时。


We propose a method which can visually explain the classification decision of deep neural networks (DNNs). There are many proposed methods in machine learning and computer vision seeking to clarify the decision of machine learning black boxes, specifically DNNs.   All of these methods try to gain insight into why the network "chose class A" as an answer. Humans, when searching for explanations, ask two types of questions. The first question is, "Why did you choose this answer? " The second question asks, "Why did you not choose answer B over A?" The previously proposed methods are either not able to provide the latter directly or efficiently.   We introduce a method capable of answering the second question both directly and efficiently. In this work, we limit the inputs to be images. In general, the proposed method generates explanations in the input space of any model capable of efficient evaluation and gradient evaluation. We provide results, showing the superiority of this approach for gaining insight into the inner representation of machine learning models.

回答 "为什么不是B类？"的方法来解释深度网络
本文提出了一种为深度神经网络提供对比性视觉解释的方法。


We flip the usual approach to study invariance and robustness of neural networks by considering the non-uniqueness and instability of the inverse mapping. We provide theoretical and numerical results on the inverse of ReLU-layers. First, we derive a necessary and sufficient condition on the existence of invariance that provides a geometric interpretation. Next, we move to robustness via analyzing local effects on the inverse. To conclude, we show how this reverse point of view not only provides insights into key effects, but also enables to view adversarial examples from different perspectives.

我们通过研究ReLU-层的预成像和逆的稳定性来分析深度神经网络的可逆性。

本文研究了ReLU网络在某一层激活的预成像量，它建立在ReLU网络的前向函数的分段线性上。

本文对ReLU网络的逆不变性进行了分析，并提供了train网络的奇异值的上限。


While deep learning has led to remarkable results on a number of challenging problems, researchers have discovered a vulnerability of neural networks in adversarial settings, where small but carefully chosen perturbations to the input can make the models produce extremely inaccurate outputs. This makes these models particularly unsuitable for safety-critical application domains (e.g. self-driving cars) where robustness is extremely important. Recent work has shown that augmenting training with adversarially generated data provides some degree of robustness against test-time attacks. In this paper we investigate how this approach scales as we increase the computational budget given to the defender. We show that increasing the number of parameters in adversarially-trained models increases their robustness, and in particular that ensembling smaller models while adversarially training the entire ensemble as a single model is a more efficient way of spending said budget than simply using a larger single model. Crucially, we show that it is the adversarial training of the ensemble, rather than the ensembling of adversarially trained models, which provides robustness.

集合体的对抗性训练提供了对对抗性例子的稳健性，超过了在对抗性训练的模型和独立训练的集合体中观察到的。

建议联合训练一个集合模型，在每个时间步骤中，将一组对集合模型本身具有对抗性的例子纳入学习。


Multi-task learning (MTL) with neural networks leverages commonalities in tasks to improve performance, but often suffers from task interference which reduces the benefits of transfer. To address this issue we introduce the routing network paradigm, a novel neural network and training algorithm. A routing network is a kind of self-organizing neural network consisting of two components: a router and a set of one or more function blocks. A function block may be any neural network – for example a fully-connected or a convolutional layer. Given an input the router makes a routing decision, choosing a function block to apply and passing the output back to the router recursively, terminating when a fixed recursion depth is reached. In this way the routing network dynamically composes different function blocks for each input. We employ a collaborative multi-agent reinforcement learning (MARL) approach to jointly train the router and function blocks. We evaluate our model against cross-stitch networks and shared-layer baselines on multi-task settings of the MNIST, mini-imagenet, and CIFAR-100 datasets. Our experiments demonstrate a significant improvement in accuracy, with sharper convergence. In addition, routing networks have nearly constant per-task training cost while cross-stitch networks scale linearly with the number of tasks. On CIFAR100 (20 tasks) we obtain cross-stitch performance levels with an 85% average reduction in training time.

路由网络：一种新的神经网络，它可以为多任务学习学习自适应地确定其输入路线

该文件建议使用一个带有控制器的模块化网络，该控制器在每个时间步骤中对下一个应用的结核作出决定。

本文提出了一种在多任务学习框架下，通过使用多代理强化学习寻找策略来学习神经网络的最佳架构的新公式，并显示出比共享层的硬编码架构的改进。


We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization. However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected $L_0$ regularized objective is differentiable with respect to the distribution parameters. We further propose the \emph{hard concrete} distribution for the gates, which is obtained by ``stretching'' a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.

我们展示了如何用梯度下降法优化参数模型的预期L_0范数，并介绍了一种有利于硬门控的新分布。

作者介绍了一种基于梯度的方法来最小化带有L0稀疏惩罚的目标函数，以帮助学习稀疏神经网络


Recently popularized graph neural networks achieve the state-of-the-art accuracy on a number of standard benchmark datasets for graph-based semi-supervised learning, improving significantly over existing approaches. These architectures alternate between a propagation layer that aggregates the hidden states of the local neighborhood and a fully-connected layer. Perhaps surprisingly, we show that a linear model, that removes all the intermediate fully-connected layers, is still able to achieve a performance comparable to the state-of-the-art models. This significantly reduces the number of parameters, which is critical for semi-supervised learning where number of labeled examples are small. This in turn allows a room for designing more innovative propagation layers. Based on this insight, we propose a novel graph neural network that removes all the intermediate fully-connected layers, and replaces the propagation layers with attention mechanisms that respect the structure of the graph. The attention mechanism allows us to learn a dynamic and adaptive local summary of the neighborhood to achieve more accurate predictions. In a number of experiments on benchmark citation networks datasets, we demonstrate that our approach outperforms competing methods. By examining the attention weights among neighbors, we show that our model provides some interesting insights on how neighbors influence each other.

我们提出了一种新型的基于注意力的可解释图神经网络架构，在标准基准数据集中的表现优于目前最先进的图神经网络。

作者提出了GCN的两个扩展，即从GCN计算中去除中间的非线性，并在聚集层中增加一个注意力机制。

本文提出了一种半监督的图节点分类学习算法，其灵感来自于图神经网络。


Modern generative models are usually designed to match target distributions directly in the data space, where the intrinsic dimensionality of data can be much lower than the ambient dimensionality. We argue that this discrepancy may contribute to the difficulties in training generative models. We therefore propose to map both the generated and target distributions to the latent space using the encoder of a standard autoencoder, and train the generator (or decoder) to match the target distribution in the latent space. The resulting method, perceptual generative autoencoder (PGA), is then incorporated with maximum likelihood or variational autoencoder (VAE) objective to train the generative model. With maximum likelihood, PGA generalizes the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities. When combined with VAE, PGA can generate sharper samples than vanilla VAE.

基于自动编码器的生成模型训练框架，具有非对抗性损失和非限制性神经网络架构。 本文使用自动编码器来做高维空间的分布匹配。


The quality of the representations achieved by embeddings is determined by how well the geometry of the embedding space matches the structure of the data.  Euclidean space has been the workhorse for embeddings; recently hyperbolic and spherical spaces have gained popularity due to their ability to better embed new types of structured data---such as hierarchical data---but most data is not structured so uniformly.  We address this problem by proposing learning embeddings in a product manifold combining multiple copies of these model spaces (spherical, hyperbolic, Euclidean), providing a space of heterogeneous curvature suitable for a wide variety of structures.  We introduce a heuristic to estimate the sectional curvature of graph data and directly determine an appropriate signature---the number of component spaces and their dimensions---of the product manifold.  Empirically, we jointly learn the curvature and the embedding in the product space via Riemannian optimization.  We discuss how to define and compute intrinsic quantities such as means---a challenging notion for product manifolds---and provably learnable optimization functions.  On a range of datasets and reconstruction tasks, our product space embeddings outperform single Euclidean or hyperbolic spaces used in previous works, reducing distortion by 32.55% on a Facebook social network dataset. We learn word embeddings and find that a product of hyperbolic spaces in 50 dimensions consistently improves on baseline Euclidean and hyperbolic embeddings, by 2.6  points in Spearman rank correlation on similarity tasks  and 3.4 points on analogy accuracy.

与各种结构的传统嵌入空间相比，具有异质曲率的积流形嵌入空间产生了更好的表述。

提出了一种降维方法，将数据嵌入球形、欧几里得和双曲流形的积流形中。该算法的基础是将积流形上的测地距离与图的距离相匹配。


Synthesizing user-intended programs from a small number of input-output exam-  ples is a challenging problem with several important applications like spreadsheet  manipulation, data wrangling and code refactoring. Existing synthesis systems  either completely rely on deductive logic techniques that are extensively hand-  engineered or on purely statistical models that need massive amounts of data, and in  general fail to provide real-time synthesis on challenging benchmarks. In this work,  we propose Neural Guided Deductive Search (NGDS), a hybrid synthesis technique  that combines the best of both symbolic logic techniques and statistical models.  Thus, it produces programs that satisfy the provided specifications by construction  and generalize well on unseen examples, similar to data-driven systems. Our  technique effectively utilizes the deductive search framework to reduce the learning  problem of the neural component to a simple supervised learning setup. Further,  this allows us to both train on sparingly available real-world data and still leverage  powerful recurrent neural network encoders. We demonstrate the effectiveness  of our method by evaluating on real-world customer scenarios by synthesizing  accurate programs with up to 12× speed-up compared to state-of-the-art systems.

我们整合了符号（演绎）和统计（基于神经）方法，以实现实时程序合成，并从1个输入输出的例子中得到几乎完美的泛化。

本文提出了一种分支和边界方法来学习好的程序，其中LSTM被用来预测搜索树中的哪些分支应该导致好的程序。

提出了从一个例子中合成程序的系统，其泛化优于先前的最先进技术。


Variational auto-encoders  (VAEs) offer a tractable approach when performing approximate inference in otherwise intractable generative models. However, standard VAEs often produce latent codes that are disperse and lack interpretability, thus making the resulting representations unsuitable for auxiliary tasks (e.g. classiﬁcation) and human interpretation. We address these issues by merging ideas from variational auto-encoders and sparse coding, and propose to explicitly model sparsity in the latent space of a VAE with a Spike and Slab prior distribution. We derive the evidence lower bound using a discrete mixture recognition function thereby making approximate posterior inference as computational efﬁcient as in the standard VAE case. With the new approach, we are able to infer truly sparse representations with generally intractable non-linear probabilistic models. We show that these sparse representations are advantageous over standard VAE representations on two benchmark classiﬁcation tasks (MNIST and Fashion-MNIST) by demonstrating improved classiﬁcation accuracy and signiﬁcantly increased robustness to the number of latent dimensions. Furthermore, we demonstrate qualitatively that the sparse elements capture subjectively understandable sources of variation.

我们探索VAE和稀疏编码的交叉点。 本文提出了一个具有稀疏先验和后验的VAEs的扩展，以学习稀疏的可解释表征。


A widely observed phenomenon in deep learning is the degradation problem: increasing  the depth of a network leads to a decrease in performance on both test and training data. Novel architectures such as ResNets and Highway networks have addressed this issue by introducing various flavors of skip-connections or gating mechanisms. However, the degradation problem persists in the context of plain feed-forward networks. In this work we propose a simple method to address this issue. The proposed method poses the learning of weights in deep networks as a constrained optimization problem where the presence of skip-connections is penalized by Lagrange multipliers. This allows for skip-connections to be introduced during the early stages of training and subsequently phased out in a principled manner. We demonstrate the benefits of such an approach with experiments on MNIST, fashion-MNIST, CIFAR-10 and CIFAR-100 where the proposed method is shown to greatly decrease the degradation effect (compared to plain networks) and is often competitive with ResNets.

以一种有原则的方式逐步淘汰跳过连接，避免了深度前馈网络的退化。

作者提出了一种新的训练策略，即VAN，用于训练没有跳过连接的非常深的前馈网络。

本文介绍了一种在ResNets和vanilla深网之间线性插值的架构，而无需跳过连接。


Deep learning is becoming more widespread in its application due to its power in solving complex classification problems. However, deep learning models often require large memory and energy consumption, which may prevent them from being deployed effectively on embedded platforms, limiting their applications. This work addresses the problem by proposing methods {\em Weight Reduction Quantisation} for compressing the memory footprint of the models, including reducing the number of weights and the number of bits to store each weight. Beside, applying with sparsity-inducing regularization, our work focuses on speeding up stochastic variance reduced gradients (SVRG) optimization on non-convex problem. Our method that mini-batch SVRG with $\ell$1 regularization on non-convex problem has faster and smoother convergence rates than SGD by using adaptive learning rates. Experimental evaluation of our approach uses MNIST and CIFAR-10 datasets on LeNet-300-100 and LeNet-5 models, showing our approach can reduce the memory requirements both in the convolutional and fully connected layers by up to 60$\times$ without affecting their test accuracy.

部署在嵌入式设备上的深度神经网络的压缩。 作者提出了一种基于l-1正则化的SVRG训练算法，能够强迫网络的许多权重为0。 这项工作降低了对内存的要求。


It has been argued that the brain is a prediction machine that continuously learns how to make better predictions about the stimuli received from the external environment. For this purpose, it builds a model of the world around us and uses this model to infer the external stimulus. Predictive coding has been proposed as a mechanism through which the brain might be able to build such a model of the external environment. However, it is not clear how predictive coding can be used to build deep neural network models of the brain while complying with the architectural constraints imposed by the brain. In this paper, we describe an algorithm to build a deep generative model using predictive coding that can be used to infer latent representations about the stimuli received from external environment. Specifically, we used predictive coding to train a deep neural network on real-world images in a unsupervised learning paradigm. To understand the capacity of the network with regards to modeling the external environment, we studied the latent representations generated by the model on images of objects that are never presented to the model during training. Despite the novel features of these objects the model is able to infer the latent representations for them. Furthermore, the reconstructions of the original images obtained from these latent representations preserve the important details of these objects.

一种基于预测编码的学习算法，用于构建大脑的深度神经网络模型 本文考虑了使用预测编码设置的生成性神经网络的学习。


In this paper, we propose deep convolutional generative adversarial networks (DCGAN) that learn to produce a 'mental image' of the input image as internal representation of a certain category of input data distribution.   This mental image is what the DCGAN 'imagines' that the input image might look like under ideal conditions.   The mental image contains a version of the input that is iconic, without any peculiarities that do not contribute to the ideal representation of the input data distribution within a category. A DCGAN learns this association by training an encoder to capture salient features from the original image and a decoder to convert salient features into its associated mental image representation.   Our new approach, which we refer to as a Mental Image DCGAN (MIDCGAN), learns features that are useful for recognizing entire classes of objects, and that this in turn has the benefit of helping single and zero shot recognition.   We demonstrate our approach on object instance recognition and handwritten digit recognition tasks.

用对抗性自动编码器进行对象实例识别时，使用了一个新的 "心理图像 "目标，它是输入图像的典范表征。

本文提出了一种学习物体识别特征的方法，该方法对物体的各种变换是不变的，最主要的是物体的姿势。

本文研究了在给定输入图像的情况下，通过生成的心理图像作为中间表征的少样本识别任务。


An obstacle that prevents the wide adoption of (deep) reinforcement learning (RL) in control systems is its need for a large number of interactions with the environment in order to master a skill. The learned skill usually generalizes poorly across domains and re-training is often necessary when presented with a new task. We present a framework that combines techniques in \textit{formal methods} with \textit{hierarchical reinforcement learning} (HRL). The set of techniques we provide allows for the convenient specification of tasks with logical expressions, learns hierarchical policies (meta-controller and low-level controllers) with well-defined intrinsic rewards using any RL methods and is able to construct new skills from existing ones without additional learning. We evaluate the proposed methods in a simple grid world simulation as well as simulation on a Baxter robot.

将时间逻辑与分层强化学习结合起来进行技能组合 本文提供了一种从原始MDP和与LTL公式相关的自动机中构建积MDP的策略。 提议将时间逻辑与分层强化学习结合起来，以简化技能构成。


The tremendous memory and computational complexity of Convolutional Neural Networks (CNNs) prevents the inference deployment on resource-constrained systems. As a result, recent research focused on CNN optimization techniques, in particular quantization, which allows weights and activations of layers to be represented with just a few bits while achieving impressive prediction performance. However, aggressive quantization techniques still fail to achieve full-precision prediction performance on state-of-the-art CNN architectures on large-scale classification tasks. In this work we propose a method for weight and activation quantization that is scalable in terms of quantization levels (n-ary representations) and easy to compute while maintaining the performance close to full-precision CNNs. Our weight quantization scheme is based on trainable scaling factors and a nested-means clustering strategy which is robust to weight updates and therefore exhibits good convergence properties. The flexibility of nested-means clustering enables exploration of various n-ary weight representations with the potential of high parameter compression. For activations, we propose a linear quantization strategy that takes the statistical properties of batch normalization into account. We demonstrate the effectiveness of our approach using state-of-the-art models on ImageNet.

我们为深度神经网络的权重和激活提出了一个量化方案。这大大减少了内存占用并加速了推理。

CNN模型压缩和使用量化的推理加速。


Reinforcement learning (RL) agents optimize only the features specified in a reward function and are indifferent to anything left out inadvertently. This means that we must not only specify what to do, but also the much larger space of what not to do. It is easy to forget these preferences, since these preferences are already satisfied in our environment. This motivates our key insight: when a robot is deployed in an environment that humans act in, the state of the environment is already optimized for what humans want. We can therefore use this implicit preference information from the state to fill in the blanks. We develop an algorithm based on Maximum Causal Entropy IRL and use it to evaluate the idea in a suite of proof-of-concept environments designed to show its properties. We find that information from the initial state can be used to infer both side effects that should be avoided as well as preferences for how the environment should be organized. Our code can be found at https://github.com/HumanCompatibleAI/rlsp.

当机器人被部署在一个人类一直在行动的环境中时，环境的状态已经被优化为人类想要的东西，我们可以利用这一点来推断出人类的偏好。

作者建议用从初始状态和状态动态模型推断出的辅助奖励/成本来增强RL代理的明确说明的奖励函数。

这项工作提出了一种利用IRL推断初始状态下的隐含信息，并将推断出的奖励与指定的奖励相结合的方法。


Regularization is one of the crucial ingredients of deep learning, yet the term regularization has various definitions, and regularization methods are often studied separately from each other. In our work we present a novel, systematic, unifying taxonomy to categorize existing methods. We distinguish methods that affect data, network architectures, error terms, regularization terms, and optimization procedures. We identify the atomic building blocks of existing methods, and decouple the assumptions they enforce from the mathematical tools they rely on. We do not provide all details about the listed methods; instead, we present an overview of how the methods can be sorted into meaningful categories and sub-categories. This helps revealing links and fundamental similarities between them. Finally, we include practical recommendations both for users and for developers of new regularization methods.

对深度学习的正则化方法进行系统分类，揭示其相似性。 试图为深度学习中采用的正则化技术建立一个分类法。


Deep neural networks are surprisingly efficient at solving practical tasks,  but the theory behind this phenomenon is only starting to catch up with  the practice. Numerous works show that depth is the key to this efficiency.  A certain class of deep convolutional networks – namely those that correspond  to the Hierarchical Tucker (HT) tensor decomposition – has been  proven to have exponentially higher expressive power than shallow networks.  I.e. a shallow network of exponential width is required to realize  the same score function as computed by the deep architecture. In this paper,  we prove the expressive power theorem (an exponential lower bound on  the width of the equivalent shallow network) for a class of recurrent neural  networks – ones that correspond to the Tensor Train (TT) decomposition.  This means that even processing an image patch by patch with an RNN  can be exponentially more efficient than a (shallow) convolutional network  with one hidden layer. Using theoretical results on the relation between  the tensor decompositions we compare expressive powers of the HT- and  TT-Networks. We also implement the recurrent TT-Networks and provide  numerical evidence of their expressivity.

我们证明了循环型神经网络超越浅层网络的指数效率。 作者比较了张量训练网络与通过CP分解结构的网络的复杂性


Probabilistic modelling is a principled framework to perform model aggregation, which has been a primary mechanism to combat mode collapse in the context of Generative Adversarial Networks (GAN). In this paper, we propose a novel probabilistic framework for GANs, ProbGAN, which iteratively learns a distribution over generators with a carefully crafted prior. Learning is efficiently triggered by a tailored stochastic gradient Hamiltonian Monte Carlo with a novel gradient approximation to perform Bayesian inference. Our theoretical analysis further reveals that our treatment is the first probabilistic framework that yields an equilibrium where generator distributions are faithful to the data distribution. Empirical evidence on synthetic high-dimensional multi-modal data and image databases (CIFAR-10, STL-10, and ImageNet) demonstrates the superiority of our method over both start-of-the-art multi-generator GANs and other probabilistic treatment for GANs.

具有理论保证的GAN的新型概率处理方法。

本文提出了一种贝叶斯GAN，它在理论上保证了对真实分布的收敛性，并把似然放在生成器和判别器上，其对数与传统GAN目标函数成正比。


In the adversarial-perturbation problem of neural networks, an adversary starts with a neural network model $F$ and a point $\bfx$ that $F$ classifies correctly, and applies a \emph{small perturbation} to  $\bfx$ to produce another point $\bfx'$ that $F$ classifies \emph{incorrectly}.  In this paper, we propose taking into account \emph{the inherent confidence information} produced by models when studying adversarial perturbations, where a natural measure of ``confidence'' is \|F(\bfx)\|_\infty$ (i.e. how confident $F$ is about its prediction?) . Motivated by a thought experiment based on the manifold assumption, we propose a ``goodness property'' of models which states that \emph{confident regions of a good model should be well separated}. We give formalizations of this property and examine existing robust training objectives in view of them. Interestingly, we find that a recent objective by Madry et al. encourages training a model that satisfies well our formal version of the goodness property, but has a weak control of points that are wrong but with low confidence. However, if Madry et al.'s model is indeed a good solution to their objective, then good and bad points are now distinguishable and we can try to embed uncertain points back to the closest confident region to get (hopefully) correct predictions. We thus propose embedding objectives and algorithms, and perform an empirical study using this method. Our experimental results are encouraging: Madry et al.'s model wrapped with our embedding procedure achieves almost perfect success rate in defending against attacks that the base model fails on, while retaining good generalization behavior.

抵御来自manifold假设的神经网络的对抗性扰动

该手稿提出了两个基于manifold假设的目标函数，作为对对抗性例子的防御机制。

防御基于自然数据的manifold假设的对抗性攻击


Recently Neural Architecture Search (NAS) has aroused great interest in both academia and industry, however it remains challenging because of its huge and non-continuous search space. Instead of applying evolutionary algorithm or reinforcement learning as previous works, this paper proposes a Direct Sparse Optimization NAS (DSO-NAS) method. In DSO-NAS, we provide a novel model pruning view to NAS problem. In specific, we start from a completely connected block, and then introduce scaling factors to scale the information flow between operations. Next, we impose sparse regularizations to prune useless connections in the architecture. Lastly, we derive an efficient and theoretically sound optimization method to solve it. Our method enjoys both advantages of differentiability and efficiency, therefore can be directly applied to large datasets like ImageNet. Particularly, On CIFAR-10 dataset, DSO-NAS achieves an average test error 2.84%, while on the ImageNet dataset DSO-NAS achieves 25.4% test error under 600M FLOPs with 8 GPUs in 18 hours.

通过直接稀疏优化进行单次神经结构搜索 提出了一种架构搜索方法，其中的连接是用稀疏的正则化来消除的。 本文提出了直接稀疏优化，这是一种以合理的计算成本获得特定问题的神经架构的方法。 本文提出了一种基于直接稀疏优化的神经结构搜索方法


Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning. One aspect of the field receiving considerable attention is efficiently executing deep models in resource-constrained environments, such as mobile or embedded devices. This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger teacher networks into smaller student networks. The first method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher, into the training of a student network whose weights are quantized to a limited set of levels. The second method,  differentiable quantization, optimizes the location of quantization points through stochastic gradient descent, to better fit the behavior of the teacher model.   We validate both methods through experiments on convolutional and recurrent architectures. We show that quantized shallow students can reach similar accuracy levels to full-precision teacher models, while providing order of magnitude compression, and inference speedup that is linear in the depth reduction. In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.

通过利用蒸馏技术，获得了最先进的量化、浅层网的准确性。 通过结合蒸馏和量化为视觉和神经机器翻译实验提出了小型和低成本的模型 本文在模型压缩的背景下，提出了一个利用教师模型来帮助深度学习模型压缩的框架。


Previous work has demonstrated the benefits of incorporating additional linguistic annotations such as syntactic trees into neural machine translation. However the cost of obtaining those syntactic annotations is expensive for many languages and the quality of unsupervised learning linguistic structures is too poor to be helpful. In this work, we aim to improve neural machine translation via source side dependency syntax but without explicit annotation. We propose a set of models that learn to induce dependency trees on the source side and learn to use that information on the target side. Importantly, we also show that our dependency trees capture important syntactic features of language and improve translation quality on two language pairs En-De and En-Ru.

用潜在的树来改进NMT 本文介绍了一种为神经机器翻译服务的诱导源端依赖结构的方法。


Model-free reinforcement learning (RL) requires a large number of trials to learn a good policy, especially in environments with sparse rewards. We explore a method to improve the sample efficiency when we have access to demonstrations. Our approach, Backplay, uses a single demonstration to construct a curriculum for a given task. Rather than starting each training episode in the environment's fixed initial state, we start the agent near the end of the demonstration and move the starting point backwards during the course of training until we reach the initial state. Our contributions are that we analytically characterize the types of environments where Backplay can improve training speed, demonstrate the effectiveness of Backplay both in large grid worlds and a complex four player zero-sum game (Pommerman), and show that Backplay compares favorably to other competitive methods known to improve sample efficiency. This includes reward shaping, behavioral cloning, and reverse curriculum generation.

通过从一个单一的示范，甚至是一个低效的示范向后学习，并逐步让代理自己做更多的解决方法。

本文提出了一种通过专家论证的后向课程提高稀疏奖励RL方法效率的方法。

本文提出了一种用RL解决稀疏奖励任务的策略，即从演示中抽取初始状态。


Episodic memory is a psychology term which refers to the ability to recall specific events from the past. We suggest one advantage of this particular type of memory is the ability to easily assign credit to a specific state when remembered information is found to be useful. Inspired by this idea, and the increasing popularity of external memory mechanisms to handle long-term dependencies in deep learning systems, we propose a novel algorithm which uses a reservoir sampling procedure to maintain an external memory consisting of a fixed number of past states. The algorithm allows a deep reinforcement learning agent to learn online to preferentially remember those states which are found to be useful to recall later on. Critically this method allows for efficient online computation of gradient estimates with respect to the write process of the external memory. Thus unlike most prior mechanisms for external memory it is feasible to use in an online reinforcement learning setting.

基于估计梯度的在线强化学习的外部记忆在一个新的水库采样技术上。

本文提出了一种改良的RL方法，即由代理保留一个额外的 "事件记忆"，并使用一个基于当前状态的 "查询网络"。


We achieve bias-variance decomposition for Boltzmann machines using an information geometric formulation. Our decomposition leads to an interesting phenomenon that the variance does not necessarily increase when more parameters are included in Boltzmann machines, while the bias always decreases. Our result gives a theoretical evidence of the generalization ability of deep learning architectures because it provides the possibility of increasing the representation power with avoiding the variance inflation.

我们利用信息几何学的公式实现了对玻尔兹曼机的偏差-方差分解。

本文的目标是通过对分级模型，特别是玻尔兹曼机的偏差-方差分解进行理论分析，来分析深度学习的有效性和可推广性。

本文得出的主要结论是，在分层模型中，有可能同时减少偏差和方差。


Recurrent Neural Networks (RNNs) are powerful tools for solving sequence-based problems, but their efficacy and execution time are dependent on the size of the network.   Following recent work in simplifying these networks with model pruning and a novel mapping of work onto GPUs, we design an efficient implementation for sparse RNNs.   We investigate several optimizations and tradeoffs: Lamport timestamps, wide memory loads, and a bank-aware weight layout.   With these optimizations, we achieve speedups of over 6x over the next best algorithm for a hidden layer of size 2304, batch size of 4, and a density of 30%.   Further, our technique allows for models of over 5x the size to fit on a GPU for a speedup of 2x, enabling larger networks to help advance the state-of-the-art.   We perform case studies on NMT and speech recognition tasks in the appendix, accelerating their recurrent layers by up to 3x.

将网络修剪和持久性内核结合起来，形成一个实用、快速、准确的网络实现。

本文介绍了稀疏持久性 RNN，这是一种在芯片上存储 RNN 权重的现有工作中添加修剪的机制。


Weight pruning has proven to be an effective method in reducing the model size and computation cost while not sacrificing the model accuracy. Conventional sparse matrix formats, however, involve irregular index structures with large storage requirement and sequential reconstruction process, resulting in inefficient use of highly parallel computing resources. Hence, pruning is usually restricted to inference with a batch size of one, for which an efficient parallel matrix-vector multiplication method exists. In this paper, a new class of sparse matrix representation utilizing Viterbi algorithm that has a high, and more importantly, fixed index compression ratio regardless of the pruning rate, is proposed. In this approach, numerous sparse matrix candidates are first generated by the Viterbi encoder, and then the one that aims to minimize the model accuracy degradation is selected by the Viterbi algorithm. The model pruning process based on the proposed Viterbi encoder and Viterbi algorithm is highly parallelizable, and can be implemented efficiently in hardware to achieve low-energy, high-performance index decoding process. Compared with the existing magnitude-based pruning methods, index data storage requirement can be further compressed by 85.2% in MNIST and 83.9% in AlexNet while achieving similar pruning rate. Even compared with the relative index compression technique, our method can still reduce the index storage requirement by 52.7% in MNIST and 35.5% in AlexNet.

我们提出了一种新的剪枝方法和稀疏矩阵格式，以实现高索引压缩率和并行索引解码过程。

作者使用Viterbi编码极大地压缩了修剪过的网络的稀疏矩阵索引，减少了主要的内存开销之一，并加快了并行设置中的推理速度。


Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL). It is also a requirement for its deployment in real-world scenarios. This paper proposes a novel framework for efficient multi-task reinforcement learning. Our framework trains agents to employ hierarchical policies that decide when to use a previously learned policy and when to learn a new skill. This enables agents to continually acquire new skills during different stages of training. Each learned task corresponds to a human language description. Because agents can only access previously learned skills through these descriptions, the agent can always provide a human-interpretable description of its choices. In order to help the agent learn the complex temporal dependencies necessary for the hierarchical policy, we provide it with a stochastic temporal grammar that modulates when to rely on previously learned skills and when to execute new skills. We validate our approach on Minecraft games designed to explicitly test the ability to reuse previously learned skills while simultaneously learning new skills.

一个新颖的分层政策网络，它可以通过发现技能之间的潜在关系，将以前学到的技能与新技能一起重用并作为新技能的子组件。

本文旨在通过使用由随机时间语法调节的递归政策结构来学习分层策略。

本文提出了一种在终身学习背景下学习分层策略的方法，通过堆叠策略，然后使用明确的 "切换 "策略。


Embeddings are a fundamental component of many modern machine learning and natural language processing models.  Understanding them and visualizing them is essential for gathering insights about the information they capture and the behavior of the models.  State of the art in analyzing embeddings consists in projecting them in two-dimensional planes without any interpretable semantics associated to the axes of the projection, which makes detailed analyses and comparison among multiple sets of embeddings challenging.  In this work, we propose to use explicit axes defined as algebraic formulae over embeddings to project them into a lower dimensional, but semantically meaningful subspace, as a simple yet effective analysis and visualization methodology.  This methodology assigns an interpretable semantics to the measures of variability and the axes of visualizations, allowing for both comparisons among different sets of embeddings and fine-grained inspection of the embedding spaces.  We demonstrate the power of the proposed methodology through a series of case studies that make use of visualizations constructed around the underlying methodology and through a user study. The results show how the methodology is effective at providing more profound insights than classical projection methods and how it is widely applicable to many other use cases.

我们提出使用明确的向量代数公式投影作为可视化嵌入空间的另一种方式，专门为面向目标的分析任务定制，在我们的用户研究中，它优于t-SNE。

以非参数的方式分析嵌入空间（基于实例的_方式


Learning deep networks which can resist large variations between training andtesting data is essential to build accurate and robust image classifiers.   Towardsthis end, a typical strategy is to apply data augmentation to enlarge the trainingset.    However,  standard  data  augmentation  is  essentially  a  brute-force  strategywhich is inefficient,  as it performs all the pre-defined transformations  to everytraining sample. In this paper, we propose a principled approach to train networkswith  significantly  improved  resistance  to  large  variations  between  training  andtesting data.   This is achieved by embedding a learnable transformation moduleinto the introspective networks (Jin et al., 2017; Lazarow et al., 2017; Lee et al.,2018), which is a convolutional neural network (CNN) classifier empowered withgenerative capabilities.   Our approach alternatively synthesizes pseudo-negativesamples with learned transformations and enhances the classifier by retraining itwith synthesized samples.   Experimental results verify that our approach signif-icantly improves the ability of deep networks to resist large variations betweentraining and testing data and achieves classification accuracy improvements onseveral benchmark datasets, including MNIST, affNIST, SVHN and CIFAR-10.

我们提出了一种原则性的方法，赋予分类器以智能和高效的方式抵御训练和测试数据之间较大变化的能力。

使用内省学习来处理测试时的数据变化

本文建议使用学习到的转换网络，嵌入到内省网络中，以提高综合实例的分类性能。


It is well known that it is possible to construct "adversarial examples"  for neural networks: inputs which are misclassified by the network  yet indistinguishable from true data. We propose a simple  modification to standard neural network architectures, thermometer  encoding, which significantly increases the robustness of the network to  adversarial examples. We demonstrate this robustness with experiments  on the MNIST, CIFAR-10, CIFAR-100, and SVHN datasets, and show that  models with thermometer-encoded inputs consistently have higher accuracy  on adversarial examples, without decreasing generalization.  State-of-the-art accuracy under the strongest known white-box attack was   increased from 93.20% to 94.30% on MNIST and 50.00% to 79.16% on CIFAR-10.  We explore the properties of these networks, providing evidence  that thermometer encodings help neural networks to  find more-non-linear decision boundaries.

输入离散化带来了对对抗性例子的稳健性 作者介绍了对输入离散化/量化的深入研究，作为对对抗性例子的一种防御。


Low-precision training is a promising way of decreasing the time and energy cost of training machine learning models.  Previous work has analyzed low-precision training algorithms, such as low-precision stochastic gradient descent, and derived theoretical bounds on their convergence rates.  These bounds tend to depend on the dimension of the model $d$ in that the number of bits needed to achieve a particular error bound increases as $d$ increases.  This is undesirable because a motivating application for low-precision training is large-scale models, such as deep learning, where $d$ can be huge.  In this paper, we prove dimension-independent bounds for low-precision training algorithms that use fixed-point arithmetic, which lets us better understand what affects the convergence of these algorithms as parameters scale.  Our methods also generalize naturally to let us prove new convergence bounds on low-precision training with other quantization schemes, such as low-precision floating-point computation and logarithmic quantization.

我们证明了低精度训练算法的维度无关边界 本文讨论了具有低精度权重的训练模型的收敛性不依赖于模型维度的条件。


We consider the problem of exploration in meta reinforcement learning. Two new meta reinforcement learning algorithms are suggested: E-MAML and ERL2. Results are presented on a novel environment we call 'Krazy World'  and a set of maze environments. We show E-MAML and ERL2 deliver better performance on tasks where exploration is important.

对MAML和RL2的修改，应该允许更好的探索。 本文提出了在最近的两种元RL算法的基础上，扩展目标函数以驱动元RL的探索的技巧


We propose a new class of probabilistic neural-symbolic models for visual question answering (VQA) that provide interpretable explanations of their decision making in the form of programs, given a small annotated set of human programs. The key idea of our approach is to learn a rich latent space which effectively propagates program annotations from known questions to novel questions. We do this by formalizing prior work on VQA, called module networks (Andreas, 2016) as discrete, structured, latent variable models on the joint distribution over questions and answers given images, and devise a procedure to train the model effectively. Our results on a dataset of compositional questions about SHAPES (Andreas, 2016) show that our model generates more interpretable programs and obtains better accuracy on VQA in the low-data regime than prior work.

一个具有潜在程序空间的概率神经符号模型，用于更多的可解释问题的回答

本文提出了一种用于视觉问答的离散的、结构化的潜变量模型，该模型涉及组合式的泛化和推理，在性能和能力方面有明显的增益。


The ability to deploy neural networks in real-world, safety-critical systems is severely limited by the presence of adversarial examples: slightly perturbed inputs that are misclassified by the network. In recent years, several techniques have been proposed for training networks that are robust to such examples; and each time stronger attacks have been devised, demonstrating the shortcomings of existing defenses. This highlights a key difficulty in designing an effective defense: the inability to assess a network's robustness against future attacks. We propose to address this difficulty through formal verification techniques. We construct ground truths: adversarial examples with a provably-minimal distance from a given input point. We demonstrate how ground truths can serve to assess the effectiveness of attack techniques, by comparing the adversarial examples produced by those attacks to the ground truths; and also of defense techniques, by computing the distance to the ground truths before and after the defense is applied, and measuring the improvement. We use this technique to assess recently suggested attack and defense techniques.

我们使用形式验证来评估寻找对抗性例子或防御对抗性例子的技术的有效性。

本文提出了一种计算与原始输入最小距离的对抗性例子的方法。

作者提议采用可证明的最小距离的例子作为工具来评估训练过的网络的稳健性。

本文介绍了一种生成对抗性例子的方法，这些例子与用于生成对抗性例子的训练例子的距离最小。


This paper introduces a new framework for open-domain question answering in which the retriever and the reader \emph{iteratively interact} with each other. The framework is agnostic to the architecture of the machine reading model provided it has \emph{access} to the token-level hidden representations of the reader. The retriever uses fast nearest neighbor search that allows it to scale to corpora containing millions of paragraphs. A gated recurrent unit updates the query at each step conditioned on the \emph{state} of the reader and the \emph{reformulated} query is used to re-rank the paragraphs by the retriever. We conduct analysis and show that iterative interaction helps in retrieving informative paragraphs from the corpus. Finally, we show that our multi-step-reasoning framework brings consistent improvement when applied to two widely used reader architectures (\drqa and \bidaf) on various large open-domain datasets ---\tqau, \quasart, \searchqa, and \squado\footnote{Code and pretrained models are available at \url{https://github.com/rajarshd/Multi-Step-Reasoning}}.

段落检索器和机器阅读器通过强化学习相互作用，在开放领域的数据集上产生巨大的改进。

本文介绍了一个文档检索器和读者之间双向互动的新框架，用于开放领域的问题回答，并提出了从读者到检索器的 "读者状态 "概念。

本文提出了一个由3个不同部分和一个算法组成的多文档抽取式机器阅读模型。


Many imaging tasks require global information about all pixels in an image. Conventional bottom-up classification networks globalize information by decreasing resolution; features are pooled and down-sampled into a single output. But for semantic segmentation and object detection tasks, a network must provide higher-resolution pixel-level outputs. To globalize information while preserving resolution, many researchers propose the inclusion of sophisticated auxiliary blocks, but these come at the cost of a considerable increase in network size and computational cost. This paper proposes stacked u-nets (SUNets), which iteratively combine features from different resolution scales while maintaining resolution. SUNets leverage the information globalization power of u-nets in a deeper net- work architectures that is capable of handling the complexity of natural images. SUNets perform extremely well on semantic segmentation tasks using a small number of parameters.

提出了新的架构，在更深的网络中利用u-net的信息全球化能力，并在各种任务中表现良好，没有任何花哨的东西。

一种用于语义图像分割的网络架构，基于对基本的U-Net架构的组合，减少了参数数量并改善了结果。

这提出了一个用于图像分割的堆叠式U-Net架构。


Asking questions is an important ability for a chatbot. This paper focuses on question generation. Although there are existing works on question generation based on a piece of descriptive text, it remains to be a very challenging problem. In the paper, we propose a new question generation problem, which also requires the input of a target topic in addition to a piece of descriptive text. The key reason for proposing the new problem is that in practical applications, we found that useful questions need to be targeted toward some relevant topics. One almost never asks a random question in a conversation. Due to the fact that given a descriptive text, it is often possible to ask many types of questions, generating a question without knowing what it is about is of limited use. To solve the problem, we propose a novel neural network that is able to generate topic-specific questions. One major advantage of this model is that it can be trained directly using a question-answering corpus without requiring any additional annotations like annotating topics in the questions or answers. Experimental results show that our model outperforms the state-of-the-art baseline.

我们提出了一个能够生成特定主题问题的神经网络。

提出了一种基于神经网络的方法来生成特定主题的问题，其动机是主题问题在实际应用中更有意义。

提出了一种基于话题的生成方法，使用LSTM的两阶段编码技术来提取话题


Brain-Machine Interfaces (BMIs) have recently emerged as a clinically viable option  to restore voluntary movements after paralysis. These devices are based on the  ability to extract information about movement intent from neural signals recorded  using multi-electrode arrays chronically implanted in the motor cortices of the  brain. However, the inherent loss and turnover of recorded neurons requires repeated  recalibrations of the interface, which can potentially alter the day-to-day  user experience. The resulting need for continued user adaptation interferes with  the natural, subconscious use of the BMI. Here, we introduce a new computational  approach that decodes movement intent from a low-dimensional latent representation  of the neural data. We implement various domain adaptation methods  to stabilize the interface over significantly long times. This includes Canonical  Correlation Analysis used to align the latent variables across days; this method  requires prior point-to-point correspondence of the time series across domains.  Alternatively, we match the empirical probability distributions of the latent variables  across days through the minimization of their Kullback-Leibler divergence.  These two methods provide a significant and comparable improvement in the performance  of the interface. However, implementation of an Adversarial Domain  Adaptation Network trained to match the empirical probability distribution of the  residuals of the reconstructed neural signals outperforms the two methods based  on latent variables, while requiring remarkably few data points to solve the domain  adaptation problem.

我们实现了一个对抗性领域适应网络，以稳定一个固定的脑机接口，对抗记录的神经信号的逐渐变化。

描述了一种植入式脑机接口的新方法，以解决校准问题和协变量转移。

作者定义了一个使用自动编码器的BMI，然后解决BMI中的数据漂移问题。


Neural network-based systems can now learn to locate the referents of words and phrases in images, answer questions about visual scenes, and even execute symbolic instructions as first-person actors in partially-observable worlds. To achieve this so-called grounded language learning, models must overcome certain well-studied learning challenges that are also fundamental to infants learning their first words. While it is notable that models with no meaningful prior knowledge overcome these learning obstacles, AI researchers and practitioners currently lack a clear understanding of exactly how they do so. Here we address this question as a way of achieving a clearer general understanding of grounded language learning, both to inform future research and to improve confidence in model predictions. For maximum control and generality, we focus on a simple neural network-based language learning agent trained via policy-gradient methods to interpret synthetic linguistic instructions in a simulated 3D world. We apply experimental paradigms from developmental psychology to this agent, exploring the conditions under which established human biases and learning effects emerge. We further propose a novel way to visualise and analyse semantic representation in grounded language learning agents that yields a plausible computational account of the observed effects.

分析和理解神经网络代理如何学习理解简单的基础语言

作者将心理学实验方法与理解深度学习方法的黑箱如何解决问题联系起来。

本文介绍了在一个简单的环境中通过强化学习来学习基础语言的代理的分析，该环境将口头指示与视觉信息相结合


Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future. In this paper, we propose a novel formulation that simultaneously learns a hierarchical, disentangled object representation and a dynamics model for object parts from unlabeled videos. Our Parts, Structure, and Dynamics (PSD) model learns to, first, recognize the object parts via a layered image representation; second, predict hierarchy via a structural descriptor that composes low-level concepts into a hierarchical structure; and third, model the system dynamics by predicting the future. Experiments on multiple real and synthetic datasets demonstrate that our PSD model works well on all three tasks: segmenting object parts, building their hierarchical structure, and capturing their motion distributions.

通过观察物体的运动方式来学习物体的部件、层次结构和动态变化

提出了一个无监督学习模型，该模型可以学习将物体分解成若干部分，预测各部分的层次结构，并根据分解的部分和层次结构来预测运动。


A successful application of convolutional architectures is to increase the resolution of single low-resolution images -- a image restoration task called super-resolution (SR). Naturally, SR is of value to resource constrained devices like mobile phones, electronic photograph frames and televisions to enhance image quality. However, SR demands perhaps the most extreme amounts of memory and compute operations of any mainstream vision task known today, preventing SR from being deployed to devices that require them. In this paper, we perform a early systematic study of system resource efficiency for SR, within the context of a variety of architectural and low-precision approaches originally developed for discriminative neural networks. We present a rich set of insights, representative SR architectures, and efficiency trade-offs; for example, the prioritization of ways to compress models to reach a specific memory and computation target and techniques to compact SR models so that they are suitable for DSPs and FPGAs. As a result of doing so, we manage to achieve better and comparable performance with previous models in the existing literature, highlighting the practicality of using existing efficiency techniques in SR tasks. Collectively, we believe these results provides the foundation for further research into the little explored area of resource efficiency for SR.

我们在超级分辨率上建立了对资源节约型技术的理解 本文对各种卷积神经网络在超级分辨率问题上取得的权衡提出了详细的实证评估。 本文提出要提高超分辨率网络的系统资源效率。


Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with 100% accuracy. In this work we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we show that deep ReLU networks are biased towards low frequency functions, meaning that they cannot have local fluctuations without affecting their global behavior. Intuitively, this property is in line with the observation that over-parameterized networks find simple patterns that generalize across data samples. We also investigate how the shape of the data manifold affects expressivity by showing evidence that learning high frequencies gets easier with increasing manifold complexity, and present a theoretical understanding of this behavior. Finally, we study the robustness of the frequency components with respect to parameter perturbation, to develop the intuition that the parameters must be finely tuned to express high frequency functions.

我们研究了傅里叶域中的ReLU网络，并展示了奇特的行为。 对ReLU网络进行傅里叶分析，发现它们偏向于学习低频 本文对神经网络的傅里叶系数课题有理论和实证上的贡献。


Instance embeddings are an efficient and versatile image representation that facilitates applications like recognition, verification, retrieval, and clustering. Many metric learning methods represent the input as a single point in the embedding space. Often the distance between points is used as a proxy for match confidence. However, this can fail to represent uncertainty which can arise when the input is ambiguous, e.g., due to occlusion or blurriness. This work addresses this issue and explicitly models the uncertainty by “hedging” the location of each input in the embedding space. We introduce the hedged instance embedding (HIB) in which embeddings are modeled as random variables and the model is trained under the variational information bottleneck principle (Alemi et al., 2016; Achille & Soatto, 2018). Empirical results on our new N-digit MNIST dataset show that our method leads to the desired behavior of “hedging its bets” across the embedding space upon encountering ambiguous inputs. This results in improved performance for image matching and classification tasks, more structure in the learned embedding space, and an ability to compute a per-exemplar uncertainty measure which is correlated with downstream performance.

本文提出在识别和验证等实例嵌入任务中使用概率分布而不是点。 论文提出了当前点嵌入的替代方案和一种训练它们的技术。 本文提出了一个使用不确定嵌入的模型来扩展深度学习在贝叶斯的应用


Convolution neural networks typically consist of many convolutional layers followed by several fully-connected layers.   While convolutional layers map between high-order activation tensors, the fully-connected layers operate on flattened activation vectors.   Despite its success, this approach has notable drawbacks. Flattening discards the multi-dimensional structure of the activations, and the fully-connected layers require a large number of parameters.   We present two new techniques to address these problems.   First, we introduce tensor contraction layers which can replace the ordinary fully-connected layers in a neural network. Second, we introduce tensor regression layers, which express the output of a neural network as a low-rank multi-linear mapping from a high-order activation tensor to the softmax layer.   Both the contraction and regression weights are learned end-to-end by backpropagation. By imposing low rank on both, we use significantly fewer parameters.   Experiments on the ImageNet dataset show that applied to the popular VGG and ResNet architectures, our methods significantly reduce the number of parameters in the fully connected layers (about 65% space savings) while negligibly impacting accuracy.

我们提出了张量收缩和低秩张量回归层，以保留和利用整个网络的多线性结构，从而节省了大量空间，对性能几乎没有影响。

本文提出了使用张量的低秩表示的神经网络的新层架构

本文通过使用一个新的张量回归层，将张量分解和张量回归纳入CNN。


We explore ways of incorporating bilingual dictionaries to enable semi-supervised  neural machine translation. Conventional back-translation methods have shown  success in leveraging target side monolingual data. However, since the quality of  back-translation models is tied to the size of the available parallel corpora, this  could adversely impact the synthetically generated sentences in a low resource  setting. We propose a simple data augmentation technique to address both this  shortcoming. We incorporate widely available bilingual dictionaries that yield  word-by-word translations to generate synthetic sentences. This automatically  expands the vocabulary of the model while maintaining high quality content. Our  method shows an appreciable improvement in performance over strong baselines.

我们使用双语词典为神经机器翻译提供数据增援

本文研究了使用双语词典为目标端单语数据创建合成源，以改善用少量平行数据训练的NMT模型。


Rewards are sparse in the real world and most of today's reinforcement learning algorithms struggle with such sparsity. One solution to this problem is to allow the agent to create rewards for itself - thus making rewards dense and more suitable for learning. In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus. Such bonus is summed up with the real task reward - making it possible for RL algorithms to learn from the combined reward. We propose a new curiosity method which uses episodic memory to form the novelty bonus. To determine the bonus, the current observation is compared with the observations in memory. Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory - which incorporates rich information about environment dynamics. This allows us to overcome the known "couch-potato" issues of prior work - when the agent finds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences. We test our approach in visually rich 3D environments in ViZDoom, DMLab and MuJoCo. In navigational tasks from ViZDoom and DMLab, our agent outperforms the state-of-the-art curiosity method ICM. In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only. The code is available at https://github.com/google-research/episodic-curiosity/.

我们提出了一个基于事件记忆和可及性思想的新颖的好奇心模型，这使我们能够克服先前工作中已知的 "沙发土豆 "问题。

建议在RL算法中给予探索奖励，对环境步骤中的父亲的观察给予更大的奖金。

作者提出了一个旨在帮助解决稀疏奖励RL问题的探索奖励，并考虑在复杂的三维环境中进行许多实验。


We introduce a new dataset of logical entailments for the purpose of measuring models' ability to capture and exploit the structure of logical expressions against an entailment prediction task. We use this task to compare a series of architectures which are ubiquitous in the sequence-processing literature, in addition to a new model class---PossibleWorldNets---which computes entailment as a ``convolution over possible worlds''. Results show that convolutional networks present the wrong inductive bias for this class of problems relative to LSTM RNNs, tree-structured neural networks outperform LSTM RNNs due to their enhanced ability to exploit the syntax of logic, and PossibleWorldNets outperform all benchmarks.

我们引入了一个新的逻辑蕴含数据集，目的是衡量模型捕捉和利用逻辑表达式结构的能力，以完成蕴含预测任务。

本文提出了一个新的模型，使用深度模型来检测作为可能世界上连续函数的乘积的逻辑蕴含。

提出了一个为机器学习设计的新模型，并预测了逻辑上蕴含。


Deep convolutional neural network (DCNN) based supervised learning is a widely practiced approach for large-scale image classification.   However, retraining these large networks to accommodate new, previously unseen data demands high computational time and energy requirements. Also, previously seen training samples may not be available at the time of retraining. We propose an efficient training methodology and incrementally growing a DCNN to allow new classes to be learned while sharing part of the base network. Our proposed methodology is inspired by transfer learning techniques, although it does not forget previously learned classes. An updated network for learning new set of classes is formed using previously learned convolutional layers (shared from initial part of base network) with addition of few newly added convolutional kernels included in the later layers of the network. We evaluated the proposed scheme on several recognition applications. The classification accuracy achieved by our approach is comparable to the regular incremental learning approach (where networks are updated with new training samples only, without any network sharing).

本文是关于增量学习的一种新的节能方法。

提出了作为迁移学习的增量学习的程序。

本文提出了一种以增量方式训练深度卷积神经网络的方法，即在一段时间内以小批量的方式获得数据。

提出了一种利用深度网络进行类增量学习的方法，在最终/最佳方法中提出了三种不同的学习策略。


Recurrent neural networks (RNNs) are widely used to model sequential data but  their non-linear dependencies between sequence elements prevent parallelizing  training over sequence length. We show the training of RNNs with only linear  sequential dependencies can be parallelized over the sequence length using the  parallel scan algorithm, leading to rapid training on long sequences even with  small minibatch size. We develop a parallel linear recurrence CUDA kernel and  show that it can be applied to immediately speed up training and inference of  several state of the art RNN architectures by up to 9x.  We abstract recent work  on linear RNNs into a new framework of linear surrogate RNNs and develop a  linear surrogate model for the long short-term memory unit, the GILR-LSTM, that  utilizes parallel linear recurrence.  We extend sequence learning to new  extremely long sequence regimes that were previously out of reach by  successfully training a GILR-LSTM on a synthetic sequence classification task  with a one million timestep dependency.

使用并行扫描来并行化线性递归神经网络。 在长度为100万的依赖关系上训练模型。

建议通过应用Blelloch的方法来加速RNN。

作者提出了一种线性代理RNN的并行算法，它比现有的准RNN、SRU和LSTM的实现方式产生了加速。


Neural text generation models are often autoregressive language models or seq2seq models. Neural autoregressive and seq2seq models that generate text by sampling words sequentially, with each word conditioned on the previous model, are state-of-the-art for several machine translation and summarization benchmarks. These benchmarks are often defined by validation perplexity even though this is not a direct measure of sample quality. Language models are typically trained via maximum likelihood and most often with teacher forcing. Teacher forcing is well-suited to optimizing perplexity but can result in poor sample quality because generating text requires conditioning on sequences of words that were never observed at training time. We propose to improve sample quality using Generative Adversarial Network (GANs), which explicitly train the generator to produce high quality samples and have shown a lot of success in image generation. GANs were originally to designed to output differentiable values, so discrete language generation is challenging for them. We introduce an actor-critic conditional GAN that fills in missing text conditioned on the surrounding context. We show qualitatively and quantitatively, evidence that this produces more realistic text samples compared to a maximum likelihood trained model.

用于填空的自然语言GAN 本文提出使用GAN来生成文本。 使用GAN和机制生成文本样本，以周围文本为条件填补缺失的单词


Parametric texture models have been applied successfully to synthesize artificial images. Psychophysical studies show that under defined conditions observers are unable to differentiate between model-generated and original natural textures. In industrial applications the reverse case is of interest: a texture analysis system should decide if human observers are able to discriminate between a reference and a novel texture. For example, in case of inspecting decorative surfaces the de- tection of visible texture anomalies without any prior knowledge is required. Here, we implemented a human-vision-inspired novelty detection approach. Assuming that the features used for texture synthesis are important for human texture percep- tion, we compare psychophysical as well as learnt texture representations based on activations of a pretrained CNN in a novelty detection scenario. Additionally, we introduce a novel objective function to train one-class neural networks for novelty detection and compare the results to standard one-class SVM approaches. Our experiments clearly show the differences between human-vision-inspired texture representations and learnt features in detecting visual anomalies. Based on a dig- ital print inspection scenario we show that psychophysical texture representations are able to outperform CNN-encoded features.

心理物理学和CNN编码的纹理表征在单类神经网络新颖性检测应用中的比较。

本文重点讨论了新颖性检测，并表明心理物理表征在这一任务的某些部分可以胜过VGG编码器特征。

本文考虑检测纹理中的异常情况，并提出了原始损失函数。

提出从三个不同的模型中训练两个异常检测器来检测视觉纹理中的感知异常。


In representation learning (RL), how to make the learned representations easy to interpret and less overfitted to training data are two important but challenging issues. To address these problems, we study a new type of regularization approach that encourages the supports of weight vectors in RL models to have small overlap, by simultaneously promoting near-orthogonality among vectors and sparsity of each vector. We apply the proposed regularizer to two models: neural networks (NNs) and sparse coding (SC), and develop an efficient ADMM-based algorithm for regularized SC. Experiments on various datasets demonstrate that weight vectors learned under our regularizer are more interpretable and have better generalization performance.

我们提出了一种新型的正则化方法，鼓励表征学习中的非重叠性，以提高可解释性并减少过拟合。

本文引入了一个矩阵正则器来同时诱导稀疏性和近似正交性。

本文研究了一种正则化方法，以促进稀疏性并减少所学表征中权重向量支持之间的重叠，从而提高可解释性并避免过度拟合。

该论文提出了一种新的正则化方法，同时鼓励权重向量（W）是稀疏的，并且是相互正交的。


Successful recurrent models such as long short-term memories (LSTMs) and gated recurrent units (GRUs) use \emph{ad hoc} gating mechanisms.   Empirically these models have been found to improve the learning of medium to long term temporal dependencies and to help with vanishing gradient issues. 	  We prove that learnable gates in a recurrent model formally provide \emph{quasi-invariance to general time transformations} in the input data. We recover part of the LSTM architecture from a simple axiomatic approach. 	  This result leads to a new way of initializing gate biases in LSTMs and GRUs. Experimentally, this new \emph{chrono initialization} is shown to greatly improve learning of long term dependencies, with minimal implementation effort.

证明了门控机制提供了对时间转换的不变性。从这一观点出发，介绍并测试了LSTM的新初始化。

论文将循环网络的设计及其对网络对时间转换的反应的影响联系起来，并利用这一点开发了一个简单的偏差初始化方案。


Teaching plays a very important role in our society, by spreading human knowledge and educating our next generations. A good teacher will select appropriate teaching materials, impact suitable methodologies, and set up targeted examinations, according to the learning behaviors of the students. In the field of artificial intelligence, however, one has not fully explored the role of teaching, and pays most attention to machine \emph{learning}. In this paper, we argue that equal attention, if not more, should be paid to teaching, and furthermore, an optimization framework (instead of heuristics) should be used to obtain good teaching strategies. We call this approach ``learning to teach''. In the approach, two intelligent agents interact with each other: a student model (which corresponds to the learner in traditional machine learning algorithms), and a teacher model (which determines the appropriate data, loss function, and hypothesis space to facilitate the training of the student model). The teacher model leverages the feedback from the student model to optimize its own teaching strategies by means of reinforcement learning, so as to achieve teacher-student co-evolution. To demonstrate the practical value of our proposed approach, we take the training of deep neural networks (DNN) as an example, and show that by using the learning to teach techniques, we are able to use much less training data and fewer iterations to achieve almost the same accuracy for different kinds of DNN models (e.g., multi-layer perceptron, convolutional neural networks and recurrent neural networks) under various machine learning tasks (e.g., image classification and text understanding).

我们提出并验证了以学定教的有效性，这是一个自动指导机器学习过程的新框架。

本文专注于 "机器教学"，并建议利用强化学习，将奖励定义为学习者的学习速度，并使用策略梯度来更新教师参数。

作者定义了一个由四个部分组成的深度学习模型：学生模型、教师模型、损失函数和数据集。

提出了一个 "学习教学 "的框架，对应于对呈现给学习者的数据的选择。


We present DL2, a system for training and querying neural networks with logical constraints. The key idea is to translate these constraints into a differentiable loss with desirable mathematical properties and to then either train with this loss in an iterative manner or to use the loss for querying the network for inputs subject to the constraints. We empirically demonstrate that DL2 is effective in both training and querying scenarios, across a range of constraints and data sets.

用于训练和查询神经网络的逻辑约束的可微损失。

将神经网络的参数和输入输出对的查询转化为可微的损失函数的框架，以及用于指定这些查询的相关声明性语言

本文通过将逻辑公式转化为神经网络的非负损失函数，解决了逻辑方法与神经网络的结合问题。


Genetic algorithms have been widely used in many practical optimization problems.  Inspired by natural selection, operators, including mutation, crossover  and selection, provide effective heuristics for search and black-box optimization.  However, they have not been shown useful for deep reinforcement learning, possibly  due to the catastrophic consequence of parameter crossovers of neural networks.  Here, we present Genetic Policy Optimization (GPO), a new genetic algorithm  for sample-efficient deep policy optimization. GPO uses imitation learning  for policy crossover in the state space and applies policy gradient methods for mutation.  Our experiments on MuJoCo tasks show that GPO as a genetic algorithm  is able to provide superior performance over the state-of-the-art policy gradient  methods and achieves comparable or higher sample efficiency.

基于遗传算法的深度神经网络策略优化方法

作者提出了一种训练政策网络组合的算法，该算法定期将组合中的不同政策混合在一起。

本文提出了一种受遗传算法启发的策略优化方法，它模拟了策略网络上的突变和交叉运算。


To make deep neural networks feasible in resource-constrained environments (such as mobile devices), it is beneficial to quantize models by using low-precision weights. One common technique for quantizing neural networks is the straight-through gradient method, which enables back-propagation through the quantization mapping. Despite its empirical success, little is understood about why the straight-through gradient method works.  Building upon a novel observation that the straight-through gradient method is in fact identical to the well-known Nesterov’s dual-averaging algorithm on a quantization constrained optimization problem, we propose a more principled alternative approach, called ProxQuant , that formulates quantized network training as a regularized learning problem instead and optimizes it via the prox-gradient method. ProxQuant does back-propagation on the underlying full-precision vector and applies an efficient prox-operator in between stochastic gradient steps to encourage quantizedness. For quantizing ResNets and LSTMs, ProxQuant outperforms state-of-the-art results on binary quantization and is on par with state-of-the-art on multi-bit quantization. We further perform theoretical analyses showing that ProxQuant converges to stationary points under mild smoothness assumptions, whereas variants such as lazy prox-gradient method can fail to converge in the same setting.

使用近端梯度法进行模型量化的原则性框架，并进行了实证评估和理论收敛分析。 提出了ProxQuant方法来训练具有量化权重的神经网络。 提出用近端梯度下降法求解二元网及其变体。


Background: Statistical mechanics results (Dauphin et al. (2014); Choromanska et al. (2015)) suggest that local minima with high error are exponentially rare in high dimensions. However, to prove low error guarantees for Multilayer Neural Networks (MNNs), previous works so far required either a heavily modified MNN model or training method, strong assumptions on the labels (e.g., “near” linear separability), or an unrealistically wide hidden layer with \Omega\(N) units.    Results: We examine a MNN with one hidden layer of piecewise linear units, a single output, and a quadratic loss. We prove that, with high probability in the limit of N\rightarrow\infty datapoints, the volume of differentiable regions of the empiric loss containing sub-optimal differentiable local minima is exponentially vanishing in comparison with the same volume of global minima, given standard normal input of dimension d_0=\tilde{\Omega}(\sqrt{N}), and a more realistic number of d_1=\tilde{\Omega}(N/d_0) hidden units. We demonstrate our results numerically: for example, 0% binary classification training error on CIFAR with only N/d_0 = 16 hidden neurons.

多层神经网中的 "坏 "局部最小值正在消失：用比以前更合理的假设进行证明
在只有一个隐藏层的网络中，与全局最小值相比，次优局部最小值的数量呈指数级下降。
本文旨在回答为什么基于神经网络的标准SGD算法会收敛到 "好 "的解决方案。


Deep neural networks are vulnerable to adversarial examples, which can mislead classifiers by adding imperceptible perturbations. An intriguing property of adversarial examples is their good transferability, making black-box attacks feasible in real-world applications. Due to the threat of adversarial attacks, many methods have been proposed to improve the robustness, and several state-of-the-art defenses are shown to be robust against transferable adversarial examples. In this paper, we identify the attention shift phenomenon, which may hinder the transferability of adversarial examples to the defense models. It indicates that the defenses rely on different discriminative regions to make predictions compared with normally trained models. Therefore, we propose an attention-invariant attack method to generate more transferable adversarial examples. Extensive experiments on the ImageNet dataset validate the effectiveness of the proposed method. Our best attack fools eight state-of-the-art defenses at an 82% success rate on average based only on the transferability, demonstrating the insecurity of the defense techniques.

我们提出了一种注意力不变的攻击方法，为黑箱攻击产生更多可转移的对抗性例子，可以以高成功率骗过最先进的防御系统。

本文提出了一种新的方法来克服对CNN的对抗性攻击的最先进的防御。

本文认为，"注意力转移 "是对抗性攻击无法转移背后的一个关键属性，并提出一种注意力不变的攻击方法


We present Merged-Averaged Classifiers via Hashing (MACH) for $K$-classification with large $K$. Compared to traditional one-vs-all classifiers that require $O(Kd)$ memory and inference cost, MACH only need $O(d\log{K})$ memory while only requiring $O(K\log{K} + d\log{K})$ operation for inference. MACH is the first generic $K$-classification algorithm, with provably theoretical guarantees, which requires $O(\log{K})$ memory without any assumption on the relationship between classes. MACH uses universal hashing to reduce classification with a large number of classes to few independent classification task with very small (constant) number of classes. We provide theoretical quantification of accuracy-memory tradeoff by showing the first connection between extreme classification and heavy hitters. With MACH we can train ODP dataset with 100,000 classes and 400,000 features on a single Titan X GPU (12GB), with the classification accuracy of 19.28\%, which is the best-reported accuracy on this dataset. Before this work, the best performing baseline is a one-vs-all classifier that requires 40 billion parameters (320 GB model size) and achieves 9\% accuracy.   In contrast, MACH can achieve 9\% accuracy with 480x reduction in the model size (of mere 0.6GB). With MACH, we also demonstrate complete training of fine-grained imagenet dataset (compressed size 104GB), with 21,000 classes, on a single GPU.

如何在单个GPU上训练100,000个类

提出了一种高效的散列方法MACH，用于大输出空间背景下的softmax逼近，既节省内存又节省计算量。

一种针对在 ODP 和 Imagenet-21K 数据集上演示的多类设置中涉及大量类的问题的分类方案方法

本文提出了一种基于散列的方案，以减少K-way分类在K较大时的内存和计算时间。


Gradient-based optimization is the foundation of deep learning and reinforcement learning.  Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy. We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables, based on gradients of a learned function.  These estimators can be jointly trained with model parameters or policies, and are applicable in both discrete and continuous settings. We give unbiased, adaptive analogs of state-of-the-art reinforcement learning methods such as advantage actor-critic. We also demonstrate this framework for training discrete latent-variable models.

我们提出了一种对随机变量的黑箱函数梯度进行无偏估计的一般方法。我们将此方法应用于离散变分推理和强化学习。

提出了一种新的方法，为黑箱优化或训练离散潜变量模型执行梯度下降。


Do GANS (Generative Adversarial Nets) actually learn the target distribution? The foundational paper of Goodfellow et al. (2014) suggested they do, if they were given sufficiently large deep nets, sample size, and computation time. A recent theoretical analysis in Arora et al. (2017) raised doubts whether the same holds when discriminator has bounded size. It showed that the training objective can approach its optimum value even if the generated distribution has very low support. In other words, the training objective is unable to prevent mode collapse. The current paper makes two contributions. (1) It proposes a novel test for estimating support size using the birthday paradox of discrete probability. Using this  evidence is presented that well-known GANs approaches do learn distributions of fairly low support.   (2) It theoretically studies encoder-decoder GANs architectures (e.g., BiGAN/ALI), which were proposed to learn more meaningful features via GANs, and consequently to also solve the mode-collapse issue. Our result shows that such encoder-decoder training objectives also cannot guarantee learning of the full distribution because they cannot prevent serious mode collapse. More seriously, they cannot prevent learning meaningless codes for data, contrary to usual intuition.

我们提出了一个GAN学习分布的支持度估计器，表明它们确实受到模式崩溃的影响，并且我们证明了编码器-解码器GAN也不能避免这个问题。

本文试图通过实验来估计典型的GAN产生的解决方案的支持大小。

本文提出了一个基于生日悖论的巧妙的新测试，用于测量生成样本的多样性，实验结果被解释为意味着模式崩溃在一些最先进的生成模型中很强。

本文利用生日悖论表明，一些GAN架构产生的分布具有相当低的支持度。


Due to their complex nature, it is hard to characterize the ways in which machine learning models can misbehave or be exploited when deployed. Recent work on adversarial examples, i.e. inputs with minor perturbations that result in substantially different model predictions, is helpful in evaluating the robustness of these models by exposing the adversarial scenarios where they fail. However, these malicious perturbations are often unnatural, not semantically meaningful, and not applicable to complicated domains such as language. In this paper, we propose a framework to generate natural and legible adversarial examples that lie on the data manifold, by searching in semantic space of dense and continuous data representation, utilizing the recent advances in generative adversarial networks. We present generated adversaries to demonstrate the potential of the proposed approach for black-box classifiers for a wide range of applications such as image classification, textual entailment, and machine translation. We include experiments to show that the generated adversaries are natural, legible to humans, and useful in evaluating and analyzing black-box classifiers.

我们提出了一个框架，通过在潜在语义空间中寻找对手，为视觉和文本领域生成针对黑箱分类器的自然对手。

提出了一种创建语义对抗例子的方法。

提出了一个框架，通过在密集和连续数据表示的潜在空间中搜索对抗者来生成自然对抗的例子


Kronecker-factor Approximate Curvature (Martens & Grosse, 2015) (K-FAC) is a 2nd-order optimization method which has been shown to give state-of-the-art performance on large-scale neural network optimization tasks (Ba et al., 2017).   It is based on an approximation to the Fisher information matrix (FIM) that makes assumptions about the particular structure of the network and the way it is parameterized. The original K-FAC method was applicable only to fully-connected networks, although it has been recently extended by Grosse & Martens (2016) to handle convolutional networks as well. In this work we extend the method to handle RNNs by introducing a novel approximation to the FIM for RNNs. This approximation works by modelling the covariance structure between the gradient contributions at different time-steps using a chain-structured linear Gaussian graphical model, summing the various cross-covariances, and computing the inverse in closed form. We demonstrate in experiments that our method significantly outperforms general purpose state-of-the-art optimizers like SGD with momentum and Adam on several challenging RNN training tasks.

我们通过开发一个新的Fisher近似系列，将K-FAC方法扩展到RNN。

作者将K-FAC方法扩展到RNN，并提出了3种近似F的方法，展示了3个数据集的优化结果，在更新次数和计算时间上都优于ADAM。

建议将克朗克系数适当曲率优化方法扩展到循环神经网络的设置中。

作者提出了一种专门为RNN设计的二阶方法


Bayesian inference is known to provide a general framework for incorporating prior knowledge or specific properties into machine learning models via carefully choosing a prior distribution. In this work, we propose a new type of prior distributions for convolutional neural networks, deep weight prior (DWP), that exploit generative models to encourage a specific structure of trained convolutional filters e.g., spatial correlations of weights. We define DWP in the form of an implicit distribution and propose a method for variational inference with such type of implicit priors. In experiments, we show that DWP improves the performance of Bayesian neural networks when training data are limited, and initialization of weights with samples from DWP accelerates training of conventional convolutional neural networks.

卷积神经网络核的生成模型，在新的数据集上训练时作为先验分布。

一种使用贝叶斯方法为卷积神经网络建模的方法。

提出了 "深度权重先验"：这个想法是在一个辅助数据集上引出一个先验，然后在CNN过滤器上使用该先验来快速启动对感兴趣的数据集的推理。

本文探讨了通过使用自动编码器来获得对训练过的网络的过滤权重的表达性先验，从而为具有类似问题域的卷积神经网络模型学习信息性先验。


The high dimensionality of hyperspectral imaging forces unique challenges in scope, size and processing requirements.   Motivated by the potential for an in-the-field cell sorting detector, we examine a Synechocystis sp. PCC 6803 dataset wherein cells are grown alternatively in nitrogen rich or deplete cultures.   We use deep learning techniques to both successfully classify cells and generate a mask segmenting the cells/condition from the background. Further, we use the classification accuracy to guide a data-driven, iterative feature selection method, allowing the design neural networks requiring 90% fewer input features with little accuracy degradation.

我们将深度学习技术应用于高光谱图像分割和迭代特征采样。 提出了一个贪婪的方案，在分类任务中选择一个高度相关的光谱特征子集。 本文探讨了使用神经网络对细胞的高光谱成像（HSI）进行分类和分割。 基于深度学习技术对细胞进行分类并实现细胞分割，减少输入特征


Data Interpretation is an important part of Quantitative Aptitude exams and requires an individual to answer questions grounded in plots such as bar charts, line graphs, scatter plots, \textit{etc}. Recently, there has been an increasing interest in building models which can perform this task by learning from datasets containing triplets of the form \{plot, question, answer\}. Two such datasets have been proposed in the recent past which contain plots generated from synthetic data with limited (i) $x-y$ axes variables (ii) question templates and (iii) answer vocabulary and hence do not adequately capture the challenges posed by this task. To overcome these limitations of existing datasets, we introduce a new dataset containing $9.7$ million question-answer pairs grounded over $270,000$ plots with three main differentiators. First, the plots in our dataset contain a wide variety of realistic $x$-$y$ variables such as CO2 emission, fertility rate, \textit{etc. } extracted from  real word data sources such as World Bank, government sites, \textit{etc}. Second, the questions in our dataset are more complex as they are based on templates extracted from interesting questions asked by a crowd of workers using a fraction of these plots. Lastly, the answers in our dataset are not restricted to a small vocabulary and a large fraction of the answers seen at test time are not present in the training vocabulary. As a result, existing models for Visual Question Answering which largely use end-to-end models in a multi-class classification framework cannot be used for this task. We establish initial results on this dataset and emphasize the complexity of the task using a multi-staged modular pipeline with various sub-components to (i) extract relevant data from the plot and convert it to a semi-structured table (ii) combine the question with this table and use compositional semantic parsing to arrive at a logical form from which the answer can be derived. We believe that such a modular framework is the best way to go forward as it would enable the research community to independently make progress on all the sub-tasks involved in plot question answering.

我们创建了一个新的数据集，用于对图的数据解释，并提出了相同的基线。

作者提出了一个解决DIP问题的管道，涉及从包含{plot, question, answer}形式的三联体的数据集中学习。

提出了一种能够解释科学图谱中显示的数据的算法。


Learning to predict complex time-series data is a fundamental challenge in a range of disciplines including Machine Learning, Robotics, and Natural Language Processing. Predictive State Recurrent Neural Networks (PSRNNs) (Downey et al.) are a state-of-the-art approach for modeling time-series data which combine the benefits of probabilistic filters and Recurrent Neural Networks into a single model. PSRNNs leverage the concept of Hilbert Space Embeddings of distributions (Smola et al.) to embed predictive states into a Reproducing Kernel Hilbert Space, then estimate, predict, and update these embedded states using Kernel Bayes Rule. Practical implementations of PSRNNs are made possible by the machinery of Random Features, where input features are mapped into a new space where dot products approximate the kernel well. Unfortunately PSRNNs often require a large number of RFs to obtain good results, resulting in large models which are slow to execute and slow to train. Orthogonal Random Features (ORFs) (Choromanski et al.) is an improvement on RFs which has been shown to decrease the number of RFs required for pointwise kernel approximation. Unfortunately, it is not clear that ORFs can be applied to PSRNNs, as PSRNNs rely on Kernel Ridge Regression as a core component of their learning algorithm, and the theoretical guarantees of ORF do not apply in this setting. In this paper, we extend the theory of ORFs to Kernel Ridge Regression and show that ORFs can be used to obtain Orthogonal PSRNNs (OPSRNNs), which are smaller and faster than PSRNNs. In particular, we show that OPSRNN models clearly outperform LSTMs and furthermore, can achieve accuracy similar to PSRNNs with an order of magnitude smaller number of features needed.

通过正交随机特征改进预测状态的循环神经网络 建议通过考虑正交随机特征来提高预测状态递归神经网络的性能。 本文解决了训练预测状态循环神经网络的问题，并做出了两个贡献。


Training deep neural networks requires many training samples, but in practice training labels are expensive to obtain and may be of varying quality, as some may be from trusted expert labelers while others might be from heuristics or other sources of weak supervision such as crowd-sourcing. This creates a fundamental quality- versus-quantity trade-off in the learning process. Do we learn from the small amount of high-quality data or the potentially large amount of weakly-labeled data? We argue that if the learner could somehow know and take the label-quality into account when learning the data representation, we could get the best of both worlds. To this end, we propose “fidelity-weighted learning” (FWL), a semi-supervised student- teacher approach for training deep neural networks using weakly-labeled data. FWL modulates the parameter updates to a student network (trained on the task we care about) on a per-sample basis according to the posterior confidence of its label-quality estimated by a teacher (who has access to the high-quality labels). Both student and teacher are learned from the data. We evaluate FWL on two tasks in information retrieval and natural language processing where we outperform state-of-the-art alternative semi-supervised methods, indicating that our approach makes better use of strong and weak labels, and leads to better task-dependent data representations.

我们提出了保真度加权学习（Fidelity-weighted Learning），这是一种半监督的师生方法，用于使用弱标记的数据训练神经网络。

本文通过使用一个干净的和一个有噪声的数据集，并假设一个教师和学生网络，提出了一种在弱监督下的学习方法。

这篇论文试图用少数标记的训练样本来训练深度神经网络模型。

作者提出了一种针对没有足够可靠的注释数据的情况下训练深度学习模型的方法。


Online learning has attracted great attention due to the increasing demand for systems that have the ability of learning and evolving. When the data to be processed is also high dimensional and dimension reduction is necessary for visualization or prediction enhancement, online dimension reduction will play an essential role. The purpose of this paper is to propose new online learning approaches for supervised dimension reduction. Our first algorithm is motivated by adapting the sliced inverse regression (SIR), a pioneer and effective algorithm for supervised dimension reduction, and making it implementable in an incremental manner. The new algorithm, called incremental sliced inverse regression (ISIR), is able to update the subspace of significant factors with intrinsic lower dimensionality fast and efficiently when new observations come in. We also refine the algorithm by using an overlapping technique  and develop an incremental overlapping sliced inverse regression (IOSIR) algorithm. We verify the effectiveness and efficiency of both algorithms by simulations and real data applications.

我们提出了两种新的方法，增量切片反回归和增量重叠切片反回归，以在线学习的方式实现监督降维。

研究充分降维问题，提出增量切片反回归算法。

本文提出了一种用于监督降维的在线学习算法，称为增量切分反回归


This paper presents a storage-efficient learning model titled Recursive Binary Neural Networks for embedded and mobile devices having a limited amount of on-chip data storage such as hundreds of kilo-Bytes. The main idea of the proposed model is to recursively recycle data storage of weights (parameters) during training. This enables a device with a given storage constraint to train and instantiate a neural network classifier with a larger number of weights on a chip, achieving better classification accuracy. Such efficient use of on-chip storage reduces off-chip storage accesses, improving energy-efficiency and speed of training. We verified the proposed training model with deep and convolutional neural network classifiers on the MNIST and voice activity detection benchmarks. For the deep neural network, our model achieves data storage requirement of as low as 2 bits/weight, whereas the conventional binary neural network learning models require data storage of 8 to 32 bits/weight. With the same amount of data storage, our model can train a bigger network having more weights, achieving 1% less test error than the conventional binary neural network learning model. To achieve the similar classification error, the conventional binary neural network model requires 4× more data storage for weights than our proposed model. For the convolution neural network classifier, the proposed model achieves 2.4% less test error for the same on-chip storage or 6× storage savings to achieve the similar accuracy.

我们提出了一个学习模型，使DNN仅用2位/权重进行学习，这对设备上的学习特别有用。

提出了一种将NN逐步离散化的方法，以提高内存和性能。


Within-class variation in a high-dimensional dataset can be modeled as being on a low-dimensional manifold due to the constraints of the physical processes producing that variation (e.g., translation, illumination, etc.). We desire a method for learning a representation of the manifolds induced by identity-preserving transformations that can be used to increase robustness, reduce the training burden, and encourage interpretability in machine learning tasks. In particular, what is needed is a representation of the transformation manifold that can robustly capture the shape of the manifold from the input data, generate new points on the manifold, and extend transformations outside of the training domain without significantly increasing the error. Previous work has proposed algorithms to efficiently learn analytic operators (called transport operators) that define the process of transporting one data point on a manifold to another.   The main contribution of this paper is to define two transfer learning methods that use this generative manifold representation to learn natural transformations and incorporate them into new data. The first method uses this representation in a novel randomized approach to transfer learning that employs the learned generative model to map out unseen regions of the data space. These results are shown through demonstrations of transfer learning in a data augmentation task for few-shot image classification. The second method use of transport operators for injecting specific transformations into new data examples which allows for realistic image animation and informed data augmentation.   These results are shown on stylized constructions using the classic swiss roll data structure and in demonstrations of transfer learning in a data augmentation task for few-shot image classification. We also propose the use of transport operators for injecting transformations into new data examples which allows for realistic image animation.

在流形上学习运输算子形成了一个有价值的表示，用于做迁移学习等任务。

使用字典学习框架，在增强的USPS数字上学习多方面的运输算子。

本文考虑了Culpepper和Olshausen（2009）的流形运输算子学习框架，并将其解释为获得概率生成模型下的MAP估计。


The seemingly infinite diversity of the natural world arises from a relatively small set of coherent rules, such as the laws of physics or chemistry. We conjecture that these rules give rise to regularities that can be discovered through primarily unsupervised experiences and represented as abstract concepts. If such representations are compositional and hierarchical, they can be recombined into an exponentially large set of new concepts. This paper describes SCAN (Symbol-Concept Association Network), a new framework for learning such abstractions in the visual domain. SCAN learns concepts through fast symbol association, grounding them in disentangled visual primitives that are discovered in an unsupervised manner. Unlike state of the art multimodal generative model baselines, our approach requires very few pairings between symbols and images and makes no assumptions about the form of symbol representations. Once trained, SCAN is capable of multimodal bi-directional inference, generating a diverse set of image samples from symbolic descriptions and vice versa. It also allows for traversal and manipulation of the implicit hierarchy of visual concepts through symbolic instructions and learnt logical recombination operations. Such manipulations enable SCAN to break away from its training data distribution and imagine novel visual concepts through symbolically instructed recombination of previously learnt concepts.

我们提出了一个神经变分模型，用于学习语言指导下的构成性视觉概念。

提出了一种新的神经网结构，通过结合β-VAE和SCAN来学习物体概念。

本文介绍了一个基于VAE的模型，用于图像和文本之间的翻译，其潜伏表示非常适合应用符号运算，使其在从文本中抽取图像时有更多的表达语言。

本文提出了一个名为SCAN（符号-概念关联网络）的新模型，用于层次化的概念学习，并允许使用逻辑运算符对由现有概念组成的新概念进行泛化。