Despite much success in many large-scale language tasks, sequence-to-sequence (seq2seq) models have not been an ideal choice for conversational modeling as they tend to generate generic and repetitive responses. In this paper, we propose a Latent Topic Conversational Model (LTCM) that augments the seq2seq model with a neural topic component to better model human-human conversations. The neural topic component encodes information from the source sentence to build a global “topic” distribution over words, which is then consulted by the seq2seq model to improve generation at each time step. The experimental results show that the proposed LTCM can generate more diverse and interesting responses by sampling from its learnt latent representations. In a subjective human evaluation, the judges also confirm that LTCM is the preferred option comparing to competitive baseline models.

Latent Topic Conversational Model，是seq2seq和神经主题模型的混合体，以产生更多样化和有趣的反应。

本文提出了话题模型和seq2seq对话模型的结合。

通过结合seq2seq模型和神经主题模型，提出了一个带有主题信息的对话模型，并表明所提出的模型优于一些基线模型seq2seq和seq2seq的其他潜在变量模型变体。

本文讨论了对话模型中持久的主题性问题，并提出了一个模型，它是神经主题模型和基于seq2seq的对话系统的结合。


Most of the existing Graph Neural Networks (GNNs) are the mere extension of the Convolutional Neural Networks (CNNs) to graphs. Generally, they consist of several steps of message passing between the nodes followed by a global indiscriminate feature pooling function. In many data-sets, however, the nodes are unlabeled or their labels provide no information about the similarity between the nodes and the locations of the nodes in the graph. Accordingly, message passing may not propagate helpful information throughout the graph. We show that this conventional approach can fail to learn to perform even simple graph classification tasks. We alleviate this serious shortcoming of the GNNs by making them a two step method. In the first of the proposed approach, a graph embedding algorithm is utilized to obtain a continuous feature vector for each node of the graph. The embedding algorithm represents the graph as a point-cloud in the embedding space. In the second step, the GNN is applied to the point-cloud representation of the graph provided by the embedding method. The GNN learns to perform the given task by inferring the topological structure of the graph encoded in the spatial distribution of the embedded vectors. In addition, we extend the proposed approach to the graph clustering problem and a new architecture for graph clustering is proposed. Moreover, the spatial representation of the graph is utilized to design a graph pooling algorithm. We turn the problem of graph down-sampling into a column sampling problem, i.e., the sampling algorithm selects a subset of the nodes whose feature vectors preserve the spatial distribution of all the feature vectors. We apply the proposed approach to several popular benchmark data-sets and it is shown that the proposed geometrical approach strongly improves the state-of-the-art result for several data-sets. For instance, for the PTC data-set, we improve the state-of-the-art result for more than 22 %.

图形分析问题被转化为点云分析问题。 提出了一个使用其自适应图池层的深度GNN网络用于图分类问题。 作者提出了一种学习图的表征的方法


Deep neural networks (DNNs) have been found to be vulnerable to adversarial examples resulting from adding small-magnitude perturbations to inputs. Such adversarial examples can mislead DNNs to produce adversary-selected results.  Different attack strategies have been proposed to generate adversarial examples, but how to produce them with high perceptual quality and more efficiently requires more research efforts.   In this paper, we propose AdvGAN to generate adversarial examples with generative adversarial networks (GANs), which can learn and approximate the distribution of original instances.   For AdvGAN, once the generator is trained, it can generate adversarial perturbations efficiently for any instance, so as to potentially accelerate adversarial training as defenses.    We apply AdvGAN in both semi-whitebox and black-box attack settings. In semi-whitebox attacks, there is no need to access the original target model after the generator is trained, in contrast to traditional white-box attacks. In black-box attacks, we dynamically train a distilled model for the black-box model and optimize the generator accordingly.  Adversarial examples generated by AdvGAN on different target models have high attack success rate under state-of-the-art defenses compared to other attacks. Our attack  has placed the first with 92.76% accuracy on a public MNIST black-box attack challenge.

我们提议在半白盒和黑盒环境下，基于生成式对抗网络生成对抗性例子。

描述了AdvGAN，一个有条件的GAN加对抗性损失，并在半白盒和黑盒设置上评估了AdvGAN，报告了最先进的结果。

本文提出了一种生成欺骗分类系统的对抗性例子的方法，并赢得了MadryLab的mnist挑战。


This paper proposes a new model for the rating prediction task in recommender systems which significantly outperforms previous state-of-the art models on a time-split Netflix data set. Our model is based on deep autoencoder with 6 layers and is trained end-to-end without any layer-wise pre-training. We empirically demonstrate that: a) deep autoencoder models generalize much better than the shallow ones, b) non-linear activation functions with negative parts are crucial for training deep models, and c) heavy use of regularization techniques such as dropout is necessary to prevent over-fitting. We also propose a new training algorithm based on iterative output re-feeding to overcome natural sparseness of collaborate filtering. The new algorithm significantly speeds up training and improves model performance. Our code is publicly available.

本文展示了如何在时间分割的Netflix数据集上端到端训练深度自动编码器以实现SoA结果。

本文提出了一个用于评级预测的深度自动编码器模型，该模型在Netflix奖项数据集上的表现优于其他最先进的方法。

建议在推荐系统中使用深度AE来做评级预测任务。

作者提出了一个更准确的Netflix推荐模型，证明了深度自动编码器可以胜过有时间信息的更复杂的基于RNN的模型。


Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times. Despite these successes, however, popular feed-forward sequence models like the Transformer fail to generalize in many simple tasks that recurrent models handle with ease, e.g. copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time. We propose the Universal Transformer (UT), a parallel-in-time self-attentive recurrent sequence model which can be cast as a generalization of the Transformer model and which addresses these issues. UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs. We also add a dynamic per-position halting mechanism and find that it improves accuracy on several tasks. In contrast to the standard Transformer, under certain assumptions UTs can be shown to be Turing-complete. Our experiments show that UTs outperform standard Transformers on a wide range of algorithmic and language understanding tasks, including the challenging LAMBADA language modeling task where UTs achieve a new state of the art, and machine translation where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De dataset.

我们介绍了Universal Transformer，这是一个自关注的并行时间循环序列模型，在广泛的序列到序列的任务中，包括机器翻译，它的性能优于Transformers和LSTMs。

提出了一个新的模型UT，基于Transformer模型，增加了循环和动态停止循环的功能。

本文通过递归地应用多头自我关注块来扩展Transformer，而不是在vanilla Transformer中叠加多个块。


We present a framework for interpretable continual learning (ICL). We show that explanations of previously performed tasks can be used to improve performance on future tasks. ICL generates a good explanation of a finished task, then uses this to focus attention on what is important when facing a new task. The ICL idea is general and may be applied to many continual learning approaches. Here we focus on the variational continual learning framework to take advantage of its flexibility and efficacy in overcoming catastrophic forgetting. We use saliency maps to provide explanations of performed tasks and propose a new metric to assess their quality. Experiments show that ICL achieves state-of-the-art results in terms of overall continual learning performance as measured by average classification accuracy, and also in terms of its explanations, which are assessed qualitatively and quantitatively using the proposed metric.

本文开发了一个可解释的持续学习框架，其中对已完成任务的解释被用来提高学习者在未来任务中的注意力，并且还提出了一个解释指标。

作者提出了一个持续学习的框架，该框架基于对以前所学任务的执行分类的解释。

本文提出了一个持续学习框架的扩展，使用现有的变分持续学习作为基础方法，并使用证据权重。


The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikevicius et al. (2017). On the other hand, while a lot of research has also happened in the domain of low and mixed-precision Integer training, these works either present results for non-SOTA networks (for instance only AlexNet for ImageNet-1K), or relatively small datasets (like CIFAR-10). In this work, we train state-of-the-art visual understanding neural networks on the ImageNet-1K dataset, with Integer operations on General Purpose (GP) hardware. In particular, we focus on Integer Fused-Multiply-and-Accumulate (FMA) operations which take two pairs of INT16 operands and accumulate results into an INT32 output.We propose a shared exponent representation of tensors and develop a Dynamic Fixed Point (DFP) scheme suitable for common neural network operations. The nuances of developing an efficient integer convolution kernel is examined, including methods to handle overflow of the INT32 accumulator. We implement CNN training for ResNet-50, GoogLeNet-v1, VGG-16 and AlexNet; and these networks achieve or exceed SOTA accuracy within the same number of iterations as their FP32 counterparts without any change in hyper-parameters and with a 1.8X improvement in end-to-end training throughput. To the best of our knowledge these results represent the first INT16 training results on GP hardware for ImageNet-1K dataset using SOTA CNNs and achieve highest reported accuracy using half precision

在通用硬件上使用16位整数的混合精度训练管道；ImageNet级CNN的SOTA精度；ImageNet-1K分类任务的最佳报告精度与任何降低精度训练。

本文展示了混合精度动态定点计算的精心实现，可以使用16位整数表示的降低精度的深度学习模型来达到最先进的精度。

提出了一种 "动态定点 "方案，分享张量的指数部分，开发了用这种格式进行NN计算的程序，并对有限精度训练进行了演示。


In this paper we introduce a new speech recognition system, leveraging a simple letter-based ConvNet acoustic model. The acoustic model requires only audio transcription for training -- no alignment annotations, nor any forced alignment step is needed. At inference, our decoder takes only a word list and a language model, and is fed with letter scores from the acoustic model -- no phonetic word lexicon is needed. Key ingredients for the acoustic model are Gated Linear Units and high dropout. We show near state-of-the-art results in word error rate on the LibriSpeech corpus with MFSC features, both on the clean and other configurations.

一个基于字母的ConvNet声学模型带来了一个简单而有竞争力的语音识别管道。 本文将门控卷积神经网络应用于语音识别，使用训练标准ASG。


Generative adversarial networks (GANs) are a powerful framework for generative tasks. However, they are difficult to train and tend to miss modes of the true data generation process. Although GANs can learn a rich representation of the covered modes of the data in their latent space, the framework misses an inverse mapping from data to this latent space. We propose Invariant Encoding Generative Adversarial Networks (IVE-GANs), a novel GAN framework that introduces such a mapping for individual samples from the data by utilizing features in the data which are invariant to certain transformations. Since the model maps individual samples to the latent space, it naturally encourages the generator to cover all modes. We demonstrate the effectiveness of our approach in terms of generative performance and learning rich representations on several datasets including common benchmark image generation tasks.

一个新的GAN框架，利用变换不变的特征来学习丰富的表示和强大的生成器。 提出了一个修正的GAN目标，由一个经典的GAN项和一个不变的编码项组成。 本文介绍了IVE-GAN，一个将编码器引入生成式对抗网络框架的模型。


We propose a method for learning the dependency structure between latent variables in deep latent variable models.   Our general modeling and inference framework combines the complementary strengths of deep generative models and probabilistic graphical models. In particular, we express the latent variable space of a variational autoencoder (VAE) in terms of a Bayesian network with a learned, flexible dependency structure.   The network parameters, variational parameters as well as the latent topology are optimized simultaneously with a single objective.   Inference is formulated via a sampling procedure that produces expectations over latent variable structures and incorporates top-down and bottom-up reasoning over latent variable values.   We validate our framework in extensive experiments on MNIST, Omniglot, and CIFAR-10. Comparisons to state-of-the-art structured variational autoencoder baselines show improvements in terms of the expressiveness of the learned model.

我们提出了一种在变分自动编码器中学习潜在依赖结构的方法。 使用二元随机变量矩阵来捕捉分层深度生成模型中潜在变量之间的依赖关系。 本文提出了一种VAE方法，在训练过程中学习潜变量的依赖结构。 作者建议用自动回归结构来增加VAE的潜空间，以提高推理网络和潜先验的表现力。


Many real-world time series, such as in activity recognition, finance, or climate science, have changepoints where the system's structure or parameters change. Detecting changes is important as they may indicate critical events. However, existing methods for changepoint detection face challenges when (1) the patterns of change cannot be modeled using simple and predefined metrics, and (2) changes can occur gradually, at multiple time-scales. To address this, we show how changepoint detection can be treated as a supervised learning problem, and propose a new deep neural network architecture that can efficiently identify both abrupt and gradual changes at multiple scales. Our proposed method, pyramid recurrent neural network (PRNN), is designed to be scale-invariant, by incorporating wavelets and pyramid analysis techniques from multi-scale signal processing. Through experiments on synthetic and real-world datasets, we show that PRNN can detect abrupt and gradual changes with higher accuracy than the state of the art and can extrapolate to detect changepoints at novel timescales that have not been seen in training.

我们介绍了一种用于多变量时间序列中变化点检测的标度不变的神经网络结构。

本文利用深度架构中的小波变换概念来解决变化点检测。

本文提出了一个基于金字塔的神经网络，并将其应用于一维信号，其基本过程发生在不同的时间尺度上，其任务是变化点检测。


We demonstrate how to learn efficient heuristics for automated reasoning algorithms through deep reinforcement learning. We focus on backtracking search algorithms for quantified Boolean logics, which already can solve formulas of impressive size - up to 100s of thousands of variables. The main challenge is to find a representation of these formulas that lends itself to making predictions in a scalable way. For challenging problems, the heuristic learned through our approach reduces execution time by a factor of 10 compared to the existing handwritten heuristics.

RL为自动推理算法找到更好的启发式方法。

旨在利用强化学习为回溯搜索算法学习启发式方法，并提出一个模型，利用图形神经网络产生字面和句子嵌入，并利用它们来预测每个字面的质量，以决定每个动作的概率。

本文提出了一种利用深度学习为QBF自动学习变量选择启发式的方法


We consider the question of how to assess generative adversarial networks, in particular with respect to whether or not they generalise beyond memorising the training data. We propose a simple procedure for assessing generative adversarial network performance based on a principled consideration of what the actual goal of generalisation is. Our approach involves using a test set to estimate the Wasserstein distance between the generative distribution produced by our procedure, and the underlying data distribution. We use this procedure to assess the performance of several modern generative adversarial network architectures. We find that this procedure is sensitive to the choice of ground metric on the underlying data space, and suggest a choice of ground metric that substantially improves performance.   We finally suggest that attending to the ground metric used in Wasserstein generative adversarial network training may be fruitful, and outline a concrete pathway towards doing so.

评估您的GAN是否实际上在做一些事情，而不是记住训练数据。

旨在为GAN提供一个质量衡量/测试，并提议通过使用由Diracs之和构成的两个分布之间的Wasserstein距离作为基线性能来评估GAN学习的分布的当前近似度。

本文提出了一个通过重新考虑观察的关键来评估GAN性能的程序，使用该程序来测试和改进目前的GAN


The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various hand-designed alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose to leverage automatic search techniques to discover new activation functions. Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions. We verify the effectiveness of the searches by conducting an empirical evaluation with the best discovered activation function. Our experiments show that the best discovered activation function, f(x) = x * sigmoid(beta * x), which we name Swish, tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.

我们使用搜索技术来发现新的激活函数，我们发现的最佳激活函数f(x) = x * sigmoid(beta * x)，在ImageNet等一些具有挑战性的任务上优于ReLU。

提出了一种基于强化学习的方法，通过从一组单数和双数运算符中搜索组合来寻找非线性。

本文利用强化学习来搜索一组一元和二元函数的组合，从而得到一个新的激活函数

作者利用强化学习，从丰富的可能候选者中找到新的潜在激活函数。


Successful training of convolutional neural networks is often associated with suffi-  ciently deep architectures composed of high amounts of features. These networks  typically rely on a variety of regularization and pruning techniques to converge  to less redundant states. We introduce a novel bottom-up approach to expand  representations in fixed-depth architectures. These architectures start from just a  single feature per layer and greedily increase width of individual layers to attain  effective representational capacities needed for a specific task. While network  growth can rely on a family of metrics, we propose a computationally efficient  version based on feature time evolution and demonstrate its potency in determin-  ing feature importance and a networks’ effective capacity. We demonstrate how  automatically expanded architectures converge to similar topologies that benefit  from lesser amount of parameters or improved accuracy and exhibit systematic  correspondence in representational complexity with the specified task. In contrast  to conventional design patterns with a typical monotonic increase in the amount of  features with increased depth, we observe that CNNs perform better when there is  more learnable parameters in intermediate, with falloffs to earlier and later layers.

一种自下而上的算法，将每层只有一个特征的CNN扩展到具有足够表征能力的架构。

提出动态调整全卷积神经网络的特征图深度，制定自我相似度的措施，提升性能。

介绍了一个简单的基于相关性的指标来衡量神经网络中的过滤器是否被有效利用，作为有效容量的代理。

旨在通过对网络中间层通道的增量添加和移除，解决深度学习架构的搜索问题。


Deep neural networks are almost universally trained with reverse-mode automatic differentiation (a.k.a. backpropagation). Biological networks, on the other hand, appear to lack any mechanism for sending gradients back to their input neurons, and thus cannot be learning in this way. In response to this, Scellier & Bengio (2017) proposed Equilibrium Propagation - a method for gradient-based train- ing of neural networks which uses only local learning rules and, crucially, does not rely on neurons having a mechanism for back-propagating an error gradient. Equilibrium propagation, however, has a major practical limitation: inference involves doing an iterative optimization of neural activations to find a fixed-point, and the number of steps required to closely approximate this fixed point scales poorly with the depth of the network. In response to this problem, we propose Initialized Equilibrium Propagation, which trains a feedforward network to initialize the iterative inference procedure for Equilibrium propagation. This feed-forward network learns to approximate the state of the fixed-point using a local learning rule. After training, we can simply use this initializing network for inference, resulting in a learned feedforward network. Our experiments show that this network appears to work as well or better than the original version of Equilibrium propagation. This shows how we might go about training deep networks without using backpropagation.

我们通过使用基于能量的模型来提供本地目标，训练一个没有反推的前馈网络

本文旨在加快用平衡传播法（EP）训练的基于能量的模型的迭代推理程序，建议训练一个前馈网络来预测 "平衡网络 "的固定点。

训练一个单独的网络来初始化用平衡传播法训练的循环网络


We propose a novel generative model architecture designed to learn representations for images that factor out a single attribute from the rest of the representation. A single object may have many attributes which when altered do not change the identity of the object itself. Consider the human face; the identity of a particular person is independent of whether or not they happen to be wearing glasses. The attribute of wearing glasses can be changed without changing the identity of the person. However, the ability to manipulate and alter image attributes without altering the object identity is not a trivial task. Here, we are interested in learning a representation of the image that separates the identity of an object (such as a human face) from an attribute (such as 'wearing glasses'). We demonstrate the success of our factorization approach by using the learned representation to synthesize the same face with and without a chosen attribute. We refer to this specific synthesis process as image attribute manipulation. We further demonstrate that our model achieves competitive scores, with state of the art, on a facial attribute classification task.

学习图像的表征，将单一属性的因素剔除。

本文在条件性VAE GAN的基础上，允许在合成过程中进行属性操作。

本文提出了一个生成模型来学习能够将对象的身份与属性分开的表征，并通过增加一个辅助网络来扩展自动编码器的对抗性。


Stochastic video prediction models take in a sequence of image frames, and generate a sequence of consecutive future image frames. These models typically generate future frames in an autoregressive fashion, which is slow and requires the input and output frames to be consecutive. We introduce a model that overcomes these drawbacks by generating a latent representation from an arbitrary set of frames that can then be used to simultaneously and efficiently sample temporally consistent frames at arbitrary time-points. For example, our model can "jump" and directly sample frames at the end of the video, without sampling intermediate frames. Synthetic video evaluations confirm substantial gains in speed and functionality without loss in fidelity. We also apply our framework to a 3D scene reconstruction dataset. Here, our model is conditioned on camera location and can sample consistent sets of images for what an occluded region of a 3D scene might look like, even if there are multiple possibilities for what that region might contain. Reconstructions and videos are available at https://bit.ly/2O4Pc4R.

我们提出了一个用于一致的三维重建和跳跃式视频预测的模型，例如在不产生中间帧的情况下产生未来多个时间步长的图像帧。

本文提出了一种索引数据建模的通用方法，将索引信息与观测信息一起编码到神经网络中，然后将观测条件解码到目标索引上。

建议使用一种以排列不变的方式编码输入视频的VAE来预测视频的未来帧。


The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn’t. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of the stochastic gradient, whereas the update magnitude is solely determined by an estimate of its relative variance. We  disentangle these two aspects and analyze them in isolation, shedding light on ADAM ’s inner workings. Transferring the "variance adaptation” to momentum- SGD gives rise to a novel method, completing the practitioner’s toolbox for problems where ADAM fails.

分析流行的Adam优化器

本文试图通过提出两种算法来改进基于动量的方差适应的Adam。

本文分析了Adam中使用的学习率的标度不变性和特殊形状，认为Adam的更新是签名更新和基于方差的学习率的结合。

本文将ADAM算法分成两部分：梯度符号的随机方向和相对方差的自适应步进式，并提出两种算法分别进行测试。


We propose a novel framework to adaptively adjust the dropout rates for the deep neural network based on a Rademacher complexity bound. The state-of-the-art deep learning algorithms impose dropout strategy to prevent feature co-adaptation. However, choosing the dropout rates remains an art of heuristics or relies on empirical grid-search over some hyperparameter space. In this work, we show the network Rademacher complexity is bounded by a function related to the dropout rate vectors and the weight coefficient matrices. Subsequently, we impose this bound as a regularizer and provide a theoretical justified way to trade-off between model complexity and representation power. Therefore, the dropout rates and the empirical loss are unified into the same objective function, which is then optimized using the block coordinate descent algorithm. We discover that the adaptively adjusted dropout rates converge to some interesting distributions that reveal meaningful patterns.Experiments on the task of image and document classification also show our method achieves better performance compared to the state-of the-art dropout algorithms.

我们提出了一个新的框架，基于Rademacher复杂度约束，自适应调整深度神经网络的丢弃率。

作者将丢弃参数与网络的Rademacher复杂性的边界联系起来

将网络的可学习性的复杂性与反向传播的丢弃率联系起来。


Sensor fusion is a key technology that integrates various sensory inputs to allow for robust decision making in many applications such as autonomous driving and robot control. Deep neural networks have been adopted for sensor fusion in a body of recent studies. Among these, the so-called netgated architecture was proposed, which has demonstrated improved performances over the conventional convolu- tional neural networks (CNN). In this paper, we address several limitations of the baseline negated architecture by proposing two further optimized architectures: a coarser-grained gated architecture employing (feature) group-level fusion weights and a two-stage gated architectures leveraging both the group-level and feature- level fusion weights. Using driving mode prediction and human activity recogni- tion datasets, we demonstrate the significant performance improvements brought by the proposed gated architectures and also their robustness in the presence of sensor noise and failures.

提出了用于传感器融合的优化门控深度学习架构。

作者通过提出更粗粒度的门控融合架构和两阶段门控融合架构，改善了基线否定架构的几个局限性

提出了两个用于传感器融合的门控深度学习架构，通过分组特征，展示了改进的性能，特别是在随机传感器噪声和故障的情况下。


We develop a mean field theory for batch normalization in fully-connected feedforward neural networks. In so doing, we provide a precise characterization of signal propagation and gradient backpropagation in wide batch-normalized networks at initialization. Our theory shows that gradient signals grow exponentially in depth and that these exploding gradients cannot be eliminated by tuning the initial weight variances or by adjusting the nonlinear activation function. Indeed, batch normalization itself is the cause of gradient explosion. As a result, vanilla batch-normalized networks without skip connections are not trainable at large depths for common initialization schemes, a prediction that we verify with a variety of empirical simulations. While gradient explosion cannot be eliminated, it can be reduced by tuning the network close to the linear regime, which improves the trainability of deep batch-normalized networks without residual connections. Finally, we investigate the learning dynamics of batch-normalized networks and observe that after a single step of optimization the networks achieve a relatively stable equilibrium in which gradients have dramatically smaller dynamic range. Our theory leverages Laplace, Fourier, and Gegenbauer transforms and we derive new identities that may be of independent interest.

批量归一化导致vanilla前馈网络的梯度爆炸。 在具有随机初始化权重的全连接网络中，发展了批量正常化（BN）的平均场理论。 利用协方差矩阵与各层的演变，为深度神经网络提供了一个动态视角。


We present NeuroSAT, a message passing neural network that learns to solve SAT problems after only being trained as a classifier to predict satisfiability.   Although it is not competitive with state-of-the-art SAT solvers, NeuroSAT can solve problems that are substantially larger and more difficult than it ever saw during training by simply running for more iterations. Moreover, NeuroSAT generalizes to novel distributions; after training only on random SAT problems, at test time it can solve SAT problems encoding graph coloring, clique detection, dominating set, and vertex cover problems, all on a range of distributions over small random graphs.

我们训练一个图网络来预测布尔可满足性，并表明它学会了搜索解决方案，而且它找到的解决方案可以从其激活中解码。

本文描述了一个用于预测可满足性的通用神经网络架构

本文介绍了NeuroSAT架构，它使用深度信息传递神经网来预测CNF实例的可满足性。


Spatiotemporal forecasting has various applications in neuroscience, climate and transportation domain. Traffic forecasting is one canonical example of such learning task. The task is challenging due to (1) complex spatial dependency on road networks, (2) non-linear temporal dynamics with changing road conditions and (3) inherent difficulty of long-term forecasting. To address these challenges, we propose to model the traffic flow as a diffusion process on a directed graph and introduce Diffusion Convolutional Recurrent Neural Network (DCRNN), a deep learning framework for traffic forecasting that incorporates both spatial and temporal dependency in the traffic flow. Specifically, DCRNN captures the spatial dependency using bidirectional random walks on the graph, and the temporal dependency using the encoder-decoder architecture with scheduled sampling. We evaluate the framework on two real-world large-scale road network traffic datasets and observe consistent improvement of 12% - 15% over state-of-the-art baselines

一个在有向图上学习预测的神经序列模型。 本文提出了用于时空流量预测问题的扩散卷积递归神经网络架构 建议利用卷积循环神经网络的扩散过程建立流量预测模型，以解决时空自相关的问题。


Obtaining reliable uncertainty estimates of neural network predictions is a long standing challenge. Bayesian neural networks have been proposed as a solution, but it remains open how to specify their prior. In particular, the common practice of a standard normal prior in weight space imposes only weak regularities, causing the function posterior to possibly generalize in unforeseen ways on inputs outside of the training distribution. We propose noise contrastive priors (NCPs) to obtain reliable uncertainty estimates. The key idea is to train the model to output high uncertainty for data points outside of the training distribution. NCPs do so using an input prior, which adds noise to the inputs of the current mini batch, and an output prior, which is a wide distribution given these inputs. NCPs are compatible with any model that can output uncertainty estimates, are easy to scale, and yield reliable uncertainty estimates throughout training. Empirically, we show that NCPs prevent overfitting outside of the training distribution and result in uncertainty estimates that are useful for active learning. We demonstrate the scalability of our method on the flight delays data set, where we significantly improve upon previously published results.

我们训练神经网络对噪声输入的不确定性，以避免在训练分布之外的过度自信的预测。

介绍了一种获得神经网络预测不确定性估计的方法，该方法在量化训练分布之外的点的预测不确定性时具有良好的性能。

本文考虑了神经网络的不确定性估计问题，并提出使用贝叶斯方法与噪声对比性先验


Convolutional neural networks (CNNs) were inspired by human vision and, in some settings, achieve a performance comparable to human object recognition. This has lead to the speculation that both systems use similar mechanisms to perform recognition. In this study, we conducted a series of simulations that indicate that there is a fundamental difference between human vision and CNNs: while object recognition in humans relies on analysing shape, CNNs do not have such a shape-bias. We teased apart the type of features selected by the model by modifying the CIFAR-10 dataset so that, in addition to containing objects with shape, the images concurrently contained non-shape features, such as a noise-like mask. When trained on these modified set of images, the model did not show any bias towards selecting shapes as features. Instead it relied on whichever feature allowed it to perform the best prediction -- even when this feature was a noise-like mask or a single predictive pixel amongst 50176 pixels. We also found that regularisation methods, such as batch normalisation or Dropout, did not change this behaviour and neither did past or concurrent experience with images from other datasets.

这项研究强调了人类视觉和CNN之间的一个关键区别：人类的物体识别依赖于对形状的分析，而CNN则没有这种形状偏差。

试图通过一系列精心设计的实验来确定，为图像分类而训练的CNN并不像人类视觉那样编码形状偏差。

本文强调了一个事实，即CNN不一定会学会根据物体的形状来识别物体，并表明它们会过度依赖基于噪声的特征。


The development of high-dimensional generative models has recently gained a great surge of interest with the introduction of variational auto-encoders and generative adversarial neural networks. Different variants have been proposed where the underlying latent space is structured, for example, based on attributes describing the data to generate. We focus on a particular problem where one aims at generating samples corresponding to a number of objects under various views. We assume that the distribution of the data is driven by two independent latent factors: the content, which represents the intrinsic features of an object, and the view, which stands for the settings of a particular observation of that object. Therefore, we propose a generative model and a conditional variant built on such a disentangled latent space. This approach allows us to generate realistic samples corresponding to various objects in a high variety of views. Unlike many multi-view approaches, our model doesn't need any supervision on the views but only on the content. Compared to other conditional generation approaches that are mostly based on binary or categorical attributes, we make no such assumption about the factors of variations. Our model can be used on problems with a huge, potentially infinite, number of categories. We experiment it on four images datasets on which we demonstrate the effectiveness of the model and its ability to generalize.

我们描述了一个新颖的多视图生成模型，它可以生成同一物体的多个视图，或者同一视图中的多个物体，而不需要对视图进行标注。

本文提出了一种基于GAN的图像生成方法，试图将描述图像内容的潜变量与描述视图属性的潜变量分开。

本文提出了一个GAN架构，旨在将某一特定类别的基础分布分解为 "内容 "和 "视图"。

提出了一种基于生成对抗网络（GAN）的新的生成模型，该模型在没有视图监督的情况下将对象的内容和视图分开，并将GMV扩展为一个条件生成模型，该模型接受一个输入图像并生成输入图像中对象的不同视图。


The huge size of deep networks hinders their use in small computing devices. In this paper, we consider compressing the network by weight quantization. We extend a recently proposed loss-aware weight binarization scheme to ternarization, with possibly different scaling parameters for the positive and negative weights, and m-bit (where m > 2) quantization. Experiments on feedforward and recurrent neural networks show that the proposed scheme outperforms state-of-the-art weight quantization algorithms, and is as accurate (or even more accurate) than the full-precision network.

提出了一种直接考虑其对损失的影响的损失感知权重量化算法。

提出了一种通过权重ternarization来压缩网络的方法。

本文提出了一种新的方法来训练具有量化权重的DNN，方法是将量化作为近似准牛顿算法的一个约束条件，同时学习量化值的比例。

本文将损失感知的权重二值化方案扩展到terarization和任意m位量化，并展示了其良好的性能。


In the pursuit of increasingly intelligent learning systems, abstraction plays a vital role in enabling sophisticated decisions to be made in complex environments. The options framework provides formalism for such abstraction over sequences of decisions.   However most models require that options be given a priori, presumably specified by hand, which is neither efficient, nor scalable. Indeed, it is preferable to learn options directly from interaction with the environment. Despite several efforts, this remains a difficult problem: many approaches require access to a model of the environmental dynamics, and inferred options are often not interpretable, which limits our ability to explain the system behavior for verification or debugging purposes.   In this work we develop a novel policy gradient method for the automatic learning of policies with options.   This algorithm uses inference methods to simultaneously improve all of the options available to an agent, and thus can be employed in an off-policy manner, without observing option labels. Experimental results show that the options learned can be interpreted. Further, we find that the method presented here is more sample efficient than existing methods, leading to faster and more stable learning of policies with options.

我们开发了一种新的策略梯度方法，用于使用可微分推理步骤自动学习带有选项的策略。 本文提出了一种新的学习选项策略梯度技术，即可以用一个样本来更新所有选项。 提出了一种在复杂连续问题中学习选择的非策略性方法。


The paper, interested in unsupervised feature selection, aims to retain the features best accounting for the local patterns in the data. The proposed approach, called Locally Linear Unsupervised Feature Selection, relies on a dimensionality reduction method to characterize such patterns; each feature is thereafter assessed according to its compliance w.r.t. the local patterns, taking inspiration from Locally Linear Embedding (Roweis and Saul, 2000). The experimental validation of the approach on the scikit-feature benchmark suite demonstrates its effectiveness compared to the state of the art.

通过捕捉数据的局部线性结构进行无监督的特征选择 提出了局部线性无监督的特征选择。 本文提出了LLUFS方法进行特征选择。


Humans can understand and produce new utterances effortlessly, thanks to their systematic compositional skills. Once a person learns the meaning of a new verb "dax," he or she can immediately understand the meaning of "dax twice" or "sing and dax." In this paper, we introduce the SCAN domain, consisting of a set of simple compositional navigation commands paired with the corresponding action sequences. We then test the zero-shot generalization capabilities of a variety of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence methods. We find that RNNs can generalize well when the differences between training and test commands are small, so that they can apply "mix-and-match" strategies to solve the task. However, when generalization requires systematic compositional skills (as in the "dax" example above), RNNs fail spectacularly. We conclude with a proof-of-concept experiment in neural machine translation, supporting the conjecture that lack of systematicity is an important factor explaining why neural networks need very large training sets.

使用一个简单的语言驱动的导航任务，我们研究了现代seq2seq循环网络的组成能力。

本文重点讨论了现代序列到序列RNN的零样本学习构成能力，并暴露了目前seq2seq RNN架构的不足之处。

本文分析了RNN的构成能力，特别是RNN在随机的SCAN命令子集上的泛化能力，在较长的SCAN命令上的泛化能力，以及对原始命令的构成能力。

作者介绍了一个新的数据集，便于分析一个Seq2Seq学习案例


This paper addresses the challenging problem of retrieval and matching of graph structured objects, and makes two key contributions. First, we demonstrate how  Graph Neural Networks (GNN), which have emerged as an effective model for various supervised prediction problems defined on structured data, can be trained to produce embedding of graphs in vector spaces that enables efficient similarity reasoning. Second, we propose a novel Graph Matching Network model that, given a pair of graphs as input, computes a similarity score between them by jointly reasoning on the pair through a new cross-graph attention-based matching mechanism. We demonstrate the effectiveness of our models on different domains including the challenging problem of control-flow-graph based function similarity search that plays an important role in the detection of vulnerabilities in software systems. The experimental analysis demonstrates that our models are not only able to exploit structure in the context of similarity learning but they can also outperform domain-specific baseline systems that have been carefully hand-engineered for these problems.

我们解决了结构化对象的相似性学习问题，特别是在计算机安全方面的应用，并提出了一个新的模型图匹配网络，在这个任务上表现出色。

作者介绍了一个图形匹配网络，用于检索和匹配图形结构的对象。

作者通过提出一个图形嵌入网络的扩展来解决图形匹配的问题

作者提出了两种学习图对之间相似性分数的方法，并展示了将图匹配的思想引入图神经网络的好处。


Context information plays an important role in human language understanding, and it is also useful for machines to learn vector representations of language. In this paper, we explore an asymmetric encoder-decoder structure for unsupervised context-based sentence representation learning. As a result, we build an encoder-decoder architecture with an RNN encoder and a CNN decoder, and we show that neither an autoregressive decoder nor an RNN decoder is required.   We further combine a suite of effective designs to significantly improve model efficiency while also achieving better performance. Our model is trained on two different large unlabeled corpora, and in both cases transferability is evaluated on a set of downstream language understanding tasks. We empirically show that our model is simple and fast while producing rich sentence representations that excel in downstream tasks.

我们提出了一个RNN-CNN编码器-解码器模型，用于快速无监督的句子表征学习。

对学习句子嵌入的跳过思考框架的修改。

本文提出了一种新的RNN编码器和CNN解码器混合设计，用于预训练，在预训练编码器时不需要自回归解码器。

作者通过使用CNN解码器只对一个目标句子进行解码来扩展Skip-thought。


Building on the success of deep learning, two modern approaches to learn a probability model of the observed data are Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs). VAEs consider an explicit probability model for the data and compute a generative distribution by maximizing a variational lower-bound on the log-likelihood function. GANs, however, compute a generative model by minimizing a distance between observed and generated probability distributions without considering an explicit model for the observed data. The lack of having explicit probability models in GANs prohibits computation of sample likelihoods in their frameworks and limits their use in statistical inference problems. In this work, we show that an optimal transport GAN with the entropy regularization can be viewed as a generative model that maximizes a lower-bound on average sample likelihoods, an approach that VAEs are based on. In particular, our proof constructs an explicit probability model for GANs that can be used to compute likelihood statistics within GAN's framework. Our numerical results on several datasets demonstrate consistent trends with the proposed theory.

计算生成式对抗网络中样本可能性的统计方法
证明具有熵正则化的WGAN能使观察到的数据分布的可能性达到最大的下限。
作者声称有可能利用熵正则化最优运输的上限，得出一个 "样本可能性 "的测量。


We introduce geomstats, a Python package for Riemannian modelization and optimization over manifolds such as hyperspheres, hyperbolic spaces, SPD matrices or Lie groups of transformations. Our contribution is threefold. First, geomstats allows the flexible modeling of many a machine learning problem through an efficient and extensively unit-tested implementations of these manifolds, as well as the set of useful Riemannian metrics, exponential and logarithm maps that we provide. Moreover, the wide choice of loss functions and our implementation of the corresponding gradients allow fast and easy optimization over manifolds. Finally, geomstats is the only package to provide a unified framework for Riemannian geometry, as the operations implemented in geomstats are available with different computing backends (numpy,tensorflow and keras), as well as with a GPU-enabled mode–-thus considerably facilitating the application of Riemannian geometry in machine learning. In this paper, we present geomstats through a review of the utility and advantages of manifolds in machine learning, using the concrete examples that they span to show the efficiency and practicality of their implementation using our package

我们介绍geomstats，这是一个高效的Python包，用于流形上的黎曼建模和优化，与numpy和tensorflow都兼容。

本文介绍了软件包geomstats，它提供了在机器学习模型中对黎曼流形和度量的简单使用。

提出了一个用于黎曼流形上的优化和应用的Python包，并强调了Geomstats包与其他包的区别。

介绍了一个几何工具箱Geomstats，用于黎曼流形上的机器学习。


We propose to execute deep neural networks (DNNs) with dynamic and sparse graph (DSG) structure for compressive memory and accelerative execution during both training and inference. The great success of DNNs motivates the pursuing of lightweight models for the deployment onto embedded devices. However, most of the previous studies optimize for inference while neglect training or even complicate it. Training is far more intractable, since (i) the neurons dominate the memory cost rather than the weights in inference; (ii) the dynamic activation makes previous sparse acceleration via one-off optimization on fixed weight invalid; (iii) batch normalization (BN) is critical for maintaining accuracy while its activation reorganization damages the sparsity. To address these issues, DSG activates only a small amount of neurons with high selectivity at each iteration via a dimensionreduction search and obtains the BN compatibility via a double-mask selection. Experiments show significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x) with little accuracy loss on various benchmarks.

我们通过降维搜索构建动态稀疏图，以减少DNN训练和推理的计算和内存成本。 作者提出使用动态稀疏计算图来减少深度神经网络（DNN）的计算内存和时间成本。 本文提出了一种利用计算图的动态修剪来加快深度神经网络的训练和推理的方法。


Efficient exploration remains a major challenge for reinforcement learning. One reason is that the variability of the returns often depends on the current state and action, and is therefore heteroscedastic. Classical exploration strategies such as upper confidence bound algorithms and Thompson sampling fail to appropriately account for heteroscedasticity, even in the bandit setting. Motivated by recent findings that address this issue in bandits, we propose to use Information-Directed Sampling (IDS) for exploration in reinforcement learning. As our main contribution, we build on recent advances in distributional reinforcement learning and propose a novel, tractable approximation of IDS for deep Q-learning. The resulting exploration strategy explicitly accounts for both parametric uncertainty and heteroscedastic observation noise. We evaluate our method on Atari games and demonstrate a significant improvement over alternative approaches.

我们开发了强化学习的信息导向抽样的实际扩展，它考虑了参数的不确定性和回报分布的异方差，以进行探索。

作者提出了一种将信息导向采样扩展到强化学习的方法，即结合两种不确定性，得到一种基于IDS的简单探索策略。

本文研究了建立在信息直接取样和分布式强化学习基础上的强化学习的复杂探索方法。


We address the problem of learning structured policies for continuous control. In traditional reinforcement learning, policies of agents are learned by MLPs which take the concatenation of all observations from the environment as input for predicting actions. In this work, we propose NerveNet to explicitly model the structure of an agent, which naturally takes the form of a graph. Specifically, serving as the agent's policy network, NerveNet first propagates information over the structure of the agent and then predict actions for different parts of the agent. In the experiments, we first show that our NerveNet is comparable to state-of-the-art methods on standard MuJoCo environments. We further propose our customized reinforcement learning environments for benchmarking two types of structure transfer learning tasks, i.e., size and disability transfer. We demonstrate that policies learned by NerveNet are significantly better than policies learned by other models and are able to transfer even in a zero-shot setting.

使用图谱神经网络对代理人的结构信息进行建模，以改善策略和可转移性
一种使用图谱神经网络表示和学习连续控制任务的结构化策略的方法
呈文提出将额外的结构纳入强化学习问题，特别是代理的形态结构
提出图形神经网络在学习控制不同长度的 "蜈蚣 "机器人的策略方面的应用。


Real-world tasks are often highly structured. Hierarchical reinforcement learning (HRL) has attracted research interest as an approach for leveraging the hierarchical structure of a given task in reinforcement learning (RL). However, identifying the hierarchical policy structure that enhances the performance of RL is not a trivial task. In this paper, we propose an HRL method that learns a latent variable of a hierarchical policy using mutual information maximization. Our approach can be interpreted as a way to learn a discrete and latent representation of the state-action space. To learn option policies that correspond to modes of the advantage function, we introduce advantage-weighted importance sampling.    In our HRL method, the gating policy learns to select option policies based on an option-value function, and these option policies are optimized based on the deterministic policy gradient method. This framework is derived by leveraging the analogy between a monolithic policy in standard RL and a hierarchical policy in HRL by using a deterministic option policy.   Experimental results indicate that our HRL approach can learn a diversity of options and that it can enhance the performance of RL in continuous control tasks.

本文提出了一个基于确定性选项策略和相互信息最大化的分层强化学习框架。

提出了一种HRL算法，该算法试图学习在最优政策下与状态行动密度相互信息最大化的选项。

本文提出了一个HRL系统，其中潜变量和状态-行动对的突变信息近似最大化。

提出了一个旨在使选项和状态动作对之间的相互信息最大化的标准，并通过经验表明，学到的选项分解了状态动作空间，但没有分解状态空间。


Deep neural networks (DNNs) have achieved impressive predictive performance due to their ability to learn complex, non-linear relationships between variables. However, the inability to effectively visualize these relationships has led to DNNs being characterized as black boxes and consequently limited their applications. To ameliorate this problem, we introduce the use of hierarchical interpretations to explain DNN predictions through our proposed method: agglomerative contextual decomposition (ACD). Given a prediction from a trained DNN, ACD produces a hierarchical clustering of the input features, along with the contribution of each cluster to the final prediction. This hierarchy is optimized to identify clusters of features that the DNN learned are predictive. We introduce ACD using examples from Stanford Sentiment Treebank and ImageNet, in order to diagnose incorrect predictions, identify dataset bias, and extract polarizing phrases of varying lengths. Through human experiments, we demonstrate that ACD enables users both to identify the more accurate of two DNNs and to better trust a DNN's outputs. We also find that ACD's hierarchy is largely robust to adversarial perturbations, implying that it captures fundamental aspects of the input and ignores spurious noise.

我们介绍并验证了分层局部解释，这是第一个自动搜索并显示LSTM和CNN所做的单个预测的重要交互作用的技术。

通过学习输入特征组的分层表征及其对最终预测的贡献来解释神经网络预测的一种新方法

将现有的LSTM的特征解释方法扩展到更通用的DNN，并引入了输入特征的分层聚类以及每个聚类对最终预测的贡献。

本文提出了背景分解的分层扩展。


Principal Filter Analysis (PFA) is an easy to implement, yet effective method for neural network compression. PFA exploits the intrinsic correlation between filter responses within network layers to recommend a smaller network footprint. We propose two compression algorithms: the first allows a user to specify the proportion of the original spectral energy that should be preserved in each layer after compression, while the second is a heuristic that leads to a parameter-free approach that automatically selects the compression used at each layer. Both algorithms are evaluated against several architectures and datasets, and we show considerable compression rates without compromising accuracy, e.g., for VGG-16 on CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x, and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively. In our tests we also demonstrate that networks compressed with PFA achieve an accuracy that is very close to the empirical upper bound for a given compression ratio. Finally, we show how PFA is an effective tool for simultaneous compression and domain adaptation.

我们提出了一种易于实现但有效的神经网络压缩方法。PFA利用网络层内滤波器响应之间的内在关联性来推荐一个更小的网络足迹。

建议通过分析同一层的过滤器之间观察到的相关性，即通过其协方差矩阵的特征值谱表示的相关性，来修整卷积网络。

本文介绍了一种压缩神经网络的方法，即通过两种策略考察每一层中滤波器响应的相关性。

本文提出了一种基于频谱分析的压缩方法


We propose a method to efficiently learn diverse strategies in reinforcement learning for query reformulation in the tasks of document retrieval and question answering. In the proposed framework an agent consists of multiple specialized sub-agents and a meta-agent that learns to aggregate the answers from sub-agents to produce a final answer. Sub-agents are trained on disjoint partitions of the training data, while the meta-agent is trained on the full training set. Our method makes learning faster, because it is highly parallelizable, and has better generalization performance than strong baselines, such as an ensemble of agents trained on the full data. We show that the improved performance is due to the increased diversity of reformulation strategies.

用强化学习训练的多种多样的查询重组代理，以改善搜索引擎。

在查询重构的重构学习中，集合方法的并行化，加快了训练速度，提高了学习到的自由格式的多样性。

作者提议训练多个不同的代理，每个代理在训练集的不同子集上训练。

作者提出了一种用于查询重构的集合方法


Network Embeddings (NEs) map the nodes of a given network into $d$-dimensional Euclidean space $\mathbb{R}^d$. Ideally, this mapping is such that 'similar' nodes are mapped onto nearby points, such that the NE can be used for purposes such as link prediction (if 'similar' means being 'more likely to be connected') or classification (if 'similar' means 'being more likely to have the same label'). In recent years various methods for NE have been introduced, all following a similar strategy: defining a notion of similarity between nodes (typically some distance measure within the network), a distance measure in the embedding space, and a loss function that penalizes large distances for similar nodes and small distances for dissimilar nodes.   A difficulty faced by existing methods is that certain networks are fundamentally hard to embed due to their structural properties: (approximate) multipartiteness, certain degree distributions, assortativity, etc. To overcome this, we introduce a conceptual innovation to the NE literature and propose to create \emph{Conditional Network Embeddings} (CNEs); embeddings that maximally add information with respect to given structural properties (e.g. node degrees, block densities, etc.). We use a simple Bayesian approach to achieve this, and propose a block stochastic gradient descent algorithm for fitting it efficiently.   We demonstrate that CNEs are superior for link prediction and multi-label classification when compared to state-of-the-art methods, and this without adding significant mathematical or computational complexity. Finally, we illustrate the potential of CNE for network visualization.

我们引入了一种网络嵌入方法，它考虑了网络的先验信息，产生了卓越的实证性能。

该论文提出使用先验分布来约束网络嵌入，对于该表述，该论文使用了非常有限的高斯分布。

提出了通过考虑网络的结构特性来学习无监督的节点嵌入。


This paper studies a class of adaptive gradient based momentum algorithms that update the  search directions and learning rates simultaneously using past gradients. This class, which we refer to as the ''``Adam-type'', includes the popular algorithms such as Adam, AMSGrad, AdaGrad. Despite their popularity in training deep neural networks (DNNs), the convergence of these algorithms for solving  non-convex problems remains an open question. In this paper, we develop an analysis framework and a set of mild sufficient conditions that guarantee the convergence of the Adam-type methods, with a convergence rate of order   $O(\log{T}/\sqrt{T})$ for non-convex stochastic optimization. Our convergence analysis applies to a new algorithm called AdaFom (AdaGrad with First Order Momentum). We show that the conditions are essential, by identifying concrete examples in which violating the conditions makes an algorithm diverge. Besides providing one of the first comprehensive analysis for Adam-type methods in the non-convex setting, our results can also help the practitioners to easily  monitor the progress of algorithms and determine their convergence behavior.

我们分析了亚当型算法的收敛性，并提供了保证其收敛性的温和的充分条件，我们还表明违反条件会使算法发生发散。

介绍了在非凸环境下对一系列优化算法的收敛性分析。

本文研究了亚当型优化器在无约束的非凸优化问题中的收敛条件。


This research paper describes a simplistic architecture named as AANN: Absolute Artificial Neural Network, which can be used to create highly interpretable representations of the input data. These representations are generated by penalizing the learning of the network in such a way that those learned representations correspond to the respective labels present in the labelled dataset used for supervised training; thereby, simultaneously giving the network the ability to classify the input data. The network can be used in the reverse direction to generate data that closely resembles the input by feeding in representation vectors as required. This research paper also explores the use of mathematical abs (absolute valued) functions as activation functions which constitutes the core part of this neural network architecture. Finally the results obtained on the MNIST dataset by using this technique are presented and discussed in brief.

以abs函数为激活函数的捆绑权重自动编码器，由于特别定义的成本函数，学会了在前向进行分类，在后向进行回归。

本文提出在自动编码器结构中使用绝对值激活函数，在目标函数中增加监督学习项

本文介绍了一个以绝对值作为激活函数的可逆网络。


Current state-of-the-art relation extraction methods typically rely on a set of lexical, syntactic, and semantic features, explicitly computed in a pre-processing step. Training feature extraction models requires additional annotated language resources, which severely restricts the applicability and portability of relation extraction to novel languages. Similarly, pre-processing introduces an additional source of error. To address these limitations, we introduce TRE, a Transformer for Relation Extraction, extending the OpenAI Generative Pre-trained Transformer [Radford et al., 2018]. Unlike previous relation extraction models, TRE uses pre-trained deep language representations instead of explicit linguistic features to inform the relation classification and combines it with the self-attentive Transformer architecture to effectively model long-range dependencies between entity mentions. TRE allows us to learn implicit linguistic features solely from plain text corpora by unsupervised pre-training, before fine-tuning the learned language representations on the relation extraction task. TRE obtains a new state-of-the-art result on the TACRED and SemEval 2010 Task 8 datasets, achieving a test F1 of 67.4 and 87.1, respectively. Furthermore, we observe a significant increase in sample efficiency. With only 20% of the training examples, TRE matches the performance of our baselines and our model trained from scratch on 100% of the TACRED dataset. We open-source our trained models, experiments, and source code.

我们提出了一个基于Transformer的关系提取模型，该模型使用预先训练好的语言表征，而不是明确的语言学特征。

提出了一个基于转化器的关系提取模型，该模型利用了对未标记文本的预训练，具有语言建模的目标。

本文介绍了Transformer网络在关系提取方面的一种新的应用。

本文提出了一个基于变形器的松弛提取架构，在两个数据集上进行了评估。


Neural networks have recently had a lot of success for many tasks. However, neural  network architectures that perform well are still typically designed manually  by experts in a cumbersome trial-and-error process. We propose a new method  to automatically search for well-performing CNN architectures based on a simple  hill climbing procedure whose operators apply network morphisms, followed  by short optimization runs by cosine annealing. Surprisingly, this simple method  yields competitive results, despite only requiring resources in the same order of  magnitude as training a single network. E.g., on CIFAR-10, our method designs  and trains networks with an error rate below 6% in only 12 hours on a single GPU;  training for one day reduces this error further, to almost 5%.

我们提出了一种简单而有效的卷积神经网络架构搜索方法。

提出了一种神经结构搜索方法，在CIFAR10上达到了接近最先进的准确度，而且占用的计算资源少得多。

提出了一种在训练的同时搜索神经网络架构的方法，极大地节省了训练时间和架构搜索时间。

提出了使用网络形态的神经架构搜索的变体，以定义使用CNN架构完成CIFAR图像分类任务的搜索空间


We propose GraphGAN - the first implicit generative model for graphs that enables to mimic real-world networks.  We pose the problem of graph generation as learning the distribution of biased random walks over a single input graph.  Our model is based on a stochastic neural network that generates discrete output samples, and is trained using the Wasserstein GAN objective. GraphGAN enables us to generate sibling graphs, which have similar properties yet are not exact replicas of the original graph. Moreover, GraphGAN learns a semantic mapping from the latent input space to the generated graph's properties. We discover that sampling from certain regions of the latent space leads to varying properties of the output graphs, with smooth transitions between them. Strong generalization properties of GraphGAN are highlighted by its competitive performance in link prediction as well as promising results on node classification, even though not specifically trained for these tasks.

使用GAN通过随机漫步生成图。

作者提出了一个图上随机漫步的生成模型，该模型可以进行模型诊断学习、可控拟合、集合图生成

提出了一个WGAN公式，用于生成基于随机漫步的图，使用节点嵌入和LSTM架构进行建模。


The ability of a classifier to recognize unknown inputs is important for many classification-based systems. We discuss the problem of simultaneous classification and novelty detection, i.e. determining whether an input is from the known set of classes and from which specific class, or from an unknown domain and does not belong to any of the known classes. We propose a method based on the Generative Adversarial Networks (GAN) framework. We show that a multi-class discriminator trained with a generator that generates samples from a mixture of nominal and novel data distributions is the optimal novelty detector. We approximate that generator with a mixture generator trained with the Feature Matching loss and empirically show that the proposed method outperforms conventional methods for novelty detection. Our findings demonstrate a simple, yet powerful new application of the GAN framework for the task of novelty detection.

我们提议在GAN框架内解决一个同时进行分类和新奇性检测的问题。

提出了一个GAN来统一分类和新颖性检测。

本文提出了一种基于多类GAN的新颖性检测方法，该方法被训练为输出由名义分布和新颖分布的混合物生成的图像。

本文提出了一个使用混合物生成器与特征匹配损失进行新颖性检测的GAN。


Verifying a person's identity based on their voice is a challenging, real-world problem in biometric security. A crucial requirement of such speaker verification systems is to be domain robust. Performance should not degrade even if speakers are talking in languages not seen during training. To this end, we present a flexible and interpretable framework for learning domain invariant speaker embeddings using Generative Adversarial Networks. We combine adversarial training with an angular margin loss function, which encourages the speaker embedding model to be discriminative by directly optimizing for cosine similarity between classes. We are able to beat a strong baseline system using a cosine distance classifier and a simple score-averaging strategy. Our results also show that models with adversarial adaptation perform significantly better than unadapted models. In an attempt to better understand this behavior, we quantitatively measure the degree of invariance induced by our proposed methods using Maximum Mean Discrepancy and Frechet distances. Our analysis shows that our proposed adversarial speaker embedding models significantly reduce the distance between source and target data distributions, while performing similarly on the former and better on the latter.

通过使用生成对抗网络使模型适应域内数据，可以显著提高说话人的验证性能。此外，这种适应可以以无监督的方式进行。

在领域不匹配条件下的说话人识别任务上提出了一些GAN变体。


Learning disentangling representations of the independent factors of variations that explain the data in an unsupervised setting is still a major challenge. In the following paper we address the task of disentanglement and introduce a new state-of-the-art approach called Non-synergistic variational Autoencoder (Non-Syn VAE). Our model draws inspiration from population coding, where the notion of synergy arises when we describe the encoded information by neurons in the form of responses from the stimuli. If those responses convey more information together than separate as independent sources of encoding information, they are acting synergetically. By penalizing the synergistic mutual information within the latents we encourage information independence and by doing that disentangle the latent factors. Notably, our approach could be added to the VAE framework easily, where the new ELBO function is still a lower bound on the log likelihood. In addition, we qualitatively compare our model with Factor VAE and show that this one implicitly minimises the synergy of the latents.

使用VAE框架使潜在和数据内的协同互信息最小化，以完成拆分的任务。

提出了一个新的目标函数，通过最小化所提供信息的协同作用，在变分框架中学习解缠表征。

作者的目标是训练一个以 "协同 "的最大方式拆分潜伏表征的VAE。

本文提出了一种新的方法来强制执行VAE中的disentanglement，使用的术语是惩罚潜变量之间的协同互信息。


Metric embeddings are   immensely useful representations of associations between entities   (images, users, search queries, words, and more).   Embeddings are learned by  optimizing a loss objective of the general form of a sum over example associations. Typically, the optimization uses stochastic gradient updates over minibatches of examples that are arranged  independently at random. In this work, we propose the use of {\em structured arrangements} through randomized {\em microbatches} of examples that are more likely to include similar ones. We make a principled argument for the properties of our arrangements  that accelerate the training and present efficient algorithms to generate microbatches that respect the marginal  distribution of training examples.   Finally, we observe experimentally that our structured arrangements accelerate training by 3-20\%. Structured arrangements emerge as a powerful and novel performance knob for SGD that is independent and complementary to other SGD  hyperparameters and thus is a candidate for wide deployment.

通过安排不同的例子来加速SGD

本文提出了一种通过将相似的训练样本分组来提高随机梯度下降法学习嵌入的收敛率的方法。

提出了一种非均匀抽样策略，在SGD中为学习对象关联的嵌入任务构建迷你批。


Ubuntu dialogue corpus is the largest public available dialogue corpus to make it feasible to build end-to-end deep neural network models directly from the conversation data. One challenge of Ubuntu dialogue corpus is  the large number of out-of-vocabulary words. In this paper we proposed an algorithm which combines the general pre-trained word embedding vectors with those  generated on the task-specific training set to address this issue.   We integrated character embedding into Chen et al's Enhanced LSTM method (ESIM) and used it to evaluate the effectiveness of our proposed method. For the task of next utterance selection, the proposed method has demonstrated a significant performance improvement against original ESIM and the new model has achieved state-of-the-art results on both Ubuntu dialogue corpus and Douban conversation corpus. In addition, we investigated the performance impact of end-of-utterance and end-of-turn token tags.

结合预先建立的单词嵌入和特定任务的单词表示之间的信息，以解决词汇量不足的问题

本文提出了一种改进词汇外嵌入预测的方法，用于模拟对话会话的任务，比基线有相当大的收益。

建议将外部预训练的词嵌入和训练数据上的预训练词嵌入结合起来，将它们作为两个视图保留。

提出了扩大预训练词嵌入覆盖范围的方法，以处理将其应用于对话数据集时出现的OOV问题，并将基于LSTM模型的新变体应用于对话建模中的反应选择任务。


Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks. Analogously, this suggests that learned update functions may similarly outperform current hand-designed optimizers, especially for specific tasks. However, learned optimizers are notoriously difficult to train and have yet to demonstrate wall-clock speedups over hand-designed optimizers, and thus are rarely used in practice. Typically, learned optimizers are trained by truncated backpropagation through an unrolled optimization process. The resulting gradients are either strongly biased (for short truncations) or have exploding norm (for long truncations). In this work we propose a training scheme which overcomes both of these difficulties, by dynamically weighting two unbiased gradient estimators for a variational loss on optimizer performance. This allows us to train neural networks to perform optimization faster than well tuned first-order methods. Moreover, by training the optimizer against validation loss, as opposed to training loss, we are able to use it to train models which generalize better than those trained by first order methods. We demonstrate these results on problems where our learned optimizer trains convolutional networks in a fifth of the wall-clock time compared to tuned first-order methods, and with an improvement

我们分析了训练学习型优化器时的问题，通过使用两个互补的梯度估计器的变分优化来解决这些问题，并训练出比基线优化器（如Adam）快5倍壁钟时间的优化器。

本文使用非滚动优化来学习神经网络进行优化。

本文解决了学习优化器的问题，特别是作者专注于从未滚动的训练程序中获得更清洁的梯度。

提出了一种通过对 "外部 "优化器损失使用变分优化来 "学习优化器 "的方法，并提出了将变分目标的再参数化梯度和得分函数估计器结合起来的想法，并使用高斯乘积公式对均值进行加权。


Asynchronous distributed gradient descent algorithms for training of deep neural  networks are usually considered as inefficient, mainly because of the Gradient delay  problem. In this paper, we propose a novel asynchronous distributed algorithm  that tackles this limitation by well-thought-out averaging of model updates, computed  by workers. The algorithm allows computing gradients along the process  of gradient merge, thus, reducing or even completely eliminating worker idle time  due to communication overhead, which is a pitfall of existing asynchronous methods.  We provide theoretical analysis of the proposed asynchronous algorithm,  and show its regret bounds. According to our analysis, the crucial parameter for  keeping high convergence rate is the maximal discrepancy between local parameter  vectors of any pair of workers. As long as it is kept relatively small, the  convergence rate of the algorithm is shown to be the same as the one of a sequential  online learning. Furthermore, in our algorithm, this discrepancy is bounded  by an expression that involves the staleness parameter of the algorithm, and is  independent on the number of workers. This is the main differentiator between  our approach and other solutions, such as Elastic Asynchronous SGD or Downpour  SGD, in which that maximal discrepancy is bounded by an expression that  depends on the number of workers, due to gradient delay problem. To demonstrate  effectiveness of our approach, we conduct a series of experiments on image  classification task on a cluster with 4 machines, equipped with a commodity communication  switch and with a single GPU card per machine. Our experiments  show a linear scaling on 4-machine cluster without sacrificing the test accuracy,  while eliminating almost completely worker idle time. Since our method allows  using commodity communication switch, it paves a way for large scale distributed  training performed on commodity clusters.

一种高效的深度学习模型异步分布式训练方法，以及理论上的遗憾界限。 本文提出了一种限制异步SGD中滞后性的算法，并提供了理论分析 提出了一种混合算法来消除异步方法的梯度延迟。


Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs. However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks. This paper aims to raise people's awareness about the security of the quantized models, and we designed a novel quantization methodology to jointly optimize the efficiency and robustness of deep learning models. We first conduct an empirical study to show that vanilla quantization suffers more from adversarial attacks. We observe that the inferior robustness comes from the error amplification effect, where the quantization operation further enlarges the distance caused by amplified noise. Then we propose a novel Defensive Quantization (DQ) method by controlling the Lipschitz constant of the network during quantization, such that the magnitude of the adversarial noise remains non-expansive during inference. Extensive experiments on CIFAR-10 and SVHN datasets demonstrate that our new quantization method can defend neural networks against adversarial examples, and even achieves superior robustness than their full-precision counterparts, while maintaining the same hardware efficiency as vanilla quantization approaches. As a by-product, DQ can also improve the accuracy of quantized models without adversarial attack.

我们设计了一种新颖的量化方法，以共同优化深度学习模型的效率和稳健性。

提出了一种正则化方案，利用内层输出的Lipschitz常数过滤来保护量化的神经网络免受对抗性攻击。


Recurrent Neural Networks (RNNs) continue to show  outstanding performance in sequence modeling tasks. However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models. Source code is publicly available at https://imatge-upc.github.io/skiprnn-2017-telecombcn/.

对现有的RNN架构进行修改，使其能够跳过状态更新，同时保留了原始架构的性能。

提出了跳过RNN模型，该模型允许循环网络有选择地跳过更新某些输入的隐藏状态，从而减少测试时的计算。

提出了一种新的RNN模型，其中循环单元的输入和状态更新在某些时间段内被自适应地跳过。


We propose a fast second-order method that can be used as a drop-in replacement for current deep learning solvers. Compared to stochastic gradient descent (SGD), it only requires two additional forward-mode automatic differentiation operations per iteration, which has a computational cost comparable to two standard forward passes and is easy to implement. Our method addresses long-standing issues with current second-order solvers, which invert an approximate Hessian matrix every iteration exactly or by conjugate-gradient methods, procedures that are much slower than a SGD step. Instead, we propose to keep a single estimate of the gradient projected by the inverse Hessian matrix, and update it once per iteration with just two passes over the network. This estimate has the same size and is similar to the momentum variable that is commonly used in SGD. No estimate of the Hessian is maintained.  We first validate our method, called CurveBall, on small problems with known solutions (noisy Rosenbrock function and degenerate 2-layer linear networks), where current deep learning solvers struggle. We then train several large models on CIFAR and ImageNet, including ResNet and VGG-f networks, where we demonstrate faster convergence with no hyperparameter tuning. We also show our optimiser's generality by testing on a large set of randomly-generated architectures.

用于深度学习的快速二阶求解器，可用于ImageNet规模的问题，无需进行超参数调整

通过使用单步梯度下降 "朝向牛顿步 "从最初的估计值选择方向，然后采取这个方向而不是原始梯度

一种新的近似二阶优化方法，计算成本低，用单一梯度步骤和热启动策略取代了Hessian矩阵的计算。


The recently presented idea to learn heuristics for combinatorial optimization problems is promising as it can save costly development. However, to push this idea towards practical implementation, we need better models and better ways of training. We contribute in both directions: we propose a model based on attention layers with benefits over the Pointer Network and we show how to train this model using REINFORCE with a simple baseline based on a deterministic greedy rollout, which we find is more efficient than using a value function. We significantly improve over recent learned heuristics for the Travelling Salesman Problem (TSP), getting close to optimal results for problems up to 100 nodes. With the same hyperparameters, we learn strong heuristics for two variants of the Vehicle Routing Problem (VRP), the Orienteering Problem (OP) and (a stochastic variant of) the Prize Collecting TSP (PCTSP), outperforming a wide range of baselines and getting results close to highly optimized and specialized algorithms.

用REINFORCE训练的基于注意力的模型，以贪婪的推出为基线，学习启发式方法，在TSP和其他路由问题上取得有竞争力的结果。

介绍了一种基于注意力的方法来学习解决TSP和其他路由型组合优化问题的策略。

本文试图学习用于解决组合优化问题的启发式方法


We propose an efficient online hyperparameter optimization method which uses a joint dynamical system to evaluate the gradient with respect to the hyperparameters. While similar methods are usually limited to hyperparameters with a smooth impact on the model, we show how to apply it to the probability of dropout in neural networks. Finally, we show its effectiveness on two distinct tasks.

训练期间优化正则化超参数的算法

本文提出了一种在每次更新lambda时重新初始化y的方法和y的剪切程序，以保持动态系统的稳定性。

提出了一种超参数优化的算法，可以看作是Franceschi 2017年的扩展，是对一些估计值进行了热重启，以提高方法的稳定性。

提出了对现有方法的扩展，以优化正则化超参数。


Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks. However, these have been evaluated using differing codebases and limited computational resources, which represent uncontrolled sources of experimental variation. We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models. We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the Hutter Prize dataset.

表明LSTM对LM来说和最近的创新一样好，甚至更好，而模型评估往往是不可靠的。

本文描述了对基于LSTM的单词和字符语言模型的全面验证，导致了语言建模的重大成果，也是深度学习的一个里程碑。


Residual and skip connections play an important role in many current   generative models. Although their theoretical and numerical advantages   are understood, their role in speech enhancement systems has not been   investigated so far. When performing spectral speech enhancement,   residual connections are very similar in nature to spectral subtraction,   which is the one of the most commonly employed speech enhancement approaches.    Highway networks, on the other hand, can be seen as a combination of spectral   masking and spectral subtraction. However, when using deep neural networks, such operations would   normally happen in a transformed spectral domain, as opposed to traditional speech   enhancement where all operations are often done directly on the spectrum.    In this paper, we aim to investigate the role of residual and highway   connections in deep neural networks for speech enhancement, and verify whether   or not they operate similarly to their traditional, digital signal processing   counterparts. We visualize the outputs of such connections, projected back to   the spectral domain, in models trained for speech denoising, and show that while   skip connections do not necessarily improve performance with regards to the   number of parameters, they make speech enhancement models more interpretable.

我们展示了使用跳过连接可以使语音增强模型更具可解释性，因为它使模型使用了DSP文献中已经探索过的类似机制。

作者提出在完全卷积管道内加入残差、Highway和Masking block，以了解在语音增强任务中如何进行输出和掩码的迭代推理。

作者通过人为地将噪音从一个完善的噪音数据集添加到一个不太为人所知的干净的语音数据集上，产生了他们自己的噪音语音。


Bayesian neural networks (BNNs) hold great promise as a flexible and principled solution to deal with uncertainty when learning from finite data. Among approaches to realize probabilistic inference in deep neural networks, variational Bayes (VB) is theoretically grounded, generally applicable, and computationally efficient. With wide recognition of potential advantages, why is it that variational Bayes has seen very limited practical use for BNNs in real applications? We argue that variational inference in neural networks is fragile: successful implementations require careful initialization and tuning of prior variances, as well as controlling the variance of Monte Carlo gradient estimates. We provide two innovations that aim to turn VB into a robust inference tool for Bayesian neural networks: first, we introduce a novel deterministic method to approximate moments in neural networks, eliminating gradient variance; second, we introduce a hierarchical prior for parameters and a novel Empirical Bayes procedure for automatically selecting prior variances. Combining these two innovations, the resulting method is highly efficient and robust. On the application of heteroscedastic regression we demonstrate good predictive performance over alternative approaches.

为有效训练贝叶斯神经网络而消除梯度方差和自动调整预案的方法

提出了一种新的方法，通过近似层状矩，对具有特定非线性激活函数的前馈BNN进行确定性的变分推理。

本文考虑了一种纯粹的确定性方法来学习贝叶斯神经网络的变分后验近似值。


Skills learned through (deep) reinforcement learning often generalizes poorly  across tasks and re-training is necessary when presented with a new task. We  present a framework that combines techniques in formal methods with reinforcement  learning (RL) that allows for the convenient specification of complex temporal  dependent tasks with logical expressions and construction of new skills from existing  ones with no additional exploration. We provide theoretical results for our  composition technique and evaluate on a simple grid world simulation as well as  a robotic manipulation task.

强化学习任务中技能构成的正式方法方法

本文通过设置scTLTL公式的自动化，将RL和由逻辑公式表达的约束结合起来。

提出了一种有助于从学习到的子任务中构建策略的方法，其主题是将RL任务与线性时间逻辑公式相结合。


The application of multi-modal generative models by means of a Variational Auto Encoder (VAE) is an upcoming research topic for sensor fusion and bi-directional modality exchange.  This contribution gives insights into the learned joint latent representation and shows that expressiveness and coherence are decisive properties for multi-modal datasets.  Furthermore, we propose a multi-modal VAE derived from the full joint marginal log-likelihood that is able to learn the most meaningful representation for ambiguous observations.  Since the properties of multi-modal sensor setups are essential for our approach but hardly available, we also propose a technique to generate correlated datasets from uni-modal ones.

从联合边际对数似然推导出多模态VAE的一般表述。

提出了一个多模态的VAE，其变分约束来自于连锁规则。

本文提出了一个针对多模态VAE的目标，即M^2VAE，它应该是学习一个更有意义的潜在空间表示。


We build on auto-encoding sequential Monte Carlo (AESMC): a method for model and proposal learning based on maximizing the lower bound to the log marginal likelihood in a broad family of structured probabilistic models. Our approach relies on the efficiency of sequential Monte Carlo (SMC) for performing inference in structured probabilistic models and the flexibility of deep neural networks to model complex conditional probability distributions. We develop additional theoretical insights and introduce a new training procedure which improves both model and proposal learning. We demonstrate that our approach provides a fast, easy-to-implement and scalable means for simultaneous model learning and proposal adaptation in deep generative models.

我们在自动编码顺序蒙特卡洛的基础上，获得了新的理论见解，并在这些见解的基础上开发了一个改进的训练程序。

本文提出了一个IWAE式训练的版本，使用SMC代替经典的重要性采样。

这项工作提出了自动编码的顺序蒙特卡洛（SMC），将VAE框架扩展到基于SMC的新蒙特卡洛目标。


A key component for many reinforcement learning agents is to learn a value function, either for policy evaluation or control. Many of the algorithms for learning values, however, are designed for linear function approximation---with a fixed basis or fixed representation. Though there have been a few sound extensions to nonlinear function approximation, such as nonlinear gradient temporal difference learning, these methods have largely not been adopted, eschewed in favour of simpler but not sound methods like temporal difference learning and Q-learning. In this work, we provide a two-timescale network (TTN) architecture that enables linear methods to be used to learn values, with a nonlinear representation learned at a slower timescale. The approach facilitates the use of algorithms developed for the linear setting, such as data-efficient least-squares methods, eligibility traces and the myriad of recently developed linear policy evaluation algorithms, to provide nonlinear value estimates. We prove convergence for TTNs, with particular care given to ensure convergence of the fast linear component under potentially dependent features provided by the learned representation. We empirically demonstrate the benefits of TTNs, compared to other nonlinear value function approximation algorithms, both for policy evaluation and control.

我们提出了一个学习价值函数的架构，它允许使用任何线性策略评估算法与非线性特征学习相配合。

本文提出了一个学习价值函数的双时间尺度框架，以及一个完全由非线性近似器构成的状态表征。

本文提出了双时间尺度网络（TTNs），并使用来自两个时间尺度随机近似的方法证明了这种方法的收敛性。

本文提出了一个双时间尺度网络（TTN），使线性方法可以用来学习数值。


Large-scale Long Short-Term Memory (LSTM) cells are often the building blocks of many state-of-the-art algorithms for tasks in Natural Language Processing (NLP). However, LSTMs are known to be computationally inefficient because the memory capacity of the models depends on the number of parameters, and the inherent recurrence that models the temporal dependency is not parallelizable. In this paper, we propose simple, but effective, low-rank matrix factorization (MF) algorithms to compress network parameters and significantly speed up LSTMs with almost no loss of performance (and sometimes even gain). To show the effectiveness of our method across different tasks, we examine two settings: 1) compressing core LSTM layers in Language Models, 2) compressing biLSTM layers of ELMo~\citep{ELMo} and evaluate in three downstream NLP tasks (Sentiment Analysis, Textual Entailment, and Question Answering). The latter is particularly interesting as embeddings from large pre-trained biLSTM Language Models are often used as contextual word representations. Finally, we discover that matrix factorization performs better in general, additive recurrence is often more important than multiplicative recurrence, and we identify an interesting correlation between matrix norms and compression performance.

我们提出了简单而有效的低秩矩阵分解（MF）算法，以加快运行时间，节省内存，并提高LSTM的性能。

建议通过使用MF作为后处理压缩策略来加速LSTM，并进行了大量的实验来展示其性能。


Manipulation and re-use of images in scientific publications is a recurring problem, at present lacking a scalable solution.   Existing tools for detecting image duplication are mostly manual or semi-automated, despite the fact that generating data for a learning-based approach is straightforward, as we here illustrate. This paper addresses the problem of determining if, given two images, one is a manipulated version of the other by means of certain geometric and statistical manipulations, e.g. copy, rotation, translation, scale, perspective transform, histogram adjustment, partial erasing, and compression artifacts. We propose a solution based on a 3-branch Siamese Convolutional Neural Network. The ConvNet model is trained to map images into a 128-dimensional space, where the Euclidean distance between duplicate (respectively, unique) images is no greater (respectively, greater) than 1. Our results suggest that such an approach can serve as tool to improve surveillance of the published and in-peer-review literature for image manipulation. We also show that as a byproduct the network learns useful representations for semantic segmentation, with performance comparable to that of domain-specific models.

一个法医指标，以确定一个给定的图像是否是来自给定数据集的另一个图像的副本（可能被操纵）。

介绍了识别重复和复制/修改图像的siamese网络，可用于改善对已发表和同行评议文献的监控。

本文介绍了深度卷积网络在重复图像检测任务中的应用

这项工作解决了从生物医学出版物中寻找重复/接近重复的图像的问题，提出了一个标准的CNN和损失函数，并将其应用于这个领域。


Training generative adversarial networks is unstable in high-dimensions as the true data distribution tends to be concentrated in a small fraction of the ambient space. The discriminator is then quickly able to classify nearly all generated samples as fake, leaving the generator without meaningful gradients and causing it to deteriorate after a point in training. In this work, we propose training a single generator simultaneously against an array of discriminators, each of which looks at a different random low-dimensional projection of the data. Individual discriminators, now provided with restricted views of the input, are unable to reject generated samples perfectly and continue to provide meaningful gradients to the generator throughout training.   Meanwhile, the generator learns to produce samples consistent with the full data distribution to satisfy all discriminators simultaneously. We  demonstrate the practical utility of this approach experimentally, and show that it is able to produce image samples with higher quality than traditional training with a single discriminator.

通过使用判别器阵列在高维度上进行稳定的GAN训练，每个判别器都有生成样本的低维视图

本文提出通过使用鉴别器的集合来稳定GAN的训练，每个鉴别器在输入数据的随机投影上工作，为生成器模型提供训练信号。

本文提出了一种提高训练稳定性的GAN训练方法。

本文提出了一种新的GAN训练方法，它提供了稳定的梯度来训练发生器。


We present a novel method to precisely impose tree-structured category information onto word-embeddings, resulting in ball embeddings in higher dimensional spaces (N-balls for short). Inclusion relations among N-balls implicitly encode subordinate relations among categories. The similarity measurement in terms of the cosine function is enriched by category information. Using a geometric construction method instead of back-propagation, we create large N-ball embeddings that satisfy two conditions: (1) category trees are precisely imposed onto word embeddings at zero energy cost; (2) pre-trained word embeddings are well preserved. A new benchmark data set is created for validating the category of unknown words. Experiments show that N-ball embeddings, carrying category information, significantly outperform word embeddings in the test of nearest neighborhoods, and demonstrate surprisingly good performance in validating categories of unknown words. Source codes and data-sets are free for public access \url{https://github.com/gnodisnait/nball4tree.git} and \url{https://github.com/gnodisnait/bp94nball.git}.

我们展示了一种几何方法，将类别树信息完美地编码到预先训练好的词嵌入中。

本文提出了分类数据的N球嵌入，其中N球是一对中心点向量和离中心点的半径。

本文介绍了一种调整现有分类对象（如单词）的向量嵌入的方法，将其转换为遵循层次结构的球状嵌入。

专注于调整预训练的词嵌入，使其通过适当的n-ball封装来尊重hypernymy/hyponymy关系。


For the challenging semantic image segmentation task the best performing models  have traditionally combined the structured modelling capabilities of Conditional  Random Fields (CRFs) with the feature extraction power of CNNs. In more recent  works however, CRF post-processing has fallen out of favour. We argue that this  is mainly due to the slow training and inference speeds of CRFs, as well as the  difficulty of learning the internal CRF parameters. To overcome both issues we  propose to add the assumption of conditional independence to the framework of  fully-connected CRFs. This allows us to reformulate the inference in terms of  convolutions, which can be implemented highly efficiently on GPUs.Doing so  speeds up inference and training by two orders of magnitude. All parameters of  the convolutional CRFs can easily be optimized using backpropagation. Towards  the goal of facilitating further CRF research we have made our implementations  publicly available.

我们提出卷积CRF是全连接CRF的快速、强大和可训练的替代方案。

作者用一个空间变化的卷积核取代了permutohedral网格中的大型过滤步骤，并表明推理更有效，训练更容易。

建议在截断的高斯核CRF上进行消息传递，使用定义的内核和GPU上的并行化消息传递。


Deep Learning NLP domain lacks procedures for the analysis of model robustness. In this paper we propose a framework which validates robustness of any Question Answering model through model explainers. We propose that output of a robust model should be invariant to alterations that do not change its semantics. We test this property by manipulating question in two ways: swapping important question word for 1) its semantically correct synonym and 2) for word vector that is close in embedding space. We estimate importance of words in asked questions with Locally Interpretable Model Agnostic Explanations method (LIME). With these two steps we compare state-of-the-art Q&A models. We show that although accuracy of state-of-the-art models is high, they are very fragile to changes in the input. We can choose architecture that is more immune to attacks and thus more robust and stable in production environment. Morevoer, we propose 2 adversarial training scenarios which raise model sensitivity to true synonyms by up to 7% accuracy measure. Our findings help to understand which models are more stable and how they can be improved. In addition, we have created and published a new dataset that may be used for validation of robustness of a Q&A model.

我们提出了一种与模型无关的方法来验证问答系统的稳健性，并在最先进的问答模型上展示结果。

解决了问题回答中对对抗性信息的稳健性问题。

提高机器理解/问题回答的稳健性。


In this paper, we propose a mix-generator generative adversarial networks (PGAN) model that works in parallel by mixing multiple disjoint generators to approximate a complex real distribution. In our model, we propose an adjustment component that collects all the generated data points from the generators, learns the boundary between each pair of generators, and provides error to separate the support of each of the generated distributions. To overcome the instability in a multiplayer game, a shrinkage adjustment component method is introduced to gradually reduce the boundary between generators during the training procedure. To address the linearly growing training time problem in a multiple generators model, we propose a method to train the generators in parallel. This means that our work can be scaled up to large parallel computation frameworks. We present an efficient loss function for the discriminator, an effective adjustment component, and a suitable generator. We also show how to introduce the decay factor to stabilize the training procedure. We have performed extensive experiments on synthetic datasets, MNIST, and CIFAR-10. These experiments reveal that the error provided by the adjustment component could successfully separate the generated distributions and each of the generators can stably learn a part of the real distribution even if only a few modes are contained in the real distribution.

多发生器捕捉Pdata，解决竞争和一锤定音的问题 提出了并行的GAN，通过多个弱生成器的组合来避免GAN的模式崩溃。


We capitalize on the natural compositional structure of images in order to learn object segmentation with weakly labeled images. The intuition behind our approach is that removing objects from images will yield natural images, however removing random patches will yield unnatural images. We leverage this signal to develop a generative model that decomposes an image into layers, and when all layers are combined, it reconstructs the input image. However, when a layer is removed, the model learns to produce a different image that still looks natural to an adversary, which is possible by removing objects. Experiments and visualizations suggest that this model automatically learns object segmentation on images labeled only by scene better than baselines.

利用图像的组成结构和生成模型进行弱监督的图像分割。

本文创建了一个分层表示法，以便更好地从未标记的图像中学习分割。

本文提出了一个基于GAN的生成模型，该模型将图像分解成多个层次，其中GAN的目标是将真实的图像和由各层组合形成的图像区分开来。

本文围绕分层场景构成的理念提出了一个神经网络架构


Adversarial examples are a pervasive phenomenon of machine learning models where seemingly imperceptible perturbations to the input lead to misclassifications for otherwise statistically accurate models. We propose a geometric framework, drawing on tools from the manifold reconstruction literature, to analyze the high-dimensional geometry of adversarial examples. In particular, we highlight the importance of codimension: for low-dimensional data manifolds embedded in high-dimensional space there are many directions off the manifold in which to construct adversarial examples. Adversarial examples are a natural consequence of learning a decision boundary that classifies the low-dimensional data manifold well, but classifies points near the manifold incorrectly. Using our geometric framework we prove (1) a tradeoff between robustness under different norms, (2) that adversarial training in balls around the data is sample inefficient, and (3) sufficient sampling conditions under which nearest neighbor classifiers and ball-based adversarial training are robust.

我们提出了一个证明鲁棒性保证的几何框架，并强调了在对抗性例子中余维度的重要性。

本文对对抗性例子进行了理论分析，表明在不同的范数下存在稳健性的权衡，对抗性训练的样本效率很低，而最近的邻居分类器在某些条件下可以是稳健的。


Character-based neural machine translation (NMT) models alleviate out-of-vocabulary issues, learn morphology, and move us closer to completely end-to-end translation systems.   Unfortunately, they are also very brittle and easily falter when presented with noisy data. In this paper, we confront NMT models with synthetic and natural sources of noise. We find that state-of-the-art models fail to translate even moderately noisy texts that humans have no trouble comprehending. We explore two approaches to increase model robustness: structure-invariant word representations and robust training on noisy texts. We find that a model based on a character convolutional neural network is able to simultaneously learn representations robust to multiple kinds of noise.

CharNMT是脆性的 本文研究了字符级噪声对4种不同的神经机器翻译系统的影响 本文实证研究了字符级NMT系统在面对字符级噪声时的表现，包括合成的和自然的。 本文研究了噪声输入对机器翻译的影响，并测试了使NMT模型更加稳健的方法。


As neural networks grow deeper and wider, learning networks with hard-threshold activations is becoming increasingly important, both for network quantization, which can drastically reduce time and energy requirements, and for creating large integrated systems of deep networks, which may have non-differentiable components and must avoid vanishing and exploding gradients for effective learning. However, since gradient descent is not applicable to hard-threshold functions, it is not clear how to learn them in a principled way. We address this problem by observing that setting targets for hard-threshold hidden units in order to minimize loss is a discrete optimization problem, and can be solved as such. The discrete optimization goal is to find a set of targets such that each unit, including the output, has a linearly separable problem to solve. Given these targets, the network decomposes into individual perceptrons, which can then be learned with standard convex approaches. Based on this, we develop a recursive mini-batch algorithm for learning deep hard-threshold networks that includes the popular but poorly justified straight-through estimator as a special case. Empirically, we show that our algorithm improves classification accuracy in a number of settings, including for AlexNet and ResNet-18 on ImageNet, when compared to the straight-through estimator.

我们通过使用组合优化设置隐藏单元目标和凸优化设置权重来学习硬阈值单元的深度网络，从而提高了在ImageNet上的性能。

本文解释并概括了学习硬激活神经网的方法。

本文研究了优化硬阈值单元的深度网络问题。

本文讨论了具有硬阈值的神经网络的优化问题，并提出了一个具有启发式/近似式集合的新颖解决方案。


The robust and efficient recognition of visual relations in images is a hallmark of biological vision. Here, we argue that, despite recent progress in visual recognition, modern machine vision algorithms are severely limited in their ability to learn visual relations. Through controlled experiments, we demonstrate that visual-relation problems strain convolutional neural networks (CNNs). The networks eventually break altogether when rote memorization becomes impossible such as when the intra-class variability exceeds their capacity. We further show that another type of feedforward network, called a relational network (RN), which was shown to successfully solve seemingly difficult visual question answering (VQA) problems on the CLEVR datasets, suffers similar limitations. Motivated by the comparable success of biological vision, we argue that feedback mechanisms including working memory and attention are the key computational components underlying abstract visual reasoning.

我们使用一个新的、受控的、视觉关系的挑战，表明相同-不同的任务严重地限制了CNN的能力；我们认为，视觉关系可以用注意力记忆策略更好地解决。

通过在人工生成的视觉关系数据上训练网络，证明卷积和关系神经网络无法解决视觉关系问题。

本文探讨了目前的CNN和关系网络如何无法识别图像中的视觉关系。