Deep learning has found numerous applications thanks to its versatility and accuracy on pattern recognition problems such as visual object detection. Learning and inference in deep neural networks, however, are memory and compute intensive and so improving efficiency is one of the major challenges for frameworks such as PyTorch, Tensorflow, and Caffe. While the efficiency problem can be partially addressed with specialized hardware and its corresponding proprietary libraries, we believe that neural network acceleration should be transparent to the user and should support all hardware platforms and deep learning libraries.    To this end, we introduce a transparent middleware layer for neural network acceleration. The system is built around a compiler for deep learning, allowing one to combine device-specific libraries and custom optimizations while supporting numerous hardware devices. In contrast to other projects, we explicitly target the optimization of both prediction and training of neural networks. We present the current development status and some preliminary but encouraging results: on a standard x86 server, using CPUs our system achieves a 11.8x speed-up for inference and a 8.0x for batched-prediction (128); on GPUs we achieve a 1.7x and 2.3x speed-up respectively.

我们引入了一个透明的神经网络加速中间件，它有自己的编译器引擎，在CPU上实现了11.8倍的速度，在GPU上实现了2.3倍的速度。

本文提出了一个用于神经网络加速的透明中间件层，并在基本的CPU和GPU架构上获得了一些加速结果


Performance of neural networks can be significantly improved by encoding known invariance for particular tasks. Many image classification tasks, such as those related to cellular imaging, exhibit invariance to rotation. In particular, to aid convolutional neural networks in learning rotation invariance, we consider a simple, efficient conic convolutional scheme that encodes rotational equivariance, along with a method for integrating the magnitude response of the 2D-discrete-Fourier transform (2D-DFT) to encode global rotational invariance. We call our new method the Conic Convolution and DFT Network (CFNet). We evaluated the efficacy of CFNet as compared to a standard CNN and group-equivariant CNN (G-CNN) for several different image classification tasks and demonstrated improved performance, including classification accuracy, computational efficiency, and its robustness to hyperparameter selection. Taken together, we believe CFNet represents a new scheme that has the potential to improve many imaging analysis applications.

我们提出了圆锥卷积和2D-DFT来将旋转等方差编码到神经网络中。

在图像分类的背景下，本文提出了一种卷积神经网络架构，该架构具有旋转等变特征图，最终通过使用2D离散傅里叶变换（DFT）的大小使旋转不变。

作者通过结合圆锥卷积和2D-DFT提供了一个旋转不变的神经网络


The problem of visual metamerism is defined as finding a family of perceptually  indistinguishable, yet physically different images. In this paper, we propose our  NeuroFovea metamer model, a foveated generative model that is based on a mixture  of peripheral representations and style transfer forward-pass algorithms. Our  gradient-descent free model is parametrized by a foveated VGG19 encoder-decoder  which allows us to encode images in high dimensional space and interpolate  between the content and texture information with adaptive instance normalization  anywhere in the visual field. Our contributions include: 1) A framework for computing metamers that resembles a noisy communication system via a foveated feed-forward encoder-decoder network – We observe that metamerism arises as a byproduct of noisy perturbations that partially lie in the perceptual null space; 2) A perceptual optimization scheme as a solution to the hyperparametric nature of our metamer model that requires tuning of the image-texture tradeoff coefficients everywhere in the visual field which are a consequence of internal noise; 3) An  ABX psychophysical evaluation of our metamers where we also find that the rate  of growth of the receptive fields in our model match V1 for reference metamers  and V2 between synthesized samples. Our model also renders metamers at roughly  a second, presenting a ×1000 speed-up compared to the previous work, which now  allows for tractable data-driven metamer experiments.

我们引入了一种新的前馈框架来生成视觉metamer

提出了一个NeuroFovea模型，通过编码器-解码器风格架构，使用风格转移方法，生成固定点metamer

对同色异谱的分析和一个能够快速产生对实验心理物理学和其他领域有价值的metamer模型。

该论文提出了一种快速的方法，用于通过注视点，快速，任意的样式转移来生成视觉同色异构体 - 物理上不同的图像，无法与原始图像分开。


Past works have shown that, somewhat surprisingly, over-parametrization can help generalization in neural networks. Towards explaining this phenomenon, we adopt a margin-based perspective. We establish: 1) for multi-layer feedforward relu networks, the global minimizer of a weakly-regularized cross-entropy loss has the maximum normalized margin among all networks, 2) as a result, increasing the over-parametrization improves the normalized margin and generalization error bounds for deep networks. In the case of two-layer networks, an infinite-width neural network enjoys the best generalization guarantees. The typical infinite feature methods are kernel methods; we compare the neural net margin with that of kernel methods and construct natural instances where kernel methods have much weaker generalization guarantees. We validate this gap between the two approaches empirically. Finally, this infinite-neuron viewpoint is also fruitful for analyzing optimization. We show that a perturbed gradient flow on infinite-size networks finds a global optimizer in polynomial time.

我们表明，用弱的正则器训练前馈网络会产生最大的余量，并分析了这个结果的含义。

研究神经集的边际理论，表明最大边际在网络规模中是单调增长的。

本文研究了具有ReLU激活的两层网络的正则化交叉熵损失的最小化器的隐性偏差，得到了一个不随网络大小而增加的泛化上限。


We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.

一个大规模的深度强化学习的分布式架构，使用并行数据生成来提高Arcade学习环境基准的技术水平，只需要以前方法的一小部分挂钟训练时间。

研究了一个分布式深层RL系统，其中经验而不是梯度是在平行工作和集中学习者之间共享的。

一种平行的DQN训练方法，基于让多个行为者平行收集数据，同时由一个学习者从中央重放存储器的经验采样来训练模型的想法。

本文提出了大规模深度强化学习的分布式架构，重点是在优先体验回放框架中增加行为者算法的并行化。


Designing neural networks for continuous-time stochastic processes is challenging, especially when observations are made irregularly. In this article, we analyze neural networks from a frame theoretic perspective to identify the sufficient conditions that enable smoothly recoverable representations of signals in L^2(R). Moreover, we show that, under certain assumptions, these properties hold even when signals are irregularly observed. As we converge to the family of (convolutional) neural networks that satisfy these conditions, we show that we can optimize our convolution filters while constraining them so that they effectively compute a Discrete Wavelet Transform. Such a neural network can efficiently divide the time-axis of a signal into orthogonal sub-spaces of different temporal scale and localization. We evaluate the resulting neural network on an assortment of synthetic and real-world tasks: parsimonious auto-encoding, video classification, and financial forecasting.

提供不规则观察信号表征的神经架构，可证明能够进行信号重建。

证明具有Leaky ReLU激活函数的卷积神经网络是非线性框架，对非均匀采样的时间序列有类似的结果

本文考虑了时间序列上的神经网络，并表明可以选择第一个卷积滤波器来表示离散小波变换。


Most state-of-the-art neural machine translation systems, despite being different  in architectural skeletons (e.g., recurrence, convolutional), share an indispensable  feature: the Attention. However, most existing attention methods are token-based  and ignore the importance of phrasal alignments, the key ingredient for the success  of phrase-based statistical machine translation. In this paper, we propose  novel phrase-based attention methods to model n-grams of tokens as attention  entities. We incorporate our phrase-based attentions into the recently proposed  Transformer network, and demonstrate that our approach yields improvements of  1.3 BLEU for English-to-German and 0.5 BLEU for German-to-English translation  tasks, and 1.75 and 1.35 BLEU points in English-to-Russian and Russian-to-English translation tasks   on WMT newstest2014 using WMT’16 training data.

基于短语的注意力机制，用于将注意力分配给短语，实现token到短语，短语到token，短语到短语的注意力对齐，以及现有的token到token注意。

论文提出了一种注意力机制，它不仅可以计算单个token的加权和，还可以计算ngrams（短语）。


Intuitively, unfamiliarity should lead to lack of confidence. In reality, current algorithms often make highly confident yet wrong predictions when faced with unexpected test samples from an unknown distribution different from training. Unlike domain adaptation methods, we cannot gather an "unexpected dataset" prior to test, and unlike novelty detection methods, a best-effort original task prediction is still expected. We compare a number of methods from related fields such as calibration and epistemic uncertainty modeling, as well as two proposed methods that reduce overconfident errors of samples from an unknown novel distribution without drastically increasing evaluation time: (1) G-distillation, training an ensemble of classifiers and then distill into a single model using both labeled and unlabeled examples, or (2) NCR, reducing prediction confidence based on its novelty detection score. Experimentally, we investigate the overconfidence problem and evaluate our solution by creating "familiar" and "novel" test splits, where "familiar" are identically distributed with training and "novel" are not. We discover that calibrating using temperature scaling on familiar data is the best single-model method for improving novel confidence, followed by our proposed methods. In addition, some methods' NLL performance are roughly equivalent to a regularly trained model with certain degree of smoothing. Calibrating can also reduce confident errors, for example, in gender recognition by 95% on demographic groups different from the training data.

在测试意外数据时，深度网络更有可能出现置信误差。我们提出了一种实验方法来研究这个问题，以及两种方法来减少未知输入分布上的置信误差。

提出了两个减少过度置信的错误预测的想法。用额外的无监督数据对集合体进行 "G-distillation"，并利用新奇性检测器降低新奇性的置信度

作者提出了两种方法来估计新的未见过的数据分布的分类置信度。第一个想法是使用集合方法作为基础方法来帮助识别不确定的情况，然后使用蒸馏方法将集合方法减少为模仿集合方法行为的单一模型。第二个想法是使用新颖性检测器分类器，并通过新颖性得分对网络输出进行加权。


Progress in deep learning is slowed by the days or weeks it takes to train large models. The natural solution of using more hardware is limited by diminishing returns, and leads to inefficient use of additional resources. In this paper, we present a large batch, stochastic optimization algorithm that is both faster than widely used algorithms for fixed amounts of computation, and also scales up substantially better as more computational resources become available. Our algorithm implicitly computes the inverse Hessian of each mini-batch to produce descent directions; we do so without either an explicit approximation to the Hessian or Hessian-vector products. We demonstrate the effectiveness of our algorithm by successfully training large ImageNet models (InceptionV3, ResnetV1-50, ResnetV1-101 and InceptionResnetV2) with mini-batch sizes of up to 32000 with no loss in validation error relative to current baselines, and no increase in the total number of steps. At smaller mini-batch sizes, our optimizer improves the validation error in these models by 0.8-0.9\%. Alternatively, we can trade off this accuracy to reduce the number of training steps needed by roughly 10-30\%. Our work is practical and easily usable by others -- only one hyperparameter (learning rate) needs tuning, and furthermore, the algorithm is as computationally cheap as the commonly used Adam optimizer.

我们描述了一种实用的深度神经网络的优化算法，与广泛使用的算法相比，它工作得更快，生成的模型更好。

提出了一种新的算法，他们声称隐含地使用了Hessian，并使用了幂级数的动机。

提出了一种隐含使用曲率信息的新二阶算法，展示了算法中近似方案背后的直觉，并在各种实验中验证了启发式算法。


Recent work has shown that performing inference with fast, very-low-bitwidth  (e.g., 1 to 2 bits) representations of values in models can yield surprisingly accurate  results. However, although 2-bit approximated networks have been shown to  be quite accurate, 1 bit approximations, which are twice as fast, have restrictively  low accuracy. We propose a method to train models whose weights are a mixture  of bitwidths, that allows us to more finely tune the accuracy/speed trade-off. We  present the “middle-out” criterion for determining the bitwidth for each value, and  show how to integrate it into training models with a desired mixture of bitwidths.  We evaluate several architectures and binarization techniques on the ImageNet  dataset. We show that our heterogeneous bitwidth approximation achieves superlinear  scaling of accuracy with bitwidth. Using an average of only 1.4 bits, we are  able to outperform state-of-the-art 2-bit architectures.

我们引入了分数位宽近似，并表明它具有显著的优势。 提出了一种在前向传播阶段改变神经网络中量化程度的方法 在使用少于2位权重的情况下，保持2位网络字的准确性


Pruning units in a deep network can help speed up inference and training as well as reduce the size of the model. We show that bias propagation is a pruning technique which consistently outperforms the common approach of merely removing units,  regardless of the architecture and the dataset.   We also show how a simple adaptation to an existing scoring function allows us to select the best units to prune.   Finally,  we show that the units selected by the best performing scoring functions are somewhat consistent over the course of training, implying the dead parts of the network appear during the stages of training.

平均替换是一种有效的方法，可以改善修剪后的损失，而基于泰勒近似的评分函数在绝对值方面效果更好。

提出了对使用 "平均替换 "的单元修剪方法的简单改进。

本文提出了一种平均替换的修剪策略，并利用绝对值泰勒扩展作为修剪的评分函数。


Due to the phenomenon of “posterior collapse,” current latent variable generative models pose a challenging design choice that either weakens the capacity of the decoder or requires altering the training objective. We develop an alternative that utilizes the most powerful generative models as decoders, optimize the variational lower bound, and ensures that the latent variables preserve and encode useful information. Our proposed δ-VAEs achieve this by constraining the variational family for the posterior to have a minimum distance to the prior. For sequential latent variable models, our approach resembles the classic representation learning approach of slow feature analysis. We demonstrate our method’s efficacy at modeling text on LM1B and modeling images: learning representations, improving sample quality, and achieving state of the art log-likelihood on CIFAR-10 and ImageNet 32 × 32.

通过对速率的下限设定来避免后验塌陷。

提出了一种通过将变分近似族限制为后部来防止VAE后验塌陷的方法

本文介绍了对变分后验族的约束，使得KL项可以被控制以对抗深层生成模型（如VAEs）中的后后延塌陷。


Mini-batch gradient descent and its variants are commonly used in deep learning. The principle of mini-batch gradient descent is to use noisy gradient calculated on a batch to estimate the real gradient, thus balancing the computation cost per iteration and the uncertainty of noisy gradient. However, its batch size is a fixed hyper-parameter requiring manual setting before training the neural network. Yin et al. (2017) proposed a batch adaptive stochastic gradient descent (BA-SGD) that can dynamically choose a proper batch size as learning proceeds. We extend the BA-SGD to momentum algorithm and evaluate both the BA-SGD and the batch adaptive momentum (BA-Momentum) on two deep learning tasks from natural language processing to image classification. Experiments confirm that batch adaptive methods can achieve a lower loss compared with mini-batch methods after scanning the same epochs of data. Furthermore, our BA-Momentum is more robust against larger step sizes, in that it can dynamically enlarge the batch size to reduce the larger uncertainty brought by larger step sizes. We also identified an interesting phenomenon, batch size boom. The code implementing batch adaptive framework is now open source, applicable to any gradient-based optimization problems.

我们开发了一种批处理自适应动量，与小批处理方法相比，在扫描相同的历时数据后，可以达到更低的损失，而且对大步长也更加稳健。

本文解决了在深度学习训练过程中自动调整批次大小的问题，并声称将批次自适应SGD扩展到自适应动量，并将算法采用于复杂的神经网络问题。

本文提出了一个算法的一般化，该算法通过向效用函数添加动量来执行具有自适应批次大小的SGD。


Deep learning models for graphs have advanced the state of the art on many tasks. Despite their recent success, little is known about their robustness. We investigate training time attacks on graph neural networks for node classification that perturb the discrete graph structure.   Our core principle is to use meta-gradients to solve the bilevel problem underlying training-time attacks, essentially treating the graph as a hyperparameter to optimize. Our experiments show that small graph perturbations consistently lead to a strong decrease in performance for graph convolutional networks, and even transfer to unsupervised embeddings. Remarkably, the perturbations created by our algorithm can misguide the graph neural networks such that they perform worse than a simple baseline that ignores all relational information. Our attacks do not assume any knowledge about or access to the target classifiers.

我们使用元梯度来攻击图的深度神经网络的训练过程。

研究学习一个更好的中毒图参数的问题，可以使图神经网络的损失最大化。

一种通过增加/删除边缘来改变图结构的算法，从而降低节点分类的全局性能，以及使用元学习来解决双级优化问题的想法。


Numerous models for grounded language understanding have been recently proposed, including (i) generic models that can be easily adapted to any given task and (ii) intuitively appealing modular models that require background knowledge to be instantiated. We compare both types of models in how much they lend themselves to a particular form of systematic generalization. Using a synthetic VQA test, we evaluate which models are capable of reasoning about all possible object pairs after training on only a small subset of them. Our findings show that the generalization of modular models is much more systematic and that it is highly sensitive to the module layout, i.e. to how exactly the modules are connected. We furthermore investigate if modular models that generalize well could be made more end-to-end by learning their layout and parametrization. We find that end-to-end methods from prior work often learn inappropriate layouts or parametrizations that do not facilitate systematic generalization. Our results suggest that, in addition to modularity, systematic generalization in language understanding may require explicit regularizers or priors.

我们表明，模块化结构模型在系统泛化方面是最好的，其端到端版本的泛化效果并不理想。

本文通过引入一个新的空间推理数据集，评估了模块化神经网络和其他通用模型之间的系统泛化。

对视觉推理模型的泛化进行了有针对性的实证评估，重点是在以字母和数字为特征的合成场景中识别（物体、关系、物体）三要素的问题。


The behavioral dynamics of multi-agent systems have a rich and orderly structure, which can be leveraged to understand these systems, and to improve how artificial agents learn to operate in them. Here we introduce Relational Forward Models (RFM) for multi-agent learning, networks that can learn to make accurate predictions of agents' future behavior in multi-agent environments. Because these models operate on the discrete entities and relations present in the environment, they produce interpretable intermediate representations which offer insights into what drives agents' behavior, and what events mediate the intensity and valence of social interactions. Furthermore, we show that embedding RFM modules inside agents results in faster learning systems compared to non-augmented baselines.   As more and more of the autonomous systems we develop and interact with become multi-agent in nature, developing richer analysis tools for characterizing how and why agents make decisions is increasingly necessary. Moreover, developing artificial agents that quickly and safely learn to coordinate with one another, and with humans in shared environments, is crucial.

用于多代理学习的关系型前向模型能够准确预测代理的未来行为，它们产生了可解释的表征，并可在代理内部使用。

一种通过具有显式模型（使用类似图形 conv 网络的体系结构）来减少模型自由学习中方差的方法，该模型包含其他代理将执行的操作。

使用带有循环成分的关系型前向模型预测多代理行为，性能优于两个基线和两个消减。


We show that gradient descent on an unregularized logistic regression  problem, for almost all separable datasets, converges to the same direction as the max-margin solution. The result generalizes also to other monotone decreasing loss functions with an infimum at infinity, and we also discuss a multi-class generalizations to the cross entropy loss. Furthermore,  we show this convergence is very slow, and only logarithmic in the  convergence of the loss itself. This can help explain the benefit  of continuing to optimize the logistic or cross-entropy loss even  after the training error is zero and the training loss is extremely  small, and, as we show, even if the validation loss increases. Our  methodology can also aid in understanding implicit regularization  in more complex models and with other optimization methods.

梯度下降在逻辑回归上的归一化解（或类似的衰减损失）慢慢收敛到可分离数据上的L2最大边际解。

本文提供了一个正式的证明，即在数据是线性可分的情况下，逻辑损失的梯度下降会非常缓慢地收敛到硬SVM的解决方案。

本文着重描述了线性可分数据上对数损失最小化的行为，并表明用梯度下降法将对数损失最小化，会导致收敛到最大边际的解决方案。


Despite impressive performance as evaluated on i.i.d. holdout data, deep neural networks depend heavily on superficial statistics of the training data and are liable to break under distribution shift. For example, subtle changes to the background or texture of an image can break a seemingly powerful classifier. Building on previous work on domain generalization, we hope to produce a classifier that will generalize to previously unseen domains, even when domain identifiers are not available during training. This setting is challenging because the model may extract many distribution-specific (superficial) signals together with distribution-agnostic (semantic) signals. To overcome this challenge, we incorporate the gray-level co-occurrence matrix (GLCM) to extract patterns that our prior knowledge suggests are superficial: they are sensitive to the texture but unable to capture the gestalt of an image. Then we introduce two techniques for improving our networks' out-of-sample performance. The first method is built on the reverse gradient method that pushes our model to learn representations from which the GLCM representation is not predictable. The second method is built on the independence introduced by projecting the model's representation onto the subspace orthogonal to GLCM representation's.  We test our method on the battery of standard domain generalization data sets and, interestingly, achieve comparable or better performance as compared to other domain generalization methods that explicitly require samples from the target distribution for training.

在以前关于领域泛化的工作基础上，我们希望产生一个分类器，即使在训练期间没有领域标识符的情况下，也能泛化到以前未见过的领域。

基于CNN和NGLCM输出层的线性投影方案，揭示语义信息的领域概括方法。

本文提出了一种无监督的方法来识别对图像分类任务没有意义的图像特征


In this paper, we conduct an intriguing experimental study about the physical adversarial attack on object detectors in the wild. In particular, we learn a camouflage pattern to hide vehicles from being detected by state-of-the-art convolutional neural network based detectors. Our approach alternates between two threads. In the first, we train a neural approximation function to imitate how a simulator applies a camouflage to vehicles and how a vehicle detector performs given images of the camouflaged vehicles. In the second, we minimize the approximated detection score by searching for the optimal camouflage. Experiments show that the learned camouflage can not only hide a vehicle from the image-based detectors under many test cases but also generalizes to different environments, vehicles, and object detectors.

我们提出了一种学习物理车辆伪装的方法，以对抗性地攻击野外的物体检测器。我们发现我们的伪装是有效的和可转移的。

作者研究了学习一种伪装模式的问题，这种模式在应用于模拟车辆时，将阻止物体探测器对其进行探测。

本文以对抗性学习为目标，通过学习伪装模式进行干扰车检测


As deep learning-based classifiers are increasingly adopted in real-world applications, the importance of understanding how a particular label is chosen grows. Single decision trees are an example of a simple, interpretable classifier, but are unsuitable for use with complex, high-dimensional data. On the other hand, the variational autoencoder (VAE) is designed to learn a factored, low-dimensional representation of data, but typically encodes high-likelihood data in an intrinsically non-separable way.   We introduce the differentiable decision tree (DDT) as a modular component of deep networks and a simple, differentiable loss function that allows for end-to-end optimization of a deep network to compress high-dimensional data for classification by a single decision tree.   We also explore the power of labeled data in a  supervised VAE (SVAE) with a Gaussian mixture prior, which leverages label information to produce a high-quality generative model with improved bounds on log-likelihood.   We combine the SVAE with the DDT to get our classifier+VAE (C+VAE), which is competitive in both classification error and log-likelihood, despite optimizing both simultaneously and using a very simple encoder/decoder architecture.

我们将可微分决策树与监督变分自动编码器相结合，以增强分类的可解释性。

本文提出了一个由变分自动编码器与可微决策树组成的混合模型，以及一个配套的训练方案，实验证明了树的分类性能、负对数似然性能和潜在空间的可解释性。

本文试图通过堆叠有监督的VAE和可微的决策树来建立一个可解释的和准确的分类器。


We propose Regularized Learning under Label shifts (RLLS), a principled and a practical domain-adaptation algorithm to correct for shifts in the label distribution between a source and a target domain. We first estimate importance weights using labeled source data and unlabeled target data, and then train a classifier on the weighted source samples. We derive a generalization bound for the classifier on the target domain which is independent of the (ambient) data dimensions, and instead only depends on the complexity of the function class. To the best of our knowledge, this is the first generalization bound for the label-shift problem where the labels in the target domain are not available. Based on this bound, we propose a regularized estimator for the small-sample regime which accounts for the uncertainty in the estimated weights. Experiments on the CIFAR-10 and MNIST datasets show that RLLS improves classification accuracy, especially in the low sample and large-shift regimes, compared to previous methods.

在源数据集和目标数据集之间存在标签转移的情况下，训练高效分类器的实用和可证明的保证方法

作者提出了一种新的算法，以两步程序提高类重要性加权估计程序的稳定性。

作者考虑了标签转移下的学习问题，即标签比例不同而条件相等，并提出了一个改进的正则化估计器。


The statistics of the real visual world presents a long-tailed distribution: a few classes have significantly more training instances than the remaining classes in a dataset. This is because the real visual world has a few classes that are common while others are rare. Unfortunately, the performance of a convolutional neural network is typically unsatisfactory when trained using a long-tailed dataset. To alleviate this issue, we propose a method that discriminatively learns an embedding in which a simple Bayesian classifier can balance the class-priors to generalize well for rare classes. To this end, the proposed approach uses a Gaussian mixture model to factor out class-likelihoods and class-priors in a long-tailed dataset. The proposed method is simple and easy-to-implement in existing deep learning frameworks. Experiments on publicly available datasets show that the proposed approach improves the performance on classes with few training instances, while maintaining a comparable performance to the state-of-the-art on classes with abundant training examples.

提高尾部类分类准确性的方法。

本文的主要目标是学习一个ConvNet分类器，该分类器对处于类发生率分布尾部的类表现得更好。

建议采用高斯混合模型的贝叶斯框架来解决分类应用中的一个问题，即不同类别的训练数据的数量是不平衡的。


As deep reinforcement learning is being applied to more and more tasks, there is a growing need to better understand and probe the learned agents. Visualizing and understanding the decision making process can be very valuable to comprehend and identify problems in the learned behavior. However, this topic has been relatively under-explored in the reinforcement learning community. In this work we present a method for synthesizing states of interest for a trained agent. Such states could be situations (e.g. crashing or damaging a car) in which specific actions are necessary. Further, critical states in which a very high or a very low reward can be achieved (e.g. risky states) are often interesting to understand the situational awareness of the system. To this end, we learn a generative model over the state space of the environment and use its latent space to optimize a target function for the state of interest. In our experiments we show that this method can generate insightful visualizations for a variety of environments and reinforcement learning methods. We explore these issues in the standard Atari benchmark games as well as in an autonomous driving simulator. Based on the efficiency with which we have been able to identify significant decision scenarios with this technique, we believe this general approach could serve as an important tool for AI safety applications.

我们提出了一种为强化学习代理合成感兴趣的状态的方法，以分析其行为。

本文提出了一个RL中视觉观察的生成模型，它能够生成感兴趣的观察结果。

一种可视化感兴趣状态的方法，涉及学习重建状态空间的变分自动编码器和查找条件参数以生成合成图像的优化步骤。


We introduce the deep abstaining classifier -- a deep neural network trained with a novel loss function that provides an abstention option during training. This allows the  DNN to abstain on confusing or difficult-to-learn examples while improving performance on the non-abstained samples. We show that such deep abstaining classifiers can: (i) learn representations for structured noise -- where noisy training labels or confusing examples are correlated with underlying features -- and then learn to abstain based on such features; (ii) enable robust learning in the presence of arbitrary or unstructured noise by identifying noisy samples; and (iii) be used as an effective out-of-category detector that learns to reliably abstain when presented with samples from  unknown classes. We provide analytical results on loss function behavior that enable automatic tuning of accuracy and coverage, and demonstrate the utility of the deep abstaining classifier using multiple image benchmarks, Results indicate significant improvement in learning in the presence of label noise.

一个用新的损失函数训练的深度弃权神经网络，学习何时弃权的表征，在不同类型的噪声存在下实现稳健学习。

一种新的损失函数，用于训练深度神经网络，该神经网络可以弃权，从存在结构化噪声，存在非结构化噪声和开放世界检测的角度看待性能。

本稿件介绍了深度弃权分类器，它用弃权损失修改了多类交叉熵损失，然后将其应用于扰动图像分类任务。


Temporal Difference learning with function approximation has been widely used recently and has led to several successful results.   However, compared with the original tabular-based methods, one major drawback of temporal difference learning with neural networks and other function approximators is that they tend to over-generalize across temporally successive states, resulting in slow convergence and even instability. In this work, we propose a novel TD learning method, Hadamard product Regularized TD (HR-TD), that reduces over-generalization and thus leads to faster convergence. This approach can be easily applied to both linear and nonlinear function approximators.   HR-TD is evaluated on several linear and nonlinear benchmark domains, where we show improvement in learning behavior and performance.

一种用于TD学习的正则化技术，避免了时间上的过度泛化，尤其是在深度网络中。

在函数近似的情况下，对时间差异学习的一种变体，试图解决跨时间连续状态的过度泛化问题。

本文介绍了HR-TD，这是TD（0）算法的变体，旨在改善传统TD中的过度泛化问题。


We present an efficient convolution kernel for Convolutional Neural Networks (CNNs) on unstructured grids using parameterized differential operators while focusing on spherical signals such as panorama images or planetary signals.   To this end, we replace conventional convolution kernels with linear combinations of differential operators that are weighted by learnable parameters. Differential operators can be efficiently estimated on unstructured grids using one-ring neighbors, and learnable parameters can be optimized through standard back-propagation. As a result, we obtain extremely efficient neural networks that match or outperform state-of-the-art network architectures in terms of performance but with a significantly lower number of network parameters. We evaluate our algorithm in an extensive series of experiments on a variety of computer vision and climate science tasks, including shape classification, climate pattern segmentation, and omnidirectional image semantic segmentation. Overall, we present (1) a novel CNN approach on unstructured grids using parameterized differential operators for spherical signals, and (2) we show that our unique kernel parameterization allows our model to achieve the same or higher accuracy with significantly fewer network parameters.

我们提出了一个新的CNN内核，用于球形信号的非结构化网格，并在三维分类和全向图像分割等任务上显示出明显的准确性和参数效率的提高。

一种高效的方法使球形数据的深度学习成为可能，与流行的方法相比，用更少的参数就能达到有竞争力/最先进的数字。

本文提出了一种用于非结构化网格的CNN的新型卷积核，并通过微分算子的线性组合来表述卷积。


Prediction is arguably one of the most basic functions of an intelligent system. In general, the problem of predicting events in the future or between two waypoints is exceedingly difficult. However, most phenomena naturally pass through relatively predictable bottlenecks---while we cannot predict the precise trajectory of a robot arm between being at rest and holding an object up, we can be certain that it must have picked the object up. To exploit this, we decouple visual prediction from a rigid notion of time. While conventional approaches predict frames at regularly spaced temporal intervals, our time-agnostic predictors (TAP) are not tied to specific times so that they may instead discover predictable "bottleneck" frames no matter when they occur. We evaluate our approach for future and intermediate frame prediction across three robotic manipulation tasks. Our predictions are not only of higher visual quality, but also correspond to coherent semantic subgoals in temporally extended tasks.

在视觉预测任务中，让你的预测模型选择哪些时间进行预测有两点好处。(i)提高预测质量，(ii)导致语义上连贯的 "瓶颈状态 "预测，这对规划很有用。

一种关于视频中帧预测的方法，该方法包括目标预测是浮动的，由预测误差的最小值解决。

重新制定视频预测/插值的任务，使预测器不被强迫在固定的时间间隔内生成帧，而是被训练成在未来任何时间点发生的帧。


In cities with tall buildings, emergency responders need an accurate floor level location to find 911 callers quickly. We introduce a system to estimate a victim's floor level via their mobile device's sensor data in a two-step process. First, we train a neural network to determine when a smartphone enters or exits a building via GPS signal changes. Second, we use a barometer equipped smartphone to measure the change in barometric pressure from the entrance of the building to the victim's indoor location. Unlike impractical previous approaches, our system is the first that does not require the use of beacons, prior knowledge of the building infrastructure, or knowledge of user behavior. We demonstrate real-world feasibility through 63 experiments across five different tall buildings throughout New York City where our system predicted the correct floor level with 100% accuracy.

我们用一个LSTM来检测智能手机何时走进大楼。然后我们利用智能手机上的传感器的数据预测设备的楼层高度。

本文介绍了一个系统，通过他们的移动设备的传感器数据，使用LSTM和气压的变化来估计一个楼层的高度。

建议采用两步法来确定移动电话在高层建筑内的哪个楼层。


Sparse reward is one of the most challenging problems in reinforcement learning (RL). Hindsight Experience Replay (HER) attempts to address this issue by converting a failure experience to a successful one by relabeling the goals. Despite its effectiveness, HER has limited applicability because it lacks a compact and universal goal representation. We present Augmenting experienCe via TeacheR's adviCE (ACTRCE), an efficient reinforcement learning technique that extends the HER framework using natural language as the goal representation. We first analyze the differences among goal representation, and show that ACTRCE can efficiently solve difficult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn. We also show that with language goal representations, the agent can generalize to unseen instructions, and even generalize to instructions with unseen lexicons. We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, but we also found that little amount of hindsight advice is sufficient for the learning to take off, showing the practical aspect of the method.

将语言目标表述与事后的经验复述结合起来。

本文考虑了后知后觉的经验复述中隐含的假设，即可以获得从状态到目标的映射，并提出了一种自然语言的目标表述。

本报告使用带有自然语言目标的Hindsight Experience Replay框架来提高指令遵循模型的样本效率。


Learning rich and compact representations is an open topic in many fields such as word embedding, visual question-answering, object recognition or image retrieval. Although deep neural networks (convolutional or not) have made a major breakthrough during the last few years by providing hierarchical, semantic and abstract representations for all of these tasks, these representations are not necessary as rich as needed nor as compact as expected. Models using higher order statistics, such as bilinear pooling, provide richer representations at the cost of higher dimensional features. Factorization schemes have been proposed but without being able to reach the original compactness of first order models, or at a heavy loss in performances. This paper addresses these two points by extending factorization schemes to codebook strategies, allowing compact representations with the same dimensionality as first order representations, but with second order performances. Moreover, we extend this framework with a joint codebook and factorization scheme, granting a reduction both in terms of parameters and computation cost. This formulation leads to state-of-the-art results and compact second-order models with few additional parameters and intermediate representations with a dimension similar to that of first-order statistics.

我们提出了一个联合编码本和因子化方案来改善二阶集合。

本文提出了一种将现有的因子化二阶表征与编码本式的硬分配相结合的方法。

提出了一种基于编码本模型的新型双线性表示法，以及一种有效的公式，其中基于编码本的投影通过共享投影进行因子化，以进一步减少参数大小。


Natural language understanding research has recently shifted towards complex Machine Learning and Deep Learning algorithms. Such models often outperform their simpler counterparts significantly. However, their performance relies on the availability of large amounts of labeled data, which are rarely available. To tackle this problem, we propose a methodology for extending training datasets to arbitrarily big sizes and training complex, data-hungry models using weak supervision. We apply this methodology on biomedical relation extraction, a task where training datasets are excessively time-consuming and expensive to create, yet has a major impact on downstream applications such as drug discovery. We demonstrate in two small-scale controlled experiments that our method consistently enhances the performance of an LSTM network, with performance improvements comparable to hand-labeled training data. Finally, we discuss the optimal setting for applying weak supervision using this methodology.

我们提出并应用了一种基于弱监督的元学习方法，在生物医学关系提取的任务上将半监督和集合学习相结合。

一种用于关系分类的半监督方法，它使用一个小的标记数据集训练多个基础学习者，并应用其中的一些来注释未标记的例子，以进行半监督学习。

本文解决了生成生物关系提取的训练数据的问题，并使用由弱分类器标记的数据的预测作为元学习算法的额外训练数据。

本文提出了半监督学习和集合学习相结合的信息提取方法，并在生物医学关系提取任务中进行了实验


We introduce contextual explanation networks (CENs)---a class of models that learn to predict by generating and leveraging intermediate explanations. CENs are deep networks that generate parameters for context-specific probabilistic graphical models which are further used for prediction and play the role of explanations. Contrary to the existing post-hoc model-explanation tools, CENs learn to predict and to explain jointly. Our approach offers two major advantages: (i) for each prediction, valid instance-specific explanations are generated with no computational overhead and (ii) prediction via explanation acts as a regularization and boosts performance in low-resource settings. We prove that local approximations to the decision boundary of our networks are consistent with the generated explanations. Our results on image and text classification and survival analysis tasks demonstrate that CENs are competitive with the state-of-the-art while offering additional insights behind each prediction, valuable for decision support.

一类网络在即时生成简单的模型（称为解释），作为正则器，使模型的诊断和可解释性一致。

作者称，以前的技术直接将神经网络作为组件整合到图形模型中，这使得模型无法被解释。

通过使用深度神经网来预测图形模型的参数，提出了神经网和图形模型相结合的建议。


The goal of imitation learning (IL) is to enable a learner to imitate an expert’s behavior given the expert’s demonstrations. Recently, generative adversarial imitation learning (GAIL) has successfully achieved it even on complex continuous control tasks. However, GAIL requires a huge number of interactions with environment during training. We believe that IL algorithm could be more applicable to the real-world environments if the number of interactions could be reduced. To this end, we propose a model free, off-policy IL algorithm for continuous control. The keys of our algorithm are two folds: 1) adopting deterministic policy that allows us to derive a novel type of policy gradient which we call deterministic policy imitation gradient (DPIG), 2) introducing a function which we call state screening function (SSF) to avoid noisy policy updates with states that are not typical of those appeared on the expert’s demonstrations. Experimental results show that our algorithm can achieve the goal of IL with at least tens of times less interactions than GAIL on a variety of continuous control tasks.

我们提出了一种无模型的模仿学习算法，与最先进的模仿学习算法即GAIL相比，该算法能够减少与环境的交互次数。

提出扩展确定性策略梯度算法，从演示中学习，同时结合专家的一种密度估计。

本文考虑了无模型模仿学习的问题，并提出了生成式对抗性模仿学习算法的扩展，即用确定性的策略取代学习者的随机策略。

本文结合了IRL、对抗性训练和确定性政策梯度的思想，目的是降低样本的复杂性。


Convolution acts as a local feature extractor in convolutional neural networks (CNNs). However, the convolution operation is not applicable when the input data is supported on an irregular graph such as with social networks, citation networks, or knowledge graphs. This paper proposes the topology adaptive graph convolutional network (TAGCN), a novel graph convolutional network that generalizes CNN architectures to graph-structured data and provides a systematic way to design a set of fixed-size learnable filters to perform convolutions on graphs. The topologies of these filters are adaptive to the topology of the graph when they scan the graph to perform convolution, replacing the square filter for the grid-structured data in traditional CNNs. The outputs are the weighted sum of these filters’ outputs, extraction of both vertex features and strength of correlation between vertices. It  can be used with both directed and undirected graphs. The proposed TAGCN not only inherits the properties of convolutions in CNN for grid-structured data, but it is also consistent with convolution as defined in graph signal processing. Further, as no approximation to the convolution is needed, TAGCN exhibits better performance than existing graph-convolution-approximation methods on a number  of data sets. As only the polynomials of degree two of the adjacency matrix are used, TAGCN is also computationally simpler than other recent methods.

计算复杂度低的图形CNN（无近似），分类精度更高

提出了一种新的CNN图形分类方法，使用基于长度增加的外向行走的过滤器，在一个传播步骤中纳入来自更远的顶点的信息。

提出了用于半监督图分类的新的神经网络结构，建立在图多项式过滤器的基础上，并将其用于具有ReLU激活函数的连续神经网络层。

本文介绍了拓扑自适应GCN，将卷积网络推广到图形结构的数据中。


Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks. Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a ``forgetting event'' to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning. Across several benchmark data sets, we find that: (i) certain examples are forgotten with high frequency, and some not at all; (ii) a data set's (un)forgettable examples generalize across neural architectures; and (iii) based on forgetting dynamics, a significant fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.

我们表明，灾难性的遗忘发生在被认为是单一任务的范围内，并发现不容易遗忘的例子可以从训练集中删除而不损失泛化。

研究了SGD过程中训练实例的遗忘行为，并表明在不同的网络结构中，神经网络训练存在 "支持实例"。

本文分析了网络在多大程度上学会了对特定的例子进行正确分类，然后在训练过程中忘记这些例子。

本文研究了训练神经网络中的一些例子是否比其他例子更难学习。这样的例子在学习中会被遗忘，并多次被重新学习。


Discovering objects and their attributes is of great importance for autonomous agents to effectively operate in human environments. This task is particularly challenging due to the ubiquitousness of objects and all their nuances in perceptual and semantic detail. In this paper we present an unsupervised approach for learning disentangled representations of objects entirely from unlabeled monocular videos. These continuous representations are not biased by or limited by a discrete set of labels determined by human labelers. The proposed representation is trained with a metric learning loss, where objects with homogeneous features are pushed together, while those with heterogeneous features are pulled apart. We show these unsupervised embeddings allow to discover object attributes and can enable robots to self-supervise in previously unseen environments. We quantitatively evaluate performance on a large-scale synthetic dataset with 12k object models, as well as on a real dataset collected by a robot and show that our unsupervised object understanding generalizes to previously unseen objects. Specifically, we demonstrate the effectiveness of our approach on robotic manipulation tasks, such as pointing at and grasping of objects. An interesting and perhaps surprising finding in this approach is that given a limited set of objects, object correspondences will naturally emerge when using metric learning without requiring explicit positive pairs.

一种无监督的方法，完全从无标签的单眼视频中学习物体的分离表征。

从不同视点拍摄的场景视频序列中设计出一个特征表征。

提出了一种用于视觉输入的无监督表征学习方法，该方法包含了一种度量学习方法，在嵌入空间中将图像块的近邻对拉近，而将其他对推开。

本文探讨了物体表征的自我监督学习，其主要思想是鼓励具有相似特征的物体进一步相互吸引。


Learning from a scalar reward in continuous action space environments is difficult and often requires millions if not billions of interactions.   We introduce state aligned vector rewards, which are easily defined in metric state spaces and allow our deep reinforcement learning agent to tackle the curse of dimensionality.   Our agent learns to map from action distributions to state change distributions implicitly defined in a quantile function neural network.    We further introduce a new reinforcement learning technique inspired by quantile regression which does not limit agents to explicitly parameterized action distributions.   Our results in high dimensional state spaces show that training with vector rewards allows our agent to learn multiple times faster than an agent training with scalar rewards.

我们用状态对齐的向量奖励来训练一个从行动分布中预测状态变化的代理，使用一种新的强化学习技术，其灵感来自于分位数回归。

提出了旨在于奖励与状态空间对齐的情况下加速强化学习的算法。

本文通过使用一个重新参数化的策略和一个新的基于向量的训练目标来解决连续行动空间中的RL。

这项工作提出将分布式RL与负责以分位数的方式对世界的演变进行建模的网混合在一起，声称可以改善样本效率。


We propose Episodic Backward Update - a new algorithm to boost the performance of a deep reinforcement learning agent by fast reward propagation. In contrast to the conventional use of the replay memory with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state into its previous states. Our computationally efficient recursive algorithm allows sparse and delayed rewards to propagate effectively throughout the sampled episode. We evaluate our algorithm on 2D MNIST Maze Environment and 49 games of the Atari 2600 Environment and show that our agent improves sample efficiency with a competitive computational cost.

我们提出了Episodic Backward Update，这是一种新颖的深度强化学习算法，可以逐集采样转换并以向后方式递归更新值，以实现快速稳定的学习。

提出了一个新的DQN，在这个DQN中，目标是通过向后更新（从头到尾）在一个完整的集上计算的，以便在集结束时更快地传播奖赏。

作者提出修改DQN算法，在有一定衰减的轨迹上递归地应用最大贝尔曼算子，以防止嵌套的最大误差积累。

在deep-Q网络中，从集结束时开始更新Q值，以促进奖励沿集快速传播。


Survival Analysis (time-to-event analysis) in the presence of multiple possible adverse events, i.e., competing risks, is a challenging, yet very important problem in medicine, finance, manufacturing, etc. Extending classical survival analysis to competing risks is not trivial since only one event (e.g. one cause of death) is observed and hence, the incidence of an event of interest is often obscured by other related competing events. This leads to the nonidentifiability of the event times’ distribution parameters, which makes the problem significantly more challenging. In this work we introduce Siamese Survival Prognosis Network, a novel Siamese Deep Neural Network architecture that is able to effectively learn from data in the presence of multiple adverse events. The Siamese Survival Network is especially crafted to issue pairwise concordant time-dependent risks, in which longer event times are assigned lower risks. Furthermore, our architecture is able to directly optimize an approximation to the C-discrimination index, rather than relying on well-known metrics of cross-entropy etc., and which are not able to capture the unique requirements of survival analysis with competing risks. Our results show consistent performance improvements on a number of publicly available medical datasets over both statistical and deep learning state-of-the-art methods.

在这项工作中，我们介绍了一种新的Siamese深度神经网络架构，该架构能够在存在多种不良事件的情况下有效地从数据中学习。

本文通过直接对C指数进行优化，将siamese神经网络引入竞争性风险框架。

作者讨论了在有竞争风险的生存分析环境中估计风险的问题，并建议使用siamese生存网络直接优化随时间变化的区分指数。


The digitization of data has resulted in making datasets available to millions of users in the form of relational databases and spreadsheet tables. However, a majority of these users come from diverse backgrounds and lack the programming expertise to query and analyze such tables. We present a system that allows for querying data tables using natural language questions, where the system translates the question into an executable SQL query. We use a deep sequence to sequence model in wich the decoder uses a simple type system of SQL expressions to structure the output prediction. Based on the type, the decoder either copies an output token from the input question using an attention-based copying mechanism or generates it from a fixed vocabulary. We also introduce a value-based loss function that transforms a distribution over locations to copy from into a distribution over the set of input tokens to improve training of our model. We evaluate our model on the recently released WikiSQL dataset and show that our model trained using only supervised learning significantly outperforms the current state-of-the-art Seq2SQL model that uses reinforcement learning.

我们提出了一个基于类型的指针网络模型以及基于值的损失方法，以有效地训练一个神经模型，将自然语言翻译成SQL。

本文声称开发了一种新颖的方法，通过使用语法来指导解码，并使用新的损失函数来指导指针/复制机制，将自然语言查询映射到SQL。


To backpropagate the gradients through stochastic binary layers, we propose the augment-REINFORCE-merge (ARM) estimator that is unbiased, exhibits low variance, and has low computational complexity. Exploiting variable augmentation, REINFORCE, and reparameterization, the ARM estimator achieves adaptive variance reduction for Monte Carlo integration by merging two expectations via common random numbers. The variance-reduction mechanism of the ARM estimator can also be attributed to either antithetic sampling in an augmented space, or the use of an optimal anti-symmetric "self-control" baseline function together with the REINFORCE estimator in that augmented space. Experimental results show the ARM estimator provides state-of-the-art performance in auto-encoding variational inference and maximum likelihood estimation, for discrete latent variable models with one or multiple stochastic binary layers. Python code for reproducible research is publicly available.

离散潜在变量模型的无偏低方差梯度估计器

提出了一种新的方差减小技术，用于计算预期损失梯度，其中预期是关于独立二元随机变量的。

在随机二元网络的特殊情况下，降低得分函数梯度估计器方差的Rao-Blackwellization和一般的随机数相结合的算法

用于计算和反向传播二元神经网络梯度的无偏和低方差的增强-REINFORCE-合并（ARM）估计器


Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training. The scheme can reach a linear speed-up with respect to the number of workers, but this is rarely seen in practice as the scheme often suffers from large network delays and bandwidth limits. To overcome this communication bottleneck recent works propose to reduce the communication frequency. An algorithm of this type is local SGD that runs SGD independently in parallel on different workers and averages the sequences only once in a while. This scheme shows promising results in practice, but eluded thorough theoretical analysis.       We prove concise convergence rates for local SGD on convex problems and show that it converges at the same rate as mini-batch SGD in terms of number of evaluated gradients, that is, the scheme achieves linear speed-up in the number of workers and mini-batch size. The number of  communication rounds can be reduced up to a factor of T^{1/2}---where T denotes the number of total steps---compared to mini-batch SGD. This also holds for asynchronous implementations.   Local SGD can also be used for large scale training of deep learning models. The results shown here aim serving as a guideline to further explore the theoretical and practical aspects of local SGD in these applications.

我们证明，并行本地SGD实现了线性加速，而通信量比并行小批量SGD少得多。

为本地SGD提供了一个收敛证明，并证明了本地SGD可以提供与minibatch相同的加速收益，但可能能够显著减少通信。

本文提出了对本地SGD的分析，以及对运行SGD得到的估计器需要多频繁地进行平均化才能产生线性并行化速度的界限。

作者分析了本地SGD算法，其中SGD的$K$并行链被运行，迭代结果偶尔通过平均化在机器间同步。


Extracting relevant information, causally inferring and predicting the future states with high accuracy is a crucial task for modeling complex systems. The endeavor to address these tasks is made even more challenging when we have to deal with high-dimensional heterogeneous data streams. Such data streams often have higher-order inter-dependencies across spatial and temporal dimensions. We propose to perform a soft-clustering of the data and learn its dynamics to produce a compact dynamical model while still ensuring the original objectives of causal inference and accurate predictions. To efficiently and rigorously process the dynamics of soft-clustering, we advocate for an information theory inspired approach that incorporates stochastic calculus and seeks to determine a trade-off between the predictive accuracy and compactness of the mathematical representation. We cast the model construction as a maximization of the compression of the state variables such that the predictive ability and causal interdependence (relatedness) constraints between the original data streams and the compact model are closely bounded. We provide theoretical guarantees concerning the convergence of the proposed learning algorithm. To further test the proposed framework, we consider a high-dimensional Gaussian case study and describe an iterative scheme for updating the new model parameters. Using numerical experiments, we demonstrate the benefits on compression and prediction accuracy for a class of dynamical systems. Finally, we apply the proposed algorithm to the real-world dataset of multimodal sentiment intensity and show improvements in prediction with reduced dimensions.

动态过程的紧凑感知 研究用信息瓶颈法紧凑地表示复杂动态系统的模型，同时保留信息的问题。 本文研究了高斯线性动态，并提出了一种计算信息瓶颈层次的算法（IBH）。


We propose the dense RNN, which has the fully connections from each hidden state to multiple preceding hidden states of all layers directly. As the density of the connection increases, the number of paths through which the gradient flows can be increased. It increases the magnitude of gradients, which help to prevent the vanishing gradient problem in time. Larger gradients, however, can also cause exploding gradient problem. To complement the trade-off between two problems, we propose an attention gate, which controls the amounts of gradient flows. We describe the relation between the attention gate and the gradient flows by approximation. The experiment on the language modeling using Penn Treebank corpus shows dense connections with the attention gate improve the model’s performance.

密集的RNN，从每个隐藏状态到所有层的多个前面的隐藏状态有完全的连接。

提出了一种新的RNN架构，可以更好地模拟长期依赖关系，可以学习顺序数据的多尺度表示，并通过使用参数化的门控单元避开梯度问题。

本文提出了一种全连接的密集型RNN结构，该结构具有每一层的门控连接和前一层的连接，以及它在PTB字符级建模任务上的结果。


We propose a new algorithm for training generative adversarial networks to jointly learn latent codes for both identities (e.g. individual humans) and observations (e.g. specific photographs). In practice, this means that by fixing the identity portion of latent codes, we can generate diverse images of the same subject, and by fixing the observation portion we can traverse the manifold of subjects while maintaining contingent aspects such as lighting and pose. Our algorithm features a pairwise training scheme in which each sample from the generator consists of two images with a common identity code. Corresponding samples from the real dataset consist of two distinct photographs of the same subject. In order to fool the discriminator, the generator must produce images that are both photorealistic, distinct, and appear to depict the same person. We augment both the DCGAN and BEGAN approaches with Siamese discriminators to accommodate pairwise training. Experiments with human judges and an off-the-shelf face verification system demonstrate our algorithm’s ability to generate convincing, identity-matched photographs.

SD-GAN根据数据集中的已知共性（例如，描绘同一个人的照片）来分解潜伏代码。

本文研究了受控图像生成的问题，并提出了一种能产生一对相同身份的图像的算法。

本文提出了，SD-GAN，一种训练GAN的方法，以分解潜伏向量输入Z中的身份和非身份信息。


The goal of unpaired cross-domain translation is to learn useful mappings between two domains, given unpaired sets of datapoints from these domains. While this formulation is highly underconstrained, recent work has shown that it is possible to learn mappings useful for downstream tasks by encouraging approximate cycle consistency in the mappings between the two domains [Zhu et al., 2017]. In this work, we propose AlignFlow, a framework for unpaired cross-domain translation that ensures exact cycle consistency in the learned mappings. Our framework uses a normalizing flow model to specify a single invertible mapping between the two domains. In contrast to prior works in cycle-consistent translations, we can learn AlignFlow via adversarial training, maximum likelihood estimation, or a hybrid of the two methods. Theoretically, we derive consistency results for AlignFlow which guarantee recovery of desirable mappings under suitable assumptions. Empirically, AlignFlow demonstrates significant improvements over relevant baselines on image-to-image translation and unsupervised domain adaptation tasks on benchmark datasets.

我们提出了一个跨域翻译的学习框架，该框架完全是循环一致的，可以通过对抗性训练、最大似然估计或混合方式学习。

提出了AlignFlow，这是一种利用可逆流实现周期一致性原则的有效方法。

非配对图像到图像转换的流模型


Program synthesis is a class of regression problems where one seeks a solution, in the form of a source-code program, that maps the inputs to their corresponding outputs exactly. Due to its precise and combinatorial nature, it is commonly formulated as a constraint satisfaction problem, where input-output examples are expressed constraints, and solved with a constraint solver. A key challenge of this formulation is that of scalability: While constraint solvers work well with few well-chosen examples, constraining the entire set of example constitutes a significant overhead in both time and memory. In this paper we address this challenge by constructing a representative subset of examples that is both small and is able to constrain the solver sufficiently. We build the subset one example at a time, using a trained discriminator to predict the probability of unchosen input-output examples conditioned on the chosen input-output examples, adding the least probable example to the subset. Experiment on a diagram drawing domain shows our approach produces subset of examples that are small and representative for the constraint solver.

在程序合成的背景下，输入一组例子，我们通过计算代表例子的子集来减少成本

提出了一种确定程序合成的代表性例子的方法，以提高现有约束性编程解决方案的可扩展性。

一种选择实例子集的方法，在此基础上运行约束解算器，以解决程序综合问题。

本文提出了一种加快通用程序合成器的方法。


Humans possess an ability to abstractly reason about objects and their interactions, an ability not shared with state-of-the-art deep learning models. Relational networks, introduced by Santoro et al. (2017), add the capacity for relational reasoning to deep neural networks, but are limited in the complexity of the reasoning tasks they can address. We introduce recurrent relational networks which increase the suite of solvable tasks to those that require an order of magnitude more steps of relational reasoning. We use recurrent relational networks to solve Sudoku puzzles and achieve state-of-the-art results by solving 96.6% of the hardest Sudoku puzzles, where relational networks fail to solve any. We also apply our model to the BaBi textual QA dataset solving 19/20 tasks which is competitive with state-of-the-art sparse differentiable neural computers. The recurrent relational network is a general purpose module that can augment any neural network model with the capacity to do many-step relational reasoning.

我们介绍了循环关系网络，这是一个用于关系推理的强大而通用的神经网络模块，并利用它解决了96.6%最难的Sudokus和19/20 BaBi任务。

引入了循环关系网络（RRN）,可以添加到任何神经网络中以增加关系推理能力。

介绍一个用于结构化预测的深度神经网络，该网络在Soduku谜题和BaBi任务上取得了最先进的性能。

本文介绍了一种称为关系网络的方法，为深度神经网络增加关系推理能力。


Empirical risk minimization (ERM), with proper loss function and regularization, is the common practice of supervised classification. In this paper, we study training arbitrary (from linear to deep) binary classifier from only unlabeled (U) data by ERM. We prove that it is impossible to estimate the risk of an arbitrary binary classifier in an unbiased manner given a single set of U data, but it becomes possible given two sets of U data with different class priors. These two facts answer a fundamental question---what the minimal supervision is for training any binary classifier from only U data. Following these findings, we propose an ERM-based learning method from two sets of U data, and then prove it is consistent. Experiments demonstrate the proposed method could train deep models and outperform state-of-the-art methods for learning from two sets of U data.

仅从 U 数据训练深度模型所需的三个类先验就足够了，而任何两个都不够。

提出了一个无偏估计器，允许在两个具有已知类先验的无标签数据集上用弱监督训练模型，并讨论了估计器的理论属性。

一种仅从无标签数据中训练任何二元分类器的方法，以及一种针对两组无标签数据的经验性风险最小化方法，其中类别先验是给定的。


Deep Neural Networks (DNNs) excel on many complex perceptual tasks but it has proven notoriously difficult to understand how they reach their decisions. We here introduce a high-performance DNN architecture on ImageNet whose decisions are considerably easier to explain. Our model, a simple variant of the ResNet-50 architecture called BagNet, classifies an image based on the occurrences of small local image features without taking into account their spatial ordering. This strategy is closely related to the bag-of-feature (BoF) models popular before the onset of deep learning and reaches a surprisingly high accuracy on ImageNet (87.6% top-5 for 32 x 32 px features and Alexnet performance for 16 x16 px features). The constraint on local features makes it straight-forward to analyse how exactly each part of the image influences the classification. Furthermore, the BagNets behave similar to state-of-the art deep neural networks such as VGG-16, ResNet-152 or DenseNet-169 in terms of feature sensitivity, error distribution and interactions between image parts. This suggests that the improvements of DNNs over previous bag-of-feature classifiers in the last few years is mostly achieved by better fine-tuning rather than by qualitatively different decision strategies.

从许多小图像块中聚集类证据足以解决ImageNet，产生更多可解释的模型，并能解释流行的DNN的决策方面。

本文提出了一种新颖而紧凑的神经网络结构，它使用了词包特征中的信息。建议的算法只独立使用补丁信息，并使用独立分类的图像块进行多数投票。


Somatic cancer mutation detection at ultra-low variant allele frequencies (VAFs) is an unmet challenge that is intractable with current state-of-the-art mutation calling methods. Specifically, the limit of VAF detection is closely related to the depth of coverage, due to the requirement of multiple supporting reads in extant methods, precluding the detection of mutations at VAFs that are orders of magnitude lower than the depth of coverage. Nevertheless, the ability to detect cancer-associated mutations in ultra low VAFs is a fundamental requirement for low-tumor burden cancer diagnostics applications such as early detection, monitoring, and therapy nomination using liquid biopsy methods (cell-free DNA). Here we defined a spatial representation of sequencing information adapted for convolutional architecture that enables variant detection at VAFs, in a manner independent of the depth of sequencing. This method enables the detection of cancer mutations even in VAFs as low as 10x-4^, >2 orders of magnitude below the current state-of-the-art. We validated our method on both simulated plasma and on clinical cfDNA plasma samples from cancer patients and non-cancer controls. This method introduces a new domain within bioinformatics and personalized medicine – somatic whole genome mutation calling for liquid biopsy.

目前的体细胞突变方法对液体活检（即低覆盖率测序）不起作用，我们将一个CNN架构应用于读数的独特表示和其识别，我们显示在低频设置中比以前的方法有显著的改进。

提出了一个基于CNN的解决方案，称为Kittyhawk，用于超低等位基因频率下的体细胞突变调用。

一种从无细胞DNA测序中检测癌症突变的新算法，该算法将识别测序错误与真正突变的序列背景。

本文提出了一种深度学习框架，用于预测在检测无细胞DNA肿瘤时发生的极低频率的体细胞突变。


This paper presents the formal release of {\em MedMentions}, a new manually annotated resource for the recognition of biomedical concepts. What distinguishes MedMentions from other annotated biomedical corpora is its size (over 4,000 abstracts and over 350,000 linked mentions), as well as the size of the concept ontology (over 3 million concepts from UMLS 2017) and its broad coverage of biomedical disciplines. In addition to the full corpus, a sub-corpus of MedMentions is also presented, comprising annotations for a subset of UMLS 2017 targeted towards document retrieval. To encourage research in Biomedical Named Entity Recognition and Linking, data splits for training and testing are included in the release, and a baseline model and its metrics for entity linking are also described.

本文介绍了一个新的黄金标准的生物医学科学文献语料库，该语料库由UMLS概念提及的人工注释而成。

详细介绍了一个涵盖生物医学概念的人工注释数据集的构建，该数据集比以前的数据集更大，并由更大的本体所覆盖。

本文使用MedMentions，一个用于端到端概念识别的TaggerOne半马尔可夫模型，并在一组Pubmed摘要上链接，为论文贴上生物医学概念/实体标签。


In this paper we propose a Deep Autoencoder Mixture Clustering (DAMIC) algorithm. It is based on a mixture of deep autoencoders where each cluster is represented by an autoencoder. A clustering network transforms the data into another space and then selects one of the clusters. Next, the autoencoder associated with this cluster is used to reconstruct the data-point. The clustering algorithm jointly learns the nonlinear data representation and the set of autoencoders. The optimal clustering is found by minimizing the reconstruction loss of the mixture of autoencoder network. Unlike other deep clustering algorithms, no regularization term is needed to avoid data collapsing to a single point. Our experimental evaluations on image and text corpora show significant improvement over state-of-the-art methods.

我们提出了一种深度聚类方法，其中每个聚类由一个自动编码器来代表，而不是中心点。

提出了基于自动编码器混合物的深度聚类，如果用自动编码器网络来表示，数据点就会被分配到一个聚类中，而这个聚类是基于表示误差的。

一种深度聚类方法，使用自动编码器框架同时学习数据的低维嵌入，同时使用深度神经网络进行数据聚类。

一种深度聚类方法，用不同的自动编码器表示每个聚类，以端到端的方式工作，也可用于对新进入的数据进行聚类，而不需要重做整个聚类程序。


We propose a new Integral Probability Metric (IPM) between distributions: the Sobolev IPM. The Sobolev IPM compares the mean discrepancy of two distributions for functions (critic) restricted to a Sobolev ball defined with respect to a dominant measure mu. We show that the Sobolev IPM compares two distributions in high dimensions based on weighted conditional Cumulative Distribution Functions (CDF) of each coordinate on a leave one out basis. The Dominant measure mu plays a crucial role as it defines the support on which conditional CDFs are compared. Sobolev IPM can be seen as an extension of the one dimensional Von-Mises Cramer statistics to high dimensional distributions. We show how Sobolev IPM can be used to train Generative Adversarial Networks (GANs). We then exploit the intrinsic conditioning implied by Sobolev IPM in text generation. Finally we show that a variant of Sobolev GAN achieves competitive results in semi-supervised learning on CIFAR-10, thanks to the smoothness enforced on the critic by Sobolev GAN which relates to Laplacian regularization.

我们定义了一个新的积分概率度量（Sobolev IPM），并展示了它如何被用于训练GAN的文本生成和半监督学习。

提出了一种基于Sobolev范数的GAN的新型正则化方案，衡量导数的L2范数之间的偏差。

作者提供了另一种类型的GAN，使用GAN的典型设置，但有不同的函数类，并制作了一个用这种函数类训练GAN的配方。

该论文为GAN critics提出了一个不同的梯度惩罚，强制梯度的预期平方范数等于1


We propose in this paper a new approach to train the Generative Adversarial Nets (GANs) with a mixture of generators to overcome the mode collapsing problem. The main intuition is to employ multiple generators, instead of using a single one as in the original GAN. The idea is simple, yet proven to be extremely effective at covering diverse data modes, easily overcoming the mode collapsing problem and delivering state-of-the-art results. A minimax formulation was able to establish among a classifier, a discriminator, and a set of generators in a similar spirit with GAN. Generators create samples that are intended to come from the same distribution as the training data, whilst the discriminator determines whether samples are true data or generated by generators, and the classifier specifies which generator a sample comes from. The distinguishing feature is that internal samples are created from multiple generators, and then one of them will be randomly selected as final output similar to the mechanism of a probabilistic mixture model. We term our method Mixture Generative Adversarial Nets (MGAN). We develop theoretical analysis to prove that, at the equilibrium, the Jensen-Shannon divergence (JSD) between the mixture of generators’ distributions and the empirical data distribution is minimal, whilst the JSD among generators’ distributions is maximal, hence effectively avoiding the mode collapsing problem. By utilizing parameter sharing, our proposed model adds minimal computational cost to the standard GAN, and thus can also efficiently scale to large-scale datasets. We conduct extensive experiments on synthetic 2D data and natural image databases (CIFAR-10, STL-10 and ImageNet) to demonstrate the superior performance of our MGAN in achieving state-of-the-art Inception scores over latest baselines, generating diverse and appealing recognizable objects at different resolutions, and specializing in capturing different types of objects by the generators.

我们提出了一种新的方法，用混合生成器训练GAN，以克服模式崩溃问题。

利用生成器的受限混合分布和预测源混合成分的辅助分类器，解决GAN中的模式崩溃问题。

本文提出了一种混合生成器来训练GAN，而没有额外的计算成本

作者提出，使用MGAN，旨在通过混合生成器克服模型崩溃问题，取得了最先进的结果。


Allowing humans to interactively train artificial agents to understand language instructions is desirable for both practical and scientific reasons.   Though, given  the lack of sample efficiency in current learning methods, reaching this goal may require substantial research efforts. We introduce the BabyAI research platform, with the goal of supporting investigations towards including humans in the loop for grounded language learning. The BabyAI platform comprises an extensible suite of 19 levels of increasing difficulty. Each level gradually leads the agent towards acquiring a combinatorially rich synthetic language, which is a proper subset of English. The platform also provides a hand-crafted bot agent, which simulates a human teacher.   We report estimated amount of supervision required for training neural reinforcement and behavioral-cloning agents on some BabyAI levels. We put forward strong evidence that current deep learning methods are not yet sufficiently sample-efficient in the context of learning a language with compositional properties.

我们提出了BabyAI平台，用于研究有人类参与的语言学习的数据效率。

提出了一个研究平台，其中有一个机器人在循环中学习执行语言指令，其中语言有构成性结构

介绍了一个基础语言学习的平台，该平台用启发式教师取代了循环中的任何人类，并使用映射到2D网格世界的合成语言。


Recently, there has been growing interest in methods that perform neural network compression, namely techniques that attempt to substantially reduce the size of a neural network without significant reduction in performance. However, most existing methods are post-processing approaches in that they take a learned neural network as input and output a compressed network by either forcing several parameters to take the same value (parameter tying via quantization) or pruning irrelevant edges (pruning) or both. In this paper, we propose a novel algorithm that jointly learns and compresses a neural network. The key idea in our approach is to change the optimization criteria by adding $k$ independent Gaussian priors over the parameters and a sparsity penalty. We show that our approach is easy to implement using existing neural network libraries, generalizes L1 and L2 regularization and elegantly enforces parameter tying as well as pruning constraints. Experimentally, we demonstrate that our new algorithm yields state-of-the-art compression on several standard benchmarks with minimal loss in accuracy while requiring little to no hyperparameter tuning as compared with related, competing approaches.

k-means先验与L1正则化相结合，产生了最先进的压缩结果。 本文探讨了DNN/CNN的软参数捆绑和压缩问题


The application of stochastic variance reduction to optimization has shown remarkable recent theoretical and practical success. The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem. We show that naive application of the SVRG technique and related approaches fail, and explore why.

SVRG方法在现代深度学习问题上失败了

本文对SVRG风格的方法进行了分析，显示出信息缺失、批量规范、数据增强（随机裁剪/轮换/翻译）往往会增加更新的偏差和/或变异。

本文研究了SVGD对现代神经网络的适用性，并表明SVGD的天真应用通常会失败。


The ground-breaking performance obtained by deep convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks. One of the main challenge in applying CNNs to 3D shape analysis is how to define a natural convolution operator on non-euclidean surfaces. In this paper, we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere. A cascade set of geodesic disk filters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks. We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance, and its effectiveness is evaluated in applications of non-rigid/ rigid shape classification and shape retrieval.

一种将深度学习应用于三维表面的方法，使用其球形描述符和2-球体上的alt-az各向异性卷积。

通过用过滤器旋转代替过滤器平移，提出了单位球体上的极性各向异性卷积方案。

本文探讨了使用alt-az各向异性2-球体卷积的3D形状的深度学习。


Recent breakthroughs in computer vision make use of large deep neural networks, utilizing the substantial speedup offered by GPUs. For applications running on limited hardware, however, high precision real-time processing can still be a challenge.   One approach to solving this problem is training networks with binary or ternary weights, thus removing the need to calculate multiplications and significantly reducing memory size. In this work, we introduce LR-nets (Local reparameterization networks), a new method for training neural networks with discrete weights using stochastic parameters. We show how a simple modification to the local reparameterization trick, previously used to train Gaussian distributed weights, enables the training of discrete weights. Using the proposed training we test both binary and ternary models on MNIST, CIFAR-10 and ImageNet benchmarks and reach state-of-the-art results on most experiments.

利用CLT近似的局部重参数化训练二元/三元网络

使用反向传播训练二元和三元权重分布网络，用重参数化技巧对神经元预激活进行采样

本文建议使用随机参数与局部重参数技巧相结合来训练具有二元或三元权重的神经网络，从而获得最先进的结果。


We present Optimal Completion Distillation (OCD), a training procedure for optimizing sequence to sequence models based on edit distance. OCD is efficient, has no hyper-parameters of its own, and does not require pre-training or joint optimization with conditional log-likelihood. Given a partial sequence generated by the model, we first identify the set of optimal suffixes that minimize the total edit distance, using an efficient dynamic programming algorithm.   Then, for each position of the generated sequence, we use a target distribution which puts equal probability on the first token of all the optimal suffixes. OCD achieves the state-of-the-art performance on end-to-end speech recognition, on both Wall Street Journal and Librispeech datasets, achieving $9.3\%$ WER and $4.5\%$ WER, respectively.

最佳完成度蒸馏法（Optimal Completion Distillation）是一种基于编辑距离的序列到序列模型的优化训练程序，它在端到端语音识别任务上达到了最先进的水平。

使用动态程序训练seq2seq模型的替代方法来计算预测前缀的最佳延续性

一种自动回归模型的训练算法，不需要任何MLE预训练，可以直接从采样中进行优化。

本文考虑了使用最大似然估计训练的序列对序列模型的缺陷，并提出了一种基于编辑距离和训练期间隐含使用给定标签序列的方法。


As an emerging field, federated learning has recently attracted considerable attention.  Compared to distributed learning in the datacenter setting, federated learning  has more strict constraints on computate efficiency of the learned model and communication  cost during the training process. In this work, we propose an efficient  federated learning framework based on variational dropout. Our approach is able  to jointly learn a sparse model while reducing the amount of gradients exchanged  during the iterative training process. We demonstrate the superior performance  of our approach on achieving significant model compression and communication  reduction ratios with no accuracy loss.

联合学习的联合模型和梯度稀疏化方法

应用变分丢弃来减少神经网络分布式训练的通信成本，并在mnist、cifar10和svhn数据集上做了实验。

作者提出了一种算法，通过将稀疏梯度从设备发送到服务器并返回，来减少联合学习中的通信成本。

将分布式优化算法与变分丢弃相结合，使本地学习者发送给主服务器的梯度更加稀疏。


We prove a multiclass boosting theory for the ResNet architectures which simultaneously creates a new technique for multiclass boosting and provides a new algorithm for ResNet-style architectures.   Our proposed training algorithm, BoostResNet, is particularly suitable in non-differentiable architectures.   Our method only requires the relatively inexpensive sequential training of T "shallow ResNets". We prove that the training error decays exponentially with the depth T if the weak module classifiers that we train perform slightly better than some weak baseline.   In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition.   A generalization error bound based on margin theory is proved and suggests that ResNet could be resistant to overfitting using a network with l_1 norm bounded weights.

我们证明了ResNet架构的多类提升理论，该理论同时创造了一种新的多类提升技术，并为ResNet风格的架构提供了一种新算法。

提出了一种训练深度残差网络的提升式算法，对训练误差进行收敛分析，并对泛化能力进行了分析。

一种使用提升框架的ResNet学习方法，分解了复杂网络的学习，并使用较少的计算成本。

作者提出将深度ResNet作为一种提升算法，他们声称这比标准的端到端反向传播更有效。


We consider the problem of learning a one-hidden-layer neural network: we assume the input x is from Gaussian distribution and the label $y = a \sigma(Bx) + \xi$, where a is a nonnegative vector and  $B$ is a full-rank weight matrix, and $\xi$ is a noise vector. We first give an analytic formula for the population risk of the standard squared loss and demonstrate that it implicitly attempts to decompose a sequence of low-rank tensors simultaneously.  	  Inspired by the formula, we design a non-convex objective function $G$ whose landscape is guaranteed to have the following properties:	  1. All local minima of $G$ are also global minima.  2. All global minima of $G$ correspond to the ground truth parameters.  3. The value and gradient of $G$ can be estimated using samples. 	  With these properties, stochastic gradient descent on $G$ provably converges to the global minimum and learn the ground-truth parameters. We also prove finite sample complexity results and validate the results by simulations.

本文分析了单隐层神经网络的优化情况，并设计了一个新的目标，证明其没有虚假的局部最小值。

本文研究了单隐层神经网络的学习问题，建立了最小二乘群体损失和Hermite多项式之间的联系，并提出了一个新的损失函数。

一种张量因子化的方法，用于倾斜一个隐藏层的神经网络


Open information extraction (OIE) systems extract relations and their   arguments from natural language text in an unsupervised manner. The resulting   extractions are a valuable resource for downstream tasks such as knowledge   base construction, open question answering, or event schema induction. In this   paper, we release, describe, and analyze an OIE corpus called OPIEC, which was   extracted from the text of English Wikipedia. OPIEC complements the available   OIE resources: It is the largest OIE corpus publicly available to date (over   340M triples) and contains valuable metadata such as provenance information,   confidence scores, linguistic annotations, and semantic annotations including   spatial and temporal information. We analyze the OPIEC corpus by comparing its   content with knowledge bases such as DBpedia or YAGO, which are also based on   Wikipedia. We found that most of the facts between entities present in OPIEC   cannot be found in DBpedia and/or YAGO, that OIE facts    often differ in the level of specificity compared to knowledge base facts, and   that OIE open relations are generally highly polysemous. We believe that the   OPIEC corpus is a valuable resource for future research on automated knowledge   base construction.

一个开放的信息提取语料库及其深入分析

为信息提取建立一个新的语料库，这个语料库比之前的公共语料库要大，并且包含当前语料库中不存在的信息。

介绍了在最近的提取系统的帮助下从维基百科上收集的开放式IE三要素的数据集。

本文描述了通过自动方式在英语维基百科上创建开放IE语料库的过程。


The process of designing neural architectures requires expert knowledge and extensive trial and error.  While automated architecture search may simplify these requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in both flexibility and components.  We propose a domain-specific language (DSL) for use in automated architecture search which can produce novel RNNs of arbitrary depth and width.  The DSL is flexible enough to define standard architectures such as the Gated Recurrent Unit and Long Short Term Memory and allows the introduction of non-standard RNN components such as trigonometric curves and layer normalization.   Using two different candidate generation techniques, random search with a ranking function and reinforcement learning,  we explore the novel architectures produced by the RNN DSL for language modeling and machine translation domains.  The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed.

我们为RNN架构的生成定义了一个灵活的DSL，允许不同大小和复杂程度的RNN，并提出了一个将RNN表示为递归神经网络的排名函数，模拟其性能以决定最有前途的架构。

介绍了一种新的方法来生成RNN架构，使用一种特定领域的语言来生成两种类型的生成器（随机和基于RL的）以及一个排序函数和评估器。

本文将寻找良好的RNN单元结构作为一个黑箱优化问题，其中例子被表示为运算符树，并根据学习到的函数或RL代理生成的函数进行评分。

本文通过使用指定RNN循环操作的DSL，研究了RNN背景下自动架构搜索的元学习策略。


Researches on deep neural networks with discrete parameters and their deployment in embedded systems have been active and promising topics. Although previous works have successfully reduced precision in inference, transferring both training and inference processes to low-bitwidth integers has not been demonstrated simultaneously. In this work, we develop a new method termed as ``"WAGE" to discretize both training and inference, where weights (W), activations (A), gradients (G) and errors (E) among layers are shifted and linearly constrained to low-bitwidth integers. To perform pure discrete dataflow for fixed-point devices, we further replace batch normalization by a constant scaling layer and simplify other components that are arduous for integer implementation. Improved accuracies can be obtained on multiple datasets, which indicates that WAGE somehow acts as a type of regularization. Empirically, we demonstrate the potential to deploy training in hardware systems such as integer-based deep learning accelerators and neuromorphic chips with comparable accuracy and higher energy efficiency, which is crucial to future AI applications in variable scenarios with transfer and continual learning demands.

我们在DNN中只应用低位宽整数的训练和推理

一种叫做WAGE的方法，它将神经网络中的所有操作数和运算符，以减少网络中的表示位数。

作者提出在神经网络的训练和测试时间内，对权重、激活、梯度和误差进行离散化处理。


Modern Convolutional Neural Networks (CNNs) are complex, encompassing millions of parameters. Their deployment exerts computational, storage and energy demands, particularly on embedded platforms. Existing approaches to prune or sparsify CNNs require retraining to maintain inference accuracy. Such retraining is not feasible in some contexts. In this paper, we explore the sparsification of CNNs by proposing three model-independent methods. Our methods are applied on-the-fly and require no retraining. We show that the state-of-the-art models' weights can be reduced by up to 73% (compression factor of 3.7x) without incurring more than 5% loss in Top-5 accuracy. Additional fine-tuning gains only 8% in sparsity, which indicates that our fast on-the-fly methods are effective.

在本文中，我们开发了快速的无再训练的稀疏化方法，可以在许多工业背景下对CNN进行即时的稀疏化部署。

本文通过引入三种方案来确定修剪权重的阈值，提出了无需再训练的CNN修剪方法。

本文介绍了一种无需再训练的CNN稀疏化方法。


Curriculum learning and Self paced learning are popular topics in the machine learning that suggest to put the training samples in order by considering their difficulty levels. Studies in these topics show that starting with a small training set and adding new samples according to difficulty levels improves the learning performance. In this paper we experimented that we can also obtain good results by adding the samples randomly without a meaningful order. We compared our method with classical training, Curriculum learning, Self paced learning and their reverse ordered versions. Results of the statistical tests show that the proposed method is better than classical method and similar with the others. These results point a new training regime that removes the process of difficulty level determination in Curriculum and Self paced learning and as successful as these methods.

我们提出，用成长集逐级训练为神经网络提供了一种优化。

作者将课程学习与随机顺序的学习进行了比较，在之前随机构建的集合中加入了新的例子样本的阶段。

本文研究了课程和自定进度学习中排序的影响，并表明在某种程度上，培训实例的排序并不重要。


We study the problem of learning to map, in an unsupervised way, between domains $A$ and $B$, such that the samples $\vb \in B$ contain all the information that exists in samples $\va\in A$ and some additional information. For example, ignoring occlusions, $B$ can be people with glasses, $A$ people without, and the glasses, would be the added information. When mapping a sample $\va$ from the first domain to the other domain, the missing information is replicated from an independent reference sample $\vb\in B$. Thus, in the above example, we can create, for every person without glasses a version with the glasses observed in any face image.    Our solution employs a single two-pathway encoder and a single decoder for both domains. The common part of the two domains and the separate part are encoded as two vectors, and the separate part is fixed at zero for domain $A$. The loss terms are minimal and involve reconstruction losses for the two domains and a domain confusion term. Our analysis shows that under mild assumptions, this architecture, which is much simpler than the literature guided-translation methods, is enough to ensure disentanglement between the two domains. We present convincing results in a few visual domains, such as no-glasses to glasses, adding facial hair based on a reference image, etc.

一种图像对图像的翻译方法，在一个图像中加入另一个图像的内容，从而形成一个新的图像。

本文处理的是内容转移的任务，其新颖性在损失上。


Mathematical reasoning---a core ability within human intelligence---presents some unique challenges as a domain: we do not come to understand and solve mathematical problems primarily on the back of experience and evidence, but on the basis of inferring, learning, and exploiting laws, axioms, and symbol manipulation rules. In this paper, we present a new challenge for the evaluation (and eventually the design) of neural architectures and similar system, developing a task suite of mathematics problems involving sequential questions and answers in a free-form textual input/output format. The structured nature of the mathematics domain, covering arithmetic, algebra, probability and calculus, enables the construction of training and test spits designed to clearly illuminate the capabilities and failure-modes of different architectures, as well as evaluate their ability to compose and relate knowledge and learned processes. Having described the data generation process and its potential future expansions, we conduct a comprehensive analysis of models from two broad classes of the most powerful sequence-to-sequence architectures and find notable differences in their ability to resolve mathematical problems and generalize their knowledge.

一个用于测试数学推理（和代数泛化）的数据集，以及关于当前序列到序列模型的结果。

提出了一个新的合成数据集来评估序列对序列模型的数学推理能力，并利用它来评估多个模型。

解决基本数学问题的模式。


Convolutional Neural Networks (CNNs) filter the input data using a series of spatial convolution operators with compactly supported stencils and point-wise nonlinearities.  Commonly, the convolution operators couple features from all channels.  For wide networks, this leads to immense computational cost in the training of and prediction with CNNs.  In this paper, we present novel ways to parameterize the convolution more efficiently, aiming to decrease the number of parameters in CNNs and their computational complexity.  We propose new architectures that use a sparser coupling between the channels and thereby reduce both the number of trainable weights and the computational cost of the CNN.  Our architectures arise as new types of residual neural network (ResNet) that can be seen as discretizations of a Partial Differential Equations (PDEs) and thus have predictable theoretical properties. Our first architecture involves a convolution operator with a special sparsity structure, and is applicable to a large class of CNNs. Next, we present an architecture that can be seen as a discretization of a diffusion reaction PDE, and use it with three different convolution operators. We outline in our experiments that the proposed architectures,  although considerably reducing the number of trainable weights, yield comparable accuracy to existing CNNs that are fully coupled in the channel dimension.

本文介绍了以偏微分方程为动机的卷积神经网络的高效和经济参数化。

介绍了标准卷积运算的四个 "低成本 "替代方案，可以用来代替标准卷积运算，以降低其计算复杂性。

本文介绍了降低CNN实现的计算成本的方法，并介绍了限制参数耦合的类似CNN架构的新参数化。

本文提出了一个基于PDE的视角来理解CNN并将其参数化


In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equivalently a lower bound on the negative log likelihood) that the model can assign to some data as one varies the prior and/or the likelihood function in a latent variable model. The core of our contribution is to formally show that the problem of optimizing priors in latent variable models is exactly an instance of the variational optimization problem that information theorists solve when computing rate-distortion functions, and then to use this to derive a lower bound on negative log likelihood. Moreover, we will show that if changing the prior can improve the log likelihood, then there is a way to change the likelihood function instead and attain the same log likelihood, and thus rate-distortion theory is of relevance to both optimizing priors as well as optimizing likelihood functions. We will experimentally argue for the usefulness of quantities derived from rate-distortion theory in latent variable modeling by applying them to a problem in image modeling.

使用速率失真理论来约束潜在变量模型的改进程度

通过提出基于负对数似然的下限的标准，解决了潜变量模型中先验的优化问题和似然函数的选择。

提出了一个定理，给出了潜在变量建模的速率扭曲的负对数似然的下限

作者认为，有损压缩的速率-失真理论为研究潜变量模型提供了一个自然的工具包，提出了一个下限。


Backprop is the primary learning algorithm used in many machine learning algorithms. In practice, however, Backprop in deep neural networks is a highly sensitive learning algorithm and successful learning depends on numerous conditions and constraints. One set of constraints is to avoid weights that lead to saturated units. The motivation for avoiding unit saturation is that gradients vanish and as a result learning comes to a halt. Careful weight initialization and re-scaling schemes such as batch normalization ensure that input activity to the neuron is within the linear regime where gradients are not vanished and can flow. Here we investigate backpropagating error terms only linearly. That is, we ignore the saturation that arise by ensuring gradients always flow. We refer to this learning rule as Linear Backprop since in the backward pass the network appears to be linear. In addition to ensuring persistent gradient flow, Linear Backprop is also favorable when computation is expensive since gradients are never computed. Our early results suggest that learning with Linear Backprop is competitive with Backprop and saves expensive gradient computations.

我们忽略了非线性，在后向传递中不计算梯度，以节省计算量，并确保梯度始终流动。 作者提出了线性反向传播算法，以确保反向传播过程中所有部分的梯度流动。


Deep neural networks with discrete latent variables offer the promise of better symbolic reasoning, and learning  abstractions that are more useful to new tasks. There has been a surge in interest in discrete latent variable models,  however, despite several recent improvements, the training of discrete latent variable models has remained  challenging and their performance has mostly failed to match their continuous counterparts. Recent work on vector quantized autoencoders (VQ-VAE) has made substantial progress in this direction, with its perplexity almost matching that of a VAE on datasets such as CIFAR-10. In this work, we investigate an alternate training technique for VQ-VAE, inspired by its connection to the Expectation Maximization (EM) algorithm. Training the discrete autoencoder with EM and combining it with sequence  level knowledge distillation alows us to develop a non-autoregressive machine translation model whose accuracy almost matches a strong greedy autoregressive baseline Transformer, while being 3.3 times faster at inference.

利用EM系统地理解VQ-VAE离散自动编码器，并利用它来设计匹配强自回归基线的非自回归翻译模型。

本文介绍了一种解释VQ-VAE的新方法，并提出了一种基于软EM聚类的新训练算法。

本文对使用软电磁算法的VQ-VAE的训练程序提出了另一种观点


Recent research about margin theory has proved that maximizing the minimum margin like support vector machines does not necessarily lead to better performance, and instead, it is crucial to optimize the margin distribution. In the meantime, margin theory has been used to explain the empirical success of deep network in recent studies. In this paper, we present ODN (the Optimal margin Distribution Network), a network which embeds a loss function in regard to the optimal margin distribution. We give a theoretical analysis for our method using the PAC-Bayesian framework, which confirms the significance of the margin distribution for classification within the framework of deep networks. In addition, empirical results show that the ODN model always outperforms the baseline cross-entropy loss model consistently across different regularization situations. And our ODN  model also outperforms the cross-entropy loss (Xent), hinge loss and soft hinge loss model in generalization task through limited training data.

本文提出了一个在最佳边际分布方面嵌入损失函数的深度神经网络，在理论上和经验上缓解了过拟合问题。


Deep network compression seeks to reduce the number of parameters in the network while maintaining a certain level of performance.   Deep network distillation seeks to train a smaller network that matches soft-max performance of a larger network.   While both regimes have led to impressive performance for their respective goals, neither provide insight into the importance of a given layer in the original model, which is useful if we are to improve our understanding of these highly parameterized models.   In this paper, we present the concept of deep net triage, which individually assesses small blocks of convolution layers to understand their collective contribution to the overall performance, which we call \emph{criticality}.  We call it triage because we assess this criticality by answering the question: what is the impact to the health of the overall network if we compress a block of layers into a single layer.  We propose a suite of triage methods and compare them on problem spaces of varying complexity.   We ultimately show that, across these problem spaces, deep net triage is able to indicate the of relative importance of different layers.   Surprisingly, our local structural compression technique also leads to an improvement in overall accuracy when the final model is fine-tuned globally.

我们试图通过一种我们称之为深度网络分流的实验机制来了解压缩网络中的学习表征

比较各种初始化和训练方法，将知识从VGG网络转移到一个较小的学生网络，用单层取代层块。

本文介绍了五种为深度网络做分流或块层压缩的方法。

本文提出了一种压缩NN中的层块的方法，评估了几种不同的子方法


In this paper, we show a phenomenon, which we named ``super-convergence'', where residual networks can be trained using an order of magnitude fewer iterations than is used with standard training methods.    The existence of super-convergence is relevant to understanding why deep networks generalize well.   One of the key elements of super-convergence is training with cyclical learning rates and a large maximum learning rate.   Furthermore, we present evidence that training with large learning rates improves performance by regularizing the network. In addition, we show that super-convergence provides a  greater boost in performance relative to standard training when the amount of labeled training data is limited.   We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate.   The architectures to replicate this work will be made available upon publication.

对一个新现象的经验证明需要新的理论见解，并与文献中关于SGD和理解泛化的积极讨论相关。

本文讨论了一个现象，即在非常特定的环境中，神经网络训练可以从包括大的学习率的时间表中获益良多

作者分析了使用大循环学习率训练残差网络的情况，并证明了循环学习率的快速收敛，以及大学习率作为正则化的证据。


Infinite-width neural networks have been extensively used to study the theoretical properties underlying the extraordinary empirical success of standard, finite-width neural networks. Nevertheless, until now, infinite-width networks have been limited to at most two hidden layers. To address this shortcoming, we study the initialisation requirements of these networks and show that the main challenge for constructing them is defining the appropriate sampling distributions for the weights. Based on these observations, we propose a principled approach to weight initialisation that correctly accounts for the functional nature of the hidden layer activations and facilitates the construction of arbitrarily many infinite-width layers, thus enabling the construction of arbitrarily deep infinite-width networks. The main idea of our approach is to iteratively reparametrise the hidden-layer activations into appropriately defined reproducing kernel Hilbert spaces and use the canonical way of constructing probability distributions over these spaces for specifying the required weight distributions in a principled way. Furthermore, we examine the practical implications of this construction for standard, finite-width networks. In particular, we derive a novel weight initialisation scheme for standard, finite-width networks that takes into account the structure of the data and information about the task at hand. We demonstrate the effectiveness of this weight initialisation approach on the MNIST, CIFAR-10 and Year Prediction MSD datasets.

我们提出了一种构建任意深度的无限宽网络的方法，在此基础上，我们得出了一种新的有限宽网络的权重初始化方案，并证明了其具有竞争力的性能。

提出了一种权重初始化方法，以实现无限深和无限宽的网络，并在小数据集上取得了实验结果。

提出了无限宽度的深度神经网络。