This paper proposes a Pruning in Training (PiT) framework of learning to reduce the parameter size of networks.Different from existing works, our PiT framework employs the sparse penalties to train networks and thus help rank the importance of weights and filters.Our PiT algorithms can directly prune the network without any fine-tuning.The pruned networks can still achieve comparable performance to the original networks.In particular, we introduce the (Group) Lasso-type Penalty (L-P /GL-P), and (Group) Split LBI Penalty (S-P / GS-P) to regularize the networks, and a pruning strategy proposed  is used in help prune the network.We conduct the extensive experiments on MNIST, Cifar-10, and miniImageNet.The results validate the efficacy of our proposed methods.Remarkably, on MNIST dataset, our PiT framework can save 17.5% parameter size of LeNet-5, which achieves the 98.47% recognition accuracy.

我们提出了一种通过强制执行结构稀疏度惩罚来学习修剪网络的算法。 本文介绍了一种在训练网络时使用套索和分割LBI惩罚的修剪方法。


We first pose the Unsupervised Continual Learning (UCL) problem: learning salient representations from a non-stationary stream of unlabeled data in which the number of object classes varies with time.Given limited labeled data just before inference, those representations can also be associated with specific object types to perform classification.To solve the UCL problem, we propose an architecture that involves a single module, called Self-Taught Associative Memory (STAM), which loosely models the function of a cortical column in the mammalian brain.Hierarchies of STAM modules learn based on a combination of Hebbian learning, online clustering, detection of novel patterns and forgetting outliers, and top-down predictions.We illustrate the operation of STAMs in the context of learning handwritten digits in a continual manner with only 3-12 labeled examples per class.STAMs suggest a promising direction to solve the UCL problem without catastrophic forgetting.

我们介绍了无监督持续学习（UCL）和一个解决UCL问题的神经启发架构。

建议使用STAM模块的层次来解决UCL问题，提供证据表明模块所学的表征非常适合于少样本分类。


Recent advances have made it possible to create deep complex-valued neural networks.Despite this progress, the potential power of fully complex intermediate computations and representations has not yet been explored for many challenging learning problems.Building on recent advances, we propose a novel mechanism for extracting signals in the frequency domain.As a case study, we perform audio source separation in the Fourier domain.Our extraction mechanism could be regarded as a local ensembling method that combines a complex-valued convolutional version of Feature-Wise Linear Modulation (FiLM) and a signal averaging operation.We also introduce a new explicit amplitude and phase-aware loss, which is scale and time invariant, taking into account the complex-valued components of the spectrogram.Using the Wall Street Journal Dataset, we compare our phase-aware loss to several others that operate both in the time and frequency domains and demonstrate the effectiveness of our proposed signal extraction method and proposed loss.When operating in the complex-valued frequency domain, our deep complex-valued network substantially outperforms its real-valued counterparts even with half the depth and a third of the parameters.Our proposed mechanism improves significantly deep complex-valued networks' performance and we demonstrate the usefulness of its regularizing effect.

傅里叶域中的新信号提取方法

贡献了一个复值卷积版本的逐特征线性调制，它允许参数优化并设计了一个考虑到幅度和相位的损失。


It is challenging to disentangle an object into two orthogonal spaces of content and style since each can influence the visual observation in a different and unpredictable way.It is rare for one to have access to a large number of data to help separate the influences.In this paper, we present a novel framework to learn this disentangled representation in a completely unsupervised manner.We address this problem in a two-branch Autoencoder framework.For the structural content branch, we project the latent factor into a soft structured point tensor and constrain it with losses derived from prior knowledge.This encourages the branch to distill geometry information.Another branch learns the complementary style information.The two branches form an effective framework that can disentangle object's content-style representation without any human annotation.We evaluate our approach on four image datasets, on which we demonstrate the superior disentanglement and visual analogy quality both in synthesized and real-world data.We are able to generate photo-realistic images with 256x256 resolution that are clearly disentangled in content and style.

我们提出了一个新的框架，以完全无监督的方式学习内容和风格的分离表示。

提出了基于自动编码器框架的模型，以分解对象的表征，结果显示，该模型可以产生捕捉内容和风格的表征。


We develop the Y-learner for estimating heterogeneous treatment effects in experimental and observational studies.The Y-learner is designed to leverage the abilities of neural networks to optimize multiple objectives and continually update, which allows for better pooling of underlying feature information between treatment and control groups.We evaluate the Y-learner on three test problems: (1) A set of six simulated data benchmarks from the literature.(2) A real-world large-scale experiment on voter persuasion.(3) A task from the literature that estimates artificially generated treatment effects on MNIST didgits.The Y-learner achieves state of the art results on two of the three tasks.On the MNIST task, it gets the second best results.

我们开发了一种CATE估计策略，利用了神经网络的一些有趣的特性。

通过对治疗响应函数、控制响应函数以及从估算的治疗效果到条件平均治疗效果的映射进行建模，显示了对X-learner的改进，作为神经网络。

作者提出用Y-learner来估计条件平均治疗效果（CATE），它同时更新结果函数的参数和CATE估计器。


With the rapid proliferation of IoT devices, our cyberspace is nowadays dominated by billions of low-cost computing nodes, which expose an unprecedented heterogeneity to our computing systems.Dynamic analysis, one of the most effective approaches to finding software bugs, has become paralyzed due to the lack of a generic emulator capable of running diverse previously-unseen firmware.In recent years, we have witnessed devastating security breaches targeting IoT devices.These security concerns have significantly hamstrung further evolution of IoT technology.In this work, we present Laelaps, a device emulator specifically designed to run diverse software on low-cost IoT devices.We do not encode into our emulator any specific information about a device.Instead, Laelaps infers the expected behavior of firmware via symbolic-execution-assisted peripheral emulation and generates proper inputs to steer concrete execution on the fly.This unique design feature makes Laelaps the first generic device emulator capable of running diverse firmware with no a priori knowledge about the target device.To demonstrate the capabilities of Laelaps, we deployed two popular dynamic analysis techniques---fuzzing testing and dynamic symbolic execution---on top of our emulator.We successfully identified both self-injected and real-world vulnerabilities.


Deep neural models, such as convolutional and recurrent networks, achieve phenomenal results over spatial data such as images and text.However, when considering tabular data, gradient boosting of decision trees (GBDT) remains the method of choice.Aiming to bridge this gap, we propose \emph{deep neural forests} (DNF)--  a novel architecture that combines elements from decision trees as well as dense residual connections. We present the results of extensive empirical study in which we examine the performance of GBDTs, DNFs and (deep) fully-connected networks. These results indicate that DNFs achieve comparable results to GBDTs on tabular data, and open the door to end-to-end neural modeling of multi-modal data.To this end, we present a successful application of DNFs as part of a hybrid architecture for a multi-modal driving scene understanding classification task.

一个用于表格数据的架构，它模拟了决策树的分支，并使用密集的残差连接性

本文提出了深度神经森林，这是一种针对表格数据的算法，整合了决策树梯度提升的强项。

一种模仿决策森林工作方式的新型神经网络架构，解决了为表格数据训练深度模型的一般问题，并展示了与GBDT相同的有效性。


Hyperparameter tuning is one of the most time-consuming workloads in deep learning.State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable.Recently researchers have shown renewed interest in simpler methods like momentum SGD as they may yield better results.Motivated by this trend, we ask: can simple adaptive methods, based on SGD perform as well or better?We revisit the momentum SGD algorithm and show that hand-tuning a single learning rate and momentum makes it competitive with Adam.We then analyze its robustness to learning rate misspecification and objective curvature variation.Based on these insights, we design YellowFin, an automatic tuner for momentum and learning rate in SGD.YellowFin optionally uses a negative-feedback loop to compensate for the momentum dynamics in asynchronous settings on the fly.We empirically show YellowFin can converge in fewer iterations than Adam on ResNets and LSTMs for image recognition, language modeling and constituency parsing, with a speedup of up to $3.28$x in synchronous and up to $2.69$x in asynchronous settings.

YellowFin是一个基于SGD的优化器，具有动量和学习率的适应性。

提出了一种在动量SGD方法中自动调整动量参数的方法，该方法取得了比最先进的Adam算法更好的结果和快速收敛速度。


Robustness and security of machine learning (ML) systems are intertwined, wherein a non-robust ML system (classifiers, regressors, etc.) can be subject to attacks using a wide variety of exploits.With the advent of scalable deep learning methodologies, a lot of emphasis has been put on the robustness of supervised, unsupervised and reinforcement learning algorithms.Here, we study the robustness of the latent space of a deep variational autoencoder (dVAE), an unsupervised generative framework, to show that it is indeed possible to perturb the latent space, flip the class predictions and keep the classification probability approximately equal before and after an attack.This means that an agent that looks at the outputs of a decoder would remain oblivious to an attack.

对变分自动编码器的潜在空间进行对抗性攻击以改变输入的语义

本文涉及安全和机器学习，提出了一种中间人攻击，改变了输入数据的VAE编码，使解码后的输出会被错误分类。


Graph-based dependency parsing consists of two steps: first, an encoder produces a feature representation for each parsing substructure of the input sentence, which is then used to compute a score for the substructure; and second, a decoder} finds the parse tree whose substructures have the largest total score.Over the past few years, powerful neural techniques have been introduced into the encoding step which substantially increases parsing accuracies.However, advanced decoding techniques, in particular high-order decoding, have seen a decline in usage.It is widely believed that contextualized features produced by neural encoders can help capture high-order decoding information and hence diminish the need for a high-order decoder.In this paper, we empirically evaluate the combinations of different neural and non-neural encoders with first- and second-order decoders and provide a comprehensive analysis about the effectiveness of these combinations with varied training data sizes.We find that: first, when there is large training data, a strong neural encoder with first-order decoding is sufficient to achieve high parsing accuracy and only slightly lags behind the combination of neural encoding and second-order decoding; second, with small training data, a non-neural encoder with a second-order decoder outperforms the other combinations in most cases.

检验不同编码器-解码器组合对依赖性解析任务的有效性的实证研究 实证性地分析了各种编码器、解码器以及它们对基于图的依赖性解析的依赖性。


Meta-learning will be crucial to creating lifelong, generalizable AI.In practice, however, it is hard to define the meta-training task distribution that is used to train meta-learners.If made too small, tasks are too similar for a model to meaningfully generalize.If made too large, generalization becomes incredibly difficult.We argue that both problems can be alleviated by introducing a teacher model that controls the sequence of tasks that a meta-learner is trained on.This teacher model is incentivized to start the student meta-learner on simple tasks then adaptively increase task difficulty in response to student progress.While this approach has been previously studied in curriculum generation, our main contribution is in extending it to meta-learning.

像人类一样训练元学习者的教师


Using higher order knowledge to reduce training data has become a popular research topic.However, the ability for available methods to draw effective decision boundaries is still limited: when training set is small, neural networks will be biased to certain labels.Based on this observation, we consider constraining output probability distribution as higher order domain knowledge.We design a novel algorithm that jointly optimizes output probability distribution on a clustered embedding space to make neural networks draw effective decision boundaries.  While directly applying probability constraint is not effective, users need to provide additional very weak supervisions: mark some batches that have output distribution greatly differ from target probability distribution.We use experiments to empirically prove that our model can converge to an accuracy higher than other state-of-art semi-supervised learning models with less high quality labeled training examples.

我们引入了一种嵌入空间的方法来约束神经网络的输出概率分布。

本文介绍了一种用深度神经网络进行半监督学习的方法，在训练规模较小的情况下，该模型取得了相对较高的准确性。

本文在训练实例数量有限的情况下，将标签分布纳入模型学习，并提出了两种处理输出标签分布错误偏差问题的技术。


We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy).  Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pretrained on a large text corpus.We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis.  We also present an analysis showing that exposing the deep internals of the pretrained network is crucial, allowing downstream models to mix different types of semi-supervision signals.

我们介绍了一种新型的深度语境化词汇表征，在一系列具有挑战性的NLP任务中显著提高了技术水平。


This work addresses the long-standing problem of robust event localization in the presence of temporally of misaligned labels in the training data.We propose a novel versatile loss function that generalizes a number of training regimes from standard fully-supervised cross-entropy to count-based weakly-supervised learning.Unlike classical models which are constrained to strictly fit the annotations during training, our soft localization learning approach relaxes the reliance on the exact position of labels instead.Training with this new loss function exhibits strong robustness to temporal misalignment of labels, thus alleviating the burden of precise annotation of temporal sequences.We demonstrate state-of-the-art performance against standard benchmarks in a number of challenging experiments and further show that robustness to label noise is not achieved at the expense of raw performance.

这项工作引入了一个新的损失函数，用于在标签不一致的情况下对时间定位DNN进行稳健性训练。

一种新的损失，用于训练模型，通过比较平滑的标签和预测序列，预测事件在具有噪声标签的训练序列中的发生位置。


The driving force behind deep networks is their ability to compactly represent rich classes of functions.The primary notion for formally reasoning about this phenomenon is expressive efficiency, which refers to a situation where one network must grow unfeasibly large in order to replicate functions of another.To date, expressive efficiency analyses focused on the architectural feature of depth, showing that deep networks are representationally superior to shallow ones.In this paper we study the expressive efficiency brought forth by connectivity, motivated by the observation that modern networks interconnect their layers in elaborate ways.We focus on dilated convolutional networks, a family of deep models delivering state of the art performance in sequence processing tasks.By introducing and analyzing the concept of mixed tensor decompositions, we prove that interconnecting dilated convolutional networks can lead to expressive efficiency.In particular, we show that even a single connection between intermediate layers can already lead to an almost quadratic gap, which in large-scale settings typically makes the difference between a model that is practical and one that is not.Empirical evaluation demonstrates how the expressive efficiency of connectivity, similarly to that of depth, translates into gains in accuracy.This leads us to believe that expressive efficiency may serve a key role in developing new tools for deep network design.

我们介绍了混合张量分解的概念，并利用它来证明扩张卷积网络的互连能提高其表达能力。

本文从理论上验证了使用混合张量分解，将不同扩张的网络相互连接可以带来表达效率。

作者研究了扩张卷积网络，并表明在不同阶段交织两个扩张卷积网络A和B，比不交织更有表达效率。

表明WaveNet的单一完美二叉树的结构假设阻碍了它的性能，具有更复杂的混合树结构的类似WaveNet的架构表现更好。


We apply multi-task learning to image classification tasks on MNIST-like datasets.MNIST dataset has been referred to as the {\em drosophila} of machine learning and has been the testbed of many learning theories.The NotMNIST dataset and the FashionMNIST dataset have been created with the MNIST dataset as reference.In this work, we exploit these MNIST-like datasets for multi-task learning.The datasets are pooled together for learning the parameters of joint classification networks.Then the learned parameters are used as the initial parameters to retrain disjoint classification networks.The baseline recognition model are all-convolution neural networks.Without multi-task learning, the recognition accuracies for MNIST, NotMNIST and FashionMNIST are 99.56\%, 97.22\% and 94.32\% respectively.With multi-task learning to pre-train the networks, the recognition accuracies are respectively 99.70\%, 97.46\% and 95.25\%.The results re-affirm that multi-task learning framework, even with data with different genres, does lead to significant improvement.

多任务学习的作用 本文介绍了一种用于类似MNIST数据集分类的多任务神经网络


Recent work has demonstrated that neural networks are vulnerable to adversarial examples, i.e., inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network.To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization.This approach provides us with a broad and unifying view on much prior work on this topic.Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal.In particular, they specify a concrete security guarantee that would protect against a well-defined class of adversaries.These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks.They also suggest robustness against a first-order adversary as a natural security guarantee.We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models.

我们对对抗性例子的概念进行了原则性的、基于优化的重新审视，并开发了产生对抗性模型的方法，这些模型对广泛的对抗者具有对抗性的稳健性。

研究深度网络学习的最小最大公式，以提高其稳健性，使用投影梯度下降作为主要对手。

本文拟通过鞍点问题的框架，研究如何使神经网络抵抗对抗性损失。


In recent years there has been a rapid increase in classification methods on graph structured data.Both in graph kernels and graph neural networks, one of the implicit assumptions of successful state-of-the-art models was that incorporating graph isomorphism features into the architecture leads to better empirical performance.However, as we discover in this work, commonly used data sets for graph classification have repeating instances which cause the problem of isomorphism bias, i.e. artificially increasing the accuracy of the models by memorizing target information from the training set.This prevents fair competition of the algorithms and raises a question of the validity of the obtained results.We analyze 54 data sets, previously extensively used for graph-related tasks, on the existence of isomorphism bias, give a set of recommendations to machine learning practitioners to properly set up their models, and open source new data sets for the future experiments.

许多图分类数据集都有重复的数据，因此提出了关于泛化能力和模型公平比较的疑问。

作者讨论了图数据集中的同构偏差，即当图同构特征被纳入模型时，学习网络的过拟合效应，理论上类似于数据泄露效应。


Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently.However, learning from demonstrations often suffers from the covariate shift problem, which resultsin cascading errors of the learned policy.We introduce a notion of conservatively extrapolated value functions, which provably lead to policies with self-correction.We design an algorithm Value Iteration with Negative Sampling (VINS) that practically learns such value functions with conservative extrapolation.We show that VINS can correct mistakes of the behavioral cloning policy on simulated robotics benchmark tasks.We also propose the algorithm of using VINS to initialize a reinforcement learning algorithm, which is shown to outperform prior works in sample efficiency.

我们引入了一个保守外推值函数的概念，它可以证明导致策略可以自我修正以保持接近示范状态，并通过一个新的负采样技术学习它们。

一种被称为负采样的价值迭代的算法，以解决模仿学习中的协变量转移问题。


A structured understanding of our world in terms of objects, relations, and hierarchies is an important component of human cognition.Learning such a structured world model from raw sensory data remains a challenge.As a step towards this goal, we introduce Contrastively-trained Structured World Models (C-SWMs).C-SWMs utilize a contrastive approach for representation learning in environments with compositional structure.We structure each state embedding as a set of object representations and their relations, modeled by a graph neural network.This allows objects to be discovered from raw pixel observations without direct supervision as part of the learning process.We evaluate C-SWMs on compositional environments involving multiple interacting objects that can be manipulated independently by an agent, simple Atari games, and a multi-object physics simulation.Our experiments demonstrate that C-SWMs can overcome limitations of models based on pixel reconstruction and outperform typical representatives of this model class in highly structured environments, while learning interpretable object-based representations.

对比训练的结构化世界模型（C-SWMs）从原始像素输入中学习面向对象的状态表示和环境的关系模型。

作者通过使用一个对比性的潜在空间，克服了在构建和学习结构化世界模型时使用基于像素的损失的问题。


Neural machine translation (NMT) models learn representations containing substantial linguistic information.However, it is not clear if such information is fully distributed or if some of it can be attributed to individual neurons.We develop unsupervised methods for discovering important neurons in NMT models.Our methods rely on the intuition that different models learn similar properties, and do not require any costly external supervision.We show experimentally that translation quality depends on the discovered neurons, and find that many of them capture common linguistic phenomena.Finally, we show how to control NMT translations in predictable ways, by modifying activations of individual neurons.

寻找、分析和控制NMT中重要神经元的无监督方法

这项工作提出在神经机器翻译模型中寻找 "有意义的 "神经元，方法是根据模型对、不同历时或不同数据集之间的相关性进行排名，并提出模型的控制机制。


Computations for the softmax function in neural network models are expensive when the number of output classes is large.This can become a significant issue in both training and inference for such models.In this paper, we present Doubly Sparse Softmax (DS-Softmax), Sparse Mixture of Sparse of Sparse Experts, to improve the efficiency for softmax inference.During training, our method learns a two-level class hierarchy by dividing entire output class space into several partially overlapping experts.Each expert is responsible for a learned subset of the output class space and each output class only belongs to a small number of those experts.During inference, our method quickly locates the most probable expert to compute small-scale softmax.Our method is learning-based and requires no knowledge of the output class partition space a priori.We empirically evaluate our method on several real-world tasks and demonstrate that we can achieve significant computation reductions without loss of

我们提出了双重稀疏softmax，即稀疏专家的稀疏混合物，通过利用两级重叠的层次结构来提高softmax推理的效率。

本文提出了新的Softmax算法实现，该算法具有两个层次的稀疏性，加快了语言建模的操作。


Our work presents empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance.Compared to previously studied indicators of generalization, we show that layer rotation has the additional benefit of being easily monitored and controlled, as well as having a network-independent optimum: the training procedures during which all layers' weights reach a cosine distance of 1 from their initialization consistently outperform other configurations -by up to 20% test accuracy.Finally, our results also suggest that the study of layer rotation can provide a unified framework to explain the impact of weight decay and adaptive gradient methods on generalization.

本文提出了支持发现泛化指标的实证证据：每个层的权重向量与其初始化之间的余弦距离在整个训练中的演变。


Models of code can learn distributed representations of a program's syntax and semantics to predict many non-trivial properties of a program.Recent state-of-the-art models leverage highly structured representations of programs, such as trees, graphs and paths therein (e.g. data-flow relations), which are precise and abundantly available for code.This provides a strong inductive bias towards semantically meaningful relations, yielding more generalizable representations than classical sequence-based models.Unfortunately, these models primarily rely on graph-based message passing to represent relations in code, which makes them de facto local due to the high cost of message-passing steps, quite in contrast to modern, global sequence-based models, such as the Transformer.In this work, we bridge this divide between global and structured models by introducing two new hybrid model families that are both global and incorporate structural bias: Graph Sandwiches, which wrap traditional (gated) graph message-passing layers in sequential message-passing layers; and Graph Relational Embedding Attention Transformers (GREAT for short), which bias traditional Transformers with relational information from graph edge types.By studying a popular, non-trivial program repair task, variable-misuse identification, we explore the relative merits of traditional and hybrid model families for code representation.Starting with a  graph-based model that already improves upon the prior state-of-the-art for this task by 20%, we show that our proposed hybrid models improve an additional 10-15%, while training both faster and using fewer parameters.

结合全局和结构特征的源代码模型可以学习到更强大的程序表征。

一种新的方法，使用像[RNN GNN RNN]这样的三明治模型来为修复错误的任务建立源代码模型，这可以显著提高定位和修复的准确性。


Recurrent neural networks (RNNs) are particularly well-suited for modeling long-term dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate.While a number of works attempt to mitigate this effect through gated recurrent units, skip-connections, parametric constraints and design choices, we propose a novel incremental RNN (iRNN), where hidden state vectors keep track of incremental changes, and as such approximate state-vector increments of Rosenblatt's (1962) continuous-time RNNs.iRNN exhibits identity gradients and is able to account for long-term dependencies (LTD).We show that our method is computationally efficient overcoming overheads of many existing methods that attempt to improve RNN training, while suffering no performance degradation.We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks.

增量式RNN通过基于先前的状态和ODE预测的状态之间的差异更新状态向量来解决梯度爆炸/消失问题。

作者通过建立一个信号转换的吸引子系统并检查它是否收敛到一个平衡点来解决递归神经网络中的信号传播问题。


Recent empirical results on over-parameterized deep networks are marked by a striking absence of the classic U-shaped test error curve: test error keeps decreasing in wider networks.Researchers are actively working on bridging this discrepancy by proposing better complexity measures.Instead, we directly measure prediction bias and variance for four classification and regression tasks on modern deep networks.We find that both bias and variance can decrease as the number of parameters grows.Qualitatively, the phenomenon persists over a number of gradient-based optimizers.To better understand the role of optimization, we decompose the total variance into variance due to training set sampling and variance due to initialization.Variance due to initialization is significant in the under-parameterized regime.In the over-parameterized regime, total variance is much lower and dominated by variance due to sampling.We provide theoretical analysis in a simplified setting that is consistent with our empirical findings.

我们提供证据反对关于偏差-方差权衡的经典说法，并提出了一个新的方差分解。


Real world images often contain large amounts of private / sensitive information that should be carefully protected without reducing their utilities.In this paper, we propose a privacy-preserving deep learning framework with a learnable ob- fuscator for the image classification task.Our framework consists of three mod- els: learnable obfuscator, classifier and reconstructor.The learnable obfuscator is used to remove the sensitive information in the images and extract the feature maps from them.The reconstructor plays the role as an attacker, which tries to recover the image from the feature maps extracted by the obfuscator.In order to best protect usersâ€™ privacy in images, we design an adversarial training methodol- ogy for our framework to optimize the obfuscator.Through extensive evaluations on real world datasets, both the numerical metrics and the visualization results demonstrate that our framework is qualified to protect usersâ€™ privacy and achieve a relatively high accuracy on the image classification task.

我们提出了一个新颖的深度学习图像分类框架，既能对图像进行准确分类，又能保护用户的隐私。

本文提出了一个框架，它保留了图像中的私人信息，并且不影响图像的可用性。

目前的这项工作建议使用对抗性网络来混淆图像，从而允许在没有隐私顾虑的情况下收集图像，将其用于训练机器学习模型。


Bitcoin is a virtual coinage system that enables users to trade virtually free of a central trusted authority.All transactions on the Bitcoin blockchain are publicly available for viewing, yet as Bitcoin is built mainly for security itâ€™s original structure does not allow for direct analysis of address transactions. Existing analysis methods of the Bitcoin blockchain can be complicated, computationally expensive or inaccurate.We propose a computationally efficient model to analyze bitcoin blockchain addresses and allow for their use with existing machine learning algorithms.We compare our approach against Multi Level Sequence Learners (MLSLs), one of the best performing models on bitcoin address data.

加密货币交易图的2vec模型

本文提出使用自动编码器、networkX和node2Vec来预测一个比特币地址是否会在一年后变成空地址，但结果比现有的基线要差。


Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood.In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations.First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable.Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients.Finally, we present anchored simGD, a new method, and show convergence with stochastic subgradients.

随机子梯度法的收敛性证明以及凸凹最小最大问题的变化

在最小凸凹博弈的背景下，对同步随机亚梯度、同步乐观梯度和同步锚定梯度进行了分析。

本文分析了随机梯度下降在应用于凸凹博弈时的动态变化，以及乐观主义的GD和一种新的锚定GD算法，该算法在比SGD或乐观主义的SGD更弱的假设下收敛。


Small spacecraft now have precise attitude control systems available commercially, allowing them to slew in 3 degrees of freedom, and capture images within short notice.When combined with appropriate software, this agility can significantly increase response rate, revisit time and coverage.In prior work, we have demonstrated an algorithmic framework that combines orbital mechanics, attitude control and scheduling optimization to plan the time-varying, full-body orientation of agile, small spacecraft in a constellation.The proposed schedule optimization would run at the ground station autonomously, and the resultant schedules uplinked to the spacecraft for execution.The algorithm is generalizable over small steerable spacecraft, control capability, sensor specs, imaging requirements, and regions of interest.In this article, we modify the algorithm to run onboard small spacecraft, such that the constellation can make time-sensitive decisions to slew and capture images autonomously, without ground control.We have developed a communication module based on Delay/Disruption Tolerant Networking (DTN) for onboard data management and routing among the satellites, which will work in conjunction with the other modules to optimize the schedule of agile communication and steering.We then apply this preliminary framework on representative constellations to simulate targeted measurements of episodic precipitation events and subsequent urban floods.The command and control efficiency of our agile algorithm is compared to non-agile (11.3x improvement) and non-DTN (21% improvement) constellations.

我们提出了一个算法框架，以安排具有3-DOF重新定向能力的小型航天器星座，并与卫星间的链接联网。

本文提出了一个优化航天器星座问题的通信模块，并比较了分布式和集中式设置中的算法。


Importance sampling (IS) is a standard Monte Carlo (MC) tool to compute information about random variables such as moments or quantiles with unknown distributions.  IS is asymptotically consistent as the number of MC samples, and hence deltas (particles) that parameterize the density estimate, go to infinity.However, retaining infinitely many particles is intractable.We propose a scheme for only keeping a \emph{finite representative subset} of particles and their augmented importance weights that is \emph{nearly consistent}. To do so in {an online manner}, we approximate importance sampling in two ways.  First, we replace the deltas by kernels, yielding kernel density estimates (KDEs).  Second, we sequentially project KDEs onto nearby lower-dimensional subspaces.We characterize the asymptotic bias of this scheme as determined by a compression parameter and kernel bandwidth, which yields a tunable tradeoff between consistency and memory.In experiments,  we observe a favorable tradeoff between memory and accuracy, providing for the first time near-consistent compressions of arbitrary posterior distributions.

我们提出了一种新型的压缩核化重要性采样算法。


We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator?(2) how to correctly use cross-validation to choose the regularization parameter?and (3) how to accelerate computation without losing too much accuracy?We consider the three problems in a unified large-data linear model.We give a precise representation of ridge regression as a covariance matrix-dependent linear combination of the true parameter and the noise. We study the bias of $K$-fold cross-validation for choosing the regularization parameter, and propose a simple bias-correction.We analyze the accuracy of primal and dual sketching for ridge regression, showing they are surprisingly accurate.Our results are illustrated by simulations and by analyzing empirical data.

我们在高维渐进框架中研究了脊回归的结构，并得到了关于交叉验证和草图的见解。 利用脊回归估计器的新渐进特征对脊回归进行理论研究。


Attention mechanisms have advanced the state of the art in several machine learning tasks.Despite significant empirical gains, there is a lack of theoretical analyses on understanding their effectiveness.In this paper, we address this problem by studying the landscape of population and empirical loss functions of attention-based neural networks.Our results show that, under mild assumptions, every local minimum of a two-layer global attention model has low prediction error, and attention models require lower sample complexity than models not employing attention.We then extend our analyses to the popular self-attention model, proving that they deliver consistent predictions with a more expressive class of functions.Additionally, our theoretical results provide several guidelines for designing attention mechanisms.Our findings are validated with satisfactory experimental results on MNIST and IMDB reviews dataset.

我们分析了有注意力的神经网络的损失情况，并解释了为什么注意力有助于训练神经网络以获得良好的性能。

本文从理论角度证明了注意力网络在单层设置中，对于固定注意力（单层和多层）和自我注意力的泛化能力比非注意力基线更好。


Recent advances in deep learning techniques has shown the usefulness of the deep neural networks in extracting features required to perform the task at hand.However, these features learnt are in particular helpful only for the initial task.This is due to the fact that the features learnt are very task specific and does not capture the most general and task agnostic features of the input.In fact the way humans are seen to learn is by disentangling features which task agnostic.This indicates that leaning task agnostic features by disentangling only the most informative features from the input data.Recently Variational Auto-Encoders (VAEs) have shown to be the de-facto models to capture the latent variables in a generative sense.As these latent features can be represented as continuous and/or discrete variables, this indicates us to use VAE with a mixture of continuous and discrete variables for the latent space.We achieve this by performing our experiments using a modified version of joint-vae to learn the disentangled features.


To improve how neural networks function it is crucial to understand their learning process.The information bottleneck theory of deep learning proposes that neural networks achieve good generalization by compressing their representations to disregard information that is not relevant to the task.However, empirical evidence for this theory is conflicting, as compression was only observed when networks used saturating activation functions.In contrast, networks with non-saturating activation functions achieved comparable levels of task performance but did not show compression.In this paper we developed more robust mutual information estimation techniques, that adapt to hidden activity of neural networks and produce more sensitive measurements of activations from all functions, especially unbounded functions.Using these adaptive estimation techniques, we explored compression in networks with a range of different activation functions.With two improved methods of estimation, firstly, we show that saturation of the activation function is not required for compression, and the amount of compression varies between different activation functions.We also find that there is a large amount of variation in compression between different network initializations.Secondary, we see that L2 regularization leads to significantly increased compression, while preventing overfitting.Finally, we show that only compression of the last layer is positively correlated with generalization.

我们为DNN开发了稳健的互信息估计，并利用它们观察具有非饱和激活函数的网络中的压缩情况

本文研究了深度神经网络为监督任务做信息压缩的普遍观点

本文提出了一种估计具有无界激活函数的网络的相互信息的方法，并使用L2正则化来引起更多的压缩。


In this work, we address the problem of musical timbre transfer, where the goal is to manipulate the timbre of a sound sample from one instrument to match another instrument while preserving other musical content, such as pitch, rhythm, and loudness.In principle, one could apply image-based style transfer techniques to a time-frequency representation of an audio signal, but this depends on having a representation that allows independent manipulation of timbre as well as high-quality waveform generation.We introduce TimbreTron, a method for musical timbre transfer which applies â€œimageâ€ domain style transfer to a time-frequency representation of the audio signal, and then produces a high-quality waveform using a conditional WaveNet synthesizer.We show that the Constant Q Transform (CQT) representation is particularly well-suited to convolutional architectures due to its approximate pitch equivariance.Based on human perceptual evaluations, we confirmed that TimbreTron recognizably transferred the timbre while otherwise preserving the musical content, for both monophonic and polyphonic samples.We made an accompanying demo video here: https://www.cs.toronto.edu/~huang/TimbreTron/index.html which we strongly encourage you to watch before reading the paper.

我们提出了TimbreTron，这是一个使用CQT域式传输在音乐波形上完成高质量音色传输的管道。

通过应用为图像风格转移而开发的CycleGAN，将特定乐器的录音转换为另一种乐器的方法，用于转移谱图。

作者使用多种技术/工具，在没有配对训练实例的情况下，实现了神经性音色转换（将音乐从一种乐器转换到另一种乐器）。

描述了一个音乐音色传递的模型，结果表明，所提出的系统对音高和节奏的传递以及音色的适应是有效的。


Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them.But also generic hardware and software implementations of deep learning run more efficiently for sparse networks.Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints.We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network.DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded.We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance.DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.

本文介绍了Deep Rewiring，一种可用于训练深度神经网络的算法，当网络连接性在训练过程中受到严重制约时，该算法可用于训练深度神经网络。

一种直接在稀疏连接图上实现深度学习的方法，允许网络有效地在线训练，并实现快速和灵活的学习。

作者提供了一个简单的算法，能够用有限的内存进行训练


Deep learning's success has led to larger and larger models to handle more and more complex tasks; trained models can contain millions of parameters.These large models are compute- and memory-intensive, which makes it a challenge to deploy them with minimized latency, throughput, and storage requirements.Some model compression methods have been successfully applied on image classification and detection or language models, but there has been very little work compressing generative adversarial networks (GANs) performing complex tasks.In this paper, we show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods.We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator.We show that this framework has a compelling performance to high degrees of sparsity, generalizes well to new tasks and models, and enables meaningful comparisons between different pruning granularities.

现有的修剪方法在应用于处理复杂任务的GAN时失败了，因此我们提出了一种简单而稳健的修剪生成器的方法，对各种网络和任务都很有效。

作者针对压缩网络的任务提出了对经典蒸馏法的修改，以解决以前的解决方案在应用于生成式对抗网络时的失败。


Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure.The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections.In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth.To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training.We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus.On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270x to 600x without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB.Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile.

我们发现分布式SGD中99.9%的梯度交换是多余的；我们将通信带宽减少了两个数量级，而不会失去准确性。

本文提出了比梯度下降更多的改进，以提高通信效率


Image-to-image translation has recently received significant attention due to advances in deep learning.Most works focus on learning either a one-to-one mapping in an unsupervised way or a many-to-many mapping in a supervised way.However, a more practical setting is many-to-many mapping in an unsupervised way, which is harder due to the lack of supervision and the complex inner- and cross-domain variations.To alleviate these issues, we propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain.We assume that an image comprises of a content component which is shared across domains, and a style component specific to each domain.Under the guidance of an exemplar from the target domain we apply Adaptive Instance Normalization to the shared content component, which allows us to transfer the style information of the target domain to the source domain.To avoid semantic inconsistencies during translation that naturally appear due to the large inner- and cross-domain variations, we introduce the concept of feature masks that provide coarse semantic guidance without requiring the use of any semantic labels.Experimental results on various datasets show that EGSC-IT does not only translate the source image to diverse instances in the target domain, but also preserves the semantic consistency during the process.

我们提出了典范引导和语义一致的图像-图像转换（EGSC-IT）网络，它将转换过程置于目标领域的典范图像上。

讨论了I2I转换模型的一个核心故障和需求。

本文探讨了一个图像有两个组成部分的想法，并应用了一个注意力模型，其中引导转换过程的特征掩码不需要语义标签。


Deep neural networks can learn meaningful representations of data.However, these representations are hard to interpret.For example, visualizing a latent layer is generally only possible for at most three dimensions.Neural networks are able to learn and benefit from much higher dimensional representations but these are not visually interpretable because nodes have arbitrary ordering within a layer.Here, we utilize the ability of the human observer to identify patterns in structured representations to visualize higher dimensions.To do so, we propose a class of regularizations we call \textit{Graph Spectral Regularizations} that impose graph-structure on latent layers.This is achieved by treating activations as signals on a predefined graph and constraining those activations using graph filters, such as low pass and wavelet-like filters.This framework allows for any kind of graph as well as filter to achieve a wide range of structured regularizations depending on the inference needs of the data.First, we show a synthetic example that the graph-structured layer can reveal topological features of the data.Next, we show that a smoothing regularization can impose semantically consistent ordering of nodes when applied to capsule nets.Further, we show that the graph-structured layer, using wavelet-like spatially localized filters, can form localized receptive fields for improved image and biomedical data interpretation.In other words, the mapping between latent layer, neurons and the output space becomes clear due to the localization of the activations.Finally, we show that when structured as a grid, the representations create coherent images that allow for image-processing techniques such as convolutions.

在神经网络层上强加图形结构以提高视觉可解释性。 一种新型的正则器，将图结构强加于神经网络的隐藏层，以提高隐藏表征的可解释性。 突出了图谱正则器对神经网络的可解释性的贡献。


Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation.The dominant parametric approach is based on locally normalized models which predict one word at a time.While these work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process.In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level.In order to make training tractable, we first work in the residual of a pretrained locally normalized language model and second we train using noise contrastive estimation.Furthermore, since the EBM works at the sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa.Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity compared to locally normalized baselines.Moreover, generation via importance sampling is very efficient and of higher quality than the baseline models according to human evaluation.

我们表明，在自动回归语言模型的残差上进行训练时，基于能量的模型可以有效地用于生成文本。

提出了一种用于文本生成的基于残余能量的模型（EBM），该模型在句子层面上运作，因此可以利用BERT，并实现较低的困惑度，而且被人类评价所青睐。


We investigate the robustness properties of image recognition models equipped with two features inspired by human vision, an explicit episodic memory and a shape bias, at the ImageNet scale.As reported in previous work, we show that an explicit episodic memory improves the robustness of image recognition models against small-norm adversarial perturbations under some threat models.It does not, however, improve the robustness against more natural, and typically larger, perturbations.Learning more robust features during training appears to be necessary for robustness in this second sense.We show that features derived from a model that was encouraged to learn global, shape-based representations (Geirhos et al., 2019) do not only improve the robustness against natural perturbations, but when used in conjunction with an episodic memory, they also provide additional robustness against adversarial perturbations.Finally, we address three important design choices for the episodic memory: memory size, dimensionality of the memories and the retrieval method.We show that to make the episodic memory more compact, it is preferable to reduce the number of memories by clustering them, instead of reducing their dimensionality.

对基于大规模缓存的图像识别模型进行系统研究，特别关注其稳健性特性

本文提出使用内存缓存来提高对对抗性图像实例的稳健性，并得出结论，使用大型连续缓存并不优于硬注意力。


Group convolutional neural networks (G-CNNs) can be used to improve classical CNNs by equipping them with the geometric structure of groups.Central in the success of G-CNNs is the lifting of feature maps to higher dimensional disentangled representations, in which data characteristics are effectively learned, geometric data-augmentations are made obsolete, and predictable behavior under geometric transformations (equivariance) is guaranteed via group theory.Currently, however, the practical implementations of G-CNNs are limited to either discrete groups (that leave the grid intact) or continuous compact groups such as rotations (that enable the use of Fourier theory).In this paper we lift these limitations and propose a modular framework for the design and implementation of G-CNNs for arbitrary Lie groups.In our approach the differential structure of Lie groups is used to expand convolution kernels in a generic basis of B-splines that is defined on the Lie algebra.This leads to a flexible framework that enables localized, atrous, and deformable convolutions in G-CNNs by means of respectively localized, sparse and non-uniform B-spline expansions.The impact and potential of our approach is studied on two benchmark datasets: cancer detection in histopathology slides (PCam dataset) in which rotation equivariance plays a key role and facial landmark localization (CelebA dataset) in which scale equivariance is important.In both cases, G-CNN architectures outperform their classical 2D counterparts and the added value of atrous and localized group convolutions is studied in detail.

本文描述了一个灵活的框架，用于构建对一大类变换组具有等效性的CNN。

一个用任意Lie群G构建群组CNN的框架，它在肿瘤分类和地标定位方面显示出比CNN更优越的性能。


Global feature pooling is a modern variant of feature pooling providing better interpretatability and regularization.Although alternative pooling methods exist (eg. max, lp norm, stochastic), the averaging operation is still the dominating global pooling scheme in popular models.As fine-grained recognition requires learning subtle, discriminative features, we consider the question: is average pooling the optimal strategy?We first ask: ``is there a difference between features learned by global average and max pooling?'' Visualization and quantitative analysis show that max pooling encourages learning features of different spatial scales.We then ask ``is there a single global feature pooling variant that's most suitable for fine-grained recognition?'' A thorough evaluation of nine representative pooling algorithms finds that: max pooling outperforms average pooling consistently across models, datasets, and image resolutions; it does so by reducing the generalization gap; and generalized pooling's performance increases almost monotonically as it changes from average to max.We finally ask: ``what's the best way to combine two heterogeneous pooling schemes?'' Common strategies struggle because of potential gradient conflict but the ``freeze-and-train'' trick works best.We also find that post-global batch normalization helps with faster convergence and improves model performance consistently.

对九个有代表性的全局池计划的基准测试显示了一些有趣的发现。

对于细粒度的分类任务，本文验证了maxpooling会鼓励比avgpooling更稀疏的特征图，并且优于avgpooling。


We present a technique to improve the generalization of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions.Although recent research has shown benefits of self-supervised learning (SSL) on large unlabeled datasets, its utility on small datasets is unknown.We find that SSL reduces the relative error rate of few-shot meta-learners by 4%-27%, even when the datasets are small and only utilizing images within the datasets.The improvements are greater when the training set is smaller or the task is more challenging.Though the benefits of SSL may increase with larger training sets, we observe that SSL can have a negative impact on performance when there is a domain shift between distribution of images used for meta-learning and SSL.Based on this analysis we present a technique that automatically select images for SSL from a large, generic pool of unlabeled images for a given dataset using a domain classifier that provides further improvements.We present results using several meta-learners and self-supervised tasks across datasets with varying degrees of domain shifts and label sizes to characterize the effectiveness of SSL for few-shot learning.

自我监督改善了小型和具有挑战性的数据集上的几张照片的识别，而不依赖额外的数据；额外的数据只有在来自相同或类似领域时才有帮助。

对不同的自我监督学习（SSL）方法的实证研究，显示SSL在数据集较难时帮助更大，领域对训练很重要，还有一种从无标签的数据集中选择样本的方法。


Abstraction of Markov Decision Processes is a useful tool for solving complex problems, as it can ignore unimportant aspects of an environment, simplifying the process of learning an optimal policy.In this paper, we propose a new algorithm for finding abstract MDPs in environments with continuous state spaces.It is based on MDP homomorphisms, a structure-preserving mapping between MDPs.We demonstrate our algorithm's ability to learns abstractions from collected experience and show how to reuse the abstractions to guide exploration in new tasks the agent encounters.Our novel task transfer method beats a baseline based on a deep Q-network.

我们从经验中创造出抽象的环境模型，并利用它们更快地学习新的任务。 一种利用MDP同构的思想将具有连续状态空间的复杂MDP转化为较简单的方法。


A number of recent methods to understand neural networks have focused on quantifying the role of individual features.  One such method, NetDissect identifies interpretable features of a model using the Broden dataset of visual semantic labels (colors, materials, textures, objects and scenes).  Given the recent rise of a number of action recognition datasets, we propose extending the Broden dataset to include actions to better analyze learned action models.  We describe the annotation process, results from interpreting action recognition models on the extended Broden dataset and examine interpretable feature paths to help us understand the conceptual hierarchy used to classify an action.

我们将网络剖析扩展到包括行动解释，并检查可解释的特征路径，以了解用于分类行动的概念层次。


Automatic melody generation for pop music has been a long-time aspiration forboth AI researchers and musicians.However, learning to generate euphoniousmelody has turned out to be highly challenging due to a number of factors.Representationof multivariate property of notes has been one of the primary challenges.It is also difficult to remain in the permissible spectrum of musical variety, outsideof which would be perceived as a plain random play without auditory pleasantness.Observing the conventional structure of pop music poses further challenges.In this paper, we propose to represent each note and its properties as a uniqueâ€˜word,â€™ thus lessening the prospect of misalignments between the properties, aswell as reducing the complexity of learning.We also enforce regularization policieson the range of notes, thus encouraging the generated melody to stay closeto what humans would find easy to follow.Furthermore, we generate melodyconditioned on song part information, thus replicating the overall structure of afull song.Experimental results demonstrate that our model can generate auditorilypleasant songs that are more indistinguishable from human-written ones thanprevious models.

我们提出了一个新的模型来表示音符及其属性，这可以提高自动生成旋律的能力。

本文提出了西方流行音乐中符号（MIDI）旋律的生成模型，它将音符符号与时间和持续时间信息联合编码，形成音乐 "词"。

本文提出通过将音符表示为 "词 "来促进旋律的生成，表示音符的所有属性，从而允许生成音乐 "句子"。


Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts.We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth.We propose robust growing and stopping policies to generalize to different network architectures and datasets.Our experiments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet.For example, in terms of accuracy-computation trade-off, AutoGrow discovers a better depth combination in ResNets than human experts.Our AutoGrow is efficient.It discovers depth within similar time of training a single DNN.

一种在神经网络中自动增长层数以发现最佳深度的方法。
一个交错训练较浅的网络和添加新层的框架，它提供了对 "成长网络 "范式的见解。


Given the importance of remote sensing, surprisingly little attention has been paid to it by the representation learning community.To address it and to speed up innovation in this domain, we provide simplified access to 5 diverse remote sensing datasets in a standardized form.We specifically explore in-domain representation learning and address the question of "what characteristics should a dataset have to be a good source for remote sensing representation learning".The established baselines achieve state-of-the-art performance on these datasets.

遥感数据集的域内表征学习的探索。

这篇论文提供了几个标准化的遥感数据集，并表明与在ImageNet上进行微调或从头开始学习相比，域内表示可以为遥感产生更好的基线结果。


Generative seq2seq dialogue systems are trained to predict the next word in dialogues that have already occurred.They can learn from large unlabeled conversation datasets, build a deep understanding of conversational context, and generate a wide variety of responses.This flexibility comes at the cost of control.Undesirable responses in the training data will be reproduced by the model at inference time, and longer generations often donâ€™t make sense.Instead of generating responses one word at a time, we train a classifier to choose from a predefined list of full responses.The classifier is trained on (conversation context, response class) pairs, where each response class is a noisily labeled group of interchangeable responses.At inference, we generate the exemplar response associated with the predicted response class.Experts can edit and improve these exemplar responses over time without retraining the classifier or invalidating old training data.Human evaluation of 775 unseen doctor/patient conversations shows that this tradeoff improves responses.Only 12% of our discriminative approachâ€™s responses are worse than the doctorâ€™s response in the same conversational context, compared to 18% for the generative model.A discriminative model trained without any manual labeling of response classes achieves equal performance to the generative model.

通过使用弱监督来训练分类器以挑选完整的响应，避免一次生成一个词的响应。

一种使用分类器为医疗对话生成响应的方法，根据对话背景从专家策划的响应中进行选择。


There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs).This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP.In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels.We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible. Surprisingly, in the absence of pooling layers, the GPs corresponding to CNNs with and without weight sharing are identical.As a consequence, translation equivariance, beneficial in finite channel CNNs trained with stochastic gradient descent (SGD), is guaranteed to play no role in the Bayesian treatment of the infinite channel limit - a qualitative difference between the two regimes that is not present in the FCN case.We confirm experimentally, that while in some scenarios the performance of SGD-trained finite CNNs approaches that of the corresponding GPs as the channel count increases, with careful tuning SGD-trained CNNs can significantly outperform their corresponding GPs, suggesting advantages from SGD training compared to fully Bayesian parameter estimation.

有限宽度的SGD训练的CNN与无限宽度的完全贝叶斯CNN。谁赢了？ 本文建立了无限通道贝叶斯卷积神经网络和高斯过程之间的联系。


Bayesian inference promises to ground and improve the performance of deep neural networks.It promises to be robust to overfitting, to simplify the training procedure and the space of hyperparameters, and to provide a calibrated measure of uncertainty that can enhance decision making, agent exploration and prediction fairness.Markov Chain Monte Carlo (MCMC) methods enable Bayesian inference by generating samples from the posterior distribution over model parameters.Despite the theoretical advantages of Bayesian inference and the similarity between MCMC and optimization methods, the performance of sampling methods has so far lagged behind  optimization methods for large scale deep learning tasks.We aim to fill this gap and introduce ATMC, an adaptive noise MCMC algorithm that estimates and is able to sample from the posterior of a neural network.ATMC dynamically adjusts the amount of momentum and noise applied to each parameter update in order to compensate for the use of stochastic gradients.We use a ResNet architecture without batch normalization to test ATMC on the Cifar10 benchmark and the large scale ImageNet benchmark and show that, despite the  absence of batch normalization, ATMC outperforms a strong optimization baseline in terms of both classification accuracy and test log-likelihood.We show that ATMC is intrinsically robust to overfitting on the training data and that ATMC provides a better calibrated measure of uncertainty compared to the optimization baseline.

我们将贝叶斯推理扩展到ImageNet分类，并取得了具有竞争力的结果准确性和不确定性校准。

一种用于图像分类的自适应噪声MCMC算法，动态调整应用于每个参数更新的动量和噪声，对过拟合具有稳健性，并提供带有预测的不确定性测量。


Now GANs can generate more and more realistic face images that can easily fool human beings.  In contrast, a common convolutional neural network(CNN), e.g. ResNet-18, can achieve more than 99.9% accuracy in discerning fake/real faces if training and testing faces are from the same source.In this paper, we performed both human studies and CNN experiments, which led us to two important findings.One finding is that the textures of fake faces are substantially different from real ones.CNNs can capture local image texture information for recognizing fake/real face, while such cues are easily overlooked by humans.The other finding is that global image texture information is more robust to image editing and generalizable to fake faces from different GANs and datasets.Based on the above findings, we propose  a  novel  architecture  coined  as  Gram-Net,  which  incorporates  â€œGram Blockâ€ in multiple semantic levels to extract global image texture representations.Experimental results demonstrate that our Gram-Net performs better than existing approaches for fake face detection.  Especially, our Gram-Net is more robust to image editing, e.g.  downsampling, JPEG compression, blur, and noise.  More importantly, our Gram-Net generalizes significantly better in detecting fake faces from GAN models not seen in the training phase.

一项关于假图像的实证研究显示，纹理是当前假图像与真实图像不同的重要线索。我们改进的捕获全局纹理统计的模型显示出更好的跨GAN假图像检测性能。

本文提出了一种改进模型性能的方法，用于由GAN生成的图像中的假人脸检测，使其在纹理信息的基础上更具有通用性。


The Wasserstein probability metric has received much attention from the machine learning community.Unlike the Kullback-Leibler divergence, which strictly measures change in probability, the Wasserstein metric reflects the underlying geometry between outcomes.The value of being sensitive to this geometry has been demonstrated, among others, in ordinal regression and generative modelling, and most recently in reinforcement learning.In this paper we describe three natural properties of probability divergences that we believe reflect requirements from machine learning: sum invariance, scale sensitivity, and unbiased sample gradients.The Wasserstein metric possesses the first two properties but, unlike the Kullback-Leibler divergence, does not possess the third.We provide empirical evidence suggesting this is a serious issue in practice.Leveraging insights from probabilistic forecasting we propose an alternative to the Wasserstein metric, the CramÃ©r distance.We show that the CramÃ©r distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences.We give empirical results on a number of domains comparing these three divergences.To illustrate the practical relevance of the CramÃ©r distance we design a new algorithm, the CramÃ©r Generative Adversarial Network (GAN), and show that it has a number of desirable properties over the related Wasserstein GAN.

Wasserstein距离很难用随机梯度下降法进行最小化，而Cramer距离可以很容易地被优化，而且效果也很好。

该稿件提出，在使用随机梯度下降法优化目标函数时，使用Cramer距离作为损失，因为它具有无偏的样本梯度。

文章的贡献与性能标准有关，特别是与瓦尔斯坦/马洛斯的指标有关。


We humans have an innate understanding of the asymmetric progression of time, which we use to efficiently and safely perceive and manipulate our environment.Drawing inspiration from that, we approach the problem of learning an arrow of time in a Markov (Decision) Process.We illustrate how a learned arrow of time can capture salient information about the environment, which in turn can be used to measure reachability, detect side-effects and to obtain an intrinsic reward signal.Finally, we propose a simple yet effective algorithm to parameterize the problem at hand and learn an arrow of time with a function approximator (here, a deep neural network).Our empirical results span a selection of discrete and continuous environments, and demonstrate for a class of stochastic processes that the learned arrow of time agrees reasonably well with a well known notion of an arrow of time due to Jordan, Kinderlehrer and Otto (1998).

我们学习MDP的时间之箭，并利用它来衡量可达性，检测副作用，并获得好奇心的奖励信号。

这项工作提出了h-potential作为衡量MDP中状态转换不对称性目标的解决方案。


We formulate stochastic gradient descent (SGD) as a novel factorised Bayesian filtering problem, in which each parameter is inferred separately, conditioned on the corresopnding backpropagated gradient.  Inference in this setting naturally gives rise to BRMSprop and BAdam: Bayesian variants of RMSprop and Adam.  Remarkably, the Bayesian approach recovers many features of state-of-the-art adaptive SGD methods, including amongst others root-mean-square normalization, Nesterov acceleration and AdamW.  As such, the Bayesian approach provides one explanation for the empirical effectiveness of state-of-the-art adaptive SGD algorithms.  Empirically comparing BRMSprop and BAdam with naive RMSprop and Adam on MNIST, we find that Bayesian methods have the potential to considerably reduce test loss and classification error.

我们将SGD表述为一个贝叶斯滤波问题，并表明这产生了RMSprop、Adam、AdamW、NAG和其他最先进的适应性方法的特征。

本文通过贝叶斯滤波分析随机梯度下降，作为分析自适应方法的框架。

作者试图在贝叶斯滤波框架下将现有的适应性梯度方法与动态先验统一起来。


Data augmentation (DA) has been widely utilized to improve generalization in training deep neural networks.Recently, human-designed data augmentation has been gradually replaced by automatically learned augmentation policy.Through finding the best policy in well-designed search space of data augmentation, AutoAugment (Cubuk et al., 2019) can significantly improve validation accuracy on image classification tasks.However, this approach is not computationally practical for large-scale problems.In this paper, we develop an adversarial method to arrive at a computationally-affordable solution called Adversarial AutoAugment, which can simultaneously optimize target related object and augmentation policy search loss.The augmentation policy network attempts to increase the training loss of a target network through generating adversarial augmentation policies, while the target network can learn more robust features from harder examples to improve the generalization.In contrast to prior work, we reuse the computation in target network training for policy evaluation, and dispense with the retraining of the target network.Compared to AutoAugment, this leads to about 12x reduction in computing cost and 11x shortening in time overhead on ImageNet.We show experimental results of our approach on CIFAR-10/CIFAR-100, ImageNet, and demonstrate significant performance improvements over state-of-the-art.On CIFAR-10, we achieve a top-1 test error of 1.36%, which is the currently best performing single model.On ImageNet, we achieve a leading performance of top-1 accuracy 79.40% on ResNet-50 and 80.00% on ResNet-50-D without extra data.

我们将对抗性学习的理念引入到自动数据增强中，以提高目标网络的泛化能力。

一种被称为对抗性自动增强的技术，在训练过程中使用对抗性方法动态地学习良好的数据增强策略。


In this study we focus on first-order meta-learning algorithms that aim to learn a parameter initialization of a network which can quickly adapt to new concepts, given a few examples.We investigate two approaches to enhance generalization and speed of learning of such algorithms, particularly expanding on the Reptile (Nichol et al., 2018) algorithm.We introduce a novel regularization technique called meta-step gradient pruning and also investigate the effects of increasing the depth of network architectures in first-order meta-learning.We present an empirical evaluation of both approaches, where we match benchmark few-shot image classification results with 10 times fewer iterations using Mini-ImageNet dataset and with the use of deeper networks, we attain accuracies that surpass the current benchmarks of few-shot image classification using Omniglot dataset.

该研究介绍了两种加强一阶元学习泛化的方法，并提出了对少样本照片图像分类的实证评估。

本文介绍了对一阶元学习Reptile算法的实证研究，研究了所提出的正则化技术和更深入的网络


In this paper, we propose the use of in-training matrix factorization to reduce the model size for neural machine translation.Using in-training matrix factorization, parameter matrices may be decomposed into the products of smaller matrices, which can compress large machine translation architectures by vastly reducing the number of learnable parameters.We apply in-training matrix factorization to different layers of standard neural architectures and show that in-training factorization is capable of reducing nearly 50% of learnable parameters without any associated loss in BLEU score.Further, we find that in-training matrix factorization is especially powerful on embedding layers, providing a simple and effective method to curtail the number of parameters with minimal impact on model performance, and, at times, an increase in performance.

本文提出在训练时使用矩阵分解法进行神经机器翻译，这可以在不损害性能的情况下减少模型大小，减少训练时间。

本文提出在机器翻译的深度神经网络训练过程中使用矩阵分解法压缩模型。


Though state-of-the-art sentence representation models can perform tasks requiring significant knowledge of grammar, it is an open question how best to evaluate their grammatical knowledge.We explore five experimental methods inspired by prior work evaluating pretrained sentence representation models.We use a single linguistic phenomenon, negative polarity item (NPI) licensing, as a case study for our experiments.NPIs like 'any' are grammatical only if they appear in a licensing environment like negation ('Sue doesn't have any cats' vs. '*Sue has any cats').This phenomenon is challenging because of the variety of NPI licensing environments that exist.We introduce an artificially generated dataset that manipulates key features of NPI licensing for the experiments.We find that BERT has significant knowledge of these features, but its success varies widely across different experimental methods.We conclude that a variety of methods is necessary to reveal all relevant aspects of a model's grammatical knowledge in a given domain.

在一个关于NPI的案例研究中，分析BERT的不同方法提出了不同（但兼容）的结论。


The primate visual system builds robust, multi-purpose representations of the external world in order to support several diverse downstream cortical processes.Such representations are required to be invariant to the sensory inconsistencies caused by dynamically varying lighting, local texture distortion, etc.A key architectural feature combating such environmental irregularities is â€˜long-range horizontal connectionsâ€™ that aid the perception of the global form of objects.In this work, we explore the introduction of such horizontal connections into standard deep convolutional networks; we present V1Net -- a novel convolutional-recurrent unit that models linear and nonlinear horizontal inhibitory and excitatory connections inspired by primate visual cortical connectivity.We introduce the Texturized Challenge -- a new benchmark to evaluate object recognition performance under perceptual noise -- which we use to evaluate V1Net against an array of carefully selected control models with/without recurrent processing.Additionally, we present results from an ablation study of V1Net demonstrating the utility of diverse neurally inspired horizontal connections for state-of-the-art AI systems on the task of object boundary detection from natural images.We also present the emergence of several biologically plausible horizontal connectivity patterns, namely center-on surround-off, association fields and border-ownership connectivity patterns in a V1Net model trained to perform boundary detection on natural images from the Berkeley Segmentation Dataset 500 (BSDS500).Our findings suggest an increased representational similarity between V1Net and biological visual systems, and highlight the importance of neurally inspired recurrent contextual processing principles for learning visual representations that are robust to perceptual noise and furthering the state-of-the-art in computer vision.

在这项工作中，我们提出了V1Net--一个新型的循环神经网络，它模拟了皮质的水平连接，通过知觉分组产生了稳健的视觉表征。

作者提议修改LSTM的卷积变体，以包括受视觉皮层中已知的相互作用启发的水平连接。


Humans understand novel sentences by composing meanings and roles of core language components.In contrast, neural network models for natural language modeling fail when such compositional generalization is required.The main contribution of this paper is to hypothesize that language compositionality is a form of group-equivariance.Based on this hypothesis, we propose a set of tools for constructing equivariant sequence-to-sequence models.Throughout a variety of experiments on the SCAN tasks, we analyze the behavior of existing models under the lens of equivariance, and demonstrate that our equivariant architecture is able to achieve the type compositional generalization required in human language understanding.

我们提出了排列等价与构成性泛化之间的联系，并提供了等值的语言模型 这项工作的重点是为SCAN任务的目的，在输入/输出词上学习局部等价表示和函数。


Variational inference (VI) is a popular approach for approximate Bayesian inference that is particularly promising for highly parameterized models such as deep neural networks.  A key challenge of variational inference is to approximate the posterior over model parameters with a distribution that is simpler and tractable yet sufficiently expressive.In this work, we propose a method for training highly flexible variational distributions by starting with a coarse approximation and iteratively refining it.Each refinement step makes cheap, local adjustments and only requires optimization of simple variational families.We demonstrate theoretically that our method always improves a bound on the approximation (the Evidence Lower BOund) and observe this empirically across a variety of benchmark tasks.  In experiments, our method consistently outperforms recent variational inference methods for deep learning in terms of log-likelihood and the ELBO.  We see that the gains are further amplified on larger scale models, significantly outperforming standard VI and deep ensembles on residual networks on CIFAR10.

本文提出一种算法，通过迭代优化提高贝叶斯神经网络中变分后验的灵活性。

一种训练柔性的变分后验分布的方法，应用于贝叶斯神经网，对权重进行变分推理（VI）。


In this paper, we propose a residual non-local attention network for high-quality image restoration.Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial- and channel-wise features.To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts.Specifically, we design trunk branch and (non-)local mask branch in each (non-)local attention block.The trunk branch is used to extract hierarchical features.Local and non-local mask branches aim to adaptively rescale these hierarchical features with mixed attentions.The local mask branch concentrates on more local structures with convolutional operations, while non-local attention considers more about long-range dependencies in the whole feature map.Furthermore, we propose residual local and non-local attention learning to train the very deep network, which further enhance the representation ability of the network.Our proposed method can be generalized for various image restoration applications, such as image denoising, demosaicing, compression artifacts reduction, and super-resolution.Experiments demonstrate that our method obtains comparable or better results compared with recently leading methods quantitatively and visually.

新的最先进的图像修复框架

本文提出了一个卷积神经网络架构，其中包括局部和非局部注意力机制的区块，据称这些区块在四个图像修复应用中取得了优异的结果。

本文提出了一种用于图像修复的残差非局部注意力网络


Most approaches to learning action planning models heavily rely on a significantly large volume of training samples or plan observations.In this paper, we adopt a different approach based on deductive learning from domain-specific knowledge, specifically from logic formulae that specify constraints about the possible states of a given domain.The minimal input observability required by our approach is a single example composed of a full initial state and a partial goal state.We will show that exploiting specific domain knowledge enables to constrain the space of possible action models as well as to complete partial observations, both of which turn out helpful to learn good-quality action models.

用专家提供的特定领域的知识来弥补缺乏可用数据的模型获取的混合方法

一种领域获取方法，考虑通过使用图式化的互斥关系代替前/后条件，对部分领域模型使用不同的表征。


We release the largest public ECG dataset of continuous raw signals for representation learning containing over 11k patients and 2 billion labelled beats.Our goal is to enable semi-supervised ECG models to be made as well as to discover unknown subtypes of arrhythmia and anomalous ECG signal events.To this end, we propose an unsupervised representation learning task, evaluated in a semi-supervised fashion.  We provide a set of baselines for different feature extractors that can be built upon.  Additionally, we perform qualitative evaluations on results from PCA embeddings, where we identify some clustering of known subtypes indicating the potential for representation learning in arrhythmia sub-type discovery.

我们发布了一个数据集，该数据集由1.1万名被规定使用{DEVICENAME}(TM)设备的患者的单导联心电图数据构建。

本文描述了作者打算发表的大规模心电图数据集，并提供了数据集的无监督分析和可视化。


As the basic building block of Convolutional Neural Networks (CNNs), the convolutional layer is designed to extract local patterns and lacks the ability to model global context in its nature.Many efforts have been recently made to complement CNNs with the global modeling ability, especially by a family of works on global feature interaction.In these works, the global context information is incorporated into local features before they are fed into convolutional layers.However, research on neuroscience reveals that, besides influences changing the inputs to our neurons, the neurons' ability of modifying their functions dynamically according to context is essential for perceptual tasks, which has been overlooked in most of CNNs.Motivated by this, we propose one novel Context-Gated Convolution (CGC) to explicitly modify the weights of convolutional layers adaptively under the guidance of global context.As such, being aware of the global context, the modulated convolution kernel of our proposed CGC can better extract representative local patterns and compose discriminative features.Moreover, our proposed CGC is lightweight, amenable to modern CNN architectures, and consistently improves the performance of CNNs according to extensive experiments on image classification, action recognition, and machine translation.

一种新型的背景关联卷积，通过明确调节卷积核，将全局背景信息纳入CNN，从而捕捉到更具代表性的局部模式，并提取出判别性的特征。

本文使用全局背景来调制卷积层的权重，并帮助CNN以高性能和比特征图调制更少的参数捕获更多的鉴别性特征。


We analyze the trade-off between quantization noise and clipping distortion in low precision networks.We identify the statistics of various tensors, and derive exact expressions for the mean-square-error degradation due to clipping.By optimizing these expressions, we show marked improvements over standard quantization schemes that normally avoid clipping.For example, just by choosing the accurate clipping values, more than 40\% accuracy improvement is obtained for the quantization of VGG-16 to 4-bits of precision.Our results have many applications for the quantization of neural networks at both training and inference time.

我们分析了低精度网络中量化噪声和削波失真之间的权衡，并显示了比通常避免削波的标准量化方案的明显改进。

推导出一个公式，用于寻找均匀量化的最小和最大削波值，使量化产生的平方误差最小，对于预量化值的拉普拉斯分布或高斯分布。


Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field.But its performance can awfully degrade with insufficient batch size.This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of memory consumption.Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and increase huge consumption.In this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before.The extra batch statistics associated with gradients also can severely affect the training of deep neural network.Based on our analysis, we propose a novel normalization method, named Moving Average Batch Normalization (MABN).MABN can completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations in inference procedure.We prove the benefits of MABN by both theoretical analysis and experiments.Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO.The code has been released in https://github.com/megvii-model/MABN.

我们提出了一种新颖的归一化方法来处理小批量的情况。

一种处理BN的小批量问题的方法，该方法应用移动平均操作，没有过多的开销，并减少了BN的统计量，以获得更好的稳定性。


We present a simple proof for the benefit of depth in multi-layer feedforward network with rectifed activation (``"depth separation").Specifically we present a sequence of classification problems f_i such that(a) for any fixed depth rectified network we can find an index m such that problems with index > m require exponential network width to fully represent the function f_m; and(b) for any problem f_m in the family, we present a concrete neural network with linear depth and bounded width that fully represents it.While there are several previous work showing similar results, our proof uses substantially simpler tools and techniques, and should be accessible to undergraduate students in computer science and people with similar backgrounds.

ReLU MLP深度分离证明与gemoteric论证
证明对于一个系列的问题，较深的网络比较浅的网络需要更多的单元。


The rich and accessible labeled data fuel the revolutionary success of deep learning.Nonetheless, massive supervision remains a luxury for many real applications, boosting great interest in label-scarce techniques such as few-shot learning (FSL).An intuitively feasible approach to FSL is to conduct data augmentation via synthesizing additional training samples.The key to this approach is how to guarantee both discriminability and diversity of the synthesized samples.In this paper, we propose a novel FSL model, called $\textrm{D}^2$GAN, which synthesizes Diverse and Discriminative features based on Generative Adversarial Networks (GAN).$\textrm{D}^2$GAN secures discriminability of the synthesized features by constraining them to have high correlation with real features of the same classes while low correlation with those of different classes.  Based on the observation that noise vectors that are closer in the latent code space are more likely to be collapsed into the same mode when mapped to feature space, $\textrm{D}^2$GAN incorporates a novel anti-collapse regularization term, which encourages feature diversity by penalizing the ratio of the logarithmic similarity of two synthesized features and the logarithmic similarity of the latent codes generating them.Experiments on three common benchmark datasets verify the effectiveness of $\textrm{D}^2$GAN by comparing with the state-of-the-art.

一个新的基于GAN的少样本学习算法，通过合成不同的和有判别力的特征

一种元学习方法，可以学习一个生成模型，该模型可以增加一个少样本学习者的支持集，该学习者优化了损失的组合。


The lack of crisp mathematical models that capture the structure of real-worlddata sets is a major obstacle to the detailed theoretical understanding of deepneural networks.Here, we first demonstrate the effect of structured data setsby experimentally comparing the dynamics and the performance of two-layernetworks trained on two different data sets:(i) an unstructured synthetic dataset containing random i.i.d. inputs, and(ii) a simple canonical data set suchas MNIST images.Our analysis reveals two phenomena related to the dynamics ofthe networks and their ability to generalise that only appear when training onstructured data sets.Second, we introduce a generative model for data sets,where high-dimensional inputs lie on a lower-dimensional manifold and havelabels that depend only on their position within this manifold.We call it the*hidden manifold model* and we experimentally demonstrate that trainingnetworks on data sets drawn from this model reproduces both the phenomena seenduring training on MNIST.

我们证明了数据集的结构是如何影响神经网络的，并为合成数据集引入了一个再现这种影响的生成模型。

本文研究了数据结构的不同设置如何影响神经网络的学习，以及在合成数据集上学习时如何模仿真实数据集上的行为。


In this paper, we study deep diagonal circulant neural networks, that is deep neural networks in which weight matrices are the product of diagonal and circulant ones.Besides making a theoretical analysis of their expressivity, we introduced principled techniques for training these models: we devise an initialization scheme and proposed a smart use of non-linearity functions in order to train deep diagonal circulant networks. Furthermore, we show that these networks outperform recently introduced deep networks with other types of structured layers.We conduct a thorough experimental study to compare the performance of deep diagonal circulant networks with state of the art models based on structured matrices and with dense models.We show that our models achieve better accuracy than other structured approaches while required 2x fewer weights as the next best approach.Finally we train deep diagonal circulant networks to build a compact and accurate models on a real world video classification dataset with over 3.8 million training examples.

我们训练基于对角线和环形矩阵的深度神经网络，并表明这种类型的网络在现实世界的应用中既紧凑又准确。

作者对对角环形神经网络（DCNN）的表现力进行了理论分析，并提出了一种深度DCNN的初始化方案。


Interpretability has largely focused on local explanations, i.e. explaining why a model made a particular prediction for a sample.These explanations are appealing due to their simplicity and local fidelity.However, they do not provide information about the general behavior of the model.We propose to leverage model distillation to learn global additive explanations that describe the relationship between input features and model predictions.These global explanations take the form of feature shapes, which are more expressive than feature attributions.Through careful experimentation, we show qualitatively and quantitatively that global additive explanations are able to describe model behavior and yield insights about models such as neural nets.A visualization of our approach applied to a neural net as it is trained is available at https://youtu.be/ErQYwNqzEdc

我们建议利用模型蒸馏来学习特征形状形式的全局加性解释（比特征归属更有表现力），用于在表格数据上训练的神经网络等模型。

本文将广义加权模型（GAMs）与模型蒸馏结合起来，以提供神经网络的全局解释。


A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner.These representations are typically used as general purpose features for words across a range of NLP problems.However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem.Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations.In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences.Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods.We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

具有不同训练目标的大规模多任务学习框架来学习固定长度的句子表征

本文是关于通过结合几个训练信号来学习句子嵌入：跳过思考、预测翻译、分类蕴含关系和预测成分解析。


In a time where neural networks are increasingly adopted in sensitive applications, algorithmic bias has emerged as an issue with moral implications.While there are myriad ways that a system may be compromised by bias, systematically isolating and evaluating existing systems on such scenarios is non-trivial, i.e., bias may be subtle, natural and inherently difficult to quantify.To this end, this paper proposes the first systematic study of benchmarking state-of-the-art neural models against biased scenarios.More concretely, we postulate that the bias annotator problem can be approximated with neural models, i.e., we propose generative models of latent bias to deliberately and unfairly associate latent features to a specific class.All in all, our framework provides a new way for principled quantification and evaluation of models against biased datasets.Consequently, we find that state-of-the-art NLP models (e.g., BERT, RoBERTa, XLNET) are readily compromised by biased data.

我们提出了一个神经性偏差注释器，以衡量模型对有偏见的文本数据集的稳健性。

一种为NLP生成有偏差的数据集的方法，依赖于条件对抗性正则化自动编码器（CARA）。


We consider the problem of topic modeling in a weakly semi-supervised setting.In this scenario, we assume that the user knows a priori a subset of the topics she wants the model to learn and is able to provide a few exemplar documents for those topics.In addition, while each document may typically consist of multiple topics, we do not assume that the user will identify all its topics exhaustively.      Recent state-of-the-art topic models such as NVDM, referred to herein as Neural Topic Models (NTMs), fall under the variational autoencoder framework.We extend NTMs to the weakly semi-supervised setting by using informative priors in the training objective.After analyzing the effect of informative priors, we propose a simple modification of the NVDM model using a logit-normal posterior that we show achieves better alignment to user-desired topics versus other NTM models.

我们建议通过在每个文档的基础上智能地调整先验来监督VAE式的主题模型。我们发现logit-normal后验提供了最好的性能。

一种对主题模型进行弱监督的灵活方法，以实现与用户直觉的更好匹配。


Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently as a tool to gain insight into, among others, their generalization ability.However, it is by no means obvious how to estimate mutual information (MI) between each hidden layer and the input/desired output, to construct the IP.For instance, hidden layers with many neurons require MI estimators with robustness towards the high dimensionality associated with such layers.MI estimators should also be able to naturally handle convolutional layers, while at the same time being computationally tractable to scale to large networks.None of the existing IP methods to date have been able to study truly deep Convolutional Neural Networks (CNNs), such as the e.g.\ VGG-16.In this paper, we propose an IP analysis using the new matrix--based R\'enyi's entropy coupled with tensor kernels over convolutional layers, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data.The obtained results shed new light on the previous literature concerning small-scale DNNs, however using a completely new approach.Importantly, the new framework enables us to provide the first comprehensive IP analysis of contemporary large-scale DNNs and CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.

首次使用基于矩阵的熵和张量核对大规模深度神经网络进行全面的信息平面分析。 作者提出了一个基于张量核的估计器，用于神经网络中高维层之间的相互信息估计。


Developing agents that can learn to follow natural language instructions has been an emerging research direction.While being accessible and flexible, natural language instructions can sometimes be ambiguous even to humans.To address this, we propose to utilize programs, structured in a formal language, as a precise and expressive way to specify tasks.We then devise a modular framework that learns to perform a task specified by a program â€“ as different circumstances give rise to diverse ways to accomplish the task, our framework can perceive which circumstance it is currently under, and instruct a multitask policy accordingly to fulfill each subtask of the overall task.Experimental results on a 2D Minecraft environment not only demonstrate that the proposed framework learns to reliably accomplish program instructions and achieves zero-shot generalization to more complex instructions but also verify the efficiency of the proposed modulation mechanism for learning the multitask policy.We also conduct an analysis comparing various models which learn from programs and natural language instructions in an end-to-end fashion.

我们提出了一个模块化的框架，可以完成由程序指定的任务，并实现对更复杂任务的零样本泛化。

本文研究了用指令和任务分解形式化为程序来训练RL代理，提出了一个程序引导代理的模型，该模型解释了一个程序并向行动模块提出了子目标。


We analyze the convergence of (stochastic) gradient descent algorithm for learning a convolutional filter with Rectified Linear Unit (ReLU) activation function.Our analysis does not rely on any specific form of the input distribution and our proofs only use the definition of ReLU, in contrast with previous works that are restricted to standard Gaussian input.We show that (stochastic) gradient descent with random initialization can learn the convolutional filter in polynomial time and the convergence rate depends on the smoothness of the input distribution and the closeness of patches.To the best of our knowledge, this is the first recovery guarantee of gradient-based algorithms for convolutional filter on non-Gaussian input distributions.Our theory also justifies the two-stage learning rate strategy in deep neural networks.While our focus is theoretical, we also present experiments that justify our theoretical findings.

我们证明了随机初始化（随机）梯度下降能在多项式时间内学习卷积滤波器。

研究了使用SGD学习单一卷积滤波器的问题，并表明在某些条件下，SGD能学习单一卷积滤波器。

本文将高斯分布假设扩展到一个更普遍的角度平滑性假设，它涵盖了更广泛的输入分布系列


Deep neural networks (DNNs) are widely adopted in real-world cognitive applications because of their high accuracy.The robustness of DNN models, however, has been recently challenged by adversarial attacks where small disturbance on input samples may result in misclassification.State-of-the-art defending algorithms, such as adversarial training or robust optimization, improve DNNs' resilience to adversarial attacks by paying high computational costs.Moreover, these approaches are usually designed to defend one or a few known attacking techniques only.The effectiveness to defend other types of attacking methods, especially those that have not yet been discovered or explored, cannot be guaranteed.This work aims for a general approach of enhancing the robustness of DNN models under adversarial attacks.In particular, we propose Bamboo -- the first data augmentation method designed for improving the general robustness of DNN without any hypothesis on the attacking algorithms.Bamboo augments the training data set with a small amount of data uniformly sampled on a fixed radius ball around each training data and hence, effectively increase the distance between natural data points and decision boundary.Our experiments show that Bamboo substantially improve the general robustness against arbitrary types of attacks and noises, achieving better results comparing to previous adversarial training methods, robust optimization methods and other data augmentation methods with the same amount of data points.

第一个数据增强方法是专门为提高DNN的一般稳健性而设计的，对攻击算法没有任何假设。

提出了一种数据增强训练方法，通过从以训练数据为中心的固定半径球体中增强均匀的随机样本，以获得模型的稳健性，对抗对抗性扰动。