The ability to synthesize realistic patterns of neural activity is crucial for studying neural information processing.Here we used the Generative Adversarial Networks (GANs) framework to simulate the concerted activity of a population of neurons.We adapted the Wasserstein-GAN variant to facilitate the generation of unconstrained neural population activity patterns while still benefiting from parameter sharing in the temporal domain.We demonstrate that our proposed GAN, which we termed Spike-GAN, generates spike trains that match accurately the first- and second-order statistics of datasets of tens of neurons and also approximates well their higher-order statistics.We applied Spike-GAN to a real dataset recorded from salamander retina and showed that it performs as well as state-of-the-art approaches based on the maximum entropy and the dichotomized Gaussian frameworks.Importantly, Spike-GAN does not require to specify a priori the statistics to be matched by the model, and so constitutes a more flexible method than these alternative approaches.Finally, we show how to exploit a trained Spike-GAN  to construct 'importance maps' to detect the most relevant statistical structures present in a spike train. Spike-GAN provides a powerful, easy-to-use technique for generating realistic spiking neural activity and for describing the most relevant features of the large-scale neural population recordings studied in modern systems neuroscience.

使用Wasserstein-GAN来产生真实的神经活动，并检测神经群体模式中存在的最相关的特征。

一种使用半卷积GAN模拟符合经验数据的神经元群的尖峰训练的方法。

本文提出使用GAN来合成真实的神经活动模式


Deep latent variable models have become a popular model choice due to the scalable learning algorithms introduced by (Kingma & Welling 2013, Rezende et al. 2014).These approaches maximize a variational lower bound on the intractable log likelihood of the observed data.Burda et al. (2015) introduced a multi-sample variational bound, IWAE, that is at least as tight as the standard variational lower bound and becomes increasingly tight as the number of samples increases.Counterintuitively, the typical inference network gradient estimator for the IWAE bound performs poorly as the number of samples increases (Rainforth et al. 2018, Le et al. 2018).Roeder et a.(2017) propose an improved gradient estimator, however, are unable to show it is unbiased.We show that it is in fact biased and that the bias can be estimated efficiently with a second application of the reparameterization trick.The doubly reparameterized gradient (DReG) estimator does not suffer as the number of samples increases, resolving the previously raised issues.The same idea can be used to improve many recently introduced training techniques for latent variable models.In particular, we show that this estimator reduces the variance of the IWAE gradient, the reweighted wake-sleep update (RWS) (Bornschein & Bengio 2014), and the jackknife variational inference (JVI) gradient (Nowozin 2018).Finally, we show that this computationally efficient, drop-in estimator translates to improved performance for all three objectives on several modeling tasks.

双重重参数化梯度估计器提供了无偏的方差减少，导致性能的提高。

作者通过实验发现，现有工作的估计器（STL）是有偏差的，并提出减少偏差以改善ELBO的梯度估计器。


Zeroth-order optimization is the process of minimizing an objective $f(x)$, given oracle access to evaluations at adaptively chosen inputs $x$.In this paper, we present two simple yet powerful GradientLess Descent (GLD) algorithms that do not rely on an underlying gradient estimate and are numerically stable.We analyze our algorithm from a novel geometric perspective and we show that for {\it any monotone transform} of a smooth and strongly convex objective with latent dimension $k \ge n$, we present a novel analysis that shows convergence within an $\epsilon$-ball of the optimum in $O(kQ\log(n)\log(R/\epsilon))$ evaluations, where the input dimension is $n$, $R$ is the diameter of the input space and $Q$ is the condition number.Our rates are the first of its kind to be both1) poly-logarithmically dependent on dimensionality and2) invariant under monotone transformations.We further leverage our geometric perspective to show that our analysis is optimal.Both monotone invariance and its ability to utilize a low latent dimensionality are key to the empirical success of our algorithms, as demonstrated on synthetic and MuJoCo benchmarks.

无梯度下降是一种被证明有效的无梯度算法，对于高维零阶优化来说是单调不变的，而且速度很快。

本文提出了稳定的无梯度下降（GLD）算法，该算法不依赖于梯度估计。


Many processes can be concisely represented as a sequence of events leading from a starting state to an end state.Given raw ingredients, and a finished cake, an experienced chef can surmise the recipe.Building upon this intuition,  we propose a new class of visual generative models: goal-conditioned predictors (GCP).Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video.GCP instead treats videos as start-goal transformations, making video generation easier by conditioning on the more informative context provided by the first and final frames. Not only do existing forward prediction approaches synthesize better and longer videos when modified to become goal-conditioned,  but GCP models can also utilize structures that are not linear in time, to accomplish hierarchical prediction. To this end, we study both auto-regressive GCP models and novel tree-structured GCP models that generate frames recursively, splitting the video iteratively into finer and finer segments delineated by subgoals. In experiments across simulated and real datasets, our GCP methods generate high-quality sequences over long horizons.  Tree-structured GCPs are also substantially easier to parallelize than auto-regressive GCPs, making training  and  inference  very  efficient, and allowing the model to train on sequences that are thousands of frames in length.Finally, we demonstrate the utility of GCP approaches for imitation learning in the setting without access to expert actions. Videos are on the supplementary website: https://sites.google.com/view/video-gcp

我们提出了一类新的视觉生成模型：目标条件的预测器。我们通过实验表明，以目标为条件可以减少不确定性，并在更长的时间内产生预测结果。

本文通过对开始和结束（目标）帧的预测进行调节，将视频预测问题重新表述为内插法而不是外推法，从而获得更高质量的预测。


Recent advances in computing technology and sensor design have made it easier to collect longitudinal or time series data from patients, resulting in a gigantic amount of available medical data.Most of the medical time series lack annotations or even when the annotations are available they could be subjective and prone to human errors.Earlier works have developed natural language processing techniques to extract concept annotations and/or clinical narratives from doctor notes.However, these approaches are slow and do not use the accompanying medical time series data.To address this issue, we introduce the problem of concept annotation for the medical time series data, i.e., the task of predicting and localizing medical concepts by using the time series data as input.We propose Relational Multi-Instance Learning (RMIL) - a deep Multi Instance Learning framework based on recurrent neural networks, which uses pooling functions and attention mechanisms for the concept annotation tasks.Empirical results on medical datasets show that our proposed models outperform various multi-instance learning models.

我们提出了一个基于循环神经网络的深度多实例学习框架，该框架使用池化函数和注意力机制来完成概念注释任务。

本文涉及医疗时间序列数据的分类，并提出使用循环神经网络架构对每个序列中的实例之间的时间关系进行建模。

提出了一种新的多实例学习（MIL）公式，称为关系MIL（RMIL），并讨论了它与LSTM、Bi-LSTM、S2S等的一些变体，探讨了RMIL与各种注意力机制的整合，并展示了它在时间序列数据的医学概念预测中的应用。


The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing.However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting.We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance.  We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.

嵌入层通过张量训练分解进行因子化，以减少其内存占用。

本文提出了一种低秩张量分解模型，用于对自然语言处理（NLP）中的嵌入矩阵进行参数化，该模型可以压缩网络，有时可以提高测试的准确性。


We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor. We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function.We provide empirical evidence that our proposed modification(i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and(ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter).We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence.Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32. Our source code will become available after the review process.

亚当等自适应梯度方法中的固定权重衰减正则化

提出了将权重衰减与优化过程中的步骤数脱钩的想法。

本文提出了一种在Adam中实施权重衰减的替代方法，其实证结果显示

研究了SGD变体中存在的权重衰减问题，提出了权重衰减和基于梯度的更新之间的解耦方法。


Lifelong learning is the problem of learning multiple consecutive tasks in a sequential manner where knowledge gained from previous tasks is retained and used for future learning.It is essential towards the development of intelligent machines that can adapt to their surroundings.In this work we focus on a lifelong learning approach to generative modeling where we continuously incorporate newly observed streaming distributions into our learnt model.We do so through a student-teacher architecture which allows us to learn and preserve all the distributions seen so far without the need to retain the past data nor the past models.Through the introduction of a novel cross-model regularizer, the student model leverages the information learnt by the teacher, which acts as a summary of everything seen till now.The regularizer has the additional benefit of reducing the effect of catastrophic interference that appears when we learn over streaming data.We demonstrate its efficacy on streaming distributions as well as its ability to learn a common latent representation across a complex transfer learning scenario.

通过学生-教师结构与跨模型后验正则的结合，实现终身分布式学习。


Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling.In this paper, we look at geometric data represented as point clouds.We introduce a deep autoencoder (AE) network with excellent reconstruction quality and generalization ability.The learned representations outperform the state of the art in 3D recognition tasks and enable basic shape editing applications via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation.We also perform a thorough study of different generative models including GANs operating on the raw point clouds, significantly improved GANs trained in the fixed latent space our AEs and, Gaussian mixture models (GMM).Interestingly, GMMs trained in the latent space of our AEs produce samples of the best fidelity and diversity.To perform our quantitative evaluation of generative models, we propose simple measures of fidelity and diversity based on optimally matching between sets point clouds.

深度自动编码器学习几何三维点云数据的良好表示；点云的生成模型。


Despite the remarkable performance of deep neural networks (DNNs) on various tasks, they are susceptible to adversarial perturbations which makes it difficult to deploy them in real-world safety-critical applications.In this paper, we aim to obtain robust networks by sparsifying DNN's latent features sensitive to adversarial perturbation.Specifically, we define vulnerability at the latent feature space and then propose a Bayesian framework to prioritize/prune features based on their contribution to both the original and adversarial loss.We also suggest regularizing the features' vulnerability during training to improve robustness further.While such network sparsification has been primarily studied in the literature for computational efficiency and regularization effect of DNNs, we confirm that it is also useful to design a defense mechanism through quantitative evaluation and qualitative analysis.We validate our method, \emph{Adversarial Neural Pruning (ANP)} on multiple benchmark datasets, which results in an improvement in test accuracy and leads to state-of-the-art robustness.ANP also tackles the practical problem of obtaining sparse and robust networks at the same time, which could be crucial to ensure adversarial robustness on lightweight networks deployed to computation and memory-limited devices.

我们提出了一种新的方法来抑制潜在特征空间的脆弱性，以实现稳健和紧凑的网络。

本文提出了 "对抗性神经修剪 "的方法，即训练一个修剪掩码和一个新的漏洞抑制损失，以提高准确性和对抗性稳健性。


In anomaly detection (AD), one seeks to identify whether a test sample is abnormal,  given a data set of normal samples.   A recent and promising approach to AD relies on deep generative models, such as variational autoencoders (VAEs),for unsupervised learning of the normal data distribution.In semi-supervised AD (SSAD), the data also includes a small sample of labeled anomalies.In this work,we propose two variational methods for training VAEs for SSAD.The intuitive idea in both methods is to train the encoder to â€˜separateâ€™ between latent vectors for normal and outlier data.We show that this idea can be derived from principled probabilistic formulations of the problem, and propose simple and effective algorithms.  Our methods can be applied to various data types, as we demonstrate on SSAD datasets ranging from natural images to astronomy and medicine, and can be combined with any VAE model architecture.When comparing to state-of-the-art SSAD methods that are not specific to particular data types, we obtain marked improvement in outlier detection.

我们提出了两个考虑到负面数据例子的VAE修改，并将其用于半监督性的异常检测。

这些论文提出了两种类似VAE的半监督新奇性检测方法，即MML-VAE和DP-VAE。


We introduce dynamic instance hardness (DIH) to facilitate the training of machine learning models.DIH is a property of each training sample and is computed as the running mean of the sample's instantaneous hardness as measured over the training history.We use DIH to evaluate how well a model retains knowledge about each training sample over time.We find that for deep neural nets (DNNs), the DIH of a sample in relatively early training stages reflects its DIH in later stages and as a result, DIH can be effectively used to reduce the set of training samples in future epochs.Specifically, during each epoch, only samples with high DIH are trained (since they are historically hard) while samples with low DIH can be safely ignored.DIH is updated each epoch only for the selected samples, so it does not require additional computation.Hence, using DIH during training leads to an appreciable speedup.Also, since the model is focused on the historically more challenging samples, resultant models are more accurate.The above, when formulated as an algorithm, can be seen as a form of curriculum learning, so we call our framework DIH curriculum learning (or DIHCL).The advantages of DIHCL, compared to other curriculum learning approaches, are: (1) DIHCL does not require additional inference steps over the data not selected by DIHCL in each epoch, (2) the dynamic instance hardness, compared to static instance hardness (e.g., instantaneous loss), is more stable as it integrates information over the entire training history up to the present time.Making certain mathematical assumptions, we formulate the problem of DIHCL as finding a curriculum that maximizes a multi-set function $f(\cdot)$, and derive an approximation bound for a DIH-produced curriculum relative to the optimal curriculum.Empirically, DIHCL-trained DNNs significantly outperform random mini-batch SGD and other recently developed curriculum learning methods in terms of efficiency, early-stage convergence, and final performance, and this is shown in training several state-of-the-art DNNs on 11 modern datasets.

对训练动力学的新理解和记忆硬度的衡量标准导致了高效和可证明的课程学习。

本文将DIH表述为一个课程学习问题，可以更有效地利用数据来训练DNN，并推导出关于近似边界的理论。


This paper explores many immediate connections between adaptive control and machine learning, both through common update laws as well as common concepts.Adaptive control as a field has focused on mathematical rigor and guaranteed convergence.The rapid advances in machine learning on the other hand have brought about a plethora of new techniques and problems for learning.This paper elucidates many of the numerous common connections between both fields such that results from both may be leveraged together to solve new problems.In particular, a specific problem related to higher order learning is solved through insights obtained from these intersections.

更新规律的平行发展史，以及机器学习中适应性控制和优化之间的概念。


Recurrent convolution (RC) shares the same convolutional kernels and unrolls them multiple times, which is originally proposed to model time-space signals.We suggest that RC can be viewed as a model compression strategy for deep convolutional neural networks.RC reduces the redundancy across layers and is complementary to most existing model compression approaches.However, the performance of an RC network can't match the performance of its corresponding standard one, i.e. with the same depth but independent convolutional kernels.  This reduces the value of RC for model compression.In this paper, we propose a simple variant which improves RC networks: The batch normalization layers of an RC module are learned independently (not shared) for different unrolling steps.We provide insights on why this works.Experiments on CIFAR show that unrolling a convolutional layer several steps can improve the performance, thus indirectly plays a role in model compression.

用于模型压缩的循环卷积和一个训练它的技巧，即在步骤上学习独立的BN分层。

作者用独立的批处理规范化修改了循环卷积神经网络（RCNN），当RCNN包含相同的层数时，RCNN的实验结果与ResNet神经网络架构兼容。


The visual world is vast and varied, but its variations divide into structured and unstructured factors.Structured factors, such as scale and orientation, admit clear theories and efficient representation design.Unstructured factors, such as what it is that makes a cat look like a cat, are too complicated to model analytically, and so require free-form representation learning.We compose structured Gaussian filters and free-form filters, optimized end-to-end, to factorize the representation for efficient yet general learning.Our experiments on dynamic structure, in which the structured filters vary with the input, equal the accuracy of dynamic inference with more degrees of freedom while improving efficiency.(Please see https://arxiv.org/abs/1904.11487 for the full edition.)

具有空间高斯结构的动态感受场是准确和有效的。 本文提出了一个结构化卷积算子来模拟图像局部区域的变形，这大大减少了参数的数量。


It is widely known that well-designed perturbations can cause state-of-the-art machine learning classifiers to mis-label an image, with sufficiently small perturbations that are imperceptible to the human eyes.However, by detecting the inconsistency between the image and wrong label, the human observer would be alerted of the attack.In this paper, we aim to design attacks that not only make classifiers generate wrong labels, but also make the wrong labels imperceptible to human observers.To achieve this, we propose an algorithm called LabelFool which identifies a target label similar to the ground truth label and finds a perturbation of the image for this target label.We first find the target label for an input image by a probability model, then move the input in the feature space towards the target label.Subjective studies on ImageNet show that in the label space, our attack is much less recognizable by human observers, while objective experimental results on ImageNet show that we maintain similar performance in the image space as well as attack rates to state-of-the-art attack algorithms.

关于对抗性样本的技巧，使错误分类的标签在标签空间中对人类观察者来说是不可察觉的。

一种构建对抗性攻击的方法，通过改变目标类别，使其与图像的原始类别相似，从而在图像空间中无成本就能被人类检测到。


This paper presents noise type/position classification of various impact noises generated in a building which is a serious conflict issue in apartment complexes.For this study, a collection of floor impact noise dataset is recorded with a single microphone.Noise types/positions are selected based on a report by the Floor Management Center under Korea Environmental Corporation.Using a convolutional neural networks based classifier, the impact noise signals converted to log-scaled Mel-spectrograms are classified into noise types or positions.Also, our model is evaluated on a standard environmental sound dataset ESC-50 to show extensibility on environmental sound classification.

本文介绍了建筑物内产生的各种影响噪音的噪音类型/位置分类，这是公寓楼内的一个严重冲突问题。

这项工作描述了卷积神经网络在建筑噪声类型和噪声位置分类这一新型应用领域中的应用。


Recordings of neural circuits in the brain reveal extraordinary dynamical richness and high variability.At the same time, dimensionality reduction techniques generally uncover low-dimensional structures underlying these dynamics.What determines the dimensionality of activity in neural circuits?What is the functional role of dimensionality in behavior and task learning?In this work we address these questions using recurrent neural network (RNN) models.We find that, depending on the dynamics of the initial network, RNNs learn to increase and reduce dimensionality in a way that matches task demands.These findings shed light on fundamental dynamical mechanisms by which neural networks solve tasks with robust representations that generalize to new cases.

循环神经网络学会以与任务相匹配的方式增加和减少其内部表征的维度，这取决于初始网络的动态变化。


Domain adaptation addresses the common problem when the target distribution generating our test data drifts from the source (training) distribution.While absent assumptions, domain adaptation is impossible, strict conditions, e.g. covariate or label shift, enable principled algorithms.Recently-proposed domain-adversarial approaches consist of aligning source and target encodings, often motivating this approach as minimizing two (of three) terms in a theoretical bound on target error.Unfortunately, this minimization can cause arbitrary increases in the third term, e.g. they can break down under shifting label distributions.We propose asymmetrically-relaxed distribution alignment, a new approach that overcomes some limitations of standard domain-adversarial algorithms.Moreover, we characterize precise assumptions under which our algorithm is theoretically principled and demonstrate empirical benefits on both synthetic and real datasets.

在传统的深度领域适应性目标中，严格的分布排列在目标标签分布转移时是失败的，而我们提出用新的分析、新的算法和实验验证来优化一个宽松的目标。

本文提出了领域适应的宽松度量，对目标误差给出了新的理论边界。


In this paper, we explore \textit{summary-to-article generation}: the task of generating long articles given a short summary, which provides finer-grained content control for the generated text.To prevent sequence-to-sequence (seq2seq) models from degenerating into language models and better controlling the long text to be generated, we propose a hierarchical generation approach which first generates a sketch of intermediate length based on the summary and then completes the article by enriching the generated sketch.To mitigate the discrepancy between the ``oracle'' sketch used during training and the noisy sketch generated during inference, we propose an end-to-end joint training framework based on multi-agent reinforcement learning.For evaluation, we use text summarization corpora by reversing their inputs and outputs, and introduce a novel evaluation method that employs a summarization system to summarize the generated article and test its match with the original input summary.Experiments show that our proposed hierarchical generation approach can generate a coherent and relevant article based on the given summary, yielding significant improvements upon conventional seq2seq models.

我们探讨了摘要到文章的生成任务，并提出了一个分层生成方案，以及一个联合的端到端强化学习框架来训练分层模型。

为了解决摘要到文章生成中的退化问题，本文提出了一种分层生成的方法，首先生成文章的中间草图，然后再生成完整的文章。


When training a deep neural network for supervised image classification, one can broadly distinguish between two types of latent features of images that will drive the classification of class Y. Following the notation of Gong et al. (2016), we can divide features broadly into the classes of(i) â€œcoreâ€ or â€œconditionally invariantâ€ features X^ci whose distribution P(X^ci | Y) does not change substantially across domains and(ii) â€œstyleâ€ or â€œorthogonalâ€ features X^orth whose distribution P(X^orth | Y) can change substantially across domains.These latter orthogonal features would generally include features such as position, rotation, image quality or brightness but also more complex ones like hair color or posture for images of persons.We try to guard against future adversarial domain shifts by ideally just using the â€œconditionally invariantâ€ features for classification.In contrast to previous work, we assume that the domain itself is not observed and hence a latent variable.We can hence not directly see the distributional change of features across different domains. We do assume, however, that we can sometimes observe a so-called identifier or ID variable.We might know, for example, that two images show the same person, with ID referring to the identity of the person.In data augmentation, we generate several images from the same original image, with ID referring to the relevant original image.The method requires only a small fraction of images to have an ID variable.We provide a causal framework for the problem by adding the ID variable to the model of Gong et al. (2016).However, we are interested in settings where we cannot observe the domain directly and we treat domain as a latent variable.If two or more samples share the same class and identifier, (Y, ID)=(y,i), then we treat those samples as counterfactuals under different style interventions on the orthogonal or style features.Using this grouping-by-ID approach, we regularize the network to provide near constant output across samples that share the same ID by penalizing with an appropriate graph Laplacian.This is shown to substantially improve performance in settings where domains change in terms of image quality, brightness, color changes, and more complex changes such as changes in movement and posture.We show links to questions of interpretability, fairness and transfer learning.

我们提出了反事实正则化，以防止因图像的潜在 "风格特征 "的分布变化而产生的对抗性领域转移。

本文讨论了通过学习一个对图像中的表面变化（或 "风格 "特征）不变的分类器，来防范具有反事实正则化的对抗性领域转移的方法。

本文的目的是针对对抗性领域的转移进行稳健的图像分类，通过避免使用变化的风格特征来实现这一目标。


Gradient-based meta-learning algorithms require several steps of gradient descent to adapt to newly incoming tasks.This process becomes more costly as the number  of  samples  increases.  Moreover,  the  gradient  updates  suffer  from  several sources of noise leading to a degraded performance.   In this work,  we propose a meta-learning algorithm equipped with the GradiEnt Component COrrections, aGECCO cell for short, which generates a multiplicative corrective low-rank matrix which (after vectorization) corrects the estimated gradients. GECCO contains a simple decoder-like network with learnable parameters, an attention module and a so-called context input parameter. The context parameter of GECCO is updated to  generate  a  low-rank  corrective  term  for  the  network  gradients.   As  a  result, meta-learning requires only a few of gradient updates to absorb new task (often, a single update is sufficient in the few-shot scenario). While previous approaches address this problem by altering the learning rates, factorising network parameters or directly learning feature corrections from features and/or gradients, GECCO is an off-the-shelf generator-like unit that performs element-wise gradient corrections without the need to â€˜observeâ€™ the features and/or the gradients directly.  We show that our GECCO(i) accelerates learning,(ii) performs robust corrections of the gradients corrupted by a noise, and(iii) leads to notable improvements over existing gradient-based meta-learning algorithms.

我们提出了一个元学习器，以便在多个任务上快速适应，甚至是在少样本的设置中一步到位。

本文提出了一种元学习梯度校正模块的方法，其中预设条件是由神经网络参数化的，并在适应过程中建立了一个两阶段的梯度更新过程。


Discriminative  question  answering  models  can  overfit  to  superficial  biases  in datasets,  because their loss function saturates when any clue makes the answer likely.  We introduce generative models of the joint distribution of questions and answers, which are trained to explain the whole question, not just to answer it.Our  question  answering  (QA)  model  is  implemented  by  learning  a  prior  over answers,  and  a  conditional  language  model  to  generate  the  question  given  the answerâ€”allowing scalable and interpretable many-hop reasoning as the question is generated word-by-word.  Our model achieves competitive performance with specialised discriminative models on the SQUAD and CLEVR benchmarks, indicating that it is a more general architecture for language understanding and reasoning than previous work.The model greatly improves generalisation both from biased training data and to adversarial testing data, achieving a new state-of-the-art on ADVERSARIAL SQUAD.We will release our code.

对问题和答案的联合分布进行建模的问答模型可以比判别性模型学到更多东西

本文提出了一种文本和视觉QA的生成方法，即在给定的背景下学习问题和答案空间的联合分布，这可以捕捉到更复杂的关系。

本文介绍了一个用于回答问题的生成模型，并建议将p(q,a|c)建模，因子为p(a|c) * p(q|a,c)。

作者提出了一个生成性的QA模型，该模型对问题的分布和给定文件/背景的回答进行联合优化。


In this paper, we turn our attention to the interworking between the activation functions and the batch normalization, which is a virtually mandatory technique to train deep networks currently.We propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization.Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of standardized VGG and Residual Networks state-of-the-art models.These convolutional neural networks were trained on CIFAR-100 and CIFAR-10, the most commonly used deep learning computer vision datasets.The results showed DReLU speeded up learning in all models and datasets.Besides, statistical significant performance assessments (p<0.05) showed DReLU enhanced the test accuracy presented by ReLU in all scenarios.Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception, in which case it presented the second best performance.Therefore, this work demonstrates that it is possible to increase performance replacing ReLU by an enhanced activation function.

提出了一个新的激活函数，称为位移整流器线性单元。它被证明可以提高批量归一化卷积神经网络的训练和推理性能。

本文比较并建议反对使用整流器线性单元后的批量规范化的用法。

本文提出了一个激活函数，称为位移ReLU，以提高使用批量归一化的CNN的性能。


Encoding the input scale information explicitly into the representation learned by a convolutional neural network (CNN) is beneficial for many vision tasks especially when dealing with multiscale input signals.We study, in this paper, a scale-equivariant CNN architecture with joint convolutions across the space and the scaling group, which is shown to be both sufficient and necessary to achieve scale-equivariant representations.To reduce the model complexity and computational burden, we decompose the convolutional filters under two pre-fixed separable bases and truncate the expansion to low-frequency components.A further benefit of the truncated filter expansion is the improved deformation robustness of the equivariant representation.Numerical experiments demonstrate that the proposed scale-equivariant neural network with decomposed convolutional filters (ScDCFNet) achieves significantly improved performance in multiscale image classification and better interpretability than regular CNNs at a reduced model size.

我们以最一般的形式构建了标度等价卷积神经网络，既具有计算效率，又被证明具有变形稳健性。

作者提出了一个CNN架构，该架构通过在激活张量上增加一个额外的标度维，在理论上对各向同性的标度和平移具有等效性。


In this paper, we diagnose deep neural networks for 3D point cloud processing to explore the utility of different network architectures.We propose a number of hypotheses on the effects of specific network architectures on the representation capacity of DNNs.In order to prove the hypotheses, we design five metrics to diagnose various types of DNNs from the following perspectives, information discarding, information concentration, rotation robustness, adversarial robustness, and neighborhood inconsistency.We conduct comparative studies based on such metrics to verify the hypotheses, which may shed new lights on the architectural design of neural networks.Experiments demonstrated the effectiveness of our method.The code will be released when this paper is accepted.

我们对三维点云处理的深度神经网络进行了诊断，以探索不同网络架构的效用。

本文研究了用于3D点云处理的不同神经网络架构，并提出了对抗性稳健性、旋转稳健性和邻域一致性的衡量标准。


In this work we construct flexible joint distributions from low-dimensional conditional semi-implicit distributions.Explicitly defining the structure of the approximation allows to make the variational lower bound tighter, resulting in more accurate inference.

利用分布的结构改善半隐式变分推理


Imitation learning from human-expert demonstrations has been shown to be greatly helpful for challenging reinforcement learning problems with sparse environment rewards.However, it is very difficult to achieve similar success without relying on expert demonstrations.Recent works on self-imitation learning showed that imitating the agent's own past good experience could indirectly drive exploration in some environments, but these methods often lead to sub-optimal and myopic behavior.To address this issue, we argue that exploration in diverse directions by imitating diverse trajectories, instead of focusing on limited good trajectories, is more desirable for the hard-exploration tasks.We propose a new method of learning a trajectory-conditioned policy to imitate diverse trajectories from the agent's own past experiences and show that such self-imitation helps avoid myopic behavior and increases the chance of finding a globally optimal solution for hard-exploration tasks, especially when there are misleading rewards.Our method significantly outperforms existing self-imitation learning and count-based exploration methods on various hard-exploration tasks with local optima.In particular, we report a state-of-the-art score of more than 20,000 points on Montezumas Revenge without using expert demonstrations or resetting to arbitrary states.

具有轨迹条件的不同轨迹的自我模仿学习策略

本文通过对过去经验中多样化的轨迹选择进行自我模仿来解决困难的探索任务，以推动稀疏奖励问题中更有效的探索，实现SOTA结果。


We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost.This is achieved by gating the deep-learning architecture on a fine-grained-level.Individual convolutional maps are turned on/off conditionally on features in the network.To achieve this, we introduce a new residual block architecture that gates convolutional channels in a fine-grained manner.We also introduce a generally applicable tool batch-shaping that matches the marginal aggregate posteriors of features in a neural network to a pre-specified prior distribution.We use this novel technique to force gates to be more conditional on the data.We present results on CIFAR-10 and ImageNet datasets for image classification, and Cityscapes for semantic segmentation.Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy.In particular, on ImageNet, our ResNet50 and ResNet34 gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity.We also show that the resulting networks automatically learn to use more features for difficult examples and fewer features for simple examples.

一种训练大容量神经网络的方法，其准确性显著提高，动态计算成本较低

使用细粒度的条件选择和新的正则化方法 "批量整形"，训练具有大容量的网络，在推理时只使用其中的一部分，取决于输入。


With a view to bridging the gap between deep learning and symbolic AI, we present a novel end-to-end neural network architecture that learns to form propositional representations with an explicitly relational structure from raw pixel data.In order to evaluate and analyse the architecture, we introduce a family of simple visual relational reasoning tasks of varying complexity.We show that the proposed architecture, when pre-trained on a curriculum of such tasks, learns to generate reusable representations that better facilitate subsequent learning on previously unseen tasks when compared to a number of baseline architectures.The workings of a successfully trained model are visualised to shed some light on how the architecture functions.

我们提出了一个端到端的可微分的架构，它可以学习将像素映射到谓词，并在一套简单的关系推理任务上对其进行评估

基于多头自我注意模块的网络架构，学习一种新的关系表征形式，在课程学习上提高了数据效率和泛化能力。


In natural language inference, the semantics of some words do not affect the inference.Such information is considered superficial and brings overfitting.How can we represent and discard such superficial information?In this paper, we use first order logic (FOL) - a classic technique from meaning representation language â€“ to explain what information is superficial for a given sentence pair.Such explanation also suggests two inductive biases according to its properties.We proposed a neural network-based approach that utilizes the two inductive biases.We obtain substantial improvements over extensive experiments.

我们通过从一阶逻辑的角度定义和识别表层信息，利用神经网络将表层信息投射出来，用于自然语言推理。

本文试图减少自然语言推理中的表面信息，以防止过度拟合，并引入图神经网络来模拟前提和假设之间的关系。

一种使用一阶逻辑处理自然语言推理的方法，并在NLI模型中注入逻辑信息，使其在推理中更加稳健。


We propose an approach to training machine learning models that are fair in the sense that their performance is invariant under certain perturbations to the features.For example, the performance of a resume screening system should be invariant under changes to the name of the applicant.We formalize this intuitive notion of fairness by connecting it to the original  notion of individual fairness put forth by Dwork et al and show that the proposed approach achieves this notion of fairness.We also demonstrate the effectiveness of the approach on two machine learning tasks that are susceptible to gender and racial biases.

利用对抗性稳健性训练个别公平分类器的算法

本文提出了一个新的算法公平性的定义，以及一个可以证明找到满足公平性约束的ML模型的算法。


In this paper, we propose a Seed-Augment-Train/Transfer (SAT) framework that contains a synthetic seed image dataset generation procedure for languages with different numeral systems using freely available open font file datasets.This seed dataset of images is then augmented to create a purely synthetic training dataset, which is in turn used to train a deep neural network and test on held-out real world handwritten digits dataset spanning five Indic scripts, Kannada, Tamil, Gujarati, Malayalam, and Devanagari.We showcase the efficacy of this approach both qualitatively, by training a Boundary-seeking GAN (BGAN) that generates realistic digit images in the five languages, and also qualitatively by testing a CNN trained on the synthetic data on the real-world datasets.This establishes not only an interesting nexus between the font-datasets-world and transfer learning but also provides a recipe for universal-digit classification in any script.

在任何语言中对数字进行分类时，播种和增殖都是你所需要的吗？

本文介绍了五种语言的新数据集，并提出了一个用于通用数字分类的字体图像数据集生成的新框架（SAT）。


An important research direction in machine learning has centered around developing meta-learning algorithms to tackle few-shot learning.An especially successful algorithm has been Model Agnostic Meta-Learning (MAML), a method that consists of two optimization loops, with the outer loop finding a meta-initialization, from which the inner loop can efficiently learn new tasks.Despite MAML's popularity, a fundamental open question remains -- is the effectiveness of MAML due to the meta-initialization being primed for rapid learning (large, efficient changes in the representations) or due to feature reuse,  with the meta initialization already containing high quality features?We investigate this question, via ablation studies and analysis of the latent representations, finding that feature reuse is the dominant factor.This leads to the ANIL (Almost No Inner Loop) algorithm, a simplification of MAML where we remove the inner loop for all but the (task-specific) head of the underlying neural network.ANIL matches MAML's performance on benchmark few-shot image classification and RL and offers computational improvements over MAML.We further study the precise contributions of the head and body of the network, showing that performance on the test tasks is entirely determined by the quality of the learned features, and we can remove even the head of the network (the NIL algorithm).We conclude with a discussion of the rapid learning vs feature reuse question for meta-learning algorithms more broadly.

MAML的成功依赖于元初始化的特征重用，这也产生了算法的自然简化，网络主体的内循环被删除，以及对头部和主体的其他见解。

论文发现，特征重用是MAML成功的主导因素，并提出了新的算法，这些算法花费的计算量比MAML少得多。


Model training remains a dominant financial cost and time investment in machine learning applications.Developing and debugging models often involve iterative training, further exacerbating this issue.With growing interest in increasingly complex models, there is a need for techniques that help to reduce overall training effort.While incremental training can save substantial time and cost by training an existing model on a small subset of data, little work has explored policies for determining when incremental training provides adequate model performance versus full retraining.We provide a method-agnostic algorithm for deciding when to incrementally train versus fully train.We call this setting of non-deterministic full- or incremental training ``Mixed Setting Training".Upon evaluation in slot-filling tasks, we find that this algorithm provides a bounded error, avoids catastrophic forgetting, and results in a significant speedup over a policy of always fully training.

我们提供了一种决定何时进行增量训练与完全训练的方法无关的算法，它比完全训练有明显的速度提升，并避免了灾难性的遗忘。

本文提出了一种方法，用于决定在填空任务中迭代模型开发的设置下，何时增量与完全重新训练模型。


Neural networks have succeeded in many reasoning tasks.Empirically, these tasks require specialized network structures, e.g., Graph Neural Networks (GNNs) perform well on many such tasks, while less structured networks fail.Theoretically, there is limited understanding of why and when a network structure generalizes better than other equally expressive ones.We develop a framework to characterize which reasoning tasks a network can learn well, by studying how well its structure aligns with the algorithmic structure of the relevant reasoning procedure.We formally define algorithmic alignment and derive a sample complexity bound that decreases with better alignment.This framework explains the empirical success of popular reasoning models and suggests their limitations.We unify seemingly different reasoning tasks, such as intuitive physics, visual question answering, and shortest paths, via the lens of a powerful algorithmic paradigm, dynamic programming (DP).We show that GNNs can learn DP and thus solve these tasks.On several reasoning tasks, our theory aligns with empirical results.

我们开发了一个理论框架来描述神经网络可以很好地学习哪些推理任务。

本文提出了一种衡量算法一致性的类别，以衡量神经网络与已知算法的 "接近 "程度，证明了几类已知算法与神经网络架构之间的联系。


Cell-cell interactions have an integral role in tumorigenesis as they are critical in governing immune responses.As such, investigating specific cell-cell interactions has the potential to not only expand upon the understanding of tumorigenesis, but also guide clinical management of patient responses to cancer immunotherapies.A recent imaging technique for exploring cell-cell interactions, multiplexed ion beam imaging by time-of-flight (MIBI-TOF), allows for cells to be quantified in 36 different protein markers at sub-cellular resolutions in situ as high resolution multiplexed images.To explore the MIBI images, we propose a GAN for multiplexed data with protein specific attention.By conditioning image generation on cell types, sizes, and neighborhoods through semantic segmentation maps, we are able to observe how these factors affect cell-cell interactions simultaneously in different protein channels.Furthermore, we design a set of metrics and offer the first insights towards cell spatial orientations, cell protein expressions, and cell neighborhoods.Our model, cell-cell interaction GAN (CCIGAN), outperforms or matches existing image synthesis methods on all conventional measures and significantly outperforms on biologically motivated metrics.To our knowledge, we are the first to systematically model multiple cellular protein behaviors and interactions under simulated conditions through image synthesis.

我们通过使用一个新的注意力GAN架构进行图像合成，探索在高度复用的图像中观察到的跨肿瘤环境背景下的细胞-细胞相互作用。

通过学习细胞类型和蛋白质标记物表达水平之间的多对多映射，建立了一种新的方法来模拟由多路离子束飞行时间成像（MIBI-TOF）产生的数据。


Machine learning models for question-answering (QA), where given a question and a passage, the learner must select some span in the passage as an answer, are known to be brittle.By inserting a single nuisance sentence into the passage, an adversary can fool the model into selecting the wrong span.A promising new approach for QA decomposes the task into two stages:(i) select relevant sentences from the passage; and(ii) select a span among those sentences.Intuitively, if the sentence selector excludes the offending sentence, then the downstream span selector will be robust.While recent work has hinted at the potential robustness of two-stage QA, these methods have never, to our knowledge, been explicitly combined with adversarial training.This paper offers a thorough empirical investigation of adversarial robustness, demonstrating that although the two-stage approach lags behind single-stage span selection, adversarial training improves its performance significantly, leading to an improvement of over 22 points in F1 score over the adversarially-trained single-stage model.

与根据完整上下文训练的单阶段模型相比，由句子选择和跨度选择组成的两阶段方法可以对对抗性攻击更加稳健。

本文调查了一个现有的模型，发现与其他方法相比，两阶段训练的QA方法对对抗性攻击并不更强大。


The aim of this study is to introduce a formal framework for analysis and synthesis of driver assistance systems.It applies formal methods to the verification of a stochastic human driver model built using the cognitive architecture ACT-R, and then bootstraps safety in semi-autonomous vehicles through the design of provably correct Advanced Driver Assistance Systems.The main contributions include the integration of probabilistic ACT-R models in the formal analysis of semi-autonomous systems and an abstraction technique that enables a finite representation of a large dimensional, continuous system in the form of a Markov model.The effectiveness of the method is illustrated in several case studies under various conditions.

基于认知架构的人类驾驶模型的验证，以及根据该模型合成的正确的ADAS。


In contrast to the older writing system of the 19th century, modern Hawaiian orthography employs characters for long vowels and glottal stops.These extra characters account for about one-third of the phonemes in Hawaiian, so including them makes a big difference to reading comprehension and pronunciation.However, transliterating between older and newer texts is a laborious task when performed manually.We introduce two related methods to help solve this transliteration problem automatically, given that there were not enough data to train an end-to-end deep learning model.One approach is implemented, end-to-end, using finite state transducers (FSTs).The other is a hybrid deep learning approach which approximately composes an FST with a recurrent neural network (RNN).We find that the hybrid approach outperforms the end-to-end FST by partitioning the original problem into one part that can be modelled by hand, using an FST, and into another part, which is easily solved by an RNN trained on the available data.

一种新颖的混合深度学习方法为有限的数据问题提供了最佳解决方案（这对保护夏威夷语言非常重要）。


In many real-world settings, a learning model must perform few-shot classification: learn to classify examples from unseen classes using only a few labeled examples per class.Additionally, to be safely deployed, it should have the ability to detect out-of-distribution inputs: examples that do not belong to any of the classes.While both few-shot classification and out-of-distribution detection are popular topics,their combination has not been studied.In this work, we propose tasks for out-of-distribution detection in the few-shot setting and establish benchmark datasets, based on four popular few-shot classification datasets.  Then, we propose two new methods for this task and investigate their performance.In sum, we establish baseline out-of-distribution detection results using standard metrics on new benchmark datasets and show improved results with our proposed methods.

我们定量地研究了在少样本下的分布外检测，用ProtoNet、MAML、ABML建立了基线结果，并对其进行了改进。

本文提出了两个新的置信度分数，它们更适合于少样本分类的分布外检测，并表明基于距离度量的方法提高了性能。


While modern  generative  models are able to synthesize high-fidelity, visually appealing images, successfully generating examples that are useful for recognition tasks remains an elusive goal.To this end, our key insight is that the examples should be synthesized to recover classifier decision boundaries that would be learned from a large amount of real examples.More concretely, we treat a classifier trained on synthetic examples as ''student'' and a classifier trained on real examples as ''teacher''.By introducing knowledge distillation into a meta-learning framework, we encourage the generative model to produce examples in a way that enables the student classifier to mimic the behavior of the teacher.To mitigate the potential gap between student and teacher classifiers, we further propose to distill the knowledge in a progressive manner, either by gradually strengthening the teacher or weakening the student.We demonstrate the use of our model-agnostic distillation approach to deal with data scarcity, significantly improving few-shot learning performance on miniImageNet and ImageNet1K benchmarks.

本文介绍了用于学习以识别任务为导向的生成模型的渐进式知识蒸馏法。 本文展示了由易到难的课程学习，以训练生成模型来改善少样本分类。


Deep neural networks provide state-of-the-art performance for many applications of interest.Unfortunately they are known to be vulnerable to adversarial examples, formed by applying small but malicious perturbations to the original inputs.Moreover, the perturbations can transfer across models: adversarial examples generated for a specific model will often mislead other unseen models.Consequently  the adversary can leverage it to attack against the deployed black-box systems. In this work, we demonstrate that the adversarial perturbation can be decomposed into two components: model-specific and data-dependent one, and it is the latter that mainly contributes to the transferability.Motivated by this understanding, we propose to craft adversarial examples by utilizing the noise reduced gradient (NRG) which approximates the data-dependent component.Experiments on various classification models trained on ImageNet demonstrates that the new approach enhances the transferability dramatically.We also find that low-capacity models have more powerful attack capability than high-capacity counterparts, under the condition that they have comparable test performance.  These insights give rise to a principled manner to construct adversarial examples with high success rates and could potentially provide us guidance for designing effective defense approaches against black-box attacks.

我们提出了一种新的方法，通过使用降噪梯度来增强对抗性例子的可转移性。

本文假设，对抗性扰动由特定模型和特定数据部分组成，而后者的放大最适合于对抗性攻击。

本文的重点是加强对抗性例子从一个模型到另一个模型的可转移性。


We present the iterative two-pass decomposition flow to accelerate existing convolutional neural networks (CNNs).  The proposed rank selection algorithm can effectively determine the proper ranks of the target convolutional layers for the low rank approximation.Our two-pass CP-decomposition helps prevent from the instability problem.The iterative flow makes the decomposition of the deeper networks systematic.The experiment results shows that VGG16 can be accelerated with a 6.2x measured speedup while the accuracy drop remains only 1.2%.

我们提出了迭代的两路CP分解流程，以有效加速现有的卷积神经网络（CNN）。

本文提出了一种新颖的CNN加速和压缩的工作流程，还提出了一种给定目标整体加速的方法来确定每层的目标等级。

本文讨论了为深度神经网络（DNN）中的过滤层学习低秩张量过滤操作的问题。


We introduce LiPopt, a polynomial optimization framework for computing increasingly tighter upper bound on the Lipschitz constant of neural networks.The underlying optimization problems boil down to either linear (LP) or semidefinite (SDP) programming.We show how to use the sparse connectivity of a network, to significantly reduce the complexity of computation.This is specially useful for convolutional as well as pruned neural networks.We conduct experiments on networks with random weights as well as networks trained on MNIST, showing that in the particular case of the $\ell_\infty$-Lipschitz constant, our approach yields superior estimates as compared to other baselines available in the literature.

基于LP的神经网络的Lipschitz常数的上界

作者研究了估算具有ELO激活函数的深度神经网络的Lipschitz常数问题，将其表述为一个多项式优化问题。


Although few-shot learning research has advanced rapidly with the help of meta-learning, its practical usefulness is still limited because most of the researches assumed that all meta-training and meta-testing examples came from a single domain.We propose a simple but effective way for few-shot classification in which a task distribution spans multiple domains including previously unseen ones during meta-training.The key idea is to build a pool of embedding models which have their own metric spaces and to learn to select the best one for a particular task through multi-domain meta-learning.This simplifies task-specific adaptation over a complex task distribution as a simple selection problem rather than modifying the model with a number of parameters at meta-testing time.Inspired by common multi-task learning techniques, we let all models in the pool share a base network and add a separate modulator to each model to refine the base network in its own way.This architecture allows the pool to maintain representational diversity and each model to have domain-invariant representation as well. Experiments show that our selection scheme outperforms other few-shot classification algorithms when target tasks could come from many different domains.They also reveal that aggregating outputs from all constituent models is effective for tasks from unseen domains showing the effectiveness of our framework.

我们通过建立多个模型，以集体的方式表示这种复杂的任务分布，并将特定任务的适应性简化为来自这些预训练模型的选择问题，来解决多领域的少样本分类。

本文通过建立一个嵌入模型池，在不大幅增加参数数量的情况下捕捉领域变量和特定领域的特征，来解决具有许多不同领域的少样本分类问题。


Still in 2019, many scanned documents come into businesses in non-digital format.Text to be extracted from real world documents is often nestled inside rich formatting, such as tabular structures or forms with fill-in-the-blank boxes or underlines whose ink often touches or even strikes through the ink of the text itself.Such ink artifacts can severely interfere with the performance of recognition algorithms or other downstream processing tasks.In this work, we propose DeepErase, a neural preprocessor to erase ink artifacts from text images.We devise a method to programmatically augment text images with real artifacts, and use them to train a segmentation network in an weakly supervised manner.In additional to high segmentation accuracy, we show that our cleansed images achieve a significant boost in downstream recognition accuracy by popular OCR software such as Tesseract 4.0.We test DeepErase on out-of-distribution datasets (NIST SDB) of scanned IRS tax return forms and achieve double-digit improvements in recognition accuracy over baseline for both printed and handwritten text.

基于神经的文档墨水人工制品（下划线、污点等）的去除，不使用人工标注的训练数据


Black-box adversarial attacks require a large number of attempts before finding successful adversarial examples that are visually indistinguishable from the original input.Current approaches relying on substitute model training, gradient estimation or genetic algorithms often require an excessive number of queries.Therefore, they are not suitable for real-world systems where the maximum query number is limited due to cost.We propose a query-efficient black-box attack which uses Bayesian optimisation in combination with Bayesian model selection to optimise over the adversarial perturbation and the optimal degree of search space dimension reduction.We demonstrate empirically that our method can achieve comparable success rates with 2-5 times fewer queries compared to previous state-of-the-art black-box attacks.

我们提出了一种查询效率高的黑箱攻击，它使用贝叶斯优化与贝叶斯模型选择相结合，对对抗性扰动和搜索空间降维的最佳程度进行优化。

作者提出利用贝叶斯优化与GP代理进行对抗性图像生成，通过利用加法结构和使用贝叶斯模型选择来确定最佳降维。


Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information.Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data:1) models must learn the complex intra-modal and cross-modal interactions for prediction and2) models must be robust to unexpected missing or noisy modalities during testing.In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels.We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors.Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction.Modality-specific generative factors are unique for each modality and contain the information required for generating data.Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets.Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance.Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning.

我们提出了一个学习因子化多模态表征的模型，它具有辨别性、生成性和可解释性。

本文提出了 "多模态因子化模型"，该模型将表征分解为共享的多模态判别因子和特定模态生成因子。


The successful application of flexible, general learning algorithms to real-world robotics applications is often limited by their poor data-efficiency.To address the challenge, domains with more than one dominant task of interest encourage the sharing of information across tasks to limit required experiment time.To this end, we investigate compositional inductive biases in the form of hierarchical policies as a mechanism for knowledge transfer across tasks in reinforcement learning (RL).We demonstrate that this type of hierarchy enables positive transfer while mitigating negative interference.Furthermore, we demonstrate the benefits of additional incentives to efficiently decompose task solutions.Our experiments show that these incentives are naturally given in multitask learning and can be easily introduced for single objectives.We design an RL algorithm that enables stable and fast learning of structured policies and the effective reuse of both behavior components and transition data across tasks in an off-policy setting.Finally, we evaluate our algorithm in simulated environments as well as physical robot experiments and  demonstrate substantial improvements in data data-efficiency over competitive baselines.

我们开发了一种分层的、行为者-批判者的算法，用于通过共享策略组件进行成分转移，并展示了组件的专业化和在多任务领域的相关直接收益，以及对单一任务的适应性。

不同学习技术的组合，用于获取结构和学习不对称的数据，用于训练HRL政策。

作者介绍了一种用于单任务和多任务强化学习的分层策略结构，并评估了该结构在复杂机器人任务上的实用性。


In this paper, we study the representational power of deep neural networks (DNN) that belong to the family of piecewise-linear (PWL) functions, based on PWL activation units such as rectifier or maxout.We investigate the complexity of such networks by studying the number of linear regions of the PWL function.Typically, a PWL function from a DNN can be seen as a large family of linear functions acting on millions of such regions.We directly build upon the work of MontÂ´ufar et al. (2014), MontÂ´ufar (2017), and Raghu et al. (2017) by refining the upper and lower bounds on the number of linear regions for rectified and maxout networks.In addition to achieving tighter bounds, we also develop a novel method to perform exact numeration or counting of the number of linear regions with a mixed-integer linear formulation that maps the input space to output.We use this new capability to visualize how the number of linear regions change while training DNNs.

我们根据经验统计了整流器网络的线性区域数量，并完善了上限和下限。 本文提出了计算ReLU网络中线性区域数量的改进界限。


Convolutional neural networks memorize part of their training data, which is why strategies such as data augmentation and drop-out are employed to mitigate over- fitting.This paper considers the related question of â€œmembership inferenceâ€, where the goal is to determine if an image was used during training.We con- sider membership tests over either ensembles of samples or individual samples.First, we show how to detect if a dataset was used to train a model, and in particular whether some validation images were used at train time.Then, we introduce a new approach to infer membership when a few of the top layers are not available or have been fine-tuned, and show that lower layers still carry information about the training samples.To support our findings, we conduct large-scale experiments on Imagenet and subsets of YFCC-100M with modern architectures such as VGG and Resnet.

我们通过训练集的convnet来分析记忆特性，并提出了几个我们可以提取训练集的一些信息的用例。

阐明了大型和深度ConvNets的泛化/记忆特性，并试图开发与识别训练过的ConvNet的输入是否真的被用于训练网络有关的程序。


While Generative Adversarial Networks (GANs) have empirically produced impressive results on learning complex real-world distributions, recent works have shown that they suffer from lack of diversity or mode collapse.The theoretical work of Arora et al. (2017a) suggests a dilemma about GANsâ€™ statistical properties: powerful discriminators cause overfitting, whereas weak discriminators cannot detect mode collapse.By contrast, we show in this paper that GANs can in principle learn distributions in Wasserstein distance (or KL-divergence in many cases) with polynomial sample complexity, if the discriminator class has strong distinguishing power against the particular generator class (instead of against all possible generators).For various generator classes such as mixture of Gaussians, exponential families, and invertible and injective neural networks generators, we design corresponding discriminators (which are often neural nets of specific architectures) such that the Integral Probability Metric (IPM) induced by the discriminators can provably approximate the Wasserstein distance and/or KL-divergence.This implies that if the training is successful, then the learned distribution is close to the true distribution in Wasserstein distance or KL divergence, and thus cannot drop modes.Our preliminary experiments show that on synthetic datasets the test IPM is well correlated with KL divergence or the Wasserstein distance, indicating that the lack of diversity in GANs may be caused by the sub-optimality in optimization instead of statistical inefficiency.

如果判别器类是紧凑的，并且对特定的生成器类有很强的判别能力，那么原则上GAN可以有效地学习分布。

提出了限制性近似的概念，并提供了一个与维度成多项式的样本复杂度约束，这对于研究GANs的多样性不足是很有用的。

分析了在一些温和的假设条件下，积分概率度量可以很好地近似于Wasserstein距离。


Understanding the optimization trajectory is critical to understand training of deep neural networks.We show how the hyperparameters of stochastic gradient descent influence the covariance of the gradients (K) and the Hessian of the training loss (H) along this trajectory.Based on a theoretical model, we predict that using a high learning rate or a small batch size in the early phase of training leads SGD to regions of the parameter space with (1) reduced spectral norm of K, and (2) improved conditioning of K and H. We show that the point on the trajectory after which these effects hold, which we refer to as the break-even point, is reached early during training.We demonstrate these effects empirically for a range of deep neural networks applied to multiple different tasks.Finally, we apply our analysis to networks with batch normalization (BN) layers and find that it is necessary to use a high learning rate to achieve loss smoothing effects attributed previously to BN alone.

在深度神经网络训练的早期阶段，存在一个 "收支平衡点"，它决定了整个优化轨迹的属性。

这项工作通过考虑批量大小和步长的超参数如何修改学习轨迹，分析了深度神经网络的优化。


Graph Convolution Network (GCN) has been recognized as one of the most effective graph models for semi-supervised learning, but it extracts merely the first-order or few-order neighborhood information through information propagation, which suffers performance drop-off for deeper structure.Existing approaches that deal with the higher-order neighbors tend to take advantage of adjacency matrix power.In this paper, we assume a seemly trivial condition that the higher-order neighborhood information may be similar to that of the first-order neighbors.Accordingly, we present an unsupervised approach to describe such similarities and learn the weight matrices of higher-order neighbors automatically through Lasso that minimizes the feature loss between the first-order and higher-order neighbors, based on which we formulate the new convolutional filter for GCN to learn the better node representations.Our model, called higher-order weighted GCN (HWGCN), has achieved the state-of-the-art results on a number of node classification tasks over Cora, Citeseer and Pubmed datasets.

我们提出HWGCN来混合不同顺序的相关邻域信息，以更好地学习节点表征。

作者提出了一个GCN的变体，即HWGCN，考虑了1步邻居以外的卷积，这与最先进的方法相媲美。


The performance of deep neural networks is often attributed to their automated, task-related feature construction.It remains an open question, though, why this leads to solutions with good generalization, even in cases where the number of parameters is larger than the number of samples.Back in the 90s, Hochreiter and Schmidhuber observed that flatness of the loss surface around a local minimum correlates with low generalization error.For several flatness measures, this correlation has been empirically validated.However, it has recently been shown that existing measures of flatness cannot theoretically be related to generalization: if a network uses ReLU activations, the network function can be reparameterized without changing its output in such a way that flatness is changed almost arbitrarily.This paper proposes a natural modification of existing flatness measures that results in invariance to reparameterization.The proposed measures imply a robustness of the network to changes in the input and the hidden layers.Connecting this feature robustness to generalization leads to a generalized definition of the representativeness of data.With this, the generalization error of a model trained on representative data can be bounded by its feature robustness which depends on our novel flatness measure.

我们引入了一种新的深度神经网络损失面局部最小值的平坦度测量方法，该方法对于逐层的重新参数化是不变的，我们将平坦度与特征鲁稳健性和泛化相联系。

作者提出了一个特征稳健性的概念，该概念在重新调整权重方面是不变的，并讨论了该概念与泛化的关系。

本文定义了一个特征稳健性的概念，并将其与函数的epsilon代表性相结合，描述了深度神经网络中最小值的平坦性和泛化之间的联系。


Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e.g.neurons.We apply and further develop this approach for gated recurrent architectures.Specifically, in addition to sparsification of individual weights and neurons, we propose to sparsify preactivations of gates and information flow in LSTM.It makes some gates and information flow components constant, speeds up forward pass and improves compression.Moreover, the resulting structure of gate sparsity is interpretable and depends on the task.

我们建议对LSTM中门的预激活和信息流进行稀疏化处理，使其成为常数，并提高神经元的稀疏程度。

本文提出了一种循环神经网络的稀疏化方法，通过消除预激活为零的神经元来获得紧凑的网络。


Improving the accuracy of numerical methods remains a central challenge in many disciplines and is especially important for nonlinear simulation problems.A representative example of such problems is fluid flow, which has been thoroughly studied to arrive at efficient simulations of complex flow phenomena.This paper presents a data-driven approach that learns to improve the accuracy of numerical solvers.The proposed method utilizes an advanced numerical scheme with a fine simulation resolution to acquire reference data.We, then, employ a neural network that infers a correction to move a coarse thus quickly obtainable result closer to the reference data.We provide insights into the targeted learning problem with different learning approaches: fully supervised learning methods with a naive and an optimized data acquisition as well as an unsupervised learning method with a differentiable Navier-Stokes solver.While our approach is very general and applicable to arbitrary partial differential equation models, we specifically highlight gains in accuracy for fluid flow simulations.

我们引入了一种神经网络方法来辅助偏微分方程求解器。

作者旨在通过在模拟参考数据上训练一个神经网络来提高数值求解器的精度，该神经网络可以纠正数值求解器。


A patientâ€™s health information is generally fragmented across silos.Though it is technically feasible to unite data for analysis in a manner that underpins a rapid learning healthcare system, privacy concerns and regulatory barriers limit data centralization.Machine learning can be conducted in a federated manner on patient datasets with the same set of variables, but separated across sites of care.But federated learning cannot handle the situation where different data types for a givenpatient are separated vertically across different organizations.We call methods that enable machine learning model training on data separated by two or more degrees â€œconfederated machine learning.â€We built and evaluated a confederated machinelearningmodel to stratify the risk of accidental falls among the elderly.

一种联合学习方法，从横向和纵向分离的医疗数据中训练模型 一种"联合"机器学习方法，可以跨学科学习水平和垂直分离的医疗数据。


Existing neural networks are vulnerable to "adversarial examples"---created by adding maliciously designed small perturbations in inputs to induce a misclassification by the networks.The most investigated defense strategy is adversarial training which augments training data with adversarial examples.However, applying single-step adversaries in adversarial training does not support the robustness of the networks, instead, they will even make the networks to be overfitted.In contrast to the single-step, multi-step training results in the state-of-the-art performance on MNIST and CIFAR10, yet it needs a massive amount of time.Therefore, we propose a method, Stochastic Quantized Activation (SQA) that solves overfitting problems in single-step adversarial training and fastly achieves the robustness comparable to the multi-step.SQA attenuates the adversarial effects by providing random selectivity to activation functions and allows the network to learn robustness with only single-step training.Throughout the experiment, our method demonstrates the state-of-the-art robustness against one of the strongest white-box attacks as PGD training, but with much less computational cost.Finally, we visualize the learning process of the network with SQA to handle strong adversaries, which is different from existing methods.

本文提出了随机量化激活法，解决了FGSM对抗性训练中的过拟合问题，并快速实现了与多步骤训练相媲美的稳健性。

本文提出了一个模型，通过在其中一个隐藏层的激活中引入随机扰动来改善对抗性训练。


Neural activity is highly variable in response to repeated stimuli.We used an open dataset, the Allen Brain Observatory, to quantify the distribution of responses to repeated natural movie presentations.A large fraction of responses are best fit by log-normal distributions or Gaussian mixtures with two components.These distributions are similar to those from units in deep neural networks with dropout.Using a separate set of electrophysiological recordings, we constructed a population coupling model as a control for state-dependent activity fluctuations and found that the model residuals also show non-Gaussian distributions.We then analyzed responses across trials from multiple sections of different movie clips and observed that the noise in cortex aligns better with in-clip versus out-of-clip stimulus variations.We argue that noise is useful for generalization when it moves along representations of different exemplars in-class, similar to the structure of cortical noise.

我们研究了大脑中的噪声结构，发现它可能通过沿着类内刺激变化移动表征来帮助泛化。


Unsupervised domain adaptation has received significant attention in recent years.Most of existing works tackle the closed-set scenario, assuming that the source and target domains share the exactly same categories.In practice, nevertheless, a target domain often contains samples of classes unseen in source domain (i.e., unknown class).The extension of domain adaptation from closed-set to such open-set situation is not trivial since the target samples in unknown class are not expected to align with the source.In this paper, we address this problem by augmenting the state-of-the-art domain adaptation technique, Self-Ensembling, with category-agnostic clusters in target domain.Specifically, we present Self-Ensembling with Category-agnostic Clusters (SE-CC) --- a novel architecture that steers domain adaptation with the additional guidance of category-agnostic clusters that are specific to target domain.These clustering information provides domain-specific visual cues, facilitating the generalization of Self-Ensembling for both closed-set and open-set scenarios.Technically, clustering is firstly performed over all the unlabeled target samples to obtain the category-agnostic clusters, which reveal the underlying data space structure peculiar to target domain.A clustering branch is capitalized on to ensure that the learnt representation preserves such underlying structure by matching the estimated assignment distribution over clusters to the inherent cluster distribution for each target sample.Furthermore, SE-CC enhances the learnt representation with mutual information maximization.Extensive experiments are conducted on Office and VisDA datasets for both open-set and closed-set domain adaptation, and superior results are reported when comparing to the state-of-the-art approaches.

我们提出了一种新的设计，即具有类别无关的集群的自我组装，用于封闭式和开放式领域的适应。

一种开放集领域适应的新方法，源领域类别包含在目标领域类别中，以便过滤掉离群的类别，实现在共享类别中的适应。


We present Spectral Inference Networks, a framework for learning eigenfunctions of linear operators by stochastic optimization.Spectral Inference Networks generalize Slow Feature Analysis to generic symmetric operators, and are closely related to Variational Monte Carlo methods from computational physics.As such, they can be a powerful tool for unsupervised representation learning from video or graph-structured data.We cast training Spectral Inference Networks as a bilevel optimization problem, which allows for online learning of multiple eigenfunctions.We show results of training Spectral Inference Networks on problems in quantum mechanics and feature learning for videos on synthetic datasets.Our results demonstrate that Spectral Inference Networks accurately recover eigenfunctions of linear operators and can discover interpretable representations from video in a fully unsupervised manner.

我们展示了如何用深度学习来学习线性算子的谱系分解，并将其用于没有生成模型的无监督学习。

作者提出使用深度学习框架来解决最大特征向量的计算问题。

本文提出了一个通过随机过程学习特征函数的框架，并建议通过使用两阶段的随机优化过程来近似计算大规模背景下的特征函数，以应对这一挑战。


The Tensor-Train factorization (TTF) is an efficient way to compress large weight matrices of fully-connected layers and recurrent layers in recurrent neural networks (RNNs).However, high Tensor-Train ranks for all the core tensors of parameters need to be element-wise fixed, which results in an unnecessary redundancy of model parameters.This work applies Riemannian stochastic gradient descent (RSGD) to train core tensors of parameters in the Riemannian Manifold before finding vectors of lower Tensor-Train ranks for parameters.The paper first presents the RSGD algorithm with a convergence analysis and then tests it on more advanced Tensor-Train RNNs such as bi-directional GRU/LSTM and Encoder-Decoder RNNs with a Tensor-Train attention model.The experiments on digit recognition and machine translation tasks suggest the effectiveness of the RSGD algorithm for Tensor-Train RNNs.

应用黎曼SGD（RSGD）算法来训练张量训练RNN，以进一步减少模型参数。

本文提出在深度网络中使用黎曼随机梯度算法进行低秩张量训练学习。

提出了一种基于黎曼优化和等级适应的Tensor Train分解的神经网络优化算法，并设计了一种双向的TT LSTM架构。


In this paper, we consider the problem of learning control policies that optimize areward function while satisfying constraints due to considerations of safety, fairness, or other costs.We propose a new algorithm - Projection Based ConstrainedPolicy Optimization (PCPO), an iterative method for optimizing policies in a two-step process - the first step performs an unconstrained update while the secondstep reconciles the constraint violation by projection the policy back onto the constraint set.We theoretically analyze PCPO and provide a lower bound on rewardimprovement, as well as an upper bound on constraint violation for each policy update.We further characterize the convergence of PCPO with projection basedon two different metrics - L2 norm and Kullback-Leibler divergence.Our empirical results over several control tasks demonstrate that our algorithm achievessuperior performance, averaging more than 3.5 times less constraint violation andaround 15% higher reward compared to state-of-the-art methods.

我们提出了一种学习满足约束条件的策略的新算法，并在有约束条件的强化学习背景下提供理论分析和实证证明。

本文介绍了一种使用两步优化过程的约束性策略优化算法，不满足约束的策略可以被投射回约束集中。


Deep networks face challenges of ensuring their robustness against inputs that cannot be effectively represented by information learned from training data.We attribute this vulnerability to the limitations inherent to activation-based representation.To complement the learned information from activation-based representation, we propose utilizing a gradient-based representation that explicitly focuses on missing information.In addition, we propose a directional constraint on the gradients as an objective during training to improve the characterization of missing information.To validate the effectiveness of the proposed approach, we compare the anomaly detection performance of gradient-based and activation-based representations.We show that the gradient-based representation outperforms the activation-based representation by 0.093 in CIFAR-10 and 0.361 in CURE-TSR datasets in terms of AUROC averaged over all classes.Also, we propose an anomaly detection algorithm that uses the gradient-based representation, denoted as GradCon, and validate its performance on three benchmarking datasets.The proposed method outperforms the majority of the state-of-the-art algorithms in CIFAR-10, MNIST, and fMNIST datasets with an average AUROC of 0.664, 0.973, and 0.934, respectively.

我们提出了一种基于梯度的表征，用于表征深度网络没有学到的信息。 作者提出创建基于梯度的权重表征，以补充深度网络训练数据集中缺失的信息。


Medical images may contain various types of artifacts with different patterns and mixtures, which depend on many factors such as scan setting, machine condition, patientsâ€™ characteristics, surrounding environment, etc.However, existing deep learning based artifact reduction methods are restricted by their training set with specific predetermined artifact type and pattern.As such, they have limited clinical adoption.In this paper, we introduce a â€œZero-Shotâ€ medical image Artifact Reduction (ZSAR) framework, which leverages the power of deep learning but without using general pre-trained networks or any clean image reference.Specifically, we utilize the low internal visual entropy of an image and train a light-weight image-specific artifact reduction network to reduce artifacts in an image at test-time.We use Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) as vehicles to show that ZSAR can reduce artifacts better than state-of-the-art both qualitatively and quantitatively, while using shorter execution time.To the best of our knowledge, this is the first deep learning framework that reduces artifacts in medical images without using a priori training set.

我们介绍了一个零样本医学图像伪影减少框架，它利用了深度学习的力量，但没有使用一般的预训练网络或任何干净的图像参考。


Attribution methods provide insights into the decision-making of machine learning models like artificial neural networks.For a given input sample, they assign a relevance score to each individual input variable, such as the pixels of an image.In this work we adapt the information bottleneck concept for attribution.By adding noise to intermediate feature maps we restrict the flow of information and can quantify (in bits) how much information image regions provide.We compare our method against ten baselines using three different metrics on VGG-16 and ResNet-50, and find that our methods outperform all baselines in five out of six settings.The methodâ€™s information-theoretic foundation provides an absolute frame of reference for attribution values (bits) and a guarantee that regions scored close to zero are not necessary for the network's decision.

我们将信息瓶颈的概念应用于归属问题。

本文提出了一种新的基于扰动的方法，用于计算基于深度神经网络的图像分类器的归属/显著图，方法是向网络的早期层注入精心制作的噪声。


Recurrent Neural Networks (RNNs) are used in state-of-the-art models in domains such as speech recognition, machine translation, and language modelling.Sparsity is a technique to reduce compute and memory requirements of deep learning models.Sparse RNNs are easier to deploy on devices and high-end server processors.Even though sparse operations need less compute and memory relative to their dense counterparts, the speed-up observed by using sparse operations is less than expected on different hardware platforms.In order to address this issue, we investigate two different approaches to induce block sparsity in RNNs: pruning blocks of weights in a layer and using group lasso regularization with pruning to create blocks of weights with zeros.Using these techniques, we can create block-sparse RNNs with sparsity ranging from 80% to 90% with a small loss in accuracy.This technique allows us to reduce the model size by roughly 10x.Additionally, we can prune a larger dense network to recover this loss in accuracy while maintaining high block sparsity and reducing the overall parameter count.Our technique works with a variety of block sizes up to 32x32.Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.

我们展示了RNN可以被修剪以诱导块状稀疏性，从而提高现有硬件上稀疏操作的速度。

作者提出了一种压缩RNN的块状稀疏性修剪方法，使用组LASSO来促进稀疏性并进行修剪，但在修剪和修剪权重方面有一个非常专业的安排。


Value iteration networks are an approximation of the value iteration (VI) algorithm implemented with convolutional neural networks to make VI fully differentiable.In this work, we study these networks in the context of robot motion planning, with a focus on applications to planetary rovers.The key challenging task in learning-based motion planning is to learn a transformation from terrain observations to a suitable navigation reward function.In order to deal with complex terrain observations and policy learning, we propose a value iteration recurrence, referred to as the soft value iteration network (SVIN).SVIN is designed to produce more effective training gradients through the value iteration network.It relies on a soft policy model, where the policy is represented with a probability distribution over all possible actions, rather than a deterministic policy that returns only the best action.We demonstrate the effectiveness of the proposed method in robot motion planning scenarios.In particular, we study the application of SVIN to very challenging problems in planetary rover navigation and present early training results on data gathered by the  Curiosity rover that is currently operating on Mars.

我们提出了对价值迭代网络的改进，并应用于行星探测器的路径规划。 本文利用价值迭代模块学习基于专家轨迹的奖励函数，以使规划步骤具有可分性


Transformer networks have lead to important progress in language modeling and machine translation.These models include two consecutive modules, a feed-forward layer and a self-attention layer.The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers.Building upon this intuition, we propose a new model that solely consists of attention layers.More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer.Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer.Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.

一种新型的注意层，结合了Transformer网络的自我注意和前馈子层。

本文提出了对Transformer模型的修改，将对 "持久性 "记忆向量的注意纳入自我注意层，使其性能与现有模型相当，同时使用较少的参数。


This work views neural networks as data generating systems and applies anomalous pattern detection techniques on that data in order to detect when a network is processing a group of anomalous inputs.  Detecting anomalies is a critical component for multiple machine learning problems including detecting the presence of adversarial noise added to inputs.More broadly, this work is a step towards giving neural networks the ability to detect groups of out-of-distribution samples.  This work introduces ``Subset Scanning methods from the anomalous pattern detection domain to the task of detecting anomalous inputs to neural networks.  Subset Scanning allows us to answer the question: "``Which subset of inputs have larger-than-expected activations at which subset of nodes?"  Framing the adversarial detection problem this way allows us to identify systematic patterns in the activation space that span multiple adversarially noised images.  Such images are ``"weird together".  Leveraging this common anomalous pattern, we show increased detection power as the proportion of noised images increases in a test set.   Detection power and accuracy results are provided for targeted adversarial noise added to CIFAR-10 images on a 20-layer ResNet using the Basic Iterative Method attack.

我们有效地找到一个图像子集，该子集的一些节点的激活度高于预期。  这些图像在作为一个群体来看时，显得更加异常，更容易被发现。

该论文提出了一种基于 "子集扫描 "的方案来检测异常输入的存在，以检测深度学习网络中的异常激活。


Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks.In this work, we conduct a thorough investigation of stable recurrent models.Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent.Empirically, we demonstrate stable recurrent models often perform as well as their unstable counterparts on benchmark sequence tasks.Taken together, these findings shed light on the effective power of recurrent networks and suggest much of sequence learning happens, or can be made to happen, in the stable regime.Moreover, our results help to explain why in many cases practitioners succeed in replacing recurrent models by feed-forward models.

稳定的循环模型可以用前馈网络来近似，根据经验，在基准任务上的表现与不稳定的模型一样好。

研究RNN的稳定性，并调查频谱归一化对序列预测的影响。


Weight-sharing plays a significant role in the success of many deep neural networks, by increasing memory efficiency and incorporating useful inductive priors about the problem into the network.But understanding how weight-sharing can be used effectively in general is a topic that has not been studied extensively.Chen et al. (2015) proposed HashedNets, which augments a multi-layer perceptron with a hash table, as a method for neural network compression.We generalize this method into a framework (ArbNets) that allows for efficient arbitrary weight-sharing, and use it to study the role of weight-sharing in neural networks.We show that common neural networks can be expressed as ArbNets with different hash functions.We also present two novel hash functions, the Dirichlet hash and the Neighborhood hash, and use them to demonstrate experimentally that balanced and deterministic weight-sharing helps with the performance of a neural network.

研究了使用哈希函数的神经网络中权重共享的作用，发现平衡和确定的哈希函数有助于网络性能。

提出ArbNets，通过将分权函数定义为哈希函数，以更系统的方式研究分权。


We introduce Neural Markov Logic Networks (NMLNs), a statistical relational learning system that borrows ideas from Markov logic.Like Markov Logic Networks (MLNs), NMLNs are an exponential-family model for modelling distributions over possible worlds, but unlike MLNs, they do not rely on explicitly specified first-order logic rules.Instead, NMLNs learn an implicit representation of such rules as a neural network that acts as a potential function on fragments of the relational structure.Interestingly, any MLN can be represented as an NMLN.Similarly to recently proposed Neural theorem provers (NTPs) (Rocktaschel at al. 2017), NMLNs can exploit embeddings of constants but, unlike NTPs, NMLNs work well also in their absence.This is extremely important for predicting in settings other than the transductive one.We showcase the potential of NMLNs on knowledge-base completion tasks and on generation of molecular (graph) data.

我们介绍了一个统计关系学习系统，它借用了马尔可夫逻辑的思想，但学习的是作为神经网络的规则隐含表征。

本文对马尔科夫逻辑网络进行了扩展，取消了其对预定的一阶逻辑规则的依赖，以在知识库完成任务中对更多的领域进行建模。


Using variational Bayes neural networks, we develop an algorithm capable of accumulating knowledge into a prior from multiple different tasks.This results in a rich prior capable of few-shot learning on new tasks.The posterior can go beyond the mean field approximation and yields good uncertainty on the performed experiments.Analysis on toy tasks show that it can learn from significantly different tasks while finding similarities among them.Experiments on Mini-Imagenet reach state of the art with 74.5% accuracy on 5 shot learning.Finally, we provide two new benchmarks, each showing a failure mode of existing meta learning algorithms such as MAML and prototypical Networks.

一种在多个任务中学习神经网络上的表达性先验的可扩展方法。

本文提出了一种训练多任务迁移学习的概率模型的方法，通过为每个任务引入一个潜在变量来捕捉任务实例的共性。

该工作提出了一种元学习的变分方法，采用了与特定任务数据集相对应的潜变量。

旨在为多个任务学习神经网络的先验。


Sequential data often originates from diverse environments.Across them exist both shared regularities and environment specifics.To learn robust cross-environment descriptions of sequences we introduce disentangled state space models (DSSM).In the latent space of DSSM environment-invariant state dynamics is explicitly disentangled from environment-specific information governing that dynamics.We empirically show that such separation enables robust prediction, sequence manipulation and environment characterization.We also propose an unsupervised VAE-based training procedure to learn DSSM as Bayesian filters.In our experiments, we demonstrate state-of-the-art performance in controlled generation and prediction of bouncing ball video sequences across varying gravitational influences.

分散的状态空间模型 本文提出了一个生成的状态空间模型，使用一个全局潜变量E来捕捉环境的特定信息。


In this work, we approach one-shot and few-shot learning problems as methods for finding good prototypes for each class, where these prototypes are generalizable to new data samples and classes.We propose a metric learner that learns a Bregman divergence by learning its underlying convex function.Bregman divergences are a good candidate for this framework given they are the only class of divergences with the property that the best representative of a set of points is given by its mean.We propose a flexible extension to prototypical networks to enable joint learning of the embedding and the divergence, while preserving computational efficiency.Our preliminary results are comparable with the prior work on the Omniglot and Mini-ImageNet datasets, two standard benchmarks for one-shot and few-shot learning.We argue that our model can be used for other tasks that involve metric learning or tasks that require approximate convexity such as structured prediction and data completion.

布雷格曼发散学习，用于少样本学习。


Motivated by the flexibility of biological neural networks whose connectivity structure changes significantly during their lifetime,we introduce the Unrestricted Recursive Network (URN) and demonstrate that it can exhibit similar flexibility during training via gradient descent.We show empirically that many of the different neural network structures commonly used in practice today (including fully connected, locally connected and residual networks of differ-ent depths and widths) can emerge dynamically from the same URN.These different structures can be derived using gradient descent on a single general loss function where the structure of the data and the relative strengths of various regulator terms determine the structure of the emergent network.We show that this loss function and the regulators arise naturally when considering the symmetries of the network as well as the geometric properties of the input data.

我们引入了一个网络框架，它可以在训练过程中修改其结构，并表明它可以收敛到各种ML网络原型，如MLP和LCN。


We present CROSSGRAD , a method to use multi-domain training data to learn a classifier that generalizes to new domains.CROSSGRAD does not need an adaptation phase via labeled or unlabeled data, or domain features in the new domain.Most existing domain adaptation methods attempt to erase domain signals using techniques like domain adversarial training.In contrast, CROSSGRAD is free to use domain signals for predicting labels, if it can prevent overfitting on training domains.We conceptualize the task in a Bayesian setting, in which a sampling step is implemented as data augmentation, based on domain-guided perturbations of input instances.CROSSGRAD jointly trains a label and a domain classifier on examples perturbed by loss gradients of each otherâ€™s objectives.This enables us to directly perturb inputs, without separating and re-mixing domain signals while making various distributional assumptions.Empirical evaluation on three different applications where this setting is natural establishes that (1) domain-guided perturbation provides consistently better generalization to unseen domains, compared to generic instance perturbation methods, and (2) data augmentation is a more stable and accurate method than domain adversarial training.

领域指导下的数据增强提供了一种稳健而稳定的领域泛化方法

本文提出了一种通过依赖领域的数据增强的领域泛化方法

作者介绍了CrossGrad方法，该方法同时训练一个标签分类任务和一个领域分类任务。


We present sketch-rnn, a recurrent neural network able to construct stroke-based drawings of common objects.The model is trained on a dataset of human-drawn images representing many different classes.We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.

我们研究了替代传统像素图像建模的方法，并提出了一个矢量图像的生成模型。 本文介绍了一种用于生成草图的神经网络架构，其灵感来自于变分自动编码器。


Wilson et al. (2017) showed that, when the stepsize schedule is properly designed, stochastic gradient generalizes better than ADAM (Kingma & Ba, 2014).In light of recent work on hypergradient methods (Baydin et al., 2018), we revisit these claims to see if such methods close the gap between the most popular optimizers.As a byproduct, we analyze the true benefit of these hypergradient methods compared to more classical schedules, such as the fixed decay of Wilson et al. (2017).In particular, we observe they are of marginal help since their performance varies significantly when tuning their hyperparameters.Finally, as robustness is a critical quality of an optimizer, we provide a sensitivity analysis of these gradient based optimizers to assess how challenging their tuning is.

我们提供了一项研究，试图看看最近的在线学习率适应如何扩展Wilson等人2018年关于自适应梯度方法的结论，同时还进行了比较和敏感性分析。

报告了几个与步骤调整有关的方法的测试结果，包括vanilla SGD、带有Neserov动量的SGD和ADAM，并比较了这些方法与超梯度和没有超梯度的情况。


Despite an ever growing literature on reinforcement learning algorithms and applications, much less is known about their statistical inference.In this paper, we investigate the large-sample behaviors of the Q-value estimates with closed-form characterizations of the asymptotic variances.This allows us to efficiently construct confidence regions for Q-value and optimal value functions, and to develop policies to minimize their estimation errors.This also leads to a policy exploration strategy that relies on estimating the relative discrepancies among the Q estimates.Numerical experiments show superior performances of our exploration strategy than other benchmark approaches.

我们研究了Q值估计的大样本行为，并提出了一个有效的探索策略，该策略依赖于对Q值估计之间的相对差异的估计。

本文提出了一种基于Q值的渐进分析及其向中心极限分布收敛的强化学习的纯探索算法，其表现优于基准探索算法。


We perform completely unsupervised one-sided image to image translation between a source domain $X$ and a target domain $Y$ such that we preserve relevant underlying shared semantics (e.g., class, size, shape, etc). In particular, we are interested in a more difficult case than those typically addressed in the literature, where the source and target are ``far" enough that reconstruction-style or pixel-wise approaches fail.We argue that transferring (i.e., \emph{translating}) said relevant information should involve both discarding source domain-specific information while incorporate target domain-specific information, the latter of which we model with a noisy prior distribution. In order to avoid the degenerate case where the generated samples are only explained by the prior distribution, we propose to minimize an estimate of the mutual information between the generated sample and the sample from the prior distribution.We discover that the architectural choices are an important factor to consider in order to preserve the shared semantic between $X$ and $Y$. We show state of the art results on the MNIST to SVHN task for unsupervised image to image translation.

我们训练一个图像转换网络，将源图像和先验分布的样本作为输入，生成目标分布的样本。

本文对无监督转换的问题进行了形式化，并提出了一个增强的GAN框架，该框架使用互相的信息来避免退化的情况。

本文提出了无监督的一对多图像转换问题，并通过最小化相互信息量来解决这个问题。


Identifying salient points in images is a crucial component for visual odometry, Structure-from-Motion or SLAM algorithms.Recently, several learned keypoint methods have demonstrated compelling performance on challenging benchmarks.  However, generating consistent and accurate training data for interest-point detection in natural images still remains challenging, especially for human annotators.We introduce IO-Net (i.e. InlierOutlierNet), a novel proxy task for the self-supervision of keypoint detection, description and matching.By making the sampling of inlier-outlier sets from point-pair correspondences fully differentiable within the keypoint learning framework, we show that are able to simultaneously self-supervise keypoint description and improve keypoint matching.Second, we introduce KeyPointNet, a keypoint-network architecture that is especially amenable to robust keypoint detection and description.We design the network to allow local keypoint aggregation to avoid artifacts due to spatial discretizations commonly used for this task, and we improve fine-grained keypoint descriptor performance by taking advantage of efficient sub-pixel convolutions to upsample the descriptor feature-maps to a higher operating resolution.Through extensive experiments and ablative analysis, we show that the proposed self-supervised keypoint learning method greatly improves the quality of feature matching and homography estimation on challenging benchmarks over the state-of-the-art.

学习从代理任务中提取可区分的关键点，剔除离群点。

本文致力于使用神经引导的RANSAC作为额外的辅助损失提供者来改进描述符插值，对局部特征进行自我监督学习。