Mixed precision training (MPT) is becoming a practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs.MPT is typically used in combination with a technique called loss scaling, that works by scaling up the loss value up before the start of backpropagation in order to minimize the impact of numerical underflow on training.Unfortunately, existing methods make this loss scale value a hyperparameter that needs to be tuned per-model, and a single scale cannot be adapted to different layers at different training stages.We introduce a loss scaling-based training method called adaptive loss scaling that makes MPT easier and more practical to use, by removing the need to tune a model-specific loss scale hyperparameter.We achieve this by introducing layer-wise loss scale values which are automatically computed during training to deal with underflow more effectively than existing methods.We present experimental results on a variety of networks and tasks that show our approach can shorten the time to convergence and improve accuracy, compared with using the existing state-of-the-art MPT and single-precision floating point.

我们设计了自适应损失缩放来改善混合精度训练，超过了最先进的结果。

在混合精度训练的反向传播过程中，提出了一种自适应损失缩放方法，该方法自动决定缩放率以减少下溢。

作者提出了一种在FP16精度下训练模型的方法，该方法采用了一种更精细的方式来同时自动减少每层的下溢。


Many real-world problems, e.g. object detection, have outputs that are naturally expressed as sets of entities.This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors.We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks.Specifically, in our formulation we incorporate the permutation as unobservable variable and estimate its distribution during the learning process using alternating optimization.We demonstrate the validity of this new formulation on two relevant vision problems: object detection, for which our formulation outperforms state-of-the-art detectors such as Faster R-CNN and YOLO, and a complex CAPTCHA test, where we observe that, surprisingly, our set based network acquired the ability of mimicking arithmetics without any rules being coded.

我们提出了一种新的方法，利用前馈深度神经网络来学习预测具有未知排列组合和基数的集合。

基于深度网络的集合预测问题，学习不可观察的排列变数分布的表述。


Foveation is an important part of human vision, and a number of deep networks have also used foveation.However, there have been few systematic comparisons between foveating and non-foveating deep networks, and between different variable-resolution downsampling methods.Here we define several such methods, and compare their performance on ImageNet recognition with a Densenet-121 network.The best variable-resolution method slightly outperforms uniform downsampling.Thus in our experiments, foveation does not substantially help or hinder object recognition in deep networks.

我们比较了统一降采样的图像和三种不同凹陷方案的物体识别性能。


We explore the concept of co-design in the context of neural network verification.Specifically, we aim to train deep neural networks that not only are robust to adversarial perturbations but also whose robustness can be verified more easily.To this end, we identify two properties of network models - weight sparsity and so-called ReLU stability - that turn out to significantly impact the complexity of the corresponding verification task.We demonstrate that improving weight sparsity alone already enables us to turn computationally intractable verification problems into tractable ones.Then, improving ReLU stability leads to an additional 4-13x speedup in verification times.An important feature of our methodology is its "universality," in the sense that it can be used with a broad range of training procedures and verification approaches.

我们开发了训练深度神经模型的方法，这些模型对对抗性扰动具有强化，而且其强化性明显更容易验证。

本文介绍了几种对普通ReLU网络进行正则化的方法，以使对抗性强化、可证明的对抗性强化和验证速度达到最佳。

本文提出了训练稳定的神经网络的方法，可以更快地验证，使用修剪方法来鼓励权重的稀疏性和正则化来鼓励ReLU稳定性。


Batch Normalization (BatchNorm) has shown to be effective for improving and accelerating the training of deep neural networks.However, recently it has been shown that it is also vulnerable to adversarial perturbations.In this work, we aim to investigate the cause of adversarial vulnerability of the BatchNorm.We hypothesize that the use of different normalization statistics during training and inference (mini-batch statistics for training and moving average of these values at inference) is the main cause of this adversarial vulnerability in the BatchNorm layer.We empirically proved this by experiments on various neural network architectures and datasets.Furthermore, we introduce Robust Normalization (RobustNorm) and experimentally show that it is not only resilient to adversarial perturbation but also inherit the benefits of BatchNorm.

调查BatchNorm如何导致对抗脆弱性以及如何避免它。

本文讨论了BatchNorm中对对抗性扰动的脆弱性，并提出了一个名为RobustNorm的替代方案，使用最小-最大重定比例代替正规化。

本文研究了BatchNorm的脆弱性背后的原因，并提出了Robust Normalization，一种在各种攻击方法下都能取得明显效果的正规化方法。


Electronic Health Records (EHR) comprise of longitudinal clinical observations portrayed with sparsity, irregularity, and high-dimensionality which become the major obstacles in drawing reliable downstream outcome.Despite greatly numbers of imputation methods are being proposed to tackle these issues, most of the existing methods ignore correlated features or temporal dynamics and entirely put aside the uncertainty.In particular, since the missing values estimates have the risk of being imprecise, it motivates us to pay attention to reliable and less certain information differently.In this work, we propose a novel variational-recurrent imputation network (V-RIN), which unified imputation and prediction network, by taking into account the correlated features, temporal dynamics, and further utilizing the uncertainty to alleviate the risk of biased missing values estimates.Specifically, we leverage the deep generative model to estimate the missing values based on the distribution among variables and a recurrent imputation network to exploit the temporal relations in conjunction with utilization of the uncertainty.We validated the effectiveness of our proposed model with publicly available real-world EHR dataset, PhysioNet Challenge 2012, and compared the results with other state-of-the-art competing methods in the literature.

我们的变异-循环网络（V-RIN）考虑到了相关的特征、时间动态，并进一步利用不确定性来减轻有误差的缺失值估计的风险。

一个缺失数据填补网络，将相关性、时间关系和数据的不确定性纳入EHRs的数据稀少问题，在死亡率分类任务中产生了更高的AUC。

该论文提出了一种结合VAE和不确定性意识的GRU的方法，用于连续缺失数据的填补和结果预测。


Despite the state-of-the-art accuracy of Deep Neural Networks (DNN) in various classification problems, their deployment onto resource constrained edge computing devices remains challenging due to their large size and complexity.Several recent studies have reported remarkable results in reducing this complexity through quantization of DNN models.However, these studies usually do not consider the changes in the loss function when performing quantization, nor do they take the different importances of DNN model parameters to the accuracy into account.We address these issues in this paper by proposing a new method, called adaptive quantization, which simplifies a trained DNN model by finding a unique, optimal precision for each network parameter such that the increase in loss is minimized.The optimization problem at the core of this method iteratively uses the loss function gradient to determine an error margin for each parameter and assigns it a precision accordingly.Since this problem uses linear functions, it is computationally cheap and, as we will show, has a closed-form approximate solution.Experiments on MNIST, CIFAR, and SVHN datasets showed that the proposed method can achieve near or better than state-of-the-art reduction in model size with similar error rates.Furthermore, it can achieve compressions close to floating-point model compression methods without loss of accuracy.

基于理论分析而非启发式方法的神经网络定点量化的自适应方法。

提出了一种量化神经网络的方法，允许根据权重的重要性对其进行不同精度的量化，同时考虑到了损失。

本文提出了一种将神经网络的权重量化的技术，其位深/精度在每个参数的基础上变化。


We study the problem of learning permutation invariant representations that can capture containment relations.We propose training a model on a novel task: predicting the size of the symmetric difference between pairs of multisets, sets which may contain multiple copies of the same object.With motivation from fuzzy set theory, we formulate both multiset representations and how to predict symmetric difference sizes given these representations.We model multiset elements as vectors on the standard simplex and multisets as the summations of such vectors, and we predict symmetric difference as the l1-distance between multiset representations.We demonstrate that our representations more effectively predict the sizes of symmetric differences than DeepSets-based approaches with unconstrained object representations.Furthermore, we demonstrate that the model learns meaningful representations, mapping objects of different classes to different standard basis vectors.

基于模糊集合理论，我们提出了一个模型，该模型只给定多集对之间的对称性差异的大小，学习这种多集及其元素的表示。

本文提出了一个集合学习的新任务，即预测多集合之间对称差的大小，并给出了一种基于模糊集合理论的解决方法。


It is important to collect credible training samples $(x,y)$ for building data-intensive learning systems (e.g., a deep learning system).In the literature, there is a line of studies on eliciting distributional information from self-interested agents who hold a relevant information.  Asking people to report complex distribution $p(x)$, though theoretically viable, is challenging in practice.This is primarily due to the heavy cognitive loads required for human agents to reason and report this high dimensional information.Consider the example where we are interested in building an image classifier via first collecting a certain category of  high-dimensional image data.While classical elicitation results apply to eliciting a complex and generative (and continuous) distribution $p(x)$ for this image data, we are interested in eliciting samples $x_i \sim p(x)$ from agents.This paper introduces a deep learning aided method to incentivize credible sample contributions from selfish and rational agents.The challenge to do so is to design an incentive-compatible score function to score each reported sample to induce truthful reports, instead of an arbitrary or even adversarial one.We show that with accurate estimation of a certain $f$-divergence function we are able to achieve approximate incentive compatibility in eliciting truthful samples.We then present an efficient estimator with theoretical guarantee via studying the variational forms of $f$-divergence function.Our work complements the literature of information elicitation via introducing the problem of \emph{sample elicitation}.  We also show a connection between this sample elicitation problem and $f$-GAN, and how this connection can help reconstruct an estimator of the distribution based on collected samples.

本文提出了一种深度学习辅助的方法，从自利的代理人那里获得可信的样本。

作者针对从代理人那里引出复杂分布的可信样本的问题提出了一个样本引出框架，提出深度神经框架可以在这个框架中应用，并将样本引出和f-GAN联系起来。

本文研究了样本诱导问题，提出了一种深度学习方法，该方法依赖于f-分歧的双重表达，该表达在一组函数t上写为最大值。


The celebrated Sequence to Sequence learning (Seq2Seq) technique and its numerous variants achieve excellent performance on many tasks.However, many machine learning tasks have inputs naturally represented as graphs; existing Seq2Seq models face a significant challenge in achieving accurate conversion from graph form to the appropriate sequence.To address this challenge, we introduce a general end-to-end graph-to-sequence neural encoder-decoder architecture that maps an input graph to a sequence of vectors and uses an attention-based LSTM method to decode the target sequence from these vectors.Our method first generates the node and graph embeddings using an improved graph-based neural network with a novel aggregation strategy to incorporate edge direction information in the node embeddings.We further introduce an attention mechanism that aligns node embeddings and the decoding sequence to better cope with large graphs.Experimental results on bAbI, Shortest Path, and Natural Language Generation tasks demonstrate that our model achieves state-of-the-art performance and significantly outperforms existing graph neural networks, Seq2Seq, and Tree2Seq models; using the proposed bi-directional node embedding aggregation strategy, the model can converge rapidly to the optimal performance.

以注意力为基础的神经网络的图形到序列学习

一个Graph2seq架构，将混合了GGNN和GCN组件的图形编码器与注意力序列编码器结合在一起，并显示出比基线有所改进。

这项工作提出了一个端到端的图形编码器到序列解码器模型，中间有一个关注机制。


We address the problem of learning to discover 3D parts for objects in unseen categories.Being able to learn the geometry prior of parts and transfer this prior to unseen categories pose fundamental challenges on data-driven shape segmentation approaches.Formulated as a contextual bandit problem, we propose a learning-based iterative grouping framework which learns a grouping policy to progressively merge small part proposals into bigger ones in a bottom-up fashion.At the core of our approach is to restrict the local context for extracting part-level features, which encourages the generalizability to novel categories.On a recently proposed large-scale fine-grained 3D part dataset, PartNet, we demonstrate that our method can transfer knowledge of parts learned from 3 training categories to 21 unseen testing categories without seeing any annotated samples.Quantitative comparisons against four strong shape segmentation baselines show that we achieve the state-of-the-art performance.

一个用于三维物体部分分割的零样本分割框架。将分割建模为一个决策过程，并作为一个上下文赌博机问题来解决。

一种将物体的三维点云分割成组合零件的方法，重点是将部件分组归纳到训练期间未见过的新物体类别中，相对于基线显示出强大的性能。

本文提出了一种在物体点云中进行部件分割的方法。


This paper presents the ballistic graph neural network.Ballistic graph neural network tackles the weight distribution from a transportation perspective and has many different properties comparing to the traditional graph neural network pipeline.The ballistic graph neural network does not require to calculate any eigenvalue.The filters propagate exponentially faster($\sigma^2 \sim T^2$) comparing to traditional graph neural network($\sigma^2 \sim T$).We use a perturbed coin operator to perturb and optimize the diffusion rate.Our results show that by selecting the diffusion speed, the network can reach a similar accuracy with fewer parameters.We also show the perturbed filters act as better representations comparing to pure ballistic ones.We provide a new perspective of training graph neural network, by adjusting the diffusion rate, the neural network's performance can be improved.

关于如何根据扩散特性收集节点之间的关联性的新观点。

一种新的图神经网络的扩散操作，不需要计算特征值，与传统的图神经网络相比，可以以指数级速度传播。

本文提出通过引入弹道行走来应对扩散速度的问题。


In this paper, we propose a \textit{weak supervision} framework for neural ranking tasks based on the data programming paradigm \citep{Ratner2016}, which enables us to leverage multiple weak supervision signals from different sources.Empirically, we consider two sources of weak supervision signals, unsupervised ranking functions and semantic feature similarities.We train a BERT-based passage-ranking model (which achieves new state-of-the-art performances on two benchmark datasets with full supervision) in our weak supervision framework.Without using ground-truth training labels, BERT-PR models outperform BM25 baseline by a large margin on all three datasets and even beat the previous state-of-the-art results with full supervision on two of datasets.

我们提出了一个基于数据编程框架的排名任务的弱监督训练管道，其中我们训练一个基于BERT的排名模型并建立新的SOTA。

作者提出了一个BERT和弱监督框架的组合来解决通道排名的问题，获得了比完全监督的最先进的结果。


We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective.We demonstrate a very universal Frequency Principle (F-Principle) --- DNNs often fit target functions from low to high frequencies --- on high-dimensional benchmark datasets, such as MNIST/CIFAR10, and deep networks, such as VGG16.This F-Principle of DNNs is opposite to the learning behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibits faster convergence for higher frequencies, for various scientific computing problems.With a naive theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions.The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function.This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.

在实际问题中，我们发现DNN在训练过程中经常从低频到高频地拟合目标函数。

本文分析了神经网络在傅里叶域的损失，发现DNN倾向于在高频成分之前学习低频成分。

本文通过傅里叶分析研究了NN的训练过程，得出结论：NN在学习高频成分之前，先学习低频成分。


The problem of accelerating drug discovery relies heavily on automatic tools to optimize precursor molecules to afford them with better biochemical properties.Our work in this paper substantially extends prior state-of-the-art on graph-to-graph translation methods for molecular optimization.In particular, we realize coherent multi-resolution representations by interweaving the encoding of substructure components with the atom-level encoding of the original molecular graph.Moreover, our graph decoder is fully autoregressive, and interleaves each step of adding a new substructure with the process of resolving its attachment to the emerging molecule.We evaluate our model on multiple molecular optimization tasks and show that our model significantly outperforms previous state-of-the-art baselines.

我们提出了一个多解析度、分层耦合的编码器-解码器，用于图到图的翻译。

用化学子结构作为构件生成分子图的分层图-图转换模型，该模型是完全自回归的，并能学习连贯的多解析度表征，性能优于以前的模型。

作者提出了一种用于生成新型有机分子的分层图到图的翻译方法。


Equivariance is a nice property to have as it produces much more parameter efficient neural architectures and preserves the structure of the input through the feature mapping.Even though some combinations of transformations might never appear (e.g. an upright face with a horizontal nose), current equivariant architectures consider the set of all possible transformations in a transformation group when learning feature representations.Contrarily, the human visual system is able to attend to the set of relevant transformations occurring in the environment and utilizes this information to assist and improve object recognition.Based on this observation, we modify conventional equivariant feature mappings such that they are able to attend to the set of co-occurring transformations in data and generalize this notion to act on groups consisting of multiple symmetries.We show that our proposed co-attentive equivariant neural networks consistently outperform conventional rotation equivariant and rotation & reflection equivariant neural networks on rotated MNIST and CIFAR-10.

我们利用注意力将等价神经网络限制在数据中的一组或共同发生的转变。

本文将注意力与群的等价性结合起来，特别是研究旋转、平移和翻转的p4m群，并推导出一种不破坏等价性的自我注意力的形式。

作者提出了一种旋转等值神经网络的自我注意机制，它比常规的旋转等值网络提高了分类性能。


The fast generation and refinement of protein backbones would constitute a major advancement to current methodology for the design and development of de novo proteins.In this study, we train Generative Adversarial Networks (GANs) to generate fixed-length full-atom protein backbones, with the goal of sampling from the distribution of realistic 3-D backbone fragments.We represent protein structures by pairwise distances between all backbone atoms, and present a method for directly recovering and refining the corresponding backbone coordinates in a differentiable manner.We show that interpolations in the latent space of the generator correspond to smooth deformations of the output backbones, and that test set structures not seen by the generator during training exist in its image.Finally, we perform sequence design, relaxation, and ab initio folding of a subset of generated structures, and show that in some cases we can recover the generated folds after forward-folding.Together, these results suggest a mechanism for fast protein structure refinement and folding using external energy functions.

我们训练一个GAN来生成和恢复全原子的蛋白质骨架，并且我们表明，在选定的情况下，我们可以在序列设计和非初始正向折叠后恢复所生成的蛋白质。

一个蛋白质骨架的生成模型，它使用了GAN、类似自动编码器的网络和细化过程，并有一组定性评价显示出正面的结果。

本文介绍了一种使用生成式对抗网络生成蛋白质骨架的端到端方法。


Few-Shot Learning (learning with limited labeled data) aims to overcome the limitations of traditional machine learning approaches which require thousands of labeled examples to train an effective model.Considered as a hallmark of human intelligence, the community has recently witnessed several contributions on this topic, in particular through meta-learning, where a model learns how to learn an effective model for few-shot learning.The main idea is to acquire prior knowledge from a set of training tasks, which is then used to perform (few-shot) test tasks.Most existing work assumes that both training and test tasks are drawn from the same distribution, and a large amount of labeled data is available in the training tasks.This is a very strong assumption which restricts the usage of meta-learning strategies in the real world where ample training tasks following the same distribution as test tasks may not be available.In this paper, we propose a novel meta-learning paradigm wherein a few-shot learning model is learnt, which simultaneously overcomes domain shift between the train and test tasks via adversarial domain adaptation.We demonstrate the efficacy the proposed method through extensive experiments.

元学习的Few Shot学习假设训练任务和测试任务是从同一分布中抽取的。如果不是这样，你会怎么做呢？用任务级的领域适应性进行元学习!

本文提出了一种将无监督的对抗性领域适应与原型网络相结合的模型，该模型在有领域转移的几次学习任务上比几次学习基线表现得更好。

作者提出了元领域适应，以解决元学习设置中的领域转移情况，在几个实验中证明了性能的改善。


Universal probabilistic programming systems (PPSs) provide a powerful framework for specifying rich and complex probabilistic models.However, this expressiveness comes at the cost of substantially complicating the process of drawing inferences from the model.In particular, inference can become challenging when the support of the model varies between executions.Though general-purpose inference engines have been designed to operate in such settings, they are typically inefficient, often relying on proposing from the prior to make transitions.To address this, we introduce a new inference framework: Divide, Conquer, and Combine (DCC).DCC divides the program into separate straight-line sub-programs, each of which has a fixed support allowing more powerful inference algorithms to be run locally, before recombining their outputs in a principled fashion.We show how DCC can be implemented as an automated and general-purpose PPS inference engine, and empirically confirm that it can provide substantial performance improvements over previous approaches.

Divide, Conquer, and Combine是一种新的推理方案，可以在具有随机支持的概率程序上进行，即变量的存在是随机的。


Detecting communities or the modular structure of real-life networks (e.g. a socialnetwork or a product purchase network) is an important task because the way anetwork functions is often determined by its communities.The traditional approaches to community detection involve modularity-based approaches,which generally speaking, construct partitions based on heuristics thatseek to maximize the ratio of the edges within the partitions to those betweenthem.Node embedding approaches, which represent each node in a graph as areal-valued vector, transform the problem of community detection in a graph tothat of clustering a set of vectors.Existing node embedding approaches are primarilybased on first initiating uniform random walks from each node to constructa context of a node and then seeks to make the vector representation ofthe node close to its context.However, standard node embedding approaches donot directly take into account the community structure of a network while constructingthe context around each node.To alleviate this, we explore two differentthreads of work.First, we investigate the use of biased random walks (specifically,maximum entropy based walks) to obtain more centrality preserving embeddingof nodes, which we hypothesize may lead to more effective clusters in the embeddedspace.Second, we propose a community structure aware node embeddingapproach where we incorporate modularity-based partitioning heuristics intothe objective function of node embedding.We demonstrate that our proposed approachfor community detection outperforms a number of modularity-based baselinesas well as K-means on a standard node-embedded vector space (specifically,node2vec) on a wide range of real-life networks of different sizes and densities.

一种保存社区的节点嵌入算法，能更有效地检测出嵌入空间上有聚类的群体


A point cloud is an agile 3D representation, efficiently modeling an object's surface geometry.However, these surface-centric properties also pose challenges on designing tools to recognize and synthesize point clouds.This work presents a novel autoregressive model, PointGrow, which generates realistic point cloud samples from scratch or conditioned from given semantic contexts.Our model operates recurrently, with each point sampled according to a conditional distribution given its previously-generated points.Since point cloud object shapes are typically encoded by long-range interpoint dependencies, we augment our model with dedicated self-attention modules to capture these relations.Extensive evaluation demonstrates that PointGrow achieves satisfying performance on both unconditional and conditional point cloud generation tasks, with respect to fidelity, diversity and semantic preservation.Further, conditional PointGrow learns a smooth manifold of given images where 3D shape interpolation and arithmetic calculation can be performed inside.

一个用于生成多样化点云的自回归深度学习模型。

一种将三维图形生成为点云的方法，它考虑了根据坐标对点进行的字典排序，并训练了一个模型来预测点的顺序。

本文介绍了一个点云的生成模型，使用一个类似像素RNN的自动回归模型和一个注意力模型来处理较长距离的相互作用。


Reinforcement learning and evolutionary algorithms can be used to create sophisticated control solutions.Unfortunately explaining how these solutions work can be difficult to due to their "black box" nature.In addition, the time-extended nature of control algorithms often prevent direct applications of explainability techniques used for standard supervised learning algorithms.This paper attempts to address explainability of blackbox control algorithms through six different techniques:1) Bayesian rule lists,2) Function analysis,3) Single time step integrated gradients,4) Grammar-based decision trees,5) Sensitivity analysis combined with temporal modeling with LSTMs, and6) Explanation templates.These techniques are tested on a simple 2d domain, where a simulated rover attempts to navigate through obstacles to reach a goal.For control, this rover uses an evolved multi-layer perception that maps an 8d field of obstacle and goal sensors to an action determining where it should go in the next time step.Results show that some simple insights in explaining the neural network are possible, but that good explanations are difficult.

描述了一系列应用于导航用的简单神经网络控制器的可解释性技术。

本文对为作为漫游车运动反控器的多层感知器提供解释的问题提出了见解和解释，以及如何解释黑盒模型的想法。


The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments.This challenging task demands that the agent be aware of which instruction was completed, which instruction is needed next, which way to go, and its navigation progress towards the goal.In this paper, we introduce a self-monitoring agent with two complementary components: (1) visual-textual co-grounding module to locate the instruction completed in the past, the instruction required for the next action, and the next moving direction from surrounding images and (2) progress monitor to ensure the grounded instruction correctly reflects the navigation progress.We test our self-monitoring agent on a standard benchmark and analyze our proposed approach through a series of ablation studies that elucidate the contributions of the primary components.Using our proposed method, we set the new state of the art by a significant margin (8% absolute increase in success rate on the unseen test set).Code is available at https://github.com/chihyaoma/selfmonitoring-agent.

我们提出了一个用于视觉和语言导航任务的自我监控代理。

一种视觉+语言导航的方法，该方法使用进度显示器和视觉-文本共同基础模块跟踪指令的进展，并在标准基准上表现良好。

本文描述了一个具有全景视觉注意力和辅助性进度监测损失的视觉和语言导航模型，给出了最先进的结果。


Environments in Reinforcement Learning (RL) are usually only partially observable.To address this problem, a possible solution is to provide the agent with information about past  observations.While common methods represent this history using a Recurrent Neural Network (RNN), in this paper we propose an alternative representation which is based on the record of the past events observed in a given episode.Inspired by the human memory, these events describe only important changes in the environment and, in our approach, are automatically discovered using self-supervision. We evaluate our history representation method using two challenging RL benchmarks: some games of the Atari-57 suite and the 3D environment Obstacle Tower.Using these benchmarks we show the advantage of our solution with respect to common RNN-based approaches.

事件发现来代表RL中代理的历史。

作者研究了部分观察环境下的RL问题，并提出了一个使用FFN但提供历史表示的解决方案，其性能优于PPO。

本文提出了一种代表过去历史的新方法，作为RL代理的输入，显示出比PPO和PPO的RN变体更好的性能。


The unconditional generation of high fidelity images is a longstanding benchmarkfor testing the performance of image decoders.Autoregressive image modelshave been able to generate small images unconditionally, but the extension ofthese methods to large images where fidelity can be more readily assessed hasremained an open problem.Among the major challenges are the capacity to encodethe vast previous context and the sheer difficulty of learning a distribution thatpreserves both global semantic coherence and exactness of detail.To address theformer challenge, we propose the Subscale Pixel Network (SPN), a conditionaldecoder architecture that generates an image as a sequence of image slices of equalsize.The SPN compactly captures image-wide spatial dependencies and requires afraction of the memory and the computation.To address the latter challenge, wepropose to use multidimensional upscaling to grow an image in both size and depthvia intermediate stages corresponding to distinct SPNs.We evaluate SPNs on theunconditional generation of CelebAHQ of size 256 and of ImageNet from size 32to 128.We achieve state-of-the-art likelihood results in multiple settings, set upnew benchmark results in previously unexplored settings and are able to generatevery high fidelity large scale samples on the basis of both datasets.

我们表明，自回归模型可以生成高保真图像。

一个利用解码器、尺寸放大解码器和深度放大解码器组件的架构，以解决学习图像中的长距离依赖关系的问题，从而获得高保真图像。

本文解决了对高保真图像的生成问题，成功地展示了具有128x128分辨率的似然密度模型的令人信服的Imagenet样本。


Real-world dynamical systems often consist of multiple stochastic subsystems that interact with each other.Modeling and forecasting the behavior of such dynamics are generally not easy, due to the inherent hardness in understanding the complicated interactions and evolutions of their constituents.This paper introduces the relational state-space model (R-SSM), a sequential hierarchical latent variable model that makes use of graph neural networks (GNNs) to simulate the joint state transitions of multiple correlated objects.By letting GNNs cooperate with SSM, R-SSM provides a flexible way to incorporate relational information into the modeling of multi-object dynamics.We further suggest augmenting the model with normalizing flows instantiated for vertex-indexed random variables and propose two auxiliary contrastive objectives to facilitate the learning.The utility of R-SSM is empirically evaluated on synthetic and real time series datasets.

一个深度分层的状态空间模型，其中相关对象的状态转换由图神经网络协调。

当每个对象表现出显着的随机性时，多个对象的顺序动态过程的分层潜变量模型。

本文提出了一个关系状态空间模型，模拟相关对象的联合状态转换，这些对象在一个图结构中是分层协调的。


Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses).When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed.While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents.This paper proposes to add such inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated.Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

我们引入了一种新的归纳偏向，将树状结构整合到循环神经网络中。

本文提出了ON-LSTM，一种新的RNN单元，它将潜在树结构整合到循环模型中，在语言建模、无监督解析、目标句法评价和逻辑推理上有良好的效果。


Skip connections made the training of very deep networks possible and have become an indispensable component in a variety of neural architectures.A completely satisfactory explanation for their success remains elusive.Here, we present a novel explanation for the benefits of skip connections in training very deep networks.The difficulty of training deep networks is partly due to the singularities caused by the non-identifiability of the model.Several such singularities have been identified in previous works:(i) overlap singularities caused by the permutation symmetry of nodes in a given layer,(ii) elimination singularities corresponding to the elimination, i.e. consistent deactivation, of nodes,(iii) singularities generated by the linear dependence of the nodes.These singularities cause degenerate manifolds in the loss landscape that slow down learning.We argue that skip connections eliminate these singularities by breaking the permutation symmetry of nodes, by reducing the possibility of node elimination and by making the nodes less linearly dependent.Moreover, for typical initializations, skip connections move the network away from the "ghosts" of these singularities and sculpt the landscape around them to alleviate the learning slow-down.These hypotheses are supported by evidence from simplified models, as well as from experiments with deep networks trained on real-world datasets.

由模型的不可识别性产生的退化流形减缓了深度网络的学习；跳过连接通过打破退化来帮助学习。

作者表明，消除奇点和重叠奇异点阻碍了深度神经网络的学习，并证明跳过连接可以减少这些奇异点的普遍性，加快学习速度。

论文研究了在深度网络中使用跳过连接作为缓解训练期间Hessian矩阵的奇异性的一种方式。


Representation learning is a central challenge across a range of machine learning areas.In reinforcement learning, effective and functional representations have the potential to tremendously accelerate learning progress and solve more challenging problems.Most prior work on representation learning has focused on generative approaches, learning representations that capture all the underlying factors of variation in the observation space in a more disentangled or well-ordered manner.In this paper, we instead aim to learn functionally salient representations: representations that are not necessarily complete in terms of capturing all factors of variation in the observation space, but rather aim to capture those factors of variation that are important for decision making -- that are "actionable".These representations are aware of the dynamics of the environment, and capture only the elements of the observation that are necessary for decision making rather than all factors of variation, eliminating the need for explicit reconstruction.We show how these learned representations can be useful to improve exploration for sparse reward problems, to enable long horizon hierarchical reinforcement learning, and as a state representation for learning policies for downstream tasks.We evaluate our method on a number of simulated environments, and compare it to prior methods for representation learning, exploration, and hierarchical reinforcement learning.

学习捕捉控制所需因素的状态表征 本文提出了一种学习表征的方法，其中欧氏距离的接近性代表了由类似政策实现的状态。 本文提出了一种学习表征的方法，其中欧氏距离的接近性代表了由类似政策实现的状态。


We explore the behavior of a standard convolutional neural net in a setting that introduces classification tasks sequentially and requires the net to master new tasks while preserving mastery of previously learned tasks.  This setting corresponds to that which human learners face as they acquire domain expertise, for example, as an individual reads a textbook chapter-by-chapter.Through simulations involving sequences of 10 related tasks, we find reason for optimism that nets will scale well as they advance from having a single skill to becoming domain experts.We observed two key phenomena.First, forward facilitation---the accelerated learning of task n+1 having learned n previous tasks---grows with n. Second, backward interference---the forgetting of the n previous tasks when learning task n+1---diminishes with n.  Forward facilitation is the goal of research on metalearning, and reduced backward interference is the goal of research on ameliorating catastrophic forgetting.We find that both of these goals are attained simply through broader exposure to a domain.

我们研究CNN的行为，因为它在掌握新任务的同时保留了对以前学习的任务的掌握。


We demonstrate a low effort method that unsupervisedly constructs task-optimized embeddings from existing word embeddings to gain performance on a supervised end-task.This avoids additional labeling or building more complex model architectures by instead providing specialized embeddings better fit for the end-task(s).Furthermore, the method can be used to roughly estimate whether a specific kind of end-task(s) can be learned form, or is represented in, a given unlabeled dataset, e.g. using publicly available probing tasks.We evaluate our method for diverse word embedding probing tasks and by size of embedding training corpus -- i.e. to explore its use in reduced (pretraining-resource) settings.

Morty改进了预训练的词嵌入，以达到以下目的。 (a)提高整体嵌入性能（对于多任务设置）或提高单任务性能，同时只需要最小的努力。


Data augmentation is commonly used to encode invariances in learning methods.However, this process is often performed in an inefficient manner, as artificial examples are created by applying a number of transformations to all points in the training set.The resulting explosion of the dataset size can be an issue in terms of storage and training costs, as well as in selecting and tuning the optimal set of transformations to apply.In this work, we demonstrate that it is possible to significantly reduce the number of data points included in data augmentation while realizing the same accuracy and invariance benefits of augmenting the entire dataset.We propose a novel set of subsampling policies, based on model influence and loss, that can achieve a 90% reduction in augmentation set size while maintaining the accuracy gains of standard data augmentation.

有选择地增强难以分类的点，从而实现高效的训练。

作者研究了识别数据增殖的子采样策略问题，并提出了基于模型影响和损失的策略，以及所提方法的经验基准。

作者提议使用影响或基于损失的方法来选择一个子集，用于增强训练模型的数据集，其中损失是对数据点的增加。


Over the last few years exciting work in deep generative models has produced models able to suggest new organic molecules by generating strings, trees, and graphs representing their structure.While such models are able to generate molecules with desirable properties, their utility in practice is limited due to the difficulty in knowing how to synthesize these molecules.We therefore propose a new molecule generation model, mirroring a more realistic real-world process, where reactants are selected and combined to form more complex molecules.More specifically, our generative model proposes a bag of initial reactants (selected from a pool of commercially-available molecules) and uses a reaction model to predict how they react together to generate new molecules.Modeling the entire process of constructing a molecule during generation offers a number of advantages.First, we show that such a model has the ability to generate a wide, diverse set of valid and unique molecules due to the useful inductive biases of modeling reactions.Second, modeling synthesis routes rather than final molecules offers practical advantages to chemists who are not only interested in new molecules but also suggestions on stable and safe synthetic routes.Third, we demonstrate the capabilities of our model to also solve one-step retrosynthesis problems, predicting a set of reactants that can produce a target product.

一个有机分子的深度生成模型，首先生成反应物构件，然后用反应预测器组合这些构件。

一个分子生成模型，通过两步过程生成分子，提供生成分子的合成路线，使用户能够检查生成化合物的合成可及性。


Deep neural networks are complex non-linear models used as predictive analytics tool and have demonstrated state-of-the-art performance on many classification tasks.  However, they have no inherent capability to recognize when their predictions might go wrong.There have been several efforts in the recent past to detect natural errors i.e.  misclassified inputs but these mechanisms pose additional energy requirements.  To address this issue, we present a novel post-hoc framework to detect natural errors in an energy efficient way.  We achieve this by appending relevant features based linear classifiers per class referred as Relevant features based Auxiliary Cells (RACs).   The proposed technique makes use of the consensus between RACs appended at few selected hidden layers to distinguish the correctly classified inputs from misclassified inputs.The combined confidence of RACs is utilized to determine if classification should terminate at an early stage.We demonstrate the effectiveness of our technique on various image classification datasets such as CIFAR10, CIFAR100 and Tiny-ImageNet.Our results show that for CIFAR100 dataset trained on VGG16 network, RACs can detect 46% of the misclassified examples along with 12% reduction in energy compared to the baseline network while 69% of the examples are correctly classified.

利用隐性表征提高深度神经网络的稳定性和能源效率。

本文旨在通过在一个或多个隐藏层之后添加基于相关特征的辅助细胞来决定是否提前结束分类，从而以节能的方式减少深度神经网络的错误分类。


Many methods have been developed to represent knowledge graph data, which implicitly exploit low-rank latent structure in the data to encode known information and enable unknown facts to be inferred.To predict whether a relationship holds between entities, their embeddings are typically compared in the latent space following a relation-specific mapping.Whilst link prediction has steadily improved, the latent structure, and hence why such models capture semantic information, remains unexplained.We build on recent theoretical interpretation of word embeddings as a basis to consider an explicit structure for representations of relations between entities.For identifiable relation types, we are able to predict properties and justify the relative performance of leading knowledge graph representation methods, including their often overlooked ability to make independent predictions.

利用词嵌入的洞察力理解知识图谱的结构。

本文试图理解知识图谱嵌入方法的潜在结构，并证明一个模型表示关系类型的能力取决于模型架构在关系条件方面的限制。

本文提出对链接预测（LP）模型的可解释性进行详细研究，利用最近对词嵌入的解释来对LP的模型性能进行更好的理解。


Many real-world applications involve multivariate, geo-tagged time series data: at each location, multiple sensors record corresponding measurements.For example, air quality monitoring system records PM2.5, CO, etc.The resulting time-series data often has missing values due to device outages or communication errors.In order to impute the missing values, state-of-the-art methods are built on Recurrent Neural Networks (RNN), which process each time stamp sequentially, prohibiting the direct modeling of the relationship between distant time stamps.Recently, the self-attention mechanism has been proposed for sequence modeling tasks such as machine translation, significantly outperforming RNN because the relationship between each two time stamps can be modeled explicitly.In this paper, we are the first to adapt the self-attention mechanism for multivariate, geo-tagged time series data.In order to jointly capture the self-attention across different dimensions (i.e. time, location and sensor measurements) while keep the size of attention maps reasonable, we propose a novel approach called Cross-Dimensional Self-Attention (CDSA) to process each dimension sequentially, yet in an order-independent manner.On three real-world datasets, including one our newly collected NYC-traffic dataset, extensive experiments demonstrate the superiority of our approach compared to state-of-the-art methods for both imputation and forecasting tasks.

用于多变量、地理标记的时间序列归纳的新型自我关注机制。

本文提出了以一种兼容有效的方式将变换器网络应用于时空数据的问题，并研究了实现三维关注的方法。

本文实证研究了转化器模型在输入的不同维度上对时间序列数据插补的有效性。


The conversion of scanned documents to digital forms is performed using an Optical Character Recognition (OCR) software.This work focuses on improving the quality of scanned documents in order to improve the OCR output.We create an end-to-end document enhancement pipeline which takes in a set of noisy documents and produces clean ones.Deep neural network based denoising auto-encoders are trained to improve the OCR quality.We train a blind model that works on different noise levels of scanned text documents.Results are shown for blurring and watermark noise removal from noisy scanned documents.

我们设计并测试了一个具有8个跳过连接的REDNET（ResNet Encoder-Decoder），以去除文件中的噪音，包括模糊和水印，从而形成了一个用于文件图像清理的高性能深度网络。


The existence of adversarial examples, or intentional mis-predictions constructed from small changes to correctly predicted examples, is one of the most significant challenges in neural network research today.Ironically, many new defenses are based on a simple observation - the adversarial inputs themselves are not robust and small perturbations to the attacking input often recover the desired prediction.While the intuition is somewhat clear, a detailed understanding of this phenomenon is missing from the research literature.This paper presents a comprehensive experimental analysis of when and why perturbation defenses work and potential mechanisms that could explain their effectiveness (or ineffectiveness) in different settings.

我们确定了一个防御技术系列，并表明确定性的有损压缩和对输入的随机扰动都导致了类似的稳定性收益。

本文讨论了破坏特定对抗性攻击的方法，是什么让对抗性图像变得不稳定，以及攻击者是否有可能使用一个通用的扰动模型来使他们的对抗性例子对这种扰动变得稳定。

本文研究了对抗性攻击对其输入的转变的稳定性。


There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD.In this work, we fill in the important, yet ambiguous concept of â€˜ease-of-useâ€™ by defining an optimizerâ€™s tunability:  How easy is it to find good hyperparameter configurations using automatic random hyperparameter search?We propose a practical and universal quantitative measure for optimizer tunability that can form the basis for a fair optimizer benchmark.  Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, we find  that Adam is the most tunable for the majority of problems, especially with a low budget for hyperparameter tuning.

我们提供了一种对优化器进行基准测试的方法，该方法对超参数调谐过程有所认识。

引入一个新的指标来捕捉优化器的可调性，并对不同数量的超参数调整下的深度学习优化器进行了全面的经验比较。

本文介绍了一种简单的可调性措施，可以在资源限制下对优化器进行比较，发现调整亚当优化器的学习率最容易找到表现良好的超参数配置。


The phase problem in diffraction physics is one of the oldest inverse problems in all of science.The central difficulty that any approach to solving this inverse problem must overcome is that half of the information, namely the phase of the diffracted beam, is always missing.In the context of electron microscopy, the phase problem is generally non-linear and solutions provided by phase-retrieval techniques are known to be poor approximations to the physics of electrons interacting with matter.Here, we show that a semi-supervised learning approach can effectively solve the phase problem in electron microscopy/scattering.In particular, we introduce a new Deep Neural Network (DNN), Y-net, which simultaneously learns a reconstruction algorithm via supervised training in addition to learning a physics-based regularization via unsupervised training.We demonstrate that this constrained, semi-supervised approach is an order of magnitude more data-efficient and accurate than the same model trained in a purely supervised fashion.In addition, the architecture of the Y-net model provides for a straightforward evaluation of the consistency of the model's prediction during inference and is generally applicable to the phase problem in other settings.

我们引入一个半监督的深度神经网络来近似解决电子显微镜中的相位问题


Word embeddings extract semantic features of words from large datasets of text.Most embedding methods rely on a log-bilinear model to predict the occurrenceof a word in a context of other words.Here we propose word2net, a method thatreplaces their linear parametrization with neural networks.For each term in thevocabulary, word2net posits a neural network that takes the context as input andoutputs a probability of occurrence.Further, word2net can use the hierarchicalorganization of its word networks to incorporate additional meta-data, such assyntactic features, into the embedding model.For example, we show how to shareparameters across word networks to develop an embedding model that includespart-of-speech information.We study word2net with two datasets, a collectionof Wikipedia articles and a corpus of U.S. Senate speeches.Quantitatively, wefound that word2net outperforms popular embedding methods on predicting held-out words and that sharing parameters based on part of speech further boostsperformance.Qualitatively, word2net learns interpretable semantic representationsand, compared to vector-based methods, better incorporates syntactic information.

Word2net是一种学习单词的神经网络表征的新方法，可以利用句法信息来学习更好的语义特征。

本文对SGNS进行了扩展，在架构上从词包模型改为前馈模型，并通过在不同的关联网络之间捆绑一个子层，贡献了一种新形式的正则化。

一种利用上下文向量的非线性组合来学习单词的向量表示的方法，其主要想法是用神经网络取代每个单词的嵌入。


A key goal in neuroscience is to understand brain mechanisms of cognitive functions.An emerging approach is to study â€œbrain statesâ€ dynamics using functional magnetic resonance imaging (fMRI).So far in the literature, brain states have typically been studied using 30 seconds of fMRI data or more, and it is unclear to which extent brain states can be reliably identified from very short time series.In this project, we applied graph convolutional networks (GCN) to decode brain activity over short time windows in a task fMRI dataset, i.e. associate a given window of fMRI time series with the task used.Starting with a populational brain graph with nodes defined by a parcellation of cerebral cortex and the adjacent matrix extracted from functional connectome, GCN takes a short series of fMRI volumes as input, generates high-level domain-specific graph representations, and then predicts the corresponding cognitive state.We investigated the performance of this GCN "cognitive state annotation" in the Human Connectome Project (HCP) database, which features 21 different experimental conditions spanning seven major cognitive domains, and high temporal resolution task fMRI data.Using a 10-second window, the 21 cognitive states were identified with an excellent average test accuracy of 89% (chance level 4.8%).As the HCP task battery was designed to selectively activate a wide range of specialized functional networks, we anticipate the GCN annotation to be applicable as a base model for other transfer learning applications, for instance, adapting to new task domains.

使用10s的fMRI信号窗口，我们的GCN模型从HCP数据集中识别了21种不同的任务条件，测试准确率为89%。


Modern deep neural networks (DNNs) require high memory consumption and large computational loads.  In order to deploy DNN algorithms efficiently on edge or mobile devices, a series of DNN compression algorithms have been explored, including the line of works on factorization methods.Factorization methods approximate the weight matrix of a DNN layer with multiplication of two or multiple low-rank matrices.However, it is hard to measure the ranks of DNN layers during the training process.Previous works mainly induce low-rank through implicit approximations or via costly singular value decomposition (SVD) process on every training step.The former approach usually induces a high accuracy loss while the latter prevents DNN factorization from efficiently reaching a high compression rate.In this work, we propose SVD training, which first applies SVD to decompose DNN's layers and then performs training on the full-rank decomposed weights.To improve the training quality and convergence, we add orthogonality regularization to the singular vectors, which ensure the valid form of SVD and avoid gradient vanishing/exploding.Low-rank is encouraged by applying sparsity-inducing regularizers on the singular values of each layer.Singular value pruning is applied at the end to reach a low-rank model.We empirically show that SVD training can significantly reduce the rank of DNN layers and achieve higher reduction on computation load under the same accuracy, comparing to not only previous factorization methods but also state-of-the-art filter pruning methods.

通过具有稀疏奇异值和正交奇异向量的SVD训练有效地诱导低等级深度神经网络。

本文介绍了一种网络压缩的方法，鼓励每一层的权重矩阵具有低等级，并明确地将权重矩阵分解成类似SVD的因子，作为新的参数进行处理。

建议在训练前对深度神经网络的每一层进行参数化，用低秩矩阵分解，相应地用两个连续的卷积取代卷积，然后训练分解后的方法。


The recent rise in popularity of few-shot learning algorithms has enabled models to quickly adapt to new tasks based on only a few training samples.Previous few-shot learning works have mainly focused on classification and reinforcement learning. In this paper, we propose a few-shot meta-learning system that focuses exclusively on regression tasks.Our model is based on the idea that the degree of freedom of the unknown function can be significantly reduced if it is represented as a linear combination of a set of appropriate basis functions.This enables a few labelled samples to approximate the function.We design a Feature Extractor network to encode basis functions for a task distribution, and a  Weights Generator to generate the weight vector for a novel task.We show that our model outperforms the current state of the art meta-learning methods in various regression tasks.

我们提出了一个专门为回归任务量身定做的少量学习模型 本文提出了一种针对小样本回归问题的新型样本学习方法。 一种用少量样本学习回归模型的方法，并优于其他方法。


Most classification and segmentation datasets assume a closed-world scenario in which predictions are expressed as distribution over a predetermined set of visual classes.However, such assumption implies unavoidable and often unnoticeable failures in presence of out-of-distribution (OOD) input.These failures are bound to happen in most real-life applications since current visual ontologies are far from being comprehensive.We propose to address this issue by discriminative detection of OOD pixels in input data.Different from recent approaches, we avoid to bring any decisions by only observing the training dataset of the primary model trained to solve the desired computer vision task.Instead, we train a dedicated OOD modelwhich discriminates the primary training set from a much larger "background" dataset which approximates the variety of the visual world.We perform our experiments on high resolution natural images in a dense prediction setup.We use several road driving datasets as our training distribution, while we approximate the background distribution with the ILSVRC dataset.We evaluate our approach on WildDash test, which is currently the only public test dataset with out-of-distribution images.The obtained results show that the proposed approach succeeds to identify out-of-distribution pixels while outperforming previous work by a wide margin.

我们提出了一种在语义分割中检测异常像素的新方法。

本文讨论了用于帮助分割过程的异常检测，并提出了一种训练二元分类器的方法，该分类器可以将已知类别集合的图像斑块与未知类别的图像斑块区分开来。

本文旨在为语义分割检测出异常像素，这项工作利用了其他领域的数据来检测未确定的类别，以更好地模拟不确定性。


Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices.Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy.The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead.To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs.However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large.The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results.It is difficult for even deep reinforcement learning (DRL) DDPG-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy.In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer.Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06%, and decrease the inference energy consumption by 50.69%, while achieving the same inference accuracy.

使用分层深度强化学习进行准确、快速和自动的核素神经网络量化，具有混合精度

一种量化神经网络权重和激活的方法，该方法使用深度强化学习来选择一个层中单个内核的位宽，并且比以前的方法取得更好的性能，或延迟。

本文提出为神经网络中的每个内核自动搜索量化方案，使用分层RL来指导搜索。


Recent visual analytics systems make use of multiple machine learning models to better fit the data as opposed to traditional single, pre-defined model systems.However, while multi-model visual analytic systems can be effective, their added complexity poses usability concerns, as users are required to interact with the parameters of multiple models.Further, the advent of various model algorithms and associated hyperparameters creates an exhaustive model space to sample models from.This poses complexity to navigate this model space to find the right model for the data and the task.In this paper, we present Gaggle, a multi-model visual analytic system that enables users to interactively navigate the model space.Further translating user interactions into inferences, Gaggle simplifies working with multiple models by automatically finding the best model from the high-dimensional model space to support various user tasks.Through a qualitative user study, we show how our approach helps users to find a best model for a classification and ranking task.The study results confirm that Gaggle is intuitive and easy to use, supporting interactive model space navigation and automated model selection without requiring any technical expertise from users.

Gaggle，一个交互式视觉分析系统，帮助用户交互式地浏览模型空间，进行分类和排名任务。

一个新的视觉分析系统，旨在使非专业用户通过使用基于示范的方法，交互式地浏览一个模型空间。

一个视觉化分析系统，帮助新手分析师在执行分类和排名任务时浏览模型空间。


Chinese text classification has received more and more attention today.However, the problem of Chinese text representation still hinders the improvement of Chinese text classification, especially the polyphone and the homophone in social media.To cope with it effectively, we propose a new structure, the Extractor, based on attention mechanisms and design novel attention networks named Extractor-attention network (EAN).Unlike most of previous works, EAN uses a combination of a word encoder and a Pinyin character encoder instead of a single encoder.It improves the capability of Chinese text representation.Moreover, compared with the hybrid encoder methods, EAN has more complex combination architecture and more reducing parameters structures.Thus, EAN can take advantage of a large amount of information that comes from multi-inputs and alleviates efficiency issues.The proposed model achieves the state of the art results on 5 large datasets for Chinese text classification.

我们提出了一种新的注意力网络和混合编码器来解决中文文本分类的文本表示问题，特别是关于发音的语言现象，如多音字和同音字。

本文为中文文本分类任务提出了一个由单词编码器和拼音编码器组成的基于注意力的模型，并为拼音字符编码器扩展了该架构。

提出了一个注意力网络，在这个网络中，单词和拼音都被考虑为中文表示，在几个数据集中用于文本分类，显示了改进的结果。


Recent advances in learning from demonstrations (LfD) with deep neural networks have enabled learning complex robot skills that involve high dimensional perception such as raw image inputs. LfD algorithms generally assume learning from single task demonstrations.In practice, however, it is more efficient for a teacher to demonstrate a multitude of tasks without careful task set up, labeling, and engineering.Unfortunately in such cases, traditional imitation learning techniques fail to represent the multi-modal nature of the data, and often result in sub-optimal behavior.In this paper we present an LfD approach for learning multiple modes of behavior from visual data.Our approach is based on a stochastic deep neural network (SNN), which represents the underlying intention in the demonstration as a stochastic activation in the network.We present an efficient algorithm for training SNNs, and for learning with vision inputs, we also propose an architecture that associates the intention with a stochastic attention module.We demonstrate our method on real robot visual object reaching tasks, and show thatit can reliably learn the multiple behavior modes in the demonstration data.Video results are available at https://vimeo.com/240212286/fd401241b9.

使用随机神经网络建模意图从非结构化演示中进行多模式模仿学习.

一种新的基于抽样的潜变量模型推理方法，适用于多模态模仿学习，在实际的视觉机器人任务中比确定性神经网络和随机性神经网络效果更好。

本文展示了如何利用随机神经网络从视觉数据中模仿学习几种模式，以及一种从演示中学习的方法，其中给出了同一任务的几种模式。


The interpretability of neural networks has become crucial for their applications in real world with respect to the reliability and trustworthiness.Existing explanation generation methods usually provide important features by scoring their individual contributions to the model prediction and ignore the interactions between features, which eventually provide a bag-of-words representation as explanation.In natural language processing, this type of explanations is challenging for human user to understand the meaning of an explanation and draw the connection between explanation and model prediction, especially for long texts.In this work, we focus on detecting the interactions between features, and propose a novel approach to build a hierarchy of explanations based on feature interactions.The proposed method is evaluated with three neural classifiers, LSTM, CNN, and BERT, on two benchmark text classification datasets.The generated explanations are assessed by both automatic evaluation measurements and human evaluators.Experiments show the effectiveness of the proposed method in providing explanations that are both faithful to models, and understandable to humans.

通过检测特征的相互作用来构建文本分类的层次解释的新方法。

一种为文本分类器做出的预测提供解释的新方法，在词级重要性得分上优于基线，还有一种新的指标，即内聚力损失，用于评估跨度级的重要性。

一种基于特征相互作用和特征重要性得分的解释方法，与独立的特征贡献相比。


Making deep convolutional neural networks more accurate typically comes at the cost of increased computational and memory resources.In this paper, we reduce this cost by exploiting the fact that the importance of features computed by convolutional layers is highly input-dependent, and propose feature boosting and suppression (FBS), a new method to predictively amplify salient convolutional channels and skip unimportant ones at run-time.FBS introduces small auxiliary connections to existing convolutional layers.In contrast to channel pruning methods which permanently remove channels, it preserves the full network structures and accelerates convolution by dynamically skipping unimportant input and output channels.FBS-augmented networks are trained with conventional stochastic gradient descent, making it readily available for many state-of-the-art CNNs.We compare FBS to a range of existing channel pruning and dynamic execution schemes and demonstrate large improvements on ImageNet classification.Experiments show that FBS can respectively provide 5Ã— and 2Ã— savings in compute on VGG-16 and ResNet-18, both with less than 0.6% top-5 accuracy loss.

我们通过在特征计算中动态地提升和抑制通道，使卷积层运行得更快。

一种用于动态通道修剪的特征提升和抑制方法，预测每个通道的重要性，然后使用仿生函数来放大/抑制通道的重要性。

在测试中动态选择通道的通道修剪方法的建议。


We propose a novel way of reducing the number of parameters in the storage-hungry fully connected layers of a neural network by using pre-defined sparsity, where the majority of connections are absent prior to starting training.Our results indicate that convolutional neural networks can operate without any loss of accuracy at less than 0.5% classification layer connection density, or less than 5% overall network connection density.We also investigate the effects of pre-defining the sparsity of networks with only fully connected layers.Based on our sparsifying technique, we introduce the `scatter' metric to characterize the quality of a particular connection pattern.As proof of concept, we show results on CIFAR, MNIST and a new dataset on classifying Morse code symbols, which highlights some interesting trends and limits of sparse connection patterns.

神经网络可以被预先定义为具有稀疏的连接性，而不会出现性能下降。

本文研究了卷积图像分类网络上层的稀疏连接模式，并介绍了在窗口/组之间分配连接的启发式方法和一种称为散点的措施，以构建连接遮蔽。

建议通过在分类层设置稀疏的连接权重来减少深度网络学习的参数数量，并引入 "散点 "的概念。


Deep neural networks are vulnerable to adversarial examples, which becomes one of the most important problems in the development of deep learning.While a lot of efforts have been made in recent years, it is of great significance to perform correct and complete evaluations of the adversarial attack and defense algorithms.In this paper, we establish a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness on image classification tasks.After briefly reviewing plenty of representative attack and defense methods, we perform large-scale experiments with two robustness curves as the fair-minded evaluation criteria to fully understand the performance of these methods.Based on the evaluation results, we draw several important findings and provide insights for future research.

我们提供了一个全面的、严格的、连贯的基准来评估深度学习模型的对抗性稳定性。

本文介绍了在各种对抗性攻击方法下对不同种类的分类模型的评估。

一个大规模的实证研究，比较了不同的对抗性攻击和防御技术，并使用准确性与扰动预算和准确性与攻击强度曲线来评估攻击和防御。


We propose a modification to traditional Artificial Neural Networks (ANNs), which provides the ANNs with new aptitudes motivated by biological neurons.  Biological neurons work far beyond linearly summing up synaptic inputs and then transforming the integrated information.  A biological neuron change firing modes accordingly to peripheral factors (e.g., neuromodulators) as well as intrinsic ones.  Our modification connects a new type of ANN nodes, which mimic the function of biological neuromodulators and are termed modulators, to enable other traditional ANN nodes to adjust their activation sensitivities in run-time based on their input patterns.  In this manner, we enable the slope of the activation function to be context dependent.  This modification produces statistically significant improvements in comparison with traditional ANN nodes in the context of Convolutional Neural Networks and Long Short-Term Memory networks.

我们提出对传统的人工神经网络进行修改，以神经元的生物学为动机，使激活函数的形状取决于上下文。

一种根据对ANN中的神经元层的输入来调整该层的激活的方法，该方法报告了高于基线的改进。

介绍了神经网络中基本神经元的结构变化，以及在将神经元线性组合输出送入激活函数之前，用调制器将其相乘的想法。


In this work, we study how the large-scale pretrain-finetune framework changes the behavior of a neural language generator.We focus on the transformer encoder-decoder model for the open-domain dialogue response generation task.We find that after standard fine-tuning, the model forgets important language generation skills acquired during large-scale pre-training.We demonstrate the forgetting phenomenon through a detailed behavior analysis from the perspectives of context sensitivity and knowledge transfer.Adopting the concept of data mixing, we propose an intuitive fine-tuning strategy named "mix-review''.We find that mix-review effectively regularize the fine-tuning process, and the forgetting problem is largely alleviated.Finally, we discuss interesting behavior of the resulting dialogue model and its implications.

我们发现了预训练的NLG模型微调中的遗忘问题，并提出了解决该问题的mix-review策略。

本文从语境敏感性和知识转移的角度分析了预训练-微调框架中的遗忘问题，并提出了一种优于权重衰减方法的微调策略。

研究预训练-调整框架中的遗忘问题，特别是在对话反应生成任务中的遗忘问题，并提出了一种mix-review策略来缓解遗忘问题。


Combining domain knowledge models with neural models has been challenging.  End-to-end trained neural models often perform better (lower Mean Square Error) than domain knowledge models or domain/neural combinations, and the combination is inefficient to train.  In this paper, we demonstrate that by composing domain models with machine learning models, by using extrapolative testing sets, and invoking decorrelation objective functions, we create models which can predict more complex systems.The models are interpretable, extrapolative, data-efficient, and capture predictable but complex non-stochastic behavior such as unmodeled degrees of freedom and systemic measurement noise.  We apply this improved modeling paradigm to several simulated systems and an actual physical system in the context of system identification.   Several ways of composing domain models with neural models are examined for time series, boosting, bagging, and auto-encoding on various systems of varying complexity and non-linearity.  Although this work is preliminary, we show that the ability to combine models is a very promising direction for neural modeling.

复杂系统的改进建模使用了混合神经/域模型的构成、新的去相关损失函数和外插测试集

本文通过实验比较了由物理模型、神经网络和随机模型组成的各种混合模型的外插预测，并解决了未建模动力学成为瓶颈的难题。

本文介绍了将神经网络与非NN模型相结合的方法，以预测复杂物理系统的行为。


Humans can learn task-agnostic priors from interactive experience and utilize the priors for novel tasks without any finetuning.In this paper, we propose Scoring-Aggregating-Planning (SAP), a framework that can learn task-agnostic semantics and dynamics priors from arbitrary quality interactions as well as the corresponding sparse rewards and then plan on unseen tasks in zero-shot condition.The framework finds a neural score function for local regional state and action pairs that can be aggregated to approximate the quality of a full trajectory; moreover, a dynamics model that is learned with self-supervision can be incorporated for planning.Many of previous works that leverage interactive data for policy learning either need massive on-policy environmental interactions or assume access to expert data while we can achieve a similar goal with pure off-policy imperfect data.Instantiating our framework results in a generalizable policy to unseen tasks.Experiments demonstrate that the proposed method can outperform baseline methods on a wide range of applications including gridworld, robotics tasks and video games.

我们从探索数据中学习密集的分数和动态模型作为先验，并利用它们在零样本条件下的新任务中诱导出一个好的政策。

本文讨论了在新环境中的零样本泛化问题，并提出了一种在《网格世界》、《超级马里奥兄弟》和《3D机器人》上取得成果的方法。

一种旨在为零样本泛化学习任务诊断先验的方法，其想法是在基于模型的RL框架之上采用一种建模方法。


Particle-based inference algorithm is a promising method to efficiently generate samples for an intractable target distribution by iteratively updating a set of particles.As a noticeable example, Stein variational gradient descent (SVGD) provides a deterministic and computationally efficient update, but it is known to underestimate the variance in high dimensions, the mechanism of which is poorly understood.In this work we explore a connection between SVGD and MMD-based inference algorithm via Stein's lemma.By comparing the two update rules, we identify the source of bias in SVGD as a combination of high variance and deterministic bias, and empirically demonstrate that the removal of either factors leads to accurate estimation of the variance.In addition, for learning high-dimensional Gaussian target, we analytically derive the converged variance for both algorithms, and confirm that only SVGD suffers from the "curse of dimensionality".

分析高维度下SVGD方差塌陷的内在机制。


We describe an approach to understand the peculiar and counterintuitive generalization properties of deep neural networks.  The approach involves going beyond worst-case theoretical capacity control frameworks that have been popular in machine learning in recent years to revisit old ideas in the statistical mechanics of neural networks.  Within this approach, we present a prototypical Very Simple Deep Learning (VSDL) model, whose behavior is controlled by two control parameters, one describing an effective amount of data, or load, on the network (that decreases when noise is added to the input), and one with an effective temperature interpretation (that increases when algorithms are early stopped).  Using this model, we describe how a very simple application of ideas from the statistical mechanics theory of generalization provides a strong qualitative description of recently-observed empirical results regarding the inability of deep neural networks not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc.

反思泛化需要重新审视旧有的想法：统计力学方法和复杂的学习行为

作者提出，统计力学思想将有助于理解深度神经网络的泛化特性，并给出了一种方法，对有关深度神经网络和学习算法的经验结果进行了强而有力的定性描述。

一组与理论上理解多层神经网络的泛化特性有关的想法，以及深度学习中的行为与单层和二层神经网络的定量统计物理分析结果之间的定性类比。


Computations for the softmax function in neural network models are expensive when the number of output classes is large.This can become a significant issue in both training and inference for such models.In this paper, we present Doubly Sparse Softmax (DS-Softmax), Sparse Mixture of Sparse of Sparse Experts, to improve the efficiency for softmax inference.During training, our method learns a two-level class hierarchy by dividing entire output class space into several partially overlapping experts.Each expert is responsible for a learned subset of the output class space and each output class only belongs to a small number of those experts.During inference, our method quickly locates the most probable expert to compute small-scale softmax.Our method is learning-based and requires no knowledge of the output class partition space a priori.We empirically evaluate our method on several real-world tasks and demonstrate that we can achieve significant computation reductions without loss of performance.

我们提出了双重稀疏softmax，即稀疏专家的稀疏混合物，通过利用两级重叠的层次结构来提高softmax推理的效率。

本文提出了一种当类的数量非常大时，对softmax计算的快速近似方法。

本文提出了一种稀疏专家的稀疏混合物，它可以学习两级类的层次结构，以实现高效的softmax推理。


Supervised machine learning models for high-value computer vision applications such as medical image classification often require large datasets labeled by domain experts, which are slow to collect, expensive to maintain, and static with respect to changes in the data distribution.In this context, we assess the utility of observational supervision, where we take advantage of passively-collected signals such as eye tracking or â€œgazeâ€ data, to reduce the amount of hand-labeled data needed for model training.Specifically, we leverage gaze information to directly supervise a visual attention layer by penalizing disagreement between the spatial regions the human labeler looked at the longest and those that most heavily influence model output.We present evidence that constraining the model in this way can reduce the number of labeled examples required to achieve a given performance level by as much as 50%, and that gaze information is most helpful on more difficult tasks.

我们探索使用被动收集的眼球追踪数据来减少训练期间所需的标注数据量。

一种利用凝视信息来降低模型的样本复杂度和所需的标签努力以获得目标性能的方法，在中等规模的样本和较难的任务中，结果有所改善。

一种将凝视信号纳入标准CNN进行图像分类的方法，在模型的类激活图和由眼动追踪信息构建的图之间的差异基础上增加损失函数项。


We study the robustness to symmetric label noise of GNNs training procedures.By combining the nonlinear neural message-passing models (e.g. Graph Isomorphism Networks, GraphSAGE, etc.) with loss correction methods, we present a noise-tolerant approach for the graph classification task.Our experiments show that test accuracy can be improved under the artificial symmetric noisy setting.

我们将损失校正应用于图神经网络，以训练一个对噪声更稳定的模型。 本文介绍了图神经网络的损失校正，以处理对称图标签噪音，重点介绍了图分类任务。 本文提出在图神经网络的背景下使用噪音校正损失来处理噪音标签。


Through many recent advances in graph representation learning, performance achieved on tasks involving graph-structured data has substantially increased in recent years---mostly on tasks involving node-level predictions.The setup of prediction tasks over entire graphs (such as property prediction for a molecule, or side-effect prediction for a drug), however, proves to be more challenging, as the algorithm must combine evidence about several structurally relevant patches of the graph into a single prediction.Most prior work attempts to predict these graph-level properties while considering only one graph at a time---not allowing the learner to directly leverage structural similarities and motifs across graphs.Here we propose a setup in which a graph neural network receives pairs of graphs at once, and extend it with a co-attentional layer that allows node representations to easily exchange structural information across them.We first show that such a setup provides natural benefits on a pairwise graph classification task (drug-drug interaction prediction), and then expand to a more generic graph regression setup: enhancing predictions over QM9, a standard molecular prediction benchmark.Our setup is flexible, powerful and makes no assumptions about the underlying dataset properties, beyond anticipating the existence of multiple training graphs.

我们在一个成对的图训练系统中使用图的共同关注，用于图分类和回归。

本文在GCN中注入了一个多头共同关注机制，允许一种药物在药物副作用预测期间关注另一种药物。

一种用共同注意力层扩展基于图的学习的方法，该方法在成对图分类任务上优于以前的其他方法。


In this paper we study image captioning as a conditional GAN training, proposing both a context-aware LSTM captioner and co-attentive discriminator, which enforces semantic alignment between images and captions.We investigate the viability of two discrete GAN training methods: Self-critical Sequence Training (SCST) and Gumbel Straight-Through (ST) and demonstrate that SCST shows more stable gradient behavior and improved results over Gumbel ST.

图像描述作为一种有条件的GAN训练的新颖架构，还研究了两种离散的GAN训练方法。

一种改进的用于图像描述的GAN模型，提出了一种上下文感知的LSTM标题机，引入了性能更强的共同注意判别器，并使用SCST进行GAN训练。


We present Newtonian Monte Carlo (NMC), a method to improve Markov Chain Monte Carlo (MCMC) convergence by analyzing the first and second order gradients of the target density to determine a suitable proposal density at each point.Existing first order gradient-based methods suffer from the problem of determining an appropriate step size.Too small a step size and it will take a large number of steps to converge, while a very large step size will cause it to overshoot the high density region.NMC is similar to the Newton-Raphson update in optimization where the second order gradient is used to automatically scale the step size in each dimension.However, our objective is not to find a maxima but instead to find a parameterized density that can best match the local curvature of the target density.  This parameterized density is then used as a single-site Metropolis-Hastings proposal.As a further improvement on first order methods, we show that random variables with constrained supports don't need to be transformed before taking a gradient step.NMC directly matches constrained random variables to a proposal density with the same support thus keeping the curvature of the target density intact.We demonstrate the efficiency of NMC on a number of different domains.For statistical models where the prior is conjugate to the likelihood, our method recovers the posterior quite trivially in one step.However, we also show results on fairly large non-conjugate models, where NMC performs better than adaptive first order methods such as NUTS or other inexact scalable inference methods such as Stochastic Variational Inference or bootstrapping.

利用曲率使MCMC方法的收敛速度快于现有技术水平。


Neural Tangents is a library designed to enable research into infinite-width neural networks.It provides a high-level API for specifying complex and hierarchical neural network architectures.These networks can then be trained and evaluated either at finite-width as usual or in their infinite-width limit.Infinite-width networks can be trained analytically using exact Bayesian inference or using gradient descent via the Neural Tangent Kernel.Additionally, Neural Tangents provides tools to study gradient descent training dynamics of wide but finite networks in either function space or weight space. The entire library runs out-of-the-box on CPU, GPU, or TPU.All computations can be automatically distributed over multiple accelerators with near-linear scaling in the number of devices. Neural Tangents is availableathttps://www.github.com/google/neural-tangentsWe also provide an accompanying interactive Colab notebook athttps://colab.sandbox.google.com/github/google/neural-tangents/blob/master/notebooks/neural_tangents_cookbook.ipynb

Keras适用于无限的神经网络。


Deep neural networks have achieved great success in classiï¬cation tasks during the last years.However, one major problem to the path towards artiï¬cial intelligence is the inability of neural networks to accurately detect samples from novel class distributions and therefore, most of the existent classiï¬cation algorithms assume that all classes are known prior to the training stage.In this work, we propose a methodology for training a neural network that allows it to efï¬ciently detect out-of-distribution (OOD) examples without compromising much of its classiï¬cation accuracy on the test examples from known classes.Based on the Outlier Exposure (OE) technique, we propose a novel loss function that achieves state-of-the-art results in out-of-distribution detection with OE both on image and text classiï¬cation tasks.Additionally, the way this method was constructed makes it suitable for training any classiï¬cation algorithm that is based on Maximum Likelihood methods.

我们提出了一个新的损失函数，在图像和文本分类任务中，通过Outlier Exposure实现了最先进的异常检测结果。

本文通过调整Outlier Exposure技术的损失函数来解决异常检测和模型校准的问题，结果表明在视觉和文本基准上的性能比OE有所提高，并改进了模型校准。

提出了一个新的损失函数，用于训练具有Outlier Exposure的网络，与使用KL发散的简单损失函数相比，它产生了更好的OOD检测。


Navigation is crucial for animal behavior and is assumed to require an internal representation of the external environment, termed a cognitive map.The precise form of this representation is often considered to be a metric representation of space.An internal representation, however, is judged by its contribution to performance on a given task, and may thus vary between different types of navigation tasks.Here we train a recurrent neural network that controls an agent performing several navigation tasks in a simple environment.To focus on internal representations, we split learning into a task-agnostic pre-training stage that modifies internal connectivity and a task-specific Q learning stage that controls the network's output.We show that pre-training shapes the attractor landscape of the networks, leading to either a continuous attractor, discrete attractors or a disordered state.These structures induce bias onto the Q-Learning phase, leading to a performance pattern across the tasks corresponding to metric and topological regularities.Our results show that, in recurrent networks, inductive bias takes the form of attractor landscapes -- which can be shaped by pre-training and analyzed using dynamical systems methods.Furthermore, we demonstrate that non-metric representations are useful for navigation tasks.

任务无关的预训练可以塑造RNN的吸引子景观，并为不同的导航任务形成不同的感应偏向

本文研究了在导航任务上训练的循环神经网络的内部表征，发现预先训练使用路径整合的RNN包含二维连续吸引子，而预先训练地标记忆的RNN包含离散吸引子。

本文探讨了在不同的导航目标上对循环网络进行预训练如何为解决下游任务带来不同的好处，并展示了不同的预训练如何在预训练后表现为网络中不同的动态结构。


Formal verification of machine learning models has attracted attention recently, and significant progress has been made on proving simple properties like robustness to small perturbations of the input features.In this context, it has also been observed that folding the verification procedure into training makes it easier to train verifiably robust models.In this paper, we extend the applicability of verified training by extending it to (1) recurrent neural network architectures and (2) complex specifications that go beyond simple adversarial robustness, particularly specifications that capture temporal properties like requiring that a robot periodically visits a charging station or that a language model always produces sentences of bounded length.Experiments show that while models trained using standard training often violate desired specifications, our verified training method produces models that both perform well (in terms of test error or reward) and can be shown to be provably consistent with specifications.

时间属性和序列生成模型的神经网络验证

本文将区间约束传播扩展到循环计算和自动回归模型，介绍并扩展了用于指定时间约束的信号时态逻辑，并提供了带有约束传播的STL可以确保神经模型符合时间规范的证明。

根据信号时态逻辑定义的一组规则，可核查地训练时间序列回归器的方法，以及在推导STL语言的约束传播规则方面的方法。


Neural Network (NN) has achieved state-of-the-art performances in many tasks within image, speech, and text domains.Such great success is mainly due to special structure design to fit the particular data patterns, such as CNN capturing spatial locality and RNN modeling sequential dependency.Essentially, these specific NNs achieve good performance by leveraging the prior knowledge over corresponding domain data.Nevertheless, there are many applications with all kinds of tabular data in other domains.Since there are no shared patterns among these diverse tabular data, it is hard to design specific structures to fit them all.Without careful architecture design based on domain knowledge, it is quite challenging for NN to reach satisfactory performance in these tabular data domains.To fill the gap of NN in tabular data learning, we propose a universal neural network solution, called TabNN, to derive effective NN architectures for tabular data in all kinds of tasks automatically.Specifically, the design of TabNN follows two principles: \emph{to explicitly leverages expressive feature combinations} and \emph{to reduce model complexity}.Since GBDT has empirically proven its strength in modeling tabular data, we use GBDT to power the implementation of TabNN.Comprehensive experimental analysis on a variety of tabular datasets demonstrate that TabNN can achieve much better performance than many baseline solutions.

我们提出了一个通用的神经网络解决方案，为表格数据自动得出有效的NN架构。

一种为表格数据而设计的新的神经网络训练程序，旨在利用从GBDTs中提取的特征集群。

提议使用梯度提升决策树和深度神经网络的混合机器学习算法，打算在表格数据上进行研究方向。


Knowledge Bases (KBs) are becoming increasingly large, sparse and probabilistic.These KBs are typically used to perform query inferences and rule mining.But their efficacy is only as high as their completeness.Efficiently utilizing incomplete KBs remains a major challenge as the current KB completion techniques either do not take into account the inherent uncertainty associated with each KB tuple or do not scale to large KBs.Probabilistic rule learning not only considers the probability of every KB tuple but also tackles the problem of KB completion in an explainable way.For any given probabilistic KB, it learns probabilistic first-order rules from its relations to identify interesting patterns.But, the current probabilistic rule learning techniques perform grounding to do probabilistic inference for evaluation of candidate rules.It does not scale well to large KBs as the time complexity of inference using grounding is exponential over the size of the KB.In this paper, we present SafeLearner -- a scalable solution to probabilistic KB completion that performs probabilistic rule learning using lifted probabilistic inference -- as faster approach instead of grounding. We compared SafeLearner to the state-of-the-art probabilistic rule learner ProbFOIL+ and to its deterministic contemporary AMIE+ on standard probabilistic KBs of NELL (Never-Ending Language Learner) and Yago.Our results demonstrate that SafeLearner scales as good as AMIE+ when learning simple rules and is also significantly faster than ProbFOIL+.

使用提升推理的概率规则学习系统

一个概率规则学习的模型，用于自动完成概率数据库，该模型使用AMIE+和提升推理来帮助计算效率。


Recent efforts in Dialogue State Tracking (DST) for task-oriented dialogues have progressed toward open-vocabulary or generation-based approaches where the models can generate slot value candidates from the dialogue history itself.These approaches have shown good performance gain, especially in complicated dialogue domains with dynamic slot values.However, they fall short in two aspects: (1) they do not allow models to explicitly learn signals across domains and slots to detect potential dependencies among \textit{(domain, slot)} pairs; and (2) existing models follow auto-regressive approaches which incur high time cost when the dialogue evolves over multiple domains and multiple turns.In this paper, we propose a novel framework of Non-Autoregressive Dialog State Tracking (NADST) which can factor in potential dependencies among domains and slots to optimize the models towards better prediction of dialogue states as a complete set rather than separate slots.In particular, the non-autoregressive nature of our method not only enables decoding in parallel to significantly reduce the latency of DST for real-time dialogue response generation, but also detect dependencies among slots at token level in addition to slot and domain level.Our empirical results show that our model achieves the state-of-the-art joint accuracy across all domains on the MultiWOZ 2.1 corpus, and the latency of our model is an order of magnitude lower than the previous state of the art as the dialogue history extends over time.

我们提出了第一个用于对话状态跟踪（DST）的非自回归神经模型，在MultiWOZ2.1基准上达到了SOTA的准确性（49.04%），并将推理延迟降低了一个数量级。

一种用于DST任务的新模型，通过非自回归解码器降低推理时间的复杂性，获得有竞争力的DST精度，并显示出比其他基线的改进。

提议建立一个能够以非循环方式追踪对话状态的模型。


The 3D-zoom operation is the positive translation of the camera in the Z-axis, perpendicular to the image plane.In contrast, the optical zoom changes the focal length and the digital zoom is used to enlarge a certain region of an image to the original image size.In this paper, we are the first to formulate an unsupervised 3D-zoom learning problem where images with an arbitrary zoom factor can be generated from a given single image.An unsupervised framework is convenient, as it is a challenging task to obtain a 3D-zoom dataset of natural scenes due to the need for special equipment to ensure camera movement is restricted to the Z-axis.Besides, the objects in the scenes should not move when being captured, which hinders the construction of a large dataset of outdoor scenes.We present a novel unsupervised framework to learn how to generate arbitrarily 3D-zoomed versions of a single image, not requiring a 3D-zoom ground truth, called the Deep 3D-Zoom Net.The Deep 3D-Zoom Net incorporates the following features:(i) transfer learning from a pre-trained disparity estimation network via a back re-projection reconstruction loss;(ii) a fully convolutional network architecture that models depth-image-based rendering (DIBR), taking into account high-frequency details without the need for estimating the intermediate disparity; and(iii) incorporating a discriminator network that acts as a no-reference penalty for unnaturally rendered areas.Even though there is no baseline to fairly compare our results, our method outperforms previous novel view synthesis research in terms of realistic appearance on large camera baselines.We performed extensive experiments to verify the effectiveness of our method on the KITTI and Cityscapes datasets.

一种新颖的网络架构，用于执行深度3D缩放或特写。

一种为给定的输入图像创建 "放大图像 "的方法，以及一种新颖的反投影重建损失，使网络能够学习潜在的三维结构并保持自然外观。

当摄像机向前移动时，合成三维变焦行为的算法，在GANs框架中纳入差异估计的网络结构，以合成新的视图，以及提议的新电脑视觉任务。


The universal approximation theorem, in one of its most general versions, says that if we consider only continuous activation functions Ïƒ, then a standard feedforward neural network with one hidden layer is able to approximate any continuous multivariate function f to any given approximation threshold Îµ, if and only if Ïƒ is non-polynomial.In this paper, we give a direct algebraic proof of the theorem.Furthermore we shall explicitly quantify the number of hidden units required for approximation.Specifically, if X in R^n is compact, then a neural network with n input units, m output units, and a single hidden layer with {n+d choose d} hidden units (independent of m and Îµ), can uniformly approximate any polynomial function f:X -> R^m whose total degree is at most d for each of its m coordinate functions.In the general case that f is any continuous function, we show there exists some N in O(Îµ^{-n}) (independent of m), such that N hidden units would suffice to approximate f.We also show that this uniform approximation property (UAP) still holds even under seemingly strong conditions imposed on the weights.We highlight several consequences:(i) For any Î´ > 0, the UAP still holds if we restrict all non-bias weights w in the last layer to satisfy |w| < Î´.(ii) There exists some Î»>0 (depending only on f and Ïƒ), such that the UAP still holds if we restrict all non-bias weights w in the first layer to satisfy |w|>Î».(iii) If the non-bias weights in the first layer are *fixed* and randomly chosen from a suitable range, then the UAP holds with probability 1.

通过代数方法对普遍近似定理进行量化改良。

作者以代数方式推导出通用近似属性证明，并断言这些结果通用于其他种类的神经网络和类似的学习者。

Leshno版本的神经网络通用近似属性的新证明，以及对通用近似属性的新见解。


In this paper, we design a generic framework for learning a robust text classification model that achieves accuracy comparable to standard full models under test-timebudget constraints.We take a different approach from existing methods and learn to dynamically delete a large fraction of unimportant words by a low-complexity selector such that the high-complexity classifier only needs to process a small fraction of important words.In addition, we propose a new data aggregation method to train the classifier, allowing it to make accurate predictions even on fragmented sequence of words.Our end-to-end method achieves state-of-the-art performance while its computational complexity scales linearly with the small fraction of important words in the whole corpus.Besides, a single deep neural network classifier trained by our framework can be dynamically tuned to different budget levels at inference time.

模块化的文档分类框架和数据聚合技术，使框架对各种失真和噪音具有稳定性，并只关注重要的词。

作者考虑在测试时间预测存在资源限制的情况下，训练基于RNN的文本分类，并提供了一种使用掩蔽机制的方法，以减少预测中使用的单词/短语/句子，然后用分类器来处理这些成分。


Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-net and searching for an optimal architecture.In this paper, we present a novel approach, namely  Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance.In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut.This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels.We solve it by introducing  edge normalization, which adds a new set of edge-level hyper-parameters to reduce uncertainty in search.Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoy both faster speed and higher training stability.Experiment results demonstrate the effectiveness of the proposed method.Specifically, we achieve an error rate of 2.57% on CIFAR10 within merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet (under the mobile setting) within 3.8 GPU-days for search.Our code has been made available at https://www.dropbox.com/sh/on9lg3rpx1r6dkf/AABG5mt0sMHjnEJyoRnLEYW4a?dl=0.

允许超级网络中的部分通道连接，以规范和加速可微分的架构搜索

神经结构搜索方法DARTS的扩展，通过使用通道的随机子集和规范化边缘的方法来解决其巨大的内存成本的缺点。

本文从大的内存和计算开销出发，提出在训练效率方面改进DARTS，并提出了一种部分通道连接和边缘正规化的DARTS。


Dialogue research tends to distinguish between chit-chat and goal-oriented tasks.While the former is arguably more naturalistic and has a wider use of language, the latter has clearer metrics and a more straightforward learning signal.Humans effortlessly combine the two, and engage in chit-chat for example with the goal of exchanging information or eliciting a specific response.Here, we bridge the divide between these two domains in the setting of a rich multi-player text-based fantasy environment where agents and humans engage in both actions and dialogue.Specifically, we train a goal-oriented model with reinforcement learning via self-play against an imitation-learned chit-chat model with two new approaches: the policy either learns to pick a topic or learns to pick an utterance given the top-k utterances.We show that both models outperform a strong inverse model baseline and can converse naturally with their dialogue partner in order to achieve goals.

代理人进行互动（说话、行动），并能在一个具有多样化语言的丰富世界中实现目标，连接了闲聊和目标导向的对话之间的差距。

本文研究了一个多代理的对话任务，在这个任务中，学习代理的目的是产生自然语言行动，以引起另一个代理的特定行动，并显示RL-代理可以达到比模仿学习基线更高的任务完成水平。

本文探讨了以目标为导向的对话设置与强化学习在一个科幻文本冒险游戏中的应用，并观察到RL方法优于监督学习模型。


We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies.Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation.We propose estimated mixture policy (EMP), a novel class of partially policy-agnostic methods to accurately estimate those quantities.With careful analysis, we show that EMP gives rise to estimates with reduced variance for estimating the state stationary distribution correction while it also offers a useful induction bias for estimating the state-action stationary distribution correction.In extensive experiments with both continuous and discrete environments, we demonstrate that our algorithm offers significantly improved accuracy compared to the state-of-the-art methods.

一种新的部分政策不可知的方法，用于具有多个已知或未知行为政策的无限期非政策政策评估。

一种估计的混合政策，它采用了非政策性政策评价的无限期估计器和重要性权重的回归重要性抽样的思想，并将其扩展到许多政策和未知政策。

一种通过估计回归下的混合政策来解决具有多种行为政策的无限期非政策评价的算法，以及估计政策比率可以减少方差的理论证明。


We introduce a more efficient neural architecture for amortized inference, which combines continuous and conditional normalizing flows using a principled choice of structure.Our gradient flow derives its sparsity pattern from the minimally faithful inverse of its underlying graphical model.We find that this factorization reduces the necessary numbers both of parameters in the neural network and of adaptive integration steps in the ODE solver.Consequently, the throughput at training time and inference time is increased, without decreasing performance in comparison to unconstrained flows.By expressing the structural inversion and the flow construction as compilation passes of a probabilistic programming language, we demonstrate their applicability to the stochastic inversion of realistic models such as convolutional neural networks (CNN).

我们为摊销推理引入了一个更有效的神经架构，它利用原则性的稀疏结构选择将连续和条件标准化流结合起来。


We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way.By defining the combinatorial search space of NAS to be the set of different edge-partitionings (colorings) into same-weight classes, we represent compact architectures via efficient learned edge-partitionings.For several RL tasks, we manage to learn colorings translating to effective policies parameterized by as few as 17 weight parameters, providing >90 % compression over vanilla policies and 6x compression over state-of-the-art compact policies based on Toeplitz matrices, while still maintaining good reward.We believe that our work is one of the first attempts to propose a rigorous approach to training structured neural network architectures for RL problems that are of interest especially in mobile robotics with limited storage and computational resources.

我们表明，在RL中具有ES优化的ENAS是高度可扩展的，并利用它通过权重共享来压缩神经网络策略。

作者通过压缩前馈神经网络，迫使其共享权重，并使用强化学习方法学习共享权重的映射，构建了极少参数的强化学习策略。

本文结合了ENAS和ES方法的优化想法，并引入了彩色网络架构，将RL网络的权重划分为绑定的子组。