We propose a novel  adversarial learning framework in this work.Existing adversarial learning methods involve two separate networks, i.e., the structured prediction models and the discriminative models, in the training.The information captured by discriminative models complements that in the structured prediction models, but few existing researches have studied on utilizing such information to improve structured prediction models at the inference stage.In this work, we propose to refine the predictions of structured prediction models by effectively integrating discriminative models into the prediction.Discriminative models are treated as energy-based models.Similar to the adversarial learning, discriminative models are trained to estimate scores which measure the quality of predicted outputs, while structured prediction models are trained to predict contrastive outputs with maximal energy scores.In this way, the gradient vanishing problem is ameliorated, and thus we are able to perform inference by following the ascent gradient directions of discriminative models to refine structured prediction models.The proposed method is able to handle a range of tasks, \emph{e.g.}, multi-label classification and image segmentation.  Empirical results on these two tasks validate the effectiveness of our learning method.

Wir schlagen ein neuartiges adversariales Lernverfahren für strukturierte Vorhersagen vor, bei dem diskriminative Modelle zur Verfeinerung strukturierter Vorhersagemodelle in der Inferenzphase verwendet werden können.


We propose RaPP, a new methodology for novelty detection by utilizing hidden space activation values obtained from a deep autoencoder.Precisely, RaPP compares input and its autoencoder reconstruction not only in the input space but also in the hidden spaces.We show that if we feed a reconstructed input to the same autoencoder again, its activated values in a hidden space are equivalent to the corresponding reconstruction in that hidden space given the original input.In order to aggregate the hidden space activation values, we propose two metrics, which enhance the novelty detection performance.Through extensive experiments using diverse datasets, we validate that RaPP improves novelty detection performances of autoencoder-based approaches.Besides, we show that RaPP outperforms recent novelty detection methods evaluated on popular benchmarks.

Eine neue Methode zur Erkennung von Neuheiten durch die Verwendung von Aktivierungswerten im verborgenen Raum, die von einem tiefen Autoencoder erhalten werden.


Learning preferences of users over plan traces can be a challenging task given a large number of features and limited queries that we can ask a single user.Additionally, the preference function itself can be quite convoluted and non-linear.Our approach uses feature-directed active learning to gather the necessary information about plan trace preferences.This data is used to train a simple feedforward neural network to learn preferences over the sequential data.We evaluate the impact of active learning on the number of traces that are needed to train a model that is accurate and interpretable.This evaluation is done by comparing the aforementioned feedforward network to a more complex neural network model that uses LSTMs and is trained with a larger dataset without active learning.

Lernen von Präferenzen über Planspuren durch aktives Lernen.


Reinforcement learning is a promising framework for solving control problems, but its use in practical situations is hampered by the fact that reward functions are often difficult to engineer.Specifying goals and tasks for autonomous machines, such as robots, is a significant challenge: conventionally, reward functions and goal states have been used to communicate objectives.But people can communicate objectives to each other simply by describing or demonstrating them.How can we build learning algorithms that will allow us to tell machines what we want them to do?In this work, we investigate the problem of grounding language commands as reward functions using inverse reinforcement learning, and argue that language-conditioned rewards are more transferable than language-conditioned policies to new environments.We propose language-conditioned reward learning (LC-RL), which grounds language commands as a reward function represented by a deep neural network.We demonstrate that our model learns rewards that transfer to novel tasks and environments on realistic, high-dimensional visual environments with natural language commands, whereas directly learning a language-conditioned policy leads to poor performance.

Wir erden Sprachbefehle in einer hochdimensionalen visuellen Umgebung, indem wir sprachbedingte Belohnungen durch inverses Reinforcement Learning erlernen.


Many biological learning systems such as the mushroom body, hippocampus, and cerebellum are built from sparsely connected networks of neurons.For a new understanding of such networks, we study the function spaces induced by sparse random features and characterize what functions may and may not be learned.A network with d inputs per neuron is found to be equivalent to an additive model of order d, whereas with a degree distribution the network combines additive terms of different orders.We identify three specific advantages of sparsity: additive function approximation is a powerful inductive bias that limits the curse of dimensionality, sparse networks are stable to outlier noise in the inputs, and sparse random features are scalable.Thus, even simple brain architectures can be powerful function approximators.Finally, we hope that this work helps popularize kernel theories of networks among computational neuroscientists.

Wir plädieren für zufällige Merkmale als Theorie biologischer neuronaler Netze und konzentrieren uns dabei auf spärlich verbundene Netze.


We propose a new application of embedding techniques to problem retrieval in adaptive tutoring.The objective is to retrieve problems similar in mathematical concepts.There are two challenges: First, like sentences, problems helpful to tutoring are never exactly the same in terms of the underlying concepts.Instead, good problems mix concepts in innovative ways, while still displaying continuity in their relationships.Second, it is difficult for humans to determine a similarity score consistent across a large enough training set.We propose a hierarchical problem embedding algorithm, called Prob2Vec, that consists of an abstraction and an embedding step.Prob2Vec achieves 96.88\% accuracy on a problem similarity test, in contrast to 75\% from directly applying state-of-the-art sentence embedding methods.It is surprising that Prob2Vec is able to distinguish very fine-grained differences among problems, an ability humans need time and effort to acquire.In addition, the sub-problem of concept labeling with imbalanced training data set is interesting in its own right.It is a multi-label problem suffering from dimensionality explosion, which we propose ways to ameliorate.We propose the novel negative pre-training algorithm that dramatically reduces false negative and positive ratios for classification, using an imbalanced training data set.

Wir schlagen die Prob2Vec-Methode zur Problemeinbettung vor, die in einem personalisierten E-Learning-Tool zusätzlich zu einer Klassifizierungsmethode auf Datenebene, dem so genannten negativen Pre-Training, eingesetzt wird, wenn der Trainingsdatensatz unausgewogen ist.


We introduce a new deep convolutional neural network, CrescendoNet, by stacking simple building blocks without residual connections.Each Crescendo block contains independent convolution paths with increased depths.The numbers of convolution layers and parameters are only increased linearly in Crescendo blocks.In experiments, CrescendoNet with only 15 layers outperforms almost all networks without residual connections on benchmark datasets, CIFAR10, CIFAR100, and SVHN.Given sufficient amount of data as in SVHN dataset, CrescendoNet with 15 layers and 4.1M parameters can match the performance of DenseNet-BC with 250 layers and 15.3M parameters.CrescendoNet provides a new way to construct high performance deep convolutional neural networks without residual connections.Moreover, through investigating the behavior and performance of subnetworks in CrescendoNet, we note that the high performance of CrescendoNet may come from its implicit ensemble behavior, which differs from the FractalNet that is also a deep convolutional neural network without residual connections.Furthermore, the independence between paths in CrescendoNet allows us to introduce a new path-wise training procedure, which can reduce the memory needed for training.

Wir stellen CrescendoNet vor, eine tiefe CNN-Architektur durch das Stapeln einfacher Bausteine ohne Residual Connections.


Gaussian processes are the leading class of distributions on random functions, but they suffer from well known issues including difficulty scaling and inflexibility with respect to certain shape constraints (such as nonnegativity).Here we propose Deep Random Splines, a flexible class of random functions obtained by transforming Gaussian noise through a deep neural network whose output are the parameters of a spline.Unlike Gaussian processes, Deep Random Splines allow us to readily enforce shape constraints while inheriting the richness and tractability of deep generative models.We also present an observational model for point process data which uses Deep Random Splines to model the intensity function of each point process and apply it to neuroscience data to obtain a low-dimensional representation of spiking activity.Inference is performed via a variational autoencoder that uses a novel recurrent encoder architecture that can handle multiple point processes as input.

Wir kombinieren Splines mit neuronalen Netzen, um eine neuartige Verteilung über Funktionen zu erhalten und sie zur Modellierung von Intensitätsfunktionen von Punktprozessen zu verwenden.


The recent development of Natural Language Processing (NLP) has achieved great success using large pre-trained models with hundreds of millions of parameters.However, these models suffer from the heavy model size and high latency such that we cannot directly deploy them to resource-limited mobile devices.In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model.Like BERT, MobileBERT is task-agnostic; that is, it can be universally applied to various downstream NLP tasks via fine-tuning.MobileBERT is a slimmed version of BERT-LARGE augmented with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.To train MobileBERT, we use a bottom-to-top progressive scheme to transfer the intrinsic knowledge of a specially designed Inverted Bottleneck BERT-LARGE teacher to it.Empirical studies show that MobileBERT is 4.3x smaller and 4.0x faster than original BERT-BASE while achieving competitive results on well-known NLP benchmarks.On the natural language inference tasks of GLUE, MobileBERT achieves 0.6 GLUE score performance degradation, and 367 ms latency on a Pixel 3 phone.On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a 90.0/79.2 dev F1 score, which is 1.5/2.1 higher than BERT-BASE.

Wir entwickeln einen aufgabenspezifisch komprimierten BERT, der 4,3x kleiner und 4,0x schneller als BERT-BASE ist und gleichzeitig eine konkurrenzfähige Leistung auf GLUE und SQuAD erreicht.


The importance weighted autoencoder (IWAE) (Burda et al., 2016) is a popular variational-inference method which achieves a tighter evidence bound (and hence a lower bias) than standard variational autoencoders by optimising a multi-sample objective, i.e. an objective that is expressible as an integral over $K > 1$ Monte Carlo samples.Unfortunately, IWAE crucially relies on the availability of reparametrisations and even if these exist, the multi-sample objective leads to inference-network gradients which break down as $K$ is increased (Rainforth et al., 2018).This breakdown can only be circumvented by removing high-variance score-function terms, either by heuristically ignoring them (which yields the 'sticking-the-landing' IWAE (IWAE-STL) gradient from Roeder et al. (2017)) or through an identity from Tucker et al. (2019) (which yields the 'doubly-reparametrised' IWAE (IWAE-DREG) gradient).In this work, we argue that directly optimising the proposal distribution in importance sampling as in the reweighted wake-sleep (RWS) algorithm from Bornschein & Bengio (2015) is preferable to optimising IWAE-type multi-sample objectives.To formalise this argument, we introduce an adaptive-importance sampling framework termed adaptive importance sampling for learning (AISLE) which slightly generalises the RWS algorithm.We then show that AISLE admits IWAE-STL and IWAE-DREG (i.e. the IWAE-gradients which avoid breakdown) as special cases.

Wir zeigen, dass sich die meisten Varianten von importance-weighted Autoencodern auf prinzipiellere Weise als Spezialfälle von adaptiven importance-sampling Ansätzen wie dem reweighted-wake sleep Algorithmus ableiten lassen.


As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed on clusters to perform model fitting in parallel.Alistarh et al. (2017) describe two variants of data-parallel SGD that quantize and encode gradients to lessen communication costs.For the first variant, QSGD, they provide strong theoretical guarantees.For the second variant, which we call QSGDinf, they demonstrate impressive empirical gains for distributed training of large neural networks.Building on their work, we propose an alternative scheme for quantizing gradients and show that it yields stronger theoretical guarantees than exist for QSGD while matching the empirical performance of QSGDinf.

NUQSGD schließt die Lücke zwischen den theoretischen Garantien von QSGD und der empirischen Leistung von QSGDinf.


The impressive lifelong learning in animal brains is primarily enabled by plastic changes in synaptic connectivity.Importantly, these changes are not passive, but are actively controlled by neuromodulation, which is itself under the control of the brain.The resulting self-modifying abilities of the brain play an important role in learning and adaptation, and are a major basis for biological reinforcement learning.Here we show for the first time that artificial neural networks with such neuromodulated plasticity can be trained with gradient descent.Extending previous work on differentiable Hebbian plasticity, we propose a differentiable formulation for the neuromodulation of plasticity.We show that neuromodulated plasticity improves the performance of neural networks on both reinforcement learning and supervised learning tasks.In one task, neuromodulated plastic LSTMs with millions of parameters outperform standard LSTMs on a benchmark language modeling task (controlling for the number of parameters).We conclude that differentiable neuromodulation of plasticity offers a powerful new framework for training neural networks.

Neuronale Netze können so trainiert werden, dass sie ihre eigene Konnektivität ändern und so ihre Online-Lernleistung bei schwierigen Aufgaben verbessern.


Deep learning has made remarkable achievement in many fields.However, learningthe parameters of neural networks usually demands a large amount of labeleddata.The algorithms of deep learning, therefore, encounter difficulties when appliedto supervised learning where only little data are available.This specific taskis called few-shot learning.To address it, we propose a novel algorithm for fewshotlearning using discrete geometry, in the sense that the samples in a class aremodeled as a reduced simplex.The volume of the simplex is used for the measurementof class scatter.During testing, combined with the test sample and thepoints in the class, a new simplex is formed.Then the similarity between the testsample and the class can be quantized with the ratio of volumes of the new simplexto the original class simplex.Moreover, we present an approach to constructingsimplices using local regions of feature maps yielded by convolutional neural networks.Experiments on Omniglot and miniImageNet verify the effectiveness ofour simplex algorithm on few-shot learning.

Es wird ein geometrisches Verfahren auf Simplex-Basis vorgeschlagen, um Few-Shot Learning Probleme zu bewältigen.


Reservoir computing is a powerful tool to explain how the brain learns temporal sequences, such as movements, but existing learning schemes are either biologically implausible or too inefficient to explain animal performance.We show that a network can learn complicated sequences with a reward-modulated Hebbian learning rule if the network of reservoir neurons is combined with a second network that serves as a dynamic working memory and provides a spatio-temporal backbone signal to the reservoir.In combination with the working memory, reward-modulated Hebbian learning of the readout neurons performs as well as FORCE learning, but with the advantage of a biologically plausible interpretation of both the learning rule and the learning paradigm.

Wir zeigen, dass ein Arbeitsgedächtnis-Input für ein Reservoir-Netzwerk eine lokale belohnungsmodulierte Hebb'sche Regel genauso gut funktionieren lässt wie die rekursive Least-Squares-Regel (auch bekannt als FORCE).


Convolutional architectures have recently been shown to be competitive on manysequence modelling tasks when compared to the de-facto standard of recurrent neural networks (RNNs) while providing computational and modelling advantages due to inherent parallelism.However, currently, there remains a performancegap to more expressive stochastic RNN variants, especially those with several layers of dependent random variables.In this work, we propose stochastic temporal convolutional networks (STCNs), a novel architecture that combines the computational advantages of temporal convolutional networks (TCN) with the representational power and robustness of stochastic latent spaces.In particular, we propose a hierarchy of stochastic latent variables that captures temporal dependencies at different time-scales.The architecture is modular and flexible due to the decoupling of the deterministic and stochastic layers.We show that the proposed architecture achieves state of the art log-likelihoods across several tasks.Finally, the model is capable of predicting high-quality synthetic samples over a long-range temporal horizon in modelling of handwritten text.

Wir kombinieren die rechnerischen Vorteile temporaler Convolutional Architekturen mit der Ausdruckskraft stochastischer latenter Variablen.


The weak contraction mapping is a self mapping that the range is always a subset of the domain, which admits a unique fixed-point.The iteration of weak contraction mapping is a Cauchy sequence that yields the unique fixed-point.A gradient-free optimization method as an application of weak contraction mapping is proposed to achieve global minimum convergence.The optimization method is robust to local minima and initial point position.

Es wird eine gradientenfreie Methode für ein nicht-konvexes Optimierungsproblem vorgeschlagen.


Over the last decade, two competing control strategies have emerged for solving complex control tasks with high efficacy.Model-based control algorithms, such as model-predictive control (MPC) and trajectory optimization, peer into the gradients of underlying system dynamics in order to solve control tasks with high sample efficiency.  However, like all gradient-based numerical optimization methods,model-based control methods are sensitive to intializations and are prone to becoming trapped in local minima.Deep reinforcement learning (DRL), on the other hand, can somewhat alleviate these issues by exploring the solution space through sampling — at the expense of computational cost.In this paper, we present a hybrid method that combines the best aspects of gradient-based methods and DRL.We base our algorithm on the deep deterministic policy gradients (DDPG) algorithm and propose a simple modification that uses true gradients from a differentiable physical simulator to increase the convergence rate of both the actor and the critic.  We demonstrate our algorithm on seven 2D robot control tasks, with the most complex one being a differentiable half cheetah with hard contact constraints.Empirical results show that our method boosts the performance of DDPGwithout sacrificing its robustness to local minima.

Wir schlagen eine neue Methode vor, die die Gradienten von differenzierbaren Simulatoren nutzt, um die Leistung von RL für die Robotersteuerung zu verbessern.


We propose Bayesian hypernetworks: a framework for approximate Bayesian inference in neural networks.A Bayesian hypernetwork, h, is a neural network which learns to transform a simple noise distribution, p(e) = N(0,I), to a distribution q(t) := q(h(e)) over the parameters t of another neural network (the ``primary network).We train q with variational inference, using an invertible h to enable efficient estimation of the variational lower bound on the posterior p(t | D) via sampling.In contrast to most methods for Bayesian deep learning, Bayesian hypernets can represent a complex multimodal approximate posterior with correlations between parameters, while enabling cheap iid sampling of q(t).  In practice, Bayesian hypernets provide a better defense against adversarial examples than dropout, and also exhibit competitive performance on a suite of tasks which evaluate model uncertainty, including regularization, active learning, and anomaly detection.

Wir schlagen Bayes'sche Hypernetze vor: einen Rahmen für approximative Bayes'sche Inferenz in neuronalen Netzen.


Evolutionary-based optimization approaches have recently shown promising results in domains such as Atari and robot locomotion but less so in solving 3D tasks directly from pixels.This paper presents a method called Deep Innovation Protection (DIP) that allows training  complex world models end-to-end for such 3D environments.The main idea behind the approach is to employ multiobjective optimization to temporally reduce the selection pressure on specific components in a world model, allowing other components to adapt.We investigate the emergent representations of these evolved networks, which learn a model of the world without the need for a specific forward-prediction loss.

Deep Innovation Protection ermöglicht die Entwicklung komplexer Weltmodelle für 3D-Aufgaben von Anfang bis Ende.


Adversarial training is one of the most popular ways to learn robust models but is usually attack-dependent and time costly.In this paper, we propose the MACER algorithm, which learns robust models without using adversarial training but performs better than all existing provable l2-defenses.Recent work shows that randomized smoothing can be used to provide certified l2 radius to smoothed classifiers, and our algorithm trains provably robust smoothed classifiers via MAximizing the CErtified Radius (MACER).The attack-free characteristic makes MACER faster to train and easier to optimize.In our experiments, we show that our method can be applied to modern deep neural networks on a wide range of datasets, including Cifar-10, ImageNet, MNIST, and SVHN.For all tasks, MACER spends less training time than state-of-the-art adversarial training algorithms, and the learned models achieve larger average certified radius.

Wir schlagen MACER vor: einen beweisbaren Verteidigungsalgorithmus, der robuste Modelle durch Maximierung des zertifizierten Radius trainiert. Er verwendet kein adversarial Training, ist aber besser als alle bestehenden beweisbaren l2-Verteidigungen.


Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic.However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter.In this paper, we propose a novel actor-critic method called the guide actor-critic (GAC).GAC firstly learns a guide actor that locally maximizes the critic and then it updates the policy parameter based on the guide actor by supervised learning.Our main theoretical contributions are two folds.First, we show that GAC updates the guide actor by performing second-order optimization in the action space where the curvature matrix is based on the Hessians of the critic.Second, we show that the deterministic policy gradient method is a special case of GAC when the Hessians are ignored.Through experiments, we show that our method is a promising reinforcement learning method for continuous controls.

Dieses Papier schlägt eine neuartige Akteur-Kritik-Methode vor, die Hessians einer Kritik verwendet, um einen Akteur zu aktualisieren.


Deep Infomax~(DIM) is an unsupervised representation learning framework by maximizing the mutual information between the inputs and the outputs of an encoder, while probabilistic constraints are imposed on the outputs.In this paper, we propose Supervised Deep InfoMax~(SDIM), which introduces supervised probabilistic constraints to the encoder outputs.The supervised probabilistic constraints are equivalent to a generative classifier on high-level data representations, where class conditional log-likelihoods of samples can be evaluated.Unlike other works building generative classifiers with conditional generative models, SDIMs scale on complex datasets, and can achieve comparable performance with discriminative counterparts.  With SDIM, we could perform \emph{classification with rejection}.Instead of always reporting a class label, SDIM only makes predictions when test samples' largest logits surpass some pre-chosen thresholds, otherwise they will be deemed as out of the data distributions, and be rejected.  Our experiments show that SDIM with rejection policy can effectively reject illegal inputs including out-of-distribution samples and adversarial examples.

Generative Klassifikatoren auf komplexen Datensätzen zu skalieren und ihre Effektivität bei der Zurückweisung illegaler Eingaben, einschließlich Proben außerhalb der Verteilung und gegnerischer Beispiele, zu bewerten.


Abstract In this work, we describe a set of rules for the design and initialization of well-conditioned neural networks, guided by the goal of naturally balancing the diagonal blocks of the Hessian at the start of training.We show how our measure of conditioning of a block relates to another natural measure of conditioning, the ratio of weight gradients to the weights.We prove that for a ReLU-based deep multilayer perceptron, a simple initialization scheme using the geometric mean of the fan-in and fan-out satisfies our scaling rule.For more sophisticated architectures, we show how our scaling principle can be used to guide design choices to produce well-conditioned neural networks, reducing guess-work.

Eine Theorie für die Initialisierung und Skalierung von Schichten eines neuronalen ReLU-Netzes.


We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model.Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al., 2019), we show that the adversary does not need any real training data to successfully mount the attack.In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks including natural language inference and question answering.Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model.Finally, we study two defense strategies against model extraction—membership classification and API watermarking—which while successful against some adversaries can also be circumvented by more clever ones.

Die Ausgaben moderner NLP-APIs auf unsinnigem Text liefern starke Signale über die Interna des Modells, was es Angreifern ermöglicht, die APIs zu stehlen.


We propose SEARNN, a novel training algorithm for recurrent neural networks (RNNs) inspired by the "learning to search" (L2S) approach to structured prediction.RNNs have been widely successful in structured prediction applications such as machine translation or parsing, and are commonly trained using maximum likelihood estimation (MLE).Unfortunately, this training loss is not always an appropriate surrogate for the test error: by only maximizing the ground truth probability, it fails to exploit the wealth of information offered by structured losses.Further, it introduces discrepancies between training and predicting (such as exposure bias) that may hurt test performance.Instead, SEARNN leverages test-alike search space exploration to introduce global-local losses that are closer to the test error.We first demonstrate improved performance over MLE on two different tasks: OCR and spelling correction.Then, we propose a subsampling strategy to enable SEARNN to scale to large vocabulary sizes.This allows us to validate the benefits of our approach on a machine translation task.

Wir stellen SeaRNN vor, einen neuartigen Algorithmus für das RNN-Training, inspiriert durch den Ansatz des "Learning to Search" zur strukturierten Vorhersage, um die Einschränkungen des MLE-Trainings zu vermeiden.


Deep reinforcement learning has demonstrated increasing capabilities for continuous control problems,including agents that can move with skill and agility through their environment. An open problem in this setting is that of developing good strategies for integrating or merging policiesfor multiple skills, where each individual skill is a specialist in a specific skill and its associated state distribution. We extend policy distillation methods to the continuous action setting and leverage this technique to combine \expert policies,as evaluated in the domain of simulated bipedal locomotion across different classes of terrain.We also introduce an input injection method for augmenting an existing policy network to exploit new input features.Lastly, our method uses transfer learning to assist in the efficient acquisition of new skills.The combination of these methods allows a policy to be incrementally augmented with new skills.We compare our progressive learning and integration via distillation (PLAID) methodagainst three alternative baselines.

Eine Methode des kontinuierlichen Lernens, die mit Hilfe von Destillation Expertenstrategien und Transferlernen kombiniert, um das Erlernen neuer Fähigkeiten zu beschleunigen.


Deep Reinforcement Learning (DRL) has led to many recent breakthroughs on complex control tasks, such as defeating the best human player in the game of Go.However, decisions made by the DRL agent are not explainable, hindering its applicability in safety-critical settings.Viper, a recently proposed technique, constructs a decision tree policy by mimicking the DRL agent.Decision trees are interpretable as each action made can be traced back to the decision rule path that lead to it.However, one global decision tree approximating the DRL policy has significant limitations with respect to the geometry of decision boundaries.We propose MoET, a more expressive, yet still interpretable model based on Mixture of Experts, consisting of a gating function that partitions the state space, and multiple decision tree experts that specialize on different partitions.We propose a training procedure to support non-differentiable decision tree experts and integrate it into imitation learning procedure of Viper.We evaluate our algorithm on four OpenAI gym environments, and show that the policy constructed in such a way is more performant and better mimics the DRL agent by lowering mispredictions and increasing the reward.We also show that MoET policies are amenable for verification using off-the-shelf automated theorem provers such as Z3.

Erklärbares Reinforcement-Learning-Modell unter Verwendung einer neuartigen Kombination von Expertenmischungen mit nicht-differenzierbaren Entscheidungsbaum-Experten.


We consider the problem of unconstrained minimization of a smooth objectivefunction in $\mathbb{R}^d$ in setting where only function evaluations are possible.We propose and analyze stochastic zeroth-order method with heavy ball momentum.In particular, we propose, SMTP, a momentum version of the stochastic three-point method (STP) Bergou et al. (2019).We show new complexity results for non-convex, convex and strongly convex functions.We test our method on a collection of learning to continuous control tasks on several MuJoCo Todorov et al. (2012) environments with varying difficulty and compare against STP, other state-of-the-art derivative-free optimization algorithms and against policy gradient methods.SMTP significantly outperforms STP and all other methods that we considered in our numerical experiments.Our second contribution is SMTP with importance sampling which we call SMTP_IS.We provide convergence analysis of this method for non-convex, convex and strongly convex objectives.

Wir entwickeln und analysieren einen neuen ableitungsfreien Optimierungsalgorithmus mit Impuls- und Wichtigkeitsabtastung mit Anwendungen auf kontinuierliche Steuerung.


Using class labels to represent class similarity is a typical approach to training deep hashing systems for retrieval; samples from the same or different classes take binary 1 or 0 similarity values.This similarity does not model the full rich knowledge of semantic relations that may be present between data points.In this work we build upon the idea of using semantic hierarchies to form distance metrics between all available sample labels; for example cat to dog has a smaller distance than cat to guitar.We combine this type of semantic distance into a loss function to promote similar distances between the deep neural network embeddings.We also introduce an empirical Kullback-Leibler divergence loss term to promote binarization and uniformity of the embeddings.We test the resulting SHREWD method and demonstrate improvements in hierarchical retrieval scores using compact, binary hash codes instead of real valued ones, and show that in a weakly supervised hashing setting we are able to learn competitively without explicitly relying on class labels, but instead on similarities between labels.

Wir schlagen eine neue Methode für das Training von Deep Hashing für die Bildsuche vor, bei der nur eine relationale Abstandsmetrik zwischen Proben verwendet wird.


In this paper, we propose a novel approach to improve a given surface mapping through local refinement.The approachreceives an established mapping between two surfaces and follows four phases:(i) inspection of the mapping and creation of a sparseset of landmarks in mismatching regions;(ii) segmentation with a low-distortion region-growing process based on flattening thesegmented parts;(iii) optimization of the deformation of segmented parts to align the landmarks in the planar parameterization domain;and(iv) aggregation of the mappings from segments to update the surface mapping.In addition, we propose a new method to deform themesh in order to meet constraints (in our case, the landmark alignment of phase(iii)).We incrementally adjust the cotangent weights forthe constraints and apply the deformation in a fashion that guarantees that the deformed mesh will be free of flipped faces and will havelow conformal distortion.Our new deformation approach, Iterative Least Squares Conformal Mapping (ILSCM), outperforms otherlow-distortion deformation methods.The approach is general, and we tested it by improving the mappings from different existing surfacemapping methods.We also tested its effectiveness by editing the mappings for a variety of 3D objects.

Wir schlagen einen neuartigen Ansatz zur Verbesserung einer gegebenen flächenübergreifenden Abbildung durch lokale Verfeinerung mit einer neuen iterativen Methode zur Verformung des Netzes vor, um die Benutzeranforderungen zu erfüllen.


Understanding the groundbreaking performance of Deep Neural Networks is oneof the greatest challenges to the scientific community today.In this work, weintroduce an information theoretic viewpoint on the behavior of deep networksoptimization processes and their generalization abilities.By studying the InformationPlane, the plane of the mutual information between the input variable andthe desired label, for each hidden layer.Specifically, we show that the training ofthe network is characterized by a rapid increase in the mutual information (MI)between the layers and the target label, followed by a longer decrease in the MIbetween the layers and the input variable.Further, we explicitly show that thesetwo fundamental information-theoretic quantities correspond to the generalizationerror of the network, as a result of introducing a new generalization bound that isexponential in the representation compression.The analysis focuses on typicalpatterns of large-scale problems.For this purpose, we introduce a novel analyticbound on the mutual information between consecutive layers in the network.An important consequence of our analysis is a super-linear boost in training timewith the number of non-degenerate hidden layers, demonstrating the computationalbenefit of the hidden layers.

Einführung eines informationstheoretischen Blickwinkels auf das Verhalten von Optimierungsprozessen in tiefen Netzwerken und deren Generalisierungsfähigkeiten.


We propose and study a method for learning interpretable representations for the task of regression.Features are represented as networks of multi-type expression trees comprised of activation functions common in neural networks in addition to other elementary functions.Differentiable features are trained via gradient descent, and the performance of features in a linear model is used to weight the rate of change among subcomponents of each representation.The search process maintains an archive of representations with accuracy-complexity trade-offs to assist in generalization and interpretation.We compare several stochastic optimization approaches within this framework.We benchmark these variants on 100 open-source regression problems in comparison to state-of-the-art machine learning approaches.Our main finding is that this approach produces the highest average test scores across problems while producing representations that are orders of magnitude smaller than the next best performing method (gradient boosting).We also report a negative result in which attempts to directly optimize the disentanglement of the representation result in more highly correlated features.

Die Darstellung der Netzarchitektur als eine Reihe von Syntaxbäumen und die Optimierung ihrer Struktur führt zu genauen und präzisen Regressionsmodellen.


Most distributed machine learning (ML) systems store a copy of the model parameters locally on each machine to minimize network communication.In practice, in order to reduce synchronization waiting time, these copies of the model are not necessarily updated in lock-step, and can become stale.Despite much development in large-scale ML, the effect of staleness on the learning efficiency is inconclusive, mainly because it is challenging to control or monitor the staleness in complex distributed environments.In this work, we study the convergence behaviors of a wide array of ML models and algorithms under delayed updates.Our extensive experiments reveal the rich diversity of the effects of staleness on the convergence of ML algorithms and offer insights into seemingly contradictory reports in the literature.The empirical findings also inspire a new convergence analysis of SGD in non-convex optimization under staleness, matching the best-known convergence rate of O(1/\sqrt{T}).

Empirische und theoretische Untersuchung der Auswirkungen von Staleness bei nicht-synchroner Ausführung auf Algorithmen des maschinellen Lernens.


System identification is the process of building a mathematical model of an unknown system from measurements of its inputs and outputs.It is a key step for model-based control, estimator design, and output prediction.This work presents an algorithm for non-linear offline system identification from partial observations, i.e. situations in which the system's full-state is not directly observable.The algorithm presented, called SISL, iteratively infers the system's full state through non-linear optimization and then updates the model parameters.We test our algorithm on a simulated system of coupled Lorenz attractors, showing our algorithm's ability to identify high-dimensional systems that prove intractable for particle-based approaches.We also use SISL to identify the dynamics of an aerobatic helicopter.By augmenting the state with unobserved fluid states, we learn a model that predicts the acceleration of the helicopter better than state-of-the-art approaches.

In dieser Arbeit wird ein skalierbarer Algorithmus zur nichtlinearen Offline-Systemidentifikation aus Teilbeobachtungen vorgestellt.


Various gradient compression schemes have been proposed to mitigate the communication cost in distributed training of large scale machine learning models.Sign-based methods, such as signSGD (Bernstein et al., 2018), have recently been gaining popularity because of their simple compression rule and connection to adaptive gradient methods, like ADAM.In this paper, we perform a general analysis of sign-based methods for non-convex optimization.Our analysis is built on intuitive bounds on success probabilities and does not rely on special noise distributions nor on the boundedness of the variance of stochastic gradients.Extending the theory to distributed setting within a parameter server framework, we assure exponentially fast variance reduction with respect to number of nodes, maintaining 1-bit compression in both directions and using small mini-batch sizes.We validate our theoretical findings experimentally.

Allgemeine Analyse von vorzeichenbasierten Methoden (z.B. signSGD) für nicht-konvexe Optimierung, aufbauend auf intuitiven Schranken für Erfolgswahrscheinlichkeiten.


Off-policy learning, the task of evaluating and improving policies using historic data collected from a logging policy, is important because on-policy evaluation is usually expensive and has adverse impacts.One of the major challenge of off-policy learning is to derive counterfactual estimators that also has low variance and thus low generalization error. In this work, inspired by learning bounds for importance sampling problems, we present a new counterfactual learning principle for off-policy learning with bandit feedbacks.Our method regularizes the generalization error by minimizing the distribution divergence between the logging policy and the new policy, and removes the need for iterating through all training samples to compute sample variance regularization in prior work.With neural network policies, our end-to-end training algorithms using variational divergence minimization showed significant improvement over conventional baseline algorithms and is also consistent with our theoretical results.

Für das Off-Policy-Lernen mit Bandit-Rückkopplungen schlagen wir einen neuen varianzregulierten kontrafaktischen Lernalgorithmus vor, der sowohl theoretische Grundlagen als auch eine überlegene empirische Leistung aufweist.


We outline new approaches to incorporate ideas from deep learning into wave-based least-squares imaging.The aim, and main contribution of this work, is the combination of handcrafted constraints with deep convolutional neural networks, as a way to harness their remarkable ease of generating natural images.The mathematical basis underlying our method is the expectation-maximization framework, where data are divided in batches and coupled to additional "latent" unknowns.These unknowns are pairs of elements from the original unknown space (but now coupled to a specific data batch) and network inputs.In this setting, the neural network controls the similarity between these additional parameters, acting as a "center" variable.The resulting problem amounts to a maximum-likelihood estimation of the network parameters when the augmented data model is marginalized over the latent variables.

Wir kombinieren harte handwerkliche Einschränkungen mit einer tiefen priorisierten schwachen Einschränkung, um seismische Bilder zu erstellen und Informationen über die "posteriore" Verteilung zu erhalten, indem wir die Multiplizität in den Daten nutzen.


When translating natural language questions into SQL queries to answer questions from a database, contemporary semantic parsing models struggle to generalize to unseen database schemas.  The generalization challenge lies in (a) encoding the database relations in an accessible way for the semantic parser, and (b) modeling alignment between database columns and their mentions in a given query.  We present a unified framework, based on the relation-aware self-attention mechanism,to address schema encoding, schema linking, and feature representation within a text-to-SQL encoder.On the challenging Spider dataset this framework boosts the exact match accuracy to 53.7%, compared to 47.4% for the previous state-of-the-art model unaugmented with BERT embeddings.In addition, we observe qualitative improvements in the model’s understanding of schema linking and alignment.

Stand der Technik bei komplexem Text-zu-SQL-Parsing durch die Kombination von hartem und weichem relationalem Schließen in der Schema-/Fragenkodierung.


As our experience shows, humans can learn and deploy a myriad of different skills to tackle the situations they encounter daily.Neural networks, in contrast, have a fixed memory capacity that prevents them from learning more than a few sets of skills before starting to forget them. In this work, we make a step to bridge neural networks with human-like learning capabilities.For this, we propose a model with a growing and open-bounded memory capacity that can be accessed based on the model’s current demands.To test this system, we introduce a continual learning task based on language modelling where the model is exposed to multiple languages and domains in sequence, without providing any explicit signal on the type of input it is currently dealing with.The proposed system exhibits improved adaptation skills in that it can recover faster than comparable baselines after a switch in the input language or domain.

Wir stellen ein kontinuierliches Lernsystem vor, das auf der Sprachmodellierung basiert und bei dem kein explizites Aufgabensegmentierungssignal gegeben ist, und schlagen ein neuronales Netzmodell mit wachsendem Langzeitgedächtnis vor, um dieses Problem zu lösen.


We propose to tackle a time series regression problem by computing temporal evolution of a probability density function to provide a probabilistic forecast.A Recurrent Neural Network (RNN) based model is employed to learn a nonlinear operator for temporal evolution of a probability density function.We use a softmax layer for a numerical discretization of a smooth probability density functions, which transforms a function approximation problem to a classification task.Explicit and implicit regularization strategies are introduced to impose a smoothness condition on the estimated probability distribution.A Monte Carlo procedure to compute the temporal evolution of the distribution for a multiple-step forecast is presented.The evaluation of the proposed algorithm on three synthetic and two real data sets shows advantage over the compared baselines.

Vorgeschlagener RNN-basierter Algorithmus zur Schätzung der wahrscheinlichen Verteilung bei ein- und mehrstufigen Prognosen in Zeitreihenvorhersageproblemen.


In cognitive systems, the role of a working memory is crucial for visual reasoning and decision making.Tremendous progress has been made in understanding the mechanisms of the human/animal working memory, as well as in formulating different frameworks of artificial neural networks.  In the case of humans, the visual working memory (VWM) task is a standard one in which the subjects are presented with a sequence of images, each of which needs to be identified as to whether it was already seen or not. Our work is a study of multiple ways to learn a working memory model using recurrent neural networks that learn to remember input images across timesteps.We train these neural networks to solve the working memory task by training them with a sequence of images in supervised and reinforcement learning settings.The supervised setting uses image sequences with their corresponding labels.The reinforcement learning setting is inspired by the popular view in neuroscience that the working memory in the prefrontal cortex is modulated by a dopaminergic mechanism.We consider the VWM task as an environment that rewards the agent when it remembers past information and penalizes it for forgetting.  We quantitatively estimate the performance of these models on sequences of images from a standard image dataset (CIFAR-100).Further, we evaluate their ability to remember and recall as they are increasingly trained over episodes.Based on our analysis, we establish that a gated recurrent neural network model with long short-term memory units trained using reinforcement learning is powerful and more efficient in temporally consolidating the input spatial information. This work is an initial analysis as a part of our ultimate goal to use artificial neural networks to model the behavior and information processing of the working memory of the brain and to use brain imaging data captured from human subjects during the VWM cognitive task to understand various memory mechanisms of the brain.

LSTMs können das Arbeitsgedächtnis effektiver modellieren, wenn sie durch Reinforcement Learning erlernt werden, ähnlich wie das Dopaminsystem, das das Gedächtnis im präfrontalen Kortex moduliert.


Nonlinearity is crucial to the performance of a deep (neural) network (DN).To date there has been little progress understanding the menagerie of available  nonlinearities, but recently progress has been made on understanding the r\^{o}le played by piecewise affine and convex nonlinearities like the ReLU and absolute value activation functions and max-pooling.In particular, DN layers constructed from these operations can be interpreted as {\em max-affine spline operators} (MASOs) that have an elegant link to vector quantization (VQ) and $K$-means.While this is good theoretical progress, the entire MASO approach is predicated on the requirement that the nonlinearities be piecewise affine and convex, which precludes important activation functions like the sigmoid, hyperbolic tangent, and softmax.{\em This paper extends the MASO framework to these and an infinitely large class of new nonlinearities by linking deterministic MASOs with probabilistic Gaussian Mixture Models (GMMs).}We show that, under a GMM, piecewise affine, convex nonlinearities like ReLU, absolute value, and max-pooling can be interpreted as solutions to certain natural ``hard'' VQ inference problems, while sigmoid, hyperbolic tangent, and softmax can be interpreted as solutions to corresponding ``soft'' VQ inference problems.We further extend the framework by hybridizing the hard and soft VQ optimizations to create a $\beta$-VQ inference that interpolates between hard, soft, and linear VQ inference.A prime example of a $\beta$-VQ DN nonlinearity is the {\em swish} nonlinearity, which offers state-of-the-art performance in a range of computer vision tasks but was developed ad hoc by experimentation.Finally, we validate with experiments an important assertion of our theory, namely that DN performance can be significantly improved by enforcing orthogonality in its linear filters.

Reformulieren Sie die Nichtlinearitäten tiefer Netzwerke aus dem Bereich der Vektorquantisierung und verbinden Sie die meisten bekannten Nichtlinearitäten miteinander.


Engineered proteins offer the potential to solve many problems in biomedicine, energy, and materials science, but creating designs that succeed is difficult in practice.A significant aspect of this challenge is the complex coupling between protein sequence and 3D structure, and the task of finding a viable design is often referred to as the inverse protein folding problem.We develop generative models for protein sequences conditioned on a graph-structured specification of the design target.Our approach efficiently captures the complex dependencies in proteins by focusing on those that are long-range in sequence but local in 3D space.Our framework significantly improves upon prior parametric models of protein sequences given structure, and takes a step toward rapid and targeted biomolecular design with the aid of deep generative models.

Wir lernen, Proteinsequenzen aus gegebenen Strukturen mit einem Modell zu generieren, das spärliche, weitreichende Abhängigkeiten erfasst.


We provide a novel perspective on the forward pass through a block of layers in a deep network.In particular, we show that a forward pass through a standard dropout layer followed by a linear layer and a non-linear activation is equivalent to optimizing a convex objective with a single iteration of a $\tau$-nice Proximal Stochastic Gradient method.We further show that replacing standard Bernoulli dropout with additive dropout is equivalent to optimizing the same convex objective with a variance-reduced proximal method.By expressing both fully-connected and convolutional layers as special cases of a high-order tensor product, we unify the underlying convex optimization problem in the tensor setting and derive a formula for the Lipschitz constant $L$ used to determine the optimal step size of the above proximal methods.We conduct experiments with standard convolutional networks applied to the CIFAR-10 and CIFAR-100 datasets and show that replacing a block of layers with multiple iterations of the corresponding solver, with step size set via $L$, consistently improves classification accuracy.

Ein Framework, das tiefe Netzwerkschichten mit stochastischen Optimierungsalgorithmen verbindet; kann zur Verbesserung der Modellgenauigkeit und zur Information über das Netzwerkdesign verwendet werden.


Deep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases.Here, we present a method for training such networks, Learned Step Size Quantization, that achieves the highest accuracy to date on the ImageNet dataset when using models, from a variety of architectures, with weights and activations quantized to 2-, 3- or 4-bits of precision, and that can train 3-bit models that reach full precision baseline accuracy.Our approach builds upon existing methods for learning weights in quantized networks by improving how the quantizer itself is configured.Specifically, we introduce a novel means to estimate and scale the task loss gradient at each weight and activation layer's quantizer step size, such that it can be learned in conjunction with other network parameters.This approach works using different levels of precision as needed for a given system and requires only a simple modification of existing training code.

Eine Methode zum Erlernen der Quantisierungskonfiguration für Netze mit geringer Genauigkeit, die den neuesten Stand der Technik für quantisierte Netze erreicht.


There have been several studies recently showing that strong natural language understanding (NLU) models are prone to relying on unwanted dataset biases without learning the underlying task, resulting in models which fail to generalize to out-of-domain datasets, and are likely to perform poorly in real-world scenarios.We propose several learning strategies to train neural models which are more robust to such biases and transfer better to out-of-domain datasets.We introduce an additional lightweight bias-only model which learns dataset biases and uses its prediction to adjust the loss of the base model to reduce the biases.In other words, our methods down-weight the importance of the biased examples, and focus training on hard examples, i.e. examples that cannot be correctly classified by only relying on biases.Our approaches are model agnostic and simple to implement.  We experiment on large-scale natural language inference and fact verification datasets and their out-of-domain datasets and show that our debiased models significantly improve the robustness in all settings, including gaining 9.76 points on the FEVER symmetric evaluation dataset, 5.45 on the HANS dataset and 4.78 points on the SNLI hard set.  These datasets are specifically designed to assess the robustness of models in the out-of-domain setting where typical biases in the training data do not exist in the evaluation set.

Wir schlagen mehrere allgemeine Debiasing-Strategien vor, um die in verschiedenen Datensätzen auftretenden Verzerrungen zu beseitigen und eine erheblich verbesserte Out-of-Domain-Leistung in allen Situationen zu erzielen.


Reconstruction of few-view x-ray Computed Tomography (CT) data is a highly ill-posed problem.It is often used in applications that require low radiation dose in clinical CT, rapid industrial scanning, or fixed-gantry CT.Existing analytic or iterative algorithms generally produce poorly reconstructed images, severely deteriorated by artifacts and noise, especially when the number of x-ray projections is considerably low.This paper presents a deep network-driven approach to address extreme few-view CT by incorporating convolutional neural network-based inference into state-of-the-art iterative reconstruction.The proposed method interprets few-view sinogram data using attention-based deep networks to infer the reconstructed image.The predicted image is then used as prior knowledge in the iterative algorithm for final reconstruction.We demonstrate effectiveness of the proposed approach by performing reconstruction experiments on a chest CT dataset.

Wir stellen einen auf CNN-Inferenz basierenden Rekonstruktionsalgorithmus vor, der für CT mit extrem wenigen Ansichten geeignet ist.


In open-domain dialogue intelligent agents should exhibit the use of knowledge, however there are few convincing demonstrations of this to date.The most popular sequence to sequence models typically “generate and hope” generic utterances that can be memorized in the weights of the model when mapping from input utterance(s) to output, rather than employing recalled knowledge as context.Use of knowledge has so far proved difficult, in part because of the lack of a supervised learning benchmark task which exhibits knowledgeable open dialogue with clear  grounding.To that end we collect and release a large dataset with conversations directly grounded with knowledge retrieved from Wikipedia.  We then design architectures capable of retrieving knowledge, reading and conditioning on it, and finally generating natural responses.Our best performing dialogue models are able to conduct knowledgeable discussions on open-domain topics as evaluated by automatic metrics and human evaluations, while our new benchmark allows for measuring further improvements in this important research direction.

Wir bauen kenntnisreiche Gesprächsagenten durch Konditionierung auf Wikipedia und eine neue überwachte Aufgabe.


We formulate a new problem at the intersection of semi-supervised learning and contextual bandits, motivated by several applications including clinical trials and dialog systems.We demonstrate how contextual bandit and graph convolutional networks can be adjusted to the new problem formulation.We then take the best of both approaches to develop multi-GCN embedded contextual bandit.Our algorithms are verified on several real world datasets.

Synthese von GCN- und LINUCB-Algorithmen für Online-Lernen mit fehlendem Feedback.


A collection of scientific papers is often accompanied by tags:  keywords, topics, concepts etc., associated with each paper.  Sometimes these tags are human-generated, sometimes they are  machine-generated.  We propose a simple measure of the consistency  of the tagging of scientific papers: whether these tags are  predictive for the citation graph links.  Since the authors tend to  cite papers about the topics close to those of their publications, a  consistent tagging system could predict citations.  We present an  algorithm to calculate consistency, and experiments with human- and  machine-generated tags.  We show that augmentation, i.e. the combination  of the manual tags with the machine-generated ones, can enhance the  consistency of the tags.  We further introduce cross-consistency,  the ability to predict citation links between papers tagged by  different taggers, e.g. manually and by a machine.  Cross-consistency can be used to evaluate the tagging quality when  the amount of labeled data is limited.

Ein guter Tagger vergibt ähnliche Tags für eine bestimmte Arbeit und die darin zitierten Arbeiten.


Recent research has intensively revealed the vulnerability of deep neural networks, especially for convolutional neural networks (CNNs) on the task of image recognition, through creating adversarial samples which `"slightly" differ from legitimate samples.This vulnerability indicates that these powerful models are sensitive to specific perturbations and cannot filter out these adversarial perturbations.In this work, we propose a quantization-based method which enables a CNN to filter out adversarial perturbations effectively.Notably, different from prior work on input quantization, we apply the quantization in the intermediate layers of a CNN.Our approach is naturally aligned with the clustering of the coarse-grained semantic information learned by a CNN.Furthermore, to compensate for the loss of information which is inevitably caused by the quantization, we propose the multi-head quantization, where we project data points to different sub-spaces and perform quantization within each sub-space.We enclose our design in a quantization layer named as the Q-Layer.The results obtained on MNIST and Fashion-MNSIT datasets demonstrate that only adding one Q-Layer into a CNN could significantly improve its robustness against both white-box and black-box attacks.

Wir schlagen eine auf Quantisierung basierende Methode vor, die die gelernten Repräsentationen eines CNN so reguliert, dass sie automatisch mit der trainierbaren Konzeptmatrix abgeglichen werden, um so effektiv negative Einflüsse herauszufiltern.


Invariant and equivariant networks have been successfully used for learning images, sets, point clouds, and graphs.A basic challenge in developing such networks is finding the maximal collection of invariant and equivariant \emph{linear} layers.Although this question is answered for the first three examples (for popular transformations, at-least), a full characterization of invariant and equivariant linear layers for graphs is not known. In this paper we provide a characterization of all permutation invariant and equivariant linear layers for (hyper-)graph data, and show that their dimension, in case of edge-value graph data, is $2$ and $15$, respectively.More generally, for graph data defined on $k$-tuples of nodes, the dimension is the $k$-th and $2k$-th Bell numbers.Orthogonal bases for the layers are computed, including generalization to multi-graph data.The constant number of basis elements and their characteristics allow successfully applying the networks to different size graphs.From the theoretical point of view, our results generalize and unify recent advancement in equivariant deep learning.In particular, we show that our model is capable of approximating any message passing neural network.Applying these new linear layers in a simple deep neural network framework is shown to achieve comparable results to state-of-the-art and to have better expressivity than previous invariant and equivariant bases.

Die Arbeit bietet eine vollständige Charakterisierung von permutationsinvarianten und äquivarianten linearen Schichten für Graphdaten.


In reinforcement learning, we can learn a model of future observations and rewards, and use it to plan the agent's next actions.However, jointly modeling future observations can be computationally expensive or even intractable if the observations are high-dimensional (e.g. images).For this reason, previous works have considered partial models, which model only part of the observation.In this paper, we show that partial models can be causally incorrect: they are confounded by the observations they don't model, and can therefore lead to incorrect planning.To address this, we introduce a general family of partial models that are provably causally correct, but avoid the need to fully model future observations.

Kausal korrekte Teilmodelle müssen nicht die gesamte Beobachtung erzeugen, um in stochastischen Umgebungen kausal korrekt zu bleiben.


In lifelong learning, the learner is presented with a sequence of tasks, incrementally building a data-driven prior which may be leveraged to speed up learning of a new task.In this work, we investigate the efficiency of current lifelong approaches, in terms of sample complexity, computational and memory cost.Towards this end, we first introduce a new and a more realistic evaluation protocol, whereby learners observe each example only once and hyper-parameter selection is done on a small and disjoint set of tasks, which is not used for the actual learning experience and evaluation.Second, we introduce a new metric measuring how quickly a learner acquires a new skill.Third, we propose an improved version of GEM (Lopez-Paz & Ranzato, 2017), dubbed Averaged GEM (A-GEM), which enjoys the same or even better performance as GEM, while being almost as computationally and memory efficient as EWC (Kirkpatrick et al., 2016) and other regularization-based methods.Finally, we show that all algorithms including A-GEM can learn even more quickly if they are provided with task descriptors specifying the classification tasks under consideration.Our experiments on several standard lifelong learning benchmarks demonstrate that A-GEM has the best trade-off between accuracy and efficiency

Ein effizienter Algorithmus für lebenslanges Lernen, der im Vergleich zu anderen Algorithmen einen besseren Kompromiss zwischen Genauigkeit und Zeit-/Speicherkomplexität bietet.


Reduced precision computation is one of the key areas addressing the widening’compute gap’, driven by an exponential growth in deep learning applications.In recent years, deep neural network training has largely migrated to 16-bit precision,with significant gains in performance and energy efficiency.However, attempts to train DNNs at 8-bit precision have met with significant challenges, because of the higher precision and dynamic range requirements of back-propagation.   In this paper,  we  propose  a  method  to  train  deep  neural  networks  using  8-bit  floating point representation for weights, activations, errors, and gradients.  We demonstrate state-of-the-art accuracy across multiple data sets (imagenet-1K, WMT16)and a broader set of workloads (Resnet-18/34/50, GNMT, and Transformer) than previously reported.   We propose an enhanced loss scaling method to augment the reduced subnormal range of 8-bit floating point, to improve error propagation.We also examine the impact of quantization noise on generalization, and propose a stochastic rounding technique to address gradient noise.As a result of applying all these techniques,  we report slightly higher validation accuracy compared to full precision baseline.

Wir haben modernste Trainingsergebnisse mit 8-Bit-Gleitkommadarstellung für Resnet, GNMT und Transformer demonstriert.


Loss functions play a crucial role in deep metric learning thus a variety of them have been proposed.Some supervise the learning process by pairwise or tripletwise similarity constraints while others take the advantage of structured similarity information among multiple data points.In this work, we approach deep metric learning from a novel perspective.We propose instance cross entropy (ICE) which measures the difference between an estimated instance-level matching distribution and its ground-truth one.ICE has three main appealing properties.Firstly, similar to categorical cross entropy (CCE), ICE has clear probabilistic interpretation and exploits structured semantic similarity information for learning supervision.Secondly, ICE is scalable to infinite training data as it learns on mini-batches iteratively and is independent of the training set size.Thirdly, motivated by our relative weight analysis, seamless sample reweighting is incorporated.It rescales samples’ gradients to control the differentiation degree over training examples instead of truncating them by sample mining.In addition to its simplicity and intuitiveness, extensive experiments on three real-world benchmarks demonstrate the superiority of ICE.

Wir schlagen eine Instanz-Cross-Entropie (ICE) vor, die die Differenz zwischen einer geschätzten Instanz-Level-Matching-Verteilung und ihrer Grundwahrheit misst.


In model-based reinforcement learning, the agent interleaves between model learning and planning.  These two components are  inextricably intertwined.If the model is not able to provide sensible long-term prediction, the executed planer would exploit model flaws, which can yield catastrophic failures.This paper focuses on building a model that reasons about the long-term future and demonstrates how to use this for efficient planning and exploration.To this end, we build a latent-variable autoregressive model by leveraging recent ideas in variational inference.We argue that forcing latent variables to carry future information through an auxiliary task substantially improves long-term predictions.Moreover, by planning in the latent space, the planner's solution is ensured to be within regions where the model is valid.An exploration strategy can be devised by searching for unlikely trajectories under the model.Our methods achieves higher reward faster compared to baselines on a variety of tasks and environments in both the imitation learning and model-based reinforcement learning settings.

Die Einbeziehung latenter Variablen in das Modell, die zukünftige Inhalte kodieren, verbessert die langfristige Vorhersagegenauigkeit, was für eine bessere Planung in modellbasierten RL entscheidend ist.


Detecting anomalies is of growing importance for various industrial applications and mission-critical infrastructures, including satellite systems.Although there have been several studies in detecting anomalies based on rule-based or machine learning-based approaches for satellite systems, a tensor-based decomposition method has not been extensively explored for anomaly detection.In this work, we introduce an Integrative Tensor-based Anomaly Detection (ITAD) framework to detect anomalies in a satellite system.Because of the high risk and cost, detecting anomalies in a satellite system is crucial.We construct 3rd-order tensors with telemetry data collected from Korea Multi-Purpose Satellite-2 (KOMPSAT-2) and calculate the anomaly score using one of the component matrices obtained by applying CANDECOMP/PARAFAC decomposition to detect anomalies.Our result shows that our tensor-based approach can be effective in achieving higher accuracy and reducing false positives in detecting anomalies as compared to other existing approaches.

Integrative Tensor-based Anomaly Detection (ITAD) für ein Satellitensystem.


Many real world tasks exhibit rich structure that is repeated across different parts of the state space or in time.In this work we study the possibility of leveraging such repeated structure to speed up and regularize learning.We start from the KL regularized expected reward objective which introduces an additional component, a default policy.Instead of relying on a fixed default policy, we learn it from data.But crucially, we restrict the amount of information the default policy receives, forcing it to learn reusable behaviors that help the policy learn faster.We formalize this strategy and discuss connections to information bottleneck approaches and to the variational EM algorithm.We present empirical results in both discrete and continuous action domains and demonstrate that, for certain tasks, learning a default policy alongside the policy can significantly speed up and improve learning.Please watch the video demonstrating learned experts and default policies on several continuous control tasks ( https://youtu.be/U2qA3llzus8 ).

Die Begrenzung der Zustandsinformationen für die Standardregeln kann die Leistung in einem KL-regulierten RL-Framework verbessern, in dem sowohl der Agent als auch die Standardregeln gemeinsam optimiert werden.


When an image classifier makes a prediction, which parts of the image are relevant and why?We can rephrase this question to ask: which parts of the image, if they were not seen by the classifier, would most change its decision?Producing an answer requires marginalizing over images that could have been seen but weren't.We can sample plausible image in-fills by conditioning a generative model on the rest of the image.We then optimize to find the image regions that most change the classifier's decision after in-fill.Our approach contrasts with ad-hoc in-filling approaches, such as blurring or injecting noise, which generate inputs far from the data distribution, and ignore informative relationships between different parts of the image.Our method produces more compact and relevant saliency maps, with fewer artifacts compared to previous methods.

Wir berechnen die Auffälligkeit mit Hilfe eines starken generativen Modells, um effizient über plausible alternative Eingaben zu marginalisieren und so konzentrierte Pixelbereiche zu erkennen, die Labelinformationen enthalten.


This paper presents the Variation Network (VarNet), a  generative model providing means to manipulate the high-level attributes of a given input.The originality of our approach is that VarNet is not only capable of handling pre-defined attributes but can also learn the relevant attributes of the dataset by itself.  These two settings can be easily combined  which makes VarNet applicable for a wide variety of tasks.Further, VarNet has a sound probabilistic interpretation which grants us with  a novel way to navigate in the latent spaces as well as means to control how the  attributes are learned.We demonstrate  experimentally that this model is capable of performing interesting input manipulation  and that the learned attributes are relevant and interpretable.

Das Variationsnetzwerk ist ein generatives Modell, das in der Lage ist, ohne Überwachung hochrangige Attribute zu erlernen, die dann für eine kontrollierte Eingabemanipulation verwendet werden können.


Despite alarm over the reliance of machine learning systems on so-called spurious patterns in training data, the term lacks coherent meaning in standard statistical frameworks.However, the language of causality offers clarity: spurious associations are those due to a common cause (confounding) vs direct or indirect effects.In this paper, we focus on NLP, introducing methods and resources for training models insensitive to spurious patterns.Given documents and their initial labels, we task humans with revise each document to accord with a counterfactual target label, asking that the revised documents be internally coherent while avoiding any gratuitous changes.Interestingly, on sentiment analysis and natural language inference tasks, classifiers trained on original data fail on their counterfactually-revised counterparts and vice versa.Classifiers trained on combined datasets perform remarkably well, just shy of those specialized to either domain.While classifiers trained on either original or manipulated data alone  are sensitive to spurious features (e.g., mentions of genre), models trained on the combined data are insensitive to this signal.We will publicly release both datasets.

Die Menschen in der Schleife überarbeiten die Dokumente, damit sie mit den kontrafaktischen Bezeichnungen übereinstimmen, was dazu beiträgt, die Abhängigkeit von falschen Assoziationen zu verringern.


Among multiple ways of interpreting a machine learning model, measuring the importance of a set of features tied to a prediction is probably one of the most intuitive way to explain a model.In this paper, we establish the link between a set of features to a prediction with a new evaluation criteria, robustness analysis, which measures the minimum tolerance of adversarial perturbation.By measuring the tolerance level for an adversarial attack, we can extract a set of features that provides most robust support for a current prediction, and also can extract a set of features that contrasts the current prediction to a target class by setting a targeted adversarial attack.By applying this methodology to various prediction tasks across multiple domains, we observed the derived explanations are indeed capturing the significant feature set qualitatively and quantitatively.

Wir schlagen ein neues objektives Maß für die Bewertung von Erklärungen vor, das auf dem Begriff der Widerstandsfähigkeit gegenüber Widrigkeiten beruht. Die Bewertungskriterien ermöglichen es uns, neue Erklärungen abzuleiten, die die relevanten Merkmale qualitativ und quantitativ erfassen.


Generative adversarial networks (GANs) have been shown to provide an effective way to model complex distributions and have obtained impressive results on various challenging tasks.However, typical GANs require fully-observed data during training.In this paper, we present a GAN-based framework for learning from complex, high-dimensional incomplete data.The proposed framework learns a complete data generator along with a mask generator that models the missing data distribution.We further demonstrate how to impute missing data by equipping our framework with an adversarially trained imputer.We evaluate the proposed framework using a series of experiments with several types of missing data processes under the missing completely at random assumption.

In dieser Arbeit wird ein GAN-basierter Rahmen für das Lernen der Verteilung aus hochdimensionalen unvollständigen Daten vorgestellt.


Due to a resource-constrained environment, network compression has become an important part of deep neural networks research.In this paper, we propose a new compression method, Inter-Layer Weight Prediction (ILWP) and quantization method which quantize the predicted residuals between the weights in all convolution layers based on an inter-frame prediction method in conventional video coding schemes.Furthermore, we found a phenomenon Smoothly Varying Weight Hypothesis (SVWH) which is that the weights in adjacent convolution layers share strong similarity in shapes and values, i.e., the weights tend to vary smoothly along with the layers.Based on SVWH, we propose a second ILWP and quantization method which quantize the predicted residuals between the weights in adjacent convolution layers.Since the predicted weight residuals tend to follow Laplace distributions with very low variance, the weight quantization can more effectively be applied, thus producing more zero weights and enhancing the weight compression ratio.In addition, we propose a new inter-layer loss for eliminating non-texture bits, which enabled us to more effectively store only texture bits.That is, the proposed loss regularizes the weights such that the collocated weights between the adjacent two layers have the same values.Finally, we propose an ILWP with an inter-layer loss and quantization method.Our comprehensive experiments show that the proposed method achieves a much higher weight compression rate at the same accuracy level compared with the previous quantization-based compression methods in deep neural networks.

Wir schlagen eine neue Kompressionsmethode, Inter-Layer Weight Prediction (ILWP) und eine Quantisierungsmethode vor, die die vorhergesagten Residuen zwischen den Gewichten in den Convolutional Layers quantisiert.


There exists a plethora of techniques for inducing structured sparsity in parametric models during the optimization process, with the final goal of resource-efficient inference.However, to the best of our knowledge, none target a specific number of floating-point operations (FLOPs) as part of a single end-to-end optimization objective, despite reporting FLOPs as part of the results.Furthermore, a one-size-fits-all approach ignores realistic system constraints, which differ significantly between, say, a GPU and a mobile phone -- FLOPs on the former incur less latency than on the latter; thus, it is important for practitioners to be able to specify a target number of FLOPs during model compression.In this work, we extend a state-of-the-art technique to directly incorporate FLOPs as part of the optimization objective and show that, given a desired FLOPs requirement, different neural networks can be successfully trained for image classification.

Wir erweitern eine moderne Technik, um FLOPs direkt als Teil des Optimierungsziels einzubeziehen, und wir zeigen, dass bei einer gewünschten FLOPs-Anforderung verschiedene neuronale Netze erfolgreich trainiert werden.


Unpaired image-to-image translation among category domains has achieved remarkable success in past decades.Recent studies mainly focus on two challenges.For one thing, such translation is inherently multimodal due to variations of domain-specific information (e.g., the domain of house cat has multiple fine-grained subcategories).For another, existing multimodal approaches have limitations in handling more than two domains, i.e. they have to independently build one model for every pair of domains.To address these problems, we propose the Hierarchical Image-to-image Translation (HIT) method which jointly formulates the multimodal and multi-domain problem in a semantic hierarchy structure, and can further control the uncertainty of multimodal.Specifically, we regard the domain-specific variations as the result of the multi-granularity property of domains, and one can control the granularity of the multimodal translation by dividing a domain with large variations into multiple subdomains which capture local and fine-grained variations.With the assumption of Gaussian prior, variations of domains are modeled in a common space such that translations can further be done among multiple domains within one model.To learn such complicated space, we propose to leverage the inclusion relation among domains to constrain distributions of parent and children to be nested.Experiments on several datasets validate the promising results and competitive performance against state-of-the-arts.

Granularitätsgesteuertes multidomänen und multimodales Bild-zu-Bild-Übersetzungsverfahren.


Recurrent Neural Networks (RNNs) are very successful at solving challenging problems with sequential data.However, this observed efficiency is not yet entirely explained by theory.It is known that a certain class of multiplicative RNNs enjoys the property of depth efficiency --- a shallow network of exponentially large width is necessary to realize the same score function as computed by such an RNN.Such networks, however, are not very often applied to real life tasks.In this work, we attempt to reduce the gap between theory and practice by extending the theoretical analysis to RNNs which employ various nonlinearities, such as Rectified Linear Unit (ReLU), and show that they also benefit from properties of universality and depth efficiency.Our theoretical results are verified by a series of extensive computational experiments.

Analyse der Ausdruckskraft und Allgemeinheit von rekurrenten neuronalen Netzen mit ReLu-Nichtlinearitäten unter Verwendung der Tensor-Train-Zerlegung.


While deep neural networks have proven to be a powerful tool for many recognition and classification tasks, their stability properties are still not well understood.In the past, image classifiers have been shown to be vulnerable to so-called adversarial attacks, which are created by additively perturbing the correctly classified image.In this paper, we propose the ADef algorithm to construct a different kind of adversarial attack created by iteratively applying small deformations to the image, found through a gradient descent step.We demonstrate our results on MNIST with convolutional neural networks and on ImageNet with Inception-v3 and ResNet-101.

Wir schlagen einen neuen, effizienten Algorithmus zur Konstruktion von adversarialen Beispielen vor, der auf Deformationen statt auf additiven Störungen beruht.


Adversarial learning methods have been proposed for a wide range of applications, but the training of adversarial models can be notoriously unstable.Effectively balancing the performance of the generator and discriminator is critical, since a discriminator that achieves very high accuracy will produce relatively uninformative gradients.In this work, we propose a simple and general technique to constrain information flow in the discriminator by means of an information bottleneck.By enforcing a constraint on the mutual information between the observations and the discriminator's internal representation, we can effectively modulate the discriminator's accuracy and maintain useful and informative gradients.We demonstrate that our proposed variational discriminator bottleneck (VDB) leads to significant improvements across three distinct application areas for adversarial learning algorithms.Our primary evaluation studies the applicability of the VDB to imitation learning of dynamic continuous control skills, such as running.We show that our method can learn such skills directly from raw video demonstrations, substantially outperforming prior adversarial imitation learning methods.The VDB can also be combined with adversarial inverse reinforcement learning to learn parsimonious reward functions that can be transferred and re-optimized in new settings.Finally, we demonstrate that VDB can train GANs more effectively for image generation, improving upon a number of prior stabilization methods.

Regularisierendes adversariales Lernen mit einem Informationsengpass, angewandt auf Imitation Learning, inverses Reinforcement Learning und generative adversariale Netzwerke.


Deploying machine learning systems in the real world requires both high accuracy on clean data and robustness to naturally occurring corruptions.While architectural advances have led to improved accuracy, building robust models remains challenging, involving major changes in training procedure and datasets.  Prior work has argued that there is an inherent trade-off between robustness and accuracy, as exemplified by standard data augmentation techniques such as Cutout, which improves clean accuracy but not robustness, and additive Gaussian noise, which improves robustness but hurts accuracy.We introduce Patch Gaussian, a simple augmentation scheme that adds noise to randomly selected patches in an input image.  Models trained with Patch Gaussian achieve state of the art on the CIFAR-10 and ImageNet Common Corruptions benchmarks while also maintaining accuracy on clean data.We find that this augmentation leads to reduced sensitivity to high frequency noise (similar to Gaussian) while retaining the ability to take advantage of relevant high frequency information in the image (similar to Cutout).We show it can be used in conjunction with other regularization methods and data augmentation policies such as AutoAugment.  Finally, we find that the idea of restricting perturbations to patches can also be useful in the context of adversarial learning, yielding models without the loss in accuracy that is found with unconstrained adversarial training.

Die einfache Augmentierungsmethode überwindet den in der Literatur beobachteten Kompromiss zwischen Robustheit und Genauigkeit und wirft Fragen zu den Auswirkungen der Trainingsverteilung auf die Generalisierung außerhalb der Verteilung auf.


Offset regression is a standard method for spatial localization in many vision tasks, including human pose estimation, object detection, and instance segmentation.However, if high localization accuracy is crucial for a task, convolutional neural networks will offset regressionusually struggle to deliver.  This can be attributed to the locality of the convolution operation, exacerbated by variance in scale, clutter, and viewpoint.An even more fundamental issue is the multi-modality of real-world images.As a consequence, they cannot be approximated adequately using a single mode model.  Instead, we propose to use mixture density networks (MDN) for offset regression, allowing the model to manage various modes efficiently and learning to predict full conditional density of the outputs given the input.On 2D human pose estimation in the wild, which requires accurate localisation of body keypoints, we show that this yields significant improvement in localization accuracy.In particular, our experiments reveal viewpoint variation as  the dominant  multi-modal factor.Further, by carefully initializing MDN parameters, we do not face any instabilities in training, which is known to be a big obstacle for widespread deployment of MDN.The method can be readily applied to any task with a spatial regression component.Our findings  highlight the multi-modal nature of real-world vision, and the significance of explicitly accounting for viewpoint variation, at least when spatial localization is concerned.

Wir verwenden Mixed-Density-Netzwerke, um eine vollständige bedingte Dichteschätzung für die räumliche Offset-Regression durchzuführen, und wenden sie auf die Aufgabe der menschlichen Posenschätzung an.


Like language, music can be represented as a sequence of discrete symbols that form a hierarchical syntax, with notes being roughly like characters and motifs of notes like words.  Unlike text however, music relies heavily on repetition on multiple timescales to build structure and meaning.The Music Transformer has shown compelling results in generating music with structure (Huang et al., 2018).  In this paper, we introduce a tool for visualizing self-attention on polyphonic music with an interactive pianoroll.  We use music transformer as both a descriptive tool and a generative model.  For the former, we use it to analyze existing music to see if the resulting self-attention structure corroborates with the musical structure known from music theory.  For the latter, we inspect the model's self-attention during generation, in order to understand how past notes affect future ones.We also compare and contrast the attention structure of regular attention to that of relative attention (Shaw et al., 2018, Huang et al., 2018), and examine its impact on the resulting generated music.  For example, for the JSB Chorales dataset, a model trained with relative attention is more consistent in attending to all the voices in the preceding timestep and the chords before, and at cadences to the beginning of a phrase, allowing it to create an arc.  We hope that our analyses will offer more evidence for relative self-attention as a powerful inductive bias for modeling music.  We invite the reader to explore our video animations of music attention and to interact with the visualizations at https://storage.googleapis.com/nips-workshop-visualization/index.html.

Visualisierung der Unterschiede zwischen regulärer und relativer Aufmerksamkeit für Music Transformer.


We study the statistical properties of the endpoint of stochastic gradient descent (SGD).We approximate SGD as a stochastic differential equation (SDE) and consider its Boltzmann Gibbs equilibrium distribution under the assumption of isotropic variance in loss gradients..Through this analysis, we find that three factors – learning rate, batch size and the variance of the loss gradients – control the trade-off between the depth and width of the minima found by SGD, with wider minima favoured by a higher ratio of learning rate to batch size. In the equilibrium distribution only the ratio of learning rate to batch size appears, implying that it’s invariant under a simultaneous rescaling of each by the same amount. We experimentally show how learning rate and batch size affect SGD from two perspectives: the endpoint of SGD and the dynamics that lead up to it. For the endpoint, the experiments suggest the endpoint of SGD is similar under simultaneous rescaling of batch size and learning rate, and also that a higher ratio leads to flatter minima, both findings are consistent with our theoretical analysis. We note experimentally that the dynamics also seem to be similar under the same rescaling of learning rate and batch size, which we explore showing that one can exchange batch size and learning rate in a cyclical learning rate schedule. Next, we illustrate how noise affects memorization, showing that high noise levels lead to better generalization. Finally, we find experimentally that the similarity under simultaneous rescaling of learning rate and batch size breaks down if the learning rate gets too large or the batch size gets too small.

Drei Faktoren (Chargengröße, Lernrate, Gradientenrauschen) verändern in vorhersehbarer Weise die Eigenschaften (z. B. Schärfe) der mit SGD gefundenen Minima.


Although word analogy problems have become a standard tool for evaluating word vectors, little is known about why word vectors are so good at solving these problems.In this paper, I attempt to further our understanding of the subject, by developing a simple, but highly accurate generative approach to solve the word analogy problem for the case when all terms involved in the problem are nouns.My results demonstrate the ambiguities associated with learning the relationship between a word pair, and the role of the training dataset in determining the relationship which gets most highlighted.Furthermore, my results show that the ability of a model to accurately solve the word analogy problem may not be indicative of a model’s ability to learn the relationship between a word pair the way a human does.

Einfacher generativer Ansatz zur Lösung des Wortanalogieproblems, der Einblicke in die Wortbeziehungen und die Probleme bei deren Schätzung liefert.


Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks.Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyper-parameters and keeping them fixed to values normally used for training from scratch.This paper re-examines several common practices of setting hyper-parameters for fine-tuning.Our findings are based on extensive empirical evaluation for fine-tuning on various transfer learning benchmarks.(1) While prior works have thoroughly investigated learning rate and batch size, momentum for fine-tuning is a relatively unexplored parameter.We find that picking the right value for momentum is critical for fine-tuning performance and connect it with previous theoretical findings.(2) Optimal hyper-parameters for fine-tuning in particular the effective learning rate are not only dataset dependent but also sensitive to the similarity between the source domain and target domain.This is in contrast to hyper-parameters for training from scratch.(3) Reference-based regularization that keeps models close to the initial model does not necessarily apply for "dissimilar" datasets.Our findings challenge common practices of fine- tuning and encourages deep learning practitioners to rethink the hyper-parameters for fine-tuning.

In diesem Papier werden mehrere gängige Praktiken zur Festlegung von Hyperparametern für das Fine-Tuning erneut untersucht.


Transfer learning through fine-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can significantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task.To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied.In this paper, we propose a novel regularized transfer learning framework DELTA, namely DEep Learning Transfer using Feature Map with Attention.Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network.Specifically, in addition to minimizing the empirical loss, DELTA intends to align the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in an supervised learning manner.We evaluate DELTA with the state-of-the-art algorithms, including L2 and L2-SP.The experiment results show that our proposed method outperforms these baselines with higher accuracy for new tasks.

Verbesserung des tiefen Transferlernens mit Regularisierung unter Verwendung aufmerksamkeitsbasierter Merkmalszuordnungen.


Neural models achieved considerable improvement for many natural language processing tasks, but they offer little transparency, and interpretability comes at a cost.In some domains, automated predictions without justifications have limited applicability.Recently, progress has been made regarding single-aspect sentiment analysis for reviews, where the ambiguity of a justification is minimal.In this context, a justification, or mask, consists of (long) word sequences from the input text, which suffice to make the prediction.Existing models cannot handle more than one aspect in one training and induce binary masks that might be ambiguous.In our work, we propose a neural model for predicting multi-aspect sentiments for reviews and generates a probabilistic multi-dimensional mask (one per aspect) simultaneously, in an unsupervised and multi-task learning manner.Our evaluation shows that on three datasets, in the beer and hotel domain, our model outperforms strong baselines and generates masks that are: strong feature predictors, meaningful, and interpretable.

Neuronales Modell zur Vorhersage von Gefühlen in mehreren Aspekten und zur gleichzeitigen Erstellung einer probabilistischen mehrdimensionalen Maske. Das Modell übertrifft starke Grundlinien und generiert Masken, die starke Merkmalsvorhersager, aussagekräftig und interpretierbar sind.