The use of deep learning models as priors for compressive sensing tasks presents new potential for inexpensive seismic data acquisition.An appropriately designed Wasserstein generative adversarial network is designed based on a generative adversarial network architecture trained on several historical surveys, capable of learning the statistical properties of the seismic wavelets.The usage of validating and performance testing of compressive sensing are three steps.First, the existence of a sparse representation with different compression rates for seismic surveys is studied.Then, non-uniform samplings are studied, using the proposed methodology.Finally, recommendations for non-uniform seismic survey grid, based on the evaluation of reconstructed seismic images and metrics, is proposed.The primary goal of the proposed deep learning model is to provide the foundations of an optimal design for seismic acquisition, with less loss in imaging quality.Along these lines, a compressive sensing design of a non-uniform grid over an asset in Gulf of Mexico, versus a traditional seismic survey grid which collects data uniformly at every few feet, is suggested, leveraging the proposed method.

Verbesserung eines GAN-basierten Pixel-Inpainting-Netzwerk für die komprimierte seismische Bildwiederherstellung und Vorschlag einer ungleichmäßigen Umfrage Empfehlung, die leicht auf medizinische und andere Bereiche für die komprimierte Sensing-Technik angewendet werden kann.


In recent years we have seen fast progress on a number of benchmark problems in AI, with modern methods achieving near or super human performance in Go, Poker and Dota.One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum.In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative.In the last year, the card game Hanabi has been established as a new benchmark environment for AI to fill this gap.In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind, i.e. the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions.Learning to be informative when observed by others is an interesting challenge for Reinforcement Learning (RL): Fundamentally, RL requires agents to explore in order to discover good policies.However, when done naively, this randomness will inherently make their actions less informative to others during training.We present a new deep multi-agent RL method, the Simplified Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase.During training SAD allows other agents to not only observe the (exploratory) action chosen, but agents instead also observe the greedy action of their team mates.By combining this simple intuition with an auxiliary task for state prediction and best practices for multi-agent learning, SAD establishes a new state of the art for 2-5 players on the self-play part of the Hanabi challenge.

Wir entwickeln Simplified Action Decoder, einen einfachen MARL-Algorithmus, der den bisherigen SOTA auf Hanabi bei Spielen mit 2 bis 5 Spielern deutlich übertrifft.


We present an end-to-end trainable approach for optical character recognition (OCR) on printed documents.It is based on predicting a two-dimensional character grid ('chargrid') representation of a document image as a semantic segmentation task.To identify individual character instances from the chargrid, we regard characters as objects and use object detection techniques from computer vision.We demonstrate experimentally that our method outperforms previous state-of-the-art approaches in accuracy while being easily parallelizable on GPU (thereby being significantly faster), as well as easier to train.

Durchgängig trainierbare optische Zeichenerkennung auf gedruckten Dokumenten; wir erzielen modernste Ergebnisse und übertreffen Tesseract4 in Benchmark-Datensätzen sowohl in Bezug auf die Genauigkeit als auch auf die Laufzeit, indem wir einen rein auf Computer Vision basierenden Ansatz verwenden.


We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay.In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with momentum and Adam/AdamW. Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has two times smaller memory footprint than Adam.

NovoGrad - eine adaptive SGD-Methode mit schichtweiser Gradienten-Normalisierung und entkoppeltem Gewichtsverfall.


Most generative models of audio directly generate samples in one of two domains: time or frequency.While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived.A third approach (vocoders/synthesizers) successfully incorporates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern auto-differentiation-based machine learning methods.In this paper, we introduce the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods.Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks.Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training, blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between disparate sources.In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning.The library will is available at https://github.com/magenta/ddsp and we encourage further contributions from the community and domain experts.

Bessere Audiosynthese durch Kombination von interpretierbarem DSP mit Ende-zu-Ende-Lernen.


Spectral Graph Convolutional Networks (GCNs) are a generalization of convolutional networks to learning on graph-structured data.Applications of spectral GCNs have been successful, but limited to a few problems where the graph is fixed, such as shape correspondence and node classification.In this work, we address this limitation by revisiting a particular family of spectral graph networks, Chebyshev GCNs, showing its efficacy in solving graph classification tasks with a variable graph structure and size.Current GCNs also restrict graphs to have at most one edge between any pair of nodes.To this end, we propose a novel multigraph network that learns from multi-relational graphs.We explicitly model different types of edges: annotated edges, learned edges with abstract meaning, and hierarchical edges.We also experiment with different ways to fuse the representations extracted from different edge types.This restriction is sometimes implied from a dataset, however, we relax this restriction for all kinds of datasets.We achieve state-of-the-art results on a variety of chemical, social, and vision graph classification benchmarks.

Ein neuartiger Ansatz zur Graphenklassifikation auf der Grundlage von spektralen Graph Convolutional Networks und seine Erweiterung auf Multigraphen mit lernbaren Beziehungen und hierarchischer Struktur. Wir zeigen Ergebnisse auf dem neuesten Stand der Technik für chemische, soziale und Bild-Datensätze.


Deep neural networks have been recently demonstrated to be vulnerable to backdoor attacks.Specifically, by altering a small set of training examples, an adversary is able to install a backdoor that can be used during inference to fully control the model’s behavior.While the attack is very powerful, it crucially relies on the adversary being able to introduce arbitrary, often clearly mislabeled, inputs to the training set and can thus be detected even by fairly rudimentary data filtering.In this paper, we introduce a new approach to executing backdoor attacks, utilizing adversarial examples and GAN-generated data.The key feature is that the resulting poisoned inputs appear to be consistent with their label and thus seem benign even upon human inspection.

Wir zeigen, wie man erfolgreich Backdoor-Angriffe durchführt, ohne die Trainingsetiketten zu ändern.


Despite their ability to memorize large datasets, deep neural networks often achieve good generalization performance.However, the differences between the learned solutions of networks which generalize and those which do not remain unclear.Additionally, the tuning properties of single directions (defined as the activation of a single unit or some linear combination of units in response to some input) have been highlighted, but their importance has not been evaluated.Here, we connect these lines of inquiry to demonstrate that a network’s reliance on single directions is a good predictor of its generalization performance, across networks trained on datasets with different fractions of corrupted labels, across ensembles of networks trained on datasets with unmodified labels, across different hyper- parameters, and over the course of training.While dropout only regularizes this quantity up to a point, batch normalization implicitly discourages single direction reliance, in part by decreasing the class selectivity of individual units.Finally, we find that class selectivity is a poor predictor of task importance, suggesting not only that networks which generalize well minimize their dependence on individual units by reducing their selectivity, but also that individually selective units may not be necessary for strong network performance.

Wir stellen fest, dass tiefe Netzwerke, die schlecht verallgemeinern, stärker auf einzelne Richtungen angewiesen sind als solche, die gut verallgemeinern, und evaluieren die Auswirkungen von Dropout und Batch-Normalisierung sowie von Klassenselektivität auf die Abhängigkeit von einer Richtung.


Typical amortized inference in variational autoencoders is specialized for a single probabilistic query.Here we propose an inference network architecture that generalizes to unseen probabilistic queries.Instead of an encoder-decoder pair, we can train a single inference network directly from data, using a cost function that is stochastic not only over samples, but also over queries.We can use this network to perform the same inference tasks as we would in an undirected graphical model with hidden variables, without having to deal with the intractable partition function.The results can be mapped to the learning of an actual undirected model, which is a notoriously hard problem.Our network also marginalizes nuisance variables as required.  We show that our approach generalizes to unseen probabilistic queries on also unseen test data, providing fast and flexible inference.Experiments show that this approach outperforms or matches PCD and AdVIL on 9 benchmark datasets.

Anstatt die Parameter eines grafischen Modells aus Daten zu lernen, lernen Sie ein Inferenznetz, das die gleichen probabilistischen Fragen beantworten kann.


A plethora of computer vision tasks, such as optical flow and image alignment, can be formulated as non-linear optimization problems.Before the resurgence of deep learning, the dominant family for solving such optimization problems was numerical optimization, e.g, Gauss-Newton (GN).More recently, several attempts were made to formulate learnable GN steps as cascade regression architectures.In this paper, we investigate recent machine learning architectures, such as deep neural networks with residual connections, under the above perspective.To this end, we first demonstrate how residual blocks (when considered as discretization of ODEs) can be viewed as GN steps.Then, we go a step further and propose a new residual block, that is reminiscent of Newton's method in numerical optimization and exhibits faster convergence.We thoroughly evaluate the proposed Newton-ResNet by conducting experiments on image and speech classification and image generation, using 4 datasets.All the experiments demonstrate that Newton-ResNet requires less parameters to achieve the same performance with the original ResNet.

Wir zeigen, wie Restblöcke als Gauß-Newton-Schritte betrachtet werden können; wir schlagen einen neuen Restblock vor, der Informationen zweiter Ordnung ausnutzt.


In competitive situations, agents may take actions to achieve their goals that unwittingly facilitate an opponent’s goals.Weconsider a domain where three agents operate: (1) a user (human), (2) an attacker (human or a software) agent and (3) an observer (a software) agent.The user and the attacker compete to achieve different goals.When there is a disparity in the domain knowledge the user and the attacker possess, the attacker may use the user’s unfamiliarity with the domain toits advantage and further its own goal.In this situation, the observer, whose goal is to support the user may need to intervene, and this intervention needs to occur online, on-time and be accurate.We formalize the online plan intervention problem and propose a solution that uses a decision tree classifier to identify intervention points in situations where agents unwittingly facilitate an opponent’s goal.We trained a classifier using domain-independent features extracted from the observer’s decision space to evaluate the “criticality” of the current state.The trained model is then used in an online setting on IPC benchmarks to identify observations that warrant intervention.Our contributions lay a foundation for further work in the area of deciding when to intervene.

Wir stellen ein maschinelles Lernmodell vor, das bereichsunabhängige Merkmale verwendet, um die Kritikalität des aktuellen Zustands für einen bekannten unerwünschten Zustand zu schätzen.


Deep generative models can emulate the perceptual properties of complex image datasets, providing a latent representation of the data.However, manipulating such representation to perform meaningful and controllable transformations in the data space remains challenging without some form of supervision.While previous work has focused on exploiting statistical independence to \textit{disentangle} latent factors, we argue that such requirement can be advantageously relaxed and propose instead a non-statistical framework that relies on identifying a modular organization of the network, based on counterfactual manipulations.Our experiments support that modularity between groups of channels is achieved to a certain degree on a variety of generative models.This allowed the design of targeted interventions on complex image datasets, opening the way to applications such as computationally efficient style transfer and the automated assessment of robustness to contextual changes in pattern recognition systems.

Wir entwickeln einen Rahmen, um modulare interne Repräsentationen in generativen Modellen zu finden und dann zu manipulieren, um kontrafaktische Beispiele zu erzeugen.


Catastrophic forgetting poses a grand challenge for continual learning systems, which prevents neural networks from protecting old knowledge while learning new tasks sequentially.We propose a Differentiable Hebbian Plasticity (DHP) Softmax layer which adds a fast learning plastic component to the slow weights of the softmax output layer.The DHP Softmax behaves as a compressed episodic memory that reactivates existing memory traces, while creating new ones.We demonstrate the flexibility of our model by combining it with existing well-known consolidation methods to prevent catastrophic forgetting.We evaluate our approach on the Permuted MNIST and Split MNIST benchmarks, and introduce Imbalanced Permuted MNIST — a dataset that combines the challenges of class imbalance and concept drift.Our model requires no additional hyperparameters and outperforms comparable baselines by reducing forgetting.

Hebbsche plastische Gewichte können sich wie ein komprimierter episodischer Gedächtnisspeicher in neuronalen Netzen verhalten, was ihre Fähigkeit verbessert, katastrophales Vergessen beim kontinuierlichen Lernen zu lindern.


While real brain networks exhibit functional modularity, we investigate whether functional mod- ularity also exists in Deep Neural Networks (DNN) trained through back-propagation.Under the hypothesis that DNN are also organized in task-specific modules, in this paper we seek to dissect a hidden layer into disjoint groups of task-specific hidden neurons with the help of relatively well- studied neuron attribution methods.By saying task-specific, we mean the hidden neurons in the same group are functionally related for predicting a set of similar data samples, i.e. samples with similar feature patterns.We argue that such groups of neurons which we call Functional Modules can serve as the basic functional unit in DNN.We propose a preliminary method to identify Functional Modules via bi- clustering attribution scores of hidden neurons.We find that first, unsurprisingly, the functional neurons are highly sparse, i.e., only a small sub- set of neurons are important for predicting a small subset of data samples and, while we do not use any label supervision, samples corresponding to the same group (bicluster) show surprisingly coherent feature patterns.We also show that these Functional Modules perform a critical role in discriminating data samples through ablation experiment.

Wir entwickeln einen Ansatz zur Parzellierung einer versteckten Schicht in einem DNN in funktional verwandte Gruppen, indem wir spektrales Co-Clustering auf die Attributionswerte der versteckten Neuronen anwenden.


Power-efficient CNN Domain Specific Accelerator (CNN-DSA) chips are currently available for wide use in mobile devices.These chips are mainly used in computer vision applications.However, the recent work of Super Characters method for text classification and sentiment analysis tasks using two-dimensional CNN models has also achieved state-of-the-art results through the method of transfer learning from vision to text.In this paper, we implemented the text classification and sentiment analysis applications on mobile devices using CNN-DSA chips.Compact network representations using one-bit and three-bits precision for coefficients and five-bits for activations are used in the CNN-DSA chip with power consumption less than 300mW.For edge devices under memory and compute constraints, the network is further compressed by approximating the external Fully Connected (FC) layers within the CNN-DSA chip.At the workshop, we have two system demonstrations for NLP tasks.The first demo classifies the input English Wikipedia sentence into one of the 14 classes.The second demo classifies the Chinese online-shopping review into positive or negative.

Einsatz von Anwendungen zur Textklassifizierung und Stimmungsanalyse für Englisch und Chinesisch auf einem 300mW CNN-Beschleunigerchip für On-Device-Anwendungsszenarien.


Comparing the inferences of diverse candidate models is an essential part of model checking and escaping local optima.To enable efficient comparison, we introduce an amortized variational inference framework that can perform fast and reliable posterior estimation across models of the same architecture.Our Any Parameter Encoder (APE) extends the encoder neural network common in amortized inference to take both a data feature vector and a model parameter vector as input.APE thus reduces posterior inference across unseen data and models to a single forward pass.In experiments comparing candidate topic models for synthetic data and product reviews, our Any Parameter Encoder yields comparable posteriors to more expensive methods in far less time, especially when the encoder architecture is designed in model-aware fashion.

Wir entwickeln VAEs, bei denen der Encoder einen Modellparametervektor als Eingabe erhält, so dass wir für viele Modelle eine schnelle Inferenz durchführen können


The ability to transfer knowledge to novel environments and tasks is a sensible desiderata for general learning agents.Despite the apparent promises, transfer in RL is still an open and little exploited research area.In this paper, we take a brand-new perspective about transfer: we suggest that the ability to assign credit unveils structural invariants in the tasks that can be transferred to make RL more sample efficient.Our main contribution is Secret, a novel approach to transfer learning for RL that uses a backward-view credit assignment mechanism based on a self-attentive architecture.Two aspects are key to its generality: it learns to assign credit as a separate offline supervised process and exclusively modifies the reward function.Consequently, it can be supplemented by transfer methods that do not modify the reward function and it can be plugged on top of any RL algorithm.

Secret ist eine Transfermethode für RL, die auf der Anrechnung von Studienleistungen beruht.


Recent advances in Neural Variational Inference allowed for a renaissance in latent variable models in a variety of domains involving high-dimensional data.In this paper, we introduce two generic Variational Inference frameworks for generative models of Knowledge Graphs; Latent Fact Model and Latent Information Model.  While traditional variational methods derive an analytical approximation for the intractable distribution over the latent variables, here we construct an inference network conditioned on the symbolic representation of entities and relation types in the Knowledge Graph, to provide the variational distributions.The new framework can create models able to discover underlying probabilistic semantics for the symbolic representation by utilising parameterisable distributions which permit training by back-propagation in the context of neural variational inference, resulting in a highly-scalable method.Under a Bernoulli sampling framework, we provide an alternative justification for commonly used techniques in large-scale stochastic variational inference, which drastically reduces training time at a cost of an additional approximation to the variational lower bound.  The generative frameworks are flexible enough to allow training under any prior distribution that permits a re-parametrisation trick, as well as under any scoring function that permits maximum likelihood estimation of the parameters.Experiment results display the potential and efficiency of this framework by improving upon multiple benchmarks with Gaussian prior representations.Code publicly available on Github.

Arbeit an generativen Wissensgraphenmodellen zur besseren Einschätzung der Vorhersageunsicherheit bei der Wissensinferenz.


We present a deep generative model, named Monge-Amp\`ere flow, which builds on continuous-time gradient flow arising from the Monge-Amp\`ere equation in optimal transport theory.The generative map from the latent space to the data space follows a dynamical system, where a learnable potential function guides a compressible fluid to flow towards the target density distribution.Training of the model amounts to solving an optimal control problem.The Monge-Amp\`ere flow has tractable likelihoods and supports efficient sampling and inference.One can easily impose symmetry constraints in the generative model by designing suitable scalar potential functions.We apply the approach to unsupervised density estimation of the MNIST dataset and variational calculation of the two-dimensional Ising model at the critical point.This approach brings insights and techniques from Monge-Amp\`ere equation, optimal transport, and fluid dynamics into reversible flow-based generative models.

Ein auf Gradientenfluss basierendes dynamisches System für invertierbare generative Modellierung


Graph Neural Networks (GNNs) are a class of deep models that operates on data with arbitrary topology and order-invariant structure represented as graphs.We introduce an efficient memory layer for GNNs that can learn to jointly perform graph representation learning and graph pooling.We also introduce two new networks based on our memory layer: Memory-Based Graph Neural Network (MemGNN) and Graph Memory Network (GMN) that can learn hierarchical graph representations by coarsening the graph throughout the layers of memory.The experimental results demonstrate that the proposed models achieve state-of-the-art results in six out of seven graph classification and regression benchmarks.We also show that the learned representations could correspond to chemical features in the molecule data.

Wir stellen eine effiziente Speicherschicht vor, die die Repräsentation und die Vergröberung von Eingabegraphen gleichzeitig erlernen kann, ohne sich auf die Weitergabe von Nachrichten zu verlassen.


Deep neural networks require extensive computing resources, and can not be efficiently applied to embedded devices such as mobile phones, which seriously limits their applicability.To address this problem, we propose a novel encoding scheme by using {-1,+1} to decompose quantized neural networks (QNNs) into multi-branch binary networks, which can be efficiently implemented by bitwise operations (xnor and bitcount) to achieve model compression, computational acceleration and resource saving.Our method can achieve at most ~59 speedup and ~32 memory saving over its full-precision counterparts.Therefore, users can easily achieve different encoding precisions arbitrarily according to their requirements and hardware resources.Our mechanism is very suitable for the use of FPGA and ASIC in terms of data storage and computation, which provides a feasible idea for smart chips.We validate the effectiveness of our method on both large-scale image classification (e.g., ImageNet) and object detection tasks.

Ein neuartiges Kodierungsschema, das {-1, +1} verwendet, um QNNs in binäre Netze mit mehreren Zweigen zu zerlegen, in denen wir bitweise Operationen (xnor und bitcount) verwenden, um eine Modellkompression, eine Beschleunigung der Berechnungen und eine Einsparung von Ressourcen zu erreichen.


We propose a novel method for incorporating conditional information into a generative adversarial network (GAN) for structured prediction tasks.This method is based on fusing features from the generated and conditional information in feature space and allows the discriminator to better capture higher-order statistics from the data.This method also increases the strength of the signals passed through the network where the real or generated data and the conditional data agree.The proposed method is conceptually simpler than the joint convolutional neural network - conditional Markov random field (CNN-CRF) models and enforces higher-order consistency without being limited to a very specific class of high-order potentials.Experimental results demonstrate that this method leads to improvement on a variety of different structured prediction tasks including image synthesis, semantic segmentation, and depth estimation.

Wir schlagen einen neuen Weg vor, um bedingte Bildinformationen in den Diskriminator von GANs zu integrieren, indem wir eine Merkmalsfusion verwenden, die für strukturierte Vorhersageaufgaben verwendet werden kann.


Intelligent creatures can explore their environments and learn useful skills without supervision.In this paper, we propose ``Diversity is All You Need''(DIAYN), a method for learning useful skills without a reward function.Our proposed method learns skills by maximizing an information theoretic objective using a maximum entropy policy.On a variety of simulated robotic tasks, we show that this simple objective results in the unsupervised emergence of diverse skills, such as walking and jumping.In a number of reinforcement learning benchmark environments, our method is able to learn a skill that solves the benchmark task despite never receiving the true task reward.We show how pretrained skills can provide a good parameter initialization for downstream tasks, and can be composed hierarchically to solve complex, sparse reward tasks.Our results suggest that unsupervised discovery of skills can serve as an effective pretraining mechanism for overcoming challenges of exploration and data efficiency in reinforcement learning.

Wir schlagen einen Algorithmus zum Erlernen nützlicher Fähigkeiten ohne Belohnungsfunktion vor und zeigen, wie diese Fähigkeiten zur Lösung nachgelagerter Aufgaben genutzt werden können.


Lexical ambiguity, i.e., the presence of two or more meanings for a single word, is an inherent and challenging problem for machine translation systems.Even though the use of recurrent neural networks and attention mechanisms are expected to solve this problem, machine translation systems are not always able to correctly translate lexically ambiguous sentences.In this work, I attempt to resolve the problem of lexical ambiguity in English--Japanese neural machine translation systems by combining a pretrained Bidirectional Encoder Representations from Transformer (BERT) language model that can produce contextualized word embeddings and a Transformer translation model, which is a state-of-the-art architecture for the machine translation task.These two proposed architectures have been shown to be more effective in translating ambiguous sentences than a vanilla Transformer model and the Google Translate system.Furthermore, one of the proposed models, the Transformer_BERT-WE, achieves a higher BLEU score compared to the vanilla Transformer model in terms of general translation, which is concrete proof that the use of contextualized word embeddings from BERT can not only solve the problem of lexical ambiguity, but also boost the translation quality in general.

Diese Arbeit löst ein lexikalisches Mehrdeutigkeitsproblem, das durch Homonyme in der neuronalen Übersetzung durch BERT verursacht wird.


This paper focuses on the synthetic generation of human mobility data in urban areas.We present a novel and scalable application of Generative Adversarial Networks (GANs) for modeling and generating human mobility data.We leverage actual ride requests from ride sharing/hailing services from four major cities in the US to train our GANs model.Our model captures the spatial and temporal variability of the ride-request patterns observed for all four cities on any typical day and over any typical week.Previous works have succinctly characterized the spatial and temporal properties of human mobility data sets using the fractal dimensionality and the densification power law, respectively, which we utilize to validate our GANs-generated synthetic data sets.Such synthetic data sets can avoid privacy concerns and be extremely useful for researchers and policy makers on urban mobility and intelligent transportation.

In diesem Beitrag geht es um die synthetische Generierung von Daten zur menschlichen Mobilität in städtischen Gebieten mit Hilfe von GANs.


While deep neural networks are a highly successful model class, their large memory footprint puts considerable strain on energy consumption, communication bandwidth, and storage requirements.Consequently, model size reduction has become an utmost goal in deep learning.Following the classical bits-back argument, we encode the network weights using a random sample, requiring only a number of bits corresponding to the Kullback-Leibler divergence between the sampled variational distribution and the encoding distribution.By imposing a constraint on the Kullback-Leibler divergence, we are able to explicitly control the compression rate, while optimizing the expected loss on the training set.The employed encoding scheme can be shown to be close to the optimal information-theoretical lower bound, with respect to the employed variational family.On benchmarks LeNet-5/MNIST and VGG-16/CIFAR-10, our approach yields the best test performance for a fixed memory budget, and vice versa, it achieves the highest compression rates for a fixed test performance.

In dieser Arbeit wird ein effektives Kodierungsschema für neuronale Netze vorgeschlagen, das einen zufälligen Satz von Gewichten aus einer Variationsverteilung kodiert.


Deep neural networks provide state-of-the-art performance for image denoising, where the goal is to recover a near noise-free image from a noisy image.The underlying principle is that neural networks trained on large datasets have empirically been shown to be able to generate natural images well from a low-dimensional latent representation of the image.Given such a generator network, or prior, a noisy image can be denoised by finding the closest image in the range of the prior.However, there is little theory to justify this success, let alone to predict the denoising performance as a function of the networks parameters.In this paper we consider the problem of denoising an image from additive Gaussian noise, assuming the image is well described by a deep neural network with ReLu activations functions, mapping a k-dimensional latent space to an n-dimensional image.We state and analyze a simple gradient-descent-like iterative algorithm that minimizes a non-convex loss function, and provably removes a fraction of (1 - O(k/n)) of the noise energy.We also demonstrate in numerical experiments that this denoising performance is, indeed, achieved by generative priors learned from data.

Durch die Analyse eines Algorithmus, der einen nicht-konvexen Verlust minimiert, zeigen wir, dass bis auf einen kleinen Bruchteil das gesamte Rauschen aus einem Bild entfernt werden kann, indem ein auf einem tiefen neuronalen Netzwerk basierendes generatives Prior verwendet wird.


Deep learning yields great results across many fields,from speech recognition, image classification, to translation.But for each problem, getting a deep model to work well involvesresearch into the architecture and a long period of tuning.We present a single model that yields good results on a numberof problems spanning multiple domains.In particular, this single modelis trained concurrently on ImageNet, multiple translation tasks,image captioning (COCO dataset), a speech recognition corpus,and an English parsing task. Our model architecture incorporates building blocks from multipledomains.It contains convolutional layers, an attention mechanism,and sparsely-gated layers.Each of these computational blocks is crucial for a subset ofthe tasks we train on.Interestingly, even if a block is notcrucial for a task, we observe that adding it never hurts performanceand in most cases improves it on all tasks.We also show that tasks with less data benefit largely from jointtraining with other tasks, while performance on large tasks degradesonly slightly if at all.

Die groß angelegte Multitasking-Architektur löst ImageNet und Übersetzung gemeinsam und zeigt Transfer-Lernen.


Machine learning algorithms for controlling devices will need to learn quickly, with few trials.Such a goal can be attained with concepts borrowed from continental philosophy and formalized using tools from the mathematical theory of categories.Illustrations of this approach are presented on a cyberphysical system: the slot car game, and also on Atari 2600 games.

Kontinentalphilosophisch inspirierter Ansatz zum Lernen mit wenigen Daten.


Audio signals are sampled at high temporal resolutions, and learning to synthesize audio requires capturing structure across a range of timescales.Generative adversarial networks (GANs) have seen wide success at generating images that are both locally and globally coherent, but they have seen little application to audio generation.In this paper we introduce WaveGAN, a first attempt at applying GANs to unsupervised synthesis of raw-waveform audio.WaveGAN is capable of synthesizing one second slices of audio waveforms with global coherence, suitable for sound effect generation.Our experiments demonstrate that—without labels—WaveGAN learns to produce intelligible words when trained on a small-vocabulary speech dataset, and can also synthesize audio from other domains such as drums, bird vocalizations, and piano.We compare WaveGAN to a method which applies GANs designed for image generation on image-like audio feature representations, finding both approaches to be promising.

Erlernen der Synthese von Audio-Rohwellenformen mit GANs


The difficulty of obtaining sufficient labeled data for supervised learning has motivated domain adaptation, in which a classifier is trained in one domain, source domain, but operates in another, target domain.Reducing domain discrepancy has improved the performance, but it is hampered by the embedded features that do not form clearly separable and aligned clusters.We address this issue by propagating labels using a manifold structure, and by enforcing cycle consistency to align the clusters of features in each domain more closely.Specifically, we prove that cycle consistency leads the embedded features distant from all but one clusters if the source domain is ideally clustered.We additionally utilize more information from approximated local manifold and pursue local manifold consistency for more improvement.Results for various domain adaptation scenarios show tighter clustering and an improvement in classification accuracy.

Ein neuartiges Verfahren zur Anpassung von Domänen, um Vielfältigkeiten aus Quell- und Zieldomänen mit Hilfe von Labelpropagation für eine bessere Genauigkeit abzugleichen.


We propose a new method for training neural networks online in a bandit setting.Similar to prior work, we model the uncertainty only in the last layer of the network, treating the rest of the network as a feature extractor.This allows us to successfully balance between exploration and exploitation due to the efficient, closed-form uncertainty estimates available for linear models.To train the rest of the network, we take advantage of the posterior we have over the last layer, optimizing over all values in the last layer distribution weighted by probability.We derive a closed form, differential approximation to this objective and show empirically over various models and datasets that training the rest of the network in this fashion leads to both better online and offline performance when compared to other methods.

Dieses Papier schlägt eine neue Methode für das Lernen von neuronalen Netzen in Online Bandit Umgebungen vor, indem es die letzte Schicht marginalisiert


Despite a growing literature on explaining neural networks, no consensus has been reached on how to explain a neural network decision or how to evaluate an explanation.	Our contributions in this paper are twofold.First, we investigate schemes to combine explanation methods and reduce model uncertainty to obtain a single aggregated explanation.The aggregation is more robust and aligns better with the neural network than any single explanation method..	Second, we propose a new approach to evaluating explanation methods that circumvents the need for manual evaluation and is not reliant on the alignment of neural networks and humans decision processes.

Wir zeigen in Theorie und in Praxis, dass die Kombination mehrerer Erklärungsmethoden für DNN die Erklärung verbessert.


We present SOSELETO (SOurce SELEction for Target Optimization), a new method for exploiting a source dataset to solve a classification problem on a target dataset.  SOSELETO is based on the following simple intuition: some source examples are more informative than others for the target problem.  To capture this intuition, source samples are each given weights; these weights are solved for jointly with the source and target classification problems via a bilevel optimization scheme.  The target therefore gets to choose the source samples which are most informative for its own classification task.  Furthermore, the bilevel nature of the optimization acts as a kind of regularization on the target, mitigating overfitting.  SOSELETO may be applied to both classic transfer learning, as well as the problem of training on datasets with noisy labels; we show state of the art results on both of these problems.

Lernen mit begrenzten Trainingsdaten durch Ausnutzung "hilfreicher" Instanzen aus einer umfangreichen Datenquelle.


We present a novel approach for training neural abstract architectures which in- corporates (partial) supervision over the machine’s interpretable components.To cleanly capture the set of neural architectures to which our method applies, we introduce the concept of a differential neural computational machine (∂NCM) and show that several existing architectures (e.g., NTMs, NRAMs) can be instantiated as a ∂NCM and can thus benefit from any amount of additional supervision over their interpretable components.Based on our method, we performed a detailed experimental evaluation with both, the NTM and NRAM architectures, and showed that the approach leads to significantly better convergence and generalization capabilities of the learning phase than when training using only input-output examples.

Wir erhöhen den Umfang der Spurüberwachung, die beim Training voll differenzierbarer neuronaler Maschinenarchitekturen genutzt werden kann.


Bayesian learning of model parameters in neural networks is important in scenarios where estimates with well-calibrated uncertainty are important.In this paper, we propose Bayesian quantized networks (BQNs), quantized neural networks (QNNs) for which we learn a posterior distribution over their discrete parameters.We provide a set of efficient algorithms for learning and prediction in BQNs without the need to sample from their parameters or activations, which not only allows for differentiable learning in quantized models but also reduces the variance in gradients estimation.We evaluate BQNs on MNIST, Fashion-MNIST and KMNIST classification datasets compared against bootstrap ensemble of QNNs (E-QNN).We demonstrate BQNs achieve both lower predictive errors and better-calibrated uncertainties than E-QNN (with less than 20% of the negative log-likelihood).

Wir schlagen Bayes'sche quantisierte Netzwerke vor, für die wir eine Posterior-Verteilung über ihre quantisierten Parameter lernen.


Injecting adversarial examples during training, known as adversarial training, can improve robustness against one-step attacks, but not for unknown iterative attacks.To address this challenge, we first show iteratively generated adversarial images easily transfer between networks trained with the same strategy.Inspired by this observation, we propose cascade adversarial training, which transfers the knowledge of the end results of adversarial training.We train a network from scratch by injecting iteratively generated adversarial images crafted from already defended networks in addition to one-step adversarial images from the network being trained.We also propose to utilize embedding space for both classification and low-level (pixel-level) similarity learning to ignore unknown pixel level perturbation.During training, we inject adversarial images without replacing their corresponding clean images and penalize the distance between the two embeddings (clean and adversarial).Experimental results show that cascade adversarial training together with our proposed low-level similarity learning efficiently enhances the robustness against iterative attacks, but at the expense of decreased robustness against one-step attacks.We show that combining those two techniques can also improve robustness under the worst case black box attack scenario.

Kaskadentraining und Ähnlichkeitslernen auf niedriger Ebene verbessern die Robustheit gegen White-Box- und Black-Box-Angriffe.


Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities ``solve'' the exploding gradient problem, we show that this is not the case and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice.We explain why exploding gradients occur and highlight the {\it collapsing domain problem}, which can arise in architectures that avoid exploding gradients. ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks, which we show is a consequence of a surprising mathematical property.By noticing that {\it any neural network is a residual network}, we devise the {\it residual trick}, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.

Wir zeigen, dass das Problem der explodierenden Gradienten entgegen der landläufigen Meinung noch nicht gelöst ist und dass es die Tiefe begrenzt, bis zu der MLPs effektiv trainiert werden können. Wir zeigen, warum Gradienten explodieren und wie ResNet mit ihnen umgeht.


In this paper, we are interested in two seemingly different concepts: \textit{adversarial training} and \textit{generative adversarial networks (GANs)}.Particularly, how these techniques work to improve each other.To this end, we analyze the limitation of adversarial training as a defense method, starting from questioning how well the robustness of a model can generalize.Then, we successfully improve the generalizability via data augmentation by the ``fake'' images sampled from generative adversarial network.After that, we are surprised to see that the resulting robust classifier leads to a better generator, for free.We intuitively explain this interesting phenomenon and leave the theoretical analysis for future work.Motivated by these observations, we propose a system that combines generator, discriminator, and adversarial attacker together in a single network.After end-to-end training and fine tuning, our method can simultaneously improve the robustness of classifiers, measured by accuracy under strong adversarial attacks, and the quality of generators, evaluated both aesthetically and quantitatively.In terms of the classifier, we achieve better robustness than the state-of-the-art adversarial training algorithm proposed in (Madry \textit{et al.}, 2017), while our generator achieves competitive performance compared with SN-GAN (Miyato and Koyama, 2018).

Wir haben herausgefunden, dass adversariales Training nicht nur das GAN-Training beschleunigt, sondern auch die Bildqualität verbessert.


We present a method to train self-binarizing neural networks, that is, networks that evolve their weights and activations during training to become binary.To obtain similar binary networks, existing methods rely on the sign activation function.This function, however, has no gradients for non-zero values, which makes standard backpropagation impossible.To circumvent the difficulty of training a network relying on the sign activation function, these methods alternate between floating-point and binary representations of the network during training, which is sub-optimal and inefficient.We approach the binarization task by training on a unique representation involving a smooth activation function, which is iteratively sharpened during training until it becomes a binary representation equivalent to the sign activation function.Additionally, we introduce a new technique to perform binary batch normalization that simplifies the conventional batch normalization by transforming it into a simple comparison operation.This is unlike existing methods, which are forced to the retain the conventional floating-point-based batch normalization.Our binary networks, apart from displaying advantages of lower memory and computation as compared to conventional floating-point and binary networks, also show higher classification accuracy than existing state-of-the-art methods on multiple benchmark datasets.

Eine Methode zur Binarisierung sowohl der Gewichte als auch der Aktivierungen eines tiefen neuronalen Netzes, die rechen- und speichereffizient ist und bessere Ergebnisse als der Stand der Technik liefert.


In many applications, the training data for a machine learning task is partitioned across multiple nodes, and aggregating this data may be infeasible due to storage, communication, or privacy constraints.In this work, we present Good-Enough Model Spaces (GEMS), a novel framework for learning a global satisficing (i.e. "good-enough") model within a few communication rounds by carefully combining the space of local nodes' satisficing models.In experiments on benchmark and medical datasets, our approach outperforms other baseline aggregation techniques such as ensembling or model averaging, and performs comparably to the ideal non-distributed models.

Wir stellen Good-Enough Model Spaces (GEMS) vor, einen Rahmen für das Lernen eines aggregierten Modells über verteilte Knoten innerhalb einer kleinen Anzahl von Kommunikationsrunden.


We propose the Wasserstein Auto-Encoder (WAE)---a new algorithm for building a generative model of the data distribution.WAE minimizes a penalized form of the Wasserstein distance between the model distribution and the target distribution, which leads to a different regularizer than the one used by the Variational Auto-Encoder (VAE).This regularizer encourages the encoded training distribution to match the prior.We compare our algorithm with several other techniques and show that it is a generalization of adversarial auto-encoders (AAE).Our experiments show that WAE shares many of the properties of VAEs (stable training, encoder-decoder architecture, nice latent manifold structure) while generating samples of better quality.

Wir schlagen einen neuen Auto-Encoder vor, der auf der Wasserstein-Distanz basiert und die Sampling-Eigenschaften der VAE verbessert.


In this paper, we aim to develop a novel mechanism to preserve differential privacy (DP) in adversarial learning for deep neural networks, with provable robustness to adversarial examples.We leverage the sequential composition theory in DP, to establish a new connection between DP preservation and provable robustness.To address the trade-off among model utility, privacy loss, and robustness, we design an original, differentially private, adversarial objective function, based on the post-processing property in DP, to tighten the sensitivity of our model.An end-to-end theoretical analysis and thorough evaluations show that our mechanism notably improves the robustness of DP deep neural networks.

Wahrung der differential privacy beim adversarial Lernen mit beweisbarer Robustheit gegenüber gegnerischen Beispielen


In high-dimensional reinforcement learning settings with sparse rewards, performingeffective exploration to even obtain any reward signal is an open challenge.While model-based approaches hold promise of better exploration via planning, itis extremely difficult to learn a reliable enough Markov Decision Process (MDP)in high dimensions (e.g., over 10^100 states).In this paper, we propose learningan abstract MDP over a much smaller number of states (e.g., 10^5), which we canplan over for effective exploration.We assume we have an abstraction functionthat maps concrete states (e.g., raw pixels) to abstract states (e.g., agent position,ignoring other objects).In our approach, a manager maintains an abstractMDP over a subset of the abstract states, which grows monotonically through targetedexploration (possible due to the abstract MDP).Concurrently, we learn aworker policy to travel between abstract states; the worker deals with the messinessof concrete states and presents a clean abstraction to the manager.On three ofthe hardest games from the Arcade Learning Environment (Montezuma's,Pitfall!, and Private Eye), our approach outperforms the previousstate-of-the-art by over a factor of 2 in each game.In Pitfall!, our approach isthe first to achieve superhuman performance without demonstrations.

Wir konstruieren und untersuchen automatisch einen kleinen abstrakten Markov-Entscheidungsprozess, der es uns ermöglicht, bei Montezumas Rache, Pitfall! und Private Eye mit deutlichem Abstand die besten Ergebnisse zu erzielen.


Most deep reinforcement learning (RL) systems are not able to learn effectively from off-policy data, especially if they cannot explore online in the environment.This is a critical shortcoming for applying RL to real-world problems where collecting data is expensive, and models must be tested offline before being deployed to interact with the environment -- e.g. systems that learn from human interaction.Thus, we develop a novel class of off-policy batch RL algorithms which use KL-control to penalize divergence from a pre-trained prior model of probable actions.This KL-constraint reduces extrapolation error, enabling effective offline learning, without exploration, from a fixed batch of data.We also use dropout-based uncertainty estimates to lower bound the target Q-values as a more efficient alternative to Double Q-Learning.This Way Off-Policy (WOP) algorithm is tested on both traditional RL tasks from OpenAI Gym, and on the problem of open-domain dialog generation; a challenging reinforcement learning problem with a 20,000 dimensional action space.WOP allows for the extraction of multiple different reward functions post-hoc from collected human interaction data, and can learn effectively from all of these.We test real-world generalization by deploying dialog models live to converse with humans in an open-domain setting, and demonstrate that WOP achieves significant improvements over state-of-the-art prior methods in batch deep RL.

Wir zeigen, dass die KL-Kontrolle durch einen vortrainierten Prior RL-Modelle in die Lage versetzt, aus einem statischen Stapel gesammelter Daten zu lernen, ohne die Möglichkeit, die Umgebung online zu erkunden.


Transfer and adaptation to new unknown environmental dynamics is a key challenge for reinforcement learning (RL).An even greater challenge is performing near-optimally in a single attempt at test time, possibly without access to dense rewards, which is not addressed by current methods that require multiple experience rollouts for adaptation.To achieve single episode transfer in a family of environments with related dynamics, we propose a general algorithm that optimizes a probe and an inference model to rapidly estimate underlying latent variables of test dynamics, which are then immediately used as input to a universal control policy.This modular approach enables integration of state-of-the-art algorithms for variational inference or RL.Moreover, our approach does not require access to rewards at test time, allowing it to perform in settings where existing adaptive approaches cannot.In diverse experimental domains with a single episode test constraint, our method significantly outperforms existing adaptive approaches and shows favorable performance against baselines for robust transfer.

Übertragung von Einzelmaßnahmen in einer Familie von Umgebungen mit verwandter Dynamik durch optimiertes Sondieren zur schnellen Ermittlung latenter Variablen und sofortige Ausführung einer universellen Maßnahme.


Domain specific goal-oriented dialogue systems typically require modeling three types of inputs, viz.,(i) the knowledge-base associated with the domain,(ii) the history of the conversation, which is a sequence of utterances and(iii) the current utterance for which the response needs to be generated.While modeling these inputs, current state-of-the-art models such as Mem2Seq typically ignore the rich structure inherent in the knowledge graph and the sentences in the conversation context.Inspired by the recent success of structure-aware Graph Convolutional Networks (GCNs) for various NLP tasks such as machine translation, semantic role labeling and document dating, we propose a memory augmented GCN for goal-oriented dialogues.Our model exploits(i) the entity relation graph in a knowledge-base  and(ii) the dependency graph associated with an utterance to compute richer representations for words and entities.Further, we take cognizance of the fact that in certain situations, such as, when the conversation is in a code-mixed language, dependency parsers may not be available.We show that in such situations we could use the global word co-occurrence graph and use it to enrich the representations of utterances.We experiment with the modified DSTC2 dataset and its recently released code-mixed versions in four languages and show that our method outperforms existing state-of-the-art methods, using a wide range of evaluation metrics.

Wir schlagen ein Graph Convolutional Network basiertes Encoder-Decoder Modell mit sequentieller Aufmerksamkeit für zielorientierte Dialogsysteme vor.


Effectively capturing graph node sequences in the form of vector embeddings is critical to many applications.We achieve this by(i) first learning vector embeddings of single graph nodes and(ii) then composing them to compactly represent node sequences.Specifically, we propose SENSE-S (Semantically Enhanced Node Sequence Embedding - for Single nodes), a skip-gram based novel embedding mechanism, for single graph nodes that co-learns graph structure as well as their textual descriptions.We demonstrate that SENSE-S vectors increase the accuracy of multi-label classification tasks by up to 50% and link-prediction tasks by up to 78% under a variety of scenarios using real datasets.Based on SENSE-S, we next propose generic SENSE to compute composite vectors that represent a sequence of nodes, where preserving the node order is important.We prove that this approach is efficient in embedding node sequences, and our experiments on real data confirm its high accuracy in node order decoding.

Mechanismus zur Einbettung von Knotenfolgen, der sowohl Graph- als auch Texteigenschaften erfasst.


Recent evidence shows that convolutional neural networks (CNNs) are biased towards textures so that CNNs are non-robust to adversarial perturbations over textures, while traditional robust visual features like SIFT (scale-invariant feature transforms) are designed to be robust across a substantial range of affine distortion, addition of noise, etc with the mimic of human perception nature.This paper aims to leverage good properties of SIFT to renovate CNN architectures towards better accuracy and robustness.We borrow the scale-space extreme value idea from SIFT, and propose EVPNet (extreme value preserving network) which contains three novel components to model the extreme values: (1) parametric differences of Gaussian (DoG) to extract extrema, (2) truncated ReLU to suppress non-stable extrema and (3) projected normalization layer (PNL) to mimic PCA-SIFT like feature normalization.Experiments demonstrate that EVPNets can achieve similar or better accuracy than conventional CNNs, while achieving much better robustness on a set of adversarial attacks (FGSM,PGD,etc) even without adversarial training.

Dieses Papier zielt darauf ab, die guten Eigenschaften von robusten visuellen Merkmalen wie SIFT zu nutzen, um CNN-Architekturen im Hinblick auf bessere Genauigkeit und Robustheit zu renovieren.


Compressed representations generalize better (Shamir et al., 2010), which may be crucial when learning from limited or noisy labeled data.The Information Bottleneck (IB) method (Tishby et al. (2000)) provides an insightful and principled approach for balancing compression and prediction in representation learning.The IB objective I(X; Z) − βI(Y ; Z) employs a Lagrange multiplier β to tune this trade-off.However, there is little theoretical guidance for how to select β.There is also a lack of theoretical understanding about the relationship between β, the dataset, model capacity, and learnability.In this work, we show that if β is improperly chosen, learning cannot happen: the trivial representation P(Z|X) = P(Z) becomes the global minimum of the IB objective.We show how this can be avoided, by identifying a sharp phase transition between the unlearnable and the learnable which arises as β varies.This phase transition defines the concept of IB-Learnability.We prove several sufficient conditions for IB-Learnability, providing theoretical guidance for selecting β.We further show that IB-learnability is determined by the largest confident, typical, and imbalanced subset of the training examples.We give a practical algorithm to estimate the minimum β for a given dataset.We test our theoretical results on synthetic datasets, MNIST, and CIFAR10 with noisy labels, and make the surprising observation that accuracy may be non-monotonic in β.

Die Theorie sagt den Phasenübergang zwischen nicht erlernbaren und erlernbaren Werten von Beta für das Ziel des Informationsengpasses voraus


We consider a new class of \emph{data poisoning} attacks on neural networks, in which the attacker takes control of a model by making small perturbations to a subset of its training data.  We formulate the task of finding poisons as a bi-level optimization problem, which can be solved using methods borrowed from the meta-learning community.  Unlike previous poisoning strategies, the meta-poisoning can poison networks that are trained from scratch using an initialization unknown to the attacker and transfer across hyperparameters.Further we show that our attacks are more versatile: they can cause misclassification of the target image into an arbitrarily chosen class.Our results show above 50% attack success rate when poisoning just 3-10% of the training dataset.

Erzeugen Sie verfälschte Trainingsbilder, die nicht wahrnehmbar sind, aber das CNN-Verhalten auf einem Ziel bei jedem neuen Training verändern.


We give a new algorithm for learning a two-layer neural network under a very general class of input distributions.Assuming there is a ground-truth two-layer network y = A \sigma(Wx) + \xi,where A, W are weight matrices, \xi represents noise, and the number of neurons in the hidden layer is no larger than the input or output,  our algorithm is guaranteed to recover the parameters A, W of the ground-truth network.The only requirement on the input x is that it is symmetric, which still allows highly complicated and structured input. Our algorithm is based on the method-of-moments framework and extends several results in tensor decompositions.We use spectral algorithms to avoid the complicated non-convex optimization in learning neural networks.Experiments show that our algorithm can robustly learn the ground-truth neural network with a small number of samples for many symmetric input distributions.

Wir geben einen Algorithmus zum Lernen eines zweischichtigen neuronalen Netzes mit symmetrischer Eingangsverteilung an.


Teachers intentionally pick the most informative examples to show their students.However, if the teacher and student are neural networks, the examples that the teacher network learns to give, although effective at teaching the student, are typically uninterpretable.We show that training the student and teacher iteratively, rather than jointly, can produce interpretable teaching strategies.We evaluate interpretability by (1) measuring the similarity of the teacher's emergent strategies to intuitive strategies in each domain and (2) conducting human experiments to evaluate how effective the teacher's strategies are at teaching humans.We show that the teacher network learns to select or generate interpretable, pedagogical examples to teach rule-based, probabilistic, boolean, and hierarchical concepts.

Wir zeigen, dass das iterative Training eines Schüler- und Lehrernetzwerks, statt eines gemeinsamen, zu neuen, interpretierbaren Lehrstrategien führen kann.


Stochastic gradient descent (SGD), which  trades off noisy gradient updates for computational efficiency, is the de-facto optimization algorithm to solve large-scale machine learning problems.SGD can make rapid learning progress by performing updates using subsampled training data, but the noisy updates also lead to slow asymptotic convergence.   Several variance reduction algorithms, such as SVRG, introduce control variates to obtain a lower variance gradient estimate and faster convergence.  Despite their appealing asymptotic guarantees, SVRG-like algorithms have not been widely adopted in deep learning.The traditional asymptotic analysis in stochastic optimization provides limited insight into training deep learning models under a fixed number of epochs.In this paper, we present a non-asymptotic analysis of SVRG under a noisy least squares regression problem.Our primary focus is to compare the exact loss of SVRG to that of SGD at each iteration t.We show that the learning dynamics of our regression model closely matches with that of neural networks on MNIST and CIFAR-10 for both the underparameterized and the overparameterized models.Our analysis and experimental results suggest there is a trade-off between the computational cost and the convergence speed in underparametrized neural networks.SVRG outperforms SGD after a few epochs in this regime.However, SGD is shown to always outperform SVRG in the overparameterized regime.

Nicht-asymptotische Analyse von SGD und SVRG, die die Stärke der beiden Algorithmen in Bezug auf Konvergenzgeschwindigkeit und Rechenkosten sowohl in unter- als auch in überparametrisierten Einstellungen zeigt.


Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models.Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance.Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear.In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training.We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data.Furthermore, a strong correlation is observed between the capacity of an NAT model and the optimal complexity of the distilled data for the best translation quality.Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models.We achieve the state-of-the-art performance for the NAT-based models, and close the gap with the autoregressive baseline on WMT14 En-De benchmark.

Wir untersuchen systematisch, warum die Wissensdestillation für das Training von nicht-autoregressiven Übersetzungsmodellen (NAT) entscheidend ist, und schlagen Methoden zur weiteren Verbesserung der destillierten Daten vor, um die Kapazität eines NAT-Modells bestmöglich zu erfüllen.


Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP), achieving state-of-the-art results in domains such as language modeling and machine translation.Harnessing the transformer's ability to process long time horizons of information could provide a similar performance boost in partially-observable reinforcement learning (RL) domains, but the large-scale transformers used in NLP have yet to be successfully applied to the RL setting.In this work we demonstrate that the standard transformer architecture is difficult to optimize, which was previously observed in the supervised learning setting but becomes especially pronounced with RL objectives.We propose architectural modifications that substantially improve the stability and learning speed of the original Transformer and XL variant.The proposed architecture, the Gated Transformer-XL (GTrXL), surpasses LSTMs on challenging memory environments and achieves state-of-the-art results on the multi-task DMLab-30 benchmark suite, exceeding the performance of an external memory architecture.We show that the GTrXL, trained using the same losses, has stability and performance that consistently matches or exceeds a competitive LSTM baseline, including on more reactive tasks where memory is less critical.GTrXL offers an easy-to-train, simple-to-implement but substantially more expressive architectural alternative to the standard multi-layer LSTM ubiquitously used for RL agents in partially-observable environments.

Es gelingt uns, Transformatoren für das Training in der RL-Umgebung zu stabilisieren und eine große Verbesserung gegenüber LSTMs auf DMLab-30 zu demonstrieren, was einer externen Speicherarchitektur entspricht.


Knowledge distillation is an effective model compression technique in which a smaller model is trained to mimic a larger pretrained model.However in order to make these compact models suitable for real world deployment, not only dowe need to reduce the performance gap but also we need to make them more robust to commonly occurring and adversarial perturbations.Noise permeates every level of the nervous system, from the perception of sensory signals to thegeneration of motor responses.We therefore believe that noise could be a crucial element in improving neural networks training and addressing the apparently contradictory goals of improving both the generalization and robustness of themodel.Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise at either the input level or the supervision signals.Our results show that noise can improve both the generalization and robustness of the model.”Fickle Teacher” which uses dropout in teacher model as a source of response variation leads to significant generalization improvement.”Soft Randomization”, which matches the output distribution ofthe student model on the image with Gaussian noise to the output of the teacher on original image, improves the adversarial robustness manifolds compared to the student model trained with Gaussian noise.We further show the surprising effect of random label corruption on a model’s adversarial robustness.The study highlights the benefits of adding constructive noise in the knowledge distillation framework and hopes to inspire further work in the area.

Inspiriert von der Versuch zu Versuch Variabilität im Gehirn, die aus mehreren Rauschquellen resultieren kann, haben wir die Variabilität durch Rauschen in den Rahmen der Wissensdestillation eingeführt und ihre Auswirkungen auf die Generalisierung und Robustheit untersucht.


We introduce a novel end-to-end approach for learning to cluster in the absence of labeled examples.Our clustering objective is based on optimizing normalized cuts, a criterion which measures both intra-cluster similarity as well as inter-cluster dissimilarity.We define a differentiable loss function equivalent to the expected normalized cuts.Unlike much of the work in unsupervised deep learning, our trained model directly outputs final cluster assignments, rather than embeddings that need further processing to be usable.Our approach generalizes to unseen datasets across a wide variety of domains, including text, and image.Specifically, we achieve state-of-the-art results on popular unsupervised clustering benchmarks (e.g., MNIST, Reuters, CIFAR-10, and CIFAR-100), outperforming the strongest baselines by up to 10.9%.Our generalization results are superior (by up to 21.9%) to the recent top-performing clustering approach with the ability to generalize.

Wir stellen einen neuartigen Ende-zu-Ende Ansatz für das Lernen von Clustern in Abwesenheit von markierten Beispielen vor. Wir definieren eine differenzierbare Verlustfunktion, die den erwarteten normalisierten Schnitten entspricht.


We introduce the largest (among publicly available) dataset for Cyrillic Handwritten Text Recognition and the first dataset for Cyrillic Text in the Wild Recognition, as well as suggest a method for recognizing Cyrillic Handwritten Text and Text in the Wild.Based on this approach, we develop a system that can reduce the document processing time for one of the largest mathematical competitions in Ukraine by 12 days and the amount of used paper by 0.5 ton.

Wir stellen verschiedene Datensätze für die kyrillische OCR und eine Methode zu deren Erkennung vor


Most deep learning for NLP represents each word with a single point or single-mode region in semantic space, while the existing multi-mode word embeddings cannot represent longer word sequences like phrases or sentences.We introduce a phrase representation (also applicable to sentences) where each phrase has a distinct set of multi-mode codebook embeddings to capture different semantic facets of the phrase's meaning.The codebook embeddings can be viewed as the cluster centers which summarize the distribution of possibly co-occurring words in a pre-trained word embedding space.We propose an end-to-end trainable neural model that directly predicts the set of cluster centers from the input text sequence (e.g., a phrase or a sentence) during test time.We find that the per-phrase/sentence codebook embeddings not only provide a more interpretable semantic representation but also outperform strong baselines (by a large margin in some tasks) on benchmark datasets for unsupervised phrase similarity, sentence similarity, hypernym detection, and extractive summarization.

Wir schlagen einen unüberwachten Weg vor, um mehrere Einbettungen für Sätze und Phrasen zu lernen


We present a meta-learning approach for adaptive text-to-speech (TTS) with few data.During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker.The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system.Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers.We introduce and benchmark three strategies:(i) learning the speaker embedding while keeping the WaveNet core fixed,(ii) fine-tuning the entire architecture with stochastic gradient descent, and(iii) predicting the speaker embedding with a trained neural network encoder.The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art results in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers.

Beispiel für effiziente Algorithmen zur Anpassung eines Text-to-Speech-Modells an einen neuen Sprachstil mit modernster Leistung.


This paper introduces a probabilistic framework for k-shot image classification.The goal is to generalise from an initial large-scale classification task to a separate task comprising new  classes and small numbers of examples.The new approach not only leverages the feature-based representation learned by a neural network from the initial task (representational transfer), but also information about the classes (concept transfer).The concept information is encapsulated in a probabilistic model for the final layer weights of the neural network which acts as a prior for probabilistic k-shot learning.We show that even a simple probabilistic model achieves state-of-the-art on a standard k-shot learning dataset by a large margin.Moreover, it is able to accurately model uncertainty, leading to well calibrated classifiers, and is easily extensible and flexible, unlike many recent approaches to k-shot learning.

In diesem Beitrag wird ein probabilistisches Verfahren zur Klassifizierung von k-shot-Bildern vorgestellt, das die besten Ergebnisse erzielt


Building on the recent successes of distributed training of RL agents, in this paper we investigate the training of RNN-based RL agents from distributed prioritized experience replay.We study the effects of parameter lag resulting in representational drift and recurrent state staleness and empirically derive an improved training strategy.Using a single network architecture and fixed set of hyper-parameters, the resulting agent, Recurrent Replay Distributed DQN, quadruples the previous state of the art on Atari-57, and matches the state of the art on DMLab-30.It is the first agent to exceed human-level performance in 52 of the 57 Atari games.

Untersuchung der Kombination von rekurrenten neuronalen Netzen und Erfahrungswiedergabe, die zu einem hochmodernen Agenten auf dem Atari-57 und dem DMLab-30 mit einem einzigen Satz von Hyperparametern führt.


The current state-of-the-art end-to-end semantic role labeling (SRL) model is a deep neural network architecture with no explicit linguistic features. However, prior work has shown that gold syntax trees can dramatically improve SRL, suggesting that neural network models could see great improvements from explicit modeling of syntax.In this work, we present linguistically-informed self-attention (LISA): a new neural network model that combines multi-head self-attention with multi-task learning across dependency parsing, part-of-speech, predicate detection and SRL.For example, syntax is incorporated by training one of the attention heads to attend to syntactic parents for each token.Our model can predict all of the above tasks, but it is also trained such that if a high-quality syntactic parse is already available, it can be beneficially injected at test time without re-training our SRL model.In experiments on the CoNLL-2005 SRL dataset LISA achieves an increase of 2.5 F1 absolute over the previous state-of-the-art on newswire with predicted predicates and more than 2.0 F1 on out-of-domain data.On ConLL-2012 English SRL we also show an improvement of more than 3.0 F1, a 13% reduction in error.

Unsere Kombination aus Multi-Task-Lernen und Self-Attention, bei der das Modell trainiert wird, die Eltern in einem syntaktischen Parse-Baum zu beachten, erzielt modernste CoNLL-2005- und CoNLL-2012-SRL-Ergebnisse für Modelle mit vorhergesagten Prädikaten.


Bottleneck structures with identity (e.g., residual) connection are now emerging popular paradigms for designing deep convolutional neural networks (CNN), for processing large-scale features efficiently.In this paper, we focus on the information-preserving nature of identity connection and utilize this to enable a convolutional layer to have a new functionality of channel-selectivity, i.e., re-distributing its computations to important channels.In particular, we propose Selective Convolutional Unit (SCU), a widely-applicable architectural unit that improves parameter efficiency of various modern CNNs with bottlenecks.During training, SCU gradually learns the channel-selectivity on-the-fly via the alternative usage of(a) pruning unimportant channels, and(b) rewiring the pruned parameters to important channels.The rewired parameters emphasize the target channel in a way that selectively enlarges the convolutional kernels corresponding to it.Our experimental results demonstrate that the SCU-based models without any postprocessing generally achieve both model compression and accuracy improvement compared to the baselines, consistently for all tested architectures.

Wir schlagen ein neues Modul vor, das alle ResNet-ähnlichen Architekturen verbessert, indem es den Faltungsschichten ein "kanalselektives" Verhalten aufzwingt.


Predictive models that generalize well under distributional shift are often desirable and sometimes crucial to machine learning applications.One example is the estimation of treatment effects from observational data, where a subtask is to predict the effect of a treatment on subjects that are systematically different from those who received the treatment in the data.A related kind of distributional shift appears in unsupervised domain adaptation, where we are tasked with generalizing to a distribution of inputs that is different from the one in which we observe labels.We pose both of these problems as prediction under a shift in design.Popular methods for overcoming distributional shift are often heuristic or rely on assumptions that are rarely true in practice, such as having a well-specified model or knowing the policy that gave rise to the observed data.Other methods are hindered by their need for a pre-specified metric for comparing observations, or by poor asymptotic properties.In this work, we devise a bound on the generalization error under design shift, based on integral probability metrics and sample re-weighting.We combine this idea with representation learning, generalizing and tightening existing results in this space.Finally, we propose an algorithmic framework inspired by our bound and verify is effectiveness in causal effect estimation.

Eine Theorie und ein algorithmischer Rahmen für die Vorhersage unter Verteilungsänderungen, einschließlich der Schätzung kausaler Effekte und der Anpassung von Bereichen


Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty  and out-of-distribution robustness of deep learning models.While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well.Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift.One possible explanation for this gap between theory and practice is that popular scalable approximate Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space.We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions.Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space.We demonstrate that while low-loss connectors between modes exist, they are not connected in the space of predictions.Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods.

Wir untersuchen tiefe Ensembles durch die Linse der Verlustlandschaft und den Raum der Vorhersagen und zeigen, dass die Dekorrelationskraft von zufälligen Initialisierungen durch Unterraum-Sampling, das nur einen einzigen Modus erforscht, nicht erreicht wird.


Existing deep learning approaches for learning visual features tend to extract more information than what is required for the task at hand.From a privacy preservation perspective, the input visual information is not protected from the model; enabling the model to become more intelligent than it is trained to be.Existing approaches for suppressing additional task learning assume the presence of ground truth labels for the tasks to be suppressed during training time.In this research, we propose a three-fold novel contribution:(i) a novel metric to measure the trust score of a trained deep learning model,(ii) a model-agnostic solution framework for trust score improvement by suppressing all the unwanted tasks, and(iii) a simulated benchmark dataset, PreserveTask, having five different fundamental image classification tasks to study the generalization nature of models.In the first set of experiments, we measure and improve the trust scores of five popular deep learning models: VGG16, VGG19, Inception-v1, MobileNet, and DenseNet and demonstrate that Inception-v1 is having the lowest trust score.Additionally, we show results of our framework on color-MNIST dataset and practical applications of face attribute preservation in Diversity in Faces (DiF) and IMDB-Wiki dataset.

Können wir unseren Deep-Learning-Modellen vertrauen? Ein Rahmen zur Messung und Verbesserung des Vertrauens eines Deep-Learning-Modells während des Trainings.


Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training.In this work, we propose to train a policy while explicitly penalizing the mismatch between these two distributions over a fixed time horizon.We do this by using a learned model of the environment dynamics which is unrolled for multiple time steps, and training a policy network to minimize a differentiable cost over this rolled-out trajectory.This cost contains two terms: a policy cost which represents the objective the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on.We propose to measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks.We evaluate our approach using a large-scale observational dataset of driving behavior recorded from traffic cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.

Ein modellbasierter RL-Ansatz, der eine differenzierbare Unsicherheitsstrafe verwendet, um Fahrstrategien aus reinen Beobachtungsdaten zu lernen.


Dynamical system models (including RNNs) often lack the ability to adapt the sequence generation or prediction to a given context, limiting their real-world application.In this paper we show that hierarchical multi-task dynamical systems (MTDSs) provide direct user control over sequence generation, via use of a latent  code z that specifies the customization to theindividual data sequence.This enables style transfer, interpolation and morphing within generated sequences.We show the MTDS can improve predictions via latent code interpolation, and avoid the long-term performance degradation of standard RNN approaches.

Anpassung der Vorhersagen von Sequenzmodellen (wie LDS und RNN) über einen expliziten latenten Code.


By injecting adversarial examples into training data, adversarial training is promising for improving the robustness of deep learning models.However, most existing adversarial training approaches are based on a specific type of adversarial attack.It may not provide sufficiently representative samples from the adversarial domain, leading to a weak generalization ability on adversarial examples from other attacks.Moreover, during the adversarial training, adversarial perturbations on inputs are usually crafted by fast single-step adversaries so as to scale to large datasets.This work is mainly focused on the adversarial training yet efficient FGSM adversary.In this scenario, it is difficult to train a model with great generalization due to the lack of representative adversarial samples, aka the samples are unable to accurately reflect the adversarial domain.To alleviate this problem, we propose a novel Adversarial Training with Domain Adaptation (ATDA) method.Our intuition is to regard the adversarial training on FGSM adversary as a domain adaption task with limited number of target domain samples.The main idea is to learn a representation that is semantically meaningful and domain invariant on the clean domain as well as the adversarial domain.Empirical evaluations on Fashion-MNIST, SVHN, CIFAR-10 and CIFAR-100 demonstrate that ATDA can greatly improve the generalization of adversarial training and the smoothness of the learned models, and outperforms state-of-the-art methods on standard benchmark datasets.To show the transfer ability of our method, we also extend ATDA to the adversarial training on iterative attacks such as PGD-Adversial Training (PAT) and the defense performance is improved considerably.

Wir schlagen ein neuartiges Training mit Domänenanpassung vor, das die Verallgemeinerungsfähigkeit von Beispielen aus verschiedenen Angriffen deutlich verbessert.


Character-level language modeling is an essential but challenging task in Natural Language Processing. Prior works have focused on identifying long-term dependencies between characters and have built deeper and wider networks for better performance.However, their models require substantial computational resources, which hinders the usability of character-level language models in applications with limited resources.In this paper, we propose a lightweight model, called Group-Transformer, that reduces the resource requirements for a Transformer, a promising method for modeling sequence with long-term dependencies.Specifically, the proposed method partitions linear operations to reduce the number of parameters and computational cost.As a result, Group-Transformer only uses 18.2\% of parameters compared to the best performing LSTM-based model, while providing better performance on two benchmark tasks, enwik8 and text8.When compared to Transformers with a comparable number of parameters and time complexity, the proposed model shows better performance.The implementation code will be available.

In diesem Beitrag wird ein neuartiger, leichtgewichtiger Transformer für die Sprachmodellierung auf Zeichenebene vorgeschlagen, der gruppenweise Operationen verwendet.


Domain adaptation tackles the problem of transferring knowledge from a label-rich source domain to an unlabeled or label-scarce target domain.Recently domain-adversarial training (DAT) has shown promising capacity to learn a domain-invariant feature space by reversing the gradient propagation of a domain classifier.However, DAT is still vulnerable in several aspects including (1) training instability due to the overwhelming discriminative ability of the domain classifier in adversarial training, (2) restrictive feature-level alignment, and (3) lack of interpretability or systematic explanation of the learned feature space.In this paper, we propose a novel Max-margin Domain-Adversarial Training (MDAT) by designing an Adversarial Reconstruction Network (ARN).The proposed MDAT stabilizes the gradient reversing in ARN by replacing the domain classifier with a reconstruction network, and in this manner ARN conducts both feature-level and pixel-level domain alignment without involving extra network structures.Furthermore, ARN demonstrates strong robustness to a wide range of hyper-parameters settings, greatly alleviating the task of model selection.Extensive empirical results validate that our approach outperforms other state-of-the-art domain alignment methods.Additionally, the reconstructed target samples are visualized to interpret the domain-invariant feature space which conforms with our intuition.

Ein stabiler domänenadversarischer Trainingsansatz für robuste und umfassende Domänenanpassung


We present an approach for expanding taxonomies with synonyms, or aliases.We target large shopping taxonomies, with thousands of nodes.A comprehensive set of entity aliases is an important component of identifying entities in unstructured text such as product reviews or search queries.Our method consists of two stages: we generate synonym candidates from WordNet and shopping search queries, then use a binary classiﬁer to ﬁlter candidates.We process taxonomies with thousands of synonyms in order to generate over 90,000 synonyms.We show that using the taxonomy to derive contextual features improves classiﬁcation performance over using features from the target node alone.We show that our approach has potential for transfer learning between diﬀerent taxonomy domains, which reduces the need to collect training data for new taxonomies.

Wir verwenden maschinelles Lernen, um Synonyme für große Einkaufstaxonomien zu generieren.


Relational reasoning, the ability to model interactions and relations between objects, is valuable for robust multi-object tracking and pivotal for trajectory prediction.In this paper, we propose MOHART, a class-agnostic, end-to-end multi-object tracking and trajectory prediction algorithm, which explicitly accounts for permutation invariance in its relational reasoning.We explore a number of permutation invariant architectures and show that multi-headed self-attention outperforms the provided baselines and better accounts for complex physical interactions in a challenging toy experiment.We show on three real-world tracking datasets that adding relational reasoning capabilities in this way increases the tracking and trajectory prediction performance, particularly in the presence of ego-motion, occlusions, crowded scenes, and faulty sensor inputs.To the best of our knowledge, MOHART is the first fully end-to-end multi-object tracking from vision approach applied to real-world data reported in the literature.

MOHART verwendet einen Mechanismus der Selbstaufmerksamkeit, um relationale Schlussfolgerungen bei der Verfolgung von mehreren Objekten zu ziehen.


We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges:(a) efficient actor-critic learning with experience replay(b) stability of very off-policy learning.We employ those insights to accelerate hyper-parameter sweeps in which all participating agents run concurrently and share their experience via a common replay module.To this end we analyze the bias-variance tradeoffs in V-trace, a form of importance sampling for actor-critic methods.Based on our analysis, we then argue for mixing experience sampled from replay with on-policy experience, and propose a new trust region scheme that scales effectively to data distributions where V-trace becomes unstable.We provide extensive empirical validation of the proposed solution.We further show the benefits of this setup by demonstrating state-of-the-art data efficiency on Atari among agents trained up until 200M environment frames.

Wir untersuchen und schlagen Lösungen für zwei Herausforderungen beim Reinforcement Learning vor: (a) effizientes akteurskritisches Lernen mit Erfahrungswiederholung (b) Stabilität des Lernens von sehr unkonventionellen Strategien.


Stochastic gradient descent (SGD) is the workhorse of modern machine learning.Sometimes, there are many different potential gradient estimators that can be used.When so, choosing the one with the best tradeoff between cost and variance is important.This paper analyzes the convergence rates of SGD as a function of time, rather than iterations.This results in a simple rule to select the estimator that leads to the best optimization convergence guarantee.This choice is the same for different variants of SGD, and with different assumptions about the objective (e.g. convexity or smoothness).Inspired by this principle, we propose a technique to automatically select an estimator when a finite pool of estimators is given.Then, we extend to infinite pools of estimators, where each one is indexed by control variate weights.This is enabled by a reduction to a mixed-integer quadratic program.Empirically, automatically choosing an estimator performs comparably to the best estimator chosen with hindsight.

Wir schlagen einen Algorithmus zur Auswahl eines Gradientenschätzers vor, der die Effizienz der Optimierung verbessern soll.


We study the problem of alleviating the instability issue in the GAN training procedure via new architecture design.The discrepancy between the minimax and maximin objective values could serve as a proxy for the difficulties that the alternating gradient descent encounters in the optimization of GANs.In this work, we give new results on the benefits of multi-generator architecture of GANs.We show that the minimax gap shrinks to \epsilon as the number of generators increases with rate O(1/\epsilon).This improves over the best-known result of O(1/\epsilon^2).At the core of our techniques is a novel application of Shapley-Folkman lemma to the generic minimax problem, where in the literature the technique was only known to work when the objective function is restricted to the Lagrangian function of a constraint optimization problem.Our proposed Stackelberg GAN performs well experimentally in both synthetic and real-world datasets, improving Frechet Inception Distance by 14.61% over the previous multi-generator GANs on the benchmark datasets.

Wir untersuchen das Problem der Instabilität im GAN-Trainingsverfahren durch ein neues Architekturdesign mit theoretischen Garantien.


Nesterov SGD is widely used for training modern neural networks and other machine learning models.Yet, its advantages over SGD have not been theoretically clarified.Indeed, as we show  in this paper, both theoretically and empirically, Nesterov SGD with any parameter selection does not in general provide acceleration over ordinary SGD.Furthermore, Nesterov SGD may diverge for step sizes that ensure convergence of ordinary SGD.This is in contrast to the classical results in the deterministic setting, where the same step size ensures accelerated convergence of the Nesterov's method over optimal gradient descent.To address the non-acceleration issue, we  introduce a compensation term to Nesterov SGD.The resulting  algorithm, which we call MaSS, converges  for same step sizes as SGD.We prove that MaSS obtains an accelerated convergence rates over SGD for any mini-batch size in the linear setting.  For full batch, the convergence rate of MaSS matches the well-known accelerated rate of the Nesterov's method. We also analyze the  practically important question of the dependence of the convergence rate and  optimal hyper-parameters on the mini-batch size, demonstrating three distinct regimes: linear scaling, diminishing returns and saturation.Experimental evaluation of MaSS for several standard  architectures of deep networks, including ResNet and convolutional networks, shows improved performance over SGD, Nesterov SGD  and Adam.

Diese Arbeit beweist die Nicht-Beschleunigung von Nesterov SGD mit beliebigen Hyper-Parametern und schlägt einen neuen Algorithmus vor, der SGD in der überparametrisierten Umgebung nachweislich beschleunigt.


We propose the Fixed Grouping Layer (FGL); a novel feedforward layer designed to incorporate the inductive bias of structured smoothness into a deep learning model.FGL achieves this goal by connecting nodes across layers based on spatial similarity.The use of structured smoothness, as implemented by FGL, is motivated by applications to structured spatial data, which is, in turn, motivated by domain knowledge.The proposed model architecture outperforms conventional neural network architectures across a variety of simulated and real datasets with structured smoothness.

Eine Feedforward-Schicht zur Einbeziehung strukturierter Glätte in ein Deep-Learning-Modell