We propose Episodic Backward Update - a new algorithm to boost the performance of a deep reinforcement learning agent by fast reward propagation. In contrast to the conventional use of the replay memory with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state into its previous states. Our computationally efficient recursive algorithm allows sparse and delayed rewards to propagate effectively throughout the sampled episode. We evaluate our algorithm on 2D MNIST Maze Environment and 49 games of the Atari 2600 Environment and show that our agent improves sample efficiency with a competitive computational cost.

Wir schlagen Episodic Backward Update vor, einen neuartigen Deep Reinforcement Learning-Algorithmus, der Übergänge episodenweise abtastet und Werte rekursiv rückwärts aktualisiert, um schnelles und stabiles Lernen zu erreichen.

Schlägt ein neues DQN vor, bei dem die Ziele für eine vollständige Episode durch eine Rückwärtsaktualisierung (von Ende zu Anfang) berechnet werden, um eine schnellere Ausbreitung der Belohnungen bis zum Ende der Episode zu erreichen.

Die Autoren schlagen vor, den DQN-Algorithmus zu modifizieren, indem sie den Max-Bellman-Operator rekursiv auf eine Trajektorie mit einem gewissen Zerfall anwenden, um die Akkumulation von Fehlern mit dem verschachtelten Max zu verhindern.

In Deep-Q-Netzen werden die Q-Werte ab dem Ende der Episode aktualisiert, um eine schnelle Ausbreitung der Belohnungen entlang der Episode zu ermöglichen.


Survival Analysis (time-to-event analysis) in the presence of multiple possible adverse events, i.e., competing risks, is a challenging, yet very important problem in medicine, finance, manufacturing, etc. Extending classical survival analysis to competing risks is not trivial since only one event (e.g. one cause of death) is observed and hence, the incidence of an event of interest is often obscured by other related competing events. This leads to the nonidentifiability of the event times’ distribution parameters, which makes the problem significantly more challenging. In this work we introduce Siamese Survival Prognosis Network, a novel Siamese Deep Neural Network architecture that is able to effectively learn from data in the presence of multiple adverse events. The Siamese Survival Network is especially crafted to issue pairwise concordant time-dependent risks, in which longer event times are assigned lower risks. Furthermore, our architecture is able to directly optimize an approximation to the C-discrimination index, rather than relying on well-known metrics of cross-entropy etc., and which are not able to capture the unique requirements of survival analysis with competing risks. Our results show consistent performance improvements on a number of publicly available medical datasets over both statistical and deep learning state-of-the-art methods.

In dieser Arbeit stellen wir eine neuartige Siamese Deep Neural Network Architektur vor, die in der Lage ist, effektiv aus Daten zu lernen, wenn mehrere unerwünschte Ereignisse vorliegen.

In dieser Arbeit werden Siamesische Neuronale Netze in den Rahmen der konkurrierenden Risiken eingeführt, indem direkt für den c-Index optimiert wird.

Die Autoren befassen sich mit Fragen der Risikoabschätzung in einem Umfeld der Überlebensanalyse mit konkurrierenden Risiken und schlagen vor, den zeitabhängigen Diskriminierungsindex unter Verwendung eines siamesischen Überlebensnetzwerks direkt zu optimieren


The digitization of data has resulted in making datasets available to millions of users in the form of relational databases and spreadsheet tables. However, a majority of these users come from diverse backgrounds and lack the programming expertise to query and analyze such tables. We present a system that allows for querying data tables using natural language questions, where the system translates the question into an executable SQL query. We use a deep sequence to sequence model in wich the decoder uses a simple type system of SQL expressions to structure the output prediction. Based on the type, the decoder either copies an output token from the input question using an attention-based copying mechanism or generates it from a fixed vocabulary. We also introduce a value-based loss function that transforms a distribution over locations to copy from into a distribution over the set of input tokens to improve training of our model. We evaluate our model on the recently released WikiSQL dataset and show that our model trained using only supervised learning significantly outperforms the current state-of-the-art Seq2SQL model that uses reinforcement learning.

Wir stellen ein typbasiertes Zeigernetzmodell zusammen mit einer wertbasierten Verlustmethode vor, um ein neuronales Modell zur Übersetzung natürlicher Sprache in SQL effektiv zu trainieren.

Die Arbeit behauptet, eine neuartige Methode zu entwickeln, um Abfragen in natürlicher Sprache auf SQL abzubilden, indem eine Grammatik zur Anleitung der Dekodierung und eine neue Verlustfunktion für Zeiger/Kopiermechanismen verwendet wird.


To backpropagate the gradients through stochastic binary layers, we propose the augment-REINFORCE-merge (ARM) estimator that is unbiased, exhibits low variance, and has low computational complexity. Exploiting variable augmentation, REINFORCE, and reparameterization, the ARM estimator achieves adaptive variance reduction for Monte Carlo integration by merging two expectations via common random numbers. The variance-reduction mechanism of the ARM estimator can also be attributed to either antithetic sampling in an augmented space, or the use of an optimal anti-symmetric "self-control" baseline function together with the REINFORCE estimator in that augmented space. Experimental results show the ARM estimator provides state-of-the-art performance in auto-encoding variational inference and maximum likelihood estimation, for discrete latent variable models with one or multiple stochastic binary layers. Python code for reproducible research is publicly available.

Ein unverzerrter und variantenarmer Gradientenschätzer für diskrete latente Variablenmodelle.

Vorschlag einer neuen Technik zur Varianzreduzierung, die bei der Berechnung eines erwarteten Verlustgradienten verwendet werden kann, wenn die Erwartung in Bezug auf unabhängige binäre Zufallsvariablen besteht.

Ein Algorithmus, der Rao-Blackwellization und gewöhnliche Zufallszahlen kombiniert, um die Varianz des Score-Funktions-Gradientenschätzers im speziellen Fall stochastischer binärer Netzwerke zu verringern.

Ein unvoreingenommener und varianzarmer Augment-REINFORCE-Merge (ARM)-Schätzer für die Berechnung und das Backpropagating von Gradienten in binären neuronalen Netzen.


Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training. The scheme can reach a linear speed-up with respect to the number of workers, but this is rarely seen in practice as the scheme often suffers from large network delays and bandwidth limits. To overcome this communication bottleneck recent works propose to reduce the communication frequency. An algorithm of this type is local SGD that runs SGD independently in parallel on different workers and averages the sequences only once in a while. This scheme shows promising results in practice, but eluded thorough theoretical analysis.

 We prove concise convergence rates for local SGD on convex problems and show that it converges at the same rate as mini-batch SGD in terms of number of evaluated gradients, that is, the scheme achieves linear speed-up in the number of workers and mini-batch size. The number of  communication rounds can be reduced up to a factor of T^{1/2}---where T denotes the number of total steps---compared to mini-batch SGD. This also holds for asynchronous implementations.

 Local SGD can also be used for large scale training of deep learning models. The results shown here aim serving as a guideline to further explore the theoretical and practical aspects of local SGD in these applications.

Wir beweisen, dass paralleles lokales SGD eine lineare Beschleunigung mit viel weniger Kommunikation als paralleles Mini-Batch SGD erreicht.

Liefert einen Konvergenzbeweis für lokale SGD und beweist, dass lokale SGD die gleichen Geschwindigkeitsgewinne wie Minibatch bieten kann, aber möglicherweise in der Lage ist, deutlich weniger zu kommunizieren.

Diese Arbeit stellt eine Analyse der lokalen SGD und Grenzen auf, wie häufig die Schätzer, die durch die Ausführung von SGD erhalten werden, gemittelt werden müssen, um lineare Parallelisierung Geschwindigkeitssteigerungen zu erhalten.

Die Autoren analysieren den lokalen SGD-Algorithmus, bei dem K parallele SGD-Ketten ausgeführt werden und die Iterate gelegentlich über Maschinen hinweg durch Mittelwertbildung synchronisiert werden.


Extracting relevant information, causally inferring and predicting the future states with high accuracy is a crucial task for modeling complex systems. The endeavor to address these tasks is made even more challenging when we have to deal with high-dimensional heterogeneous data streams. Such data streams often have higher-order inter-dependencies across spatial and temporal dimensions. We propose to perform a soft-clustering of the data and learn its dynamics to produce a compact dynamical model while still ensuring the original objectives of causal inference and accurate predictions. To efficiently and rigorously process the dynamics of soft-clustering, we advocate for an information theory inspired approach that incorporates stochastic calculus and seeks to determine a trade-off between the predictive accuracy and compactness of the mathematical representation. We cast the model construction as a maximization of the compression of the state variables such that the predictive ability and causal interdependence (relatedness) constraints between the original data streams and the compact model are closely bounded. We provide theoretical guarantees concerning the convergence of the proposed learning algorithm. To further test the proposed framework, we consider a high-dimensional Gaussian case study and describe an iterative scheme for updating the new model parameters. Using numerical experiments, we demonstrate the benefits on compression and prediction accuracy for a class of dynamical systems. Finally, we apply the proposed algorithm to the real-world dataset of multimodal sentiment intensity and show improvements in prediction with reduced dimensions.

Kompakte Wahrnehmung von dynamischen Prozessen.

Untersucht das Problem der kompakten Darstellung des Modells eines komplexen dynamischen Systems bei gleichzeitiger Erhaltung der Informationen durch die Verwendung einer Informationsengpassmethode.

In diesem Beitrag wurde die Gaußsche lineare Dynamik untersucht und ein Algorithmus zur Berechnung der Informationsengpasshierarchie (IBH) vorgeschlagen.


We propose the dense RNN, which has the fully connections from each hidden state to multiple preceding hidden states of all layers directly. As the density of the connection increases, the number of paths through which the gradient flows can be increased. It increases the magnitude of gradients, which help to prevent the vanishing gradient problem in time. Larger gradients, however, can also cause exploding gradient problem. To complement the trade-off between two problems, we propose an attention gate, which controls the amounts of gradient flows. We describe the relation between the attention gate and the gradient flows by approximation. The experiment on the language modeling using Penn Treebank corpus shows dense connections with the attention gate improve the model’s performance.

Dichtes RNN, das von jedem versteckten Zustand direkt zu mehreren vorhergehenden versteckten Zuständen aller Schichten vollständige Verbindungen hat.

Schlägt eine neue RNN-Architektur vor, die langfristige Abhängigkeiten besser modelliert, eine multiskalige Darstellung von sequentiellen Daten erlernen kann und das Gradientenproblem durch die Verwendung parametrisierter Gating-Einheiten umgeht.

In diesem Beitrag wird eine vollständig vernetzte dichte RNN-Architektur mit Gated-Verbindungen zu jeder Schicht und Verbindungen zu den vorangehenden Schichten vorgeschlagen und die Ergebnisse auf PTB-Charakterebene modelliert.


We propose a new algorithm for training generative adversarial networks to jointly learn latent codes for both identities (e.g. individual humans) and observations (e.g. specific photographs). In practice, this means that by fixing the identity portion of latent codes, we can generate diverse images of the same subject, and by fixing the observation portion we can traverse the manifold of subjects while maintaining contingent aspects such as lighting and pose. Our algorithm features a pairwise training scheme in which each sample from the generator consists of two images with a common identity code. Corresponding samples from the real dataset consist of two distinct photographs of the same subject. In order to fool the discriminator, the generator must produce images that are both photorealistic, distinct, and appear to depict the same person. We augment both the DCGAN and BEGAN approaches with Siamese discriminators to accommodate pairwise training. Experiments with human judges and an off-the-shelf face verification system demonstrate our algorithm’s ability to generate convincing, identity-matched photographs.

SD-GANs entwirren latente Codes anhand bekannter Gemeinsamkeiten in einem Datensatz (z. B. Fotos, die dieselbe Person abbilden).

In diesem Beitrag wird das Problem der kontrollierten Bilderzeugung untersucht und ein Algorithmus vorgeschlagen, der ein Bildpaar mit derselben Identität erzeugt.

Diese Arbeit schlägt SD-GAN vor, eine Methode zum Training von GANs, um die Identitäts- und Nicht-Identitätsinformationen im latenten Eingangsvektor Z zu entflechten.


The goal of unpaired cross-domain translation is to learn useful mappings between two domains, given unpaired sets of datapoints from these domains. While this formulation is highly underconstrained, recent work has shown that it is possible to learn mappings useful for downstream tasks by encouraging approximate cycle consistency in the mappings between the two domains [Zhu et al., 2017]. In this work, we propose AlignFlow, a framework for unpaired cross-domain translation that ensures exact cycle consistency in the learned mappings. Our framework uses a normalizing flow model to specify a single invertible mapping between the two domains. In contrast to prior works in cycle-consistent translations, we can learn AlignFlow via adversarial training, maximum likelihood estimation, or a hybrid of the two methods. Theoretically, we derive consistency results for AlignFlow which guarantee recovery of desirable mappings under suitable assumptions. Empirically, AlignFlow demonstrates significant improvements over relevant baselines on image-to-image translation and unsupervised domain adaptation tasks on benchmark datasets.

Wir schlagen einen Lernrahmen für bereichsübergreifende Übersetzungen vor, der exakt zykluskonsistent ist und über adversariales Training, Maximum-Likelihood-Schätzung oder eine Mischform gelernt werden kann.

schlägt AlignFlow vor, eine effiziente Methode zur Umsetzung des Prinzips der Zykluskonsistenz unter Verwendung invertierbarer Flüsse.

Flussmodelle für ungepaarte Bild-zu-Bild-Übersetzung.


Program synthesis is a class of regression problems where one seeks a solution, in the form of a source-code program, that maps the inputs to their corresponding outputs exactly. Due to its precise and combinatorial nature, it is commonly formulated as a constraint satisfaction problem, where input-output examples are expressed constraints, and solved with a constraint solver. A key challenge of this formulation is that of scalability: While constraint solvers work well with few well-chosen examples, constraining the entire set of example constitutes a significant overhead in both time and memory. In this paper we address this challenge by constructing a representative subset of examples that is both small and is able to constrain the solver sufficiently. We build the subset one example at a time, using a trained discriminator to predict the probability of unchosen input-output examples conditioned on the chosen input-output examples, adding the least probable example to the subset. Experiment on a diagram drawing domain shows our approach produces subset of examples that are small and representative for the constraint solver.

In einem Programmsynthesekontext, in dem die Eingabe eine Menge von Beispielen ist, reduzieren wir die Kosten, indem wir eine Teilmenge von repräsentativen Beispielen berechnen.

Vorschlagen einer Methode zur Identifizierung repräsentativer Beispiele für die Programmsynthese, um die Skalierbarkeit bestehender Constraint-Programming Lösungen zu erhöhen.

Eine Methode zur Auswahl einer Teilmenge von Beispielen, auf die ein Constraint Solver angewendet wird, um Probleme der Programmsynthese zu lösen.

In dieser Arbeit wird eine Methode zur Beschleunigung von Mehrzweck-Programmsynthesizern vorgeschlagen.


Humans possess an ability to abstractly reason about objects and their interactions, an ability not shared with state-of-the-art deep learning models. Relational networks, introduced by Santoro et al. (2017), add the capacity for relational reasoning to deep neural networks, but are limited in the complexity of the reasoning tasks they can address. We introduce recurrent relational networks which increase the suite of solvable tasks to those that require an order of magnitude more steps of relational reasoning. We use recurrent relational networks to solve Sudoku puzzles and achieve state-of-the-art results by solving 96.6% of the hardest Sudoku puzzles, where relational networks fail to solve any. We also apply our model to the BaBi textual QA dataset solving 19/20 tasks which is competitive with state-of-the-art sparse differentiable neural computers. The recurrent relational network is a general purpose module that can augment any neural network model with the capacity to do many-step relational reasoning.

Wir stellen Recurrent Relational Networks vor, ein leistungsfähiges und allgemeines neuronales Netzmodul für relationales Denken, und verwenden es, um 96,6% der schwierigsten Sudokus und 19/20 BaBi-Aufgaben zu lösen.

Einführung rekurrenter relationaler Netze (RRNs), die zu beliebigen neuronalen Netzen hinzugefügt werden können, um die Fähigkeit zur relationalen Schlussfolgerung zu erhöhen.

Einführung eines tiefen neuronalen Netzes für strukturierte Vorhersagen, das bei Soduku-Rätseln und der BaBi-Aufgabe die beste Leistung erzielt.

In diesem Beitrag wird eine Methode beschrieben, die als relationales Netzwerk bezeichnet wird, um tiefe neuronale Netze mit relationalen Schlussfolgerungen zu versehen.


Empirical risk minimization (ERM), with proper loss function and regularization, is the common practice of supervised classification. In this paper, we study training arbitrary (from linear to deep) binary classifier from only unlabeled (U) data by ERM. We prove that it is impossible to estimate the risk of an arbitrary binary classifier in an unbiased manner given a single set of U data, but it becomes possible given two sets of U data with different class priors. These two facts answer a fundamental question---what the minimal supervision is for training any binary classifier from only U data. Following these findings, we propose an ERM-based learning method from two sets of U data, and then prove it is consistent. Experiments demonstrate the proposed method could train deep models and outperform state-of-the-art methods for learning from two sets of U data.

Drei Klassenprioritäten sind alles, was Sie brauchen, um tiefe Modelle aus nur U-Daten zu trainieren, während zwei nicht ausreichen sollten.

Schlägt einen unvoreingenommenen Schätzer vor, der das Training von Modellen mit schwacher Überwachung auf zwei unbeschrifteten Datensätzen mit bekannten Klassenprioritäten ermöglicht, und erörtert die theoretischen Eigenschaften der Schätzer.

Eine Methodik für das Training eines beliebigen binären Klassifikators aus nur unbeschrifteten Daten und eine empirische Risikominimierungsmethode für zwei Sätze unbeschrifteter Daten, bei denen Klassenprioritäten gegeben sind.


Deep Neural Networks (DNNs) excel on many complex perceptual tasks but it has proven notoriously difficult to understand how they reach their decisions. We here introduce a high-performance DNN architecture on ImageNet whose decisions are considerably easier to explain. Our model, a simple variant of the ResNet-50 architecture called BagNet, classifies an image based on the occurrences of small local image features without taking into account their spatial ordering. This strategy is closely related to the bag-of-feature (BoF) models popular before the onset of deep learning and reaches a surprisingly high accuracy on ImageNet (87.6% top-5 for 32 x 32 px features and Alexnet performance for 16 x16 px features). The constraint on local features makes it straight-forward to analyse how exactly each part of the image influences the classification. Furthermore, the BagNets behave similar to state-of-the art deep neural networks such as VGG-16, ResNet-152 or DenseNet-169 in terms of feature sensitivity, error distribution and interactions between image parts. This suggests that the improvements of DNNs over previous bag-of-feature classifiers in the last few years is mostly achieved by better fine-tuning rather than by qualitatively different decision strategies.

Die Aggregation von Klassenevidenz aus vielen kleinen Bildfeldern reicht aus, um ImageNet zu lösen, führt zu besser interpretierbaren Modellen und kann Aspekte der Entscheidungsfindung von populären DNNs erklären.

In diesem Beitrag wird eine neuartige und kompakte neuronale Netzarchitektur vorgeschlagen, die die Informationen in Bag-of-Words-Merkmalen nutzt. Der vorgeschlagene Algorithmus verwendet nur die Patch-Informationen unabhängig und führt eine Mehrheitsabstimmung mit unabhängig klassifizierten Patches durch.


Somatic cancer mutation detection at ultra-low variant allele frequencies (VAFs) is an unmet challenge that is intractable with current state-of-the-art mutation calling methods. Specifically, the limit of VAF detection is closely related to the depth of coverage, due to the requirement of multiple supporting reads in extant methods, precluding the detection of mutations at VAFs that are orders of magnitude lower than the depth of coverage. Nevertheless, the ability to detect cancer-associated mutations in ultra low VAFs is a fundamental requirement for low-tumor burden cancer diagnostics applications such as early detection, monitoring, and therapy nomination using liquid biopsy methods (cell-free DNA). Here we defined a spatial representation of sequencing information adapted for convolutional architecture that enables variant detection at VAFs, in a manner independent of the depth of sequencing. This method enables the detection of cancer mutations even in VAFs as low as 10x-4^, >2 orders of magnitude below the current state-of-the-art. We validated our method on both simulated plasma and on clinical cfDNA plasma samples from cancer patients and non-cancer controls. This method introduces a new domain within bioinformatics and personalized medicine – somatic whole genome mutation calling for liquid biopsy.

Aktuelle somatische Mutationsmethoden funktionieren nicht mit Flüssigbiopsien (d.h. Sequenzierung mit geringer Abdeckung). Wir wenden eine CNN-Architektur auf eine eindeutige Darstellung einer Lesung und deren Bewertung an und zeigen eine signifikante Verbesserung gegenüber früheren Methoden in der Niedrigfrequenz-Einstellung.

Vorschlagen einer CNN-basierten Lösung namens Kittyhawk für somatische Mutationserkennung bei extrem niedrigen Allelefrequenzen.

Ein neuer Algorithmus zur Erkennung von Krebsmutationen aus der Sequenzierung zellfreier DNA, der den Sequenzkontext identifiziert, der Sequenzierungsfehler von echten Mutationen unterscheidet.

In dieser Arbeit wird ein Deep Learning Framework zur Vorhersage von somatischen Mutationen bei extrem niedrigen Frequenzen vorgeschlagen, die bei der Erkennung von Tumoren aus zellfreier DNA auftreten.


This paper presents the formal release of {\em MedMentions}, a new manually annotated resource for the recognition of biomedical concepts. What distinguishes MedMentions from other annotated biomedical corpora is its size (over 4,000 abstracts and over 350,000 linked mentions), as well as the size of the concept ontology (over 3 million concepts from UMLS 2017) and its broad coverage of biomedical disciplines. In addition to the full corpus, a sub-corpus of MedMentions is also presented, comprising annotations for a subset of UMLS 2017 targeted towards document retrieval. To encourage research in Biomedical Named Entity Recognition and Linking, data splits for training and testing are included in the release, and a baseline model and its metrics for entity linking are also described.

Die Arbeit stellt ein neues Gold-Standard-Korpus der biomedizinischen wissenschaftlichen Literatur vor, das manuell mit UMLS Konzept Erwähnungen annotiert wurde.

Einzelheiten zum Aufbau eines manuell annotierten Datensatzes für biomedizinische Konzepte, der größer ist und von einer umfangreicheren Ontologie abgedeckt wird als bisherige Datensätze.

Dieses Papier verwendet MedMentions, ein TaggerOne Semi-Markov-Modell für die Ende-zu-Ende Konzepterkennung und -Verknüpfung auf einem Satz von Pubmed-Zusammenfassungen, um Arbeiten mit biomedizinischen Konzepten/Entitäten zu kennzeichnen


In this paper we propose a Deep Autoencoder Mixture Clustering (DAMIC) algorithm. It is based on a mixture of deep autoencoders where each cluster is represented by an autoencoder. A clustering network transforms the data into another space and then selects one of the clusters. Next, the autoencoder associated with this cluster is used to reconstruct the data-point. The clustering algorithm jointly learns the nonlinear data representation and the set of autoencoders. The optimal clustering is found by minimizing the reconstruction loss of the mixture of autoencoder network. Unlike other deep clustering algorithms, no regularization term is needed to avoid data collapsing to a single point. Our experimental evaluations on image and text corpora show significant improvement over state-of-the-art methods.

Wir schlagen eine tiefe Clustermethode vor, bei der jedes Cluster anstelle eines Centroids durch einen Autoencoder repräsentiert wird.

Stellt ein tiefes Clustering vor, das auf einer Mischung von Autoencodern basiert, wobei die Datenpunkte einem Cluster auf der Grundlage des Darstellungsfehlers zugeordnet werden, wenn das Autoencodernetzwerk zu ihrer Darstellung verwendet wurde.

Ein Deep-Clustering-Ansatz, der ein Autoencoder-Framework verwendet, um eine niedrigdimensionale Einbettung der Daten zu erlernen und gleichzeitig Daten mithilfe eines tiefen neuronalen Netzwerks zu clustern.

Eine tiefe Clustering-Methode, bei der jedes Cluster mit verschiedenen Autoencodern dargestellt wird, funktioniert durchgängig und kann auch zum Clustern neu eingehender Daten verwendet werden, ohne dass das gesamte Clustering-Verfahren erneut durchgeführt werden muss.


We propose a new Integral Probability Metric (IPM) between distributions: the Sobolev IPM. The Sobolev IPM compares the mean discrepancy of two distributions for functions (critic) restricted to a Sobolev ball defined with respect to a dominant measure mu. We show that the Sobolev IPM compares two distributions in high dimensions based on weighted conditional Cumulative Distribution Functions (CDF) of each coordinate on a leave one out basis. The Dominant measure mu plays a crucial role as it defines the support on which conditional CDFs are compared. Sobolev IPM can be seen as an extension of the one dimensional Von-Mises Cramer statistics to high dimensional distributions. We show how Sobolev IPM can be used to train Generative Adversarial Networks (GANs). We then exploit the intrinsic conditioning implied by Sobolev IPM in text generation. Finally we show that a variant of Sobolev GAN achieves competitive results in semi-supervised learning on CIFAR-10, thanks to the smoothness enforced on the critic by Sobolev GAN which relates to Laplacian regularization.

Wir definieren eine neue Integrale Wahrscheinlichkeitsmetrik (Sobolev IPM) und zeigen, wie sie für das Training von GANs zur Texterzeugung und zum halbüberwachten Lernen verwendet werden kann.

Schlägt ein neuartiges Regularisierungsschema für GANs vor, das auf einer Sobolev-Norm basiert und Abweichungen zwischen L2-Normen von Ableitungen misst.

Die Autoren stellen einen anderen GAN-Typ vor, der den typischen Aufbau eines GANs verwendet, aber eine andere Funktionsklasse hat, und geben ein Rezept für das Training von GANs mit dieser Art von Funktionsklasse.

In dem Beitrag wird eine andere Gradientenstrafe für GAN-Kritiker vorgeschlagen, die die erwartete quadratische Norm des Gradienten auf 1 setzt.


We propose in this paper a new approach to train the Generative Adversarial Nets (GANs) with a mixture of generators to overcome the mode collapsing problem. The main intuition is to employ multiple generators, instead of using a single one as in the original GAN. The idea is simple, yet proven to be extremely effective at covering diverse data modes, easily overcoming the mode collapsing problem and delivering state-of-the-art results. A minimax formulation was able to establish among a classifier, a discriminator, and a set of generators in a similar spirit with GAN. Generators create samples that are intended to come from the same distribution as the training data, whilst the discriminator determines whether samples are true data or generated by generators, and the classifier specifies which generator a sample comes from. The distinguishing feature is that internal samples are created from multiple generators, and then one of them will be randomly selected as final output similar to the mechanism of a probabilistic mixture model. We term our method Mixture Generative Adversarial Nets (MGAN). We develop theoretical analysis to prove that, at the equilibrium, the Jensen-Shannon divergence (JSD) between the mixture of generators’ distributions and the empirical data distribution is minimal, whilst the JSD among generators’ distributions is maximal, hence effectively avoiding the mode collapsing problem. By utilizing parameter sharing, our proposed model adds minimal computational cost to the standard GAN, and thus can also efficiently scale to large-scale datasets. We conduct extensive experiments on synthetic 2D data and natural image databases (CIFAR-10, STL-10 and ImageNet) to demonstrate the superior performance of our MGAN in achieving state-of-the-art Inception scores over latest baselines, generating diverse and appealing recognizable objects at different resolutions, and specializing in capturing different types of objects by the generators.

Wir schlagen einen neuen Ansatz vor, um GANs mit einer Mischung von Generatoren zu trainieren, um das Problem des Zusammenbrechens der Modi zu überwinden.

Bewältigung des Problems des Modus-Kollapses in GANs unter Verwendung einer eingeschränkten Mischungsverteilung für den Generator und eines Hilfsklassifikators, der die Quellmischungskomponente vorhersagt.

In der Arbeit wird eine Mischung von Generatoren vorgeschlagen, um GANs ohne zusätzliche Rechenkosten zu trainieren

Die Autoren zeigen, dass die Verwendung von MGAN, das darauf abzielt, das Problem des Modellzusammenbruchs durch Mischungsgeneratoren zu überwinden, hochmoderne Ergebnisse erzielt.


Allowing humans to interactively train artificial agents to understand language instructions is desirable for both practical and scientific reasons.   Though, given  the lack of sample efficiency in current learning methods, reaching this goal may require substantial research efforts. We introduce the BabyAI research platform, with the goal of supporting investigations towards including humans in the loop for grounded language learning. The BabyAI platform comprises an extensible suite of 19 levels of increasing difficulty. Each level gradually leads the agent towards acquiring a combinatorially rich synthetic language, which is a proper subset of English. The platform also provides a hand-crafted bot agent, which simulates a human teacher.   We report estimated amount of supervision required for training neural reinforcement and behavioral-cloning agents on some BabyAI levels. We put forward strong evidence that current deep learning methods are not yet sufficiently sample-efficient in the context of learning a language with compositional properties.

Wir präsentieren die BabyAI-Plattform zur Untersuchung der Dateneffizienz beim Sprachenlernen mit einem Menschen in der Schleife.

Stellt eine Forschungsplattform mit einem Bot in der Schleife vor, der lernt, Sprachbefehle auszuführen, wobei die Sprache kompositorische Strukturen aufweist.

Stellt eine Plattform für das Erlernen von Sprachen vor, die jeden Menschen in der Lernschleife durch einen heuristischen Lehrer ersetzt und eine synthetische Sprache verwendet, die einer 2D-Gitterwelt zugeordnet ist.


Recently, there has been growing interest in methods that perform neural network compression, namely techniques that attempt to substantially reduce the size of a neural network without significant reduction in performance. However, most existing methods are post-processing approaches in that they take a learned neural network as input and output a compressed network by either forcing several parameters to take the same value (parameter tying via quantization) or pruning irrelevant edges (pruning) or both. In this paper, we propose a novel algorithm that jointly learns and compresses a neural network. The key idea in our approach is to change the optimization criteria by adding $k$ independent Gaussian priors over the parameters and a sparsity penalty. We show that our approach is easy to implement using existing neural network libraries, generalizes L1 and L2 regularization and elegantly enforces parameter tying as well as pruning constraints. Experimentally, we demonstrate that our new algorithm yields state-of-the-art compression on several standard benchmarks with minimal loss in accuracy while requiring little to no hyperparameter tuning as compared with related, competing approaches.

Ein k-means-Prior in Kombination mit einer L1-Regularisierung führt zu hochmodernen Kompressionsergebnissen.

Dieses Papier untersucht die weiche Parameterbindung und Kompression von DNNs/CNNs.


The application of stochastic variance reduction to optimization has shown remarkable recent theoretical and practical success. The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem. We show that naive application of the SVRG technique and related approaches fail, and explore why.

Die SVRG-Methode versagt bei modernen Deep-Learning-Problemen.

In diesem Beitrag wird eine Analyse von SVRG-Methoden vorgestellt, die zeigt, dass Dropout, Batch-Norm und Datenerweiterung (zufällige Ernte/Rotation/Translation) dazu neigen, die Verzerrung und/oder Varianz der Aktualisierungen zu erhöhen.

Diese Arbeit untersucht die Anwendbarkeit von SVGD auf moderne neuronale Netze und zeigt, dass die naive Anwendung von SVGD in der Regel scheitert.


The ground-breaking performance obtained by deep convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks. One of the main challenge in applying CNNs to 3D shape analysis is how to define a natural convolution operator on non-euclidean surfaces. In this paper, we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere. A cascade set of geodesic disk filters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks. We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance, and its effectiveness is evaluated in applications of non-rigid/ rigid shape classification and shape retrieval.

Eine Methode zur Anwendung von Deep Learning auf 3D-Oberflächen unter Verwendung ihrer sphärischen Deskriptoren und alt-az anisotroper Convolution auf der 2-Sphäre.

Stellt ein polares anisotropes Convolution Schema auf einer Einheitskugel vor, bei dem die Filtertranslation durch Filterrotation ersetzt wird.

Diese Arbeit untersucht tiefes Lernen von 3D-Formen mit alt-az anisotroper 2-Sphären-Faltung.


Recent breakthroughs in computer vision make use of large deep neural networks, utilizing the substantial speedup offered by GPUs. For applications running on limited hardware, however, high precision real-time processing can still be a challenge.   One approach to solving this problem is training networks with binary or ternary weights, thus removing the need to calculate multiplications and significantly reducing memory size. In this work, we introduce LR-nets (Local reparameterization networks), a new method for training neural networks with discrete weights using stochastic parameters. We show how a simple modification to the local reparameterization trick, previously used to train Gaussian distributed weights, enables the training of discrete weights. Using the proposed training we test both binary and ternary models on MNIST, CIFAR-10 and ImageNet benchmarks and reach state-of-the-art results on most experiments.

Training von binären/alternären Netzen durch lokale Umparametrisierung mit der CLT-Approximation.

Trainiert binäre und ternäre Gewichtsverteilungsnetze unter Verwendung von Backpropagation, um Neuronenvoraktivierungen mit einem Reparametrisierungstrick zu testen.

In diesem Beitrag wird vorgeschlagen, stochastische Parameter in Kombination mit dem Trick der lokalen Reparametrisierung zu verwenden, um neuronale Netze mit binären oder ternären Gewichten zu trainieren, was zu Ergebnissen auf dem neuesten Stand der Technik führt.


We present Optimal Completion Distillation (OCD), a training procedure for optimizing sequence to sequence models based on edit distance. OCD is efficient, has no hyper-parameters of its own, and does not require pre-training or joint optimization with conditional log-likelihood. Given a partial sequence generated by the model, we first identify the set of optimal suffixes that minimize the total edit distance, using an efficient dynamic programming algorithm.   Then, for each position of the generated sequence, we use a target distribution which puts equal probability on the first token of all the optimal suffixes. OCD achieves the state-of-the-art performance on end-to-end speech recognition, on both Wall Street Journal and Librispeech datasets, achieving $9.3\%$ WER and $4.5\%$ WER, respectively.

Optimal Completion Distillation (OCD) ist ein Trainingsverfahren zur Optimierung von Sequenz-zu-Sequenz-Modellen auf der Basis von Edit-Distanz, das bei Ende-zu-Ende Spracherkennungsaufgaben den Stand der Technik erreicht.

Alternativer Ansatz für das Training von seq2seq-Modellen unter Verwendung eines dynamischen Programms zur Berechnung optimaler Fortsetzungen von vorhergesagten Präfixen.

Ein Trainingsalgorithmus für autoregressive Modelle, der kein MLE-Vortraining benötigt und direkt aus dem Sampling optimieren kann.

Die Arbeit geht auf einen Mangel von Sequenz-zu-Sequenz-Modellen ein, die mit Hilfe von Maximum-Likelihood-Schätzungen trainiert werden, und schlägt einen Ansatz vor, der auf Edit-Distanzen und der impliziten Verwendung vorgegebener Label-Sequenzen während des Trainings basiert.


As an emerging field, federated learning has recently attracted considerable attention.
 Compared to distributed learning in the datacenter setting, federated learning
 has more strict constraints on computate efficiency of the learned model and communication
 cost during the training process. In this work, we propose an efficient
 federated learning framework based on variational dropout. Our approach is able
 to jointly learn a sparse model while reducing the amount of gradients exchanged
 during the iterative training process. We demonstrate the superior performance
 of our approach on achieving significant model compression and communication
 reduction ratios with no accuracy loss.

Eine gemeinsame Modell- und Gradientensparsamkeitsmethode für föderiertes Lernen.

Wendet Variational Dropout an, um die Kommunikationskosten beim verteilten Training neuronaler Netze zu reduzieren, und führt Experimente mit den Datensätzen mnist, cifar10 und svhn durch.

Die Autoren schlagen einen Algorithmus vor, der die Kommunikationskosten beim föderierten Lernen reduziert, indem spärliche Gradienten vom Gerät zum Server und zurück gesendet werden.

Kombiniert einen verteilten Optimierungsalgorithmus mit Variational Dropout, um die von den lokalen Lernern an den Master-Server gesendeten Gradienten zu strecken.


We prove a multiclass boosting theory for the ResNet architectures which simultaneously creates a new technique for multiclass boosting and provides a new algorithm for ResNet-style architectures.   Our proposed training algorithm, BoostResNet, is particularly suitable in non-differentiable architectures.   Our method only requires the relatively inexpensive sequential training of T "shallow ResNets". We prove that the training error decays exponentially with the depth T if the weak module classifiers that we train perform slightly better than some weak baseline.   In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition.   A generalization error bound based on margin theory is proved and suggests that ResNet could be resistant to overfitting using a network with l_1 norm bounded weights.

Wir beweisen eine Multiklassen-Boosting-Theorie für die ResNet-Architekturen, die gleichzeitig eine neue Technik für Multiklassen-Boosting schafft und einen neuen Algorithmus für ResNet-artige Architekturen bietet.

Präsentiert einen Boosting-Algorithmus für das Training von tiefen Residual Networks, eine Konvergenzanalyse für Trainingsfehler und eine Analyse der Generalisierungsfähigkeit.

Eine Lernmethode für ResNet unter Verwendung des Boosting-Frameworks, die das Lernen komplexer Netzwerke zerlegt und weniger Rechenaufwand erfordert.

Die Autoren schlagen das tiefe ResNet als Boosting-Algorithmus vor und behaupten, dieser sei effizienter als die standardmäßige Ende-zu-Ende Backpropagation.


We consider the problem of learning a one-hidden-layer neural network: we assume the input x is from Gaussian distribution and the label $y = a \sigma(Bx) + \xi$, where a is a nonnegative vector and  $B$ is a full-rank weight matrix, and $\xi$ is a noise vector. We first give an analytic formula for the population risk of the standard squared loss and demonstrate that it implicitly attempts to decompose a sequence of low-rank tensors simultaneously.

 Inspired by the formula, we design a non-convex objective function $G$ whose landscape is guaranteed to have the following properties:

1. All local minima of $G$ are also global minima.
 2. All global minima of $G$ correspond to the ground truth parameters.
 3. The value and gradient of $G$ can be estimated using samples.

 With these properties, stochastic gradient descent on $G$ provably converges to the global minimum and learn the ground-truth parameters. We also prove finite sample complexity results and validate the results by simulations.

Die Arbeit analysiert die Optimierungslandschaft von einschichtigen neuronalen Netzen und entwirft ein neues Ziel, das nachweislich kein ungewolltes lokales Minimum aufweist.

Dieses Papier untersucht das Problem des Lernens von neuronalen Netzen mit einer verborgenen Schicht, stellt eine Verbindung zwischen dem Least squares population loss und dem Hermite-Polynomen her und schlägt eine neue Verlustfunktion vor.

Eine Tensor-Faktorisierungs-Methode zur Verschlankung eines neuronalen Netzes mit einer versteckten Schicht.


Open information extraction (OIE) systems extract relations and their
  arguments from natural language text in an unsupervised manner. The resulting
  extractions are a valuable resource for downstream tasks such as knowledge
  base construction, open question answering, or event schema induction. In this
  paper, we release, describe, and analyze an OIE corpus called OPIEC, which was
  extracted from the text of English Wikipedia. OPIEC complements the available
  OIE resources: It is the largest OIE corpus publicly available to date (over
  340M triples) and contains valuable metadata such as provenance information,
  confidence scores, linguistic annotations, and semantic annotations including
  spatial and temporal information. We analyze the OPIEC corpus by comparing its
  content with knowledge bases such as DBpedia or YAGO, which are also based on
  Wikipedia. We found that most of the facts between entities present in OPIEC
  cannot be found in DBpedia and/or YAGO, that OIE facts
  often differ in the level of specificity compared to knowledge base facts, and
  that OIE open relations are generally highly polysemous. We believe that the
  OPIEC corpus is a valuable resource for future research on automated knowledge
  base construction.

Ein offener Korpus zur Informationsextraktion und seine eingehende Analyse.

Erstellt ein neues Korpus für die Informationsextraktion, das größer ist als die bisherigen öffentlichen Korpora und Informationen enthält, die in den bisherigen Korpora nicht vorhanden sind.

Präsentiert einen Datensatz von Open-IE-Triples, die mit Hilfe eines neuen Extraktionssystems aus Wikipedia gesammelt wurden.

Die Arbeit beschreibt die Erstellung eines Open IE-Korpus über die englische Wikipedia durch eine automatische Methode


The process of designing neural architectures requires expert knowledge and extensive trial and error.
 While automated architecture search may simplify these requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in both flexibility and components.
 We propose a domain-specific language (DSL) for use in automated architecture search which can produce novel RNNs of arbitrary depth and width.
 The DSL is flexible enough to define standard architectures such as the Gated Recurrent Unit and Long Short Term Memory and allows the introduction of non-standard RNN components such as trigonometric curves and layer normalization.   Using two different candidate generation techniques, random search with a ranking function and reinforcement learning,
we explore the novel architectures produced by the RNN DSL for language modeling and machine translation domains.
 The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed.

Wir definieren eine flexible DSL für die Generierung von RNN-Architekturen, die RNNs unterschiedlicher Größe und Komplexität zulässt, und schlagen eine Ranking-Funktion vor, die RNNs als rekursive neuronale Netze darstellt und ihre Leistung simuliert, um die vielversprechendsten Architekturen auszuwählen.

Es wird eine neue Methode zur Erzeugung von RNN-Architekturen vorgestellt, die eine domänenspezifische Sprache für zwei Arten von Generatoren (zufällig und RL-basiert) zusammen mit einer Ranking-Funktion und einem Evaluator verwendet.

In dieser Arbeit wird die Suche nach guten RNN Cell-Architekturen als Black-Box-Optimierungsproblem dargestellt, bei dem Beispiele als Operatorbaum dargestellt und auf der Grundlage gelernter Funktionen bewertet oder von einem RL-Agenten erzeugt werden.

In diesem Beitrag wird eine Meta-Lernstrategie für die automatische Architektursuche im Kontext von RNN untersucht, indem ein DSL verwendet wird, die rekurrente RNN-Operationen spezifiziert.


Researches on deep neural networks with discrete parameters and their deployment in embedded systems have been active and promising topics. Although previous works have successfully reduced precision in inference, transferring both training and inference processes to low-bitwidth integers has not been demonstrated simultaneously. In this work, we develop a new method termed as ``"WAGE" to discretize both training and inference, where weights (W), activations (A), gradients (G) and errors (E) among layers are shifted and linearly constrained to low-bitwidth integers. To perform pure discrete dataflow for fixed-point devices, we further replace batch normalization by a constant scaling layer and simplify other components that are arduous for integer implementation. Improved accuracies can be obtained on multiple datasets, which indicates that WAGE somehow acts as a type of regularization. Empirically, we demonstrate the potential to deploy training in hardware systems such as integer-based deep learning accelerators and neuromorphic chips with comparable accuracy and higher energy efficiency, which is crucial to future AI applications in variable scenarios with transfer and continual learning demands.

Wir trainieren und schlussfolgern nur mit ganzen Zahlen mit geringer Bitbreite in DNNs.

Eine Methode namens WAGE, die alle Operanden und Operatoren in einem neuronalen Netz quantisiert, um die Anzahl der Bits für die Darstellung in einem Netz zu reduzieren.

Die Autoren schlagen diskrete Gewichte, Aktivierungen, Gradienten und Fehler sowohl beim Training als auch beim Testen von neuronalen Netzen vor.


Modern Convolutional Neural Networks (CNNs) are complex, encompassing millions of parameters. Their deployment exerts computational, storage and energy demands, particularly on embedded platforms. Existing approaches to prune or sparsify CNNs require retraining to maintain inference accuracy. Such retraining is not feasible in some contexts. In this paper, we explore the sparsification of CNNs by proposing three model-independent methods. Our methods are applied on-the-fly and require no retraining. We show that the state-of-the-art models' weights can be reduced by up to 73% (compression factor of 3.7x) without incurring more than 5% loss in Top-5 accuracy. Additional fine-tuning gains only 8% in sparsity, which indicates that our fast on-the-fly methods are effective.

In diesem Beitrag entwickeln wir schnelle, umschulungsfreie Sparsifizierungsmethoden, die für die fliegende Sparsifizierung von CNNs in vielen industriellen Kontexten eingesetzt werden können.

In diesem Papier werden Ansätze für das Pruning von CNNs ohne erneutes Training vorgeschlagen, indem drei Schemata zur Bestimmung der Schwellenwerte für die Pruning-Gewichte eingeführt werden.

Diese Arbeit beschreibt eine Methode zur Sparsifizierung von CNNs ohne Neutraining.


Curriculum learning and Self paced learning are popular topics in the machine learning that suggest to put the training samples in order by considering their difficulty levels. Studies in these topics show that starting with a small training set and adding new samples according to difficulty levels improves the learning performance. In this paper we experimented that we can also obtain good results by adding the samples randomly without a meaningful order. We compared our method with classical training, Curriculum learning, Self paced learning and their reverse ordered versions. Results of the statistical tests show that the proposed method is better than classical method and similar with the others. These results point a new training regime that removes the process of difficulty level determination in Curriculum and Self paced learning and as successful as these methods.

Wir schlagen vor, dass das Training mit stufenweise wachsenden Mengen eine Optimierung für neuronale Netze darstellt.

Die Autoren vergleichen das Curriculum Learning mit dem Lernen in einer zufälligen Reihenfolge mit Phasen, die eine neue Stichprobe von Beispielen zu der zuvor zufällig zusammengestellten Menge hinzufügen.

In diesem Beitrag wird der Einfluss der Reihenfolge im Curriculum und beim selbstgesteuerten Lernen untersucht, und es wird gezeigt, dass die Reihenfolge der Trainingsinstanzen bis zu einem gewissen Grad nicht wichtig ist.


We study the problem of learning to map, in an unsupervised way, between domains $A$ and $B$, such that the samples $\vb \in B$ contain all the information that exists in samples $\va\in A$ and some additional information. For example, ignoring occlusions, $B$ can be people with glasses, $A$ people without, and the glasses, would be the added information. When mapping a sample $\va$ from the first domain to the other domain, the missing information is replicated from an independent reference sample $\vb\in B$. Thus, in the above example, we can create, for every person without glasses a version with the glasses observed in any face image.

 Our solution employs a single two-pathway encoder and a single decoder for both domains. The common part of the two domains and the separate part are encoded as two vectors, and the separate part is fixed at zero for domain $A$. The loss terms are minimal and involve reconstruction losses for the two domains and a domain confusion term. Our analysis shows that under mild assumptions, this architecture, which is much simpler than the literature guided-translation methods, is enough to ensure disentanglement between the two domains. We present convincing results in a few visual domains, such as no-glasses to glasses, adding facial hair based on a reference image, etc.

Ein Bild-zu-Bild-Übersetzungsverfahren, das einem Bild den Inhalt eines anderen Bildes hinzufügt und so ein neues Bild erzeugt.

Dieses Arbeit befasst sich mit der Aufgabe der Übertragung von Inhalten, wobei die Neuigkeit im Verlust liegt.


Mathematical reasoning---a core ability within human intelligence---presents some unique challenges as a domain: we do not come to understand and solve mathematical problems primarily on the back of experience and evidence, but on the basis of inferring, learning, and exploiting laws, axioms, and symbol manipulation rules. In this paper, we present a new challenge for the evaluation (and eventually the design) of neural architectures and similar system, developing a task suite of mathematics problems involving sequential questions and answers in a free-form textual input/output format. The structured nature of the mathematics domain, covering arithmetic, algebra, probability and calculus, enables the construction of training and test spits designed to clearly illuminate the capabilities and failure-modes of different architectures, as well as evaluate their ability to compose and relate knowledge and learned processes. Having described the data generation process and its potential future expansions, we conduct a comprehensive analysis of models from two broad classes of the most powerful sequence-to-sequence architectures and find notable differences in their ability to resolve mathematical problems and generalize their knowledge.

Ein Datensatz zum Testen des mathematischen Denkens (und der algebraischen Verallgemeinerung) sowie Ergebnisse zu aktuellen Sequenz-zu-Sequenz-Modellen.

Es wird ein neuer synthetischer Datensatz zur Bewertung der mathematischen Argumentationsfähigkeit von Sequenz-zu-Sequenz-Modellen vorgestellt und zur Bewertung verschiedener Modelle verwendet.

Modell zum Lösen grundlegender mathematischer Probleme.


Convolutional Neural Networks (CNNs) filter the input data using a series of spatial convolution operators with compactly supported stencils and point-wise nonlinearities.
 Commonly, the convolution operators couple features from all channels.
 For wide networks, this leads to immense computational cost in the training of and prediction with CNNs.
 In this paper, we present novel ways to parameterize the convolution more efficiently, aiming to decrease the number of parameters in CNNs and their computational complexity.
 We propose new architectures that use a sparser coupling between the channels and thereby reduce both the number of trainable weights and the computational cost of the CNN.
 Our architectures arise as new types of residual neural network (ResNet) that can be seen as discretizations of a Partial Differential Equations (PDEs) and thus have predictable theoretical properties. Our first architecture involves a convolution operator with a special sparsity structure, and is applicable to a large class of CNNs. Next, we present an architecture that can be seen as a discretization of a diffusion reaction PDE, and use it with three different convolution operators. We outline in our experiments that the proposed architectures,  although considerably reducing the number of trainable weights, yield comparable accuracy to existing CNNs that are fully coupled in the channel dimension.

In diesem Beitrag werden effiziente und ökonomische Parametrisierungen von Convolutional Neural Networks vorgestellt, die durch partielle Differentialgleichungen motiviert sind.

Es werden vier "kostengünstige" Alternativen zur Standard Convolution Operation vorgestellt, die anstelle der Standard Convolution Operation verwendet werden können, um deren Rechenaufwand zu verringern.

In diesem Beitrag werden Methoden zur Reduzierung der Rechenkosten von CNN-Implementierungen vorgestellt und neue Parametrisierungen von CNN-ähnlichen Architekturen eingeführt, die die Parameterkopplung begrenzen.

Die Arbeit schlägt eine PDE-basierte Perspektive zum Verständnis und zur Parametrisierung von CNNs vor.


In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equivalently a lower bound on the negative log likelihood) that the model can assign to some data as one varies the prior and/or the likelihood function in a latent variable model. The core of our contribution is to formally show that the problem of optimizing priors in latent variable models is exactly an instance of the variational optimization problem that information theorists solve when computing rate-distortion functions, and then to use this to derive a lower bound on negative log likelihood. Moreover, we will show that if changing the prior can improve the log likelihood, then there is a way to change the likelihood function instead and attain the same log likelihood, and thus rate-distortion theory is of relevance to both optimizing priors as well as optimizing likelihood functions. We will experimentally argue for the usefulness of quantities derived from rate-distortion theory in latent variable modeling by applying them to a problem in image modeling.

Verwendung der Theorie der Ratenverzerrung, um zu bestimmen, wie stark ein Modell mit latenten Variablen verbessert werden kann.

Befasst sich mit Problemen der Optimierung des Priors im Modell der latenten Variablen und der Auswahl der Likelihood-Funktion, indem  Kriterien vorgeschlagen werden, die auf einer Untergrenze für die negative Log-Likelihood basieren.

Stellt ein Theorem vor, das eine untere Schranke für die negative logarithmische Wahrscheinlichkeit der Ratenverzerrung bei der Modellierung latenter Variablen liefert.

Die Autoren argumentieren, dass die Theorie der Ratenverzerrung für verlustbehaftete Kompression ein natürliches Instrumentarium für die Untersuchung von Modellen mit latenten Variablen bietet und schlagen eine untere Grenze vor.


Backprop is the primary learning algorithm used in many machine learning algorithms. In practice, however, Backprop in deep neural networks is a highly sensitive learning algorithm and successful learning depends on numerous conditions and constraints. One set of constraints is to avoid weights that lead to saturated units. The motivation for avoiding unit saturation is that gradients vanish and as a result learning comes to a halt. Careful weight initialization and re-scaling schemes such as batch normalization ensure that input activity to the neuron is within the linear regime where gradients are not vanished and can flow. Here we investigate backpropagating error terms only linearly. That is, we ignore the saturation that arise by ensuring gradients always flow. We refer to this learning rule as Linear Backprop since in the backward pass the network appears to be linear. In addition to ensuring persistent gradient flow, Linear Backprop is also favorable when computation is expensive since gradients are never computed. Our early results suggest that learning with Linear Backprop is competitive with Backprop and saves expensive gradient computations.

Wir ignorieren Nichtlinearitäten und berechnen keine Gradienten im Rückwärtsdurchlauf, um Berechnungen zu sparen und sicherzustellen, dass Gradienten immer fließen.

Der Autor schlug lineare Backprop-Algorithmen vor, um den Gradientenfluss für alle Teile während der Backpropagation zu gewährleisten.


Deep neural networks with discrete latent variables offer the promise of better symbolic reasoning, and learning  abstractions that are more useful to new tasks. There has been a surge in interest in discrete latent variable models,  however, despite several recent improvements, the training of discrete latent variable models has remained  challenging and their performance has mostly failed to match their continuous counterparts. Recent work on vector quantized autoencoders (VQ-VAE) has made substantial progress in this direction, with its perplexity almost matching that of a VAE on datasets such as CIFAR-10. In this work, we investigate an alternate training technique for VQ-VAE, inspired by its connection to the Expectation Maximization (EM) algorithm. Training the discrete autoencoder with EM and combining it with sequence  level knowledge distillation alows us to develop a non-autoregressive machine translation model whose accuracy almost matches a strong greedy autoregressive baseline Transformer, while being 3.3 times faster at inference.

Systematisches Verständnis des diskreten Autoencoders VQ-VAE unter Verwendung von EM und dessen Verwendung zum Entwurf eines nicht-autoregressiven Übersetzungsmodells, das einer starken autoregressiven Basislinie entspricht.

In diesem Beitrag wird eine neue Art der Interpretation des VQ-VAE vorgestellt und ein neuer Trainingsalgorithmus auf der Grundlage des Soft EM Clustering vorgeschlagen.

Die Arbeit präsentiert eine alternative Sichtweise auf das Trainingsverfahren für den VQ-VAE unter Verwendung des Soft EM Algorithmus.


Recent research about margin theory has proved that maximizing the minimum margin like support vector machines does not necessarily lead to better performance, and instead, it is crucial to optimize the margin distribution. In the meantime, margin theory has been used to explain the empirical success of deep network in recent studies. In this paper, we present ODN (the Optimal margin Distribution Network), a network which embeds a loss function in regard to the optimal margin distribution. We give a theoretical analysis for our method using the PAC-Bayesian framework, which confirms the significance of the margin distribution for classification within the framework of deep networks. In addition, empirical results show that the ODN model always outperforms the baseline cross-entropy loss model consistently across different regularization situations. And our ODN
 model also outperforms the cross-entropy loss (Xent), hinge loss and soft hinge loss model in generalization task through limited training data.

In diesem Beitrag wird ein tiefes neuronales Netz vorgestellt, in das eine Verlustfunktion in Bezug auf die optimale Randverteilung eingebettet ist, die das Overfitting-Problem theoretisch und empirisch entschärft.

Präsentiert eine PAC-Bayes'sche Grenze für einen Margenverlust.


Deep network compression seeks to reduce the number of parameters in the network while maintaining a certain level of performance.   Deep network distillation seeks to train a smaller network that matches soft-max performance of a larger network.   While both regimes have led to impressive performance for their respective goals, neither provide insight into the importance of a given layer in the original model, which is useful if we are to improve our understanding of these highly parameterized models.   In this paper, we present the concept of deep net triage, which individually assesses small blocks of convolution layers to understand their collective contribution to the overall performance, which we call \emph{criticality}.  We call it triage because we assess this criticality by answering the question: what is the impact to the health of the overall network if we compress a block of layers into a single layer.
 We propose a suite of triage methods and compare them on problem spaces of varying complexity.   We ultimately show that, across these problem spaces, deep net triage is able to indicate the of relative importance of different layers.   Surprisingly, our local structural compression technique also leads to an improvement in overall accuracy when the final model is fine-tuned globally.

Wir versuchen, gelernte Repräsentationen in komprimierten Netzen durch ein experimentelles System zu verstehen, das wir Deep Net Triage nennen.

Vergleich verschiedener Initialisierungs- und Trainingsmethoden zur Übertragung von Wissen von einem VGG-Netz auf ein kleineres Studentennetz durch Ersetzen von Blöcken von Schichten durch einzelne Schichten.

In dieser Arbeit werden fünf Methoden für das Triaging oder die Komprimierung von Blockschichten für tiefe Netze vorgestellt.

Die Arbeit schlägt eine Methode zur Komprimierung eines Blocks von Schichten in einem NN vor, bei der mehrere verschiedene Teilansätze bewertet werden.


In this paper, we show a phenomenon, which we named ``super-convergence'', where residual networks can be trained using an order of magnitude fewer iterations than is used with standard training methods.    The existence of super-convergence is relevant to understanding why deep networks generalize well.   One of the key elements of super-convergence is training with cyclical learning rates and a large maximum learning rate.   Furthermore, we present evidence that training with large learning rates improves performance by regularizing the network. In addition, we show that super-convergence provides a  greater boost in performance relative to standard training when the amount of labeled training data is limited.   We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate.   The architectures to replicate this work will be made available upon publication.

Der empirische Nachweis eines neuen Phänomens erfordert neue theoretische Erkenntnisse und ist für die aktive Diskussion in der Literatur über SGD und das Verständnis von Generalisierung von Bedeutung.

In der Arbeit wird ein Phänomen erörtert, bei dem das Training neuronaler Netze in sehr spezifischen Situationen von einem Zeitplan mit hohen Lernraten stark profitieren kann.

Die Autoren analysieren das Training von Residual Networks mit großen zyklischen Lernraten und zeigen schnelle Konvergenz mit zyklischen Lernraten und Beweise für große Lernraten, die als Regularisierung wirken.


Infinite-width neural networks have been extensively used to study the theoretical properties underlying the extraordinary empirical success of standard, finite-width neural networks. Nevertheless, until now, infinite-width networks have been limited to at most two hidden layers. To address this shortcoming, we study the initialisation requirements of these networks and show that the main challenge for constructing them is defining the appropriate sampling distributions for the weights. Based on these observations, we propose a principled approach to weight initialisation that correctly accounts for the functional nature of the hidden layer activations and facilitates the construction of arbitrarily many infinite-width layers, thus enabling the construction of arbitrarily deep infinite-width networks. The main idea of our approach is to iteratively reparametrise the hidden-layer activations into appropriately defined reproducing kernel Hilbert spaces and use the canonical way of constructing probability distributions over these spaces for specifying the required weight distributions in a principled way. Furthermore, we examine the practical implications of this construction for standard, finite-width networks. In particular, we derive a novel weight initialisation scheme for standard, finite-width networks that takes into account the structure of the data and information about the task at hand. We demonstrate the effectiveness of this weight initialisation approach on the MNIST, CIFAR-10 and Year Prediction MSD datasets.

Wir schlagen eine Methode für die Konstruktion beliebig tiefer Netze mit unendlicher Breite vor, auf deren Grundlage wir ein neuartiges Gewichtungsinitialisierungsschema für Netze mit endlicher Breite ableiten und dessen konkurrenzfähige Leistung demonstrieren.

Schlägt einen Ansatz zur Initialisierung von Gewichten vor, um unendlich tiefe und unendlich breite Netzwerke zu ermöglichen, mit experimentellen Ergebnissen auf kleinen Datensätzen.

Vorschlagen tiefer neuronaler Netze von unendlicher Breite.


Working memory requires information about external stimuli to be represented in the brain even after those stimuli go away. This information is encoded in the activities of neurons, and neural activities change over timescales of tens of milliseconds. Information in working memory, however, is retained for tens of seconds, suggesting the question of how time-varying neural activities maintain stable representations. Prior work shows that, if the neural dynamics are in the `  null space' of the representation - so that changes to neural activity do not affect the downstream read-out of stimulus information - then information can be retained for periods much longer than the time-scale of individual-neuronal activities. The prior work, however, requires precisely constructed synaptic connectivity matrices, without explaining how this would arise in a biological neural network. To identify mechanisms through which biological networks can self-organize to learn  memory function, we derived biologically plausible synaptic plasticity rules that dynamically modify the connectivity matrix to enable information storing. Networks implementing this plasticity rule can successfully learn to form memory representations even if only 10% of the synapses are plastic, they are robust to synaptic noise, and they can represent information about multiple stimuli.

Wir haben biologisch plausible Lernregeln für die synaptische Plastizität eines rekurrenten neuronalen Netzes zur Speicherung von Reizrepräsentationen abgeleitet.

Ein neuronales Netzmodell, das aus rekurrent verbundenen Neuronen und einem oder mehreren Redouts besteht und darauf abzielt, eine bestimmte Ausgabe über die Zeit hinweg beizubehalten.

In diesem Beitrag wird ein selbstorganisierender Speichermechanismus in einem neuronalen Modell vorgestellt und eine Zielfunktion eingeführt, die die Änderungen des zu speichernden Signals minimiert.


Generative Adversarial Networks (GANs) have been proposed as an approach to learning generative models. While GANs have demonstrated promising performance on multiple vision tasks, their learning dynamics are not yet well understood, neither in theory nor in practice. In particular, the work in this domain has been focused so far only on understanding the properties of the stationary solutions that this dynamics might converge to, and of the behavior of that dynamics in this solutions’ immediate neighborhood.

 To address this issue, in this work we take a first step towards a principled study of the GAN dynamics itself. To this end, we propose a model that, on one hand, exhibits several of the common problematic convergence behaviors (e.g., vanishing gradient, mode collapse, diverging or oscillatory behavior), but on the other hand, is sufficiently simple to enable rigorous convergence analysis.

 This methodology enables us to exhibit an interesting phenomena: a GAN with an optimal discriminator provably converges, while guiding the GAN training using only a first order approximation of the discriminator leads to unstable GAN dynamics and mode collapse. This suggests that such usage of the first order approximation of the discriminator, which is a de-facto standard in all the existing GAN dynamics, might be one of the factors that makes GAN training so challenging in practice. Additionally, our convergence result constitutes the first rigorous analysis of a dynamics of a concrete parametric GAN.

Um das GAN-Training zu verstehen, definieren wir eine einfache GAN-Dynamik und zeigen die quantitativen Unterschiede zwischen optimalen Updates und Updates erster Ordnung in diesem Modell.

Die Autoren untersuchen die Auswirkungen von GANs in Situationen, in denen bei jeder Iteration der Diskriminator bis zur Konvergenz trainiert und der Generator mit Gradientenschritten aktualisiert wird, oder in denen einige wenige Gradientenschritte für den Diskriminator und den Generator durchgeführt werden.

In diesem Beitrag wird die Dynamik des gegnerischen Trainings von GANs auf einem Gaußschen Mischmodell untersucht.


The machine learning and computer vision community is witnessing an unprecedented rate of new tasks being proposed and addressed, thanks to the power of deep convolutional networks to find complex mappings from X to Y. The advent of each task often accompanies the release of a large-scale human-labeled dataset, for supervised training of the deep network. However, it is expensive and time-consuming to manually label sufficient amount of training data. Therefore, it is important to develop algorithms that can leverage off-the-shelf labeled dataset to learn useful knowledge for the target task. While previous works mostly focus on transfer learning from a single source, we study multi-source transfer across domains and tasks (MS-DTT), in a semi-supervised setting. We propose GradMix, a model-agnostic method applicable to any model trained with gradient-based learning rule. GradMix transfers knowledge via gradient descent, by weighting and mixing the gradients from all sources during training. Our method follows a meta-learning objective, by assigning layer-wise weights to the source gradients, such that the combined gradient follows the direction that can minimize the loss for a small set of samples from the target dataset. In addition, we propose to adaptively adjust the learning rate for each mini-batch based on its importance to the target task, and a pseudo-labeling method to leverage the unlabeled samples in the target domain. We perform experiments on two MS-DTT tasks: digit recognition and action recognition, and demonstrate the advantageous performance of the proposed method against multiple baselines.

Wir schlagen eine gradientenbasierte Methode vor, um Wissen aus verschiedenen Quellen über unterschiedliche Bereiche und Aufgaben hinweg zu übertragen.

In diesem Beitrag wird vorgeschlagen, die Gradienten der Ausgangsdomänen zu kombinieren, um das Lernen in der Zieldomäne zu unterstützen.


Bayesian phylogenetic inference is currently done via Markov chain Monte Carlo with simple mechanisms for proposing new states, which hinders exploration efficiency and often requires long runs to deliver accurate posterior estimates. In this paper we present an alternative approach: a variational framework for Bayesian phylogenetic analysis. We approximate the true posterior using an expressive graphical model for tree distributions, called a subsplit Bayesian network, together with appropriate branch length distributions. We train the variational approximation via stochastic gradient ascent and adopt multi-sample based gradient estimators for different latent variables separately to handle the composite latent space of phylogenetic models. We show that our structured variational approximations are flexible enough to provide comparable posterior estimation to MCMC, while requiring less computation due to a more efficient tree exploration mechanism enabled by variational inference. Moreover, the variational approximations can be readily used for further statistical analysis such as marginal likelihood estimation for model comparison via importance sampling. Experiments on both synthetic data and real data Bayesian phylogenetic inference problems demonstrate the effectiveness and efficiency of our methods.

Die erste Variational-Bayes-Formulierung der phylogenetischen Inferenz, ein anspruchsvolles Inferenzproblem über Strukturen mit verflochtenen diskreten und kontinuierlichen Komponenten.

Erforscht eine Näherungslösung für das Problem der Bayes'schen Inferenz von phylogenetischen Bäumen durch die Nutzung kürzlich vorgeschlagener subsplit Bayes'scher Netzwerke und moderner Gradientenschätzer für VI.

Vorschlagen eines Variationsansatz für die Bayes'sche Posterior-Inferenz in phylogenetischen Bäumen.


This paper introduces HybridNet, a hybrid neural network to speed-up autoregressive
 models for raw audio waveform generation. As an example, we propose
 a hybrid model that combines an autoregressive network named WaveNet and a
 conventional LSTM model to address speech synthesis. Instead of generating
 one sample per time-step, the proposed HybridNet generates multiple samples per
 time-step by exploiting the long-term memory utilization property of LSTMs. In
 the evaluation, when applied to text-to-speech, HybridNet yields state-of-art performance.
 HybridNet achieves a 3.83 subjective 5-scale mean opinion score on
 US English, largely outperforming the same size WaveNet in terms of naturalness
 and provide 2x speed up at inference.

Es handelt sich um eine hybride neuronale Architektur zur Beschleunigung des autoregressiven Modells.

Die Schlussfolgerung lautet, dass ein Modell, das mehrere Zeitschritte gleichzeitig vorhersagt, verwendet werden sollte, um die Modellgröße zu erhöhen, ohne die Inferenzzeit für die sequenzielle Vorhersage zu verlängern.

In diesem Beitrag wird HybridNet vorgestellt, ein neuronales Sprachsynthese- und Audiosynthesesystem, das das WaveNet-Modell mit einem LSTM kombiniert, mit dem Ziel, ein Modell mit schnellerer Inferenzzeit für die Audioerzeugung anzubieten.


Visual Interpretation and explanation of deep models is critical towards wide adoption of systems that rely on them. In this paper, we propose a novel scheme for both interpretation as well as explanation in which, given a pretrained model, we automatically identify internal features relevant for the set of classes considered by the model, without relying on additional annotations. We interpret the model through average visualizations of this reduced set of features. Then, at test time, we explain the network prediction by accompanying the predicted class label with supporting visualizations derived from the identified features. In addition, we propose a method to address the artifacts introduced by strided operations in deconvNet-based visualizations. Moreover, we introduce an8Flower , a dataset specifically designed for objective quantitative evaluation of methods for visual explanation. Experiments on the MNIST , ILSVRC 12, Fashion 144k and an8Flower datasets show that our method produces detailed explanations with good coverage of relevant features of the classes of interest.

Interpretation durch Identifizierung der vom Modell gelernten Merkmale, die als Indikatoren für die interessierende Aufgabe dienen. Erklären von Modellentscheidungen durch Hervorheben der Reaktion dieser Merkmale in Testdaten. Objektive Evaluierung der Erklärungen anhand eines kontrollierten Datensatzes.

In diesem Beitrag wird eine Methode zur Erstellung visueller Erklärungen für die Ergebnisse tiefer neuronaler Netze vorgeschlagen und ein neuer synthetischer Datensatz veröffentlicht.

Eine Methode für tiefe neuronale Netze, die automatisch relevante Merkmale des Klassensatzes identifiziert und die Interpretation und Erklärung unterstützt, ohne auf zusätzliche Annotationen angewiesen zu sein.


In this work we propose a simple and efficient framework for learning sentence representations from unlabelled data. Drawing inspiration from the distributional hypothesis and recent work on learning sentence representations, we reformulate the problem of predicting the context in which a sentence appears as a classification problem. Given a sentence and the context in which it appears, a classifier distinguishes context sentences from other contrastive sentences based on their vector representations. This allows us to efficiently learn different types of encoding functions, and we show that the model learns high-quality sentence representations. We demonstrate that our sentence representations outperform state-of-the-art unsupervised and supervised representation learning methods on several downstream NLP tasks that involve understanding sentence semantics while achieving an order of magnitude speedup in training time.

Ein Rahmenwerk zum effizienten Erlernen hochwertiger Satzrepräsentationen.

Schlägt einen schnelleren Algorithmus für das Lernen von Satzrepräsentationen im Stil von SkipThought aus Korpora geordneter Sätze vor, der den Decoder auf Wortebene durch einen kontrastiven Klassifikationsverlust ersetzt.

Diese Arbeit schlägt einen Rahmen für das unbeaufsichtigte Lernen von Satzrepräsentationen vor, indem ein Modell der Wahrscheinlichkeit wahrer Kontextsätze relativ zu zufälligen Kandidatensätzen maximiert wird.


Many regularization methods have been proposed to prevent overfitting in neural networks. Recently, a regularization method has been proposed to optimize the variational lower bound of the Information Bottleneck Lagrangian. However, this method cannot be generalized to regular neural network architectures. We present the activation norm penalty that is derived from the information bottleneck principle and is theoretically grounded in a variation dropout framework. Unlike in previous literature, it can be applied to any general neural network. We demonstrate that this penalty can give consistent improvements to different state of the art architectures both in language modeling and image classification. We present analyses on the properties of this penalty and compare it to other methods that also reduce mutual information.

Aus der Perspektive des Informationsengpasses leiten wir eine Normstrafe für den Ausgang des neuronalen Netzes ab.

Setzt eine Aktivierungsnorm-Strafe ein, eine Regularisierung vom Typ L_2 auf die Aktivierungen, die sich aus dem Prinzip des Informationsengpasses ableitet.

In diesem Beitrag wird eine Zuordnung zwischen Aktivierungsnorm Strafen und Informationsengpass Frameworks unter Verwendung des Variational Dropout Frameworks erstellt.


Unsupervised learning of timeseries data is a challenging problem in machine learning. Here,
we propose a novel algorithm, Deep Temporal Clustering (DTC), a fully unsupervised method, to naturally integrate dimensionality reduction and temporal clustering into a single end to end learning framework. The algorithm starts with an initial cluster estimates using an autoencoder for dimensionality reduction and a novel temporal clustering layer for cluster assignment. Then it jointly optimizes the clustering objective and the dimensionality reduction objective. Based on requirement and application, the temporal clustering layer can be customized with any temporal similarity metric. Several similarity metrics are considered and compared.   To gain insight into features that the network has learned for its clustering, we apply a visualization method that generates a heat map of regions of interest in the timeseries. The viability of the algorithm is demonstrated using timeseries data from diverse domains, ranging from earthquakes to sensor data from spacecraft. In each case, we show that our algorithm outperforms traditional methods. This performance is attributed to fully integrated temporal dimensionality reduction and clustering criterion.

Eine vollständig unbeaufsichtigte Methode, die Dimensionalitätsreduktion und zeitliches Clustering auf natürliche Weise in ein einziges Ende-zu-Ende Lernsystem integriert.

Schlägt einen Algorithmus vor, der Autoencoder mit Zeitreihendaten-Clustering unter Verwendung einer Netzwerkstruktur integriert, die für Zeitreihendaten geeignet ist.

Ein Algorithmus für die gemeinsame Durchführung von Dimensionalitätsreduktion und zeitlichem Clustering in einem Deep-Learning-Kontext, der einen Autoencoder und ein Clustering-Ziel verwendet.

Die Autoren schlugen eine unbeaufsichtigte Zeitreihen-Clustermethode vor, die auf tiefen neuronalen Netzen aufbaut und mit einem Encoder-Decoder und einem Clustermodus ausgestattet ist, um die Zeitreihen zu verkürzen, lokale zeitliche Merkmale zu extrahieren und die kodierten Darstellungen zu erhalten.


We study many-class few-shot (MCFS) problem in both supervised learning and meta-learning scenarios. Compared to the well-studied many-class many-shot and few-class few-shot problems, MCFS problem commonly occurs in practical applications but is rarely studied. MCFS brings new challenges because it needs to distinguish between many classes, but only a few samples per class are available for training. In this paper, we propose ``memory-augmented hierarchical-classification network (MahiNet)'' for MCFS learning. It addresses the ``many-class'' problem by exploring the class hierarchy, e.g., the coarse-class label that covers a subset of fine classes, which helps to narrow down the candidates for the fine class and is cheaper to obtain. MahiNet uses a convolutional neural network (CNN) to extract features, and integrates a memory-augmented attention module with a multi-layer perceptron (MLP) to produce the probabilities over coarse and fine classes. While the MLP extends the linear classifier, the attention module extends a KNN classifier, both together targeting the ''`few-shot'' problem. We design different training strategies of MahiNet for supervised learning and meta-learning. Moreover, we propose two novel benchmark datasets ''mcfsImageNet'' (as a subset of ImageNet) and ''mcfsOmniglot'' (re-splitted Omniglot) specifically for MCFS problem. In experiments, we show that MahiNet outperforms several state-of-the-art models on MCFS classification tasks in both supervised learning and meta-learning scenarios.

Ein gedächtniserweitertes neuronales Netzwerk, das das Many-Class Few-Shot Problem angeht, indem es die Klassenhierarchie sowohl beim überwachten Lernen als auch beim Meta-Lernen nutzt.

In diesem Beitrag werden Methoden vorgestellt, mit denen ein Klassifikator durch grobe bis feine Vorhersagen entlang einer Klassenhierarchie und durch das Lernen eines speicherbasierten KNN-Klassifikators, der während des Lernens falsch beschriftete Instanzen verfolgt, induktiv beeinflusst werden kann.

In diesem Beitrag wird das Problem der Many-Class Few-Shot Klassifizierung aus der Perspektive des überwachten Lernens und des Meta-Lernens formuliert.


Learning a better representation with neural networks is a challenging problem, which has been tackled from different perspectives in the past few years. In this work, we focus on learning a representation that would be useful in a clustering task. We introduce two novel loss components that substantially improve the quality of produced clusters, are simple to apply to arbitrary models and cost functions, and do not require a complicated training procedure. We perform an extensive set of experiments, supervised and unsupervised, and evaluate the proposed loss components on two most common types of models, Recurrent Neural Networks and Convolutional Neural Networks, showing that the approach we propose consistently improves the quality of KMeans clustering in terms of mutual information scores and outperforms previously proposed methods.

Eine neuartige Verlustkomponente, die das Netz dazu zwingt, während des Trainings für eine Klassifizierungsaufgabe eine Repräsentation zu erlernen, die für die Clusterbildung gut geeignet ist.

Dieses Arbeit schlägt zwei Regularisierungsbedingungen vor, die auf einem zusammengesetzten Scharnierverlust über die KL-Divergenz zwischen zwei softmax-normalisierten Eingangsargumenten basieren, um das Lernen von entkoppelten Repräsentationen zu fördern.

Vorschlag für zwei Regularisierer, die dafür sorgen sollen, dass die in der vorletzten Schicht eines Klassifikators gelernten Repräsentationen besser mit der inhärenten Struktur der Daten übereinstimmen.


In high dimensions, the performance of nearest neighbor algorithms depends crucially on structure in the data.
 While traditional nearest neighbor datasets consisted mostly of hand-crafted feature vectors, an increasing number of datasets comes from representations learned with neural networks.
 We study the interaction between nearest neighbor algorithms and neural networks in more detail.
 We find that the network architecture can significantly influence the efficacy of nearest neighbor algorithms even when the classification accuracy is unchanged.
 Based on our experiments, we propose a number of training modifications that lead to significantly better datasets for nearest neighbor algorithms.
 Our modifications lead to learned representations that can accelerate nearest neighbor queries by 5x.

Wir zeigen, wie man gute Darstellungen aus der Sicht der Ähnlichkeitssuche erhält.

Untersucht die Auswirkungen eines Wechsels des Bildklassifikationsteils über dem DNN auf die Fähigkeit, die Deskriptoren mit einem LSH- oder einem kd-Baum-Algorithmus zu indizieren.

Es wird vorgeschlagen, den Softmax-Kreuzentropieverlust zu verwenden, um ein Netzwerk zu lernen, das versucht, die Winkel zwischen den Eingaben und den entsprechenden Klassenvektoren in einem überwachten Rahmen zu reduzieren.


Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid. These are subsequently relaxed to continuous surrogates that can allow for efficient gradient-based optimization. We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent. We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classification.

Wir stellen eine Technik vor, die ein gradientenbasiertes Training von quantisierten neuronalen Netzen ermöglicht.

Schlägt eine einheitliche und allgemeine Methode für das Training neuronaler Netze mit quantisierten synaptischen Gewichten und Aktivierungen reduzierter Präzision vor.

Ein neuer Ansatz zur Quantisierung von Aktivierungen, der bei mehreren realen Bildproblemen Stand der Technik oder wettbewerbsfähig ist.

Verfahren zum Lernen neuronaler Netze mit quantisierten Gewichten und Aktivierungen durch stochastische Quantisierung von Werten und Ersetzen der resultierenden kategorischen Verteilung durch eine kontinuierliche Entspannung.


In most current formulations of adversarial training, the discriminators can be expressed as single-input operators, that is, the mapping they define is separable over observations. In this work, we argue that this property might help explain the infamous mode collapse phenomenon in adversarially-trained generative models. Inspired by discrepancy measures and two-sample tests between probability distributions, we propose distributional adversaries that operate on samples, i.e., on sets of multiple points drawn from a distribution, rather than on single observations. We show how they can be easily implemented on top of existing models. Various experimental results show that generators trained in combination with our distributional adversaries are much more stable and are remarkably less prone to mode collapse than traditional models trained with observation-wise prediction discriminators. In addition, the application of our framework to domain adaptation results in strong improvement over recent state-of-the-art.

Wir zeigen, dass das Problem des Modus-Kollapses in GANs durch einen Mangel an Informationsaustausch zwischen Beobachtungen in einem Trainingsstapel erklärt werden kann, und schlagen einen verteilungsbasierten Rahmen für den globalen Informationsaustausch zwischen Gradienten vor, der zu einem stabileren und effektiveren adversen Training führt.

Es wird vorgeschlagen, Diskriminatoren, die nur eine einzige Probe berücksichtigen, durch Diskriminatoren zu ersetzen, die explizit mit Verteilungen von Beispielen arbeiten.

Theorie über Two-Sample Tests und MMD und wie sie vorteilhaft in den GAN-Rahmen integriert werden können.


Chemical information extraction is to convert chemical knowledge in text into true chemical database, which is a text processing task heavily relying on chemical compound name identification and standardization. Once a systematic name for a chemical compound is given, it will naturally and much simply convert the name into the eventually required molecular formula. However, for many chemical substances, they have been shown in many other names besides their systematic names which poses a great challenge for this task. In this paper, we propose a framework to do the auto standardization from the non-systematic names to the corresponding systematic names by using the spelling error correction, byte pair encoding tokenization and neural sequence to sequence model. Our framework is trained end to end and is fully data-driven. Our standardization accuracy on the test dataset achieves 54.04% which has a great improvement compared to previous state-of-the-art result.

Wir haben ein Ende-zu-Ende Framework entwickelt, das ein Sequenz zu Sequenz Modell für die Standardisierung chemischer Namen verwendet.

Standardisiert nicht systematische Namen in der chemischen Informationsextraktion durch Erstellung eines parallelen Korpus von nicht systematischen und systematischen Namen und Aufbau eines seq2seq-Modells.

In dieser Arbeit wird eine Methode vorgestellt, mit der nicht systematische Namen chemischer Verbindungen durch eine Kombination von Mechanismen in ihre systematischen Äquivalente übersetzt werden können.


The training of deep neural networks with Stochastic Gradient Descent (SGD) with a large learning rate or a small batch-size typically ends in flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. This was found to correlate with a good final generalization performance.   In this paper we extend previous work by investigating the curvature of the loss surface along the whole training trajectory, rather than only at the endpoint. We find that initially SGD visits increasingly sharp regions, reaching a maximum sharpness determined by both the learning rate and the batch-size of SGD. At this peak value SGD starts to fail to minimize the loss along directions in the loss surface corresponding to the largest curvature (sharpest directions). To further investigate the effect of these dynamics in the training process, we study a variant of SGD using a reduced learning rate along the sharpest directions which we show can improve training speed while finding both sharper and better generalizing solution, compared to vanilla SGD. Overall, our results show that the SGD dynamics in the subspace of the sharpest directions influence the regions that SGD steers to (where larger learning rate or smaller batch size result in wider regions visited), the overall training speed, and the generalization ability of the final model.

Die SGD wird zu Beginn des Trainings in einen Bereich gelenkt, in dem ihr Schritt im Vergleich zur Krümmung zu groß ist, was sich auf den Rest des Trainings auswirkt.

Analysiert die Beziehung zwischen der Konvergenz/Generalisierung und der Aktualisierung der größten Eigenvektoren der Hessian der empirischen Verluste von DNNs.

In dieser Arbeit wird die Beziehung zwischen der SGD-Schrittgröße und der Krümmung der Verlustfläche untersucht


We introduce a new approach to estimate continuous actions using actor-critic algorithms for reinforcement learning problems. Policy gradient methods usually predict one continuous action estimate or parameters of a presumed distribution (most commonly Gaussian) for any given state which might not be optimal as it may not capture the complete description of the target distribution. Our approach instead predicts M actions with the policy network (actor) and then uniformly sample one action during training as well as testing at each state. This allows the agent to learn a simple stochastic policy that has an easy to compute expected return. In all experiments, this facilitates better exploration of the state space during training and converges to a better policy.

Wir stellen einen neuartigen Reinforcement Learning Algorithmus vor, der mehrere Handlungen vorhersagt und daraus Proben zieht.

In dieser Arbeit wird eine einheitliche Mischung aus deterministischen Richtlinien eingeführt, und es wird festgestellt, dass diese Parametrisierung stochastischer Richtlinien DDPG bei mehreren OpenAI-Gym-Benchmarks übertrifft.

Die Autoren untersuchen eine Methode zur Verbesserung der Leistung von Netzwerken, die mit DDPG trainiert wurden, und zeigen eine verbesserte Leistung bei einer großen Anzahl von standardmäßigen kontinuierlichen Kontrollumgebungen.