Despite much success in many large-scale language tasks, sequence-to-sequence (seq2seq) models have not been an ideal choice for conversational modeling as they tend to generate generic and repetitive responses. In this paper, we propose a Latent Topic Conversational Model (LTCM) that augments the seq2seq model with a neural topic component to better model human-human conversations. The neural topic component encodes information from the source sentence to build a global “topic” distribution over words, which is then consulted by the seq2seq model to improve generation at each time step. The experimental results show that the proposed LTCM can generate more diverse and interesting responses by sampling from its learnt latent representations. In a subjective human evaluation, the judges also confirm that LTCM is the preferred option comparing to competitive baseline models.

Latent Topic Conversational Model, eine Mischung aus seq2seq und neuronalem Themenmodell, um vielfältigere und interessantere Antworten zu generieren.

In diesem Beitrag wird eine Kombination aus Themenmodell und seq2seq-Konversationsmodell vorgeschlagen.

Schlägt ein Gesprächsmodell mit thematischen Informationen durch die Kombination seq2seq Modell mit neuronalen Thema Modelle und zeigt das vorgeschlagene Modell übertrifft einige der Grundlinien seq2seq Modelle und andere latente Variablen seq2seq Modell Varianten.

Der Beitrag befasst sich mit dem Problem der anhaltenden Aktualität in Gesprächsmodellen und schlägt ein Modell vor, das eine Kombination aus einem neuronalen Themenmodell und einem seq2seq-basierten Dialogsystem ist.


Most of the existing Graph Neural Networks (GNNs) are the mere extension of the Convolutional Neural Networks (CNNs) to graphs. Generally, they consist of several steps of message passing between the nodes followed by a global indiscriminate feature pooling function. In many data-sets, however, the nodes are unlabeled or their labels provide no information about the similarity between the nodes and the locations of the nodes in the graph. Accordingly, message passing may not propagate helpful information throughout the graph. We show that this conventional approach can fail to learn to perform even simple graph classification tasks. We alleviate this serious shortcoming of the GNNs by making them a two step method. In the first of the proposed approach, a graph embedding algorithm is utilized to obtain a continuous feature vector for each node of the graph. The embedding algorithm represents the graph as a point-cloud in the embedding space. In the second step, the GNN is applied to the point-cloud representation of the graph provided by the embedding method. The GNN learns to perform the given task by inferring the topological structure of the graph encoded in the spatial distribution of the embedded vectors. In addition, we extend the proposed approach to the graph clustering problem and a new architecture for graph clustering is proposed. Moreover, the spatial representation of the graph is utilized to design a graph pooling algorithm. We turn the problem of graph down-sampling into a column sampling problem, i.e., the sampling algorithm selects a subset of the nodes whose feature vectors preserve the spatial distribution of all the feature vectors. We apply the proposed approach to several popular benchmark data-sets and it is shown that the proposed geometrical approach strongly improves the state-of-the-art result for several data-sets. For instance, for the PTC data-set, we improve the state-of-the-art result for more than 22 %.

Das Problem der Graphenanalyse wird in ein Problem der Punktwolkenanalyse umgewandelt.

Schlägt ein tiefes GNN-Netzwerk für Graphenklassifizierungsprobleme vor, das eine adaptive Graphenpooling-Schicht verwendet.

Die Autoren schlagen eine Methode zum Lernen von Darstellungen für Graphen vor.


Deep neural networks (DNNs) have been found to be vulnerable to adversarial examples resulting from adding small-magnitude perturbations to inputs. Such adversarial examples can mislead DNNs to produce adversary-selected results.
 Different attack strategies have been proposed to generate adversarial examples, but how to produce them with high perceptual quality and more efficiently requires more research efforts.
 In this paper, we propose AdvGAN to generate adversarial examples with generative adversarial networks (GANs), which can learn and approximate the distribution of original instances.
 For AdvGAN, once the generator is trained, it can generate adversarial perturbations efficiently for any instance, so as to potentially accelerate adversarial training as defenses.
 We apply AdvGAN in both semi-whitebox and black-box attack settings. In semi-whitebox attacks, there is no need to access the original target model after the generator is trained, in contrast to traditional white-box attacks. In black-box attacks, we dynamically train a distilled model for the black-box model and optimize the generator accordingly.
 Adversarial examples generated by AdvGAN on different target models have high attack success rate under state-of-the-art defenses compared to other attacks. Our attack  has placed the first with 92.76% accuracy on a public MNIST black-box attack challenge.

Wir schlagen vor, adversarische Beispiele auf der Grundlage generativer adversarischer Netze in einer Semi-Whitebox- und Blackbox-Umgebung zu generieren.

Beschreibt AdvGAN, ein Conditional GAN plus Adversarial Loss, und evaluiert AdvGAN in Semi-White-Box- und Black-Box-Settings und berichtet über den aktuellen Stand der Technik.

In dieser Arbeit wird eine Methode zur Erzeugung von Gegenbeispielen vorgeschlagen, die Klassifizierungssysteme täuschen und die MadryLab mnist-Herausforderung gewinnen.


This paper proposes a new model for the rating prediction task in recommender systems which significantly outperforms previous state-of-the art models on a time-split Netflix data set. Our model is based on deep autoencoder with 6 layers and is trained end-to-end without any layer-wise pre-training. We empirically demonstrate that: a) deep autoencoder models generalize much better than the shallow ones, b) non-linear activation functions with negative parts are crucial for training deep models, and c) heavy use of regularization techniques such as dropout is necessary to prevent over-fitting. We also propose a new training algorithm based on iterative output re-feeding to overcome natural sparseness of collaborate filtering. The new algorithm significantly speeds up training and improves model performance. Our code is publicly available.

In diesem Beitrag wird gezeigt, wie tiefe Autoencoder durchgängig trainiert werden können, um SoA-Ergebnisse auf einem zeitlich geteilten Netflix-Datensatz zu erzielen.

In diesem Beitrag wird ein Deep Autoencoder Modell für die Vorhersage von Bewertungen vorgestellt, das andere State-of-the-Art-Ansätze auf dem Netflix Preisdatensatz übertrifft.

Schlägt vor, eine tiefe AE für die Vorhersage von Bewertungen in Empfehlungssystemen zu verwenden.

Die Autoren stellen ein Modell für genauere Netflix-Empfehlungen vor, das zeigt, dass ein Deep Autoencoder komplexere RNN-basierte Modelle mit zeitlichen Informationen übertreffen kann.


Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times. Despite these successes, however, popular feed-forward sequence models like the Transformer fail to generalize in many simple tasks that recurrent models handle with ease, e.g. copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time. We propose the Universal Transformer (UT), a parallel-in-time self-attentive recurrent sequence model which can be cast as a generalization of the Transformer model and which addresses these issues. UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs. We also add a dynamic per-position halting mechanism and find that it improves accuracy on several tasks. In contrast to the standard Transformer, under certain assumptions UTs can be shown to be Turing-complete. Our experiments show that UTs outperform standard Transformers on a wide range of algorithmic and language understanding tasks, including the challenging LAMBADA language modeling task where UTs achieve a new state of the art, and machine translation where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De dataset.

Wir stellen den Universal Transformer vor, ein selbstaufmerksames, parallel-in-time rekurrentes Sequenzmodell, das Transformers und LSTMs bei einer Vielzahl von Sequenz-zu-Sequenz-Aufgaben, einschließlich maschineller Übersetzung, übertrifft.

Schlägt ein neues Modell UT vor, das auf dem Transformer-Modell basiert, mit zusätzlicher Rekursion und dynamischem Stoppen der Rekursion.

Diese Arbeit erweitert Transformer durch die rekursive Anwendung eines Mult-Head Self-Attention Blocks, anstatt mehrere Blöcke im Standard Transformer zu stapeln.


We present a framework for interpretable continual learning (ICL). We show that explanations of previously performed tasks can be used to improve performance on future tasks. ICL generates a good explanation of a finished task, then uses this to focus attention on what is important when facing a new task. The ICL idea is general and may be applied to many continual learning approaches. Here we focus on the variational continual learning framework to take advantage of its flexibility and efficacy in overcoming catastrophic forgetting. We use saliency maps to provide explanations of performed tasks and propose a new metric to assess their quality. Experiments show that ICL achieves state-of-the-art results in terms of overall continual learning performance as measured by average classification accuracy, and also in terms of its explanations, which are assessed qualitatively and quantitatively using the proposed metric.

In dem Beitrag wird ein interpretierbarer Rahmen für kontinuierliches Lernen entwickelt, in dem Erklärungen zu den abgeschlossenen Aufgaben verwendet werden, um die Aufmerksamkeit des Lernenden bei zukünftigen Aufgaben zu erhöhen, und in dem auch eine Erklärungsmetrik vorgeschlagen wird.

Die Autoren schlagen ein Framework für kontinuierliches Lernen vor, das auf Erklärungen für durchgeführte Klassifizierungen von zuvor gelernten Aufgaben basiert.

In diesem Beitrag wird eine Erweiterung des Rahmens für kontinuierliches Lernen vorgeschlagen, die auf der Grundlage der bestehenden Methode des variablen kontinuierlichen Lernens und der Beweiskraft der Daten beruht.


The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikevicius et al. (2017). On the other hand, while a lot of research has also happened in the domain of low and mixed-precision Integer training, these works either present results for non-SOTA networks (for instance only AlexNet for ImageNet-1K), or relatively small datasets (like CIFAR-10). In this work, we train state-of-the-art visual understanding neural networks on the ImageNet-1K dataset, with Integer operations on General Purpose (GP) hardware. In particular, we focus on Integer Fused-Multiply-and-Accumulate (FMA) operations which take two pairs of INT16 operands and accumulate results into an INT32 output.We propose a shared exponent representation of tensors and develop a Dynamic Fixed Point (DFP) scheme suitable for common neural network operations. The nuances of developing an efficient integer convolution kernel is examined, including methods to handle overflow of the INT32 accumulator. We implement CNN training for ResNet-50, GoogLeNet-v1, VGG-16 and AlexNet; and these networks achieve or exceed SOTA accuracy within the same number of iterations as their FP32 counterparts without any change in hyper-parameters and with a 1.8X improvement in end-to-end training throughput. To the best of our knowledge these results represent the first INT16 training results on GP hardware for ImageNet-1K dataset using SOTA CNNs and achieve highest reported accuracy using half precision

Trainingspipeline mit gemischter Genauigkeit unter Verwendung von 16-Bit Ganzzahlen auf Allzweck Hardware; SOTA-Genauigkeit für CNNs der ImageNet-Klasse; beste gemeldete Genauigkeit für ImageNet-1K Klassifizierungsaufgabe mit Training mit reduzierter Genauigkeit;

In diesem Beitrag wird gezeigt, dass eine sorgfältige Implementierung der dynamischen Festkommaberechnung mit gemischter Genauigkeit den neuesten Stand der Technik unter Verwendung eines Deep Learning Modells mit reduzierter Genauigkeit und einer 16-Bit Ganzzahldarstellung erreichen kann.

Schlägt ein "dynamisches Festkomma" Schema vor, das den Exponententeil für einen Tensor teilt, und entwickelt Verfahren, um NN-Berechnungen mit diesem Format durchzuführen, und demonstriert dies für Training mit begrenzter Genauigkeit.


In this paper we introduce a new speech recognition system, leveraging a simple letter-based ConvNet acoustic model. The acoustic model requires only audio transcription for training -- no alignment annotations, nor any forced alignment step is needed. At inference, our decoder takes only a word list and a language model, and is fed with letter scores from the acoustic model -- no phonetic word lexicon is needed. Key ingredients for the acoustic model are Gated Linear Units and high dropout. We show near state-of-the-art results in word error rate on the LibriSpeech corpus with MFSC features, both on the clean and other configurations.

Ein buchstabenbasiertes akustisches ConvNet-Modell führt zu einer einfachen und konkurrenzfähigen Spracherkennungspipeline.

Diese Arbeit wendet Gated Convolutional Neural Networks auf die Spracherkennung an, wobei das Trainingskriterium ASG verwendet wird.


Generative adversarial networks (GANs) are a powerful framework for generative tasks. However, they are difficult to train and tend to miss modes of the true data generation process. Although GANs can learn a rich representation of the covered modes of the data in their latent space, the framework misses an inverse mapping from data to this latent space. We propose Invariant Encoding Generative Adversarial Networks (IVE-GANs), a novel GAN framework that introduces such a mapping for individual samples from the data by utilizing features in the data which are invariant to certain transformations. Since the model maps individual samples to the latent space, it naturally encourages the generator to cover all modes. We demonstrate the effectiveness of our approach in terms of generative performance and learning rich representations on several datasets including common benchmark image generation tasks.

Ein neuartiger GAN-Rahmen, der transformationsinvariante Merkmale nutzt, um umfangreiche Repräsentationen und starke Generatoren zu erlernen.

Schlägt ein modifiziertes GAN-Ziel vor, das aus einem klassischen GAN-Term und einem invarianten Codierungsterm besteht.

In diesem Beitrag wird das IVE-GAN vorgestellt, ein Modell, das einen Encoder in das Generative Adversarial Network Framework einführt.


We propose a method for learning the dependency structure between latent variables in deep latent variable models.   Our general modeling and inference framework combines the complementary strengths of deep generative models and probabilistic graphical models. In particular, we express the latent variable space of a variational autoencoder (VAE) in terms of a Bayesian network with a learned, flexible dependency structure.   The network parameters, variational parameters as well as the latent topology are optimized simultaneously with a single objective.   Inference is formulated via a sampling procedure that produces expectations over latent variable structures and incorporates top-down and bottom-up reasoning over latent variable values.   We validate our framework in extensive experiments on MNIST, Omniglot, and CIFAR-10. Comparisons to state-of-the-art structured variational autoencoder baselines show improvements in terms of the expressiveness of the learned model.

Wir schlagen eine Methode zum Erlernen latenter Abhängigkeitsstrukturen in variablen Autoencodern vor.

Verwendet eine Matrix binärer Zufallsvariablen zur Erfassung von Abhängigkeiten zwischen latenten Variablen in einem hierarchischen tiefen generativen Modell.

In dieser Arbeit wird ein VAE-Ansatz vorgestellt, bei dem während des Trainings eine Abhängigkeitsstruktur auf der latenten Variable gelernt wird.

Die Autoren schlagen vor, den latenten Raum einer VAE um eine autoregressive Struktur zu erweitern, um die Aussagekraft sowohl des Inferenznetzwerks als auch des latenten Priors zu verbessern.


Many real-world time series, such as in activity recognition, finance, or climate science, have changepoints where the system's structure or parameters change. Detecting changes is important as they may indicate critical events. However, existing methods for changepoint detection face challenges when (1) the patterns of change cannot be modeled using simple and predefined metrics, and (2) changes can occur gradually, at multiple time-scales. To address this, we show how changepoint detection can be treated as a supervised learning problem, and propose a new deep neural network architecture that can efficiently identify both abrupt and gradual changes at multiple scales. Our proposed method, pyramid recurrent neural network (PRNN), is designed to be scale-invariant, by incorporating wavelets and pyramid analysis techniques from multi-scale signal processing. Through experiments on synthetic and real-world datasets, we show that PRNN can detect abrupt and gradual changes with higher accuracy than the state of the art and can extrapolate to detect changepoints at novel timescales that have not been seen in training.

Wir stellen eine skaleninvariante neuronale Netzwerkarchitektur für die Erkennung von Veränderungspunkten in multivariaten Zeitreihen vor.

Die Arbeit nutzt das Konzept der Wavelet-Transformation innerhalb einer tiefen Architektur, um die Erkennung von Änderungspunkten zu lösen.

In diesem Beitrag wird ein pyramidenbasiertes neuronales Netz vorgeschlagen und auf 1D-Signale angewendet, deren zugrunde liegende Prozesse auf verschiedenen Zeitskalen ablaufen, wobei die Aufgabe darin besteht, Veränderungen zu erkennen.


We demonstrate how to learn efficient heuristics for automated reasoning algorithms through deep reinforcement learning. We focus on backtracking search algorithms for quantified Boolean logics, which already can solve formulas of impressive size - up to 100s of thousands of variables. The main challenge is to find a representation of these formulas that lends itself to making predictions in a scalable way. For challenging problems, the heuristic learned through our approach reduces execution time by a factor of 10 compared to the existing handwritten heuristics.

RL findet bessere Heuristiken für automatische Schlussfolgerungsalgorithmen.

Ziel ist es, eine Heuristik für einen Backtracking-Suchalgorithmus unter Verwendung von Reinforcement Learning zu erlernen, und vorschlagen eines Modells, das grafische neuronale Netze verwendet, um die Einbettung von Wörtern und Klauseln zu erzeugen und sie zur Vorhersage der Qualität jedes Worts zu verwenden, um die Wahrscheinlichkeit jeder Aktion zu bestimmen.

Die Arbeit schlägt einen Ansatz zum automatischen Lernen von Variablenauswahlheuristiken für QBF unter Verwendung von Deep Learning vor.


We consider the question of how to assess generative adversarial networks, in particular with respect to whether or not they generalise beyond memorising the training data. We propose a simple procedure for assessing generative adversarial network performance based on a principled consideration of what the actual goal of generalisation is. Our approach involves using a test set to estimate the Wasserstein distance between the generative distribution produced by our procedure, and the underlying data distribution. We use this procedure to assess the performance of several modern generative adversarial network architectures. We find that this procedure is sensitive to the choice of ground metric on the underlying data space, and suggest a choice of ground metric that substantially improves performance.   We finally suggest that attending to the ground metric used in Wasserstein generative adversarial network training may be fruitful, and outline a concrete pathway towards doing so.

Beurteilen Sie, ob Ihr GAN tatsächlich etwas anderes tut, als sich die Trainingsdaten zu merken oder nicht.

Ziel ist es, ein Qualitätsmaß/einen Test für GANs bereitzustellen, und es wird vorgeschlagen, die aktuelle Annäherung an eine von einem GAN gelernte Verteilung zu bewerten, indem der Wasserstein Abstand zwischen zwei Verteilungen, die aus einer Summe von Diracs bestehen, als Basisleistung verwendet wird.

In diesem Beitrag wird ein Verfahren zur Bewertung der Leistung von GANs vorgeschlagen, bei dem der Beobachtungsschlüssel erneut berücksichtigt wird. Das Verfahren wird verwendet, um die aktuellen GANs zu testen und zu verbessern.


The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various hand-designed alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose to leverage automatic search techniques to discover new activation functions. Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions. We verify the effectiveness of the searches by conducting an empirical evaluation with the best discovered activation function. Our experiments show that the best discovered activation function, f(x) = x * sigmoid(beta * x), which we name Swish, tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.

Wir verwenden Suchtechniken, um neue Aktivierungsfunktionen zu entdecken, und unsere beste entdeckte Aktivierungsfunktion, f(x) = x * sigmoid(beta * x), übertrifft ReLU bei einer Reihe von anspruchsvollen Aufgaben wie ImageNet.

Schlägt einen auf Reinforcement Learning basierenden Ansatz zum Auffinden von Nichtlinearität vor, indem er Kombinationen aus einer Reihe von unären und binären Operatoren durchsucht.

In dieser Arbeit wird das Reinforcement Learning genutzt, um die Kombination einer Reihe von unären und binären Funktionen zu suchen, die zu einer neuen Aktivierungsfunktion führen.

Der Autor verwendet Reinforcement Learning, um neue potenzielle Aktivierungsfunktionen aus einer Vielzahl von möglichen Kandidaten zu finden.


Successful training of convolutional neural networks is often associated with suffi-
 ciently deep architectures composed of high amounts of features. These networks
 typically rely on a variety of regularization and pruning techniques to converge
 to less redundant states. We introduce a novel bottom-up approach to expand
 representations in fixed-depth architectures. These architectures start from just a
 single feature per layer and greedily increase width of individual layers to attain
 effective representational capacities needed for a specific task. While network
 growth can rely on a family of metrics, we propose a computationally efficient
 version based on feature time evolution and demonstrate its potency in determin-
 ing feature importance and a networks’ effective capacity. We demonstrate how
 automatically expanded architectures converge to similar topologies that benefit
 from lesser amount of parameters or improved accuracy and exhibit systematic
 correspondence in representational complexity with the specified task. In contrast
 to conventional design patterns with a typical monotonic increase in the amount of
 features with increased depth, we observe that CNNs perform better when there is
 more learnable parameters in intermediate, with falloffs to earlier and later layers.

Ein Bottom-up-Algorithmus, der CNNs, die mit einem Merkmal pro Schicht beginnen, zu Architekturen mit ausreichender Darstellungskapazität ausbaut.

Es wird vorgeschlagen, die Tiefe der Merkmalszuordnungen eines voll Convolutional Neural Networks dynamisch anzupassen, ein Maß für die Selbstähnlichkeit zu formulieren und die Leistung zu steigern.

Einführung einer einfachen korrelationsbasierten Metrik zur Messung der effektiven Nutzung von Filtern in neuronalen Netzen als Indikator für die effektive Kapazität.

Ziel ist es, das Problem der Suche nach Deep Learning Architekturen durch inkrementelles Hinzufügen und Entfernen von Kanälen in den Zwischenschichten des Netzwerks zu lösen.


Deep neural networks are almost universally trained with reverse-mode automatic differentiation (a.k.a. backpropagation). Biological networks, on the other hand, appear to lack any mechanism for sending gradients back to their input neurons, and thus cannot be learning in this way. In response to this, Scellier & Bengio (2017) proposed Equilibrium Propagation - a method for gradient-based train- ing of neural networks which uses only local learning rules and, crucially, does not rely on neurons having a mechanism for back-propagating an error gradient. Equilibrium propagation, however, has a major practical limitation: inference involves doing an iterative optimization of neural activations to find a fixed-point, and the number of steps required to closely approximate this fixed point scales poorly with the depth of the network. In response to this problem, we propose Initialized Equilibrium Propagation, which trains a feedforward network to initialize the iterative inference procedure for Equilibrium propagation. This feed-forward network learns to approximate the state of the fixed-point using a local learning rule. After training, we can simply use this initializing network for inference, resulting in a learned feedforward network. Our experiments show that this network appears to work as well or better than the original version of Equilibrium propagation. This shows how we might go about training deep networks without using backpropagation.

Wir trainieren ein Feedforward Netzwerk ohne Backpropagation, indem wir ein energiebasiertes Modell verwenden, um lokale Ziele zu liefern.

Diese Arbeit zielt darauf ab, die iterative Inferenzprozedur in energiebasierten Modellen, die mit Equilibrium Propagation (EP) trainiert werden, zu beschleunigen, indem vorgeschlagen wird, ein Feedforward Netzwerk zu trainieren, um einen Fixpunkt des "equilibrating network" vorherzusagen.

Training eines separaten Netzes zur Initialisierung von rekurrenten Netzen, die mit Gleichgewichtsvermehrung trainiert wurden.


We propose a novel generative model architecture designed to learn representations for images that factor out a single attribute from the rest of the representation. A single object may have many attributes which when altered do not change the identity of the object itself. Consider the human face; the identity of a particular person is independent of whether or not they happen to be wearing glasses. The attribute of wearing glasses can be changed without changing the identity of the person. However, the ability to manipulate and alter image attributes without altering the object identity is not a trivial task. Here, we are interested in learning a representation of the image that separates the identity of an object (such as a human face) from an attribute (such as 'wearing glasses'). We demonstrate the success of our factorization approach by using the learned representation to synthesize the same face with and without a chosen attribute. We refer to this specific synthesis process as image attribute manipulation. We further demonstrate that our model achieves competitive scores, with state of the art, on a facial attribute classification task.

Lernen von Darstellungen für Bilder, die ein einzelnes Attribut herausrechnen.

Diese Arbeit baut auf bedingten VAE GANs auf, um die Manipulation von Attributen während des Syntheseprozesses zu ermöglichen.

Diese Arbeit schlägt ein generatives Modell vor, um die Repräsentation zu erlernen, die die Identität eines Objekts von einem Attribut trennen kann, und erweitert den Autoencoder adversarial durch Hinzufügen eines Hilfsnetzwerks.


Stochastic video prediction models take in a sequence of image frames, and generate a sequence of consecutive future image frames. These models typically generate future frames in an autoregressive fashion, which is slow and requires the input and output frames to be consecutive. We introduce a model that overcomes these drawbacks by generating a latent representation from an arbitrary set of frames that can then be used to simultaneously and efficiently sample temporally consistent frames at arbitrary time-points. For example, our model can "jump" and directly sample frames at the end of the video, without sampling intermediate frames. Synthetic video evaluations confirm substantial gains in speed and functionality without loss in fidelity. We also apply our framework to a 3D scene reconstruction dataset. Here, our model is conditioned on camera location and can sample consistent sets of images for what an occluded region of a 3D scene might look like, even if there are multiple possibilities for what that region might contain. Reconstructions and videos are available at https://bit.ly/2O4Pc4R.

Wir stellen ein Modell für eine konsistente 3D-Rekonstruktion und eine sprunghafte Videovorhersage vor, d.h. die Erzeugung von Bildern in mehreren Zeitschritten in der Zukunft, ohne Zwischenbilder zu erzeugen.

In diesem Beitrag wird eine allgemeine Methode zur Modellierung indizierter Daten vorgeschlagen, bei der die Indexinformationen zusammen mit der Beobachtung in einem neuronalen Netz kodiert werden und dann die Beobachtungsbedingung anhand des Zielindexes dekodiert wird.

Es wird vorgeschlagen, eine VAE zu verwenden, die das Eingangsvideo auf eine permutationsinvariante Weise kodiert, um zukünftige Bilder eines Videos vorherzusagen.


The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn’t. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of the stochastic gradient, whereas the update magnitude is solely determined by an estimate of its relative variance. We  disentangle these two aspects and analyze them in isolation, shedding light on ADAM ’s inner workings. Transferring the "variance adaptation” to momentum- SGD gives rise to a novel method, completing the practitioner’s toolbox for problems where ADAM fails.

In der Arbeit wird versucht, Adam auf der Grundlage der Varianzanpassung mit Schwung zu verbessern, indem zwei Algorithmen vorgeschlagen werden.

Dieser Beitrag analysiert die Skaleninvarianz und die besondere Form der in Adam verwendeten Lernrate und argumentiert, dass Adams Update eine Kombination aus einem Sign-Update und einer varianzbasierten Lernrate ist.

Die Arbeit teilt den ADAM-Algorithmus in zwei Komponenten: Stochastische Richtung im Zeichen der Steigung und adaptive schrittweise mit relativer Varianz, und zwei Algorithmen werden vorgeschlagen, um jede von ihnen zu testen.


We propose a novel framework to adaptively adjust the dropout rates for the deep neural network based on a Rademacher complexity bound. The state-of-the-art deep learning algorithms impose dropout strategy to prevent feature co-adaptation. However, choosing the dropout rates remains an art of heuristics or relies on empirical grid-search over some hyperparameter space. In this work, we show the network Rademacher complexity is bounded by a function related to the dropout rate vectors and the weight coefficient matrices. Subsequently, we impose this bound as a regularizer and provide a theoretical justified way to trade-off between model complexity and representation power. Therefore, the dropout rates and the empirical loss are unified into the same objective function, which is then optimized using the block coordinate descent algorithm. We discover that the adaptively adjusted dropout rates converge to some interesting distributions that reveal meaningful patterns.Experiments on the task of image and document classification also show our method achieves better performance compared to the state-of the-art dropout algorithms.

Wir schlagen ein neues Framework vor, um die Dropout-Raten für das tiefe neuronale Netz auf der Grundlage einer Rademacher-Komplexitätsgrenze adaptiv anzupassen.

Die Autoren verbinden Dropout-Parameter mit einer Begrenzung der Rademacher-Komplexität des Netzes.

Bezieht die Komplexität der Lernfähigkeit von Netzen auf die Abbruchraten bei der Backpropagation.


Sensor fusion is a key technology that integrates various sensory inputs to allow for robust decision making in many applications such as autonomous driving and robot control. Deep neural networks have been adopted for sensor fusion in a body of recent studies. Among these, the so-called netgated architecture was proposed, which has demonstrated improved performances over the conventional convolu- tional neural networks (CNN). In this paper, we address several limitations of the baseline negated architecture by proposing two further optimized architectures: a coarser-grained gated architecture employing (feature) group-level fusion weights and a two-stage gated architectures leveraging both the group-level and feature- level fusion weights. Using driving mode prediction and human activity recogni- tion datasets, we demonstrate the significant performance improvements brought by the proposed gated architectures and also their robustness in the presence of sensor noise and failures.

Es werden optimierte Gated Deep Learning Architekturen für die Sensorfusion vorgeschlagen.

Die Autoren verbessern mehrere Einschränkungen der grundlegenden negierten Architektur, indem sie eine grobkörnigere Gated-Fusion Architektur und eine zweistufige Gated-Fusion Architektur vorschlagen.

Er schlägt zwei Gated Deep Learning Architekturen für die Sensorfusion vor und zeigt durch die gruppierten Merkmale eine verbesserte Leistung, insbesondere bei zufälligen Sensorstörungen und Ausfällen.


We develop a mean field theory for batch normalization in fully-connected feedforward neural networks. In so doing, we provide a precise characterization of signal propagation and gradient backpropagation in wide batch-normalized networks at initialization. Our theory shows that gradient signals grow exponentially in depth and that these exploding gradients cannot be eliminated by tuning the initial weight variances or by adjusting the nonlinear activation function. Indeed, batch normalization itself is the cause of gradient explosion. As a result, vanilla batch-normalized networks without skip connections are not trainable at large depths for common initialization schemes, a prediction that we verify with a variety of empirical simulations. While gradient explosion cannot be eliminated, it can be reduced by tuning the network close to the linear regime, which improves the trainability of deep batch-normalized networks without residual connections. Finally, we investigate the learning dynamics of batch-normalized networks and observe that after a single step of optimization the networks achieve a relatively stable equilibrium in which gradients have dramatically smaller dynamic range. Our theory leverages Laplace, Fourier, and Gegenbauer transforms and we derive new identities that may be of independent interest.

Batch-Normalisierung verursacht explodierende Gradienten in einfachen Feedforward Netzen.

Entwickelt eine Mean Field Theorie für Batch Normalisierung (BN) in vollständig verbundenen Netzwerken mit zufällig initialisierten Gewichten.

Bietet eine dynamische Perspektive auf tiefe neuronale Netze unter Verwendung der Entwicklung der Kovarianzmatrix zusammen mit den Schichten.


We present NeuroSAT, a message passing neural network that learns to solve SAT problems after only being trained as a classifier to predict satisfiability.   Although it is not competitive with state-of-the-art SAT solvers, NeuroSAT can solve problems that are substantially larger and more difficult than it ever saw during training by simply running for more iterations. Moreover, NeuroSAT generalizes to novel distributions; after training only on random SAT problems, at test time it can solve SAT problems encoding graph coloring, clique detection, dominating set, and vertex cover problems, all on a range of distributions over small random graphs.

Wir trainieren ein Graphennetz zur Vorhersage boolescher Erfüllbarkeit und zeigen, dass es lernt, nach Lösungen zu suchen, und dass die gefundenen Lösungen aus seinen Aktivierungen entschlüsselt werden können.

Die Arbeit beschreibt eine allgemeine neuronale Netzarchitektur zur Vorhersage der Erfüllbarkeit.

In diesem Beitrag wird die NeuroSAT-Architektur vorgestellt, die ein tiefes neuronales Netz zur Vorhersage der Erfüllbarkeit von CNF-Instanzen verwendet.


Spatiotemporal forecasting has various applications in neuroscience, climate and transportation domain. Traffic forecasting is one canonical example of such learning task. The task is challenging due to (1) complex spatial dependency on road networks, (2) non-linear temporal dynamics with changing road conditions and (3) inherent difficulty of long-term forecasting. To address these challenges, we propose to model the traffic flow as a diffusion process on a directed graph and introduce Diffusion Convolutional Recurrent Neural Network (DCRNN), a deep learning framework for traffic forecasting that incorporates both spatial and temporal dependency in the traffic flow. Specifically, DCRNN captures the spatial dependency using bidirectional random walks on the graph, and the temporal dependency using the encoder-decoder architecture with scheduled sampling. We evaluate the framework on two real-world large-scale road network traffic datasets and observe consistent improvement of 12% - 15% over state-of-the-art baselines

Ein neuronales Sequenzmodell, das lernt, Prognosen auf einem gerichteten Graphen zu erstellen.

Die Arbeit schlägt die Diffusion Convolutional Recurrent Neural Network Architektur für das räumlich-zeitliche Verkehrsprognose Problem vor.

Es wird vorgeschlagen, ein Verkehrsprognosemodell unter Verwendung eines Diffusionsprozesses für Convolutional Recurrent Neural Networks zu erstellen, um die saptio-temporale Autokorrelation zu berücksichtigen.


Obtaining reliable uncertainty estimates of neural network predictions is a long standing challenge. Bayesian neural networks have been proposed as a solution, but it remains open how to specify their prior. In particular, the common practice of a standard normal prior in weight space imposes only weak regularities, causing the function posterior to possibly generalize in unforeseen ways on inputs outside of the training distribution. We propose noise contrastive priors (NCPs) to obtain reliable uncertainty estimates. The key idea is to train the model to output high uncertainty for data points outside of the training distribution. NCPs do so using an input prior, which adds noise to the inputs of the current mini batch, and an output prior, which is a wide distribution given these inputs. NCPs are compatible with any model that can output uncertainty estimates, are easy to scale, and yield reliable uncertainty estimates throughout training. Empirically, we show that NCPs prevent overfitting outside of the training distribution and result in uncertainty estimates that are useful for active learning. We demonstrate the scalability of our method on the flight delays data set, where we significantly improve upon previously published results.

Wir trainieren neuronale Netze so, dass sie bei verrauschten Eingaben unsicher sind, um übermäßige Vorhersagen außerhalb der Trainingsverteilung zu vermeiden.

Es wird ein Ansatz zur Ermittlung von Unsicherheitsschätzungen für Vorhersagen in neuronalen Netzen vorgestellt, der eine gute Leistung bei der Quantifizierung der Vorhersageunsicherheit an Punkten außerhalb der Trainingsverteilung aufweist.

Die Arbeit befasst sich mit dem Problem der Unsicherheitsabschätzung von neuronalen Netzen und schlägt vor, einen Bayes'schen Ansatz mit einem kontrastiven Prior zu verwenden.


Convolutional neural networks (CNNs) were inspired by human vision and, in some settings, achieve a performance comparable to human object recognition. This has lead to the speculation that both systems use similar mechanisms to perform recognition. In this study, we conducted a series of simulations that indicate that there is a fundamental difference between human vision and CNNs: while object recognition in humans relies on analysing shape, CNNs do not have such a shape-bias. We teased apart the type of features selected by the model by modifying the CIFAR-10 dataset so that, in addition to containing objects with shape, the images concurrently contained non-shape features, such as a noise-like mask. When trained on these modified set of images, the model did not show any bias towards selecting shapes as features. Instead it relied on whichever feature allowed it to perform the best prediction -- even when this feature was a noise-like mask or a single predictive pixel amongst 50176 pixels. We also found that regularisation methods, such as batch normalisation or Dropout, did not change this behaviour and neither did past or concurrent experience with images from other datasets.

Diese Studie hebt einen wesentlichen Unterschied zwischen dem menschlichen Sehen und CNNs hervor: Während die Objekterkennung beim Menschen auf der Analyse der Form beruht, haben CNNs keine solche Formvorliebe.

Versucht anhand einer Reihe von gut konzipierten Experimenten nachzuweisen, dass CNNs, die für die Bildklassifizierung trainiert wurden, keine Formverzerrungen wie das menschliche Sehen kodieren.

In diesem Beitrag wird die Tatsache hervorgehoben, dass CNNs nicht unbedingt lernen, Objekte anhand ihrer Form zu erkennen, und dass sie bei auf Rauschen basierenden Merkmalen überreagieren.


The development of high-dimensional generative models has recently gained a great surge of interest with the introduction of variational auto-encoders and generative adversarial neural networks. Different variants have been proposed where the underlying latent space is structured, for example, based on attributes describing the data to generate. We focus on a particular problem where one aims at generating samples corresponding to a number of objects under various views. We assume that the distribution of the data is driven by two independent latent factors: the content, which represents the intrinsic features of an object, and the view, which stands for the settings of a particular observation of that object. Therefore, we propose a generative model and a conditional variant built on such a disentangled latent space. This approach allows us to generate realistic samples corresponding to various objects in a high variety of views. Unlike many multi-view approaches, our model doesn't need any supervision on the views but only on the content. Compared to other conditional generation approaches that are mostly based on binary or categorical attributes, we make no such assumption about the factors of variations. Our model can be used on problems with a huge, potentially infinite, number of categories. We experiment it on four images datasets on which we demonstrate the effectiveness of the model and its ability to generalize.

Wir beschreiben ein neuartiges generatives Modell mit mehreren Ansichten, das mehrere Ansichten desselben Objekts oder mehrere Objekte in derselben Ansicht generieren kann, ohne dass eine Kennzeichnung der Ansichten erforderlich ist.

In diesem Beitrag wird eine GAN basierte Methode zur Bilderzeugung vorgestellt, die versucht, latente Variablen, die den Bildinhalt beschreiben, von denen zu trennen, die die Eigenschaften der Ansicht beschreiben.

In diesem Beitrag wird eine GAN Architektur vorgeschlagen, die darauf abzielt, die zugrunde liegende Verteilung einer bestimmten Klasse in "Inhalt" und "Ansicht" zu zerlegen.

Es wird ein neues generatives Modell auf der Grundlage des Generative Adversarial Network (GAN) vorgeschlagen, das den Inhalt und die Ansicht von Objekten ohne Überwachung der Ansicht entkoppelt und GMV zu einem bedingten generativen Modell erweitert, das ein Eingabebild nimmt und verschiedene Ansichten des Objekts im Eingabebild erzeugt.


The huge size of deep networks hinders their use in small computing devices. In this paper, we consider compressing the network by weight quantization. We extend a recently proposed loss-aware weight binarization scheme to ternarization, with possibly different scaling parameters for the positive and negative weights, and m-bit (where m > 2) quantization. Experiments on feedforward and recurrent neural networks show that the proposed scheme outperforms state-of-the-art weight quantization algorithms, and is as accurate (or even more accurate) than the full-precision network.

Es wird ein verlustsensitiver Gewichtsquantisierungsalgorithmus vorgeschlagen, der seine Auswirkungen auf den Verlust direkt berücksichtigt.

Schlägt eine Methode zur Komprimierung von Netzen durch Gewichtsternarisierung vor.

Die Arbeit schlägt eine neue Methode vor, DNNs mit quantisierten Gewichten zu trainieren, indem die Quantisierung als Einschränkung in einen proximalen Quasi-Newton-Algorithmus aufgenommen wird, der gleichzeitig eine Skalierung für die quantisierten Werte lernt.

Die Arbeit erweitert das verlustbewusste Gewichts-Binarisierungsschema auf Terarisierung und beliebige m-Bit Quantisierung und demonstriert eine vielversprechende Leistung.


In the pursuit of increasingly intelligent learning systems, abstraction plays a vital role in enabling sophisticated decisions to be made in complex environments. The options framework provides formalism for such abstraction over sequences of decisions.   However most models require that options be given a priori, presumably specified by hand, which is neither efficient, nor scalable. Indeed, it is preferable to learn options directly from interaction with the environment. Despite several efforts, this remains a difficult problem: many approaches require access to a model of the environmental dynamics, and inferred options are often not interpretable, which limits our ability to explain the system behavior for verification or debugging purposes.   In this work we develop a novel policy gradient method for the automatic learning of policies with options.   This algorithm uses inference methods to simultaneously improve all of the options available to an agent, and thus can be employed in an off-policy manner, without observing option labels. Experimental results show that the options learned can be interpreted. Further, we find that the method presented here is more sample efficient than existing methods, leading to faster and more stable learning of policies with options.

Wir entwickeln eine neuartige Gradientenmethode für das automatische Lernen von Regeln mit Optionen unter Verwendung eines differenzierbaren Inferenzschritts.

In der Arbeit wird eine neue Gradiententechnik für das Lernen von Optionen vorgestellt, bei der eine einzige Stichprobe zur Aktualisierung aller Optionen verwendet werden kann.

Schlägt eine Methode zum Lernen von Optionen bei komplexen kontinuierlichen Problemen vor.


The paper, interested in unsupervised feature selection, aims to retain the features best accounting for the local patterns in the data. The proposed approach, called Locally Linear Unsupervised Feature Selection, relies on a dimensionality reduction method to characterize such patterns; each feature is thereafter assessed according to its compliance w.r.t. the local patterns, taking inspiration from Locally Linear Embedding (Roweis and Saul, 2000). The experimental validation of the approach on the scikit-feature benchmark suite demonstrates its effectiveness compared to the state of the art.

Unüberwachte Merkmalsauswahl durch Erfassung der lokalen linearen Struktur von Daten.

Schlägt eine lokal lineare unüberwachte Merkmalsauswahl vor.

In dem Papier wird die LLUFS-Methode für die Merkmalsauswahl vorgeschlagen.


Humans can understand and produce new utterances effortlessly, thanks to their systematic compositional skills. Once a person learns the meaning of a new verb "dax," he or she can immediately understand the meaning of "dax twice" or "sing and dax." In this paper, we introduce the SCAN domain, consisting of a set of simple compositional navigation commands paired with the corresponding action sequences. We then test the zero-shot generalization capabilities of a variety of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence methods. We find that RNNs can generalize well when the differences between training and test commands are small, so that they can apply "mix-and-match" strategies to solve the task. However, when generalization requires systematic compositional skills (as in the "dax" example above), RNNs fail spectacularly. We conclude with a proof-of-concept experiment in neural machine translation, supporting the conjecture that lack of systematicity is an important factor explaining why neural networks need very large training sets.

Anhand einer einfachen sprachgesteuerten Navigationsaufgabe untersuchen wir die kompositorischen Fähigkeiten moderner rekurrenter seq2seq-Netzwerke.

Dieser Beitrag konzentriert sich auf die kompositorischen Fähigkeiten moderner Sequenz-zu-Sequenz RNNs und zeigt die Schwächen der aktuellen seq2seq RNN-Architekturen auf.

Die Arbeit analysiert die Komposition Fähigkeiten von RNNs, insbesondere die Verallgemeinerung Fähigkeit der RNNs auf zufällige Teilmenge von SCAN-Befehle, auf längere SCAN-Befehle, und der Zusammensetzung über primitive Befehle.

Die Autoren stellen einen neuen Datensatz vor, der die Analyse eines Seq2Seq Lernfalls erleichtert.


This paper addresses the challenging problem of retrieval and matching of graph structured objects, and makes two key contributions. First, we demonstrate how  Graph Neural Networks (GNN), which have emerged as an effective model for various supervised prediction problems defined on structured data, can be trained to produce embedding of graphs in vector spaces that enables efficient similarity reasoning. Second, we propose a novel Graph Matching Network model that, given a pair of graphs as input, computes a similarity score between them by jointly reasoning on the pair through a new cross-graph attention-based matching mechanism. We demonstrate the effectiveness of our models on different domains including the challenging problem of control-flow-graph based function similarity search that plays an important role in the detection of vulnerabilities in software systems. The experimental analysis demonstrates that our models are not only able to exploit structure in the context of similarity learning but they can also outperform domain-specific baseline systems that have been carefully hand-engineered for these problems.

Wir befassen uns mit dem Problem des Ähnlichkeitslernens für strukturierte Objekte mit Anwendungen insbesondere im Bereich der Computersicherheit und schlagen ein neues Modell für Graph-Matching Netzwerke vor, das sich bei dieser Aufgabe auszeichnet.

Die Autoren stellen ein Graph-Matching Netzwerk für die Wiederauffindung und das Matching von graphisch strukturierten Objekten vor.

Die Autoren gehen das Problem des Graphenabgleichs an, indem sie eine Erweiterung der Grapheneinbettungsnetze vorschlagen.

Die Autoren stellen zwei Methoden zum Erlernen einer Ähnlichkeitsbewertung zwischen Graphenpaaren vor und zeigen die Vorteile der Einführung von Ideen aus dem Graphenabgleich in neuronale Netze.


Context information plays an important role in human language understanding, and it is also useful for machines to learn vector representations of language. In this paper, we explore an asymmetric encoder-decoder structure for unsupervised context-based sentence representation learning. As a result, we build an encoder-decoder architecture with an RNN encoder and a CNN decoder, and we show that neither an autoregressive decoder nor an RNN decoder is required.   We further combine a suite of effective designs to significantly improve model efficiency while also achieving better performance. Our model is trained on two different large unlabeled corpora, and in both cases transferability is evaluated on a set of downstream language understanding tasks. We empirically show that our model is simple and fast while producing rich sentence representations that excel in downstream tasks.

Wir haben ein RNN-CNN Encoder-Decoder Modell für schnelles unbeaufsichtigtes Lernen von Satzrepräsentationen vorgeschlagen.

Modifikationen des Skip-thought Frameworks für das Lernen von Satzeinbettungen.

In diesem Beitrag wird ein neues hybrides Design für RNN-Encoder und CNN-Decoder vorgestellt, das beim Pretraining von Encodern keinen autoregressiven Decoder benötigt.

Die Autoren erweitern Skip-thought, indem sie nur einen Zielsatz mit einem CNN-Decoder dekodieren.


Building on the success of deep learning, two modern approaches to learn a probability model of the observed data are Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs). VAEs consider an explicit probability model for the data and compute a generative distribution by maximizing a variational lower-bound on the log-likelihood function. GANs, however, compute a generative model by minimizing a distance between observed and generated probability distributions without considering an explicit model for the observed data. The lack of having explicit probability models in GANs prohibits computation of sample likelihoods in their frameworks and limits their use in statistical inference problems. In this work, we show that an optimal transport GAN with the entropy regularization can be viewed as a generative model that maximizes a lower-bound on average sample likelihoods, an approach that VAEs are based on. In particular, our proof constructs an explicit probability model for GANs that can be used to compute likelihood statistics within GAN's framework. Our numerical results on several datasets demonstrate consistent trends with the proposed theory.

Ein statistischer Ansatz zur Berechnung von Beispielwahrscheinlichkeiten in generativen adversarial Netzen.

Zeigen Sie, dass WGAN mit entropischer Regularisierung eine untere Schranke für die Wahrscheinlichkeit der beobachteten Datenverteilung maximiert.

Die Autoren behaupten, dass es möglich ist, die obere Schranke eines entropie-regulierten optimalen Transports zu nutzen, um ein Maß für die "Stichprobenwahrscheinlichkeit" zu finden.


We introduce geomstats, a Python package for Riemannian modelization and optimization over manifolds such as hyperspheres, hyperbolic spaces, SPD matrices or Lie groups of transformations. Our contribution is threefold. First, geomstats allows the flexible modeling of many a machine learning problem through an efficient and extensively unit-tested implementations of these manifolds, as well as the set of useful Riemannian metrics, exponential and logarithm maps that we provide. Moreover, the wide choice of loss functions and our implementation of the corresponding gradients allow fast and easy optimization over manifolds. Finally, geomstats is the only package to provide a unified framework for Riemannian geometry, as the operations implemented in geomstats are available with different computing backends (numpy,tensorflow and keras), as well as with a GPU-enabled mode–-thus considerably facilitating the application of Riemannian geometry in machine learning. In this paper, we present geomstats through a review of the utility and advantages of manifolds in machine learning, using the concrete examples that they span to show the efficiency and practicality of their implementation using our package

Wir stellen geomstats vor, ein effizientes Python-Paket für Riemannsche Modellierung und Optimierung über Mannigfaltigkeiten, das sowohl mit numpy als auch mit tensorflow kompatibel ist.

Die Arbeit stellt das Softwarepaket geomstats vor, das die einfache Nutzung von Riemannschen Mannigfaltigkeiten und Metriken in maschinellen Lernmodellen ermöglicht.

Schlägt ein Python-Paket für Optimierung und Anwendungen auf Riemannschen Mannigfaltigkeiten vor und hebt die Unterschiede zwischen dem Geomstats-Paket und anderen Paketen hervor.

Stellt eine geometrische Toolbox, Geomstats, für maschinelles Lernen auf Riemannschen Mannigfaltigkeiten vor.


We propose to execute deep neural networks (DNNs) with dynamic and sparse graph (DSG) structure for compressive memory and accelerative execution during both training and inference. The great success of DNNs motivates the pursuing of lightweight models for the deployment onto embedded devices. However, most of the previous studies optimize for inference while neglect training or even complicate it. Training is far more intractable, since (i) the neurons dominate the memory cost rather than the weights in inference; (ii) the dynamic activation makes previous sparse acceleration via one-off optimization on fixed weight invalid; (iii) batch normalization (BN) is critical for maintaining accuracy while its activation reorganization damages the sparsity. To address these issues, DSG activates only a small amount of neurons with high selectivity at each iteration via a dimensionreduction search and obtains the BN compatibility via a double-mask selection. Experiments show significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x) with little accuracy loss on various benchmarks.

Wir konstruieren einen dynamischen spärlichen Graphen mittels Dimensionsreduktionssuche, um die Rechen- und Speicherkosten sowohl beim DNN-Training als auch bei der Inferenz zu reduzieren.

Die Autoren schlagen vor, einen dynamischen, spärlichen Berechnungsgraphen zu verwenden, um die Speicher- und Zeitkosten in tiefen neuronalen Netzen (DNN) zu reduzieren.

In diesem Beitrag wird eine Methode zur Beschleunigung des Trainings und der Inferenz von tiefen neuronalen Netzen durch dynamisches Pruning des Berechnungsgraphen vorgeschlagen.


Efficient exploration remains a major challenge for reinforcement learning. One reason is that the variability of the returns often depends on the current state and action, and is therefore heteroscedastic. Classical exploration strategies such as upper confidence bound algorithms and Thompson sampling fail to appropriately account for heteroscedasticity, even in the bandit setting. Motivated by recent findings that address this issue in bandits, we propose to use Information-Directed Sampling (IDS) for exploration in reinforcement learning. As our main contribution, we build on recent advances in distributional reinforcement learning and propose a novel, tractable approximation of IDS for deep Q-learning. The resulting exploration strategy explicitly accounts for both parametric uncertainty and heteroscedastic observation noise. We evaluate our method on Atari games and demonstrate a significant improvement over alternative approaches.

Wir entwickeln eine praktische Erweiterung des Information-Directed Sampling für Reinforcement Learning, die parametrische Unsicherheit und Heteroskedastizität in der Renditeverteilung für Exploration berücksichtigt.

Die Autoren schlagen einen Weg vor, Information-Directed Sampling auf Reinforcement Learning zu erweitern, indem sie zwei Arten von Unsicherheiten kombinieren, um eine einfache, auf IDS basierende Explorationsstrategie zu erhalten.

Diese Arbeit untersucht sophistische Explorationsansätze für Reinforcement Learning, die auf Information Direct Sampling und auf Distributional Reinforcement Learning aufbauen.


We address the problem of learning structured policies for continuous control. In traditional reinforcement learning, policies of agents are learned by MLPs which take the concatenation of all observations from the environment as input for predicting actions. In this work, we propose NerveNet to explicitly model the structure of an agent, which naturally takes the form of a graph. Specifically, serving as the agent's policy network, NerveNet first propagates information over the structure of the agent and then predict actions for different parts of the agent. In the experiments, we first show that our NerveNet is comparable to state-of-the-art methods on standard MuJoCo environments. We further propose our customized reinforcement learning environments for benchmarking two types of structure transfer learning tasks, i.e., size and disability transfer. We demonstrate that policies learned by NerveNet are significantly better than policies learned by other models and are able to transfer even in a zero-shot setting.

Verwendung eines neuronalen Graphen Netzes zur Modellierung struktureller Informationen der Agenten, um die Politik und die Übertragbarkeit zu verbessern.

Eine Methode zur Darstellung und zum Erlernen strukturierter Strategien für kontinuierliche Steuerungsaufgaben unter Verwendung neuronaler Graphennetze.

In der Vorlage wird vorgeschlagen, zusätzliche Strukturen in Probleme des Reinforcement Learnings einzubeziehen, insbesondere die Struktur der Morphologie des Agenten.

Vorschlag für eine Anwendung graph neuronaler Netze zum Erlernen von Strategien zur Steuerung von "Tausendfüßler"-Robotern unterschiedlicher Länge.


Real-world tasks are often highly structured. Hierarchical reinforcement learning (HRL) has attracted research interest as an approach for leveraging the hierarchical structure of a given task in reinforcement learning (RL). However, identifying the hierarchical policy structure that enhances the performance of RL is not a trivial task. In this paper, we propose an HRL method that learns a latent variable of a hierarchical policy using mutual information maximization. Our approach can be interpreted as a way to learn a discrete and latent representation of the state-action space. To learn option policies that correspond to modes of the advantage function, we introduce advantage-weighted importance sampling.
 In our HRL method, the gating policy learns to select option policies based on an option-value function, and these option policies are optimized based on the deterministic policy gradient method. This framework is derived by leveraging the analogy between a monolithic policy in standard RL and a hierarchical policy in HRL by using a deterministic option policy.   Experimental results indicate that our HRL approach can learn a diversity of options and that it can enhance the performance of RL in continuous control tasks.

In diesem Beitrag wird ein hierarchischer Rahmen für das Reinforcement Learning vorgestellt, der auf deterministischen Optionsstrategien und der Maximierung der gegenseitigen Information basiert.

Schlägt einen HRL-Algorithmus vor, der versucht, Optionen zu lernen, die ihre gegenseitige Information mit der Zustands-Aktions Dichte unter optimalen Regeln maximieren.

In diesem Beitrag wird ein HRL-System vorgeschlagen, bei dem die wechselseitige Information der latenten Variablen und der Zustands-Aktions-Paare annähernd maximiert wird.

Schlägt ein Kriterium vor, das darauf abzielt, die gegenseitige Information zwischen Optionen und Zustands-Aktions Paaren zu maximieren, und zeigt empirisch, dass die gelernten Optionen den Zustands-Aktions Raum zerlegen, nicht aber den Zustandsraum.


Deep neural networks (DNNs) have achieved impressive predictive performance due to their ability to learn complex, non-linear relationships between variables. However, the inability to effectively visualize these relationships has led to DNNs being characterized as black boxes and consequently limited their applications. To ameliorate this problem, we introduce the use of hierarchical interpretations to explain DNN predictions through our proposed method: agglomerative contextual decomposition (ACD). Given a prediction from a trained DNN, ACD produces a hierarchical clustering of the input features, along with the contribution of each cluster to the final prediction. This hierarchy is optimized to identify clusters of features that the DNN learned are predictive. We introduce ACD using examples from Stanford Sentiment Treebank and ImageNet, in order to diagnose incorrect predictions, identify dataset bias, and extract polarizing phrases of varying lengths. Through human experiments, we demonstrate that ACD enables users both to identify the more accurate of two DNNs and to better trust a DNN's outputs. We also find that ACD's hierarchy is largely robust to adversarial perturbations, implying that it captures fundamental aspects of the input and ignores spurious noise.

Wir führen hierarchische lokale Interpretationen ein und validieren sie. Dies ist die erste Technik, die automatisch nach wichtigen Interaktionen für individuelle Vorhersagen von LSTMs und CNNs sucht und diese anzeigt.

Ein neuartiger Ansatz zur Erklärung der Vorhersagen neuronaler Netze durch das Erlernen hierarchischer Darstellungen von Gruppen von Eingangsmerkmalen und deren Beitrag zur endgültigen Vorhersage.

Erweitert eine bestehende Merkmalsinterpretationsmethode für LSTMs auf allgemeinere DNNs und führt ein hierarchisches Clustering der Eingangsmerkmale und die Beiträge jedes Clusters zur endgültigen Vorhersage ein.

In diesem Beitrag wird eine hierarchische Erweiterung der kontextuellen Dekomposition vorgeschlagen.


Principal Filter Analysis (PFA) is an easy to implement, yet effective method for neural network compression. PFA exploits the intrinsic correlation between filter responses within network layers to recommend a smaller network footprint. We propose two compression algorithms: the first allows a user to specify the proportion of the original spectral energy that should be preserved in each layer after compression, while the second is a heuristic that leads to a parameter-free approach that automatically selects the compression used at each layer. Both algorithms are evaluated against several architectures and datasets, and we show considerable compression rates without compromising accuracy, e.g., for VGG-16 on CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x, and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively. In our tests we also demonstrate that networks compressed with PFA achieve an accuracy that is very close to the empirical upper bound for a given compression ratio. Finally, we show how PFA is an effective tool for simultaneous compression and domain adaptation.

Wir schlagen eine einfach zu implementierende, aber effektive Methode zur Komprimierung neuronaler Netze vor. PFA nutzt die intrinsische Korrelation zwischen den Filterantworten innerhalb der Netzwerkschichten, um einen kleineren Netzwerkfußabdruck zu empfehlen.

Schlägt vor, Convolutional Networks durch Analyse der beobachteten Korrelation zwischen den Filtern einer Schicht zu Prunen, ausgedrückt durch das Eigenwertspektrum ihrer Kovarianzmatrix.

In diesem Beitrag wird ein Ansatz zur Komprimierung neuronaler Netze vorgestellt, bei dem die Korrelation der Filterantworten in jeder Schicht durch zwei Strategien berücksichtigt wird.

In diesem Papier wird eine auf der Spektralanalyse basierende Komprimierungsmethode vorgeschlagen.


We propose a method to efficiently learn diverse strategies in reinforcement learning for query reformulation in the tasks of document retrieval and question answering. In the proposed framework an agent consists of multiple specialized sub-agents and a meta-agent that learns to aggregate the answers from sub-agents to produce a final answer. Sub-agents are trained on disjoint partitions of the training data, while the meta-agent is trained on the full training set. Our method makes learning faster, because it is highly parallelizable, and has better generalization performance than strong baselines, such as an ensemble of agents trained on the full data. We show that the improved performance is due to the increased diversity of reformulation strategies.

Mehrere verschiedene Agenten für die Umformulierung von Suchanfragen, die mit Reinforcement Learning trainiert wurden, um Suchmaschinen zu verbessern.

Parellelisierung der Ensemble-Methode beim Reinforcement Learning für die Umformulierung von Anfragen, Beschleunigung des Trainings und Verbesserung der Vielfalt der gelernten Formulierungen.

Die Autoren schlagen vor, mehrere verschiedene Agenten zu trainieren, jeder mit einer anderen Teilmenge der Trainingsmenge.

Die Autoren schlagen einen Ensemble-Ansatz für die Reformulierung von Anfragen vor.


Network Embeddings (NEs) map the nodes of a given network into $d$-dimensional Euclidean space $\mathbb{R}^d$. Ideally, this mapping is such that 'similar' nodes are mapped onto nearby points, such that the NE can be used for purposes such as link prediction (if 'similar' means being 'more likely to be connected') or classification (if 'similar' means 'being more likely to have the same label'). In recent years various methods for NE have been introduced, all following a similar strategy: defining a notion of similarity between nodes (typically some distance measure within the network), a distance measure in the embedding space, and a loss function that penalizes large distances for similar nodes and small distances for dissimilar nodes.

 A difficulty faced by existing methods is that certain networks are fundamentally hard to embed due to their structural properties: (approximate) multipartiteness, certain degree distributions, assortativity, etc. To overcome this, we introduce a conceptual innovation to the NE literature and propose to create \emph{Conditional Network Embeddings} (CNEs); embeddings that maximally add information with respect to given structural properties (e.g. node degrees, block densities, etc.). We use a simple Bayesian approach to achieve this, and propose a block stochastic gradient descent algorithm for fitting it efficiently.

 We demonstrate that CNEs are superior for link prediction and multi-label classification when compared to state-of-the-art methods, and this without adding significant mathematical or computational complexity. Finally, we illustrate the potential of CNE for network visualization.

Wir stellen eine Methode zur Einbettung von Netzwerken vor, die Vorabinformationen über das Netzwerk berücksichtigt und dadurch eine bessere empirische Leistung erzielt.

Die Arbeit schlägt vor, eine Prioritätsverteilung zu verwenden, um die Netzwerkeinbettung einzuschränken. Für die Formulierung dieser Arbeit wurden sehr eingeschränkte Gaußsche Verteilungen verwendet.

Schlägt vor, unbeaufsichtigte Knoteneinbettungen zu lernen, indem die strukturellen Eigenschaften von Netzwerken berücksichtigt werden.


This paper studies a class of adaptive gradient based momentum algorithms that update the  search directions and learning rates simultaneously using past gradients. This class, which we refer to as the ''``Adam-type'', includes the popular algorithms such as Adam, AMSGrad, AdaGrad. Despite their popularity in training deep neural networks (DNNs), the convergence of these algorithms for solving  non-convex problems remains an open question. In this paper, we develop an analysis framework and a set of mild sufficient conditions that guarantee the convergence of the Adam-type methods, with a convergence rate of order   $O(\log{T}/\sqrt{T})$ for non-convex stochastic optimization. Our convergence analysis applies to a new algorithm called AdaFom (AdaGrad with First Order Momentum). We show that the conditions are essential, by identifying concrete examples in which violating the conditions makes an algorithm diverge. Besides providing one of the first comprehensive analysis for Adam-type methods in the non-convex setting, our results can also help the practitioners to easily  monitor the progress of algorithms and determine their convergence behavior.

Wir analysieren die Konvergenz von Algorithmen des Adam-Typs und stellen milde hinreichende Bedingungen zur Verfügung, um ihre Konvergenz zu garantieren. Wir zeigen auch, dass eine Verletzung der Bedingungen dazu führen kann, dass ein Algorithmus divergiert.

Präsentiert eine Konvergenzanalyse im nicht-konvexen Umfeld für eine Familie von Optimierungsalgorithmen.

In diesem Beitrag wird die Konvergenzbedingung von Optimierern des Adam-Typs bei ungebundenen nicht-konvexen Optimierungsproblemen untersucht.


This research paper describes a simplistic architecture named as AANN: Absolute Artificial Neural Network, which can be used to create highly interpretable representations of the input data. These representations are generated by penalizing the learning of the network in such a way that those learned representations correspond to the respective labels present in the labelled dataset used for supervised training; thereby, simultaneously giving the network the ability to classify the input data. The network can be used in the reverse direction to generate data that closely resembles the input by feeding in representation vectors as required. This research paper also explores the use of mathematical abs (absolute valued) functions as activation functions which constitutes the core part of this neural network architecture. Finally the results obtained on the MNIST dataset by using this technique are presented and discussed in brief.

Auto-Encoder mit gebundenen Gewichten und abs-Funktion als Aktivierungsfunktion, lernt die Klassifizierung in Vorwärtsrichtung und die Regression in Rückwärtsrichtung aufgrund einer speziell definierten Kostenfunktion.

In dem Beitrag wird die Verwendung der Absolutwert-Aktivierungsfunktion in einer Autoencoder-Architektur mit einem zusätzlichen überwachten Lernterm in der Zielfunktion vorgeschlagen.

In dieser Arbeit wird ein reversibles Netz mit dem absoluten Wert als Aktivierungsfunktion eingeführt.


Current state-of-the-art relation extraction methods typically rely on a set of lexical, syntactic, and semantic features, explicitly computed in a pre-processing step. Training feature extraction models requires additional annotated language resources, which severely restricts the applicability and portability of relation extraction to novel languages. Similarly, pre-processing introduces an additional source of error. To address these limitations, we introduce TRE, a Transformer for Relation Extraction, extending the OpenAI Generative Pre-trained Transformer [Radford et al., 2018]. Unlike previous relation extraction models, TRE uses pre-trained deep language representations instead of explicit linguistic features to inform the relation classification and combines it with the self-attentive Transformer architecture to effectively model long-range dependencies between entity mentions. TRE allows us to learn implicit linguistic features solely from plain text corpora by unsupervised pre-training, before fine-tuning the learned language representations on the relation extraction task. TRE obtains a new state-of-the-art result on the TACRED and SemEval 2010 Task 8 datasets, achieving a test F1 of 67.4 and 87.1, respectively. Furthermore, we observe a significant increase in sample efficiency. With only 20% of the training examples, TRE matches the performance of our baselines and our model trained from scratch on 100% of the TACRED dataset. We open-source our trained models, experiments, and source code.

Wir schlagen ein auf Transformer basierendes Modell zur Extraktion von Beziehungen vor, das anstelle von expliziten linguistischen Merkmalen vortrainierte Sprachrepräsentationen verwendet.

Es wird ein Transformer basiertes Modell zur Extraktion von Beziehungen vorgestellt, das ein Vortraining auf unbeschriftetem Text mit einem Sprachmodellierungsziel nutzt.

Dieser Artikel beschreibt eine neuartige Anwendung von Transformer-Netzwerken zur Extraktion von Beziehungen.

Die Arbeit stellt eine auf Transformer basierende Architektur zur Ausspannungsextraktion vor, die an zwei Datensätzen evaluiert wurde.


Neural networks have recently had a lot of success for many tasks. However, neural
 network architectures that perform well are still typically designed manually
 by experts in a cumbersome trial-and-error process. We propose a new method
 to automatically search for well-performing CNN architectures based on a simple
 hill climbing procedure whose operators apply network morphisms, followed
 by short optimization runs by cosine annealing. Surprisingly, this simple method
 yields competitive results, despite only requiring resources in the same order of
 magnitude as training a single network. E.g., on CIFAR-10, our method designs
 and trains networks with an error rate below 6% in only 12 hours on a single GPU;
 training for one day reduces this error further, to almost 5%.

Wir schlagen ein einfaches und effizientes Verfahren zur Architektursuche für Convolutional Neural Networks vor.

Es wird eine Suchmethode für neuronale Architekturen vorgeschlagen, die bei CIFAR10 eine Genauigkeit erreicht, die dem neuesten Stand der Technik entspricht, und die viel weniger Rechenressourcen benötigt.

Es wird eine Methode zur Suche von Architekturen für neuronale Netze zur gleichen Zeit wie das Training vorgestellt, die eine erhebliche Zeitersparnis beim Training und bei der Architektursuche mit sich bringt.

Schlägt eine Variante der neuronalen Architektursuche unter Verwendung von Netzwerkmorphismen vor, um einen Suchraum unter Verwendung von CNN-Architekturen zu definieren, der die CIFAR-Bildklassifizierungsaufgabe erfüllt.


We propose GraphGAN - the first implicit generative model for graphs that enables to mimic real-world networks.
 We pose the problem of graph generation as learning the distribution of biased random walks over a single input graph.
 Our model is based on a stochastic neural network that generates discrete output samples, and is trained using the Wasserstein GAN objective. GraphGAN enables us to generate sibling graphs, which have similar properties yet are not exact replicas of the original graph. Moreover, GraphGAN learns a semantic mapping from the latent input space to the generated graph's properties. We discover that sampling from certain regions of the latent space leads to varying properties of the output graphs, with smooth transitions between them. Strong generalization properties of GraphGAN are highlighted by its competitive performance in link prediction as well as promising results on node classification, even though not specifically trained for these tasks.

Verwendung von GANs zur Erzeugung von Graphen über Random Walks.

Die Autoren schlagen ein generatives Modell von Random Walks auf Graphen vor, das modellagnostisches Lernen, kontrollierbare Anpassung und die Erzeugung von Ensemble-Graphen ermöglicht.

Schlägt eine WGAN-Formulierung zur Erzeugung von Graphen auf der Grundlage von Random Walks unter Verwendung von Knoteneinbettungen und einer LSTM-Architektur zur Modellierung vor.


The ability of a classifier to recognize unknown inputs is important for many classification-based systems. We discuss the problem of simultaneous classification and novelty detection, i.e. determining whether an input is from the known set of classes and from which specific class, or from an unknown domain and does not belong to any of the known classes. We propose a method based on the Generative Adversarial Networks (GAN) framework. We show that a multi-class discriminator trained with a generator that generates samples from a mixture of nominal and novel data distributions is the optimal novelty detector. We approximate that generator with a mixture generator trained with the Feature Matching loss and empirically show that the proposed method outperforms conventional methods for novelty detection. Our findings demonstrate a simple, yet powerful new application of the GAN framework for the task of novelty detection.

Wir schlagen vor, ein Problem der gleichzeitigen Klassifizierung und Neuheitserkennung im GAN Framework zu lösen.

Er schlägt ein GAN vor, um Klassifizierung und Neuheitserkennung zu vereinen.

In diesem Beitrag wird eine Methode zur Neuheitserkennung vorgestellt, die auf einem Mehrklassen GAN basiert, das auf die Ausgabe von Bildern trainiert wird, die aus einer Mischung von nominalen und neuartigen Verteilungen erzeugt werden.

In der Arbeit wird ein GAN für die Erkennung von Neuheiten vorgeschlagen, das einen Mischungsgenerator mit Verlusten bei der Merkmalsanpassung verwendet.


Verifying a person's identity based on their voice is a challenging, real-world problem in biometric security. A crucial requirement of such speaker verification systems is to be domain robust. Performance should not degrade even if speakers are talking in languages not seen during training. To this end, we present a flexible and interpretable framework for learning domain invariant speaker embeddings using Generative Adversarial Networks. We combine adversarial training with an angular margin loss function, which encourages the speaker embedding model to be discriminative by directly optimizing for cosine similarity between classes. We are able to beat a strong baseline system using a cosine distance classifier and a simple score-averaging strategy. Our results also show that models with adversarial adaptation perform significantly better than unadapted models. In an attempt to better understand this behavior, we quantitatively measure the degree of invariance induced by our proposed methods using Maximum Mean Discrepancy and Frechet distances. Our analysis shows that our proposed adversarial speaker embedding models significantly reduce the distance between source and target data distributions, while performing similarly on the former and better on the latter.

Die Leistung der Sprecherverifikation kann durch die Anpassung des Modells an In-Domain-Daten mit Hilfe generativer adversarialer Netze erheblich verbessert werden. Außerdem kann die Anpassung auf unüberwachte Weise erfolgen.

Vorschlag einer Reihe von GAN-Varianten für die Aufgabe der Sprechererkennung unter der Bedingung, dass die Domäne nicht übereinstimmt.


Learning disentangling representations of the independent factors of variations that explain the data in an unsupervised setting is still a major challenge. In the following paper we address the task of disentanglement and introduce a new state-of-the-art approach called Non-synergistic variational Autoencoder (Non-Syn VAE). Our model draws inspiration from population coding, where the notion of synergy arises when we describe the encoded information by neurons in the form of responses from the stimuli. If those responses convey more information together than separate as independent sources of encoding information, they are acting synergetically. By penalizing the synergistic mutual information within the latents we encourage information independence and by doing that disentangle the latent factors. Notably, our approach could be added to the VAE framework easily, where the new ELBO function is still a lower bound on the log likelihood. In addition, we qualitatively compare our model with Factor VAE and show that this one implicitly minimises the synergy of the latents.

Minimierung der synergetischen wechselseitigen Information innerhalb der Latenzen und der Daten für die Aufgabe der Entflechtung unter Verwendung des VAE-Rahmens.

Schlägt eine neue Zielfunktion für das Erlernen von verschränkten Darstellungen in einem Variationsrahmen vor, indem die Synergie der bereitgestellten Informationen minimiert wird.

Die Autoren zielen darauf ab, eine VAE zu trainieren, die latente Repräsentationen in einer "synergetischen" maximalen Weise entwirrt.

In diesem Beitrag wird ein neuer Ansatz zur Durchsetzung der Entflechtung in VAEs vorgeschlagen, der einen Term verwendet, der die synergetische gegenseitige Information zwischen den latenten Variablen bestraft.


Metric embeddings are   immensely useful representations of associations between entities   (images, users, search queries, words, and more).   Embeddings are learned by  optimizing a loss objective of the general form of a sum over example associations. Typically, the optimization uses stochastic gradient updates over minibatches of examples that are arranged  independently at random. In this work, we propose the use of {\em structured arrangements} through randomized {\em microbatches} of examples that are more likely to include similar ones. We make a principled argument for the properties of our arrangements  that accelerate the training and present efficient algorithms to generate microbatches that respect the marginal  distribution of training examples.   Finally, we observe experimentally that our structured arrangements accelerate training by 3-20\%. Structured arrangements emerge as a powerful and novel performance knob for SGD that is independent and complementary to other SGD  hyperparameters and thus is a candidate for wide deployment.

Beschleunigung von SGD durch eine andere Anordnung der Beispiele.

Die Arbeit stellt eine Methode zur Verbesserung der Konvergenzrate des stochastischen Gradientenabstiegs für das Lernen von Einbettungen vor, indem ähnliche Trainingsbeispiele gruppiert werden.

Schlägt eine ungleichmäßige Sampling-Strategie zur Konstruktion von Minibatches in SGD für die Aufgabe des Lernens von Einbettungen für Objektassoziationen vor.


Ubuntu dialogue corpus is the largest public available dialogue corpus to make it feasible to build end-to-end
deep neural network models directly from the conversation data. One challenge of Ubuntu dialogue corpus is
the large number of out-of-vocabulary words. In this paper we proposed an algorithm which combines the general pre-trained word embedding vectors with those  generated on the task-specific training set to address this issue.   We integrated character embedding into Chen et al's Enhanced LSTM method (ESIM) and used it to evaluate the effectiveness of our proposed method. For the task of next utterance selection, the proposed method has demonstrated a significant performance improvement against original ESIM and the new model has achieved state-of-the-art results on both Ubuntu dialogue corpus and Douban conversation corpus. In addition, we investigated the performance impact of end-of-utterance and end-of-turn token tags.

Kombinieren von Informationen zwischen vorgefertigten Worteinbettungen und aufgabenspezifischen Wortdarstellungen, um das Problem des fehlenden Vokabulars zu lösen.

Diese Arbeit schlägt einen Ansatz vor, um die Vorhersage der Einbettung außerhalb des Vokabulars für die Aufgabe der Modellierung von Dialoggesprächen mit beträchtlichen Gewinnen gegenüber den Basislinien zu verbessern.

Schlägt vor, externe vortrainierte Worteinbettungen und vortrainierte Worteinbettungen auf Trainingsdaten zu kombinieren, indem sie als zwei Ansichten beibehalten werden.

Schlägt eine Methode zur Erweiterung der Abdeckung von vortrainierten Worteinbettungen vor, um das OOV-Problem zu bewältigen, das bei der Anwendung auf Gesprächsdatensätze auftritt, und wendet neue Varianten von LSTM basierten Modellen auf die Aufgabe der Antwortauswahl bei der Dialogmodellierung an.


Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks. Analogously, this suggests that learned update functions may similarly outperform current hand-designed optimizers, especially for specific tasks. However, learned optimizers are notoriously difficult to train and have yet to demonstrate wall-clock speedups over hand-designed optimizers, and thus are rarely used in practice. Typically, learned optimizers are trained by truncated backpropagation through an unrolled optimization process. The resulting gradients are either strongly biased (for short truncations) or have exploding norm (for long truncations). In this work we propose a training scheme which overcomes both of these difficulties, by dynamically weighting two unbiased gradient estimators for a variational loss on optimizer performance. This allows us to train neural networks to perform optimization faster than well tuned first-order methods. Moreover, by training the optimizer against validation loss, as opposed to training loss, we are able to use it to train models which generalize better than those trained by first order methods. We demonstrate these results on problems where our learned optimizer trains convolutional networks in a fifth of the wall-clock time compared to tuned first-order methods, and with an improvement

Wir analysieren Probleme beim Training von gelernten Optimierern, lösen diese Probleme durch Variationsoptimierung unter Verwendung von zwei komplementären Gradientenschätzern und trainieren Optimierer, die in der Wanduhrzeit 5x schneller sind als Basisoptimierer (z.B. Adam).

In diesem Beitrag wird die ungerollte Optimierung verwendet, um neuronale Netze für die Optimierung zu lernen.

Diese Arbeit befasst sich mit dem Problem des Lernens eines Optimierers, insbesondere konzentrieren sich die Autoren darauf, sauberere Gradienten aus dem aufgerollten Trainingsverfahren zu erhalten.

Stellt eine Methode zum "Lernen eines Optimierers" vor, indem eine Variationsoptimierung für den "äußeren" Optimiererverlust verwendet wird, und schlägt die Idee vor, sowohl den reparametrisierten Gradienten als auch den Score-Funktionsschätzer für das Variationsziel zu kombinieren und sie mit Hilfe einer Produkt-Gauß-Formel für den Mittelwert zu gewichten.


Asynchronous distributed gradient descent algorithms for training of deep neural
 networks are usually considered as inefficient, mainly because of the Gradient delay
 problem. In this paper, we propose a novel asynchronous distributed algorithm
 that tackles this limitation by well-thought-out averaging of model updates, computed
 by workers. The algorithm allows computing gradients along the process
 of gradient merge, thus, reducing or even completely eliminating worker idle time
 due to communication overhead, which is a pitfall of existing asynchronous methods.
 We provide theoretical analysis of the proposed asynchronous algorithm,
 and show its regret bounds. According to our analysis, the crucial parameter for
 keeping high convergence rate is the maximal discrepancy between local parameter
 vectors of any pair of workers. As long as it is kept relatively small, the
 convergence rate of the algorithm is shown to be the same as the one of a sequential
 online learning. Furthermore, in our algorithm, this discrepancy is bounded
 by an expression that involves the staleness parameter of the algorithm, and is
 independent on the number of workers. This is the main differentiator between
 our approach and other solutions, such as Elastic Asynchronous SGD or Downpour
 SGD, in which that maximal discrepancy is bounded by an expression that
 depends on the number of workers, due to gradient delay problem. To demonstrate
 effectiveness of our approach, we conduct a series of experiments on image
 classification task on a cluster with 4 machines, equipped with a commodity communication
 switch and with a single GPU card per machine. Our experiments
 show a linear scaling on 4-machine cluster without sacrificing the test accuracy,
 while eliminating almost completely worker idle time. Since our method allows
 using commodity communication switch, it paves a way for large scale distributed
 training performed on commodity clusters.

Eine Methode für ein effizientes asynchrones verteiltes Training von Deep-Learning-Modellen zusammen mit theoretischen Regret-Grenzen.

Die Arbeit schlägt einen Algorithmus zur Begrenzung der Staleness in asynchronen SGD und bietet eine theoretische Analyse.

Schlägt einen Hybrid-Algorithmus vor, um die Gradientenverzögerung von asynchronen Methoden zu eliminieren.


Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs. However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks. This paper aims to raise people's awareness about the security of the quantized models, and we designed a novel quantization methodology to jointly optimize the efficiency and robustness of deep learning models. We first conduct an empirical study to show that vanilla quantization suffers more from adversarial attacks. We observe that the inferior robustness comes from the error amplification effect, where the quantization operation further enlarges the distance caused by amplified noise. Then we propose a novel Defensive Quantization (DQ) method by controlling the Lipschitz constant of the network during quantization, such that the magnitude of the adversarial noise remains non-expansive during inference. Extensive experiments on CIFAR-10 and SVHN datasets demonstrate that our new quantization method can defend neural networks against adversarial examples, and even achieves superior robustness than their full-precision counterparts, while maintaining the same hardware efficiency as vanilla quantization approaches. As a by-product, DQ can also improve the accuracy of quantized models without adversarial attack.

Wir haben eine neuartige Quantisierungsmethode entwickelt, um die Effizienz und Robustheit von Deep-Learning-Modellen gemeinsam zu optimieren.

Schlägt ein Regularisierungsschema vor, um quantisierte neuronale Netze vor adversarial Angriffen zu schützen, indem es eine Lipschitz-Konstante zur Filterung des Inpout-Outputs der inneren Schichten verwendet.


Recurrent Neural Networks (RNNs) continue to show  outstanding performance in sequence modeling tasks. However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models. Source code is publicly available at https://imatge-upc.github.io/skiprnn-2017-telecombcn/.

Eine Modifikation für bestehende RNN-Architekturen, die es ihnen ermöglicht, Zustandsaktualisierungen zu überspringen und dabei die Leistung der ursprünglichen Architekturen beizubehalten.

Schlägt das Skip-RNN-Modell vor, das es einem rekurrenten Netzwerk ermöglicht, die Aktualisierung seines verborgenen Zustands für einige Eingaben selektiv zu überspringen, was zu einer reduzierten Berechnung zur Testzeit führt.

Schlägt ein neuartiges RNN-Modell vor, bei dem sowohl die Eingabe als auch die Zustandsaktualisierung der rekurrenten Zellen adaptiv für einige Zeitschritte übersprungen wird.


We propose a fast second-order method that can be used as a drop-in replacement for current deep learning solvers. Compared to stochastic gradient descent (SGD), it only requires two additional forward-mode automatic differentiation operations per iteration, which has a computational cost comparable to two standard forward passes and is easy to implement. Our method addresses long-standing issues with current second-order solvers, which invert an approximate Hessian matrix every iteration exactly or by conjugate-gradient methods, procedures that are much slower than a SGD step. Instead, we propose to keep a single estimate of the gradient projected by the inverse Hessian matrix, and update it once per iteration with just two passes over the network. This estimate has the same size and is similar to the momentum variable that is commonly used in SGD. No estimate of the Hessian is maintained.
 We first validate our method, called CurveBall, on small problems with known solutions (noisy Rosenbrock function and degenerate 2-layer linear networks), where current deep learning solvers struggle. We then train several large models on CIFAR and ImageNet, including ResNet and VGG-f networks, where we demonstrate faster convergence with no hyperparameter tuning. We also show our optimiser's generality by testing on a large set of randomly-generated architectures.

Ein schneller Solver zweiter Ordnung für Deep Learning, der bei ImageNet-Problemen ohne Abstimmung der Hyperparameter funktioniert.

Wahl der Richtung durch Verwendung eines einzelnen Schritts des Gradientenabstiegs "in Richtung Newton-Schritt" von einer ursprünglichen Schätzung aus, und dann Übernahme dieser Richtung anstelle des ursprünglichen Gradienten.

Ein neues approximatives Optimierungsverfahren zweiter Ordnung mit geringem Rechenaufwand, das die Berechnung der Hessian Matrix durch einen einzigen Gradientenschritt und eine Warmstart Strategie ersetzt.