Machine translation is an important real-world application, and neural network-based AutoRegressive Translation (ART) models have achieved very promising accuracy. Due to the unparallelizable nature of the autoregressive factorization, ART models have to generate tokens one by one during decoding and thus suffer from high inference latency. Recently, Non-AutoRegressive Translation (NART) models were proposed to reduce the inference time. However, they could only achieve inferior accuracy compared with ART models. To improve the accuracy of NART models, in this paper, we propose to leverage the hints from a well-trained ART model to train the NART model. We define two hints for the machine translation task: hints from hidden states and hints from word alignments, and use such hints to regularize the optimization of NART models. Experimental results show that the NART model trained with hints could achieve significantly better translation performance than previous NART models on several tasks. In particular, for the WMT14 En-De and De-En task, we obtain BLEU scores of 25.20 and 29.52 respectively, which largely outperforms the previous non-autoregressive baselines. It is even comparable to a strong LSTM-based ART model (24.60 on WMT14 En-De), but one order of magnitude faster in inference.

Wir entwickeln einen Trainingsalgorithmus für nicht-autoregressive maschinelle Übersetzungsmodelle, der eine vergleichbare Genauigkeit wie starke autoregressive Grundmodelle erreicht, aber eine Größenordnung schneller in der Inferenz ist.

Entnimmt Wissen aus versteckten Zwischenzuständen und Aufmerksamkeitsgewichten, um die nicht-autoregressive neuronale maschinelle Übersetzung zu verbessern.

Schlägt vor, ein gut trainiertes autoregressives Modell zu nutzen, um die versteckten Zustände und die Wortausrichtung von nicht-autoregressiven neuronalen maschinellen Übersetzungsmodellen zu informieren.


Artificial neural networks are built on the basic operation of linear combination and non-linear activation function. Theoretically this structure can approximate any continuous function with three layer architecture. But in practice learning  the parameters of such network can be hard. Also the choice of activation function can greatly impact the performance of the network. In this paper we are proposing to replace the basic linear combination operation with non-linear operations that do away with the need of additional non-linear activation function. To this end we are proposing the use of elementary  morphological operations (dilation and erosion) as the basic operation in neurons. We show that these networks (Denoted as Morph-Net) with morphological operations can approximate any smooth function requiring less number of parameters than what is necessary for normal neural networks. The results show that our network perform favorably when compared with similar structured network. We have carried out our experiments on  MNIST, Fashion-MNIST, CIFAR10 and CIFAR100.

Mit Hilfe mophologischer Operationen (Dilatation und Erosion) haben wir eine Klasse von Netzen definiert, die eine beliebige kontinuierliche Funktion annähern können.

In dieser Arbeit wird vorgeschlagen, die Standard RELU/tanh Einheiten durch eine Kombination von Dilatations- und Erosionsoperationen zu ersetzen, wobei festgestellt wird, dass der neue Operator mehr Hyperebenen erzeugt und eine größere Ausdruckskraft hat.

Die Autoren stellen Morph-Net vor, ein einschichtiges neuronales Netz, bei dem die Abbildung durch morphologische Dilatation und Erosion erfolgt.


With the rapidly scaling up of deep neural networks (DNNs), extensive research studies on network model compression such as weight pruning have been performed for efficient deployment. This work aims to advance the compression beyond the weights to the activations of DNNs. We propose the Integral Pruning (IP) technique which integrates the activation pruning with the weight pruning. Through the learning on the different importance of neuron responses and connections, the generated network, namely IPnet, balances the sparsity between activations and weights and therefore further improves execution efficiency. The feasibility and effectiveness of IPnet are thoroughly evaluated through various network models with different activation functions and on different datasets. With <0.5% disturbance on the testing accuracy, IPnet saves 71.1% ~ 96.35% of computation cost, compared to the original dense models with up to 5.8x and 10x reductions in activation and weight numbers, respectively.

Diese Arbeit erweitert die DNN Kompression über die Gewichte hinaus auf die Aktivierungen, durch Integration des Prunings von Aktivierungen in das Pruning der Gewichte.

Eine integrale Modellkompressionsmethode, die sowohl Gewichts- als auch Aktivierungs- Pruning handhabt, was zu einer effizienteren Netzberechnung und einer effektiven Reduzierung der Anzahl von Multiplikationen und Akkumulationen führt.

In diesem Artikel wird ein neuartiger Ansatz zur Verringerung der Rechenkosten von tiefen neuronalen Netzen durch die Integration von Aktivierungs-Pruning zusammen mit Gewichts-Pruning vorgestellt. Es wird gezeigt, dass gängige Techniken des ausschließlichen Gewichts-Pruning die Anzahl der Aktivierungen ungleich Null nach ReLU erhöhen.


The Variational Auto Encoder (VAE) is a popular generative
latent variable model that is often
applied for representation learning.
 Standard VAEs assume continuous valued
latent variables and are trained by maximization
of the evidence lower bound (ELBO). Conventional methods obtain a
differentiable estimate of the ELBO with reparametrized sampling and
optimize it with Stochastic Gradient Descend (SGD). However, this is not possible if
we want to train VAEs with discrete valued latent variables,
since reparametrized sampling is not possible. Till now, there
exist no simple solutions to circumvent this problem.
 In this paper, we propose an easy method to train VAEs
with binary or categorically valued latent representations. Therefore, we use a differentiable
estimator for the ELBO which is based on importance sampling. In experiments, we verify the approach and
train two different VAEs architectures with Bernoulli and
Categorically distributed latent representations on two different benchmark
datasets.

Wir schlagen eine einfache Methode zum Trainieren von variationalen Auto Encoders (VAE) mit diskreten latenten Repräsentationen unter Verwendung von Wichtigkeitssampling vor.

Einführung einer Wichtigkeits-Sampling Verteilung und Verwendung von Stichproben aus der Verteilung zur Berechnung der Wichtigkeitsgewichteten Schätzung des Gradienten

In dieser Arbeit wird vorgeschlagen, wichtige Stichproben zur Optimierung der VAE mit diskreten latenten Variablen zu verwenden.


Distributed computing can significantly reduce the training time of neural networks. Despite its potential, however, distributed training has not been widely adopted: scaling the training process is difficult, and existing SGD methods require substantial tuning of hyperparameters and learning schedules to achieve sufficient accuracy when increasing the number of workers. In practice, such tuning can be prohibitively expensive given the huge number of potential hyperparameter configurations and the effort required to test each one.

 We propose DANA, a novel approach that scales out-of-the-box to large clusters using the same hyperparameters and learning schedule optimized for training on a single worker, while maintaining similar final accuracy without additional overhead. DANA estimates the future value of model parameters by adapting Nesterov Accelerated Gradient to a distributed setting, and so mitigates the effect of gradient staleness, one of the main difficulties in scaling SGD to more workers.

 Evaluation on three state-of-the-art network architectures and three datasets shows that DANA scales as well as or better than existing work without having to tune any hyperparameters or tweak the learning schedule. For example, DANA achieves 75.73% accuracy on ImageNet when training ResNet-50 with 16 workers, similar to the non-distributed baseline.

Ein neuer verteilter asynchroner SGD Algorithmus, der auf bestehenden Architekturen ohne zusätzliches Tuning oder Overhead die höchste Genauigkeit erreicht.

Vorschlagen einer Verbesserung bestehender ASGD-Ansätze bei mittlerer Skalierung, die Momentum mit SGD für asynchrones Training über einen verteilten Worker-Pool verwendet.

Diese Arbeit befasst sich mit der Staleness des Gradienten im Vergleich zur parallelen Leistung Problem beim verteilten Deep Learning-Training und schlägt einen Ansatz zur Schätzung zukünftiger Modellparameter an den Slaves vor, um die Auswirkungen der Kommunikationslatenz zu reduzieren.


This paper proposes a novel approach to train deep neural networks by unlocking the layer-wise dependency of backpropagation training. The approach employs additional modules called local critic networks besides the main network model to be trained, which are used to obtain error gradients without complete feedforward and backward propagation processes. We propose a cascaded learning strategy for these local networks. In addition, the approach is also useful from multi-model perspectives, including structural optimization of neural networks, computationally efficient progressive inference, and ensemble classification for performance improvement. Experimental results show the effectiveness of the proposed approach and suggest guidelines for determining appropriate algorithm parameters.

Wir schlagen einen neuen Lernalgorithmus für tiefe neuronale Netze vor, der die schichtweise Abhängigkeit der Backpropagation aufhebt.

Ein alternatives Trainingsparadigma für DNIs, bei dem das Hilfsmodul so trainiert wird, dass es die endgültige Ausgabe des ursprünglichen Modells direkt annähert, bietet Nebeneffekte.

Beschreibt eine Methode zum Training neuronaler Netze ohne Aktualisierungssperre.


\emph{Truncated Backpropagation Through Time} (truncated BPTT, \cite{jaeger2002tutorial}) is a widespread method for learning recurrent computational graphs. Truncated BPTT keeps the computational benefits of \emph{Backpropagation Through Time} (BPTT \cite{werbos:bptt}) while relieving the need for a complete backtrack through the whole data sequence at every step.   However, truncation favors short-term dependencies: the gradient estimate of truncated BPTT is biased, so that it does not benefit from the convergence guarantees from stochastic gradient theory. We introduce \emph{Anticipated Reweighted Truncated Backpropagation} (ARTBP), an algorithm that keeps the computational benefits of truncated BPTT, while providing unbiasedness. ARTBP works by using variable truncation lengths together with carefully chosen compensation factors in the backpropagation equation. We check the viability of ARTBP on two tasks. First, a simple synthetic task where careful balancing of temporal dependencies at different scales is needed: truncated BPTT displays unreliable performance, and in worst case scenarios, divergence, while ARTBP converges reliably. Second, on Penn Treebank character-level language modelling \cite{ptb_proc}, ARTBP slightly outperforms truncated BPTT.

Bietet eine unverzerrte Version der verkürzten Backpropagation durch Stichproben der Verkürzungslängen und entsprechende Neugewichtung.

Vorschlagen einer stochastische Methoden zur Bestimmung von Abbruchpunkten in der Backpropagation durch die Zeit.

Eine neue Annäherung an die Backpropagation durch die Zeit, um die Rechen- und Speicherbelastung zu überwinden, die entsteht, wenn man aus langen Sequenzen lernen muss.


Graph convolutional networks (GCNs) have been widely used for classifying graph nodes in the semi-supervised setting.
 Previous works have shown that GCNs are vulnerable to the perturbation on adjacency and feature matrices of existing nodes. However, it is unrealistic to change the connections of  existing nodes in many applications, such as existing users in social networks. In this paper, we investigate methods attacking GCNs by adding fake nodes. A greedy algorithm is proposed to generate adjacency and feature matrices of fake nodes, aiming to minimize the classification accuracy on the existing ones. In additional, we introduce a discriminator to classify fake nodes from real nodes, and propose a Greedy-GAN algorithm to simultaneously update the discriminator and the attacker, to make fake nodes indistinguishable to the real ones.   Our non-targeted attack decreases the accuracy of GCN down to 0.10, and our targeted attack reaches a success rate of 0.99 for attacking the whole datasets, and 0.94 on average for attacking a single node.

nicht gezielte und gezielte Angriffe auf GCN durch Hinzufügen gefälschter Knotenpunkte

Die Autoren schlagen eine neue Technik vor, mit der "falsche" Knoten hinzugefügt werden können, um einen GCN-basierten Klassifikator zu täuschen.


Transfer learning aims to solve the data sparsity for a specific domain by applying information of another domain. Given a sequence (e.g. a natural language sentence), the transfer learning, usually enabled by recurrent neural network (RNN), represent the sequential information transfer. RNN uses a chain of repeating cells to model the sequence data. However, previous studies of neural network based transfer learning simply transfer the information across the whole layers, which are unfeasible for seq2seq and sequence labeling. Meanwhile, such layer-wise transfer learning mechanisms also lose the fine-grained cell-level information from the source domain.

 In this paper, we proposed the aligned recurrent transfer, ART, to achieve cell-level information transfer. ART is in a recurrent manner that different cells share the same parameters. Besides transferring the corresponding information at the same position, ART transfers information from all collocated words in the source domain. This strategy enables ART to capture the word collocation across domains in a more flexible way. We conducted extensive experiments on both sequence labeling tasks (POS tagging, NER) and sentence classification (sentiment analysis). ART outperforms the state-of-the-arts over all experiments.

Transferlernen für Sequenzen durch Lernen, um Informationen auf Zellebene über Domänen hinweg abzugleichen.

In dem Papier wird vorgeschlagen, RNN/LSTM mit Kollokationsabgleich als Repräsentationslernmethode für Transferlernen/Domänenanpassung in NLP zu verwenden.


Addressing uncertainty is critical for autonomous systems to robustly adapt to the real world. We formulate the problem of model uncertainty as a continuous Bayes-Adaptive Markov Decision Process (BAMDP), where an agent maintains a posterior distribution over latent model parameters given a history of observations and maximizes its expected long-term reward with respect to this belief distribution. Our algorithm, Bayesian Policy Optimization, builds on recent policy optimization algorithms to learn a universal policy that navigates the exploration-exploitation trade-off to maximize the Bayesian value function. To address challenges from discretizing the continuous latent parameter space, we propose a new policy network architecture that encodes the belief distribution independently from the observable state. Our method significantly outperforms algorithms that address model uncertainty without explicitly reasoning about belief distributions and is competitive with state-of-the-art Partially Observable Markov Decision Process solvers.

Wir formulieren die Modellunsicherheit beim Reinforcement Learning als einen kontinuierlichen Bayes Adaptiven Markov Entscheidungsprozess und präsentieren eine Methode zur praktischen und skalierbaren Bayes'schen Optimierung von Strategien.

Mit einem Bayes'schen Ansatz lässt sich ein besseres Gleichgewicht zwischen Erkundung und Ausbeutung in RL erreichen.


For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic.
 We clarify a necessary condition for an evaluation metric not to behave this way: estimating the function must require a large sample from the model. In search of such a metric, we turn to neural network divergences (NNDs), which are defined in terms of a neural network trained to distinguish between distributions. The resulting benchmarks cannot be ``won'' by training set memorization, while still being perceptually correlated and computable only from samples. We survey past work on using NNDs for evaluation, implement an example black-box metric based on these ideas, and validate experimentally that it can measure a notion of generalization.

Wir argumentieren, dass GAN-Benchmarks eine große Stichprobe des Modells erfordern müssen, um das Auswendiglernen zu bestrafen, und untersuchen, ob die Divergenzen neuronaler Netze diese Eigenschaft aufweisen.

Die Autoren schlagen ein Kriterium für die Bewertung der Qualität der von einem generativen adversarial Netzwerk erzeugten Beispielen vor.


Conventional methods model open domain dialogue generation as a black box through end-to-end learning from large scale conversation data. In this work, we make the first step to open the black box by introducing dialogue acts into open domain dialogue generation. The dialogue acts are generally designed and reveal how people engage in social chat. Inspired by analysis on real data, we propose jointly modeling dialogue act selection and response generation, and perform learning with human-human conversations tagged with a dialogue act classifier and a reinforcement approach to further optimizing the model for long-term conversation. With the dialogue acts, we not only achieve significant improvement over state-of-the-art methods on response quality for given contexts and long-term conversation in both machine-machine simulation and human-machine conversation, but also are capable of explaining why such achievements can be made.

Erzeugung von Dialogen im offenen Bereich mit Dialogakten.

Die Autoren verwenden eine Fernüberwachungsmethode, um Tags für Dialoghandlungen als Konditionierungsfaktor für die Generierung von Antworten in Dialogen mit offener Domäne hinzuzufügen.

Das Papier beschreibt eine Technik zur Einbindung von Dialoghandlungen in neuronale Konversationsagenten.


We discuss the feasibility of the following learning problem: given unmatched samples from two domains and nothing else, learn a mapping between the two, which preserves semantics. Due to the lack of paired samples and without any definition of the semantic information, the problem might seem ill-posed. Specifically, in typical cases, it seems possible to build infinitely many alternative mappings  from every target mapping. This apparent ambiguity stands in sharp contrast to the recent empirical success in solving this problem.

 We identify the abstract notion of aligning two domains in a semantic way with concrete terms of minimal relative complexity. A theoretical framework for measuring the complexity of compositions of functions is developed in order to show that it is reasonable to expect the minimal complexity mapping to be unique. The measured complexity used is directly related to the depth of the neural networks being learned and a semantically aligned mapping could then be captured simply by learning using architectures that are not much bigger than the minimal architecture.

 Various predictions are made based on the hypothesis that semantic alignment can be captured by the minimal mapping. These are verified extensively. In addition, a new mapping algorithm is proposed and shown to lead to better mapping results.

Unsere Hypothese ist, dass bei zwei Bereichen die Abbildung mit der geringsten Komplexität und der geringsten Diskrepanz der Zielabbildung am nächsten kommt.

Die Arbeit befasst sich mit dem Problem des Lernens von Zuordnungen zwischen verschiedenen Domänen ohne jegliche Überwachung und stellt drei Vermutungen auf.

Es wird gezeigt, dass es beim unüberwachten Lernen auf unausgerichteten Daten möglich ist, die Abbildung zwischen den Domänen nur mit GAN ohne Rekonstruktionsverlust zu lernen.


We present a novel approach for the certification of neural networks against adversarial perturbations which combines scalable overapproximation methods with precise (mixed integer) linear programming. This results in significantly better precision than state-of-the-art verifiers on challenging feedforward and convolutional neural networks with piecewise linear activation functions.

Wir verfeinern die Ergebnisse der Über-Approximation von unvollständigen Verifikatoren mit MILP-Lösern, um mehr Robustheitseigenschaften als der Stand der Technik zu beweisen.

Vorstellen eines Verifizierer, der die Genauigkeit von unvollständigen Verifizierern und die Skalierbarkeit von vollständigen Verifizierern durch Überparametrisierung, gemischt ganzzahlige lineare Programmierung und Entspannung der linearen Programmierung verbessert.

Eine gemischte Strategie zur Erzielung einer besseren Präzision bei Robustheitsüberprüfungen von Feed-Forward neuronalen Netzen mit stückweise linearen Aktivierungsfunktionen, wobei eine bessere Präzision als bei unvollständigen Überprüfern und eine bessere Skalierbarkeit als bei vollständigen Überprüfern erreicht wird.


A distinct commonality between HMMs and RNNs is that they both learn hidden representations for sequential data. In addition, it has been noted that the backward computation of the Baum-Welch algorithm for HMMs is a special case of the back-propagation algorithm used for neural networks (Eisner (2016)). Do these observations suggest that, despite their many apparent differences, HMMs are a special case of RNNs? In this paper, we show that that is indeed the case, and investigate a series of architectural transformations between HMMs and RNNs, both through theoretical derivations and empirical hybridization. In particular, we investigate three key design factors—independence assumptions between the hidden states and the observation, the placement of softmaxes, and the use of non-linearities—in order to pin down their empirical effects. We present a comprehensive empirical study to provide insights into the interplay between expressivity and interpretability in this model family with respect to language modeling and parts-of-speech induction.

Sind HMMs ein Spezialfall von RNNs? Wir untersuchen eine Reihe von architektonischen Transformationen zwischen HMMs und RNNs, sowohl durch theoretische Ableitungen als auch durch empirische Hybridisierung und liefern neue Erkenntnisse.

In diesem Beitrag wird untersucht, ob HMMs ein Spezialfall von RNNs sind, die Sprachmodellierung und POS-Tagging verwenden.


Deep neural networks have been tremendously successful in a number of tasks.
 One of the main reasons for this is their capability to automatically
 learn representations of data in levels of abstraction,
 increasingly disentangling the data as the internal transformations are applied.
 In this paper we propose a novel regularization method that penalize covariance between dimensions of the hidden layers in a network, something that benefits the disentanglement.
 This makes the network learn nonlinear representations that are linearly uncorrelated, yet allows the model to obtain good results on a number of tasks, as demonstrated by our experimental evaluation.
 The proposed technique can be used to find the dimensionality of the underlying data, because it effectively disables dimensions that aren't needed.
 Our approach is simple and computationally cheap, as it can be applied as a regularizer to any gradient-based learning model.

Wir schlagen eine neuartige Regularisierungsmethode vor, die die Kovarianz zwischen den Dimensionen der versteckten Schichten in einem Netzwerk bestraft.

Diese Arbeit stellt einen Regularisierungsmechanismus vor, der die Kovarianz zwischen allen Dimensionen in der latenten Repräsentation eines neuronalen Netzes bestraft, um die latente Repräsentation zu entflechten.


This report introduces a training and recognition scheme, in which classification is realized via class-wise discerning. Trained with datasets whose labels are randomly shuffled except for one class of interest, a neural network learns class-wise parameter values, and remolds itself from a feature sorter into feature filters, each of which discerns objects belonging to one of the classes only. Classification of an input can be inferred from the maximum response of the filters. A multiple check with multiple versions of filters can diminish fluctuation and yields better performance. This scheme of discerning, maximum response and multiple check is a method of general viability to improve performance of feedforward networks, and the filter training itself is a promising feature abstraction procedure. In contrast to the direct sorting, the scheme mimics the classification process mediated by a series of one component picking.

Das vorgeschlagene System ahmt den Klassifizierungsprozess nach, der durch eine Reihe von Ein-Komponenten Pickings vermittelt wird.

Eine Methode zur Erhöhung der Genauigkeit von Deep-Nets bei Mehrklassen-Klassifizierungsaufgaben, die scheinbar durch eine Reduktion von Mehrklassen- auf Binärklassifizierung erfolgt.

Ein neuartiges Klassifizierungsverfahren mit Unterscheidung, Maximalantwort und Mehrfachprüfung zur Verbesserung der Genauigkeit mittelmäßiger Netzwerke und zur Verbesserung von Feedforward Netzwerken.


A long-held conventional wisdom states that larger models train more slowly when using gradient descent. This work challenges this widely-held belief, showing that larger models can potentially train faster despite the increasing computational requirements of each training step. In particular, we study the effect of network structure (depth and width) on halting time and show that larger models---wider models in particular---take fewer training steps to converge.

 We design simple experiments to quantitatively characterize the effect of overparametrization on weight space traversal. Results show that halting time improves when growing model's width for three different applications, and the improvement comes from each factor: The distance from initialized weights to converged weights shrinks with a power-law-like relationship, the average step size grows with a power-law-like relationship, and gradient vectors become more aligned with each other during traversal.

Die Erfahrung zeigt, dass größere Modelle in weniger Trainingsschritten trainiert werden können, da sich alle Faktoren bei der Durchquerung des Gewichtsraums verbessern.

Diese Arbeit zeigt, dass breitere RNNs die Konvergenzgeschwindigkeit verbessern, wenn sie auf NLP-Probleme angewandt werden, und dass die Auswirkung der Erhöhung der Breiten in tiefen neuronalen Netzen auf die Konvergenz der Optimierung.

In dieser Arbeit werden die Auswirkungen einer Überparametrisierung auf die Anzahl der Wiederholungen, die ein Algorithmus benötigt, um zu konvergieren, beschrieben und weitere empirische Beobachtungen zu den Auswirkungen einer Überparametrisierung beim Training neuronaler Netze vorgestellt.


Due to its potential to improve programmer productivity and software quality, automated program repair has been an active topic of research. Newer techniques harness neural networks to learn directly from examples of buggy programs and their fixes. In this work, we consider a recently identified class of bugs called variable-misuse bugs. The state-of-the-art solution for variable misuse enumerates potential fixes for all possible bug locations in a program, before selecting the best prediction. We show that it is beneficial to train a model that jointly and directly localizes and repairs variable-misuse bugs. We present multi-headed pointer networks for this purpose, with one head each for localization and repair. The experimental results show that the joint model significantly outperforms an enumerative solution that uses a pointer based model for repair alone.

Mehrköpfige Zeigernetzwerke für gemeinsames Lernen zur Lokalisierung und Behebung von Fehlern durch Variablenmissbrauch.

Schlägt ein LSTM basiertes Modell mit Zeigern vor, um das Problem des VarMisuse in mehrere Schritte aufzuteilen.

In diesem Beitrag wird ein LSTM basiertes Modell zur Fehlererkennung und -behebung des VarMisuse Fehlers vorgestellt, das im Vergleich zu früheren Ansätzen in mehreren Datensätzen deutliche Verbesserungen aufweist.


Classification and clustering have been studied separately in machine learning and computer vision. Inspired by the recent success of deep learning models in solving various vision problems (e.g., object recognition, semantic segmentation) and the fact that humans serve as the gold standard in assessing clustering algorithms, here, we advocate for a unified treatment of the two problems and suggest that hierarchical frameworks that progressively build complex patterns on top of the simpler ones (e.g., convolutional neural networks) offer a promising solution. We do not dwell much on the learning mechanisms in these frameworks as they are still a matter of debate, with respect to biological constraints. Instead, we emphasize on the compositionality of the real world structures and objects. In particular, we show that CNNs, trained end to end using back propagation with noisy labels, are able to cluster data points belonging to several overlapping shapes, and do so much better than the state of the art algorithms. The main takeaway lesson from our study is that mechanisms of human vision, particularly the hierarchal organization of the visual ventral stream should be taken into account in clustering algorithms (e.g., for learning representations in an unsupervised manner or with minimum supervision) to reach human level clustering performance. This, by no means, suggests that other methods do not hold merits. For example, methods relying on pairwise affinities (e.g., spectral clustering) have been very successful in many cases but still fail in some cases (e.g., overlapping clusters).

Das Papier bestätigt die Idee, dass tiefe Convolutional Neural Networks lernen könnten, Eingabedaten besser zu clustern als andere Clustering-Methoden, indem sie den Kontext jedes Eingabepunkts aufgrund eines großen Sichtfelds interpretieren können.

Diese Arbeit kombiniert Deep Learning für die Merkmalsdarstellung mit der Aufgabe der menschenähnlichen, unbeaufsichtigten Gruppierung.


Instancewise feature scoring is a method for model interpretation, which yields, for each test instance, a vector of importance scores associated with features. Methods based on the Shapley score have been proposed as a fair way of computing feature attributions, but incur an exponential complexity in the number of features.   This combinatorial explosion arises from the definition of Shapley value and prevents these methods from being scalable to large data sets and complex models. We focus on settings in which the data have a graph structure, and the contribution of features to the target variable is well-approximated by a graph-structured factorization.   In such settings, we develop two algorithms with linear complexity for instancewise feature importance scoring on black-box models.   We establish the relationship of our methods to the Shapley value and a closely related concept known as the Myerson value from cooperative game theory. We demonstrate on both language and image data that our algorithms compare favorably with other methods using both quantitative metrics and human evaluation.

Wir entwickeln zwei Algorithmen mit linearer Komplexität für die modellagnostische Modellinterpretation auf der Grundlage des Shapley-Wertes, wenn der Beitrag der Merkmale zum Ziel durch eine graphisch-strukturierte Faktorisierung gut angenähert ist.

In der Arbeit werden zwei Annäherungen an den Shapley-Wert vorgeschlagen, der zur Erstellung von Merkmalsbewertungen für die Interpretierbarkeit verwendet wird.

In dieser Arbeit werden zwei Methoden für die instanzielle Bewertung der Wichtigkeit von Merkmalen unter Verwendung von Shapely-Werten vorgeschlagen und zwei effiziente Methoden zur Berechnung von ungefähren Shapely-Werten bereitgestellt, wenn eine bekannte Struktur zwischen den Merkmalen vorhanden ist.


According to parallel distributed processing (PDP) theory in psychology, neural networks (NN) learn distributed rather than interpretable localist representations. This view has been held so strongly that few researchers have analysed single units to determine if this assumption is correct. However, recent results from psychology, neuroscience and computer science have shown the occasional existence of local codes emerging in artificial and biological neural networks. In this paper, we undertake the first systematic survey of when local codes emerge in a feed-forward neural network, using generated input and output data with known qualities. We find that the number of local codes that emerge from a NN follows a well-defined distribution across the number of hidden layer neurons, with a peak determined by the size of input data, number of examples presented and the sparsity of input data. Using a 1-hot output code drastically decreases the number of local codes on the hidden layer. The number of emergent local codes increases with the percentage of dropout applied to the hidden layer, suggesting that the localist encoding may offer a resilience to noisy networks. This data suggests that localist coding can emerge from feed-forward PDP networks and suggests some of the conditions that may lead to interpretable localist representations in the cortex. The findings highlight how local codes should not be dismissed out of hand.

Lokale Codes wurden in neuronalen Feed-Forward Netzen gefunden.

Eine Methode zur Bestimmung, inwieweit einzelne Neuronen in einer versteckten Schicht eines MLP einen lokalistischen Code kodieren, der für verschiedene Eingabedarstellungen untersucht wird.

Untersucht die Entwicklung von lokalistischen Darstellungen in den verborgenen Schichten von neuronalen Feed-Forward-Netzen.


Representing entities and relations in an embedding space is a well-studied approach for machine learning on relational data. Existing approaches however primarily focus on simple link structure between a finite set of entities, ignoring the variety of data types that are often used in relational databases, such as text, images, and numerical values. In our approach, we propose a multimodal embedding using different neural encoders for this variety of data, and combine with existing models to learn embeddings of the entities. We extend existing datasets to create two novel benchmarks, YAGO-10-plus and MovieLens-100k-plus, that contain additional relations such as textual descriptions and images of the original entities. We demonstrate that our model utilizes the additional information effectively to provide further gains in accuracy. Moreover, we test our learned multimodal embeddings by using them to predict missing multimodal attributes.

Erweiterung der relationalen Modellierung zur Unterstützung multimodaler Daten unter Verwendung neuronaler Kodierer.

In diesem Beitrag wird vorgeschlagen, Link-Vorhersagen in Wissensdatenbanken durchzuführen, indem die ursprünglichen Entitäten durch multimodale Informationen ergänzt werden, und es wird ein Modell vorgestellt, das in der Lage ist, bei der Bewertung von Triples alle Arten von Informationen zu kodieren.

In dem Beitrag geht es um die Einbeziehung von Informationen aus verschiedenen Modalitäten in Ansätzen zur Vorhersage von Verbindungen.


An ensemble of neural networks is known to be more robust and accurate than an individual network, however usually with linearly-increased cost in both training and testing.
 In this work, we propose a two-stage method to learn Sparse Structured Ensembles (SSEs) for neural networks.
 In the first stage, we run SG-MCMC with group sparse priors to draw an ensemble of samples from the posterior distribution of network parameters. In the second stage, we apply weight-pruning to each sampled network and then perform retraining over the remained connections.
 In this way of learning SSEs with SG-MCMC and pruning, we not only achieve high prediction accuracy since SG-MCMC enhances exploration of the model-parameter space, but also reduce memory and computation cost significantly in both training and testing of NN ensembles.
 This is thoroughly evaluated in the experiments of learning SSE ensembles of both FNNs and LSTMs.
 For example, in LSTM based language modeling (LM), we obtain 21\% relative reduction in LM perplexity by learning a SSE of 4 large LSTM models, which has only 30\% of model parameters and 70\% of computations in total, as compared to the baseline large LSTM LM.
 To the best of our knowledge, this work represents the first methodology and empirical study of integrating SG-MCMC, group sparse prior and network pruning together for learning NN ensembles.

Wir schlagen eine neuartige Methode vor, die SG-MCMC-Sampling, Gruppensparsamkeit und Netzwerk-Pruning integriert, um ein Sparse Structured Ensemble (SSE) mit verbesserter Leistung und deutlich geringeren Kosten als traditionelle Methoden zu lernen.

Die Autoren schlagen ein Verfahren zur Erzeugung eines Ensembles von spärlich strukturierten Modellen vor.

Ein neuer Rahmen für das Training von neuronalen Netzwerken, der SG-MCMC-Methoden innerhalb des Deep Learning verwendet und dann die Recheneffizienz durch Gruppensparsamkeit und Pruning erhöht.

In diesem Beitrag wird die Verwendung von FNN und LSTMs untersucht, um die Durchschnittsbildung von bayesianischen Modellen rechnerisch machbar zu machen und die durchschnittliche Modellleistung zu verbessern.


This paper introduces a new framework for data efficient and versatile learning. Specifically:
 1) We develop ML-PIP, a general framework for Meta-Learning approximate Probabilistic Inference for Prediction. ML-PIP extends existing probabilistic interpretations of meta-learning to cover a broad class of methods.
 2) We introduce \Versa{}, an instance of the framework employing a flexible and versatile amortization network that takes few-shot learning datasets as inputs, with arbitrary numbers of shots, and outputs a distribution over task-specific parameters in a single forward pass. \Versa{} substitutes optimization at test time with forward passes through inference networks, amortizing the cost of inference and relieving the need for second derivatives during training.
 3) We evaluate \Versa{} on benchmark datasets where the method sets new state-of-the-art results, and can handle arbitrary number of shots, and for classification, arbitrary numbers of classes at train and test time. The power of the approach is then demonstrated through a challenging few-shot ShapeNet view reconstruction task.

Neuartiger Rahmen für das Meta-Lernen, der eine breite Klasse bestehender Few-Shot Lernmethoden vereinheitlicht und erweitert. Erzielt eine starke Leistung bei Few-Shot Learning Benchmarks, ohne iterative Testzeitinferenz zu benötigen.

Diese Arbeit befasst sich mit dem Few-Shot Learning aus der Sicht der probabilistischen Inferenz und erreicht den neuesten Stand der Technik trotz eines einfacheren Aufbaus als viele Wettbewerber.


In recent years, softmax together with its fast approximations has become the de-facto loss function for deep neural networks with multiclass predictions. However, softmax is used in many problems that do not fully fit the multiclass framework and where the softmax assumption of mutually exclusive outcomes can lead to biased results. This is often the case for applications such as language modeling, next event prediction and matrix factorization, where many of the potential outcomes are not mutually exclusive, but are more likely to be independent conditionally on the state. To this end, for the set of problems with positive and unlabeled data, we propose a relaxation of the original softmax formulation, where, given the observed state, each of the outcomes are conditionally independent but share a common set of negatives. Since we operate in a regime where explicit negatives are missing, we create an adversarially-trained model of negatives and derive a new negative sampling and weighting scheme which we denote as Cooperative Importance Sampling (CIS). We show empirically the advantages of our newly introduced negative sampling scheme by pluging it in the Word2Vec algorithm and benching it extensively against other negative sampling schemes on both language modeling and matrix factorization tasks and show large lifts in performance.

Definition eines sich teilweise gegenseitig ausschließenden Softmax-Verlustes für positive Daten und Implementierung eines kooperativen Stichprobenverfahrens.

In diesem Beitrag wird Cooperative Importance Sampling vorgestellt, um das Problem zu lösen, dass die sich gegenseitig ausschließende Annahme der traditionellen Softmax verzerrt ist, wenn negative Stichproben nicht explizit definiert sind.

In dieser Arebit werden PMES-Methoden vorgeschlagen, um die Annahme des ausschließlichen Ergebnisses bei Softmax-Verlusten zu lockern, und es wird der empirische Nutzen zur Verbesserung von Einbettungsmodellen des Typs word2vec aufgezeigt.


Over the past few years, various tasks involving videos such as classification, description, summarization and question answering have received a lot of attention. Current models for these tasks compute an encoding of the video by treating it as a sequence of images and going over every image in the sequence, which becomes computationally expensive for longer videos. In this paper, we focus on the task of video classification and aim to reduce the computational cost by using the idea of distillation. Specifically, we propose a Teacher-Student network wherein the teacher looks at all the frames in the video but the student looks at only a small fraction of the frames in the video. The idea is to then train the student to minimize  (i)  the difference between the final representation computed by the student and the teacher and/or (ii) the difference between the distributions predicted by the teacher and the student. This smaller student network which involves fewer computations but still learns to mimic the teacher can then be employed at inference time for video classification. We experiment with the YouTube-8M dataset and show  that the proposed student network can reduce the inference time by upto 30% with a negligent drop in the performance.

Lehrer-Schüler Framework für eine effiziente Videoklassifizierung mit weniger Bildern.

In der Arbeit wird eine Idee vorgeschlagen, aus einem vollständigen Videoklassifizierungsmodell ein kleines Modell zu destillieren, das nur eine geringere Anzahl von Einzelbildern erhält.

Die Autoren stellen ein Lehrer-Schüler-Netzwerk zur Lösung von Videoklassifizierungsproblemen vor und schlagen serielle und parallele Trainingsalgorithmen vor, um die Rechenkosten zu reduzieren.


Deep generative models have achieved impressive success in recent years. Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), as powerful frameworks for deep generative model learning, have largely been considered as two distinct paradigms and received extensive independent studies respectively. This paper aims to establish formal connections between GANs and VAEs through a new formulation of them. We interpret sample generation in GANs as performing posterior inference, and show that GANs and VAEs involve minimizing KL divergences of respective posterior and inference distributions with opposite directions, extending the two learning phases of classic wake-sleep algorithm, respectively. The unified view provides a powerful tool to analyze a diverse set of existing model variants, and enables to transfer techniques across research lines in a principled way. For example, we apply the importance weighting method in VAE literatures for improved GAN learning, and enhance VAEs with an adversarial mechanism that leverages generated samples. Experiments show generality and effectiveness of the transfered techniques.

Eine einheitliche statistische Sicht auf die breite Klasse der tiefen generativen Modelle.

Die Arbeit entwickelt einen Rahmen, in dem GAN-Algorithmen als eine Form der Variationsinferenz auf einem generativen Modell interpretiert werden, das eine Indikatorvariable rekonstruiert, die angibt, ob eine Stichprobe zu den wahren generativen Datenverteilungen gehört.


Deep neural networks have demonstrated promising prediction and classification performance on many healthcare applications. However, the interpretability of those models are often lacking. On the other hand, classical interpretable models such as rule lists or decision trees do not lead to the same level of accuracy as deep neural networks and can often be too complex to interpret (due to the potentially large depth of rule lists). In this work, we present PEARL,  Prototype lEArning via Rule Lists, which iteratively uses rule lists to guide a neural network to learn representative data prototypes. The resulting prototype neural network provides  accurate prediction, and the prediction can be easily explained by  prototype and its guiding rule lists. Thanks to the prediction power of neural networks, the rule lists from				 prototypes are more concise and hence provide better interpretability. On two real-world electronic healthcare records (EHR) datasets, PEARL consistently outperforms all baselines across both datasets, especially achieving performance improvement over conventional rule learning by up to 28% and over prototype learning by up to 3%. Experimental results also show the resulting interpretation of PEARL is  simpler than the standard rule learning.

Ein Verfahren, das das Lernen von Regeln und das Lernen von Prototypen kombiniert.

Es wird ein neuer Framework für interpretierbare Vorhersagen vorgestellt, der regelbasiertes Lernen, Prototyp-Lernen und NNs kombiniert und besonders auf longitudinale Daten anwendbar ist.

Diese Arbeit zielt darauf ab, den Mangel an Interpretierbarkeit von Deep Learning Modellen zu beheben, und schlägt Prototype lEArning via Rule Lists (PEARL) vor, das Regellernen und Prototyplernen kombiniert, um eine genauere Klassifizierung zu erreichen und die Aufgabe der Interpretierbarkeit zu vereinfachen.


Generative Adversarial Networks (GANs) are powerful tools for realistic image generation. However, a major drawback of GANs is that they are especially hard to train, often requiring large amounts of data and long training time. In this paper we propose the Deli-Fisher GAN, a GAN that generates photo-realistic images by enforcing structure on the latent generative space using similar approaches in \cite{deligan}. The structure of the latent space we consider in this paper is modeled as a mixture of Gaussians, whose parameters are learned in the training process. Furthermore, to improve stability and efficiency, we use the Fisher Integral Probability Metric as the divergence measure in our GAN model, instead of the Jensen-Shannon divergence. We show by experiments that the Deli-Fisher GAN performs better than DCGAN, WGAN, and the Fisher GAN as measured by inception score.

In diesem Beitrag wird ein neues Generatives Adversariales Netzwerk vorgeschlagen, das stabiler und effizienter ist und bessere Bilder erzeugt als die bisherigen .

Diese Arbeit kombiniert Fisher-GAN und Deli-GAN.

Diese Arbeit kombiniert Deli-GAN, das eine gemischte Prioritätsverteilung im latenten Raum hat, und Fisher GAN, das Fisher IPM anstelle von JSD als Ziel verwendet.


Recent work on encoder-decoder models for sequence-to-sequence mapping has shown that integrating both temporal and spatial attentional mechanisms into neural networks increases the performance of the system substantially. We report on a new modular network architecture that applies an attentional mechanism not on temporal and spatial regions of the input, but on sensor selection for multi-sensor setups. This network called the sensor transformation attention network (STAN) is evaluated in scenarios which include the presence of natural noise or synthetic dynamic noise. We demonstrate how the attentional signal responds dynamically to changing noise levels and sensor-specific noise, leading to reduced word error rates (WERs) on both audio and visual tasks using TIDIGITS and GRID; and also on CHiME-3, a multi-microphone real-world noisy dataset. The improvement grows as more channels are corrupted as demonstrated on the CHiME-3 dataset. Moreover, the proposed STAN architecture naturally introduces a number of advantages including ease of removing sensors from existing architectures, attentional interpretability, and increased robustness to a variety of noise environments.

Wir stellen eine modulare Multisensornetzwerk-Architektur mit einem Aufmerksamkeitsmechanismus vor, der eine dynamische Sensorauswahl auf realen, verrauschten Daten von CHiME-3 ermöglicht.

Eine generische neuronale Architektur, die in der Lage ist, die Aufmerksamkeit zu erlernen, die den verschiedenen Eingangskanälen in Abhängigkeit von der relativen Qualität der einzelnen Sensoren im Vergleich zu den anderen zukommen muss.

Betrachtet die Verwendung von Aufmerksamkeit für die Sensor- oder Kanalauswahl mit Ergebnissen zu TIDIGITS und GRID, die einen Vorteil von Aufmerksamkeit gegenüber der Zusammenstellung von Merkmalen zeigen.


Massive data exist among user local platforms that usually cannot support deep neural network (DNN) training due to computation and storage resource constraints. Cloud-based training schemes provide beneficial services but suffer from potential privacy risks due to excessive user data collection. To enable cloud-based DNN training while protecting the data privacy simultaneously, we propose to leverage the intermediate representations of the data, which is achieved by splitting the DNNs and deploying them separately onto local platforms and the cloud. The local neural network (NN) is used to generate the feature representations. To avoid local training and protect data privacy, the local NN is derived from pre-trained NNs. The cloud NN is then trained based on the extracted intermediate representations for the target learning task. We validate the idea of DNN splitting by characterizing the dependency of privacy loss and classification accuracy on the local NN topology for a convolutional NN (CNN) based image classification task. Based on the characterization, we further propose PrivyNet to determine the local NN topology, which optimizes the accuracy of the target learning task under the constraints on privacy loss, local computation, and storage. The efficiency and effectiveness of PrivyNet are demonstrated with CIFAR-10 dataset.

Um ein Cloud-basiertes DNN-Training zu ermöglichen und gleichzeitig den Datenschutz zu gewährleisten, schlagen wir vor, die Zwischendaten-Darstellungen zu nutzen, indem die DNNs aufgeteilt und separat auf lokalen Plattformen und in der Cloud bereitgestellt werden.

In diesem Beitrag wird eine Technik zur Privatisierung von Daten durch das Erlernen einer Merkmalsrepräsentation vorgeschlagen, die für die Bildrekonstruktion schwierig, aber für die Bildklassifizierung hilfreich ist.


Generative Adversarial Networks (GANs) have shown remarkable success as a framework for training models to produce realistic-looking data. In this work, we propose a Recurrent GAN (RGAN) and Recurrent Conditional GAN (RCGAN) to produce realistic real-valued multi-dimensional time series, with an emphasis on their application to medical data. RGANs make use of recurrent neural networks (RNNs) in the generator and the discriminator. In the case of RCGANs, both of these RNNs are conditioned on auxiliary information. We demonstrate our models in a set of toy datasets, where we show visually and quantitatively (using sample likelihood and maximum mean discrepancy) that they can successfully generate realistic time-series. We also describe novel evaluation methods for GANs, where we generate a synthetic labelled training dataset, and evaluate on a real test set the performance of a model trained on the synthetic data, and vice-versa. We illustrate with these metrics that RCGANs can generate time-series data useful for supervised training, with only minor degradation in performance on real test data. This is demonstrated on digit classification from ‘serialised’ MNIST and by training an early warning system on a medical dataset of 17,000 patients from an intensive care unit. We further discuss and analyse the privacy concerns that may arise when using RCGANs to generate realistic synthetic medical time series data, and demonstrate results from differentially private training of the RCGAN.

Bedingte rekurrente GANs für die Generierung reellwertiger medizinischer Sequenzen, mit neuartigen Bewertungsansätzen und einer empirischen Datenschutzanalyse.

Schlägt vor, synthetische Daten, die von GANs generiert werden, als Ersatz für persönlich identifizierbare Daten beim Training von ML-Modellen für datenschutzsensible Anwendungen zu verwenden.

Die Autoren schlagen eine neuartige rekurrente GAN-Architektur vor, die kontinuierliche Domänensequenzen erzeugt, und evaluieren sie anhand mehrerer synthetischer Aufgaben und einer Aufgabe mit ICU-Zeitreihen.

Schlägt vor, RGANs und RCGANs zu verwenden, um synthetische Sequenzen aus aktuellen Daten zu erzeugen.


Emphasis effects – visual changes that make certain elements more
 prominent – are commonly used in information visualization to draw
 the user’s attention or to indicate importance. Although theoretical
 frameworks of emphasis exist (that link visually diverse emphasis
 effects through the idea of visual prominence compared to background
 elements), most metrics for predicting how emphasis effects
 will be perceived by users come from abstract models of human
 vision which may not apply to visualization design. In particular,
 it is difficult for designers to know, when designing a visualization,
 how different emphasis effects will compare and what level of one
 effect is equivalent to what level of another. To address this gap,
 we carried out two studies that provide empirical evidence about
 how users perceive different emphasis effects, using three visual
 variables (colour, size, and blur/focus) and eight strength levels.
 Results from gaze tracking, mouse clicks, and subjective responses
 show that there are significant differences between visual variables
 and between levels, and allow us to develop an initial understanding
 of perceptual equivalence. We developed a model from the data in
 our first study, and used it to predict the results in the second; the
 model was accurate, with high correlations between predictions and
 real values. Our studies and empirical models provide valuable new
 information for designers who want to understand and control how
 emphasis effects will be perceived by users.

Unsere Studien und empirischen Modelle liefern wertvolle neue Informationen für Designer, die verstehen und kontrollieren wollen, wie Betonungseffekte von Nutzern wahrgenommen werden.

In diesem Beitrag wird untersucht, welche visuellen Hervorhebungen bei der Datenvisualisierung schneller wahrgenommen werden und wie verschiedene Hervorhebungsmethoden im Vergleich zueinander abschneiden.

Zwei Studien über die Wirksamkeit von Betonungseffekten, von denen eine die Niveaus nützlicher Unterschiede bewertet, und eine weitere, die für eine ökologisch validere Untersuchung tatsächlich unterschiedliche Visualisierungen verwendet.


Memory Network based models have shown a remarkable progress on the task of relational reasoning.
 Recently, a simpler yet powerful neural network module called Relation Network (RN) has been introduced.
 Despite its architectural simplicity, the time complexity of relation network grows quadratically with data, hence limiting its application to tasks with a large-scaled memory.
 We introduce Related Memory Network, an end-to-end neural network architecture exploiting both memory network and relation network structures.
 We follow memory network's four components while each component operates similar to the relation network without taking a pair of objects.
 As a result, our model is as simple as RN but the computational complexity is reduced to linear time.
 It achieves the state-of-the-art results in jointly trained bAbI-10k story-based question answering and  bAbI dialog dataset.

Eine einfache, auf dem Gedächtnisnetzwerk (MemNN) und dem Beziehungsnetzwerk (RN) basierende Argumentationsarchitektur, die die Zeitkomplexität im Vergleich zum RN reduziert und ein State-of-the-Art-Ergebnis für die auf bAbI-Geschichten basierende QA und den bAbI-Dialog erzielt.

Einführung des Related Memory Network (RMN), einer Verbesserung gegenüber dem Relationship Network (RN).


We investigate in this paper the architecture of deep convolutional networks. Building on existing state of the art models, we propose a reconfiguration of the model parameters into several parallel branches at the global network level, with each branch being a standalone CNN. We show that this arrangement is an efficient way to significantly reduce the number of parameters while at the same time improving the performance. The use of branches brings an additional form of regularization. In addition to splitting the parameters into parallel branches, we propose a tighter coupling of these branches by averaging their log-probabilities. The tighter coupling favours the learning of better representations, even at the level of the individual branches, as compared to when each branch is trained independently. We refer to this branched architecture as "coupled ensembles". The approach is very generic and can be applied with almost any neural network architecture. With coupled ensembles of DenseNet-BC and parameter budget of 25M, we obtain error rates of 2.92%, 15.68% and 1.50% respectively on CIFAR-10, CIFAR-100 and SVHN tasks. For the same parameter budget, DenseNet-BC has an error rate of 3.46%, 17.18%, and 1.8% respectively.   With ensembles of coupled ensembles, of DenseNet-BC networks, with 50M total parameters, we obtain error rates of 2.72%, 15.13% and 1.42% respectively on these tasks.

Wir zeigen, dass die Aufteilung eines neuronalen Netzes in parallele Zweige die Leistung verbessert und dass die richtige Kopplung der Zweige die Leistung noch weiter erhöht.

In der Arbeit wird eine Rekonfiguration des bestehenden CNN-Modells nach dem Stand der Technik vorgeschlagen, die eine neue Verzweigungsarchitektur mit besserer Leistung verwendet.

Diese Arbeit zeigt die Vorteile der Parametereinsparung durch gekoppeltes Ensembling.

Stellt eine Deep-Network-Architektur vor, die Daten mit Hilfe mehrerer paralleler Zweige verarbeitet und die Ergebnisse dieser Zweige kombiniert, um die endgültigen Ergebnisse zu berechnen.


Convolutional Neural Networks (CNN) are very popular in many fields including computer vision, speech recognition, natural language processing, to name a few. Though deep learning leads to groundbreaking performance in these domains, the networks used are very demanding computationally and are far from real-time even on a GPU, which is not power efficient and therefore does not suit low power systems such as mobile devices. To overcome this challenge, some solutions have been proposed for quantizing the weights and activations of these networks, which accelerate the runtime significantly. Yet, this acceleration comes at the cost of a larger error. The NICE method proposed in this work trains quantized neural networks by noise injection and a learned clamping, which improve the accuracy. This leads to state-of-the-art results on various regression and classification tasks, e.g., ImageNet classification with architectures such as ResNet-18/34/50 with low as 3-bit weights and 3 -bit activations. We implement the proposed solution on an FPGA to demonstrate its applicability for low power real-time applications.

Kombination von Störungsinjektion, schrittweiser Quantisierung und Aktivierungsklemmung zur Erzielung modernster 3, 4 und 5 Bit Quantisierung.

Schlägt vor, während des Trainings Störungen zu injizieren und die Parameterwerte in einer Schicht sowie die Aktivierungsausgabe in der Quantisierung des neuronalen Netzes zu klammern.

Eine Methode zur Quantisierung von tiefen neuronalen Netzen für Klassifizierung und Regression, die Störungsinjektion, Clamping mit gelernten maximalen Aktivierungen und graduelle Blockquantisierung verwendet, um gleichwertige oder bessere Leistungen als die modernsten Methoden zu erzielen.


In complex transfer learning scenarios new tasks might not be tightly linked to previous tasks. Approaches that transfer information contained only in the final parameters of a source model will therefore struggle. Instead, transfer learning at at higher level of abstraction is needed. We propose Leap, a framework that achieves this by transferring knowledge across learning processes. We associate each task with a manifold on which the training process travels from initialization to final parameters and construct a meta-learning objective that minimizes the expected length of this path. Our framework leverages only information obtained during training and can be computed on the fly at negligible cost. We demonstrate that our framework outperforms competing methods, both in meta-learning and transfer learning, on a set of computer vision tasks. Finally, we demonstrate that Leap can transfer knowledge across learning processes in demanding reinforcement learning environments (Atari) that involve millions of gradient steps.

Wir schlagen Leap vor, ein System, das Wissen über Lernprozesse hinweg überträgt, indem es die erwartete Distanz minimiert, die der Trainingsprozess auf der Verlustfläche einer Aufgabe zurücklegt.

Der Artikel schlägt ein neuartiges Meta-Lernziel vor, das darauf abzielt, bei der Bewältigung von Aufgabensammlungen, die eine beträchtliche Vielfalt zwischen den einzelnen Aufgaben aufweisen, die modernsten Ansätze zu übertreffen.


Given an existing trained neural network, it is often desirable to learn new capabilities without hindering performance of those already learned. Existing approaches either learn sub-optimal solutions, require joint training, or incur a substantial increment in the number of parameters for each added task, typically as many as the original network. We propose a method called Deep Adaptation Networks (DAN) that constrains newly learned filters to be linear combinations of existing ones. DANs preserve performance on the original task, require a fraction (typically 13%) of the number of parameters compared to standard fine-tuning procedures and converge in less cycles of training to a comparable or better level of performance. When coupled with standard network quantization techniques, we further reduce the parameter cost to around 3% of the original with negligible or no loss in accuracy. The learned architecture can be controlled to switch between various learned representations, enabling a single network to solve a task from multiple different domains. We conduct extensive experiments showing the effectiveness of our method on a range of image classification tasks and explore different aspects of its behavior.

Eine Alternative zum Transfer-Lernen, die schneller lernt, viel weniger Parameter benötigt (3-13 %), in der Regel bessere Ergebnisse erzielt und die Leistung bei alten Aufgaben genau beibehält.

Controller Module für inkrementelles Lernen auf Bildklassifizierungsdatensätzen.


High throughput and low latency inference of deep neural networks are critical for the deployment of deep learning applications. This paper presents a general technique toward 8-bit low precision inference of convolutional neural networks, including 1) channel-wise scale factors of weights, especially for depthwise convolution, 2) Winograd convolution, and 3) topology-wise 8-bit support. We experiment the techniques on top of a widely-used deep learning framework. The 8-bit optimized model is automatically generated with a calibration process from FP32 model without the need of fine-tuning or retraining. We perform a systematical and comprehensive study on 18 widely-used convolutional neural networks and demonstrate the effectiveness of 8-bit low precision inference across a wide range of applications and use cases, including image classification, object detection, image segmentation, and super resolution. We show that the inference throughput
 and latency are improved by 1.6X and 1.5X respectively with minimal within 0.6%1to no loss in accuracy from FP32 baseline. We believe the methodology can provide the guidance and reference design of 8-bit low precision inference for other frameworks. All the code and models will be publicly available soon.

Wir stellen ein allgemeines Verfahren zur 8-Bit Inferenz mit niedriger Präzision für Convolutional Neural Networks vor.

In diesem Beitrag wird ein System zur automatischen Quantisierung der vortrainierten CNN-Modelle entwickelt.


Recent approaches have successfully demonstrated the benefits of learning the parameters of shallow networks in hyperbolic space. We extend this line of work by imposing hyperbolic geometry on the embeddings used to compute the ubiquitous attention mechanisms for different neural networks architectures. By only changing the geometry of embedding of object representations, we can use the embedding space more efficiently without increasing the number of parameters of the model. Mainly as the number of objects grows exponentially for any semantic distance from the query, hyperbolic geometry  --as opposed to Euclidean geometry-- can encode those objects without having any interference. Our method shows improvements in generalization on neural machine translation on WMT'14 (English to German), learning on graphs (both on synthetic and real-world graph tasks) and visual question answering (CLEVR) tasks while keeping the neural representations compact.

Wir schlagen vor, induktive Verzerrungen und Operationen aus der hyperbolischen Geometrie einzubeziehen, um den Aufmerksamkeitsmechanismus der neuronalen Netze zu verbessern.

In dieser Arbeit wird die in den Aufmerksamkeitsmechanismen verwendete Punktproduktähnlichkeit durch die negative hyperbolische Distanz ersetzt und auf das bestehende Transformer-Modell, Graph-Attention-Networks und Relation-Networks angewendet.

Die Autoren schlagen einen neuartigen Ansatz zur Verbesserung der relationalen Aufmerksamkeit vor, indem sie die Anpassungs- und Aggregationsfunktionen so ändern, dass sie hyperbolische Geometrie verwenden.


We present a method for evaluating the sensitivity of deep reinforcement learning (RL) policies. We also formulate a zero-sum dynamic game for designing robust deep reinforcement learning policies. Our approach mitigates the brittleness of policies when agents are trained in a simulated environment and are later exposed to the real world where it is hazardous to employ RL policies. This framework for training deep RL policies involve a zero-sum  dynamic game against an adversarial agent, where the goal is to drive the system dynamics to a saddle region. Using a variant of the guided policy search algorithm, our agent learns to adopt robust policies that require less samples for learning the dynamics and performs better than the GPS algorithm. Without loss of generality, we demonstrate that deep RL policies trained in this fashion will be maximally robust to a ``worst" possible adversarial disturbances.

In diesem Beitrag wird gezeigt, wie die H-Infinity-Control-Theorie dazu beitragen kann, robuste, tiefgreifende Strategien für Robotermotoren zu entwickeln.

Es wird vorgeschlagen, Elemente der robusten Kontrolle in die Forschung zu gelenkten Strategien einzubeziehen, um eine Methode zu entwickeln, die gegenüber Störungen und Modellfehlanpassungen widerstandsfähig ist.

Die Arbeit stellt eine Methode zur Bewertung der Sensitivität und Robustheit von Deep RL Strategien vor und schlägt einen dynamischen Spielansatz zum Erlernen robuster Strategien vor.


Deep networks have recently been shown to be vulnerable to universal perturbations: there exist very small image-agnostic perturbations that cause most natural images to be misclassified by such classifiers. In this paper, we provide a quantitative analysis of the robustness of classifiers to universal perturbations, and draw a formal link between the robustness to universal perturbations, and the geometry of the decision boundary. Specifically, we establish theoretical bounds on the robustness of classifiers under two decision boundary models (flat and curved models). We show in particular that the robustness of deep networks to universal perturbations is driven by a key property of their curvature: there exist shared directions along which the decision boundary of deep networks is systematically positively curved. Under such conditions, we prove the existence of small universal perturbations. Our analysis further provides a novel geometric method for computing universal perturbations, in addition to explaining their properties.

Analyse der Anfälligkeit von Klassifikatoren für universelle Störungen und Beziehung zur Krümmung der Entscheidungsgrenze.

Die Arbeit bietet eine interessante Analyse, die die Geometrie der Entscheidungsgrenzen von Klassifizierern mit kleinen universellen Störfaktoren in Verbindung bringt.

In diesem Beitrag werden universelle Störungen erörtert - Störungen, die einen trainierten Klassifikator in die Irre führen können, wenn sie den meisten Eingabedatenpunkten hinzugefügt werden.

Es werden Modelle entwickelt, die versuchen, die Existenz universeller Störungen zu erklären, die neuronale Netze täuschen.


Behavioral skills or policies for autonomous agents are conventionally learned from reward functions, via reinforcement learning, or from demonstrations, via imitation learning. However, both modes of task specification have their disadvantages: reward functions require manual engineering, while demonstrations require a human expert to be able to actually perform the task in order to generate the demonstration. Instruction following from natural language instructions provides an appealing alternative: in the same way that we can specify goals to other humans simply by speaking or writing, we would like to be able to specify tasks for our machines. However, a single instruction may be insufficient to fully communicate our intent or, even if it is, may be insufficient for an autonomous agent to actually understand how to perform the desired task. In this work, we propose an interactive formulation of the task specification problem, where iterative language corrections are provided to an autonomous agent, guiding it in acquiring the desired skill. Our proposed language-guided policy learning algorithm can integrate an instruction and a sequence of corrections to acquire new skills very quickly. In our experiments, we show that this method can enable a policy to follow instructions and corrections for simulated navigation and manipulation tasks, substantially outperforming direct, non-interactive instruction following.

Wir schlagen eine Meta-Lernmethode für die interaktive Korrektur von Richtlinien mit natürlicher Sprache vor.

Diese Arbeit bietet ein Meta Learning Framework, das zeigt, wie neue Aufgaben in einer interaktiven Umgebung gelernt werden können. Jede Aufgabe wird durch eine Reinforcement Learning Umgebung gelernt, und dann wird die Aufgabe durch die Beobachtung neuer Anweisungen aktualisiert.

In diesem Beitrag wird Agenten beigebracht, Aufgaben mittels natürlichsprachlicher Anweisungen in einem iterativen Prozess zu erledigen.


Deep generative models such as Generative Adversarial Networks (GANs) and
 Variational Auto-Encoders (VAEs) are important tools to capture and investigate
 the properties of complex empirical data. However, the complexity of their inner
 elements makes their functionment challenging to assess and modify. In this
 respect, these architectures behave as black box models. In order to better
 understand the function of such networks, we analyze their modularity based on
 the counterfactual manipulation of their internal variables. Our experiments on the
 generation of human faces with VAEs and GANs support that modularity between
 activation maps distributed over channels of generator architectures is achieved
 to some degree, can be used to better understand how these systems operate and allow meaningful transformations of the generated images without further training.
 erate and edit the content of generated images.

Wir untersuchen die Modularität von tiefen generativen Modellen.

Die Arbeit bietet eine Möglichkeit, die modulare Struktur des tiefen generativen Modells zu untersuchen, mit dem Schlüsselkonzept, über Kanäle von Generatorarchitekturen zu verteilen.


Relational databases store a significant amount of the worlds data. However, accessing this data currently requires users to understand a query language such as SQL. We propose Seq2SQL, a deep neural network for translating natural language questions to corresponding SQL queries. Our model uses rewards from in the loop query execution over the database to learn a policy to generate the query, which contains unordered parts that are less suitable for optimization via cross entropy loss. Moreover, Seq2SQL leverages the structure of SQL to prune the space of generated queries and significantly simplify the generation problem. In addition to the model, we release WikiSQL, a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables fromWikipedia that is an order of magnitude larger than comparable datasets. By applying policy based reinforcement learning with a query execution environment to WikiSQL, Seq2SQL outperforms a state-of-the-art semantic parser, improving execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.

Wir stellen Seq2SQL vor, das Fragen in SQL-Abfragen übersetzt, indem es Belohnungen aus der Online-Abfrageausführung nutzt, und WikiSQL, einen SQL-Tabellen-/Fragen-/Abfragedatensatz, der um Größenordnungen größer ist als bestehende Datensätze.

Ein neuer semantischer Parsing-Datensatz, der sich auf die Generierung von SQL aus natürlicher Sprache unter Verwendung eines auf Reinforcement Learning basierenden Modells konzentriert.


We introduce Explainable Adversarial Learning, ExL, an approach for training neural networks that are intrinsically robust to adversarial attacks. We find that the implicit generative modeling of random noise with the same loss function used during posterior maximization, improves a model's understanding of the data manifold furthering adversarial robustness. We prove our approach's efficacy and provide a simplistic visualization tool for understanding adversarial data, using Principal Component Analysis. Our analysis reveals that adversarial robustness, in general, manifests in models with higher variance along the high-ranked principal components. We show that models learnt with our approach perform remarkably well against a wide-range of attacks. Furthermore, combining ExL with state-of-the-art adversarial training extends the robustness of a model, even beyond what it is adversarially trained for, in both white-box and black-box attack scenarios.

Die Modellierung von Störungen am Eingang während des diskriminierenden Trainings verbessert die Robustheit gegenüber nachteiligen Einflüssen. Vorschlag einer PCA-basierten Bewertungsmetrik für die adversial Robustheit.

Diese Arbeit schlägt ExL vor, eine adversarische Trainingsmethode, die multiplizierte Störungen verwendet und sich als hilfreich bei der Abwehr von Blackbox-Angriffen auf drei Datensätze erweist.

In diesem Beitrag werden multiplikative Störungen N in die Trainingsdaten aufgenommen, um adversial Robustheit zu erreichen, wenn sowohl auf die Modellparameter theta als auch auf den Störungen selbst trainiert wird.


We propose a method which can visually explain the classification decision of deep neural networks (DNNs). There are many proposed methods in machine learning and computer vision seeking to clarify the decision of machine learning black boxes, specifically DNNs.   All of these methods try to gain insight into why the network "chose class A" as an answer. Humans, when searching for explanations, ask two types of questions. The first question is, "Why did you choose this answer? " The second question asks, "Why did you not choose answer B over A?" The previously proposed methods are either not able to provide the latter directly or efficiently.

 We introduce a method capable of answering the second question both directly and efficiently. In this work, we limit the inputs to be images. In general, the proposed method generates explanations in the input space of any model capable of efficient evaluation and gradient evaluation. We provide results, showing the superiority of this approach for gaining insight into the inner representation of machine learning models.

Eine Methode zur Beantwortung der Frage "Warum nicht Klasse B?" zur Erklärung von tiefen Netzen.

Die Arbeit schlägt einen Ansatz vor, der kontrastive visuelle Erklärungen für tiefe neuronale Netze liefert.


We flip the usual approach to study invariance and robustness of neural networks by considering the non-uniqueness and instability of the inverse mapping. We provide theoretical and numerical results on the inverse of ReLU-layers. First, we derive a necessary and sufficient condition on the existence of invariance that provides a geometric interpretation. Next, we move to robustness via analyzing local effects on the inverse. To conclude, we show how this reverse point of view not only provides insights into key effects, but also enables to view adversarial examples from different perspectives.

Wir analysieren die Invertierbarkeit von tiefen neuronalen Netzen, indem wir die Präimages von ReLU-Schichten und die Stabilität der Inverse untersuchen.

Diese Arbeit untersucht das Volumen der Vorabbildung der Aktivierung eines ReLU-Netzwerks in einer bestimmten Schicht und baut auf der stückweisen Linearität der Forward Funktion eines ReLU-Netzwerks auf.

Dieses Arbeit stellt eine Analyse der inversen Invarianz von ReLU-Netzen vor und liefert obere Schranken für singuläre Werte eines Zugnetzes.


While deep learning has led to remarkable results on a number of challenging problems, researchers have discovered a vulnerability of neural networks in adversarial settings, where small but carefully chosen perturbations to the input can make the models produce extremely inaccurate outputs. This makes these models particularly unsuitable for safety-critical application domains (e.g. self-driving cars) where robustness is extremely important. Recent work has shown that augmenting training with adversarially generated data provides some degree of robustness against test-time attacks. In this paper we investigate how this approach scales as we increase the computational budget given to the defender. We show that increasing the number of parameters in adversarially-trained models increases their robustness, and in particular that ensembling smaller models while adversarially training the entire ensemble as a single model is a more efficient way of spending said budget than simply using a larger single model. Crucially, we show that it is the adversarial training of the ensemble, rather than the ensembling of adversarially trained models, which provides robustness.

Das adversial Training von Ensembles bietet eine Robustheit gegenüber adversial Beispielen, die über die von gegnerisch trainierten Modellen und unabhängig trainierten Ensembles beobachtete Robustheit hinausgeht.

Es wird vorgeschlagen, ein Ensemble von Modellen gemeinsam zu trainieren, wobei in jedem Zeitschritt eine Reihe von Beispielen, die für das Ensemble selbst adversial sind, in das Lernen einbezogen wird.


Multi-task learning (MTL) with neural networks leverages commonalities in tasks to improve performance, but often suffers from task interference which reduces the benefits of transfer. To address this issue we introduce the routing network paradigm, a novel neural network and training algorithm. A routing network is a kind of self-organizing neural network consisting of two components: a router and a set of one or more function blocks. A function block may be any neural network – for example a fully-connected or a convolutional layer. Given an input the router makes a routing decision, choosing a function block to apply and passing the output back to the router recursively, terminating when a fixed recursion depth is reached. In this way the routing network dynamically composes different function blocks for each input. We employ a collaborative multi-agent reinforcement learning (MARL) approach to jointly train the router and function blocks. We evaluate our model against cross-stitch networks and shared-layer baselines on multi-task settings of the MNIST, mini-imagenet, and CIFAR-100 datasets. Our experiments demonstrate a significant improvement in accuracy, with sharper convergence. In addition, routing networks have nearly constant per-task training cost while cross-stitch networks scale linearly with the number of tasks. On CIFAR100 (20 tasks) we obtain cross-stitch performance levels with an 85% average reduction in training time.

Routing-Netzwerke: eine neue Art von neuronalem Netzwerk, das lernt, seine Eingaben adaptiv zu routen, um Multi-Task-Lernen zu ermöglichen.

In dem Beitrag wird vorgeschlagen, ein modulares Netz mit einem Controller zu verwenden, der bei jedem Zeitschritt Entscheidungen über den nächsten zu verwendenden Knoten trifft.

Die Arbeit stellt eine neuartige Formulierung für das Lernen der optimalen Architektur eines neuronalen Netzes in einem Multi-Task-Learning-Rahmen vor, indem es Multi-Agenten-Reinforcement Learning verwendet, um eine Strategie zu finden, und zeigt eine Verbesserung gegenüber fest kodierten Architekturen mit geteilten Schichten.


We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization. However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected $L_0$ regularized objective is differentiable with respect to the distribution parameters. We further propose the \emph{hard concrete} distribution for the gates, which is obtained by ``stretching'' a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.

Wir zeigen, wie man die erwartete L_0-Norm parametrischer Modelle mit Gradientenabstieg optimieren kann, und führen eine neue Verteilung ein, die das Hard Gating erleichtert.

Die Autoren stellen einen gradientenbasierten Ansatz zur Minimierung einer Zielfunktion mit einem L0 Sparse Penalty vor, um das Erlernen spärlicher neuronaler Netze zu unterstützen.


Recently popularized graph neural networks achieve the state-of-the-art accuracy on a number of standard benchmark datasets for graph-based semi-supervised learning, improving significantly over existing approaches. These architectures alternate between a propagation layer that aggregates the hidden states of the local neighborhood and a fully-connected layer. Perhaps surprisingly, we show that a linear model, that removes all the intermediate fully-connected layers, is still able to achieve a performance comparable to the state-of-the-art models. This significantly reduces the number of parameters, which is critical for semi-supervised learning where number of labeled examples are small. This in turn allows a room for designing more innovative propagation layers. Based on this insight, we propose a novel graph neural network that removes all the intermediate fully-connected layers, and replaces the propagation layers with attention mechanisms that respect the structure of the graph. The attention mechanism allows us to learn a dynamic and adaptive local summary of the neighborhood to achieve more accurate predictions. In a number of experiments on benchmark citation networks datasets, we demonstrate that our approach outperforms competing methods. By examining the attention weights among neighbors, we show that our model provides some interesting insights on how neighbors influence each other.

Wir schlagen eine neuartige, auf Aufmerksamkeit basierende, interpretierbare Graph Neural Network-Architektur vor, die den aktuellen Stand der Technik in Standard-Benchmark-Datensätzen übertrifft.

Die Autoren schlagen zwei Erweiterungen von GCNs vor, indem sie zwischengeschaltete Nichtlinearitäten aus der GCN-Berechnung entfernen und einen Aufmerksamkeitsmechanismus in der Aggregationsschicht hinzufügen.

Die Arbeit schlägt einen halbüberwachten Lernalgorithmus für die Klassifizierung von Graphknoten vor, der von Graph Neural Networks inspiriert ist.


Modern generative models are usually designed to match target distributions directly in the data space, where the intrinsic dimensionality of data can be much lower than the ambient dimensionality. We argue that this discrepancy may contribute to the difficulties in training generative models. We therefore propose to map both the generated and target distributions to the latent space using the encoder of a standard autoencoder, and train the generator (or decoder) to match the target distribution in the latent space. The resulting method, perceptual generative autoencoder (PGA), is then incorporated with maximum likelihood or variational autoencoder (VAE) objective to train the generative model. With maximum likelihood, PGA generalizes the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities. When combined with VAE, PGA can generate sharper samples than vanilla VAE.

Ein Framework für das Training von generativen Modellen auf der Basis von Autoencodern, mit nicht-adversen Verlusten und uneingeschränkten neuronalen Netzwerkarchitekturen.

In diesem Beitrag werden Autoencoder verwendet, um eine Verteilungsanpassung im hochdimensionalen Raum vorzunehmen.


The quality of the representations achieved by embeddings is determined by how well the geometry of the embedding space matches the structure of the data.
 Euclidean space has been the workhorse for embeddings; recently hyperbolic and spherical spaces have gained popularity due to their ability to better embed new types of structured data---such as hierarchical data---but most data is not structured so uniformly.
 We address this problem by proposing learning embeddings in a product manifold combining multiple copies of these model spaces (spherical, hyperbolic, Euclidean), providing a space of heterogeneous curvature suitable for a wide variety of structures.
 We introduce a heuristic to estimate the sectional curvature of graph data and directly determine an appropriate signature---the number of component spaces and their dimensions---of the product manifold.
 Empirically, we jointly learn the curvature and the embedding in the product space via Riemannian optimization.
 We discuss how to define and compute intrinsic quantities such as means---a challenging notion for product manifolds---and provably learnable optimization functions.
 On a range of datasets and reconstruction tasks, our product space embeddings outperform single Euclidean or hyperbolic spaces used in previous works, reducing distortion by 32.55% on a Facebook social network dataset. We learn word embeddings and find that a product of hyperbolic spaces in 50 dimensions consistently improves on baseline Euclidean and hyperbolic embeddings, by 2.6
 points in Spearman rank correlation on similarity tasks
 and 3.4 points on analogy accuracy.

Produktvielfältige Einbettungsräume mit heterogener Krümmung liefern im Vergleich zu traditionellen Einbettungsräumen verbesserte Darstellungen für eine Vielzahl von Strukturen.

Schlägt eine Methode zur Dimensionalitätsreduktion vor, die Daten in eine Produktmannigfaltigkeit von sphärischen, euklidischen und hyperbolischen Mannigfaltigkeiten einbettet. Der Algorithmus basiert auf dem Abgleich der geodätischen Abstände auf der Produktmannigfaltigkeit mit Graphenabständen.


Synthesizing user-intended programs from a small number of input-output exam-
 ples is a challenging problem with several important applications like spreadsheet
 manipulation, data wrangling and code refactoring. Existing synthesis systems
 either completely rely on deductive logic techniques that are extensively hand-
 engineered or on purely statistical models that need massive amounts of data, and in
 general fail to provide real-time synthesis on challenging benchmarks. In this work,
 we propose Neural Guided Deductive Search (NGDS), a hybrid synthesis technique
 that combines the best of both symbolic logic techniques and statistical models.
 Thus, it produces programs that satisfy the provided specifications by construction
 and generalize well on unseen examples, similar to data-driven systems. Our
 technique effectively utilizes the deductive search framework to reduce the learning
 problem of the neural component to a simple supervised learning setup. Further,
 this allows us to both train on sparingly available real-world data and still leverage
 powerful recurrent neural network encoders. We demonstrate the effectiveness
 of our method by evaluating on real-world customer scenarios by synthesizing
 accurate programs with up to 12× speed-up compared to state-of-the-art systems.

Wir integrieren symbolische (deduktive) und statistische (neural-basierte) Methoden, um eine Programmsynthese in Echtzeit mit nahezu perfekter Generalisierung von einem Eingabe-Ausgabe-Beispiel zu ermöglichen.

Die Arbeit präsentiert einen Branch-and-Bound-Ansatz zum Erlernen guter Programme, bei dem ein LSTM verwendet wird, um vorherzusagen, welche Zweige im Suchbaum zu guten Programmen führen sollten.

Schlägt ein System vor, das aus einem einzigen Beispiel Programme synthetisiert, die besser verallgemeinert werden können als der bisherige Stand der Technik.


Variational auto-encoders
 (VAEs) offer a tractable approach when performing approximate inference in otherwise intractable generative models. However, standard VAEs often produce latent codes that are disperse and lack interpretability, thus making the resulting representations unsuitable for auxiliary tasks (e.g. classiﬁcation) and human interpretation. We address these issues by merging ideas from variational auto-encoders and sparse coding, and propose to explicitly model sparsity in the latent space of a VAE with a Spike and Slab prior distribution. We derive the evidence lower bound using a discrete mixture recognition function thereby making approximate posterior inference as computational efﬁcient as in the standard VAE case. With the new approach, we are able to infer truly sparse representations with generally intractable non-linear probabilistic models. We show that these sparse representations are advantageous over standard VAE representations on two benchmark classiﬁcation tasks (MNIST and Fashion-MNIST) by demonstrating improved classiﬁcation accuracy and signiﬁcantly increased robustness to the number of latent dimensions. Furthermore, we demonstrate qualitatively that the sparse elements capture subjectively understandable sources of variation.

Wir untersuchen die Überschneidung von VAEs und Sparse Coding.

Diese Arbeit schlägt eine Erweiterung von VAEs mit spärlichen Prioren und Posterioren vor, um spärliche interpretierbare Repräsentationen zu lernen.


A widely observed phenomenon in deep learning is the degradation problem: increasing
 the depth of a network leads to a decrease in performance on both test and training data. Novel architectures such as ResNets and Highway networks have addressed this issue by introducing various flavors of skip-connections or gating mechanisms. However, the degradation problem persists in the context of plain feed-forward networks. In this work we propose a simple method to address this issue. The proposed method poses the learning of weights in deep networks as a constrained optimization problem where the presence of skip-connections is penalized by Lagrange multipliers. This allows for skip-connections to be introduced during the early stages of training and subsequently phased out in a principled manner. We demonstrate the benefits of such an approach with experiments on MNIST, fashion-MNIST, CIFAR-10 and CIFAR-100 where the proposed method is shown to greatly decrease the degradation effect (compared to plain networks) and is often competitive with ResNets.

Die prinzipielle Einstellung von Skip-Verbindungen verhindert eine Verschlechterung in tiefen Feed-Forward-Netzen.

Die Autoren stellen eine neue Trainingsstrategie, VAN, für das Training sehr tiefer Feed-Forward-Netze ohne Skip-Verbindungen vor.

Die Arbeit stellt eine Architektur vor, die linear zwischen ResNets und Vanilla Deep Nets interpoliert, ohne Verbindungen zu überspringen.


Deep learning is becoming more widespread in its application due to its power in solving complex classification problems. However, deep learning models often require large memory and energy consumption, which may prevent them from being deployed effectively on embedded platforms, limiting their applications. This work addresses the problem by proposing methods {\em Weight Reduction Quantisation} for compressing the memory footprint of the models, including reducing the number of weights and the number of bits to store each weight. Beside, applying with sparsity-inducing regularization, our work focuses on speeding up stochastic variance reduced gradients (SVRG) optimization on non-convex problem. Our method that mini-batch SVRG with $\ell$1 regularization on non-convex problem has faster and smoother convergence rates than SGD by using adaptive learning rates. Experimental evaluation of our approach uses MNIST and CIFAR-10 datasets on LeNet-300-100 and LeNet-5 models, showing our approach can reduce the memory requirements both in the convolutional and fully connected layers by up to 60$\times$ without affecting their test accuracy.

Komprimierung von tiefen neuronalen Netzen, die auf eingebetteten Geräten eingesetzt werden.

Die Autoren stellen einen l-1 regularisierten SVRG-basierten Trainingsalgorithmus vor, der in der Lage ist, viele Gewichte des Netzwerks auf 0 zu setzen.

Diese Arbeit reduziert den Speicherbedarf.


It has been argued that the brain is a prediction machine that continuously learns how to make better predictions about the stimuli received from the external environment. For this purpose, it builds a model of the world around us and uses this model to infer the external stimulus. Predictive coding has been proposed as a mechanism through which the brain might be able to build such a model of the external environment. However, it is not clear how predictive coding can be used to build deep neural network models of the brain while complying with the architectural constraints imposed by the brain. In this paper, we describe an algorithm to build a deep generative model using predictive coding that can be used to infer latent representations about the stimuli received from external environment. Specifically, we used predictive coding to train a deep neural network on real-world images in a unsupervised learning paradigm. To understand the capacity of the network with regards to modeling the external environment, we studied the latent representations generated by the model on images of objects that are never presented to the model during training. Despite the novel features of these objects the model is able to infer the latent representations for them. Furthermore, the reconstructions of the original images obtained from these latent representations preserve the important details of these objects.

Ein auf vorhersehender Kodierung basierender Lernalgorithmus für den Aufbau tiefer neuronaler Netzmodelle des Gehirns.

Der Beitrag befasst sich mit dem Lernen eines generativen neuronalen Netzes unter Verwendung eines prädiktiven Kodierungssystems.


In this paper, we propose deep convolutional generative adversarial networks (DCGAN) that learn to produce a 'mental image' of the input image as internal representation of a certain category of input data distribution.   This mental image is what the DCGAN 'imagines' that the input image might look like under ideal conditions.   The mental image contains a version of the input that is iconic, without any peculiarities that do not contribute to the ideal representation of the input data distribution within a category. A DCGAN learns this association by training an encoder to capture salient features from the original image and a decoder to convert salient features into its associated mental image representation.   Our new approach, which we refer to as a Mental Image DCGAN (MIDCGAN), learns features that are useful for recognizing entire classes of objects, and that this in turn has the benefit of helping single and zero shot recognition.   We demonstrate our approach on object instance recognition and handwritten digit recognition tasks.

Die Erkennung von Objektinstanzen mit adversial Autoencodern wurde mit einem neuartigen "mentalen Bild" durchgeführt, das eine kanonische Repräsentation des Eingangsbildes darstellt.

In dem Beitrag wird eine Methode zum Erlernen von Merkmalen für die Objekterkennung vorgeschlagen, die gegenüber verschiedenen Transformationen des Objekts, insbesondere der Objektposition, invariant ist.

Diese Arbeit untersuchte die Aufgabe der Few-Shot Erkennung mittels eines generierten mentalen Bildes als Zwischendarstellung des Eingangsbildes.