With innovations in architecture design, deeper and wider neural network models deliver improved performance on a diverse variety of tasks.But the increased memory footprint of these models presents a challenge during training, when all intermediate layer activations need to be stored for back-propagation.Limited GPU memory forces practitioners to make sub-optimal choices: either train inefficiently with smaller batches of examples; or limit the architecture to have lower depth and width, and fewer layers at higher spatial resolutions.This work introduces an approximation strategy that significantly reduces a network's memory footprint during training, but has negligible effect on training performance and computational expense.During the forward pass, we replace activations with lower-precision approximations immediately after they have been used by subsequent layers, thus freeing up memory.The approximate activations are then used during the backward pass.This approach limits the accumulation of errors across the forward and backward pass---because the forward computation across the network still happens at full precision, and the approximation has a limited effect when computing gradients to a layer's input.Experiments, on CIFAR and ImageNet, show that using our approach with 8- and even 4-bit fixed-point approximations of 32-bit floating-point activations has only a minor effect on training and validation performance, while affording significant savings in memory usage.

Ein Algorithmus zur Verringerung des Speicherbedarfs für das Training tiefer Netzwerke, der auf einer Annäherungsstrategie basiert.


Estimating the frequencies of elements in a data stream is a fundamental task in data analysis and machine learning.The problem is typically addressed using streaming algorithms which can process very large data using limited storage.Today's streaming algorithms, however, cannot exploit patterns in their input to improve performance.We propose a new class of algorithms that automatically learn relevant patterns in the input data and use them to improve its frequency estimates.   The proposed algorithms combine the benefits of machine learning with the formal guarantees available through algorithm theory.  We prove that our learning-based algorithms have lower estimation errors than their non-learning counterparts.  We also evaluate our algorithms on two real-world datasets and demonstrate empirically their performance gains.

Algorithmen für Datenströme können durch Deep Learning verbessert werden, ohne dass die Leistung beeinträchtigt wird.


Link prediction in simple graphs is a fundamental problem in which new links between nodes are predicted based on the observed structure of the graph.However, in many real-world applications, there is a need to model relationships among nodes which go beyond pairwise associations.For example, in a chemical reaction, relationship among the reactants and products is inherently higher-order.Additionally, there is need to represent the direction from reactants to products.Hypergraphs provide a natural way to represent such complex higher-order relationships.Even though Graph Convolutional Networks (GCN) have recently emerged as a powerful deep learning-based approach for link prediction over simple graphs, their suitability for link prediction in hypergraphs is unexplored -- we fill this gap in this paper and propose Neural Hyperlink Predictor (NHP). NHP adapts GCNs for link prediction in hypergraphs. We propose two variants of NHP --NHP-U and NHP-D -- for link prediction over undirected and directed hypergraphs, respectively.To the best of our knowledge, NHP-D is the first method for link prediction over directed hypergraphs.Through extensive experiments on multiple real-world datasets, we show NHP's effectiveness.

Wir schlagen den Neural Hyperlink Predictor (NHP) vor. NHP adaptiert graphische Convolutional Networks für die Link-Vorhersage in Hypergraphen


In this paper, we present a method for adversarial decomposition of text representation.This method can be used to decompose a representation of an input sentence into several independent vectors, where each vector is responsible for a specific aspect of the input sentence.We evaluate the proposed method on two case studies: the conversion between different social registers and diachronic language change.We show that the proposed method is capable of fine-grained con- trolled change of these aspects of the input sentence.For example, our model is capable of learning a continuous (rather than categorical) representation of the style of the sentence, in line with the reality of language use.The model uses adversarial-motivational training and includes a special motivational loss, which acts opposite to the discriminator and encourages a better decomposition.Finally, we evaluate the obtained meaning embeddings on a downstream task of para- phrase detection and show that they are significantly better than embeddings of a regular autoencoder.

Eine Methode, die separate Repräsentationen für die Bedeutung und die Form eines Satzes lernt


We were approached by a group of healthcare providers who are involved in the care of chronic patients looking for potential technologies to facilitate the process of reviewing patient-generated data during clinical visits.  Aiming at understanding the healthcare providers' attitudes towards reviewing patient-generated data, we (1) conducted a focus group with a mixed group of healthcare providers.Next, to gain the patients' perspectives, we (2) interviewed eight chronic patients, collected a sample of their data and designed a series of visualizations representing patient data we collected.Last, we (3) sought feedback on the visualization designs from healthcare providers who requested this exploration.We found four factors shaping patient-generated data: data & context, patient's motivation, patient's time commitment, and patient's support circle.Informed by the results of our studies, we discussed the importance of designing patient-generated visualizations for individuals by considering both patient and healthcare provider rather than designing with the purpose of generalization and provided guidelines for designing future patient-generated data visualizations.

Wir haben untersucht, welche Visualisierungen chronische Patienten dabei unterstützen können, ihre Gesundheitsdaten bei Arztbesuchen zu präsentieren und zu überprüfen.


Recently,  neural-network  based  forward  dynamics  models  have  been  proposed that  attempt  to  learn  the  dynamics  of  physical  systems  in  a  deterministic  way.While near-term motion can be predicted accurately, long-term predictions suffer from accumulating input and prediction errors which can lead to plausible but different trajectories that diverge from the ground truth.  A system that predicts distributions of the future physical states for long time horizons based on its uncertainty is thus a promising solution.  In this work, we introduce a novel robust Monte Carlo sampling based graph-convolutional dropout method that allows us to sample multiple plausible trajectories for an initial state given a neural-network based forward dynamics predictor.  By introducing a new shape preservation loss and training our dynamics model recurrently, we stabilize long-term predictions.We show that our model’s long-term forward dynamics prediction errors on complicated physical interactions of rigid and deformable objects of various shapes are significantly lower than existing strong baselines.Lastly, we demonstrate how generating multiple trajectories with our Monte Carlo dropout method can be used to train model-free reinforcement learning agents faster and to better solutions on simple manipulation tasks.

Wir schlagen einen stochastisch differenzierbaren Vorwärtsdynamik-Prädiktor vor, der in der Lage ist, mehrere physikalisch plausible Trajektorien unter demselben anfänglichen Eingangszustand abzutasten, und zeigen, dass er verwendet werden kann, um modellfreie Strategien effizienter zu trainieren.


There has been a large amount of interest, both in the past and particularly recently, into the relative advantage of different families of universal function approximators, for instance neural networks, polynomials, rational functions, etc.However, current research has focused almost exclusively on understanding this problem in a worst case setting: e.g. characterizing the best L1 or L_{infty} approximation in a box (or sometimes, even under an adversarially constructed data distribution.)In this setting many classical tools from approximation theory can be effectively used.However, in typical applications we expect data to be high dimensional, but structured -- so, it would only be important to approximate the desired function well on the relevant part of its domain, e.g. a small manifold on which real input data actually lies.Moreover, even within this domain the desired quality of approximation may not be uniform; for instance in classification problems, the approximation needs to be more accurate near the decision boundary.These issues, to the best of our knowledge, have remain unexplored until now.	With this in mind, we analyze the performance of neural networks and polynomial kernels in a natural regression setting where the data enjoys sparse latent structure, and the labels depend in a simple way on the latent variables.We give an almost-tight theoretical analysis of the performance of both neural networks and polynomials for this problem, as well as verify our theory with simulations.Our results both involve new (complex-analytic) techniques, which may be of independent interest, and show substantial qualitative differences with what is known in the worst-case setting.

Beyond-Worst-Case-Analyse der Repräsentationskraft von ReLU-Netzen und polynomialen Kerneln - insbesondere bei Vorhandensein einer spärlichen latenten Struktur.


Recent deep generative models can provide photo-realistic images as well as visual or textual content embeddings useful to address various tasks of computer vision and natural language processing.Their usefulness is nevertheless often limited by the lack of control over the generative process or the poor understanding of the learned representation.To overcome these major issues, very recent works have shown the interest of studying the semantics of the latent space of generative models.In this paper, we propose to advance on the interpretability of the latent space of generative models by introducing a new method to find meaningful directions in the latent space of any generative model along which we can move to control precisely specific properties of the generated image like position or scale of the object in the image.Our method is weakly supervised and particularly well suited for the search of directions encoding simple transformations of the generated image, such as translation, zoom or color variations.We demonstrate the effectiveness of our method qualitatively and quantitatively, both for GANs and variational auto-encoders.

Ein Modell zur Steuerung der Bilderzeugung mit GAN und beta-VAE in Bezug auf Maßstab und Position der Objekte


Modern neural network architectures use structured linear transformations, such as low-rank matrices, sparse matrices, permutations, and the Fourier transform, to improve inference speed and reduce memory usage compared to general linear maps.However, choosing which of the myriad structured transformations to use (and its associated parameterization) is a laborious task that requires trading off speed, space, and accuracy.We consider a different approach: we introduce a family of matrices called kaleidoscope matrices (K-matrices) that provably capture any structured matrix with near-optimal space (parameter) and time (arithmetic operation) complexity.We empirically validate that K-matrices can be automatically learned within end-to-end pipelines to replace hand-crafted procedures, in order to improve model quality.For example, replacing channel shuffles in ShuffleNet improves classification accuracy on ImageNet by up to 5%.Learnable K-matrices can also simplify hand-engineered pipelines---we replace filter bank feature computation in speech data preprocessing with a kaleidoscope layer, resulting in only 0.4% loss in accuracy on the TIMIT speech recognition task.K-matrices can also capture latent structure in models: for a challenging permuted image classification task, adding a K-matrix to a standard convolutional architecture can enable learning the latent permutation and improve accuracy by over 8 points.We provide a practically efficient implementation of our approach, and use K-matrices in a Transformer network to attain 36% faster end-to-end inference speed on a language translation task.

Wir schlagen eine differenzierbare Familie von "Kaleidoskop-Matrizen" vor, beweisen, dass alle strukturierten Matrizen in dieser Form dargestellt werden können, und verwenden sie, um handgefertigte lineare Zuordnungen in Deep-Learning-Modellen zu ersetzen.


We propose a method, called Label Embedding Network, which can learn label representation (label embedding) during the training process of deep networks.With the proposed method, the label embedding is adaptively and automatically learned through back propagation.The original one-hot represented loss function is converted into a new loss function with soft distributions, such that the originally unrelated labels have continuous interactions with each other during the training process.As a result, the trained model can achieve substantially higher accuracy and with faster convergence speed.Experimental results based on competitive tasks demonstrate the effectiveness of the proposed method, and the learned label embedding is reasonable and interpretable.The proposed method achieves comparable or even better results than the state-of-the-art systems.

Lernen der Label-Repräsentation für tiefe Netzwerke


Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters.As current approaches are limited by network bandwidth, we propose the use of communication compression in the decentralized training context.We show that Choco-SGD achieves linear speedup in the number of workers for arbitrary high compression ratios on general non-convex functions, and non-IID training data. We demonstrate the practical performance of the algorithm in two key scenarios: the training of deep learning models(i) over decentralized user devices, connected by a peer-to-peer network and(ii) in a datacenter.

Wir schlagen Choco-SGD--dezentralisiertes SGD mit komprimierter Kommunikation--für nicht-konvexe Ziele vor und zeigen seine starke Leistung in verschiedenen Deep-Learning-Anwendungen (On-Device-Lernen, Rechenzentrumsfall).


We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier obtained by a risk-sensitive perturbation of the weights of a learned classifier.Entropy-SGD works by optimizing the bound’s prior, violating the hypothesis of the PAC-Bayes theorem that the prior is chosen independently of the data.Indeed, available implementations of Entropy-SGD rapidly obtain zero training error on random labels and the same holds of the Gibbs posterior.In order to obtain a valid generalization bound, we show that an ε-differentially private prior yields a valid PAC-Bayes bound, a straightforward consequence of results connecting generalization with differential privacy.Using stochastic gradient Langevin dynamics (SGLD) to approximate the well-known exponential release mechanism, we observe that generalization error on MNIST (measured on held out data) falls within the (empirically nonvacuous) bounds computed under the assumption that SGLD produces perfect samples.In particular, Entropy-SGLD can be configured to yield relatively tight generalization bounds and still fit real labels, although these same settings do not obtain state-of-the-art performance.

Wir zeigen, dass Entropie-SGD den Vorgänger einer PAC-Bayes-Schranke optimiert und damit gegen die Anforderung verstößt, dass der Vorgänger unabhängig von den Daten sein muss; wir verwenden Differential Privacy, um dieses Problem zu lösen und die Verallgemeinerung zu verbessern.


In this paper, we investigate learning the deep neural networks for automated optical inspection in industrial manufacturing.Our preliminary result has shown the stunning performance improvement by transfer learning from the completely dissimilar source domain: ImageNet.Further study for demystifying this improvement shows that the transfer learning produces a highly compressible network, which was not the case for the network learned from scratch.The experimental result shows that there is a negligible accuracy drop in the network learned by transfer learning until it is compressed to 1/128 reduction of the number of convolution ﬁlters.This result is contrary to the compression without transfer learning which loses more than 5% accuracy at the same compression rate.

Wir zeigen experimentell, dass Transfer-Lernen spärliche Merkmale im Netzwerk erzeugt und dadurch ein komprimierbareres Netzwerk produziert.


The key challenge in semi-supervised learning is how to effectively leverage unlabeled data to improve learning performance.The classical label propagation method, despite its popularity, has limited modeling capability in that it only exploits graph information for making predictions.In this paper, we consider label propagation from a graph signal processing perspective and decompose it into three components: signal, filter, and classifier.By extending the three components, we propose a simple generalized label propagation (GLP) framework for semi-supervised learning.GLP naturally integrates graph and data feature information, and offers the flexibility of selecting appropriate filters and domain-specific classifiers for different applications.Interestingly, GLP also provides new insight into the popular graph convolutional network and elucidates its working mechanisms.Extensive experiments on three citation networks, one knowledge graph, and one image dataset demonstrate the efficiency and effectiveness of GLP.

Wir erweitern die klassischen Label-Propation-Methoden zur gemeinsamen Modellierung von Graphen- und Merkmalsinformationen aus der Perspektive der Graphenfilterung und zeigen Verbindungen zu den Graph Convolutional Networks auf.


Because the choice and tuning of the optimizer affects the speed, and ultimately the performance of deep learning, there is significant past and recent research in this area.Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of optimization strategies for deep learning.We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization.As the primary contribution, we present DeepOBS, a Python package of deep learning optimization benchmarks.The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking.The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for image classification on ImageNet or character-level language prediction models, as well as popular classics like MNIST and CIFAR-10.The package also provides realistic baseline results for the most popular optimizers on these test problems, ensuring a fair comparison to the competition when benchmarking new optimizers, and without having to run costly experiments.It comes with output back-ends that directly produce LaTeX code for inclusion in academic publications.It supports TensorFlow and is available open source.

Wir bieten ein Softwarepaket an, das die Bewertung von Deep-Learning-Optimierern drastisch vereinfacht, automatisiert und verbessert.


Pre-trained word embeddings are the primarymethod for transfer learning in several Natural Language Processing (NLP) tasks.Recentworks have focused on using unsupervisedtechniques such as language modeling to obtain these embeddings.In contrast, this workfocuses on extracting representations frommultiple pre-trained supervised models, whichenriches word embeddings with task and domain specific knowledge.Experiments performed in cross-task, cross-domain and crosslingual settings indicate that such supervisedembeddings are helpful, especially in the lowresource setting, but the extent of gains is dependent on the nature of the task and domain.

Extrahieren von kontextuellen Einbettungen aus einem standardmäßigen überwachten Modell. Unterstützt nachgelagerte NLP-Modelle in ressourcenarmen Umgebungen


We build a theoretical framework for understanding practical meta-learning methods that enables the integration of sophisticated formalizations of task-similarity with the extensive literature on online convex optimization and sequential prediction algorithms in order to provide within-task performance guarantees.Our approach improves upon recent analyses of parameter-transfer by enabling the task-similarity to be learned adaptively and by improving transfer-risk bounds in the setting of statistical learning-to-learn.It also leads to straightforward derivations of average-case regret bounds for efficient algorithms in settings where the task-environment changes dynamically or the tasks share a certain geometric structure.

Praktische adaptive Algorithmen für gradientenbasiertes Meta-Lernen mit nachweisbaren Garantien.


In this work, we propose a self-supervised method to learn sentence representations with an injection of linguistic knowledge.Multiple linguistic frameworks propose diverse sentence structures from which semantic meaning might be expressed out of compositional words operations.We aim to take advantage of this linguist diversity and learn to represent sentences by contrasting these diverse views.Formally, multiple views of the same sentence are mapped to close representations.On the contrary, views from other sentences are mapped further.By contrasting different linguistic views, we aim at building embeddings which better capture semantic and which are less sensitive to the sentence outward form.

Unser Ziel ist es, die Vielfalt der sprachlichen Strukturen für die Erstellung von Satzrepräsentationen zu nutzen.


The peripheral nervous system represents the input/output system for the brain.Cuff electrodes implanted on the peripheral nervous system allow observation and control over this system, however, the data produced by these electrodes have a low signal-to-noise ratio and a complex signal content.In this paper, we consider the analysis of neural data recorded from the vagus nerve in animal models, and develop an unsupervised learner based on convolutional neural networks that is able to simultaneously de-noise and cluster regions of the data by signal content.

Die unüberwachte Analyse von Daten, die vom peripheren Nervensystem aufgezeichnet wurden, entrauscht und kategorisiert die Signale.


Adversarial attacks on convolutional neural networks (CNN) have gained significant attention and there have been active research efforts on defense mechanisms.Stochastic input transformation methods have been proposed, where the idea is to recover the image from adversarial attack by random transformation, and to take the majority vote as consensus among the random samples.However, the transformation improves the accuracy on adversarial images at the expense of the accuracy on clean images.While it is intuitive that the accuracy on clean images would deteriorate, the exact mechanism in which how this occurs is unclear.In this paper, we study the distribution of softmax induced by stochastic transformations.We observe that with random transformations on the clean images, although the mass of the softmax distribution could shift to the wrong class, the resulting distribution of softmax could be used to correct the prediction.Furthermore, on the adversarial counterparts, with the image transformation, the resulting shapes of the distribution of softmax are similar to the distributions from the clean images.With these observations, we propose a method to improve existing transformation-based defenses.We train a separate lightweight distribution classifier to recognize distinct features in the distributions of softmax outputs of transformed images.Our empirical studies show that our distribution classifier, by training on distributions obtained from clean images only, outperforms majority voting for both clean and adversarial images.Our method is generic and can be integrated with existing transformation-based defenses.

Wir verbessern die bestehenden transformationsbasierten Abwehrmechanismen, indem wir einen Verteilungsklassifikator für die Verteilung von Softmax aus transformierten Bildern verwenden.


Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings.We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep.This attention mechanism enables more effective and scalable learning in complex multi-agent environments, when compared to recent approaches.Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, and it makes no assumptions about the action spaces of the agents.As such, it is flexible enough to be applied to most multi-agent learning problems

Wir schlagen einen Ansatz zum Erlernen dezentraler Strategien in Multi-Agenten-Umgebungen unter Verwendung aufmerksamkeitsbasierter Kritiker vor und zeigen vielversprechende Ergebnisse in Umgebungen mit komplexen Interaktionen.


The recent expansion of machine learning applications to molecular biology proved to have a significant contribution to our understanding of biological systems, and genome functioning in particular.Technological advances enabled the collection of large epigenetic datasets, including information about various DNA binding factors (ChIP-Seq) and DNA spatial structure (Hi-C).Several studies have confirmed the correlation between DNA binding factors and Topologically Associating Domains (TADs) in DNA structure.However, the information about physical proximity represented by genomic coordinate was not yet used for the improvement of the prediction models.In this research, we focus on Machine Learning methods for prediction of folding patterns of DNA in a classical model organism Drosophila melanogaster.The paper considers linear models with four types of regularization, Gradient Boosting and Recurrent Neural Networks for the prediction of chromatin folding patterns from epigenetic marks.The bidirectional LSTM RNN model outperformed all the models and gained the best prediction scores.This demonstrates the utilization of complex models and the importance of memory of sequential DNA states for the chromatin folding.We identify informative epigenetic features that lead to the further conclusion of their biological significance.

Wir wenden RNN an, um das biologische Problem der Vorhersage von Chromatinfaltungsmustern aus epigenetischen Markierungen zu lösen, und zeigen zum ersten Mal, dass die Nutzung des Speichers für sequenzielle Zustände auf dem DNA-Molekül für die beste Leistung von Bedeutung ist.


Previous work showed empirically that large neural networks can be significantly reduced in size while preserving their accuracy.Model compression became a central research topic, as it is crucial for deployment of neural networks on devices with limited computational and memory resources.The majority of the compression methods are based on heuristics and offer no worst-case guarantees on the trade-off between the compression rate and the approximation error for an arbitrarily new sample.We propose the first efficient, data-independent neural pruning algorithm with a provable trade-off between its compression rate and the approximation error for any future test sample.Our method is based on the coreset framework, which finds a small weighted subset of points that provably approximates the original inputs.Specifically, we approximate the output of a layer of neurons by a coreset of neurons in the previous layer and discard the rest.We apply this framework in a layer-by-layer fashion from the top to the bottom.Unlike previous works, our coreset is data independent, meaning that it provably guarantees the accuracy of the function for any input $x\in \mathbb{R}^d$, including an adversarial one.We demonstrate the effectiveness of our method on popular network architectures.In particular, our coresets yield 90% compression of the LeNet-300-100 architecture on MNIST while improving the accuracy.

Wir schlagen eine effiziente, nachweisbare und datenunabhängige Methode zur Netzwerkkompression durch neuronales Pruning unter Verwendung von Neuronen-Coresets vor - eine neuartige Konstruktion, die in dieser Arbeit vorgeschlagen wird.


Planning problems in partially observable environments cannot be solved directly with convolutional networks and require some form of memory.But, even memory networks with sophisticated addressing schemes are unable to learn intelligent reasoning satisfactorily due to the complexity of simultaneously learning to access memory and plan.To mitigate these challenges we propose the Memory Augmented Control Network (MACN).The network splits planning into a hierarchical process.At a lower level, it learns to plan in a locally observed space.At a higher level, it uses a collection of policies computed on locally observed spaces to learn an optimal plan in the global environment it is operating in.The performance of the network is evaluated on path planning tasks in environments in the presence of simple and complex obstacles and in addition, is tested for its ability to generalize to new environments not seen in the training set.

Memory Augmented Network zur Planung in teilweise beobachtbaren Umgebungen.


Contextualized word representations such as ELMo and BERT have become the de facto starting point for incorporating pretrained representations for downstream NLP tasks.In these settings, contextual representations have largely made obsolete their static embedding predecessors such as Word2Vec and GloVe.However, static embeddings do have their advantages in that they are straightforward to understand and faster to use.Additionally, embedding analysis methods for static embeddings are far more diverse and mature than those available for their dynamic counterparts.In this work, we introduce simple methods for generating static lookup table embeddings from existing pretrained contextual representations and demonstrate they outperform Word2Vec and GloVe embeddings on a variety of word similarity and word relatedness tasks.In doing so, our results also reveal insights that may be useful for subsequent downstream tasks using our embeddings or the original contextual models.Further, we demonstrate the increased potential for analysis by applying existing approaches for estimating social bias in word embeddings.Our analysis constitutes the most comprehensive study of social bias in contextual word representations (via the proxy of our distilled embeddings) and reveals a number of inconsistencies in current techniques for quantifying social bias in word embeddings.We publicly release our code and distilled word embeddings to support reproducible research and the broader NLP community.

Ein Verfahren zur Destillation von kontextuellen Modellen in statische Einbettungen; wir wenden unsere Methode auf 9 populäre Modelle an und zeigen deutliche Verbesserungen der Darstellungsqualität im Vergleich zu Word2Vec/GloVe sowie ein verbessertes Analysepotenzial durch eine gründliche Untersuchung sozialer Verzerrungen.


The brain performs unsupervised learning and (perhaps) simultaneous supervised learning.This raises the question as to whether a hybrid of supervised and unsupervised methods will produce better learning.Inspired by the rich space of Hebbian learning rules, we set out to directly learn the unsupervised learning rule on local information that best augments a supervised signal.We present the Hebbian-augmented training algorithm (HAT) for combining gradient-based learning with an unsupervised rule on pre-synpatic activity, post-synaptic activities, and current weights.We test HAT's effect on a simple problem (Fashion-MNIST) and find consistently higher performance than supervised learning alone.This finding provides empirical evidence that unsupervised learning on synaptic activities provides a strong signal that can be used to augment gradient-based methods.        We further find that the meta-learned update rule is a time-varying function; thus, it is difficult to pinpoint an interpretable Hebbian update rule that aids in training.  We do find that the meta-learner eventually degenerates into a non-Hebbian rule that preserves important weights so as not to disturb the learner's convergence.

Das Metalernen von unbeaufsichtigten Aktualisierungsregeln für neuronale Netze verbessert die Leistung und zeigt möglicherweise, wie Neuronen im Gehirn ohne Zugang zu globalen Bezeichnungen lernen.


Deep convolutional networks often append additive constant ("bias") terms to their convolution operations, enabling a richer repertoire of functional mappings.Biases are also used to facilitate training, by subtracting mean response over batches of training images (a component of "batch normalization").Recent state-of-the-art blind denoising methods seem to require these terms for their success.Here, however, we show that bias terms used in most CNNs (additive constants, including those used for batch normalization) interfere with the interpretability of these networks, do not help performance, and in fact prevent generalization of performance to noise levels not including in the training data.In particular, bias-free CNNs (BF-CNNs) are locally linear, and hence amenable to direct analysis with linear-algebraic tools.These analyses provide interpretations of network functionality in terms of projection onto a union of low-dimensional subspaces, connecting the learning-based method to more traditional denoising methodology.Additionally, BF-CNNs generalize robustly, achieving near-state-of-the-art performance at noise levels well beyond the range over which they have been trained.

Wir zeigen, dass das Entfernen von konstanten Termen aus CNN-Architekturen die Interpretierbarkeit der Entrauschungsmethode mittels linearer Algebra-Techniken ermöglicht und außerdem die Generalisierungsleistung über verschiedene Lärmpegel hinweg erhöht.


The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance.Yet despite significant empirical and theoretical analysis, relatively little has been proved about the concrete effects of different initialization schemes.In this work, we analyze the effect of initialization in deep linear networks, and provide for the first time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights.We show that for deep networks, the width needed for efficient convergence to a global minimum with orthogonal initializations is independent of the depth, whereas the width needed for efficient convergence with Gaussian initializations scales linearly in the depth.Our results demonstrate how the benefits of a good initialization can persist throughout learning, suggesting an explanation for the recent empirical successes found by initializing very deep non-linear networks according to the principle of dynamical isometry.

Wir liefern zum ersten Mal einen rigorosen Beweis dafür, dass orthogonale Initialisierung die Konvergenz für tiefe lineare Netze im Vergleich zur Gaußschen Initialisierung beschleunigt.


Survival function estimation is used in many disciplines, but it is most common in medical analytics in the form of the Kaplan-Meier estimator.Sensitive data (patient records) is used in the estimation without any explicit control on the information leakage, which is a significant privacy concern.We propose a first differentially private estimator of the survival function and show that it can be easily extended to provide differentially private confidence intervals and test statistics without spending any extra privacy budget.We further provide extensions for differentially private estimation of the competing risk cumulative incidence function.Using nine real-life clinical datasets, we provide empirical evidence that our proposed method provides good utility while simultaneously providing strong privacy guarantees.

Eine erste differentially private Schätzung der Überlebensfunktion


Neural networks can learn to extract statistical properties from data, but they seldom make use of structured information from the label space to help representation learning.Although some label structure can implicitly be obtained when training on huge amounts of data, in a few-shot learning context where little data is available, making explicit use of the label structure can inform the model to reshape the representation space to reflect a global sense of class dependencies.  We propose a meta-learning framework, Conditional class-Aware Meta-Learning (CAML), that conditionally transforms feature representations based on a metric space that is trained to capture inter-class dependencies.This enables a conditional modulation of the feature representations of the base-learner to impose regularities informed by the label space.Experiments show that the conditional transformation in CAML leads to more disentangled representations and achieves competitive results on the miniImageNet benchmark.

CAML ist eine Instanz von MAML mit bedingten Klassenabhängigkeiten.


We study the problem of multiset prediction.The goal of multiset prediction is to train a predictor that maps an input to a multiset consisting of multiple items.Unlike existing problems in supervised learning, such as classification, ranking and sequence generation, there is no known order among items in a target multiset, and each item in the multiset may appear more than once, making this problem extremely challenging.In this paper, we propose a novel multiset loss function by viewing this problem from the perspective of sequential decision making.The proposed multiset loss function is empirically evaluated on two families of datasets, one synthetic and the other real, with varying levels of difficulty, against various baseline loss functions including reinforcement learning, sequence, and aggregated distribution matching loss functions.The experiments reveal the effectiveness of the proposed loss function over the others.

Wir untersuchen das Problem der Multiset-Vorhersage und schlagen eine neuartige Multiset-Verlustfunktion vor, die durch Analysen und empirische Belege ihre Wirksamkeit unter Beweis stellt.


Understanding theoretical properties of deep and locally connected nonlinear network, such as deep convolutional neural network (DCNN), is still a hard problem despite its empirical success.In this paper, we propose a novel theoretical framework for such networks with ReLU nonlinearity.The framework bridges data distribution with gradient descent rules, favors disentangled representations and is compatible with common regularization techniques such as Batch Norm, after a novel discovery of its projection nature.The framework is built upon teacher-student setting, by projecting the student's forward/backward pass onto the teacher's computational graph.We do not impose unrealistic assumptions (e.g., Gaussian inputs, independence of activation, etc).Our framework could help facilitate theoretical analysis of many practical issues, e.g. disentangled representations in deep networks.

In diesem Beitrag wird ein theoretischer Rahmen vorgestellt, der die Datenverteilung für tiefe und lokal verbundene ReLU-Netze explizit modelliert.


Multi-agent cooperation is an important feature of the natural world.Many tasks involve individual incentives that are misaligned with the common good, yet a wide range of organisms from bacteria to insects and humans are able to overcome their differences and collaborate.Therefore, the emergence of cooperative behavior amongst self-interested individuals is an important question for the fields of multi-agent reinforcement learning (MARL) and evolutionary theory.Here, we study a particular class of multi-agent problems called intertemporal social dilemmas (ISDs), where the conflict between the individual and the group is particularly sharp.By combining MARL with appropriately structured natural selection, we demonstrate that individual inductive biases for cooperation can be learned in a model-free way.To achieve this, we introduce an innovative modular architecture for deep reinforcement learning agents which supports multi-level selection.We present results in two challenging environments, and interpret these in the context of cultural and ecological evolution.

Wir stellen einen biologisch inspirierten modularen Evolutionsalgorithmus vor, in dem Deep-RL-Agenten lernen, in einem schwierigen sozialen Multi-Agenten-Spiel zu kooperieren, was helfen könnte, die Evolution des Altruismus zu erklären.


In adversarial attacks to machine-learning classifiers, small perturbations are added to input that is correctly classified.The perturbations yield adversarial examples, which are virtually indistinguishable from the unperturbed input, and yet are misclassified.In standard neural networks used for deep learning, attackers can craft adversarial examples from most input to cause a misclassification of their choice. We introduce a new type of network units, called RBFI units, whose non-linear structure makes them inherently resistant to adversarial attacks.On permutation-invariant MNIST, in absence of adversarial attacks, networks using RBFI units match the performance of networks using sigmoid units, and are slightly below the accuracy of networks with ReLU units.When subjected to adversarial attacks based on projected gradient descent or fast gradient-sign methods, networks with RBFI units retain accuracies above 75%, while ReLU or Sigmoid see their accuracies reduced to below 1%.Further, RBFI networks trained on regular input either exceed or closely match the accuracy of sigmoid and ReLU network trained with the help of adversarial examples.The non-linear structure of RBFI units makes them difficult to train using standard gradient descent.We show that RBFI networks of RBFI units can be efficiently trained to high accuracies using pseudogradients, computed using functions especially crafted to facilitate learning instead of their true derivatives.

Wir stellen eine Art von neuronalem Netz vor, das strukturell resistent gegen feindliche Angriffe ist, selbst wenn es auf unaugmentierten Trainingssätzen trainiert wird.  Diese Widerstandsfähigkeit ist auf die Stabilität der Netzeinheiten gegenüber Eingabestörungen zurückzuführen.


We show that there exists an inherent tension between the goal of adversarial robustness and that of standard generalization. Specifically, training robust models may not only be more resource-consuming, but also lead to a reduction of standard accuracy.We demonstrate that this trade-off between the standard accuracy of a model and its robustness to adversarial perturbations provably exists even in a fairly simple and natural setting.These findings also corroborate a similar phenomenon observed in practice.Further, we argue that this phenomenon is a consequence of robust classifiers learning fundamentally different feature representations than  standard classifiers.These differences, in particular, seem to result in unexpected benefits: the features learned by robust models tend to align better with salient data characteristics and human perception.

Wir zeigen, dass die adversariale Robustheit zwar auf Kosten der normalen Klassifizierungsleistung geht, aber auch unerwartete Vorteile mit sich bringt.


Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples.In this work, we propose mixup, a simple learning principle to alleviate these issues.In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels.  By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples.  Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures.  We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.

Training auf konvexen Kombinationen zwischen zufälligen Trainingsbeispielen und ihren Bezeichnungen verbessert die Generalisierung in tiefen neuronalen Netzen


We present a novel approach to spike sorting for high-density multielectrode probes using the Neural Clustering Process (NCP), a recently introduced neural architecture that performs scalable amortized approximate Bayesian inference for efficient probabilistic clustering.To optimally encode spike waveforms for clustering, we extended NCP by adding a convolutional spike encoder, which is learned end-to-end with the NCP network.Trained purely on labeled synthetic spikes from a simple generative model, the NCP spike sorting model shows promising performance for clustering multi-channel spike waveforms.The model provides higher clustering quality than an alternative Bayesian algorithm, finds more spike templates with clear receptive fields on real data and recovers more ground truth neurons on hybrid test data compared to a recent spike sorting algorithm.Furthermore, NCP is able to handle the clustering uncertainty of ambiguous small spikes by GPU-parallelized posterior sampling.The source code is publicly available.

Wir stellen einen neuartigen Ansatz zur Spike-Sortierung vor, der den Neural Clustering Process (NCP) verwendet, eine kürzlich eingeführte neuronale Architektur, die skalierbare amortisierte approximative Bayes'sche Inferenz für effizientes probabilistisches Clustering durchführt.


The goal of the paper is to propose an algorithm for learning the most generalizable solution from given training data.It is shown that Bayesian approach leads to a solution that dependent on statistics of training data and not on particularsamples.The solution is stable under perturbations of training data because it is defined by an integral contribution of multiple maxima of the likelihood and not by a single global maximum.Specifically, the Bayesian probability distributionof parameters (weights) of a probabilistic model given by a neural network is estimated via recurrent variational approximations.Derived recurrent update rules correspond to SGD-type rules for finding a minimum of an effective loss that is an average of an original negative log-likelihood over the Gaussian distributions of weights, which makes it a function of means and variances.The effective loss is convex for large variances and non-convex in the limit of small variances.Among stationary solutions of the update rules there are trivial solutions with zero variances at local minima of the original loss and a single non-trivial solution with finite variances that is a critical point at the end of convexity of the effective lossin the mean-variance space.At the critical point both first- and second-order gradients of the effective loss w.r.t. means are zero.The empirical study confirms that the critical point represents the most generalizable solution.While the location ofthe critical point in the weight space depends on specifics of the used probabilistic model some properties at the critical point are universal and model independent.

Vorgeschlagene Methode zum Finden der am besten verallgemeinerbaren Lösung, die gegenüber Störungen der Trainingsdaten stabil ist.


Humans rely on episodic memory constantly, in remembering the name of someone they met 10 minutes ago, the plot of a movie as it unfolds, or where they parked the car.Endowing reinforcement learning agents with episodic memory is a key step on the path toward replicating human-like general intelligence.We analyze why standard RL agents lack episodic memory today, and why existing RL tasks don't require it.We design a new form of external memory called Masked Experience Memory, or MEM, modeled after key features of human episodic memory.To evaluate episodic memory we define an RL task based on the common children's game of Concentration.We find that a MEM RL agent leverages episodic memory effectively to master Concentration, unlike the baseline agents we tested.

Implementierung und Evaluierung des episodischen Speichers für RL.


Parameters are one of the most critical components of machine learning models.As datasets and learning domains change, it is often necessary and time-consuming to re-learn entire models.Rather than re-learning the parameters from scratch, replacing learning with optimization, we propose a framework building upon the theory of \emph{optimal transport} to adapt model parameters by discovering correspondences between models and data, significantly amortizing the training cost.We demonstrate our idea on the challenging problem of creating probabilistic spatial representations for autonomous robots.Although recent mapping techniques have facilitated robust occupancy mapping, learning all spatially-diverse parameters in such approximate Bayesian models demand considerable computational time, discouraging them to be used in real-world robotic mapping.Considering the fact that the geometric features a robot would observe with its sensors are similar across various environments, in this paper, we demonstrate how to re-use parameters and hyperparameters learned in different domains.This adaptation is computationally more efficient than variational inference and Monte Carlo techniques.A series of experiments conducted on realistic settings verified the possibility of transferring thousands of such parameters with a negligible time and memory cost, enabling large-scale mapping in urban environments.

Wir stellen eine Methode zur Anpassung von Hyperparametern probabilistischer Modelle durch optimalen Transport mit Anwendungen in der Robotik vor.


An algorithm is introduced for learning a predictive state representation with off-policy temporal difference (TD) learning that is then used to learn to steer a vehicle with reinforcement learning.  There are three components being learned simultaneously:  (1) the off-policy predictions as a compact representation of state, (2) the behavior policy distribution for estimating the off-policy predictions, and (3) the deterministic policy gradient for learning to act.  A behavior policy discriminator is learned and used for estimating the important sampling ratios needed to learn the predictive representation off-policy with general value functions (GVFs).  A linear deterministic policy gradient method is used to train the agent with only the predictive representations while the predictions are being learned.  All three components are combined, demonstrated and evaluated on the problem of steering the vehicle from images in the TORCS racing simulator environment.Steering from only images is a challenging problem where evaluation is completed on a held-out set of tracks that were never seen during training in order to measure the generalization of the predictions and controller.  Experiments show the proposed method is able to steer smoothly and navigate many but not all of the tracks available in TORCS with performance that exceeds DDPG using only images as input and approaches the performance of an ideal non-vision based kinematics model.

Ein Algorithmus zum Erlernen einer prädiktiven Zustandsrepräsentation mit allgemeinen Wertfunktionen und Off-Policy-Lernen wird auf das Problem der sichtbasierten Lenkung beim autonomen Fahren angewendet.


For numerous domains, including for instance earth observation, medical imaging, astrophysics,..., available image and signal datasets often irregular space-time sampling patterns and large missing data rates.These sampling properties is a critical issue to apply state-of-the-art learning-based (e.g., auto-encoders, CNNs,...) to fully benefit from the available large-scale observations and reach breakthroughs in the reconstruction and identification of processes of interest.In this paper, we address the end-to-end learning of representations of signals, images and image sequences from irregularly-sampled data, {\em i.e.} when the training data involved missing data.From an analogy to Bayesian formulation, we consider energy-based representations.Two energy forms are investigated: one derived from auto-encoders and one relating to Gibbs energies.The learning stage of these energy-based representations (or priors) involve a joint interpolation issue, which resorts to solving an energy minimization problem under observation constraints.Using a neural-network-based implementation of the considered energy forms, we can state an end-to-end learning scheme from irregularly-sampled data.We demonstrate the relevance of the proposed representations for different case-studies: namely, multivariate time series, 2{\sc } images and image sequences.

Wir befassen uns mit dem durchgängigen Lernen von energiebasierten Repräsentationen für Signal- und Bildbeobachtungsdaten mit unregelmäßigen Probemustern.


Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood.Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively.This leads to poor sample-efficiency during meta-training as well as ineffective task identification strategies.This paper provides a theoretical analysis of credit assignment in gradient-based Meta-RL.Building on the gained insights we develop a novel meta-learning algorithm that overcomes both the issue of poor credit assignment and previous difficulties in estimating meta-policy gradients.By controlling the statistical distance of both pre-adaptation and adapted policies during meta-policy search, the proposed algorithm endows efficient and stable meta-learning.Our approach leads to superior pre-adaptation policy behavior and consistently outperforms previous Meta-RL algorithms in sample-efficiency, wall-clock time, and asymptotic performance.

Ein neuartiger und theoretisch fundierter Algorithmus für Meta-Reinforcement Learning


When training a neural network for a desired task, one may prefer to adapt a pretrained network rather than start with a randomly initialized one -- due to lacking enough training data, performing lifelong learning where the system has to learn a new task while being previously trained for other tasks, or wishing to encode priors in the network via preset weights.The most commonly employed approaches for network adaptation are fine-tuning and using the pre-trained network as a fixed feature extractor, among others. In this paper we propose a straightforward alternative: Side-Tuning.Side-tuning adapts a pretrained network by training a lightweight "side" network that is fused with the (unchanged) pre-rained network using a simple additive process.This simple method works as well as or better than existing solutions while it resolves some of the basic issues with fine-tuning, fixed features, and several other common baselines.In particular, side-tuning is less prone to overfitting when little training data is available, yields better results than using a fixed feature extractor, and doesn't suffer from catastrophic forgetting in lifelong learning.  We demonstrate the performance of side-tuning under a diverse set of scenarios, including lifelong learning (iCIFAR, Taskonomy), reinforcement learning, imitation learning (visual navigation in Habitat), NLP question-answering (SQuAD v2), and single-task transfer learning (Taskonomy), with consistently promising results.

Beim Side-Tuning wird ein bereits trainiertes Netz angepasst, indem ein leichtgewichtiges "Side"-Netz trainiert wird, das mit dem (unveränderten) bereits trainierten Netz durch einen einfachen additiven Prozess verschmolzen wird.


In this paper, we present a technique for generating artificial datasets that retain statistical properties of the real data while providing differential privacy guarantees with respect to this data.We include a Gaussian noise layer in the discriminator of a generative adversarial network to make the output and the gradients differentially private with respect to the training data, and then use the generator component to synthesise privacy-preserving artificial dataset.Our experiments show that under a reasonably small privacy budget we are able to generate data of high quality and successfully train machine learning models on this artificial data.

Trainieren von GANs mit differential privacy, um künstliche Datensätze zu erzeugen, die den Datenschutz wahren.


This paper presents two methods to disentangle and interpret contextual effects that are encoded in a pre-trained deep neural network.Unlike convolutional studies that visualize image appearances corresponding to the network output or a neural activation from a global perspective, our research aims to clarify how a certain input unit (dimension) collaborates with other units (dimensions) to constitute inference patterns of the neural network and thus contribute to the network output.The analysis of local contextual effects w.r.t. certain input units is of special values in real applications.In particular, we used our methods to explain the gaming strategy of the alphaGo Zero model in experiments, and our method successfully disentangled the rationale of each move during the game.

In diesem Beitrag werden Methoden zur Entflechtung und Interpretation von Kontexteffekten vorgestellt, die in einem tiefen neuronalen Netz kodiert sind.


The main goal of network pruning is imposing sparsity on the neural network by increasing the number of parameters with zero value in order to reduce the architecture size and the computational speedup.

Vorschlag einer neuen Methode, die auf der geführten Aufmerksamkeit basiert, um die Seltenheit in tiefen neuronalen Netzen zu verstärken.


Giving provable guarantees for learning neural networks is a core challenge of machine learning theory.Most prior work gives parameter recovery guarantees for one hidden layer networks, however, the networks used in practice have multiple non-linear layers.In this work, we show how we can strengthen such results to deeper networks -- we address the problem of uncovering the lowest layer in a deep neural network under the assumption that the lowest layer uses a high threshold before applying the activation, the upper network can be modeled as a well-behaved polynomial and the input distribution is gaussian.

Wir können die unterste Schicht eines tiefen neuronalen Netzes nachweislich wiederherstellen, wenn die unterste Schicht eine Aktivierung mit "hohem Schwellenwert" verwendet und das obige Netz ein "wohlerzogenes" Polynom ist.


Federated learning involves training and effectively combining machine learning models from distributed partitions of data (i.e., tasks) on edge devices, and be naturally viewed as a multi- task learning problem.While Federated Averaging (FedAvg) is the leading optimization method for training non-convex models in this setting, its behavior is not well understood in realistic federated settings when the devices/tasks are statistically heterogeneous, i.e., where each device collects data in a non-identical fashion.In this work, we introduce a framework, called FedProx, to tackle statistical heterogeneity.FedProx encompasses FedAvg as a special case.We provide convergence guarantees for FedProx through a device dissimilarity assumption.Our empirical evaluation validates our theoretical analysis and demonstrates the improved robustness and stability of FedProx for learning in heterogeneous networks.

Wir stellen FedProx vor, einen Rahmen zur Bewältigung statistischer Heterogenität in föderierten Umgebungen mit Konvergenzgarantien und verbesserter Robustheit und Stabilität.


Referential games offer a grounded learning environment for neural agents which accounts for the fact that language is functionally used to communicate.However, they do not take into account a second constraint considered to be fundamental for the shape of human language: that it must be learnable by new language learners and thus has to overcome a transmission bottleneck.In this work, we insert such a bottleneck in a referential game, by introducing a changing population of agents in which new agents learn by playing with more experienced agents.We show that mere cultural transmission results in a substantial improvement in language efficiency and communicative success, measured in convergence speed, degree of structure in the emerged languages and within-population consistency of the language.However, as our core contribution, we show that the optimal situation is to co-evolve language and agents.When we allow the agent population to evolve through genotypical evolution, we achieve across the board improvements on all considered metrics.These results stress that for language emergence studies cultural evolution is important, but also the suitability of the architecture itself should be considered.

Wir ermöglichen sowohl die kulturelle Evolution von Sprache als auch die genetische Evolution von Agenten in einem referentiellen Spiel, indem wir eine neue Sprachübertragungs Engine verwenden.


The need for large amounts of training image data with clearly defined features is a major obstacle to applying generative adversarial networks(GAN) on image generation where training data is limited but diverse, since insufficient latent feature representation in the already scarce data often leads to instability and mode collapse during GAN training.To overcome the hurdle of limited data when applying GAN to limited datasets, we propose in this paper the strategy of \textit{parallel recurrent data augmentation}, where the GAN model progressively enriches its training set with sample images constructed from GANs trained in parallel at consecutive training epochs.Experiments on a variety of small yet diverse datasets demonstrate that our method, with little model-specific considerations, produces images of better quality as compared to the images generated  without such strategy.The source code and generated images of this paper will be made public after review.

Wir haben eine neuartige, einfache und effiziente Methode zur Datenerweiterung eingeführt, die die Leistung bestehender GANs steigert, wenn die Trainingsdaten begrenzt und vielfältig sind.


We develop a stochastic whole-brain and body simulator of the nematode roundworm Caenorhabditis elegans (C. elegans) and show that it is sufficiently regularizing to allow imputation of latent membrane potentials from partial calcium fluorescence imaging observations.This is the first attempt we know of to ``complete the circle,'' where an anatomically grounded whole-connectome simulator is used to impute a time-varying ``brain'' state at single-cell fidelity from covariates that are measurable in practice.  Using state of the art Bayesian machine learning methods to condition on readily obtainable data, our method paves the way for neuroscientists to recover interpretable connectome-wide state representations, automatically estimate physiologically relevant parameter values from data, and perform simulations investigating intelligent lifeforms in silico.

Wir entwickeln einen Simulator für das gesamte Konnektom und den Körper von C. elegans und demonstrieren eine gemeinsame Zustandsraum- und Parameterinferenz in diesem Simulator.


The high-quality node embeddings learned from the Graph Neural Networks (GNNs) have been applied to a wide range of node-based applications and some of them have achieved state-of-the-art (SOTA) performance.However, when applying node embeddings learned from GNNs to generate graph embeddings, the scalar node representation may not suffice to preserve the node/graph properties efficiently, resulting in sub-optimal graph embeddings.Inspired by the Capsule Neural Network (CapsNet), we propose the Capsule Graph Neural Network (CapsGNN), which adopts the concept of capsules to address the weakness in existing GNN-based graph embeddings algorithms.By extracting node features in the form of capsules, routing mechanism can be utilized to capture important information at the graph level.As a result, our model generates multiple embeddings for each graph to capture graph properties from different aspects.The attention module incorporated in CapsGNN is used to tackle graphs with various sizes which also enables the model to focus on critical parts of the graphs.Our extensive evaluations with 10 graph-structured datasets demonstrate that CapsGNN has a powerful mechanism that operates to capture macroscopic properties of the whole graph by data-driven.It outperforms other SOTA techniques on several graph classification tasks, by virtue of the new instrument.

Inspiriert von CapsNet schlagen wir eine neue Architektur für Grapheneinbettungen auf der Basis von Knotenmerkmalen vor, die aus GNN extrahiert werden.


We introduce a novel framework for generative models based on Restricted Kernel Machines (RKMs) with multi-view generation and uncorrelated feature learning capabilities, called Gen-RKM.To incorporate multi-view generation, this mechanism uses a shared representation of data from various views.The mechanism is flexible to incorporate both kernel-based, (deep) neural network and convolutional based models within the same setting.To update the parameters of the network, we propose a novel training procedure which jointly learns the features and shared representation.Experiments demonstrate the potential of the framework through qualitative evaluation of generated samples.

Gen-RKM: ein neuer Rahmen für generative Modelle unter Verwendung von Restricted Kernel Machines mit Multiview-Generierung und unkorreliertem Merkmalslernen.


Work on the problem of contextualized word representation—the development of reusable neural network components for sentence understanding—has recently seen a  surge of progress centered on the unsupervised pretraining task of language modeling with methods like ELMo (Peters et al., 2018).This paper contributes the first large-scale systematic study comparing different pretraining tasks in this context, both as complements to language modeling and as potential alternatives.The primary results of the study support the use of language modeling as a pretraining task and set a new state of the art among comparable models using multitask learning with language models.However, a closer look at these results reveals worryingly strong baselines and strikingly varied results across target tasks, suggesting that the widely-used paradigm of pretraining and freezing sentence encoders may not be an ideal platform for further work.

Wir vergleichen viele Aufgaben und Aufgabenkombinationen zum Vortraining von BiLSTMs auf Satzebene für NLP-Aufgaben. Die Sprachmodellierung ist die beste einzelne Pretraining-Aufgabe, aber auch einfache Basisaufgaben schneiden gut ab.


In this paper, we study the adversarial attack and defence problem in deep learning from the perspective of Fourier analysis.We first explicitly compute the Fourier transform of deep ReLU neural networks and show that there exist decaying but non-zero high frequency components in the Fourier spectrum of neural networks.We then demonstrate that the vulnerability of neural networks towards adversarial samples can be attributed to these insignificant but non-zero high frequency components.Based on this analysis, we propose to use a simple post-averaging technique to smooth out these high frequency components to improve the robustness of neural networks against adversarial attacks.Experimental results on the ImageNet and the CIFAR-10 datasets have shown that our proposed method is universally effective to defend many existing adversarial attacking methods proposed in the literature, including FGSM, PGD, DeepFool and C&W attacks.Our post-averaging method is simple since it does not require any re-training, and meanwhile it can successfully defend over 80-96% of the adversarial samples generated by these methods without introducing significant performance degradation (less than 2%) on the original clean images.

Ein Einblick in den Grund der gegnerischen Verwundbarkeit, eine effektive Verteidigungsmethode gegen Adversarial Attacks.


Reinforcement learning methods that continuously learn neural networks by episode generation with game tree search have been successful in two-person complete information deterministic games such as chess, shogi, and Go.However, there are only reports of practical cases and there are little evidence to guarantee the stability and the final performance of learning process.In this research, the coordination of episode generation was focused on.By means of regarding the entire system as game tree search, the new method can handle the trade-off between exploitation and exploration during episode generation.The experiments with a small problem showed that it had robust performance compared to the existing method, Alpha Zero.

Anwendung der Monte-Carlo-Baumsuche auf die Episodenerzeugung in Alpha Zero


Message-passing neural networks (MPNNs) have been successfully applied in a wide variety of applications in the real world.However, two fundamental weaknesses of MPNNs' aggregators limit their ability to represent graph-structured data: losing the structural information of nodes in neighborhoods and lacking the ability to capture long-range dependencies in disassortative graphs.Few studies have noticed the weaknesses from different perspectives.From the observations on classical neural network and network geometry, we propose a novel geometric aggregation scheme for graph neural networks to overcome the two weaknesses.  The behind basic idea is the aggregation on a graph can benefit from a continuous space underlying the graph.The proposed aggregation scheme is permutation-invariant and consists of three modules, node embedding, structural neighborhood, and bi-level aggregation.We also present an implementation of the scheme in graph convolutional networks, termed Geom-GCN, to perform transductive learning on graphs.Experimental results show the proposed Geom-GCN achieved state-of-the-art performance on a wide range of open datasets of graphs.

Bei neuronalen Netzen mit Graphen kann die Aggregation in einem Graphen von einem kontinuierlichen Raum profitieren, der dem Graphen zugrunde liegt.


We consider the task of program synthesis in the presence of a reward function over the output of programs, where the goal is to find programs with maximal rewards.We introduce a novel iterative optimization scheme, where we train an RNN on a dataset of K best programs from a priority queue of the generated programs so far.Then, we synthesize new programs and add them to the priority queue by sampling from the RNN.We benchmark our algorithm called priority queue training (PQT) against genetic algorithm and reinforcement learning baselines on a simple but expressive Turing complete programming language called BF.Our experimental results show that our deceptively simple PQT algorithm significantly outperforms the baselines.By adding a program length penalty to the reward function, we are able to synthesize short, human readable programs.

Wir verwenden einen einfachen Suchalgorithmus mit einem RNN und einer Prioritätswarteschlange, um Lösungen für Codierungsaufgaben zu finden.


We present graph wavelet neural network (GWNN), a novel graph convolutional neural network (CNN), leveraging graph wavelet transform to address the shortcomings of previous spectral graph CNN methods that depend on graph Fourier transform.Different from graph Fourier transform, graph wavelet transform can be obtained via a fast algorithm without requiring matrix eigendecomposition with high computational cost.Moreover, graph wavelets are sparse and localized in vertex domain, offering high efficiency and good interpretability for graph convolution.The proposed GWNN significantly outperforms previous spectral graph CNNs in the task of graph-based semi-supervised classification on three benchmark datasets: Cora, Citeseer and Pubmed.

Wir präsentieren das Graph Wavelet Neural Network (GWNN), ein neuartiges Graph-Convolutional Neural Network (CNN), das die Graph Wavelet Transformation nutzt, um die Unzulänglichkeit früherer spektraler Graph CNN Methoden zu beheben, die auf der Graph Fourier Transformation beruhen.


Learning rate decay (lrDecay) is a \emph{de facto} technique for training modern neural networks.It starts with a large learning rate and then decays it multiple times.It is empirically observed to help both optimization and generalization.Common beliefs in how lrDecay works come from the optimization analysis of (Stochastic) Gradient Descent:1) an initially large learning rate accelerates training or helps the network escape spurious local minima;2) decaying the learning rate helps the network converge to a local minimum and avoid oscillation.Despite the popularity of these common beliefs, experiments suggest that they are insufficient in explaining the general effectiveness of lrDecay in training modern neural networks that are deep, wide, and nonconvex.We provide another novel explanation: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns.The proposed explanation is validated on a carefully-constructed dataset with tractable pattern complexity.And its implication, that additional patterns learned in later stages of lrDecay are more complex and thus less transferable, is justified in real-world datasets.We believe that this alternative explanation will shed light into the design of better training strategies for modern neural networks.

Wir liefern eine weitere neue Erklärung für den Verfall der Lernrate: Eine anfänglich hohe Lernrate verhindert, dass sich das Netzwerk verrauschte Daten merkt, während ein Verfall der Lernrate das Erlernen komplexer Muster verbessert.


We show how an ensemble of $Q^*$-functions can be leveraged for more effective exploration in deep reinforcement learning.We build on well established algorithms from the bandit setting, and adapt them to the $Q$-learning setting.We propose an exploration strategy based on upper-confidence bounds (UCB).Our experiments show significant gains on the Atari benchmark.

Die Anpassung der UCB-Exploration an das Ensemble Q-learning verbessert frühere Methoden wie Double DQN, A3C+ im Atari-Benchmark


We focus on the problem of learning a single motor module that can flexibly express a range of behaviors for the control of high-dimensional physically simulated humanoids.To do this, we propose a motor architecture that has the general structure of an inverse model with a latent-variable bottleneck.We show that it is possible to train this model entirely offline to compress thousands of expert policies and learn a motor primitive embedding space.The trained neural probabilistic motor primitive system can perform one-shot imitation of whole-body humanoid behaviors, robustly mimicking unseen trajectories.Additionally, we demonstrate that it is also straightforward to train controllers to reuse the learned motor primitive space to solve tasks, and the resulting movements are relatively naturalistic.To support the training of our model, we compare two approaches for offline policy cloning, including an experience efficient method which we call linear feedback policy cloning.We encourage readers to view a supplementary video (https://youtu.be/CaDEf-QcKwA ) summarizing our results.

Neural Probabilistic Motor Primitives komprimieren die Richtlinien zur Bewegungserfassung in ein flexibles Modell, das one-shot Imitation und Wiederverwendung als Low-Level-Controller kann.


Data augmentation is a useful technique to enlarge the size of the training set and prevent overfitting for different machine learning tasks when training data is scarce.However, current data augmentation techniques rely heavily on human design and domain knowledge, and existing automated approaches are yet to fully exploit the latent features in the training dataset.In this paper we propose  \textit{Parallel Adaptive GAN Data Augmentation}(PAGANDA), where the training set adaptively enriches itself with sample images automatically constructed from Generative Adversarial Networks (GANs) trained in parallel.We demonstrate by experiments that our data augmentation strategy, with little model-specific considerations, can be easily adapted to cross-domain deep learning/machine learning tasks such as image classification and image inpainting, while significantly improving model performance in both tasks.Our source code and experimental details are available at \url{https://github.com/miaojiang1987/k-folder-data-augmentation-gan/}.

Wir stellen eine automatisierte adaptive Datenerweiterung vor, die für mehrere unterschiedliche Aufgaben funktioniert.


Deep neural networks with millions of parameters may suffer from poor generalizations due to overfitting.To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples.In particular, we distill the predictive distribution between different samples of the same label and augmented samples of the same source during training.In other words, we regularize the dark knowledge (i.e., the knowledge on wrong predictions) of a single network, i.e., a self-knowledge distillation technique, to force it output more meaningful predictions.  We demonstrate the effectiveness of the proposed method  via  experiments  on  various  image  classification  tasks:  it  improves  not only the generalization ability, but also the calibration accuracy of modern neural networks.

Wir schlagen eine neue Regularisierungstechnik vor, die auf der Wissensdestillation basiert.


In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models.We propose the weight-dropped LSTM, which uses DropConnect on hidden-to-hidden weights, as a form of recurrent regularization.Further, we introduce NT-ASGD, a non-monotonically triggered  (NT) variant of the averaged stochastic gradient method (ASGD), wherein the averaging trigger is determined using a NT condition as opposed to being tuned by the user.Using these and other regularization strategies, our ASGD Weight-Dropped LSTM (AWD-LSTM) achieves state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2.In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.We also explore the viability of the proposed regularization and optimization strategies in the context of the quasi-recurrent neural network (QRNN) and demonstrate comparable performance to the AWD-LSTM counterpart.The code for reproducing the results is open sourced and is available at https://github.com/salesforce/awd-lstm-lm.

Effektive Regularisierungs- und Optimierungsstrategien für LSTM-basierte Sprachmodelle erreichen SOTA auf PTB und WT2.


Various methods of measuring unit selectivity have been developed with the aim of better understanding how neural networks work.  But the different measures provide divergent estimates of selectivity, and this has led to different conclusions regarding the conditions in which selective object representations are learned and the functional relevance of these representations.In an attempt to better characterize object selectivity, we undertake a comparison of various selectivity measures on a large set of units in AlexNet, including localist selectivity, precision, class-conditional mean activity selectivity (CCMAS), network dissection, the human interpretation of activation maximization (AM) images, and standard signal-detection measures.  We find that the different measures provide different estimates of object selectivity, with precision and CCMAS measures providing misleadingly high estimates.Indeed, the most selective units had a poor hit-rate or a high false-alarm rate (or both) in object classification, making them poor object detectors.  We fail to find any units that are even remotely as selective as the 'grandmother cell' units reported in recurrent neural networks.In order to generalize these results, we compared selectivity measures on a few units in VGG-16 and GoogLeNet trained on the ImageNet or Places-365 datasets that have been described as 'object detectors'.Again, we find poor hit-rates and high false-alarm rates for object classification.

Auf der Suche nach Objektdetektoren unter Verwendung vieler verschiedener Selektivitätsmaße; CNNs sind leicht selektiv, aber nicht genug, um als Objektdetektoren bezeichnet zu werden.


The folding structure of the DNA molecule combined with helper molecules, also referred to as the chromatin, is highly relevant for the functional properties of DNA.The chromatin structure is largely determined by the underlying primary DNA sequence, though the interaction is not yet fully understood.In this paper we develop a convolutional neural network that takes an image-representation of primary DNA sequence as its input, and predicts key determinants of chromatin structure.The method is developed such that it is capable of detecting interactions between distal elements in the DNA sequence, which are known to be highly relevant.Our experiments show that the method outperforms several existing methods both in terms of prediction accuracy and training time.

Eine Methode zur Umwandlung von DNA-Sequenzen in 2D-Bilder unter Verwendung von raumfüllenden Hilbert-Kurven zur Verbesserung der Stärken von CNNs


This paper introduces a novel method to perform transfer learning across domains and tasks, formulating it as a problem of learning to cluster.The key insight is that, in addition to features, we can transfer similarity information and this is sufficient to learn a similarity function and clustering network to perform both domain adaptation and cross-task transfer learning.We begin by reducing categorical information to pairwise constraints, which only considers whether two instances belong to the same class or not (pairwise semantic similarity).This similarity is category-agnostic and can be learned from data in the source domain using a similarity network.We then present two novel approaches for performing transfer learning using this similarity function.First, for unsupervised domain adaptation, we design a new loss function to regularize classification with a constrained clustering loss, hence learning a clustering network with the transferred similarity metric generating the training inputs.Second, for cross-task learning (i.e., unsupervised clustering with unseen categories), we propose a framework to reconstruct and estimate the number of semantic clusters, again using the clustering network.Since the similarity network is noisy, the key is to use a robust clustering algorithm, and we show that our formulation is more robust than the alternative constrained and unconstrained clustering approaches.Using this method, we first show state of the art results for the challenging cross-task problem, applied on Omniglot and ImageNet.Our results show that we can reconstruct semantic clusters with high accuracy.We then evaluate the performance of cross-domain transfer using images from the Office-31 and SVHN-MNIST tasks and present top accuracy on both datasets.  Our approach doesn't explicitly deal with domain discrepancy.If we combine with a domain adaptation loss, it shows further improvement.

Ein lernfähiges Clustering-Ziel zur Erleichterung des bereichs- und aufgabenübergreifenden Transfer Lernens


Recent advances in cross-lingual word embeddings have primarily relied on mapping-based methods, which project pretrained word embeddings from different languages into a shared space through a linear transformation.However, these approaches assume word embedding spaces are isomorphic between different languages, which has been shown not to  hold in practice (Søgaard et al., 2018), and fundamentally limits their performance.This motivates investigating joint learning methods which can overcome this impediment, by simultaneously learning embeddings across languages via a cross-lingual term in the training objective.Given the abundance of parallel data available (Tiedemann, 2012), we propose a bilingual extension of the CBOW method which leverages sentence-aligned corpora to obtain robust cross-lingual word and sentence representations.Our approach significantly improves cross-lingual sentence retrieval performance over all other approaches, as well as convincingly outscores mapping methods while maintaining parity with jointly trained methods on word-translation.It also achieves parity with a deep RNN method on a zero-shot cross-lingual document classification task, requiring far fewer computational resources for training and inference.As an additional advantage, our bilingual method also improves the quality of monolingual word vectors despite training on much smaller datasets.  We make our code and models publicly available.

Gemeinsame Methode zum Erlernen von sprachenübergreifenden Einbettungen mit modernster Leistung für sprachenübergreifende Aufgaben und einsprachige Qualität


To act and plan in complex environments, we posit that agents should have a mental simulator of the world with three characteristics:(a) it should build an abstract state representing the condition of the world;(b) it should form a belief which represents uncertainty on the world;(c) it should go beyond simple step-by-step simulation, and exhibit temporal abstraction.Motivated by the absence of a model satisfying all these requirements, we propose TD-VAE, a generative sequence model that learns representations containing explicit beliefs about states several steps into the future, and that can be rolled out directly without single-step transitions.TD-VAE is trained on pairs of temporally separated time points, using an analogue of temporal difference learning used in reinforcement learning.

Generatives Modell zeitlicher Daten, das einen online belief state aufbaut, im latenten Raum operiert und sprunghafte Vorhersagen und Zustandsänderungen vornimmt.


This paper introduces an information theoretic co-training objective for unsupervised learning. We consider the problem of predicting the future. Rather than predict future sensations (image pixels or sound waves) we predict ``hypotheses'' to be confirmed by future sensations. More formally, we assume a population distribution on pairs $(x,y)$ where we can think of $x$ as a past sensation and $y$ as a future sensation. We train both a predictor model $P_\Phi(z|x)$ and a confirmation model $P_\Psi(z|y)$ where we view $z$ as hypotheses (when predicted) or facts (when confirmed). For a population distribution on pairs $(x,y)$ we focus on the problem of measuring the mutual information between $x$ and $y$. By the data processing inequality this mutual information is at least as large as the mutual information between $x$ and $z$ under the distribution on triples $(x,z,y)$ defined by the confirmation model $P_\Psi(z|y)$.The information theoretic training objective for $P_\Phi(z|x)$ and $P_\Psi(z|y)$ can be viewed as a form of co-training where we want the prediction from $x$ to match the confirmation from $y$.We give experiments on applications to learning phonetics on the TIMIT dataset.

Stellt ein informationstheoretisches Trainingsziel für Co-Training vor und demonstriert dessen Leistungsfähigkeit beim unüberwachten Lernen von Phonetik.


Generative models for singing voice have been mostly concerned with the task of "singing voice synthesis," i.e., to produce singing voice waveforms given musical scores and text lyrics.In this work, we explore a novel yet challenging alternative: singing voice generation without pre-assigned scores and lyrics, in both training and inference time.In particular, we experiment with three different schemes:1) free singer, where the model generates singing voices without taking any conditions;2) accompanied singer, where the model generates singing voices over a waveform of instrumental music; and3) solo singer, where the model improvises a chord sequence first and then uses that to generate voices.We outline the associated challenges and propose a pipeline to tackle these new tasks.This involves the development of source separation and transcription models for data preparation, adversarial networks for audio generation, and customized metrics for evaluation.

Unsere Modelle erzeugen Gesangsstimmen ohne Text und Partitur. Sie nehmen eine Begleitung als Eingabe und geben Gesangsstimmen aus.


The carbon footprint of natural language processing (NLP) research has been increasing in recent years due to its reliance on large and inefficient neural network implementations.Distillation is a network compression technique which attempts to impart knowledge from a large model to a smaller one.We use teacher-student distillation  to  improve  the  efficiency  of  the  Biaffine  dependency  parser  which obtains state-of-the-art performance with respect to accuracy and parsing speed (Dozat & Manning, 2016).  When distilling to 20% of the original model’s trainable parameters, we only observe an average decrease of ∼1 point for both UAS and LAS across a number of diverse Universal Dependency treebanks while being 2.26x (1.21x) faster than the baseline model on CPU (GPU) at inference time.We also observe a small increase in performance when compressing to 80% for some treebanks.  Finally, through distillation we attain a parser which is not only faster but also more accurate than the fastest modern parser on the Penn Treebank.

Wir steigern die Effizienz von Dependenzparsern mit neuronalen Netzen durch Lehrer-Schüler-Destillation.


While autoencoders are a key technique in representation learning for continuous structures, such as images or wave forms, developing general-purpose autoencoders for discrete structures, such as text sequence or discretized images, has proven to be more challenging.In particular, discrete inputs make it more difficult to learn a smooth encoder that preserves the complex local relationships in the input space.In this work, we propose an adversarially regularized autoencoder (ARAE) with the goal of learning more robust discrete-space representations.ARAE jointly trains both a rich discrete-space encoder, such as an RNN, and a simpler continuous space generator function, while using generative adversarial network (GAN) training to constrain the distributions to be similar.This method yields a smoother contracted code space that maps similar inputs to nearby codes, and also an implicit latent variable GAN model for generation.Experiments on text and discretized images demonstrate that the GAN model produces clean interpolations and captures the multimodality of the original space, and that the autoencoder produces improvements in semi-supervised learning as well as state-of-the-art results in unaligned text style transfer task using only a shared continuous-space representation.

Adversarially Regularized Autoencoders lernen glatte Repräsentationen diskreter Strukturen und ermöglichen so interessante Ergebnisse bei der Texterzeugung, wie z.B. unaligned style transfer, semi-supervised learning und latent space interpolation and arithmetic.


When data arise from multiple latent subpopulations, machine learning frameworks typically estimate parameter values independently for each sub-population.In this paper, we proposeto overcome these limits by considering samples as tasks in a multitask learning framework.

Wir stellen eine Methode zur Schätzung von Sammlungen von Regressionsmodellen vor, bei der jedes Modell auf eine einzelne Stichprobe zugeschnitten ist.


In this work, we first conduct mathematical analysis on the memory, which isdefined as a function that maps an element in a sequence to the current output,of three RNN cells; namely, the simple recurrent neural network (SRN), the longshort-term memory (LSTM) and the gated recurrent unit (GRU).Based on theanalysis, we propose a new design, called the extended-long short-term memory(ELSTM), to extend the memory length of a cell.Next, we present a multi-taskRNN model that is robust to previous erroneous predictions, called the dependentbidirectional recurrent neural network (DBRNN), for the sequence-in-sequenceout(SISO) problem.Finally, the performance of the DBRNN model with theELSTM cell is demonstrated by experimental results.

Eine rekurrente neuronale Netzwerkzelle mit erweitertem Long Short Term Memory und ein Multi-Task-RNN-Modell für Sequence-in-Sequence-out Probleme


Many recently trained neural networks employ large numbers of parameters to achieve good performance.One may intuitively use the number of parameters required as a rough gauge of the difficulty of a problem.But how accurate are such notions?How many parameters are really needed?In this paper we attempt to answer this question by training networks not in their native parameter space, but instead in a smaller, randomly oriented subspace.We slowly increase the dimension of this subspace, note at which dimension solutions first appear, and define this to be the intrinsic dimension of the objective landscape.The approach is simple to implement, computationally tractable, and produces several suggestive conclusions.Many problems have smaller intrinsic dimensions than one might suspect, and the intrinsic dimension for a given dataset varies little across a family of models with vastly different sizes.This latter result has the profound implication that once a parameter space is large enough to solve a problem, extra parameters serve directly to increase the dimensionality of the solution manifold.Intrinsic dimension allows some quantitative comparison of problem difficulty across supervised, reinforcement, and other types of learning where we conclude, for example, that solving the inverted pendulum problem is 100 times easier than classifying digits from MNIST, and playing Atari Pong from pixels is about as hard as classifying CIFAR-10.In addition to providing new cartography of the objective landscapes wandered by parameterized models, the method is a simple technique for constructively obtaining an upper bound on the minimum description length of a solution.A byproduct of this construction is a simple approach for compressing networks, in some cases by more than 100 times.

Wir trainieren in zufälligen Unterräumen des Parameterraums, um zu messen, wie viele Dimensionen tatsächlich erforderlich sind, um eine Lösung zu finden.


Graph Neural Networks as a combination of Graph Signal Processing and Deep Convolutional Networks shows great power in pattern recognition in non-Euclidean domains.In this paper, we propose a new method to deploy two pipelines based on the duality of a graph to improve accuracy.By exploring the primal graph and its dual graph where nodes and edges can be treated as one another, we have exploited the benefits of both vertex features and edge features.As a result, we have arrived at a framework that has great potential in both semisupervised and unsupervised learning.

Ein primäres duales neuronales Netzmodell für halb-überwachtes Lernen


We describe two end-to-end autoencoding models for semi-supervised graph-based dependency parsing.The first model is a Local Autoencoding Parser (LAP) encoding the input using continuous latent variables in a sequential manner; The second model is a Global Autoencoding Parser (GAP) encoding the input into dependency trees as latent variables, with exact inference.Both models consist of two parts: an encoder enhanced by deep neural networks (DNN) that can utilize the contextual information to encode the input into latent variables, and a decoder which is a generative model able to reconstruct the input.Both LAP and GAP admit a unified structure with different loss functions for labeled and unlabeled data with shared parameters.We conducted experiments on WSJ and UD dependency parsing data sets, showing that our models can exploit the unlabeled data to boost the performance given a limited amount of labeled data.

Wir beschreiben zwei End-to-End-Auto-Coding-Parser für semi-supervised graph-based dependency parsing.


We improve previous end-to-end differentiable neural networks (NNs) with fastweight memories.A gate mechanism updates fast weights at every time step ofa sequence through two separate outer-product-based matrices generated by slowparts of the net.The system is trained on a complex sequence to sequence variationof the Associative Retrieval Problem with roughly 70 times more temporalmemory (i.e. time-varying variables) than similar-sized standard recurrent NNs(RNNs).In terms of accuracy and number of parameters, our architecture outperformsa variety of RNNs, including Long Short-Term Memory, Hypernetworks,and related fast weight architectures.

Ein verbessertes Fast-Weight-Netzwerk, das bessere Ergebnisse bei einer allgemeinen Spielzeugaufgabe zeigt.


The field of deep learning has been craving for an optimization method that shows outstanding property for both optimization and generalization.  We propose a method for mathematical optimization based on flows along geodesics, that is, the shortest paths between two points, with respect to the Riemannian metric induced by a non-linear function.In our method, the flows refer to Exponentially Decaying Flows (EDF), as they can be designed to converge on the local solutions exponentially.In this paper, we conduct experiments to show its high performance on optimization benchmarks (i.e., convergence properties), as well as its potential for producing good machine learning benchmarks (i.e., generalization properties).

Einführung einer neuen Optimierungsmethode und ihre Anwendung auf Deep Learning.