Deep Convolution Neural Networks (CNNs), rooted by the pioneer work of  \cite{Hinton1986,LeCun1985,Alex2012}, and summarized in \cite{LeCunBengioHinton2015},   have been shown to be very useful in a variety of fields.  The  state-of-the art CNN machines such as image rest net \cite{He_2016_CVPR} are described by real value inputs and kernel convolutions followed by the local and  non-linear rectified linear outputs.  Understanding the role of these layers, the accuracy and limitations of them,  as well as making them more efficient (fewer parameters)  are all ongoing research questions.   Inspired in quantum theory, we propose the use of complex value kernel functions, followed by the local non-linear  absolute (modulus) operator square.We argue that an advantage of quantum inspired complex kernels is robustness to realistic unpredictable scenarios (such as clutter noise,  data deformations).We study a concrete problem of shape detection and show that when multiple overlapping shapes are deformed and/or clutter noise is added, a convolution layer with quantum inspired complex kernels outperforms the statistical/classical kernel counterpart and a "Bayesian shape estimator" .The superior performance is due to the quantum phenomena of interference, not present in classical CNNs.

Ein quanteninspirierter Kernel für ein Convolutional Network, das Interferenzphänomene aufweist, kann sehr nützlich sein (und mit dem realen Gegenstück verglichen werden).


We present an artificial intelligence research platform inspired by the human game genre of MMORPGs (Massively Multiplayer Online Role-Playing Games, a.k.a. MMOs).We demonstrate how this platform can be used to study behavior and learning in large populations of neural agents.Unlike currently popular game environments, our platform supports persistent environments, with variable number of agents, and open-ended task descriptions.The emergence of complex life on Earth is often attributed to the arms race that ensued from a huge number of organisms all competing for finite resources.Our platform aims to simulate this setting in microcosm: we conduct a series of experiments to test how large-scale multiagent competition can incentivize the development of skillful behavior.We find that population size magnifies the complexity of the behaviors that emerge and results in agents that out-compete agents trained in smaller populations.

Eine MMO-inspirierte Forschungsspielplattform zur Untersuchung des aufkommenden Verhaltens großer Populationen in einer komplexen Umgebung.


In this paper, we propose an improved quantitative evaluation framework for Generative Adversarial Networks (GANs) on generating domain-specific images, where we improve conventional evaluation methods on two levels: the feature representation and the evaluation metric.Unlike most existing evaluation frameworks which transfer the representation of ImageNet inception model to map images onto the feature space, our framework uses a specialized encoder to acquire fine-grained domain-specific representation.Moreover, for datasets with multiple classes, we propose Class-Aware Frechet Distance (CAFD), which employs a Gaussian mixture model on the feature space to better fit the multi-manifold feature distribution.Experiments and analysis on both the feature level and the image level were conducted to demonstrate improvements of our proposed framework over the recently proposed state-of-the-art FID method.To our best knowledge, we are the first to provide counter examples where FID gives inconsistent results with human judgments.It is shown in the experiments that our framework is able to overcome the shortness of FID and improves robustness.Code will be made available.

Diese Arbeit verbessert die bestehende stichprobenbasierte Bewertung für GANs und enthält einige aufschlussreiche Experimente.


Recent efforts on training light-weight binary neural networks offer promising execution/memory efficiency.This paper introduces ResBinNet, which is a composition of two interlinked methodologies aiming to address the slow convergence speed and limited accuracy of binary convolutional neural networks.The first method, called residual binarization, learns a multi-level binary representation for the features within a certain neural network layer.The second method, called temperature adjustment, gradually binarizes the weights of a particular layer.The two methods jointly learn a set of soft-binarized parameters that improve the convergence rate and accuracy of binary neural networks.We corroborate the applicability and scalability of ResBinNet by implementing a prototype hardware accelerator.The accelerator is reconfigurable in terms of the numerical precision of the binarized features, offering a trade-off between runtime and inference accuracy.

Residuale binäre neuronale Netze verbessern die Konvergenzrate und die Inferenzgenauigkeit der binären neuronalen Netze erheblich.


In real-world machine learning applications, large outliers and pervasive noise are commonplace, and access to clean training data as required by standard deep autoencoders is unlikely.Reliably detecting anomalies in a given set of images is a task of high practical relevance for visual quality inspection, surveillance, or medical image analysis.Autoencoder neural networks learn to reconstruct normal images, and hence can classify those images as anomalous if the reconstruction error exceeds some threshold.In this paper, we proposed an unsupervised method based on subset scanning over autoencoder activations.The contributions of our work are threefold.First, we propose a novel method combining detection with reconstruction error and subset scanning scores to improve the anomaly score of current autoencoders without requiring any retraining.Second, we provide the ability to inspect and visualize the set of anomalous nodes in the reconstruction error space that make a sample noised.Third, we show that subset scanning can be used for anomaly detection in the inner layers of the autoencoder.We provide detection power results for several untargeted adversarial noise models under standard datasets.

Unüberwachtes Verfahren zur Erkennung gegnerischer Proben im Aktivierungs- und Rekonstruktionsfehlerraum des Autoencoders.


Learning knowledge graph embeddings (KGEs) is an efficient approach to knowledge graph completion.Conventional KGEs often suffer from limited knowledge representation, which causes less accuracy especially when training on sparse knowledge graphs.To remedy this, we present Pretrain-KGEs, a training framework for learning better knowledgeable entity and relation embeddings, leveraging the abundant linguistic knowledge from pretrained language models.Specifically, we propose a unified approach in which we first learn entity and relation representations via pretrained language models and use the representations to initialize entity and relation embeddings for training KGE models.Our proposed method is model agnostic in the sense that it can be applied to any variant of KGE models.Experimental results show that our method can consistently improve results and achieve state-of-the-art performance using different KGE models such as TransE and QuatE, across four benchmark KG datasets in link prediction and triplet classification tasks.

Wir schlagen vor, wissensbasierte Entitäts- und Beziehungsrepräsentationen von Bert für die Einbettung von Wissensgraphen zu lernen.


We describe a novel way of representing a symbolic knowledge base (KB) called a sparse-matrix reified KB.  This representation enables neural modules that are fully differentiable, faithful to the original semantics of the KB, expressive enough to model multi-hop inferences, and scalable enough to use with realistically large KBs.The sparse-matrix reified KB can be distributed across multiple GPUs, can scale to tens of millions of entities and facts, and is orders of magnitude faster than naive sparse-matrix implementations.  The reified KB enables very simple end-to-end architectures to obtain competitive performance on several benchmarks representing two families of tasks: KB completion, and learning semantic parsers from denotations.

Ein skalierbares differenzierbares neuronales Modul, das Schlussfolgerungen auf symbolischen KBs implementiert.


We study the Cross-Entropy Method (CEM) for the non-convex optimization of a continuous and parameterized objective function and introduce a differentiable variant (DCEM) that enables us to differentiate the output of CEM with respect to the objective function's parameters.In the machine learning setting this brings CEM inside of the end-to-end learning pipeline in cases this has otherwise been impossible.We show applications in a synthetic energy-based structured prediction task and in non-convex continuous control.In the control setting we show on the simulated cheetah and walker tasks that we can embed their optimal action sequences with DCEM and then use policy optimization to fine-tune components of the controller as a step towards combining model-based and model-free RL.

DCEM lernt latente Bereiche für Optimierungsprobleme und hilft, die Lücke zwischen modellbasiertem und modellfreiem RL zu schließen --- wir erstellen einen differenzierbaren Controller und fine-tunen Teile davon mit PPO.


We propose a new framework for entity and event extraction based on generative adversarial imitation learning -- an inverse reinforcement learning method using generative adversarial network (GAN).We assume that instances and labels yield to various extents of difficulty and the gains and penalties (rewards) are expected to be diverse.We utilize discriminators to estimate proper rewards according to the difference between the labels committed by ground-truth (expert) and the extractor (agent).  Experiments also demonstrate that the proposed framework outperforms state-of-the-art methods.

Wir verwenden dynamische Belohnungen, um Ereignisextraktoren zu trainieren.


Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal.To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specific to such scale.We find that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick", allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input.Our modifications lead to models which set the new state of the art in class-conditional image synthesis.When trained on ImageNet at 128x128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.3 and Frechet Inception Distance (FID) of 9.6, improving over the previous best IS of 52.52 and FID of 18.65.

GANs profitieren von der Hochskalierung.


In this work, we present a novel upper bound of target error to address the problem for unsupervised domain adaptation.Recent studies reveal that a deep neural network can learn transferable features which generalize well to novel tasks.Furthermore,  Ben-David et al. (2010) provide an upper bound for target error when transferring the knowledge, which can be summarized as minimizing the source error and  distance between marginal distributions simultaneously.However, common methods based on the theory usually ignore the joint error such that samples from different classes might be mixed together when matching marginal distribution.And in such case, no matter how we minimize the marginal discrepancy, the target error is not bounded due to an increasing joint error.To address this problem, we propose a general upper bound taking joint error into account, such that the undesirable case can be properly penalized.In addition, we utilize constrained hypothesis space to further formalize a tighter bound as well as a novel cross margin discrepancy to measure the dissimilarity between hypotheses which alleviates instability during adversarial learning.Extensive empirical evidence shows that our proposal outperforms related approaches in image classification error rates on standard domain adaptation benchmarks.

Der gemeinsame Fehler ist für die unüberwachte Bereichsanpassung von Bedeutung, insbesondere wenn die Bereichsverschiebung groß ist.


A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model.To address this issue, in this paper, we develop knowledge flow which moves ‘knowledge’ from multiple deep nets, referred to as teachers, to a new deep net model, called the student.The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too.Upon training with knowledge flow the student is independent of the teachers.We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming fine-tuning and other ‘knowledge exchange’ methods.

"Knowledge Flow" trainiert ein tiefes Netz (Student), indem es Informationen von mehreren Netzen (Lehrern) einspeist. Der Student ist nach dem Training unabhängig und erbringt sehr gute Leistungen bei den gelernten Aufgaben, unabhängig von der Einstellung (Reinforcement oder supervised Learning).


Despite the impressive performance of deep neural networks (DNNs) on numerous learning  tasks, they still exhibit uncouth behaviours.One puzzling behaviour is the subtle sensitive reaction of DNNs to various noise attacks.Such a nuisance has strengthened the line of research around developing and training noise-robust networks.In this work, we propose a new training regularizer that aims to minimize the probabilistic expected training loss of a DNN subject to a generic Gaussian input.We provide an efficient and simple approach to approximate such a regularizer for arbitrarily deep networks.This is done by leveraging the analytic expression of the output mean of a shallow neural network, avoiding the need for memory and computation expensive data augmentation.We conduct extensive experiments on LeNet and AlexNet on various datasets including MNIST, CIFAR10, and CIFAR100 to demonstrate the effectiveness of our proposed regularizer.In particular, we show that networks that are trained with the proposed regularizer benefit from a boost in robustness against Gaussian noise to an equivalent amount of performing 3-21 folds of noisy data augmentation.Moreover, we empirically show on several architectures and datasets that improving robustness against Gaussian noise, by using the new regularizer, can improve the overall robustness against 6 other types of attacks by two orders of magnitude.

Eine effiziente Schätzung des ersten Gaußschen Moments von DNNs als Regularisierer für das Training robuster Netzwerke.


Reading comprehension is a challenging task, especially when executed across longer or across multiple evidence documents, where the answer is likely to reoccur.Existing neural architectures typically do not scale to the entire evidence, and hence, resort to selecting a single passage in the document (either via truncation or other means), and carefully searching for the answer within that passage.However, in some cases, this strategy can be suboptimal,  since by focusing on a specific passage, it becomes difficult to leverage multiple mentions of the same answer throughout the document.In this work, we take a different approach by constructing lightweight models that are combined in a cascade to find the answer.Each submodel consists only of feed-forward networks equipped with an attention mechanism, making it trivially parallelizable.We show that our approach can scale to approximately an order of magnitude larger evidence documents and can aggregate information from multiple mentions of each answer candidate across the document.Empirically, our approach achieves state-of-the-art performance on both the Wikipedia and web domains of the TriviaQA dataset, outperforming more complex, recurrent architectures.

Wir schlagen neuronale Kaskaden vor, einen einfachen und trivial parallelisierbaren Ansatz für das Leseverständnis, der nur aus Feed-Forward-Netzen und Aufmerksamkeit besteht und der auf dem TriviaQA-Datensatz die beste Leistung erzielt.


Compressed forms of deep neural networks are essential in deploying large-scalecomputational models on resource-constrained devices.Contrary to analogousdomains where large-scale systems are build as a hierarchical repetition of small-scale units, the current practice in Machine Learning largely relies on models withnon-repetitive components.In the spirit of molecular composition with repeatingatoms, we advance the state-of-the-art in model compression by proposing AtomicCompression Networks (ACNs), a novel architecture that is constructed by recursiverepetition of a small set of neurons.In other words, the same neurons with thesame weights are stochastically re-positioned in subsequent layers of the network.Empirical evidence suggests that ACNs achieve compression rates of up to threeorders of magnitudes compared to fine-tuned fully-connected neural networks (88×to 1116× reduction) with only a fractional deterioration of classification accuracy(0.15% to 5.33%).Moreover our method can yield sub-linear model complexitiesand permits learning deep ACNs with less parameters than a logistic regressionwith no decline in classification accuracy.

Wir bringen den Stand der Technik bei der Modellkompression voran, indem wir Atomic Compression Networks (ACNs) vorschlagen, eine neuartige Architektur, die durch rekursive Wiederholung einer kleinen Anzahl von Neuronen aufgebaut ist.


Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions.However, a central challenge in video prediction is that the future is highly uncertain: a sequence of past observations of events can imply many possible futures.Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally as in the case of pixel-level autoregressive models, or do not directly optimize the likelihood of the data.To our knowledge, our work is the first to propose multi-frame video prediction with normalizing flows, which allows for direct optimization of the data likelihood, and produces high-quality stochastic predictions.We describe an approach for modeling the latent space dynamics, and demonstrate that flow-based generative models offer a viable and competitive approach to generative modeling of video.

Wir zeigen, dass flussbasierte generative Modelle einen praktikablen und wettbewerbsfähigen Ansatz für die generative Modellierung von Videos bieten.


Understanding the behavior of  stochastic gradient descent (SGD) in the context of deep neural networks has raised lots of concerns recently.Along this line, we theoretically study a general form of gradient based optimization dynamics with unbiased noise, which unifies SGD and standard Langevin dynamics.Through investigating this general optimization dynamics, we analyze the behavior of SGD on escaping from minima and its regularization effects.A novel indicator is derived to characterize the efficiency of escaping from minima through measuring the alignment of noise covariance and the curvature of loss function.Based on this indicator, two conditions are established to show which type of noise structure is superior to isotropic noise in term of escaping efficiency.We further show that the anisotropic noise in SGD satisfies the two conditions, and thus helps to  escape from sharp and poor minima effectively, towards more stable and flat minima that typically generalize well.We verify our understanding through comparingthis anisotropic diffusion with full gradient descent plus isotropic diffusion (i.e. Langevin dynamics) and other types of position-dependent noise.

Wir bieten eine theoretische und empirische Analyse der Rolle des anisotropen Rauschens, das durch den stochastischen Gradienten eingeführt wird, bei der Überwindung von Minima.


Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator to augment the data for policy optimization or value function learning.In this paper, we show how to make more effective use of the model by exploiting its differentiability.We construct a policy optimization algorithm that uses the pathwise derivative of the learned model and policy across future timesteps.Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion.Furthermore, we present a derivation on the monotonic improvement of our objective in terms of the gradient error in the model and value function.We show that our approach(i) is consistently more sample efficient than existing state-of-the-art model-based algorithms,(ii) matches the asymptotic performance of model-free algorithms, and(iii) scales to long horizons, a regime where typically past model-based approaches have struggled.

Policy-Gradient durch Backpropagation über die Zeit unter Verwendung gelernter Modelle und Q-Funktionen. SOTA-Ergebnisse in Benchmark-Umgebungen mit Reinforcement Learning.


Meta-learning algorithms learn to acquire new tasks more quickly from past experience.In the context of reinforcement learning, meta-learning algorithms can acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks.The performance of meta-learning algorithms depends on the tasks available for meta-training: in the same way that supervised learning generalizes best to test points drawn from the same distribution as the training points, meta-learning methods generalize best to tasks from the same distribution as the meta-training tasks.In effect, meta-reinforcement learning offloads the design burden from algorithm design to task design.If we can automate the process of task design as well, we can devise a meta-learning algorithm that is truly automated.In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning.We motivate and describe a general recipe for unsupervised meta-reinforcement learning, and present an instantiation of this approach.Our conceptual and theoretical contributions consist of formulating the unsupervised meta-reinforcement learning problem and describing how task proposals based on mutual information can in principle be used to train optimal meta-learners.Our experimental results indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design and significantly exceeds the performance of learning from scratch.

Meta-Lernen auf selbst vorgeschlagenen Aufgabenverteilungen zur Beschleunigung des Reinforcement Learnings ohne vom Menschen vorgegebene Aufgabenverteilungen.


We introduce a novel method that enables parameter-efficient transfer and multi-task learning with deep neural networks.The basic approach is to learn a model patch - a small set of parameters - that will specialize to each task, instead of fine-tuning the last layer or the entire network.For instance, we show that learning a set of scales and biases is sufficient to convert a pretrained network to perform well on qualitatively different problems (e.g. converting a Single Shot MultiBox Detection (SSD) model into a 1000-class image classification model while reusing 98% of parameters of the SSD feature extractor).Similarly, we show that re-learning existing low-parameter layers (such as depth-wise convolutions) while keeping the rest of the network frozen also improves transfer-learning accuracy significantly.Our approach allows both simultaneous (multi-task) as well as sequential transfer learning.In several multi-task learning problems, despite using much fewer parameters than traditional logits-only fine-tuning, we match single-task performance.

Eine neuartige und praktisch wirksame Methode zur Anpassung vortrainierter neuronaler Netze an neue Aufgaben durch Neutraining einer minimalen (z.B. weniger als 2%) Anzahl von Parametern.


Adversarial examples have somewhat disrupted the enormous success of machine learning (ML) and are causing concern with regards to its trustworthiness: A small perturbation of an input results in an arbitrary failure of an otherwise seemingly well-trained ML system.While studies are being conducted to discover the intrinsic properties of adversarial examples, such as their transferability and universality, there is insufficient theoretic analysis to help understand the phenomenon in a way that can influence the design process of ML experiments.In this paper, we deduce an information-theoretic model which explains adversarial attacks universally as the abuse of feature redundancies in ML algorithms.We prove that feature redundancy is a necessary condition for the existence of adversarial examples.Our model helps to explain the major questions raised in many anecdotal studies on adversarial examples.Our theory is backed up by empirical measurements of the information content of benign and adversarial examples on both image and text datasets.Our measurements show that typical adversarial examples introduce just enough redundancy to overflow the decision making of a machine learner trained on corresponding benign examples.We conclude with actionable recommendations to improve the robustness of machine learners against adversarial examples.

Eine neue theoretische Erklärung für die Existenz adversarialer Beispiele.


We propose a novel unsupervised generative model, Elastic-InfoGAN, that learns to disentangle object identity from other low-level aspects in class-imbalanced datasets.We first investigate the issues surrounding the assumptions about uniformity made by InfoGAN, and demonstrate its ineffectiveness to properly disentangle object identity in imbalanced data.Our key idea is to make the discovery of the discrete latent factor of variation invariant to identity-preserving transformations in real images, and use that as the signal to learn the latent distribution's parameters.Experiments on both artificial (MNIST) and real-world (YouTube-Faces) datasets demonstrate the effectiveness of our approach in imbalanced data by:(i) better disentanglement of object identity as a latent factor of variation; and(ii) better approximation of class imbalance in the data, as reflected in the learned parameters of the latent distribution.

Elastic-InfoGAN ist eine Abwandlung von InfoGAN, die ohne jegliche Überwachung entwirrte Repräsentationen in klassenungleichgewichtigen Daten lernt.


Many real applications show a great deal of interest in learning multiple tasks from different data sources/modalities with unbalanced samples and dimensions.Unfortunately, existing cutting-edge deep multi-task learning (MTL) approaches cannot be directly applied to these settings, due to either heterogeneous input dimensions or the heterogeneity in the optimal network architectures of different tasks.It is thus demanding to develop knowledge-sharing mechanism to handle the intrinsic discrepancies among network architectures across tasks.To this end, we propose a flexible knowledge-sharing framework for jointly learning multiple tasks from distinct data sources/modalities.The proposed framework allows each task to own its task (data)-specific network design, via utilizing a compact tensor representation, while the sharing is achieved through the partially shared latent cores.By providing more elaborate sharing control with latent cores, our framework is effective in transferring task-invariant knowledge, yet also being efficient in learning task-specific features.Experiments on both single and multiple data sources/modalities settings display the promising results of the proposed method, especially favourable in insufficient data scenarios.

Ein verteilter, auf einem latenten Raum basierender Rahmen für die gemeinsame Nutzung von Wissen für tiefes Multi-Task-Lernen.


Board games often rely on visual information such as the location of the game pieces and textual information on cards.Due to this reliance on visual feedback, blind players are at a disadvantage because they cannot read the cards or see the location of the game pieces and may be unable to play a game without sighted help.We present Game Changer, an augmented workspace that provides both audio descriptions and tactile additions to make the state of the board game accessible to blind and visually impaired players.In this paper, we describe the design of Game Changer and present findings from a user study in which 7 blind participants used Game Changer to play against a sighted partner.Most players stated the game was more accessible with the additions from Game Changer and felt that Game Changer could be used to augment other games.

Game Changer ist ein System, das sowohl Audiobeschreibungen als auch taktile Ergänzungen bietet, um den Zustand des Brettspiels für blinde und sehbehinderte Spieler zugänglich zu machen.


In many applications labeled data is not readily available, and needs to be collected via pain-staking human supervision.We propose a rule-exemplar model for collecting human supervision to combine the scalability of rules with the quality of instance labels.  The supervision is coupled such that it is both natural for humans and synergistic for learning.We propose a training algorithm that jointly denoises rules via latent coverage variables, and trains the model through a soft implication loss over the coverage and label variables.  Empirical evaluation on five different tasks shows that (1) our algorithm is more accurate than several existing methods of learning from a mix of clean and noisy supervision, and (2)  the coupled rule-exemplar supervision is effective in denoising rules.

Gekoppelte Regel-Beispiel-Überwachung und ein Implikationsverlust helfen, gemeinsam zu lernen, Störungen aus Regeln zu entfernen und Labels zu implizieren.


We consider a problem of learning the reward and policy from expert examples under unknown dynamics.Our proposed method builds on the framework of generative adversarial networks and introduces the empowerment-regularized maximum-entropy inverse reinforcement learning to learn near-optimal rewards and policies.Empowerment-based regularization prevents the policy from overfitting to expert demonstrations, which advantageously leads to more generalized behaviors that result in learning near-optimal rewards.Our method simultaneously learns empowerment through variational information maximization along with the reward and policy under the adversarial learning formulation.We evaluate our approach on various high-dimensional complex control tasks.We also test our learned rewards in challenging transfer learning problems where training and testing environments are made to be different from each other in terms of dynamics or structure.The results show that our proposed method not only learns near-optimal rewards and policies that are matching expert behavior but also performs significantly better than state-of-the-art inverse reinforcement learning algorithms.

Unsere Methode führt das empowerment-regularized maximum-entropy inverse reinforcement learning ein, um nahezu optimale Belohnungen und Strategien aus Expertendemonstrationen zu lernen.


Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies). Such regularities motivate our hypothesis that RNNs that show such regularities implicitly compile symbolic structures into tensor product representations (TPRs; Smolensky, 1990), which additively combine tensor products of vectors representing roles (e.g.,  sequence positions) and vectors representing fillers (e.g., particular words). To test this hypothesis, we introduce Tensor Product Decomposition Networks (TPDNs), which use TPRs to approximate existing vector representations. We demonstrate using synthetic data that TPDNs can successfully approximate linear and tree-based RNN autoencoder representations, suggesting that these representations exhibit interpretable compositional structure; we explore the settings that lead RNNs to induce such structure-sensitive representations.  By contrast, further TPDN experiments show that the representations of four models trained to encode naturally-occurring sentences can be largely approximated with a bag of words, with only marginal improvements from more sophisticated structures. We conclude that TPDNs provide a powerful method for interpreting vector representations, and that standard RNNs can induce compositional sequence representations that are remarkably well approximated byTPRs; at the same time, existing training tasks for sentence representation learning may not be sufficient for inducing robust structural representations

RNNs implementieren implizit Tensor-Produkt-Darstellungen, eine prinzipielle und interpretierbare Methode zur Darstellung symbolischer Strukturen im kontinuierlichen Raum.


We address the problem of teaching an RNN to approximate list-processing algorithms given a small number of input-output training examples.Our approach is to generalize the idea of parametricity from programming language theory to formulate a semantic property that distinguishes common algorithms from arbitrary non-algorithmic functions.This characterization leads naturally to a learned data augmentation scheme that encourages RNNs to learn algorithmic behavior and enables small-sample learning in a variety of list-processing tasks.

Die gelernte Datenerweiterung bewirkt, dass Algorithmen durch induktive Verzerrungen begünstigt werden, so dass RNNs Listenverarbeitungsalgorithmen aus weniger Beispielen lernen können.


The goal of multi-label learning (MLL) is to associate a given instance with its relevant labels from a set of concepts.Previous works of MLL mainly focused on the setting where the concept set is assumed to be fixed, while many real-world applications require introducing new concepts into the set to meet new demands.One common need is to refine the original coarse concepts and split them into finer-grained ones, where the refinement process typically begins with limited labeled data for the finer-grained concepts.To address the need, we propose a special weakly supervised MLL problem that not only focuses on the situation of limited fine-grained supervision but also leverages the hierarchical relationship between the coarse concepts and the fine-grained ones.The problem can be reduced to a multi-label version of negative-unlabeled learning problem using the hierarchical relationship.We tackle the reduced problem with a meta-learning approach that learns to assign pseudo-labels to the unlabeled entries.Experimental results demonstrate that our proposed method is able to assign accurate pseudo-labels, and in turn achieves superior classification performance when compared with other existing methods.

Wir schlagen ein spezielles schwach überwachtes Multi-Label-Lernproblem zusammen mit einem neu zugeschnittenen Algorithmus vor, der den zugrundeliegenden Klassifikator lernt, indem er lernt, Pseudo-Labels zuzuweisen.


We develop a reinforcement learning based search assistant which can assist users through a set of actions and sequence of interactions to enable them realize their intent.Our approach caters to subjective search where the user is seeking digital assets such as images which is fundamentally different from the tasks which have objective and limited search modalities.Labeled conversational data is generally not available in such search tasks and training the agent through human interactions can be time consuming.We propose a stochastic virtual user which impersonates a real user and can be used to sample user behavior efficiently to train the agent which accelerates the bootstrapping of the agent.We develop A3C algorithm based context preserving architecture which enables the agent to provide contextual assistance to the user.We compare the A3C agent with Q-learning and evaluate its performance on average rewards and state values it obtains with the virtual user in validation episodes.Our experiments show that the agent learns to achieve higher rewards and better states.

Ein auf Reinforcement Learning basierender konversationeller Suchassistent, der kontextbezogene Unterstützung bei der subjektiven Suche (z.B. nach digitalen Assets) bietet.


We present a simple approach based on pixel-wise nearest neighbors to understand and interpret the functioning of state-of-the-art neural networks for pixel-level tasks.We aim to understand and uncover the synthesis/prediction mechanisms of state-of-the-art convolutional neural networks.To this end, we primarily analyze the synthesis process of generative models and the prediction mechanism of discriminative models.The main hypothesis of this work is that convolutional neural networks for pixel-level tasks learn a fast compositional nearest neighbor synthesis/prediction function.Our experiments on semantic segmentation and image-to-image translation show qualitative and quantitative evidence supporting this hypothesis.

Convolutional Neural Networks verhalten sich wie kompositorische Nearest Neighbors!


Consider a world in which events occur that involve various entities.Learning how to predict future events from patterns of past events becomes more difficult as we consider more types of events.Many of the patterns detected in the dataset by an ordinary LSTM will be spurious since the number of potential pairwise correlations, for example, grows quadratically with the number of events.We propose a type of factorial LSTM architecture where different blocks of LSTM cells are responsible for capturing different aspects of the world state.We use Datalog rules to specify how to derive the LSTM structure from a database of facts about the entities in the world.This is analogous to how a probabilistic relational model (Getoor & Taskar, 2007) specifies a recipe for deriving a graphical model structure from a database.In both cases, the goal is to obtain useful inductive biases by encoding informed independence assumptions into the model.We specifically consider the neural Hawkes process, which uses an LSTM to modulate the rate of instantaneous events in continuous time.In both synthetic and real-world domains, we show that we obtain better generalization by using appropriate factorial designs specified by simple Datalog programs.

Faktorisierung von LSTM-Zuständen und LSTM-Gewichtsmatrizen mit Nullen und Bindungen entsprechend den strukturellen Verzerrungen in der realen Welt, die durch Datalog-Programme ausgedrückt werden.


In this work, we aim to solve data-driven optimization problems, where the goal is to find an input that maximizes an unknown score function given access to a dataset of input, score pairs.Inputs may lie on extremely thin manifolds in high-dimensional spaces, making the optimization prone to falling-off the manifold.Further, evaluating the unknown function may be expensive, so the algorithm should be able to exploit static, offline data.We propose model inversion networks (MINs) as an approach to solve such problems.Unlike prior work, MINs scale to extremely high-dimensional input spaces and can efficiently leverage offline logged datasets for optimization in both contextual and non-contextual settings.We show that MINs can also be extended to the active setting, commonly studied in prior work, via a simple, novel and effective scheme for active data collection.Our experiments show that MINs act as powerful optimizers on a range of contextual/non-contextual, static/active problems including optimization over images and protein designs and learning from logged bandit feedback.

Wir schlagen einen neuartigen Ansatz zur Lösung von datengesteuerten, modellbasierten Optimierungsproblemen sowohl in passiven als auch in aktiven Umgebungen vor, der auf hochdimensionale Eingaberäume skaliert werden kann.


Generating formal-language represented by relational tuples, such as Lisp programs or mathematical expressions, from a natural-language input is an extremely challenging task because it requires to explicitly capture discrete symbolic structural information from the input to generate the output.Most state-of-the-art neural sequence models do not explicitly capture such structure information, and thus do not perform well on these tasks.In this paper, we propose a new encoder-decoder model based on Tensor Product Representations (TPRs) for Natural- to Formal-language generation, called TP-N2F.The encoder of TP-N2F employs TPR 'binding' to encode natural-language symbolic structure in vector space and the decoder uses TPR 'unbinding' to generate a sequence of relational tuples, each consisting of a relation (or operation) and a number of arguments, in symbolic space.TP-N2F considerably outperforms LSTM-based Seq2Seq models, creating a new state of the art results on two benchmarks: the MathQA dataset for math problem solving, and the AlgoList dataset for program synthesis.Ablation studies show that improvements are mainly attributed to the use of TPRs in both the encoder and decoder to explicitly capture relational structure information for symbolic reasoning.

In dieser Arbeit schlagen wir ein neues Encoder-Decoder-Modell vor, das auf Tensor-Produkt-Darstellungen für die Generierung natürlicher und formaler Sprache basiert, genannt TP-N2F.


This paper we present a defogger, a model that learns to predict future hidden information from partial observations.We formulate this model in the context of forward modeling and leverage spatial and sequential constraints and correlations via convolutional neural networks and long short-term memory networks, respectively.We evaluate our approach on a large dataset of human games of StarCraft: Brood War, a real-time strategy video game.Our models consistently beat strong rule-based baselines and qualitatively produce sensible future game states.

In dieser Arbeit wird ein Defogger vorgestellt, ein Modell, das lernt, zukünftige verborgene Informationen aus Teilbeobachtungen vorherzusagen, angewandt auf einen StarCraft-Datensatz.


In this work we study generalization of neural networks in gradient-based meta-learning by analyzing various properties of the objective landscapes.We experimentally demonstrate that as meta-training progresses, the meta-test solutions obtained by adapting the meta-train solution of the model to new tasks via few steps of gradient-based fine-tuning, become flatter, lower in loss, and further away from the meta-train solution.We also show that those meta-test solutions become flatter even as generalization starts to degrade, thus providing an experimental evidence against the correlation between generalization and flat minima in the paradigm of gradient-based meta-leaning.Furthermore, we provide empirical evidence that generalization to new tasks is correlated with the coherence between their adaptation trajectories in parameter space, measured by the average cosine similarity between task-specific trajectory directions, starting from a same meta-train solution.We also show that coherence of meta-test gradients, measured by the average inner product between the task-specific gradient vectors evaluated at meta-train solution, is also correlated with generalization.

Wir untersuchen die Generalisierung neuronaler Netze beim gradientenbasierten Meta-Lernen, indem wir verschiedene Eigenschaften der Ziellandschaft analysieren.


There have been multiple attempts with variational auto-encoders (VAE) to learn powerful global representations of complex data using a combination of latent stochastic variables and an autoregressive model over the dimensions of the data.However, for the most challenging natural image tasks the purely autoregressive model with stochastic variables still outperform the combined stochastic autoregressive models.In this paper, we present simple additions to the VAE framework that generalize to natural images by embedding spatial information in the stochastic layers.We significantly improve the state-of-the-art results on MNIST, OMNIGLOT, CIFAR10 and ImageNet when the feature map parameterization of the stochastic variables are combined with the autoregressive PixelCNN approach.Interestingly, we also observe close to state-of-the-art results without the autoregressive part.This opens the possibility for high quality image generation with only one forward-pass.

Wir stellen ein generatives Modell vor, das bei Graustufen- und Naturbildern den neuesten Stand der Technik beweist.


Plain recurrent networks greatly suffer from the vanishing gradient problem while Gated Neural Networks (GNNs) such as Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU) deliver promising results in many sequence learning tasks through sophisticated network designs.This paper shows how we can address this problem in a plain recurrent network by analyzing the gating mechanisms in GNNs.We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates.We compare this model with IRNNs and LSTMs on multiple sequence modeling benchmarks.The RINs demonstrate competitive performance and converge faster in all tasks.Notably, small RIN models produce 12%–67% higher accuracy on the Sequential and Permuted MNIST datasets and reach state-of-the-art performance on the bAbI question answering dataset.

Wir schlagen ein neuartiges Netzwerk mit dem Namen Recurrent Identity Network (RIN) vor, das es einem einfachen rekurrenten Netzwerk ermöglicht, das Problem des verschwindenden Gradienten zu überwinden und gleichzeitig sehr tiefe Modelle ohne den Einsatz von Gates zu trainieren.


Recently, there has been a surge in interest in safe and robust techniques within reinforcement learning (RL). Current notions of risk in RL fail to capture the potential for systemic failures such as abrupt stoppages from system failures or surpassing of safety thresholds and the appropriate responsive controls in such instances.We propose a novel approach to fault-tolerance within RL in which the controller learns a policy can cope with adversarial attacks and random stoppages that lead to failures of the system subcomponents.The results of the paper also cover fault-tolerant (FT) control so that the controller learns to avoid states that carry risk of system failures.By demonstrating that the class of problems is represented by a variant of SGs, we prove the existence of a solution which is a unique fixed point equilibrium of the game and characterise the optimal controller behaviour.We then introduce a value function approximation algorithm that converges to the solution through simulation in unknown environments.

Diese Arbeit befasst sich mit der Fehlertoleranz bei zufälligen und gegnerischen Ausfällen.


We propose a novel framework to generate clean video frames from a single motion-blurred image.While a broad range of literature focuses on recovering a single image from a blurred image, in this work, we tackle a more challenging task i.e.  video restoration from a blurred image.We formulate video restoration from a single blurred image as an inverse problem by setting clean image sequence and their respective motion as latent factors, and the blurred image as an observation.Our framework is based on an encoder-decoder structure with spatial transformer network modules to restore a video sequence and its underlying motion in an end-to-end manner.We design a loss function and regularizers with complementary properties to stabilize the training and analyze variant models of the proposed network.The effectiveness and transferability of our network are highlighted through a large set of experiments on two different types of datasets: camera rotation blurs generated from panorama scenes and dynamic motion blurs in high speed videos.Our code and models will be publicly available.

Wir stellen eine neuartige, einheitliche Architektur vor, die Videobilder aus einem einzigen bewegungsunscharfen Bild durchgängig wiederherstellt.


High performance of deep learning models typically comes at cost of considerable model size and computation time.These factors limit applicability for deployment on memory and battery constraint devices such as mobile phones or embedded systems.In this work we propose a novel pruning technique that eliminates entire filters and neurons according to their relative L1-norm as compared to the rest of the network, yielding more compression and decreased redundancy in the parameters.The resulting network is non-sparse, however, much more compact and requires no special infrastructure for its deployment.We prove the viability of our method by achieving 97.4%, 47.8% and 53% compression of LeNet-5, ResNet-56 and ResNet-110 respectively, exceeding state-of-the-art compression results reported on ResNet without losing any performance compared to the baseline.Our approach does not only exhibit good performance, but is also easy to implement on many architectures.

Wir schlagen ein neuartiges strukturiertes, klassenblindes Pruning-Verfahren vor, um hochkomprimierte neuronale Netze zu erzeugen.


The recent success of neural networks for solving difficult decision tasks has incentivized incorporating smart decision making "at the edge."However, this work has traditionally focused on neural network inference, rather than training, due to memory and compute limitations, especially in emerging non-volatile memory systems, where writes are energetically costly and reduce lifespan.Yet, the ability to train at the edge is becoming increasingly important as it enables applications such as real-time adaptability to device drift and environmental variation, user customization, and federated learning across devices.In this work, we address four key challenges for training on edge devices with non-volatile memory: low weight update density, weight quantization, low auxiliary memory, and online learning.We present a low-rank training scheme that addresses these four challenges while maintaining computational efficiency.We then demonstrate the technique on a representative convolutional neural network across several adaptation problems, where it out-performs standard SGD both in accuracy and in number of weight updates.

Wir verwenden Kronecker-Summen-Approximationen für Low-Rank-Training, um die Herausforderungen beim Training neuronaler Netze auf Edge-Geräten zu bewältigen, die neue Speichertechnologien nutzen.


Knowledge extraction techniques are used to convert neural networks into symbolic descriptions with the objective of producing more comprehensible learning models.The central challenge is to find an explanation which is more comprehensible than the original model while still representing that model faithfully.The distributed nature of deep networks has led many to believe that the hidden features of a neural network cannot be explained by logical descriptions simple enough to be understood by humans, and that decompositional knowledge extraction should be abandoned in favour of other methods.In this paper we examine this question systematically by proposing a knowledge extraction method using \textit{M-of-N} rules which allows us to map the complexity/accuracy landscape of rules describing hidden features in a Convolutional Neural Network (CNN).Experiments reported in this paper show that the shape of this landscape reveals an optimal trade off between comprehensibility and accuracy, showing that each latent variable has an optimal \textit{M-of-N} rule to describe its behaviour.We find that the rules with optimal tradeoff in the first and final layer have a high degree of explainability whereas the rules with the optimal tradeoff in the second and third layer are less explainable.The results shed light on the feasibility of rule extraction from deep networks, and point to the value of decompositional knowledge extraction as a method of explainability.

Untersucht systematisch, wie gut wir die verborgenen Merkmale eines tiefen Netzes mit Hilfe logischer Regeln erklären können.


Recent findings show that deep generative models can judge out-of-distribution samples as more likely than those drawn from the same distribution as the training data.In this work, we focus on variational autoencoders (VAEs) and address the problem of misaligned likelihood estimates on image data.We develop a novel likelihood function that is based not only on the parameters returned by the VAE but also on the features of the data learned in a self-supervised fashion.In this way, the model additionally captures the semantic information that is disregarded by the usual VAE likelihood function.We demonstrate the improvements in reliability of the estimates with experiments on the FashionMNIST and MNIST datasets.

Verbesserte Likelihood-Schätzungen in variablen Autokodierern durch selbstüberwachtes Feauture Lernen.


Direct policy gradient methods for reinforcement learning and continuous control problems are a popularapproach for a variety of reasons: 1) they are easy to implement without explicit knowledge of the underlying model;2) they are an "end-to-end" approach, directly optimizing the performance metric of interest;3) they inherently allow for richly parameterized policies.A notable drawback is that even in the most basic continuous control problem (that of linear quadratic regulators), these methods must solve a non-convex optimization problem, where little is understood about their efficiency from both computational and statistical perspectives.In contrast, system identification and model based planning in optimal control theory have a much more solid theoretical footing, where much is known with regards to their computational and statistical properties.  This work bridges this gap showing that (model free) policy gradient methods globally converge to the optimal solution and are efficient (polynomially so in relevant problem dependent quantities) with regards to their sample and computational complexities.

Dieses Papier zeigt, dass modellfreie Policy-Gradienten-Methoden zur globalen optimalen Lösung für nicht-konvexe linearisierte Kontrollprobleme konvergieren können.


Low-dimensional vector embeddings, computed using LSTMs or simpler techniques, are a popular approach for capturing the “meaning” of text and a form of unsupervised learning useful for downstream tasks.However, their power is not theoretically understood.The current paper derives formal understanding by looking at the subcase of linear embedding schemes.Using the theory of compressed sensing we show that representations combining the constituent word vectors are essentially information-preserving linear measurements of Bag-of-n-Grams (BonG) representations of text.This leads to a new theoretical result about LSTMs: low-dimensional embeddings derived from a low-memory LSTM are provably at least as powerful on classification tasks, up to small error, as a linear classifier over BonG vectors, a result that extensive empirical work has thus far been unable to show.Our experiments support these theoretical findings and establish strong, simple, and unsupervised baselines on standard benchmarks that in some cases are state of the art among word-level methods.We also show a surprising new property of embeddings such as GloVe and word2vec: they form a good sensing matrix for text that is more efficient than random matrices, the standard sparse recovery tool, which may explain why they lead to better representations in practice.

Wir nutzen die Theorie des Compressed Sensing, um zu beweisen, dass LSTMs bei der linearen Textklassifikation mindestens genauso gut abschneiden wie Bag-of-n-Grams.


Some conventional transforms such as Discrete Walsh-Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT) have been widely used as feature extractors in image processing but rarely applied in neural networks.However, we found that these conventional transforms have the ability to capture the cross-channel correlations without any learnable parameters in DNNs.This paper firstly proposes to apply conventional transforms on pointwise convolution, showing that such transforms significantly reduce the computational complexity of neural networks without accuracy performance degradation.Especially for DWHT, it requires no floating point multiplications but only additions and subtractions, which can considerably reduce computation overheads.In addition, its fast algorithm further reduces complexity of floating point addition from O(n^2) to O(nlog n).These non-parametric and low computational properties construct extremely efficient networks in the number parameters and operations, enjoying accuracy gain.Our proposed DWHT-based model gained 1.49% accuracy increase with 79.4% reduced parameters and 48.4% reduced FLOPs compared with its baseline model (MoblieNet-V1) on the CIFAR 100 dataset.

Wir führen neue punktweise Convolution Layers ein, die mit extrem schnellen konventionellen Transformationen in tiefen neuronalen Netzen ausgestattet sind.


We introduce the notion of \emph{lattice representation learning}, in which the representation for some object of interest (e.g. a sentence or an image) is a lattice point in an Euclidean space.Our main contribution is a result for replacing an objective function which employs lattice quantization with an objective function in which quantization is absent, thus allowing optimization techniques based on gradient descent to apply; we call the resulting algorithms \emph{dithered stochastic gradient descent} algorithms as they are designed explicitly to allow for an optimization procedure where only local information is employed.We also argue that a technique commonly used in Variational Auto-Encoders (Gaussian priors and Gaussian approximate posteriors) is tightly connected with the idea of lattice representations, as the quantization error in good high dimensional lattices can be modeled as a Gaussian distribution.We use a traditional encoder/decoder architecture to explore the idea of latticed valued representations, and provide experimental evidence of the potential of using lattice representations by modifying the \texttt{OpenNMT-py} generic \texttt{seq2seq} architecture  so that it can implement not only Gaussian dithering of representations, but also the well known straight-through estimator and its application to vector quantization.

Wir schlagen die Verwendung von Gittern zur Darstellung von Objekten vor und beweisen ein grundlegendes Ergebnis, wie man Netze trainiert, die diese verwenden.


There were many attempts to explain the trade-off between accuracy and adversarial robustness.However,  there  was  no  clear  understanding  of  the  behaviors  of  a  robust  classifier  which  has human-like robustness.We  argue  (1)  why  we  need  to  consider  adversarial  robustness  against  varying  magnitudes  of perturbations not only focusing on a fixed perturbation threshold, (2) why we need to use different method to generate adversarially perturbed samples that can be used to train a robust classifier and measure the robustness of classifiers and (3) why we need to prioritize adversarial accuracies with different magnitudes.We introduce Lexicographical Genuine Robustness (LGR) of classifiers that combines the above requirements.  We also suggest a candidate oracle classifier called "Optimal Lexicographically Genuinely  Robust  Classifier  (OLGRC)"  that  prioritizes  accuracy  on  meaningful  adversarially perturbed  examples  generated  by  smaller  magnitude  perturbations.   The  training  algorithm  for estimating OLGRC requires lexicographical optimization unlike existing adversarial training methods.To apply lexicographical optimization to neural network, we utilize Gradient Episodic Memory (GEM) which was originally developed for continual learning by preventing catastrophic forgetting.

Wir versuchen, einen Klassifikator zu entwerfen und zu trainieren, dessen adversariale Robustheit mehr Ähnlichkeit mit der Robustheit des Menschen hat.


Generating complex discrete distributions remains as one of the challenging problems in machine learning.Existing techniques for generating complex distributions with high degrees of freedom depend on standard generative models like Generative Adversarial Networks (GAN), Wasserstein GAN, and associated variations.Such models are based on an optimization involving the distance between two continuous distributions.We introduce a Discrete Wasserstein GAN (DWGAN) model which is based on a dual formulation of the Wasserstein distance between two discrete distributions.We derive a novel training algorithm and corresponding network architecture based on the formulation.Experimental results are provided for both synthetic discrete data, and real discretized data from MNIST handwritten digits.

Wir schlagen ein diskretes Wasserstein-GAN-Modell (DWGAN) vor, das auf einer dualen Formulierung des Wasserstein-Abstands zwischen zwei diskreten Verteilungen basiert.


We introduce the open-ended, modular, self-improving Omega AI unification architecture which is a refinement of Solomonoff's Alpha architecture, as considered from first principles.The architecture embodies several crucial principles of general intelligence including diversity of representations, diversity of data types, integrated memory, modularity, and higher-order cognition.We retain the basic design of a fundamental algorithmic substrate called an ``AI kernel'' for problem solving and basic cognitive functions like memory, and a larger, modular architecture that re-uses the kernel in many ways.Omega includes eight representation languages and six classes of neural networks, which are briefly introduced.The architecture is intended to initially address data science automation, hence it includes many problem solving methods for statistical tasks.We review the broad software architecture, higher-order cognition, self-improvement, modular neural architectures, intelligent agents, the process and memory hierarchy, hardware abstraction, peer-to-peer computing, and data abstraction facility.

Dies ist ein Überblick über die Omega AGI-Architektur, die die Grundlage für ein Automatisierungssystem für die Datenwissenschaft bildet.


Conversational machine comprehension requires a deep understanding of the conversation history.To enable traditional, single-turn models to encode the history comprehensively, we introduce Flow, a mechanism that can incorporate intermediate representations generated during the process of answering previous questions, through an alternating parallel processing structure.Compared to shallow approaches that concatenate previous questions/answers as input, Flow integrates the latent semantics of the conversation history more deeply.Our model, FlowQA, shows superior performance on two recently proposed conversational challenges (+7.2% F1 on CoQA and +4.0% on QuAC).The effectiveness of Flow also shows in other tasks.By reducing sequential instruction understanding to conversational machine comprehension, FlowQA outperforms the best models on all three domains in SCONE, with +1.8% to +4.4% improvement in accuracy.

Wir schlagen den Flow-Mechanismus und eine Ende-zu-Ende Architektur, FlowQA, vor, die SotA auf zwei QA-Datensätzen für Konversation und eine Aufgabe zum Verstehen sequenzieller Anweisungen ermöglicht.


We consider reinforcement learning and bandit structured prediction problems with very sparse loss feedback: only at the end of an episode.We introduce a novel algorithm, RESIDUAL LOSS PREDICTION (RESLOPE), that solves such problems by automatically learning an internal representation of a denser reward function.RESLOPE operates as a reduction to contextual bandits, using its learned loss representation to solve the credit assignment problem, and a contextual bandit oracle to trade-off exploration and exploitation.RESLOPE enjoys a no-regret reduction-style theoretical guarantee and outperforms state of the art reinforcement learning algorithms in both MDP environments and bandit structured prediction settings.

Wir stellen einen neuartigen Algorithmus zur Lösung von Reinforcement Learning und Bandit-strukturierten Vorhersageproblemen mit sehr spärlicher Verlustrückmeldung vor.


Adversarial neural networks solve many important problems in data science, but are notoriously difficult to train.These difficulties come from the fact that optimal weights for adversarial nets correspond to saddle points, and not minimizers, of the loss function.The alternating stochastic gradient methods typically used for such problems do not reliably converge to saddle points, and when convergence does happen it is often highly sensitive to learning rates.We propose a simple modification of stochastic gradient descent that stabilizes adversarial networks.We show, both in theory and practice, that the proposed method reliably converges to saddle points.This makes adversarial networks less likely to "collapse," and enables faster training with larger learning rates.

Wir stellen eine einfache Modifikation der alternierenden SGD-Methode vor, die als Vorhersageschritt bezeichnet wird und die Stabilität der adversarischen Netzwerke verbessert.


An important type of question that arises in Explainable Planning is a contrastive question, of the form "Why action A instead of action B?".These kinds of questions can be answered with a contrastive explanation that compares properties of the original plan containing A against the contrastive plan containing B. An effective explanation of this type serves to highlight the differences between the decisions that have been made by the planner and what the user would expect, as well as to provide further insight into the model and the planning process.Producing this kind of explanation requires the generation of the contrastive plan.This paper introduces domain-independent compilations of user questions into constraints.These constraints are added to the planning model, so that a solution to the new model represents the contrastive plan.We introduce a formal description of the compilation from user question to constraints in a temporal and numeric PDDL2.1 planning setting.

In diesem Beitrag werden domänenunabhängige Kompilationen von Benutzerfragen in Constraints für kontrastive Erklärungen vorgestellt.


Methods that learn representations of nodes in a graph play a critical role in network analysis since they enable many downstream learning tasks.We propose Graph2Gauss - an approach that can efficiently learn versatile node embeddings on large scale (attributed) graphs that show strong performance on tasks such as link prediction and node classification.Unlike most approaches that represent nodes as point vectors in a low-dimensional continuous space, we embed each node as a Gaussian distribution, allowing us to capture uncertainty about the representation.Furthermore, we propose an unsupervised method that handles inductive learning scenarios and is applicable to different types of graphs: plain/attributed, directed/undirected.By leveraging both the network structure and the associated node attributes, we are able to generalize to unseen nodes without additional training.To learn the embeddings we adopt a personalized ranking formulation w.r.t. the node distances that exploits the natural ordering of the nodes imposed by the network structure.Experiments on real world networks demonstrate the high performance of our approach, outperforming state-of-the-art network embedding methods on several different tasks.Additionally, we demonstrate the benefits of modeling uncertainty - by analyzing it we can estimate neighborhood diversity and detect the intrinsic latent dimensionality of a graph.

Wir betten die Knoten in einem Graphen als Gaußsche Verteilungen ein, die es uns ermöglichen, die Ungewissheit über ihre Darstellung zu erfassen.


While great progress has been made at making neural networks effective across a wide range of tasks, many are surprisingly vulnerable to small, carefully chosen perturbations of their input, known as adversarial examples.In this paper, we advocate for and experimentally investigate the use of logit regularization techniques as an adversarial defense, which can be used in conjunction with other methods for creating adversarial robustness at little to no cost.We demonstrate that much of the effectiveness of one recent adversarial defense mechanism can be attributed to logit regularization and show how to improve its defense against both white-box and black-box attacks, in the process creating a stronger black-box attacks against PGD-based models.

Logit-Regularisierungsmethoden helfen bei der Erklärung und Verbesserung des Stands der Technik der gegnerischen Verteidigung.


In deep learning, performance is strongly affected by the choice of architectureand hyperparameters.While there has been extensive work on automatic hyperpa-rameter optimization for simple spaces, complex spaces such as the space of deeparchitectures remain largely unexplored.As a result, the choice of architecture isdone manually by the human expert through a slow trial and error process guidedmainly by intuition.In this paper we describe a framework for automaticallydesigning and training deep models.We propose an extensible and modular lan-guage that allows the human expert to compactly represent complex search spacesover architectures and their hyperparameters.The resulting search spaces are tree-structured and therefore easy to traverse.Models can be automatically compiled tocomputational graphs once values for all hyperparameters have been chosen.Wecan leverage the structure of the search space to introduce different model searchalgorithms, such as random search, Monte Carlo tree search (MCTS), and sequen-tial model-based optimization (SMBO).We present experiments comparing thedifferent algorithms on CIFAR-10 and show that MCTS and SMBO outperformrandom search.We also present experiments on MNIST, showing that the samesearch space achieves near state-of-the-art performance with a few samples.Theseexperiments show that our framework can be used effectively for model discov-ery, as it is possible to describe expressive search spaces and discover competitivemodels without much effort from the human expert.Code for our framework andexperiments has been made publicly available

Wir beschreiben eine modulare und komponierbare Sprache zur Beschreibung von ausdrucksstarken Suchräumen über Architekturen und einfachen Modellsuchalgorithmen, die auf diese Suchräume angewendet werden.


Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection.The performant systems, however, typically involve big models with numerous parameters.Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems -- the models (often deep networks or wide networks or both) are compute and memory intensive.Low precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footprint of these deployed models.In this paper, we study the combination of these two techniques and show that the performance of low precision networks can be significantly improved by using knowledge distillation techniques.We call our approach Apprentice and show state-of-the-art accuracies using ternary precision and 4-bit precision for many variants of ResNet architecture on ImageNet dataset.We study three schemes in which one can apply knowledge distillation techniques to various stages of the train-and-deploy pipeline.

Wir zeigen, dass Wissenstransfertechniken die Genauigkeit von Netzwerken mit geringer Genauigkeit verbessern können und eine neue Spitzengenauigkeit für ternäre und 4-Bit-Genauigkeit erreichen.


Deep neural networks (DNN) are widely used in many applications.However, their deployment on edge devices has been difficult because they are resource hungry.Binary neural networks (BNN) help to alleviate the prohibitive resource requirements of DNN, where both activations and weights are limited to 1-bit.We propose an improved binary training method (BNN+), by introducing a regularization function that encourages training weights around binary values.In addition to this, to enhance model performance we add trainable scaling factors to our regularization functions.Furthermore, we use an improved approximation of the derivative of the sign activation function in the backward computation.These additions are based on linear operations that are easily implementable into the binary training framework.We show experimental results on CIFAR-10 obtaining an accuracy of 86.5%, on AlexNet and 91.3% with VGG network.On ImageNet, our method also outperforms the traditional BNN method and XNOR-net, using AlexNet by a margin of 4% and 2% top-1 accuracy respectively.

Diese Arbeit stellt einen verbesserten Trainingsmechanismus vor, mit dem binäre Netze mit geringerem Genauigkeitsabfall erzeugt werden können, was dazu beiträgt, die Lücke zu ihrem Gegenstück mit voller Präzision zu schließen.


Clustering is a fundamental machine learning method.The quality of its results is dependent on the data distribution.For this reason, deep neural networks can be used for learning better representations of the data.In this paper, we propose a systematic taxonomy for clustering with deep learning, in addition to a review of methods from the field.Based on our taxonomy, creating new methods is more straightforward.We also propose a new approach which is built on the taxonomy and surpasses some of the limitations of some previous work.Our experimental evaluation on image datasets shows that the method approaches state-of-the-art clustering quality, and performs better in some cases.

Vereinheitlichender Rahmen zur Durchführung von Clustering mit tiefen neuronalen Netzen


Generative models often use human evaluations to determine and justify progress.Unfortunately, existing human evaluation methods are ad-hoc: there is currently no standardized, validated evaluation that: (1) measures perceptual fidelity, (2) is reliable, (3) separates models into clear rank order, and (4) ensures high-quality measurement without intractable cost.In response, we construct Human-eYe Perceptual Evaluation (HYPE), a human metric that is (1) grounded in psychophysics research in perception, (2) reliable across different sets of randomly sampled outputs from a model, (3) results in separable model performances, and (4) efficient in cost and time.We introduce two methods.The first, HYPE-Time, measures visual perception under adaptive time constraints to determine the minimum length of time (e.g., 250ms) that model output such as a generated face needs to be visible for people to distinguish it as real or fake.The second, HYPE-Infinity, measures human error rate on fake and real images with no time constraints, maintaining stability and drastically reducing time and cost.We test HYPE across four state-of-the-art generative adversarial networks (GANs) on unconditional image generation using two datasets, the popular CelebA and the newer higher-resolution FFHQ, and two sampling techniques of model outputs.By simulating HYPE's evaluation multiple times, we demonstrate consistent ranking of different models, identifying StyleGAN with truncation trick sampling (27.6% HYPE-Infinity deception rate, with roughly one quarter of images being misclassified by humans) as superior to StyleGAN without truncation (19.0%) on FFHQ.

HYPE ist ein zuverlässiger Bewertungsmaßstab für die Bewertung generativer Modelle, beginnend mit der Erzeugung menschlicher Gesichter über 4 GANs.


Machine translation has recently achieved impressive performance thanks to recent advances in deep learning and the availability of large-scale parallel corpora.There have been numerous attempts to extend these successes to low-resource language pairs, yet requiring tens of thousands of parallel sentences.In this work, we take this research direction to the extreme and investigate whether it is possible to learn to translate even without any parallel data.We propose a model that takes sentences from monolingual corpora in two different languages and maps them into the same latent space.By learning to reconstruct in both languages from this shared feature space, the model effectively learns to translate without using any labeled data.We demonstrate our model on two widely used datasets and two language pairs, reporting BLEU scores of 32.8 and 15.1 on the Multi30k and WMT English-French datasets, without using even a single parallel sentence at training time.

Wir schlagen ein neues unüberwachtes maschinelles Übersetzungsmodell vor, das ohne die Verwendung paralleler Korpora lernen kann; experimentelle Ergebnisse zeigen eine beeindruckende Leistung bei mehreren Korpora und Sprachpaaren.


We derive a new intrinsic social motivation for multi-agent reinforcement learning (MARL), in which agents are rewarded for having causal influence over another agent's actions, where causal influence is assessed using counterfactual reasoning.The reward does not depend on observing another agent's reward function, and is thus a more realistic approach to MARL than taken in previous work.We show that the causal influence reward is related to maximizing the mutual information between agents' actions.We test the approach in challenging social dilemma environments, where it consistently leads to enhanced cooperation between agents and higher collective reward.Moreover, we find that rewarding influence can lead agents to develop emergent communication protocols.Therefore, we also employ influence to train agents to use an explicit communication channel, and find that it leads to more effective communication and higher collective reward.Finally, we show that influence can be computed by equipping each agent with an internal model that predicts the actions of other agents.This allows the social influence reward to be computed without the use of a centralised controller, and as such represents a significantly more general and scalable inductive bias for MARL with independent agents.

Wir belohnen Agenten dafür, dass sie einen kausalen Einfluss auf die Handlungen anderer Agenten haben, und zeigen, dass dies zu einer besseren Zusammenarbeit und sinnvolleren emergenten Kommunikationsprotokollen führt.


This work adopts the very successful distributional perspective on reinforcement learning and adapts it to the continuous control setting.We combine this within a distributed framework for off-policy learning in order to develop what we call the Distributed Distributional Deep Deterministic Policy Gradient algorithm, D4PG.We also combine this technique with a number of additional, simple improvements such as the use of N-step returns and prioritized experience replay.Experimentally we examine the contribution of each of these individual components, and show how they interact, as well as their combined contributions.Our results show that across a wide variety of simple control tasks, difficult manipulation tasks, and a set of hard obstacle-based locomotion tasks the D4PG algorithm achieves state of the art performance.

Wir entwickeln einen Agenten, den wir Distributional Deterministic Deep Policy Gradient Algorithmus nennen, der bei einer Reihe von anspruchsvollen kontinuierlichen Steuerungsproblemen den Stand der Technik erreicht.


State-action value functions (i.e., Q-values) are ubiquitous in reinforcement learning (RL), giving rise to popular algorithms such as SARSA and Q-learning.We propose a new notion of action value defined by a Gaussian smoothed version of the expected Q-value used in SARSA.We show that such smoothed Q-values still satisfy a Bellman equation, making them naturally learnable from experience sampled from an environment.Moreover, the gradients of expected reward with respect to the mean and covariance of a parameterized Gaussian policy can be recovered from the gradient and Hessian of the smoothed Q-value function.Based on these relationships we develop new algorithms for training a Gaussian policy directly from a learned Q-value approximator.The approach is also amenable to proximal optimization techniques by augmenting the objective with a penalty on KL-divergence from a previous policy.We find that the ability to learn both a mean and covariance during training allows this approach to achieve strong results on standard continuous control benchmarks.

Wir schlagen eine neue Q-Wert-Funktion vor, die ein besseres Lernen von Gaußschen Strategien ermöglicht.


Interactive Fiction games are text-based simulations in which an agent interacts with the world purely through natural language.They are ideal environments for studying how to extend reinforcement learning agents to meet the challenges of natural language understanding, partial observability, and action generation in combinatorially-large text-based action spaces.We present KG-A2C, an agent that builds a dynamic knowledge graph while exploring and generates actions using a template-based action space.We contend that the dual uses of the knowledge graph to reason about game state and to constrain natural language generation are the keys to scalable exploration of combinatorially large natural language actions.Results across a wide variety of IF games show that KG-A2C outperforms current IF agents despite the exponential increase in action space size.

Wir stellen KG-A2C vor, einen Reinforcement Learning Agenten, der einen dynamischen Wissensgraphen aufbaut, während er natürliche Sprache unter Verwendung eines schablonenbasierten Aktionsraums erforscht und generiert - und dabei alle derzeitigen Agenten in einer Vielzahl textbasierter Spiele übertrifft.


It is well-known that neural networks are universal approximators, but that deeper networks tend in practice to be more powerful than shallower ones.We shed light on this by proving that the total number of neurons m required to approximate natural classes of multivariate polynomials of n variables grows only linearly with n for deep neural networks, but grows exponentially when merely a single hidden layer is allowed.We also provide evidence that when the number of hidden layers is increased from 1 to k, the neuron requirement grows exponentially not with n but with n^{1/k}, suggesting that the minimum number of layers required for practical expressibility grows only logarithmically with n.

Wir beweisen, dass tiefe neuronale Netze bei der Annäherung an spärliche multivariate Polynome exponentiell effizienter sind als flache Netze.


Convolutional neural networks (CNNs) in recent years have made a dramatic impact in science, technology and industry, yet the theoretical mechanism of CNN architecture design remains surprisingly vague.The CNN neurons, including its distinctive element, convolutional filters, are known to be learnable features, yet their individual role in producing the output is rather unclear.The thesis of this work is that not all neurons are equally important and some of them contain more useful information to perform a given task.Hence, we propose to quantify and rank neuron importance, and directly incorporate neuron importance in the objective function under two formulations: (1) a game theoretical approach based on Shapley value which computes the marginal contribution of each filter; and (2) a probabilistic approach based on what-we-call, the importance switch using variational inference.Using these two methods we confirm the general theory that some of the neurons are inherently more important than the others.Various experiments illustrate that learned ranks can be readily useable for structured network compression and interpretability of learned features.

Wir schlagen ein CNN-Neuronen-Ranking mit zwei verschiedenen Methoden vor und zeigen deren Konsistenz bei der Erstellung des Ergebnisses, das es erlaubt, zu interpretieren, was das Netzwerk für wichtig hält, und das Netzwerk zu komprimieren, indem die wichtigsten Knoten beibehalten werden.


This work presents a modular and hierarchical approach to learn policies for exploring 3D environments.Our approach leverages the strengths of both classical and learning-based methods, by using analytical path planners with learned mappers, and global and local policies.Use of learning provides flexibility with respect to input modalities (in mapper), leverages structural regularities of the world (in global policies), and provides robustness to errors in state estimation (in local policies).Such use of learning within each module retains its benefits, while at the same time, hierarchical decomposition and modular training allow us to sidestep the high sample complexities associated with training end-to-end policies.Our experiments in visually and physically realistic simulated 3D environments demonstrate the effectiveness of our proposed approach over past learning and geometry-based approaches.

Ein modularer und hierarchischer Ansatz zum Erlernen von Strategien zur Erkundung von 3D-Umgebungen.


Deep Learning for Computer Vision depends mainly on the source of supervision.Photo-realistic simulators can generate large-scale automatically labeled synthetic data, but introduce a domain gap negatively impacting performance.We propose a new unsupervised domain adaptation algorithm, called SPIGAN, relying on Simulator Privileged Information (PI) and Generative Adversarial Networks (GAN).We use internal data from the simulator as PI during the training of a target task network.We experimentally evaluate our approach on semantic segmentation.We train the networks on real-world Cityscapes and Vistas datasets, using only unlabeled real-world images and synthetic labeled data with z-buffer (depth) PI from the SYNTHIA dataset.Our method improves over no adaptation and state-of-the-art unsupervised domain adaptation techniques.

Eine unüberwachte Methode zur Anpassung von Simulationen an den realen Bereich für die semantische Segmentierung unter Verwendung privilegierter Informationen aus einem Simulator mit GAN-basierter Bildübersetzung.


Adversarial training is one of the main defenses against adversarial attacks.In this paper, we provide the first rigorous study on diagnosing elements of large-scale adversarial training on ImageNet, which reveals two intriguing properties. First, we study the role of normalization.Batch normalization (BN) is a crucial element for achieving state-of-the-art performance on many vision tasks, but we show it may prevent networks from obtaining strong robustness in adversarial training.One unexpected observation is that, for models trained with BN, simply removing clean images from training data largely boosts adversarial robustness, i.e., 18.3%.We relate this phenomenon to the hypothesis that clean images and adversarial images are drawn from two different domains.This two-domain hypothesis may explain the issue of BN when training with a mixture of clean and adversarial images, as estimating normalization statistics of this mixture distribution is challenging.Guided by this two-domain hypothesis, we show disentangling the mixture distribution for normalization, i.e., applying separate BNs to clean and adversarial images for statistics estimation, achieves much stronger robustness.Additionally, we find that enforcing BNs to behave consistently at training and testing can further enhance robustness.Second, we study the role of network capacity.We find our so-called "deep" networks are still shallow for the task of adversarial learning.Unlike traditional classification tasks where accuracy is only marginally improved by adding more layers to "deep" networks (e.g., ResNet-152), adversarial training exhibits a much stronger demand on deeper networks to achieve higher adversarial robustness.This robustness improvement can be observed substantially and consistently even by pushing the network capacity to an unprecedented scale, i.e., ResNet-638.

Die erste Strengheitsdiagnose von groß angelegtem adversarialem Training auf ImageNet


The gradient of a deep neural network (DNN) w.r.t. the input provides information that can be used to explain the output prediction in terms of the input features and has been widely studied to assist in interpreting DNNs. In a linear model (i.e., $g(x)=wx+b$), the gradient corresponds solely to the weights $w$.Such a model can reasonably locally linearly approximate a smooth nonlinear DNN, and hence the weights of this local model are the gradient.The other part, however, of a local linear model, i.e., the bias $b$, is usually overlooked in attribution methods since it is not part of the gradient.In this paper, we observe that since the bias in a DNN also has a non-negligible contribution to the correctness of predictions, it can also play a significant role in understanding DNN behaviors.In particular, we study how to attribute a DNN's bias to its input features.We propose a backpropagation-type algorithm ``bias back-propagation (BBp)'' that starts at the output layer and iteratively attributes the bias of each layer to its input nodes as well as combining the resulting bias term of the previous layer.This process stops at the input layer, where summing up the attributions over all the input features exactly recovers $b$.Together with the backpropagation of the gradient generating $w$, we can fully recover the locally linear model $g(x)=wx+b$.Hence, the attribution of the DNN outputs to its inputs is decomposed into two parts, the gradient $w$ and the bias attribution, providing separate and complementary explanations.We study several possible attribution methods applied to the bias of each layer in BBp.In experiments, we show that BBp can generate complementary and highly interpretable explanations of DNNs in addition to gradient-based attributions.

Zuordnung der Bias-Terme tiefer neuronaler Netze zu Eingangsmerkmalen durch einen Backpropagation-Algorithmus; Generierung ergänzender und hochinterpretierbarer Erklärungen von DNNs zusätzlich zu gradientenbasierten Zuordnungen.


This paper presents a method to autonomously find periodicities in a signal.It is based on the same idea of using Fourier Transform and autocorrelation function presented in Vlachos et al. 2005.While showing interesting results this method does not perform well on noisy signals or signals with multiple periodicities.Thus, our method adds several new extra steps (hints clustering, filtering and detrending) to fix these issues.Experimental results show that the proposed method outperforms the state of the art algorithms.

In diesem Beitrag wird eine Methode vorgestellt, mit der sich mehrere Periodizitäten in einem Signal unter Verwendung von FFT und ACF selbständig finden lassen, wobei drei neue Schritte (Clustering/Filterung/Detrending) hinzugefügt werden


We present an adversarial exploration strategy, a simple yet effective imitation learning scheme that incentivizes exploration of an environment without any extrinsic reward or human demonstration.Our framework consists of a deep reinforcement learning (DRL) agent and an inverse dynamics model contesting with each other.The former collects training samples for the latter, and its objective is to maximize the error of the latter.The latter is trained with samples collected by the former, and generates rewards for the former when it fails to predict the actual action taken by the former.In such a competitive setting, the DRL agent learns to generate samples that the inverse dynamics model fails to predict correctly, and the inverse dynamics model learns to adapt to the challenging samples.We further propose a reward structure that ensures the DRL agent collects only moderately hard samples and not overly hard ones that prevent the inverse model from imitating effectively.We evaluate the effectiveness of our method on several OpenAI gym robotic arm and hand manipulation tasks against a number of baseline models.Experimental results show that our method is comparable to that directly trained with expert demonstrations, and superior to the other baselines even without any human priors.

Ein einfaches, aber effektives Imitation Learning Schema, das Anreize für die Erkundung einer Umgebung ohne äußere Belohnung oder menschliche Vorführung schafft.


This paper proposes a dual variational autoencoder (DualVAE), a framework for generating images corresponding to multiclass labels.Recent research on conditional generative models, such as the Conditional VAE, exhibit image transfer by changing labels.However, when the dimension of multiclass labels is large, these models cannot change images corresponding to labels, because learning multiple distributions of the corresponding class is necessary to transfer an image.This leads to the lack of training data.Therefore, instead of conditioning with labels, we condition with latent vectors that include label information.DualVAE divides one distribution of the latent space by linear decision boundaries using labels.Consequently, DualVAE can easily transfer an image by moving a latent vector toward a decision boundary and is robust to the missing values of multiclass labels.To evaluate our proposed method, we introduce a conditional inception score (CIS) for measuring how much an image changes to the target class.We evaluate the images transferred by DualVAE using the CIS in CelebA datasets and demonstrate state-of-the-art performance in a multiclass setting.

Ein neues Framework, das den dualen Raum für die Erzeugung von Bildern nutzt, die bei einer großen Anzahl von Klassen mit mehreren Etiketten übereinstimmen.


One of the most notable contributions of deep learning is the application of convolutional neural networks (ConvNets) to structured signal classification, and in particular image classification.Beyond their impressive performances in supervised learning, the structure of such networks inspired the development of deep filter banks referred to as scattering transforms.These transforms apply a cascade of wavelet transforms and complex modulus operators to extract features that are invariant to group operations and stable to deformations.Furthermore, ConvNets inspired recent advances in geometric deep learning, which aim to generalize these networks to graph data by applying notions from graph signal processing to learn deep graph filter cascades.We further advance these lines of research by proposing a geometric scattering transform using graph wavelets defined in terms of random walks on the graph.We demonstrate the utility of features extracted with this designed deep filter bank in graph classification of biochemistry and social network data (incl. state of the art results in the latter case), and in data exploration, where they enable inference of EC exchange preferences in enzyme evolution.

Wir stellen ein neues Feed-Forward-Graph-ConvNet vor, das auf der Verallgemeinerung der Wavelet-Streuungstransformation von Mallat basiert, und demonstrieren seine Nützlichkeit bei Graphenklassifizierungs- und Datenexplorationsaufgaben.


OCR is inevitably linked to NLP since its final output is in text.Advances in document intelligence are driving the need for a unified technology that integrates OCR with various NLP tasks, especially semantic parsing.Since OCR and semantic parsing have been studied as separate tasks so far, the datasets for each task on their own are rich, while those for the integrated post-OCR parsing tasks are relatively insufficient.In this study, we publish a consolidated dataset for receipt parsing as the first step towards post-OCR parsing tasks.The dataset consists of thousands of Indonesian receipts, which contains images and box/text annotations for OCR, and multi-level semantic labels for parsing.The proposed dataset can be used to address various OCR and parsing tasks.

Wir stellen einen groß angelegten Belegdatensatz für Post-OCR-Parsing-Aufgaben vor.


The growth in the complexity of Convolutional Neural Networks (CNNs) is increasing interest in partitioning a network across multiple accelerators during training and pipelining the backpropagation computations over the accelerators.Existing approaches avoid or limit the use of stale weights through techniques such as micro-batching or weight stashing.These techniques either underutilize of accelerators or increase memory footprint.We explore the impact of stale weights on the statistical efficiency and performance in a pipelined backpropagation scheme that maximizes accelerator utilization and keeps memory overhead modest.We use 4 CNNs (LeNet-5, AlexNet, VGG and ResNet) and show that when pipelining is limited to early layers in a network, training with stale weights converges and results in models with comparable inference accuracies to those resulting from non-pipelined training on MNIST and CIFAR-10 datasets; a drop in accuracy of 0.4%, 4%, 0.83% and 1.45% for the 4 networks, respectively.However, when pipelining is deeper in the network, inference accuracies drop significantly.We propose combining pipelined and non-pipelined training in a hybrid scheme to address this drop.We demonstrate the implementation and performance of our pipelined backpropagation in PyTorch on 2 GPUs using ResNet, achieving speedups of up to 1.8X over a 1-GPU baseline, with a small drop in inference accuracy.

Beschleunigung des CNN-Trainings auf einer Pipeline von Beschleunigern mit veralteten Gewichten.