We introduce Quantum Graph Neural Networks (QGNN), a new class of quantum neural network ansatze which are tailored to represent quantum processes which have a graph structure, and are particularly suitable to be executed on distributed quantum systems over a quantum network.Along with this general class of ansatze, we introduce further specialized architectures, namely, Quantum Graph Recurrent Neural Networks (QGRNN) and Quantum Graph Convolutional Neural Networks (QGCNN).We provide four example applications of QGNN's: learning Hamiltonian dynamics of quantum systems, learning how to create multipartite entanglement in a quantum network, unsupervised learning for spectral clustering, and supervised learning for graph isomorphism classification.

Einführung einer neuen Klasse von neuronalen Quantennetzen zum Erlernen graphbasierter Darstellungen auf Quantencomputern.


Data breaches involve information being accessed by unauthorized parties.Our research concerns user perception of data breaches, especially issues relating to accountability.A preliminary study indicated many people had weak understanding of the issues, and felt they themselves were somehow responsible.We speculated that this impression might stem from organizational communication strategies.We therefore compared texts from organizations with external sources, such as the news media.This suggested that organizations use well-known crisis communication methods to reduce their reputational damage, and that these strategies align with repositioning of the narrative elements involved in the story.We then conducted a quantitative study, asking participants to rate either organizational texts or news texts about breaches.The findings of this study were in line with our document analysis, and suggest that organizational communication affects the users' perception of victimization, attitudes in data protection, and accountability.Our study suggests some software design and legal implications supporting users to protect themselves and develop better mental models of security breaches.

In dieser Arbeit haben wir den Einfluss von Kommunikationsstrategien auf die mentalen Modelle von Nutzern einer Datenschutzverletzung untersucht.


Goal recognition is the problem of inferring the correct goal towards which an agent executes a plan, given a set of goal hypotheses, a domain model, and a (possibly noisy) sample of the plan being executed.   This is a key problem in both cooperative and competitive agent interactions and recent approaches have produced fast and accurate goal recognition algorithms.   In this paper, we leverage advances in operator-counting heuristics computed using linear programs over constraints derived from classical planning problems to solve goal recognition problems.   Our approach uses additional operator-counting constraints derived from the observations to efficiently infer the correct goal, and serves as basis for a number of further methods with additional constraints.

Ein Zielerkennungsansatz, der auf einer Heuristik der Operatorenzählung basiert, um das Störungen im Datensatz zu berücksichtigen.


It can be challenging to train multi-task neural networks that outperform or even match their single-task counterparts.To help address this, we propose using knowledge distillation where single-task models teach a multi-task model.We enhance this training with teacher annealing, a novel method that gradually transitions the model from distillation to supervised learning, helping the multi-task model surpass its single-task teachers.We evaluate our approach by multi-task fine-tuning BERT on the GLUE benchmark.Our method consistently improves over standard single-task and multi-task training.

die Destillation von Einzelaufgabenmodellen in ein Multiaufgabenmodell verbessert die Leistung beim Verstehen natürlicher Sprache


The detection of out of distribution samples for image classification has been widely researched.Safety critical applications, such as autonomous driving, would benefit from the ability to localise the unusual objects causing the image to be out of distribution.This paper adapts state-of-the-art methods for detecting out of distribution images for image classification to the new task of detecting out of distribution pixels, which can localise the unusual objects.It further experimentally compares the adapted methods on two new datasets derived from existing semantic segmentation datasets using PSPNet and DeeplabV3+ architectures, as well as proposing a new metric for the task.The evaluation shows that the performance ranking of the compared methods does not transfer to the new task and every method performs significantly worse than their image-level counterparts.

Evaluierung von Methoden zur Erkennung von Verteilungsfehlern auf Pixelebene an zwei neuen realen Datensätzen mit PSPNet und DeeplabV3+.


We present a domain adaptation method for transferring neural representations from label-rich source domains to unlabeled target domains.Recent adversarial methods proposed for this task learn to align features across domains by ``fooling'' a special domain classifier network.However, a drawback of this approach is that the domain classifier simply labels the generated features as in-domain or not, without considering the boundaries between classes.This means that ambiguous target features can be generated near class boundaries, reducing target classification accuracy.We propose a novel approach, Adversarial Dropout Regularization (ADR), which encourages the generator to output more discriminative features for the target domain.Our key idea is to replace the traditional domain critic with a critic that detects non-discriminative features by using dropout on the classifier network.The generator then learns to avoid these areas of the feature space and thus creates better features.We apply our ADR approach to the problem of unsupervised domain adaptation for image classification and semantic segmentation tasks, and demonstrate significant improvements over the state of the art.

Wir stellen eine neue adversarische Methode zur Anpassung neuronaler Repräsentationen vor, die auf einer Kritik basiert, die nicht-diskriminierende Merkmale erkennt.


The use of deep learning for a wide range of data problems has increased the need for understanding and diagnosing these models, and deep learning interpretation techniques have become an essential tool for data analysts.Although numerous model interpretation methods have been proposed in recent years, most of these procedures are based on heuristics with little or no theoretical guarantees.In this work, we propose a statistical framework for saliency estimation for black box computer vision models.We build a model-agnostic estimation procedure that is statistically consistent and passes the saliency checks of Adebayo et al. (2018).Our method requires solving a linear program, whose solution can be efficiently computed in polynomial time.Through our theoretical analysis, we establish an upper bound on the number of model evaluations needed to recover the region of importance with high probability, and build a new perturbation scheme for estimation of local gradients that is shown to be more efficient than the commonly used random perturbation schemes.Validity of the new method is demonstrated through sensitivity analysis.

Wir schlagen einen statistischen Rahmen und ein theoretisch konsistentes Verfahren zur Schätzung der Auffälligkeit vor.


We explore the idea of compositional set embeddings that can be used to infer notjust a single class, but the set of classes associated with the input data (e.g., image,video, audio signal).This can be useful, for example, in multi-object detection inimages, or multi-speaker diarization (one-shot learning) in audio.In particular, wedevise and implement two novel models consisting of (1) an embedding functionf trained jointly with a “composite” function g that computes set union opera-tions between the classes encoded in two embedding vectors; and (2) embeddingf trained jointly with a “query” function h that computes whether the classes en-coded in one embedding subsume the classes encoded in another embedding.Incontrast to prior work, these models must both perceive the classes associatedwith the input examples, and also encode the relationships between different classlabel sets.In experiments conducted on simulated data, OmniGlot, and COCOdatasets, the proposed composite embedding models outperform baselines basedon traditional embedding approaches.

Wir haben untersucht, wie eine neuartige Methode der kompositorischen Mengeneinbettung nicht nur eine einzelne Klasse, sondern eine ganze Menge von Klassen, die mit den Eingabedaten verbunden sind, wahrnehmen und darstellen kann.


In this work, we propose a goal-driven collaborative task that contains language, vision, and action in a virtual environment as its core components.Specifically, we develop a Collaborative image-Drawing game between two agents, called CoDraw.Our game is grounded in a virtual world that contains movable clip art objects.The game involves two players: a Teller and a Drawer.The Teller sees an abstract scene containing multiple clip art pieces in a semantically meaningful configuration, while the Drawer tries to reconstruct the scene on an empty canvas using available clip art pieces.The two players communicate via two-way communication using natural language.We collect the CoDraw dataset of ~10K dialogs consisting of ~138K messages exchanged between human agents.We define protocols and metrics to evaluate the effectiveness of learned agents on this testbed, highlighting the need for a novel "crosstalk" condition which pairs agents trained independently on disjoint subsets of the training data for evaluation.We present models for our task, including simple but effective baselines and neural network approaches trained using a combination of imitation learning and goal-driven training.All models are benchmarked using both fully automated evaluation and by playing the game with live human agents.

Wir stellen einen Datensatz, Modelle und Trainings- und Evaluationsprotokolle für eine kollaborative Zeichenaufgabe vor, die es ermöglicht, zielgerichtete und wahrnehmungs- und handlungsbasierte Spracherzeugung und Sprachverständnis zu untersuchen.


Presence of bias and confounding effects is inarguably one of the most critical challenges in machine learning applications that has alluded to pivotal debates in the recent years.Such challenges range from spurious associations of confounding variables in medical studies to the bias of race in gender or face recognition systems.One solution is to enhance datasets and organize them such that they do not reflect biases, which is a cumbersome and intensive task.The alternative is to make use of available data and build models considering these biases.Traditional statistical methods apply straightforward techniques such as residualization or stratification to precomputed features to account for confounding variables.However, these techniques are not in general applicable to end-to-end deep learning methods.In this paper, we propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s).This is enabled by incorporating a new adversarial loss function that encourages a vanished correlation between the bias and learned features.We apply our method to a synthetic, a medical diagnosis, and a gender classification (Gender Shades) dataset.Our results show that the learned features by our method not only result in superior prediction performance but also are uncorrelated with the bias or confounder variables.The code is available at http://blinded_for_review/.

Wir schlagen eine Methode vor, die auf der adversen Trainingsstrategie basiert, um diskriminierende Merkmale zu erlernen, die unvoreingenommen und invariant gegenüber den Störfaktoren sind, indem wir eine Verlustfunktion einbauen, die eine verschwindende Korrelation zwischen der Verzerrung und den erlernten Merkmalen fördert.


Existing neural question answering (QA) models are required to reason over and draw complicated inferences from a long context for most large-scale QA datasets.However, if we view QA as a combined retrieval and reasoning task, we can assume the existence of a minimal context which is necessary and sufficient to answer a given question.Recent work has shown that a sentence selector module that selects a shorter context and feeds it to the downstream QA model achieves performance comparable to a QA model trained on full context, while also being more interpretable.Recent work has also shown that most state-of-the-art QA models break when adversarially generated sentences are appended to the context.While humans are immune to such distractor sentences, QA models get easily misled into selecting answers from these sentences.We hypothesize that the sentence selector module can filter out extraneous context, thereby allowing the downstream QA model to focus and reason over the parts of the context that are relevant to the question.In this paper, we show that the sentence selector itself is susceptible to adversarial inputs.However, we demonstrate that a pipeline consisting of a sentence selector module followed by the QA model can be made more robust to adversarial attacks in comparison to a QA model trained on full context.Thus, we provide evidence towards a modular approach for question answering that is more robust and interpretable.

Ein modularer Ansatz, der aus einem Satzselektionsmodul und einem anschließenden QS-Modell besteht, kann im Vergleich zu einem QS-Modell, das auf der Grundlage des gesamten Kontexts trainiert wurde, robuster gegenüber Angriffen durch Angreifer gemacht werden.


Multi-relational graph embedding which aims at achieving effective representations with reduced low-dimensional parameters, has been widely used in knowledge base completion.Although knowledge base data usually contains tree-like or cyclic structure, none of existing approaches can embed these data into a compatible space that in line with the structure.To overcome this problem, a novel framework, called Riemannian TransE, is proposed in this paper to embed the entities in a Riemannian manifold.Riemannian TransE models each relation as a move to a point and defines specific novel distance dissimilarity for each relation, so that all the relations are naturally embedded in correspondence to the structure of data.Experiments on several knowledge base completion tasks have shown that, based on an appropriate choice of manifold, Riemannian TransE achieves good performance even with a significantly reduced parameters.

Multirelationale Grapheneinbettung mit Riemannschen Mannigfaltigkeiten und TransE-ähnlicher Verlustfunktion.


Catastrophic forgetting in neural networks is one of the most well-known problems in continual learning.Previous attempts on addressing the problem focus on preventing important weights from changing.Such methods often require task boundaries to learn effectively and do not support backward transfer learning.In this paper, we propose a meta-learning algorithm which learns to reconstruct the gradients of old tasks w.r.t. the current parameters and combines these reconstructed gradients with the current gradient to enable continual learning and backward transfer learning from the current task to previous tasks.Experiments on standard continual learning benchmarks show that our algorithm can effectively prevent catastrophic forgetting and supports backward transfer learning.

Wir schlagen einen Meta-Lernalgorithmus für kontinuierliches Lernen vor, der das Problem des katastrophalen Vergessens wirksam verhindern und das Rückwärts-Transferlernen unterstützen kann.


We give a formal procedure for computing preimages of convolutional  network outputs using the dual basis defined from the set of  hyperplanes associated with the layers of the network.We point out  the special symmetry associated with arrangements of hyperplanes of  convolutional networks that take the form of regular  multidimensional polyhedral cones.We discuss  the efficiency of of  large number of layers of nested cones that result from incremental  small size convolutions in order to give a good compromise between  efficient contraction of data to low dimensions and shaping of  preimage manifolds.We demonstrate how a specific network flattens a  non linear input manifold to an affine output manifold and discuss  it's relevance to understanding classification properties of deep  networks.

Analyse von tiefen Convolutional Networks in Bezug auf die zugehörige Anordnung von Hyperebenen


Meta learning has been making impressive progress for fast model adaptation.However, limited work has been done on learning fast uncertainty adaption for Bayesian modeling.In this paper, we propose to achieve the goal by placing meta learning on the space of probability measures, inducing the concept of meta sampling for fast uncertainty adaption.Specifically, we propose a Bayesian meta sampling framework consisting of two main components: a meta sampler and a sample adapter.The meta sampler is constructed by adopting a neural-inverse-autoregressive-flow (NIAF) structure, a variant of the recently proposed neural autoregressive flows, to efficiently generate meta samples to be adapted.The sample adapter moves meta samples to task-specific samples, based on a newly proposed and general Bayesian sampling technique, called optimal-transport Bayesian sampling.The combination of the two components allows a simple learning procedure for themeta sampler to be developed, which can be efficiently optimized via standard back-propagation.Extensive experimental results demonstrate the efficiency and effectiveness of the proposed framework, obtaining better sample quality and fasteruncertainty adaption compared to related methods.

Wir haben eine Bayes'sche Meta-Sampling-Methode zur Anpassung der Modellunsicherheit beim Meta-Lernen vorgeschlagen


We propose a context-adaptive entropy model for use in end-to-end optimized image compression.Our model exploits two types of contexts, bit-consuming contexts and bit-free contexts, distinguished based upon whether additional bitallocation is required.Based on these contexts, we allow the model to more accurately estimate the distribution of each latent representation with a more generalized form of the approximation models, which accordingly leads to anenhanced compression performance.Based on the experimental results, the proposed method outperforms the traditional image codecs, such as BPG and JPEG2000, as well as other previous artificial-neural-network (ANN) based approaches, in terms of the peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) index.The test code is publicly available at https://github.com/JooyoungLeeETRI/CA_Entropy_Model.

Kontextadaptives Entropiemodell zur Verwendung in der durchgängig optimierten Bildkompression, das die Kompressionsleistung erheblich verbessert


Deep networks often perform well on the data distribution on which they are trained, yet give incorrect (and often very confident) answers when evaluated on points from off of the training distribution.This is exemplified by the adversarial examples phenomenon but can also be seen in terms of model generalization and domain shift.  Ideally, a model would assign lower confidence to points unlike those from the training distribution.  We propose a regularizer which addresses this issue by training with interpolated hidden states and encouraging the classifier to be less confident at these points.  Because the hidden states are learned, this has an important effect of encouraging the hidden states for a class to be concentrated in such a way so that interpolations within the same class or between two different classes do not intersect with the real data points from other classes.  This has a major advantage in that it avoids the underfitting which can result from interpolating in the input space.  We prove that the exact condition for this problem of underfitting to be avoided by Manifold Mixup is that the dimensionality of the hidden states exceeds the number of classes, which is often the case in practice.  Additionally, this concentration can be seen as making the features in earlier layers more discriminative.  We show that despite requiring no significant additional computation, Manifold Mixup achieves large improvements over strong baselines in supervised learning, robustness to single-step adversarial attacks, semi-supervised learning, and Negative Log-Likelihood on held out samples.

Eine Methode zum Erlernen besserer Repräsentationen, die als Regularisierer fungiert und trotz ihres geringen zusätzlichen Rechenaufwands Verbesserungen im Vergleich zu starken Baselines bei überwachten und halb-überwachten Lernaufgaben erzielt.


Sometimes SRS (Stereotactic Radio Surgery) requires using sphere packing on a Region of Interest (ROI) such as cancer to determine a treatment plan.  We have developed a sphere packing algorithm which packs non-intersecting spheres inside the ROI.  The region of interest in our case are those voxels which are identified as cancer tissues.  In this paper, we analyze the rotational invariant properties of  our sphere-packing algorithm which is based on distance transformations.Epsilon-Rotation invariant means the ability to arbitrary rotate the 3D ROI while keeping the volume properties remaining (almost) same within some limit of epsilon.The applied rotations produce spherical packing which remains highly correlated as we analyze the geometrically properties of sphere packing before and after the rotation of the volume data for the ROI.Our novel sphere packing algorithm has high degree of rotation invariance within the range of +/- epsilon.Our method used a shape descriptor derived from the values of the disjoint set of spheres form the distance-based sphere packing algorithm to extract the invariant descriptor from the ROI.We demonstrated by implementing these ideas using Slicer3D platform available for our research.  The data is based on sing MRI Stereotactic images.We presented several performance results on different benchmarks data of over 30 patients in Slicer3D platform.

Packen von Interessensgebieten (ROI) wie z. B. Krebsregionen, die in 3D-Volumendaten identifiziert wurden, Packen von Bereichen innerhalb des ROI, Drehen des ROI, Messen des Unterschieds in der gepackten Bereichen vor und nach der Drehung.


We show implicit filter level sparsity manifests in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained using adaptive gradient descent techniques with L2 regularization or weight decay.Through an extensive empirical study (Anonymous, 2019) we hypothesize the mechanism be hind the sparsification process.We find that the interplay  of  various  phenomena  influences  the strength of L2 and weight decay regularizers, leading the supposedly non sparsity inducing regularizers to induce filter sparsity.  In this workshop article we summarize some of our key findings and experiments, and present additional results on modern network architectures such as ResNet-50.

Seltenheit auf Filterebene entsteht implizit in CNNs, die mit adaptiven Gradientenabstiegsansätzen trainiert werden, aufgrund verschiedener Phänomene, und das Ausmaß der Seltenheit kann versehentlich durch verschiedene, scheinbar nicht zusammenhängende Hyperparameter beeinflusst werden.


Humans can learn a variety of concepts and skills incrementally over the course of their lives while exhibiting an array of desirable properties, such as non-forgetting, concept rehearsal, forward transfer and backward transfer of knowledge, few-shot learning, and selective forgetting.Previous approaches to lifelong machine learning can only demonstrate subsets of these properties, often by combining multiple complex mechanisms.  In this Perspective, we propose a powerful unified framework that can demonstrate all of the properties by utilizing a small number of weight consolidation parameters in deep neural networks.In addition, we are able to draw many parallels between the behaviours and mechanisms of our proposed framework and those surrounding human learning, such as memory loss or sleep deprivation.This Perspective serves as a conduit for two-way inspiration to further understand lifelong learning in machines and humans.

Indem wir Parallelen zum menschlichen Lernen ziehen, schlagen wir einen einheitlichen Rahmen vor, um viele lebenslange Lernfähigkeiten in neuronalen Netzen zu zeigen, indem wir eine kleine Anzahl von Gewichtskonsolidierungsparametern verwenden.


Limited angle CT reconstruction is an under-determined linear inverse problem that requires appropriate regularization techniques to be solved.In this work we study how pre-trained generative adversarial networks (GANs) can be used to clean noisy, highly artifact laden reconstructions from conventional techniques, by effectively projecting onto the inferred image manifold.In particular, we use a robust version of the popularly used GAN prior for inverse problems, based on a recent technique called corruption mimicking, that significantly improves the reconstruction quality.The proposed approach operates in the image space directly, as a result of which it does not need to be trained or require access to the measurement model, is scanner agnostic, and can work over a wide range of sensing scenarios.

Wir zeigen, dass robuste GAN priors besser funktionieren als GAN priors für die CT-Rekonstruktion mit begrenztem Winkel, die ein stark unterbestimmtes inverses Problem darstellt.


We propose an effective multitask learning setup for reducing distant supervision noise by leveraging sentence-level supervision.We show how sentence-level supervision can be used to improve the encoding of individual sentences, and to learn which input sentences are more likely to express the relationship between a pair of entities.We also introduce a novel neural architecture for collecting signals from multiple input sentences, which combines the benefits of attention and maxpooling.The proposed method increases AUC by 10% (from 0.261 to 0.284), and outperforms recently published results on the FB-NYT dataset.

Eine neue Form der Aufmerksamkeit, die sich gut für die Überwachung aus der Ferne eignet, und ein Multitasking-Ansatz, um Anmerkungen auf Satzebene hinzuzufügen.


The attention layer in a neural network model provides insights into the model’s reasoning behind its prediction, which are usually criticized for being opaque.Recently, seemingly contradictory viewpoints have emerged about the interpretability of attention weights (Jain & Wallace, 2019; Vig & Belinkov, 2019).Amid such confusion arises the need to understand attention mechanism more systematically.In this work, we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not).Through a series of experiments on diverse NLP tasks, we validate our observations and reinforce our claim of interpretability of attention through manual evaluation.

Analyse des Aufmerksamkeitsmechanismus bei verschiedenen NLP-Aufgaben.


Generative models forsource code are an interesting structured prediction problem, requiring to reason about both hard syntactic and semantic constraints as well as about natural, likely programs.We present a novel model for this problem that uses a graph to represent the intermediate state of the generated output.Our model generates code by interleaving grammar-driven expansion steps with graph augmentation and neural message passing steps.An experimental evaluation shows that our new model can generate semantically meaningful expressions, outperforming a range of strong baselines.

Die Darstellung von Programmen als Graphen inklusive Semantik hilft bei der Programmerstellung


The inference of models, prediction of future symbols, and entropy rate estimation of discrete-time, discrete-event processes is well-worn ground.However, many time series are better conceptualized as continuous-time, discrete-event processes.Here, we provide new methods for inferring models, predicting future symbols, and estimating the entropy rate of continuous-time, discrete-event processes.The methods rely on an extension of Bayesian structural inference that takes advantage of neural network’s universal approximation power.Based on experiments with simple synthetic data, these new methods seem to be competitive with state-of- the-art methods for prediction and entropy rate estimation as long as the correct model is inferred.

Eine neue Methode zur Ableitung eines Modells, zur Schätzung der Entropierate und zur Vorhersage zeitkontinuierlicher, ereignisdiskreter Prozesse.


Recent studies show that convolutional neural networks (CNNs) are vulnerable under various settings, including adversarial examples, backdoor attacks, and distribution shifting.  Motivated by the findings that human visual system pays more attention to global structure (e.g., shape) for recognition while CNNs are biased towards local texture features in images, we propose a unified framework EdgeGANRob based on robust edge features to improve the robustness of CNNs in general, which first explicitly extracts shape/structure features from a given image and then reconstructs a new image by refilling the texture information with a trained generative adversarial network (GAN).In addition, to reduce the sensitivity of edge detection algorithm to adversarial perturbation, we propose a robust edge detection approach Robust Canny based on the vanilla Canny algorithm.To gain more insights, we also compare EdgeGANRob with its simplified backbone procedure EdgeNetRob, which performs learning tasks directly on the extracted robust edge features.We find that EdgeNetRob can help boost model robustness significantly but at the cost of the clean model accuracy.EdgeGANRob, on the other hand, is able to improve clean model accuracy compared with EdgeNetRob and without losing the robustness benefits introduced by EdgeNetRob.Extensive experiments show that EdgeGANRob is resilient in different learning tasks under diverse settings.

Ein vereinheitlichtes Modell zur Verbesserung der Robustheit des Modells bei mehreren Aufgaben


Learning rich representation from data is an important task for deep generative models such as variational auto-encoder (VAE).However, by extracting high-level abstractions in the bottom-up inference process, the goal of preserving all factors of variations for top-down generation is compromised.Motivated by the concept of “starting small”, we present a strategy to progressively learn independent hierarchical representations from high- to low-levels of abstractions.The model starts with learning the most abstract representation, and then progressively grow the network architecture to introduce new  representations at different levels of abstraction.We quantitatively demonstrate the ability of the presented model to improve disentanglement in comparison to existing works on two benchmark datasets using three disentanglement metrics, including a new metric we proposed to complement the previously-presented metric of mutual information gap.We further present both qualitative and quantitative evidence on how the progression of learning improves disentangling of hierarchical representations.By drawing on the respective advantage of hierarchical representation learning and progressive learning, this is to our knowledge the first attempt to improve disentanglement by progressively growing the capacity of VAE to learn hierarchical representations.

Wir haben eine progressive Lernmethode vorgeschlagen, um das Lernen und die Entflechtung latenter Repräsentationen auf verschiedenen Abstraktionsebenen zu verbessern.


Deep latent variable models are powerful tools for representation learning.In this paper, we adopt the deep information bottleneck model, identify its shortcomings and propose a model that circumvents them.To this end, we apply a copula transformation which, by restoring the invariance properties of the information bottleneck method, leads to disentanglement of the features in the latent space.Building on that, we show how this transformation translates to sparsity of the latent space in the new model.  We evaluate our method on artificial and real data.

Wir wenden die Copula-Transformation auf den Deep Information Bottleneck an, was zu wiederhergestellten Invarianzeigenschaften und einem entwirrten latenten Raum mit überlegenen Vorhersagefähigkeiten führt.


State-of-the-art machine learning methods exhibit limited compositional generalization.At the same time, there is a lack of realistic benchmarks that comprehensively measure this ability, which makes it challenging to find and evaluate improvements.We introduce a novel method to systematically construct such benchmarks by maximizing compound divergence while guaranteeing a small atom divergence between train and test sets, and we quantitatively compare this method to other approaches for creating compositional generalization benchmarks.We present a large and realistic natural language question answering dataset that is constructed according to this method, and we use it to analyze the compositional generalization ability of three machine learning architectures.We find that they fail to generalize compositionally and that there is a surprisingly strong negative correlation between compound divergence and accuracy.We also demonstrate how our method can be used to create new compositionality benchmarks on top of the existing SCAN dataset, which confirms these findings.

Benchmark und Methode zur Messung der kompositorischen Generalisierung durch Maximierung der Divergenz der Verbindungshäufigkeit bei geringer Divergenz der Atomhäufigkeit.


To understand how object vision develops in infancy and childhood, it will be necessary to develop testable computational models.Deep neural networks (DNNs) have proven valuable as models of adult vision, but it is not yet clear if they have any value as models of development.As a first model, we measured learning in a DNN designed to mimic the architecture and representational geometry of the visual system (CORnet).We quantified the development of explicit object representations at each level of this network through training by freezing the convolutional layers and training an additional linear decoding layer.We evaluate decoding accuracy on the whole ImageNet validation set, and also for individual visual classes.CORnet, however, uses supervised training and because infants have only extremely impoverished access to labels they must instead learn in an unsupervised manner.We therefore also measured learning in a state-of-the-art unsupervised network (DeepCluster).CORnet and DeepCluster differ in both supervision and in the convolutional networks at their heart, thus to isolate the effect of supervision, we ran a control experiment in which we trained the convolutional network from DeepCluster (an AlexNet variant) in a supervised manner.We make predictions on how learning should develop across brain regions in infants.In all three networks, we also tested for a relationship in the order in which infants and machines acquire visual classes, and found only evidence for a counter-intuitive relationship.We discuss the potential reasons for this.

Unüberwachte Netze lernen von unten nach oben; Maschinen und Kleinkinder lernen visuelle Klassen in unterschiedlicher Reihenfolge


We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training.The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training.A simple argument then suggests that gradient descent may happen mostly in this subspace.We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.

Für Klassifizierungsprobleme mit k Klassen zeigen wir, dass der Gradient dazu neigt, in einem winzigen, sich langsam entwickelnden Unterraum zu leben, der von den Eigenvektoren aufgespannt wird, die den k-größten Eigenwerten der Hessian entsprechen.


Answering questions that require multi-hop reasoning at web-scale necessitates retrieving multiple evidence documents, one of which often has little lexical or semantic relationship to the question.This paper introduces a new graph-based recurrent retrieval approach that learns to retrieve reasoning paths over the Wikipedia graph to answer multi-hop open-domain questions.Our retriever model trains a recurrent neural network that learns to sequentially retrieve evidence paragraphs in the reasoning path by conditioning on the previously retrieved documents. Our reader model ranks the reasoning paths and extracts the answer span included in the best reasoning path.Experimental results show state-of-the-art results in three open-domain QA datasets, showcasing the effectiveness and robustness of our method.Notably, our method achieves significant improvement in HotpotQA, outperforming the previous best model by more than 14 points.

Ein graphenbasierter rekurrenter Retriever, der lernt, Argumentationspfade über den Wikipedia-Graphen abzurufen, übertrifft den aktuellen Stand der Technik bei HotpotQA um mehr als 14 Punkte.


Stereo matching is one of the important basic tasks in the computer vision field.In recent years, stereo matching algorithms based on deep learning have achieved excellent performance and become the mainstream research direction.Existing algorithms generally use deep convolutional neural networks (DCNNs) to extract more abstract semantic information, but we believe that the detailed information of the spatial structure is more important for stereo matching tasks.Based on this point of view, this paper proposes a shallow feature extraction network with a large receptive field.The network consists of three parts: a primary feature extraction module, an atrous spatial pyramid pooling (ASPP) module and a feature fusion module.The primary feature extraction network contains only three convolution layers.This network utilizes the basic feature extraction ability of the shallow network to extract and retain the detailed information of the spatial structure.In this paper, the dilated convolution and atrous spatial pyramid pooling (ASPP) module is introduced to increase the size of receptive field.In addition, a feature fusion module is designed, which integrates the feature maps with multiscale receptive fields and mutually complements the feature information of different scales.We replaced the feature extraction part of the existing stereo matching algorithms with our shallow feature extraction network, and achieved state-of-the-art performance on the KITTI 2015 dataset.Compared with the reference network, the number of parameters is reduced by 42%, and the matching accuracy is improved by 1.9%.

Wir haben ein flaches Feature-Extraktionsnetzwerk mit einem großen rezeptiven Feld für Stereo-Matching-Aufgaben eingeführt, das eine einfache Struktur verwendet, um eine bessere Leistung zu erzielen.


We propose a new output layer for deep neural networks that permits the use of logged contextual bandit feedback for training.Such contextual bandit feedback can be available in huge quantities (e.g., logs of search engines, recommender systems) at little cost, opening up a path for training deep networks on orders of magnitude more data.To this effect, we propose a Counterfactual Risk Minimization (CRM) approach for training deep networks using an equivariant empirical risk estimator with variance regularization, BanditNet, and show how the resulting objective can be decomposed in a way that allows Stochastic Gradient Descent (SGD) training.We empirically demonstrate the effectiveness of the method by showing how deep networks -- ResNets in particular -- can be trained for object recognition without conventionally labeled images.

Diese Arbeit schlägt eine neue Ausgabeschicht für tiefe Netzwerke vor, die die Verwendung von protokolliertem kontextuellem Bandit-Feedback für das Training ermöglicht.


Protein classification is responsible for the biological sequence, we came up with an idea whichdeals with the classification of proteomics using deep learning algorithm.This algorithm focusesmainly to classify sequences of protein-vector which is used for the representation of proteomics.Selection  of  the  type  protein  representation  is  challenging  based  on  which  output  in  terms  ofaccuracy is depended on, The protein representation used here is n-gram i.e.  3-gram and Kerasembedding used for biological sequences like protein.In this paper we are working on the Proteinclassification to show the strength and representation of biological sequence of the proteins

Klassifizierung von Proteinfamilien mit Deep Learning


In this work, we attempt to answer a critical question: whether there exists some input sequence that will cause a well-trained discrete-space neural network sequence-to-sequence (seq2seq)  model to generate egregious outputs (aggressive, malicious, attacking, etc.).And if such inputs exist, how to find them efficiently.We adopt an empirical methodology, in which we first create lists of egregious output sequences, and then design a discrete optimization algorithm to find input sequences that will cause the model to generate them.Moreover, the optimization algorithm is enhanced for large vocabulary search and constrained to search for input sequences that are likely to be input by real-world users.In our experiments, we apply this approach to  dialogue response generation models trained on three real-world dialogue data-sets: Ubuntu, Switchboard and OpenSubtitles, testing whether the model can generate malicious responses.We demonstrate that given the trigger inputs our algorithm finds, a significant number of malicious sentences are assigned large probability by the model, which reveals an undesirable consequence of standard seq2seq training.

Ziel dieses Beitrags ist es, eine empirische Antwort auf die Frage zu geben, ob ein gut trainiertes Dialogantwortmodell bösartige Antworten ausgeben kann.


In the problem of unsupervised learning of disentangled representations, one of the promising methods is to penalize the total correlation of sampled latent vari-ables.  Unfortunately, this well-motivated strategy often fail to achieve disentanglement due to a problematic difference between the sampled latent representation and its corresponding mean representation.  We provide a theoretical explanation that low total correlation of sample distribution cannot guarantee low total correlation of the  mean representation.We prove that for the mean representation of arbitrarily high total correlation, there exist distributions of latent variables of abounded total correlation.  However, we still believe that total correlation could be a key to the disentanglement of unsupervised representative learning, and we propose a remedy,  RTC-VAE, which rectifies the total correlation penalty.   Experiments show that our model has a more reasonable distribution of the mean representation compared with baseline models, e.g.,β-TCVAE and FactorVAE.

das gesamte Problem der STOA VAEs theoretisch und qualitativ diagnostiziert


Visual attention mechanisms have been widely used in image captioning models.In this paper, to better link the image structure with the generated text, we replace the traditional softmax attention mechanism by two alternative sparsity-promoting transformations: sparsemax and Total-Variation Sparse Attention (TVmax).With sparsemax, we obtain sparse attention weights, selecting relevant features.  In order to promote sparsity and encourage fusing of the related adjacent spatial locations, we propose TVmax.  By selecting relevant groups of features, the TVmax transformation improves interpretability.We present results in the Microsoft COCO and Flickr30k datasets, obtaining gains in comparison to softmax.  TVmax outperforms the other compared attention mechanisms in terms of human-rated caption quality and attention relevance.

Wir schlagen einen neuen, spärlichen und strukturierten Aufmerksamkeitsmechanismus, TVmax, vor, der die Spärlichkeit fördert und dazu anregt, dass die Gewichtung verwandter benachbarter Orte gleich ist.


Although there are more than 65,000 languages in the world, the pronunciations of many phonemes sound similar across the languages.When people learn a foreign language, their pronunciation often reflect their native language's characteristics.That motivates us to investigate how the speech synthesis network learns the pronunciation when multi-lingual dataset is given.In this study, we train the speech synthesis network bilingually in English and Korean, and analyze how the network learns the relations of phoneme pronunciation between the languages.Our experimental result shows that the learned phoneme embedding vectors are located closer if their pronunciations are similar across the languages.Based on the result, we also show that it is possible to train networks that synthesize English speaker's Korean speech and vice versa.In another experiment, we train the network with limited amount of English dataset and large Korean dataset, and analyze the required amount of dataset to train a resource-poor language with the help of resource-rich languages.

Gelernte Phonemeeinbettungen mehrsprachiger neuronaler Sprachsynthesenetzwerke könnten die Beziehungen der Phonemeaussprache zwischen den Sprachen darstellen.


Generative Adversarial Networks (GANs) have recently achieved impressive results for many real-world applications, and many GAN variants have emerged with improvements in sample quality and training stability.However, visualization and understanding of GANs is largely missing.How does a GAN represent our visual world internally?What causes the artifacts in GAN results?How do architectural choices affect GAN learning?Answering such questions could enable us to develop new insights and better models.In this work, we present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level.We first identify a group of interpretable units that are closely related to object concepts with a segmentation-based network dissection method.Then, we quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.Finally, we examine the contextual relationship between these units and their surrounding by inserting the discovered object concepts into new images.We show several practical applications enabled by our framework, from comparing internal representations across different layers, models, and datasets, to improving GANs by locating and removing artifact-causing units, to interactively manipulating objects in the scene.We provide open source interpretation tools to help peer researchers and practitioners better understand their GAN models.

GAN-Darstellungen werden im Detail untersucht, und es werden Sätze von Darstellungseinheiten gefunden, die die Erzeugung semantischer Konzepte im Output steuern.


Historically, the pursuit of efficient inference has been one of the driving forces be-hind the research into new deep learning architectures and building blocks.Some of the recent examples include:  the squeeze-and-excitation module of (Hu et al.,2018), depthwise separable convolutions in Xception (Chollet, 2017), and the inverted bottleneck in MobileNet v2 (Sandler et al., 2018).  Notably, in all of these cases, the resulting building blocks enabled not only higher efficiency, but also higher accuracy, and found wide adoption in the field.In this work, we further expand the arsenal of efficient building blocks for neural network architectures; but instead of combining standard primitives (such as convolution), we advocate for the replacement of these dense primitives with their sparse counterparts.  While the idea of using sparsity to decrease the parameter count is not new (Mozer & Smolensky, 1989), the conventional wisdom is that this reduction in theoretical FLOPs does not translate into real-world efficiency gains.  We aim to correct this misconception by introducing a family of efficient sparse kernels for several hardware platforms, which we plan to open-source for the benefit of the community.Equipped with our efficient implementation of sparse primitives,  we show that sparse versions of MobileNet  v1 and MobileNet  v2  architectures  substantially outperform strong dense baselines on the efficiency-accuracy curve.   On Snapdragon 835 our sparse networks outperform their dense equivalents by 1.1−2.2x – equivalent to approximately one entire generation of improvement.  We hope that our findings will facilitate wider adoption of sparsity as a tool for creating efficient and accurate deep learning architectures.

Sparse MobileNets sind schneller als Dense Nets mit den entsprechenden Kerneln.


In this work, we address the semi-supervised classification of graph data, where the categories of those unlabeled nodes are inferred from labeled nodes as well as graph structures.Recent works often solve this problem with the advanced graph convolution in a conventional supervised manner, but the performance could be heavily affected when labeled data is scarce.Here we propose a Graph Inference Learning (GIL) framework to boost the performance of node classification by learning the inference of node labels on graph topology.To bridge the connection of two nodes, we formally define a structure relation by encapsulating node attributes, between-node paths and local topological structures together, which can make inference conveniently deduced from one node to another node.For learning the inference process, we further introduce meta-optimization on structure relations from training nodes to validation nodes, such that the learnt graph inference capability can be better self-adapted into test nodes.Comprehensive evaluations on four benchmark datasets (including Cora, Citeseer, Pubmed and NELL) demonstrate the superiority of our GIL when compared with other state-of-the-art methods in the semi-supervised node classification task.

Wir schlagen einen neuartigen Rahmen für das Erlernen von Grapheninferenzen vor, indem wir Strukturbeziehungen aufbauen, um unbekannte Knotenbeschriftungen aus den beschrifteten Knoten in einer durchgängigen Weise abzuleiten.


This paper proposes a method for efficient training of Q-function for continuous-state Markov Decision Processes (MDP), such that the traces of the resulting policies satisfy a Linear Temporal Logic (LTL) property.LTL, a modal logic, can express a wide range of time-dependent logical properties including safety and liveness.We convert the LTL property into a limit deterministic Buchi automaton with which a synchronized product MDP is constructed.The control policy is then synthesised by a reinforcement learning algorithm assuming that no prior knowledge is available from the MDP.The proposed method is evaluated in a numerical study to test the quality of the generated control policy and is compared against conventional methods for policy synthesis such as MDP abstraction (Voronoi quantizer) and approximate dynamic programming (fitted value iteration).

Da Sicherheit zu einem kritischen Begriff beim maschinellen Lernen wird, glauben wir, dass diese Arbeit als Grundlage für eine Reihe von Forschungsrichtungen dienen kann, wie z. B. für sicherheitsbewusste Lernalgorithmen.


We introduce two approaches for conducting efficient Bayesian inference in stochastic simulators containing nested stochastic sub-procedures, i.e., internal procedures for which the density cannot be calculated directly such as rejection sampling loops.The resulting class of simulators are used extensively throughout the sciences and can be interpreted as probabilistic generative models.However, drawing inferences from them poses a substantial challenge due to the inability to evaluate even their unnormalised density, preventing the use of many standard inference procedures like Markov Chain Monte Carlo (MCMC).To address this, we introduce inference algorithms based on a two-step approach that first approximates the conditional densities of the individual sub-procedures, before using these approximations to run MCMC methods on the full program.Because the sub-procedures can be dealt with separately and are lower-dimensional than that of the overall problem, this two-step process allows them to be isolated and thus be tractably dealt with, without placing restrictions on the overall dimensionality of the problem.We demonstrate the utility of our approach on a simple, artificially constructed simulator.

Wir stellen zwei Ansätze für eine effiziente und skalierbare Inferenz in stochastischen Simulatoren vor, für die die Dichte nicht direkt ausgewertet werden kann, z.B. aufgrund von Rückweisungsschleifen.


While adversarial training can improve robust accuracy (against an adversary), it sometimes hurts standard accuracy (when there is no adversary).Previous work has studied this tradeoff between standard and robust accuracy, but only in the setting where no predictor performs well on both objectives in the infinite data limit.In this paper, we show that even when the optimal predictor with infinite data performs well on both objectives, a tradeoff can still manifest itself with finite data.Furthermore, since our construction is based on a convex learning problem, we rule out optimization concerns, thus laying bare a fundamental tension between robustness and generalization.Finally, we show that robust self-training mostly eliminates this tradeoff by leveraging unlabeled data.

Selbst wenn es bei unendlich vielen Daten keinen Kompromiss gibt, kann das adversarische Training selbst bei einem konvexen Problem eine schlechtere Standardgenauigkeit aufweisen.


Skip connections are increasingly utilized by deep neural networks to improve accuracy and cost-efficiency.In particular, the recent DenseNet is efficient in computation and parameters, and achieves state-of-the-art predictions by directly connecting each feature layer to all previous ones.However, DenseNet's extreme connectivity pattern may hinder its scalability to high depths, and in applications like fully convolutional networks, full DenseNet connections are prohibitively expensive. This work first experimentally shows that one key advantage of skip connections is to have short distances among feature layers during backpropagation.Specifically, using a fixed number of skip connections, the connection patterns with shorter backpropagation distance among layers have more accurate predictions.Following this insight, we propose a connection template, Log-DenseNet, which, in comparison to DenseNet,  only slightly increases the backpropagation distances among layers from 1 to  ($1 + \log_2 L$), but uses only $L\log_2 L$ total connections instead of $O(L^2)$.Hence, \logdenses are easier to scale than DenseNets, and no longer require careful GPU memory management.We demonstrate the effectiveness of our design principle by showing better performance than DenseNets on tabula rasa semantic segmentation, and competitive results on visual recognition.

Wir zeigen, dass Shortcut-Verbindungen in Mustern platziert werden sollten, die die Abstände zwischen den Schichten während der Backpropagation minimieren, und entwerfen Netzwerke, die log L Abstände mit L log(L) Verbindungen erreichen.


Building agents to interact with the web would allow for significant improvements in knowledge understanding and representation learning.However, web navigation tasks are difficult for current deep reinforcement learning (RL) models due to the large discrete action space and the varying number of actions between the states.In this work, we introduce DOM-Q-NET, a novel architecture for RL-based web navigation to address both of these problems.It parametrizes Q functions with separate networks for different action categories: clicking a DOM element and typing a string input.  Our model utilizes a graph neural network to represent the tree-structured HTML of a standard web page.  We demonstrate the capabilities of our model on the MiniWoB environment where we can match or outperform existing work without the use of expert demonstrations.Furthermore, we show 2x improvements in sample efficiency when training in the multi-task setting, allowing our model to transfer learned behaviours across tasks.

Graphenbasiertes Deep Q Network für die Webnavigation


Learning disentangled representations that correspond to factors of variation in real-world data is critical to interpretable and human-controllable machine learning.Recently, concerns about the viability of learning disentangled representations in a purely unsupervised manner has spurred a shift toward the incorporation of weak supervision.However, there is currently no formalism that identifies when and how weak supervision will guarantee disentanglement.To address this issue, we provide a theoretical framework—including a calculus of disentanglement— to assist in analyzing the disentanglement guarantees (or lack thereof) conferred by weak supervision when coupled with learning algorithms based on distribution matching.We empirically verify the guarantees and limitations of several weak supervision methods (restricted labeling, match-pairing, and rank-pairing), demonstrating the predictive power and usefulness of our theoretical framework.

Wir haben einen theoretischen Rahmen für die schwach überwachte Entflechtung konstruiert und viele Experimente durchgeführt, um die Theorie zu untermauern.


Despite the growing interest in continual learning, most of its contemporary works have been studied in a rather restricted setting where tasks are clearly distinguishable, and task boundaries are known during training.However, if our goal is to develop an algorithm that learns as humans do, this setting is far from realistic, and it is essential to develop a methodology that works in a task-free manner.Meanwhile, among several branches of continual learning, expansion-based methods have the advantage of eliminating catastrophic forgetting by allocating new resources to learn new data.In this work, we propose an expansion-based approach for task-free continual learning.Our model, named Continual Neural Dirichlet Process Mixture (CN-DPM), consists of a set of neural network experts that are in charge of a subset of the data.CN-DPM expands the number of experts in a principled way under the Bayesian nonparametric framework.With extensive experiments, we show that our model successfully performs task-free continual learning for both discriminative and generative tasks such as image classification and image generation.

Wir schlagen zum ersten Mal einen expansionsbasierten Ansatz für aufgabenfreies kontinuierliches Lernen vor. Unser Modell besteht aus einer Reihe von neuronalen Netz Experten und erweitert die Anzahl der Experten nach dem Bayes'schen nichtparametrischen Prinzip.


Stochastic Gradient Descent (SGD) with Nesterov's momentum is a widely used optimizer in deep learning, which is observed to have excellent generalization performance.However, due to the large stochasticity, SGD with Nesterov's momentum is not robust, i.e., its performance may deviate significantly from the expectation.In this work, we propose Amortized Nesterov's Momentum, a special variant of Nesterov's momentum which has more robust iterates, faster convergence in the early stage and higher efficiency.Our experimental results show that this new momentum achieves similar (sometimes better) generalization performance with little-to-no tuning.In the convex case, we provide optimal convergence rates for our new methods and discuss how the theorems explain the empirical results.

Amortisierung der Impulse von Nesterov für ein robusteres, leichteres und schnelleres Deep Learning Training.


A state-of-the-art generative model, a ”factorized action variational autoencoder (FAVAE),” is presented for learning disentangled and interpretable representations from sequential data via the information bottleneck without supervision.The purpose of disentangled representation learning is to obtain interpretable and transferable representations from data.We focused on the disentangled representation of sequential data because there is a wide range of potential applications if disentanglement representation is extended to sequential data such as video, speech, and stock price data.Sequential data is characterized by dynamic factors and static factors: dynamic factors are time-dependent, and static factors are independent of time.Previous works succeed in disentangling static factors and dynamic factors by explicitly modeling the priors of latent variables to distinguish between static and dynamic factors.However, this model can not disentangle representations between dynamic factors, such as disentangling ”picking” and ”throwing” in robotic tasks.In this paper, we propose new model that can disentangle multiple dynamic factors.Since our method does not require modeling priors, it is capable of disentangling ”between” dynamic factors.In experiments, we show that FAVAE can extract the disentangled dynamic factors.

Wir schlagen ein neues Modell vor, das mehrere dynamische Faktoren in sequentiellen Daten entflechten kann


Generative networks are promising models for specifying visual transformations.Unfortunately, certification of generative models is challenging as one needs to capture sufficient non-convexity so to produce precise bounds on the output.Existing verification methods either fail to scale to generative networks or do not capture enough non-convexity.In this work, we present a new verifier, called ApproxLine, that can certify non-trivial properties of generative networks.ApproxLine performs both deterministic and probabilistic abstract interpretation and captures infinite sets of outputs of generative networks.We show that ApproxLine can verify interesting interpolations in the network's latent space.

Wir verifizieren deterministische und probabilistische Eigenschaften neuronaler Netze mit Hilfe nicht-konvexer Relaxationen über sichtbare Transformationen, die durch generative Modelle spezifiziert werden


Multi-view video summarization (MVS) lacks researchers’ attention due to their major challenges of inter-view correlations and overlapping of cameras.Most of the prior MVS works are offline, relying on only summary, needing extra communication bandwidth and transmission time with no focus on uncertain environments.Different from the existing methods, we propose edge intelligence based MVS and spatio-temporal features based activity recognition for IoT environments.We segment the multi-view videos on each slave device over edge into shots using light-weight CNN object detection model and compute mutual information among them to generate summary.Our system does not rely on summary only but encode and transmit it to a master device with neural computing stick (NCS) for intelligently computing inter-view correlations and efficiently recognizing activities, thereby saving computation resources, communication bandwidth, and transmission time.Experiments report an increase of 0.4 in F-measure score on MVS Office dataset as well as 0.2% and 2% increase in activity recognition accuracy over UCF-50 and YouTube 11 datasets, respectively, with lower storage and transmission time compared to state-of-the-art.The time complexity is decreased from 1.23 to 0.45 secs for a single frame processing, thereby generating 0.75 secs faster MVS.Furthermore, we made a new dataset by synthetically adding fog to an MVS dataset to show the adaptability of our system for both certain and uncertain surveillance environments.

Ein effizientes Multi-View-Video-Zusammenfassungsschema für die Aktivitätserkennung in IoT-Umgebungen.


A central goal in the study of the primate visual cortex and hierarchical models for object recognition is understanding how and why single units trade off invariance versus sensitivity to image transformations.For example, in both deep networks and visual cortex there is substantial variation from layer-to-layer and unit-to-unit in the degree of translation invariance.Here, we provide theoretical insight into this variation and its consequences for encoding in a deep network.Our critical insight comes from the fact that rectification simultaneously decreases response variance and correlation across responses to transformed stimuli, naturally inducing a positive relationship between invariance and dynamic range.Invariant input units then tend to drive the network more than those sensitive to small image transformations.We discuss consequences of this relationship for AI: deep nets naturally weight invariant units over sensitive units, and this can be strengthened with training, perhaps contributing to generalization performance.Our results predict a signature relationship between invariance and dynamic range that can now be tested in future neurophysiological studies.

Die Entzerrung in tiefen neuronalen Netzen führt natürlich dazu, dass sie eine invariante Darstellung bevorzugen.


We study the use of the Wave-U-Net architecture for speech enhancement, a model introduced by Stoller et al for the separation of music vocals and accompaniment.  This end-to-end learning method for audio source separation operates directly in the time domain, permitting the integrated modelling of phase information and being able to take large  temporal contexts into account.  Our experiments show that the proposed method improves several metrics, namely PESQ, CSIG, CBAK, COVL and SSNR, over the  state-of-the-art with respect to the speech enhancement task on the Voice Bank corpus (VCTK) dataset.We find that a reduced number of hidden layers is sufficient for speech enhancement in comparison to the original system designed for singing voice separation in music.We see this initial result as an encouraging signal to further explore speech enhancement in the time-domain, both as an end in itself and as a pre-processing step to speech recognition systems.

Die Wave-U-Net-Architektur, die vor kurzem von Stoller et al. für die Trennung von Musikquellen eingeführt wurde, ist äußerst effektiv für die Sprachanhebung und übertrifft den Stand der Technik.


Reinforcement learning (RL) is a powerful framework for solving problems by exploring and learning from mistakes.However, in the context of autonomous vehicle (AV) control, requiring an agent to make mistakes, or even allowing mistakes, can be quite dangerous and costly in the real world.For this reason, AV RL is generally only viable in simulation.Because these simulations have imperfect representations, particularly with respect to graphics, physics, and human interaction, we find motivation for a framework similar to RL, suitable to the real world.To this end, we formulate a learning framework that learns from restricted exploration by having a human demonstrator do the exploration.Existing work on learning from demonstration typically either assumes the collected data is performed by an optimal expert, or requires potentially dangerous exploration to find the optimal policy.We propose an alternative framework that learns continuous control from only safe behavior.One of our key insights is that the problem becomes tractable if the feedback score that rates the demonstration applies to the atomic action, as opposed to the entire sequence of actions.We use human experts to collect driving data as well as to label the driving data through a framework we call ``Backseat Driver'', giving us state-action pairs matched with scalar values representing the score for the action.We call the more general learning framework ReNeg, since it learns a regression from states to actions given negative as well as positive examples.We empirically validate several models in the ReNeg framework, testing on lane-following with limited data.We find that the best solution in this context outperforms behavioral cloning has strong connections to stochastic policy gradient approaches.

Wir stellen einen neuartigen Rahmen für das Lernen aus Demonstrationen vor, der kontinuierliches menschliches Feedback nutzt; wir evaluieren diesen Rahmen für die kontinuierliche Steuerung von autonomen Fahrzeugen.


We explore the use of Vector Quantized Variational AutoEncoder (VQ-VAE) models for large scale image generation.To this end, we scale and enhance the autoregressive priors used in VQ-VAE to generate synthetic samples of much higher coherence and fidelity than possible before.  We use simple feed-forward encoder and decoder networks, thus our model is an attractive candidate for applications where the encoding and decoding speed is critical.Additionally, this  allows us to only sample autoregressively in the compressed latent space, which is an order of magnitude faster than sampling in the pixel space, especially for large images.We demonstrate that a multi-scale hierarchical organization of  VQ-VAE, augmented with powerful priors over the latent codes, is able to generate samples with quality that rivals that of state of the art Generative Adversarial Networks on multifaceted datasets such as ImageNet, while not suffering from GAN's known shortcomings such as mode collapse and lack of diversity.

skalieren und verbessern VQ-VAE mit leistungsstarken Prioren, um nahezu realistische Bilder zu erzeugen.


We present a graph neural network assisted Monte Carlo Tree Search approach for the classical traveling salesman problem (TSP).We adopt a greedy algorithm framework to construct the optimal solution to TSP by adding the nodes successively.A graph neural network (GNN) is trained to capture the local and global graph structure and give the prior probability of selecting each vertex every step.The prior probability provides a heuristics for MCTS, and the MCTS output is an improved probability for selecting the successive vertex, as it is the feedback information by fusing the prior with the scouting procedure.Experimental results on TSP up to 100 nodes demonstrate that the proposed method obtains shorter tours than other learning-based methods.

Ein Graph Neural Network Assisted Monte Carlo Tree Search Ansatz für das Problem des Handlungsreisenden


Graph neural networks have recently achieved great successes in predicting quantum mechanical properties of molecules.These models represent a molecule as a graph using only the distance between atoms (nodes) and not the spatial direction from one atom to another.However, directional information plays a central role in empirical potentials for molecules, e.g. in angular potentials.To alleviate this limitation we propose directional message passing, in which we embed the messages passed between atoms instead of the atoms themselves.Each message is associated with a direction in coordinate space.These directional message embeddings are rotationally equivariant since the associated directions rotate with the molecule.We propose a message passing scheme analogous to belief propagation, which uses the directional information by transforming messages based on the angle between them.Additionally, we use spherical Bessel functions to construct a theoretically well-founded, orthogonal radial basis that achieves better performance than the currently prevalent Gaussian radial basis functions while using more than 4x fewer parameters.We leverage these innovations to construct the directional message passing neural network (DimeNet).DimeNet outperforms previous GNNs on average by 77% on MD17 and by 41% on QM9.

Direktionale Nachrichtenübermittlung beinhaltet räumliche Richtungsinformationen zur Verbesserung von neuronalen Graphennetzen.


Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult.The agent needs to learn  a latent representation together with a control policy to perform the task.Fitting a high-capacity encoder using a scarce reward signal is not only extremely sample inefficient, but also prone to suboptimal convergence.Two ways to improve sample efficiency are to learn a good feature representation and use off-policy algorithms.We dissect various approaches of learning good latent features, and conclude that the image reconstruction loss is the essential ingredient that enables efficient and stable representation learning in image-based RL.Following these findings, we devise an off-policy actor-critic algorithm with an auxiliary decoder that trains end-to-end and matches state-of-the-art performance across both model-free and model-based algorithms on many challenging control tasks.We release our code to encourage future research on image-based RL.

Wir entwerfen eine einfache und effiziente modellfreie Off-Policy-Methode für bildbasiertes Reinforcement Learning, die den modernsten modellbasierten Methoden in Bezug auf die Effizienz der Beispiele entspricht.


Large deep neural networks require huge memory to run and their running speed is sometimes too slow for real applications.Therefore network size reduction with keeping accuracy is crucial for practical applications.We present a novel neural network operator, chopout, with which neural networks are trained, even in a single training process, so as to truncated sub-networks perform as well as possible.Chopout is easy to implement and integrate into most type of existing neural networks.Furthermore it enables to reduce size of networks and latent representations even after training just by truncating layers.We show its effectiveness through several experiments.

Wir stellen einen neuartigen, einfachen Operator, Chopout, vor, mit dem neuronale Netze sogar in einem einzigen Trainingsprozess so trainiert werden, dass die abgeschnittenen Teilnetze die bestmögliche Leistung erbringen.


Generative Adversarial Networks (GANs) have been shown to produce realistically looking synthetic images with remarkable success, yet their performance seems less impressive when the training set is highly diverse.In order to provide a better fit to the target data distribution when the dataset includes many different classes, we propose a variant of the basic GAN model, a Multi-Modal Gaussian-Mixture GAN (GM-GAN), where the probability distribution over the latent space is a mixture of Gaussians.We also propose a supervised variant which is capable of conditional sample synthesis.In order to evaluate the model's performance, we propose a new scoring method which separately takes into account two (typically conflicting) measures - diversity vs. quality of the generated data.  Through a series of experiments, using both synthetic and real-world datasets, we quantitatively show that GM-GANs outperform baselines, both when evaluated using the commonly used Inception Score, and when evaluated using our own alternative scoring method.In addition, we qualitatively demonstrate how the unsupervised variant of GM-GAN tends to map latent vectors sampled from different Gaussians in the latent space to samples of different classes in the data space.We show how this phenomenon can be exploited for the task of unsupervised clustering, and provide quantitative evaluation showing the superiority of our method for the unsupervised clustering of image datasets.Finally, we demonstrate a feature which further sets our model apart from other GAN models: the option to control the quality-diversity trade-off by altering, post-training, the probability distribution of the latent space.This allows one to sample higher quality and lower diversity samples, or vice versa, according to one's needs.

Multimodale Guassianische Verteilung des latenten Raums in GAN-Modellen verbessert die Leistung und ermöglicht einen Kompromiss zwischen Qualität und Vielfalt


Distributed optimization is essential for training large models on large datasets.Multiple approaches have been proposed to reduce the communication overhead in distributed training, such as synchronizing only after performing multiple local SGD steps, and decentralized methods (e.g., using gossip algorithms) to decouple communications among workers.Although these methods run faster than AllReduce-based methods, which use blocking communication before every update, the resulting models may be less accurate after the same number of updates.Inspired by the BMUF method of Chen & Huo (2016), we propose a slow momentum (SloMo) framework, where workers periodically synchronize and perform a momentum update, after multiple iterations of a base optimization algorithm.Experiments on image classification and machine translation tasks demonstrate that SloMo consistently yields improvements in optimization and generalization performance relative to the base optimizer, even when the additional overhead is amortized over many updates so that the SloMo runtime is on par with that of the base optimizer.We provide theoretical convergence guarantees showing that SloMo converges to a stationary point of smooth non-convex losses.Since BMUF is a particular instance of the SloMo framework, our results also correspond to the first theoretical convergence guarantees for BMUF.

SlowMo verbessert die Optimierungs- und Generalisierungsleistung kommunikationseffizienter dezentraler Algorithmen ohne Geschwindigkeitseinbußen.


Structural planning is important for producing long sentences, which is a missing part in current language generation models.In this work, we add a planning phase in neural machine translation to control the coarse structure of output sentences.The model first generates some planner codes, then predicts real output words conditioned on them.The codes are learned to capture the coarse structure of the target sentence.In order to learn the codes, we design an end-to-end neural network with a discretization bottleneck, which predicts the simplified part-of-speech tags of target sentences.Experiments show that the translation performance are generally improved by planning ahead.We also find that translations with different structures can be obtained by manipulating the planner codes.

Die syntaktische Struktur der Übersetzung mit Hilfe von Codes planen


Interpolation of data in deep neural networks has become a subject of significant research interest.  We prove that over-parameterized single layer fully connected autoencoders do not merely interpolate, but rather, memorize training data: they produce outputs in (a non-linear version of) the span of the training examples.In contrast to fully connected autoencoders, we prove that depth is necessary for memorization in convolutional autoencoders.  Moreover, we observe that adding nonlinearity to deep convolutional autoencoders results in a stronger form of memorization: instead of outputting points in the span of the training images, deep convolutional autoencoders tend to output individual training images.  Since convolutional autoencoder components are building blocks of deep convolutional networks, we envision that our findings will shed light on the important question of the inductive bias in over-parameterized deep networks.

Wir identifizieren Memorisierung als induktive Verzerrung der Interpolation in überparametrisierten, vollständig vernetzten und faltbaren Autokodierern.


The tremendous success of deep neural networks has motivated the need to better understand the fundamental properties of these networks, but many of the theoretical results proposed have only been for shallow networks.In this paper, we study an important primitive for understanding the meaningful input space of a deep network: span recovery.For $k<n$, let $\mathbf{A} \in \mathbb{R}^{k \times n}$ be the innermost weight matrix of an arbitrary feed forward neural network $M: \mathbb{R}^n \to  \mathbb{R}$, so $M(x)$ can be written as $M(x) = \sigma(\mathbf{A} x)$, for some network $\sigma: \mathbb{R}^k \to  \mathbb{R}$.The goal is then to recover the row span of $\mathbf{A}$ given only oracle access to the value of $M(x)$. We show that if $M$ is a multi-layered network with ReLU activation functions, then partial recovery is possible: namely, we can provably recover $k/2$ linearly independent vectors in the row span of $\mathbf{A}$ using poly$(n)$ non-adaptive queries to $M(x)$.  Furthermore, if $M$ has differentiable activation functions, we demonstrate that \textit{full} span recovery is possible even when the output is first passed through a sign or $0/1$ thresholding function; in this case our algorithm is adaptive.Empirically, we confirm that full span recovery is not always possible, but only for unrealistically thin layers.For reasonably wide networks, we obtain full span recovery on both random networks and networks trained on MNIST data.Furthermore, we demonstrate the utility of span recovery as an attack by inducing neural networks to misclassify data obfuscated by controlled random noise as sensical inputs.

Wir stellen die Spanne eines tiefen, mehrschichtigen neuronalen Netzes mit latenter Struktur nachweislich wieder her und wenden empirisch effiziente Algorithmen zur Wiederherstellung der Spanne an, um Netzwerke durch Verschleierung der Eingaben anzugreifen.


In this paper, we study the implicit regularization of the gradient descent algorithm in homogeneous neural networks, including fully-connected and convolutional neural networks with ReLU or LeakyReLU activations.In particular, we study the gradient descent or gradient flow (i.e., gradient descent with infinitesimal step size) optimizing the logistic loss or cross-entropy loss of any homogeneous model (possibly non-smooth), and show that if the training loss decreases below a certain threshold, then we can define a smoothed version of the normalized margin which increases over time.We also formulate a natural constrained optimization problem related to margin maximization, and prove that both the normalized margin and its smoothed version converge to the objective value at a KKT point of the optimization problem.Our results generalize the previous results for logistic regression with one-layer or multi-layer linear networks, and provide more quantitative convergence results with weaker assumptions than previous results for homogeneous smooth neural networks.We conduct several experiments to justify our theoretical finding on MNIST and CIFAR-10 datasets.Finally, as margin is closely related to robustness, we discuss potential benefits of training longer for improving the robustness of the model.

Wir untersuchen die implizite Verzerrung des Gradientenabstiegs und beweisen unter minimalen Annahmen, dass die Parameterrichtung von homogenen Modellen zu KKT-Punkten eines natürlichen Margenmaximierungsproblems konvergiert.


Long-term video prediction is highly challenging since it entails simultaneously capturing spatial and temporal information across a long range of image frames.Standard recurrent models are ineffective since they are prone to error propagation and cannot effectively capture higher-order correlations.A potential solution is to extend to higher-order spatio-temporal recurrent models.However, such a model requires  a  large number of parameters and operations, making it intractable  to learn in practice and is prone to overfitting.In this work, we propose convolutional tensor-train LSTM (Conv-TT-LSTM), which  learns higher-orderConvolutional LSTM (ConvLSTM) efficiently using convolutional  tensor-train decomposition (CTTD).Our proposed model naturally incorporates higher-order spatio-temporal information at a small cost of memory and computation by using efficient low-rank tensor representations.We evaluate our model on Moving-MNIST and KTH datasets and show improvements over standard ConvLSTM and better/comparable results to other ConvLSTM-based approaches, but with much fewer parameters.

wir schlagen Convolutional Tensor-Train LSTM vor, die effizient Convolutional LSTM höherer Ordnung unter Verwendung von Convolutional Tensor-Train Dekomposition erlernen.


While deep neural networks have shown outstanding results in a wide range of applications,learning from a very limited number of examples is still a challengingtask.Despite the difficulties of the few-shot learning, metric-learning techniquesshowed the potential of the neural networks for this task.While these methodsperform well, they don’t provide satisfactory results.In this work, the idea ofmetric-learning is extended with Support Vector Machines (SVM) working mechanism,which is well known for generalization capabilities on a small dataset.Furthermore, this paper presents an end-to-end learning framework for trainingadaptive kernel SVMs, which eliminates the problem of choosing a correct kerneland good features for SVMs.Next, the one-shot learning problem is redefinedfor audio signals.Then the model was tested on vision task (using Omniglotdataset) and speech task (using TIMIT dataset) as well.Actually, the algorithmusing Omniglot dataset improved accuracy from 98.1% to 98.5% on the one-shotclassification task and from 98.9% to 99.3% on the few-shot classification task.

Bei der vorgeschlagenen Methode handelt es sich um eine neuronale End-to-End-SVM, die für das Few-Shot Learning optimiert ist.


Most existing 3D CNN structures for video representation learning are clip-based methods, and do not consider video-level temporal evolution of spatio-temporal features.In this paper, we propose Video-level 4D Convolutional Neural Networks, namely V4D, to model the evolution of long-range spatio-temporal representation with 4D convolutions, as well as preserving 3D spatio-temporal representations with residual connections.We further introduce the training and inference methods for the proposed V4D.Extensive experiments are conducted on three video recognition benchmarks, where V4D achieves excellent results, surpassing recent 3D CNNs by a large margin.

Eine neuartige 4D CNN-Struktur für das Lernen von Repräsentationen auf Videoebene, die die bisherigen 3D CNNs übertrifft.


We study the problem of learning and optimizing through physical simulations via differentiable programming.We present DiffSim, a new differentiable programming language tailored for building high-performance differentiable physical simulations.We demonstrate the performance and productivity of our language in gradient-based learning and optimization tasks on 10 different physical simulators.For example, a differentiable elastic object simulator written in our language is 4.6x faster than the hand-engineered CUDA version yet runs as fast, and is 188x faster than TensorFlow.Using our differentiable programs, neural network controllers are typically optimized within only tens of iterations.Finally, we share the lessons learned from our experience developing these simulators, that is, differentiating physical simulators does not always yield useful gradients of the physical system being simulated.We systematically study the underlying reasons and propose solutions to improve gradient quality.

Wir untersuchen das Problem des Lernens und Optimierens durch physikalische Simulationen mittels differenzierbarer Programmierung unter Verwendung der von uns vorgeschlagenen DiffSim-Programmiersprache und des Compilers.


Local explanation frameworks aim to rationalize particular decisions made by a black-box prediction model.Existing techniques are often restricted to a specific type of predictor or based on input saliency, which may be undesirably sensitive to factors unrelated to the model's decision making process.We instead propose sufficient input subsets that identify minimal subsets of features whose observed values alone suffice for the same decision to be reached, even if all other input feature values are missing.General principles that globally govern a model's decision-making can also be revealed by searching for clusters of such input patterns across many data points.Our approach is conceptually straightforward, entirely model-agnostic, simply implemented using instance-wise backward selection, and able to produce more concise rationales than existing techniques.We demonstrate the utility of our interpretation method on neural network models trained on text and image data.

Wir stellen eine Methode zur Interpretation von Black-Box-Modellen vor, bei der durch instanzielle Rückwärtsselektion minimale Teilmengen von Merkmalen ermittelt werden, die allein ausreichen, um eine bestimmte Entscheidung des Modells zu rechtfertigen.


In this paper, we tackle the problem of detecting samples that are not drawn from the training distribution, i.e., out-of-distribution (OOD) samples, in classification.Many previous studies have attempted to solve this problem by regarding samples with low classification confidence as OOD examples using deep neural networks (DNNs).However, on difficult datasets or models with low classification ability, these methods incorrectly regard in-distribution samples close to the decision boundary as OOD samples.This problem arises because their approaches use only the features close to the output layer and disregard the uncertainty of the features.Therefore, we propose a method that extracts the uncertainties of features in each layer of DNNs using a reparameterization trick and combines them.In experiments, our method outperforms the existing methods by a large margin, achieving state-of-the-art detection performance on several datasets and classification models.For example, our method increases the AUROC score of prior work (83.8%) to 99.8% in DenseNet on the CIFAR-100 and Tiny-ImageNet datasets.

Wir schlagen eine Methode vor, die die Unsicherheiten von Merkmalen in jeder Schicht von DNNs extrahiert und sie zur Erkennung von OOD-Proben bei der Lösung von Klassifizierungsaufgaben kombiniert.


In this paper, we explore new approaches to combining information encoded within the learned representations of autoencoders.We explore models that are capable of combining the attributes of multiple inputs such that a resynthesised output is trained to fool an adversarial discriminator for real versus synthesised data.Furthermore, we explore the use of such an architecture in the context of semi-supervised learning, where we learn a mixing function whose objective is to produce interpolations of hidden states, or masked combinations of latent representations that are consistent with a conditioned class label.We show quantitative and qualitative evidence that such a formulation is an interesting avenue of research.

Wir setzen deterministische Autoencoder als generative Modelle ein, indem wir Mischfunktionen vorschlagen, die verborgene Zustände aus Bildpaaren kombinieren. Diese Mischungen werden durch ein adversarial Framework realistisch dargestellt.


We outline the problem of concept drifts for time series data.In this work, we analyze the temporal inconsistency of streaming wireless signals in the context of device-free passive indoor localization.We show that data obtained from WiFi channel state information (CSI) can be used to train a robust system capable of performing room level localization.One of the most challenging issues for such a system is the movement of input data distribution to an unexplored space over time, which leads to an unwanted shift in the learned boundaries of the output space.In this work, we propose a phase and magnitude augmented feature space along with a standardization technique that is little affected by drifts.We show that this robust representation of the data yields better learning accuracy and requires less number of retraining.

Wir stellen einen erweiterten robusten Merkmalsraum für Streaming-WiFi-Daten vor, der in der Lage ist, die Konzeptabweichung für die Lokalisierung in Innenräumen zu bewältigen.


Federated learning improves data privacy and efficiency in machine learning performed over networks of distributed devices, such as mobile phones, IoT and wearable devices, etc.Yet models trained with federated learning can still fail to generalize to new devices due to the problem of domain shift.Domain shift occurs when the labeled data collected by source nodes statistically differs from the target node's unlabeled data.In this work, we present a principled approach to the problem of federated domain adaptation, which aims to align the representations learned among the different nodes with the data distribution of the target node.Our approach extends adversarial adaptation techniques to the constraints of the federated setting.In addition, we devise a dynamic attention mechanism and leverage feature disentanglement to enhance knowledge transfer.Empirically, we perform extensive experiments on several image and text classification tasks and show promising results under unsupervised federated domain adaptation setting.

wir stellen einen prinzipiellen Ansatz für das Problem der föderierten Domänenanpassung vor, der darauf abzielt, die zwischen den verschiedenen Knoten erlernten Repräsentationen an die Datenverteilung des Zielknotens anzupassen.


A capsule is a group of neurons whose outputs represent different properties of the same entity.Each layer in a capsule network contains many capsules.We describe a version of capsules in which each capsule has a logistic unit to represent the presence of an entity and a 4x4 matrix which could learn to represent the relationship between that entity and the viewer (the pose).A capsule in one layer votes for the pose matrix of many different capsules in the layer above by multiplying its own pose matrix by trainable viewpoint-invariant transformation matrices that could learn to represent part-whole relationships.Each of these votes is weighted by an assignment coefficient.These coefficients are iteratively updated for each image using the Expectation-Maximization algorithm such that the output of each capsule is routed to a capsule in the layer above that receives a cluster of similar votes.The transformation matrices are trained discriminatively by backpropagating through the unrolled iterations of EM between each pair of adjacent capsule layers.On the smallNORB benchmark, capsules reduce the number of test errors by 45\% compared to the state-of-the-art.Capsules also show far more resistance to white box adversarial attacks than our baseline convolutional neural network.

Kapselnetzwerke mit gelernten Pose-Matrizen und EM-Routing verbessern den Stand der Technik bei der Klassifizierung auf smallNORB, verbessern die Verallgemeinerbarkeit auf neue Blickwinkel und die White Box Adversarial Robustheit.


We study a general formulation of program synthesis called syntax-guided synthesis(SyGuS) that concerns synthesizing a program that follows a given grammar and satisfies a given logical specification.Both the logical specification and the grammar have complex structures and can vary from task to task, posing significant challenges for learning across different tasks.Furthermore, training data is often unavailable for domain specific synthesis tasks.To address these challenges, we propose a meta-learning framework that learns a transferable policy from only weak supervision.Our framework consists of three components:1) an encoder, which embeds both the logical specification and grammar at the same time using a graph neural network;2) a grammar adaptive policy network which enables learning a transferable policy; and3) a reinforcement learning algorithm that jointly trains the embedding and adaptive policy.We evaluate the framework on 214 cryptographic circuit synthesis tasks.It solves 141 of them in the out-of-box solver setting, significantly outperforming a similar search-based approach but without learning, which solves only 31.The result is comparable to two state-of-the-art classical synthesis engines, which solve 129 and 153 respectively.In the meta-solver setting, the framework can efficiently adapt to unseen tasks and achieves speedup ranging from 2x up to 100x.

Wir schlagen ein Meta-Lernverfahren vor, das eine übertragbare Strategie aus nur schwacher Überwachung lernt, um Syntheseaufgaben mit unterschiedlichen logischen Spezifikationen und Grammatiken zu lösen.


Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence.Recent approach for this task either apply a fixed policy on transformer, or a learnable monotonic attention on a weaker recurrent neural network based structure.In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which introduced the monotonic attention mechanism to multihead attention.We also introduced two novel interpretable approaches for latency control that are specifically designed for multiple attentions.We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach.Code will be released upon publication.

Machen Sie den Transformer streamfähig mit monotoner Aufmerksamkeit.


This paper presents a novel two-step approach for the fundamental problem of learning an optimal map from one distribution to another.First, we learn an optimal transport (OT) plan, which can be thought as a one-to-many map between the two distributions.To that end, we propose a stochastic dual approach of regularized OT, and show empirically that it scales better than a recent related approach when the amount of samples is very large.Second, we estimate a Monge map as a deep neural network learned by approximating the barycentric projection of the previously-obtained OT plan.This parameterization allows generalization of the mapping outside the support of the input measure.We prove two theoretical stability results of regularized OT which show that our estimations converge to the OT and Monge map between the underlying continuous measures.We showcase our proposed approach on two applications: domain adaptation and generative modeling.

Erlernen einer optimalen Abbildung mit deepNN zwischen den Verteilungen zusammen mit theoretischen Garantien.