Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales.Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments.Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave.This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (~3 ms) between note labels and audio waveforms.The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music.

Wir trainieren eine Reihe von Modellen, die in der Lage sind, Audiowellenformen mit kohärenter musikalischer Struktur zu transkribieren, zu komponieren und zu synthetisieren, was durch den neuen MAESTRO-Datensatz ermöglicht wird.


Variational inference based on chi-square divergence minimization (CHIVI) provides a way to approximate a model's posterior while obtaining an upper bound on the marginal likelihood.However, in practice CHIVI relies on Monte Carlo (MC) estimates of an upper bound objective that at modest sample sizes are not guaranteed to be true bounds on the marginal likelihood.This paper provides an empirical study of CHIVI performance on a series of synthetic inference tasks.We show that CHIVI is far more sensitive to initialization than classic VI based on KL minimization, often needs a very large number of samples (over a million), and may not be a reliable upper bound.We also suggest possible ways to detect and alleviate some of these pathologies, including diagnostic bounds and initialization strategies.

Eine empirische Untersuchung der Variationsinferenz auf der Grundlage der Minimierung der Chi-Quadrat-Divergenz, die zeigt, dass die Minimierung des CUBO schwieriger ist als die Maximierung des ELBO.


It has been widely recognized that adversarial examples can be easily crafted to fool deep networks, which mainly root from the locally non-linear behavior nearby input examples.Applying mixup in training provides an effective mechanism to improve generalization performance and model robustness against adversarial perturbations, which introduces the globally linear behavior in-between training examples.However, in previous work, the mixup-trained models only passively defend adversarial attacks in inference by directly classifying the inputs, where the induced global linearity is not well exploited.Namely, since the locality of the adversarial perturbations, it would be more efficient to actively break the locality via the globality of the model predictions.Inspired by simple geometric intuition, we develop an inference principle, named mixup inference (MI), for mixup-trained models.MI mixups the input with other random clean samples, which can shrink and transfer the equivalent perturbation if the input is adversarial.Our experiments on CIFAR-10 and CIFAR-100 demonstrate that MI can further improve the adversarial robustness for the models trained by mixup and its variants.

Wir nutzen die globale Linearität der durch die Verwechslung trainierten Modelle bei der Inferenz, um die Lokalität der gegnerischen Störungen zu brechen.


Fine-tuning language models, such as BERT, on domain specific corpora has proven to be valuable in domains like scientific papers and biomedical text.In this paper, we show that fine-tuning BERT on legal documents similarly provides valuable improvements on NLP tasks in the legal domain.Demonstrating this outcome is significant for analyzing commercial agreements, because obtaining large legal corpora is challenging due to their confidential nature.As such, we show that having access to large legal corpora is a competitive advantage for commercial applications, and academic research on analyzing contracts.

Die Feinabstimmung von BERT auf juristische Korpora bietet marginale, aber wertvolle Verbesserungen bei NLP-Aufgaben im juristischen Bereich.


We present a deep generative model for unsupervised text style transfer that unifies previously proposed non-generative techniques.Our probabilistic approach models non-parallel data from two domains as a partially observed parallel corpus.By hypothesizing a parallel latent sequence that generates each observed sequence, our model learns to transform sequences from one domain to another in a completely unsupervised fashion.In contrast with traditional generative sequence models (e.g. the HMM), our model makes few assumptions about the data it generates: it uses a recurrent language model as a prior and an encoder-decoder as a transduction distribution.While computation of marginal data likelihood is intractable in this model class, we show that amortized variational inference admits a practical surrogate.Further, by drawing connections between our variational objective and other recent unsupervised style transfer and machine translation techniques, we show how our probabilistic view can unify some known non-generative objectives such as backtranslation and adversarial loss.Finally, we demonstrate the effectiveness of our method on a wide range of unsupervised style transfer tasks, including sentiment transfer, formality transfer, word decipherment, author imitation, and related language translation.Across all style transfer tasks, our approach yields substantial gains over state-of-the-art non-generative baselines, including the state-of-the-art unsupervised machine translation techniques that our approach generalizes.Further, we conduct experiments on a standard unsupervised machine translation task and find that our unified approach matches the current state-of-the-art.

Wir formulieren ein probabilistisches latentes Sequenzmodell, um den unüberwachten Textstiltransfer zu bewältigen, und zeigen seine Wirksamkeit in einer Reihe von unüberwachten Textstiltransferaufgaben.


Current practice in machine learning is to employ deep nets in an overparametrized limit, with the nominal number of parameters typically exceeding the number of measurements.This resembles the situation in compressed sensing, or in sparse regression with $l_1$ penalty terms, and provides a theoretical avenue for understanding phenomena that arise in the context of deep nets.One such phenonemon is the success of deep nets in providing good generalization in an interpolating regime with zero training error.Traditional statistical practice calls for regularization or smoothing to prevent "overfitting" (poor generalization performance).However, recent work shows that there exist data interpolation procedures which are statistically consistent and provide good generalization performance\cite{belkin2018overfitting} ("perfect fitting").In this context, it has been suggested that "classical" and "modern" regimes for machine learning are separated by a peak in the generalization error ("risk") curve, a phenomenon dubbed "double descent"\cite{belkin2019reconciling}.While such overfitting peaks do exist and arise from ill-conditioned design matrices, here we challenge the interpretation of the overfitting peak as demarcating the regime where good generalization occurs under overparametrization. We propose a model of Misparamatrized Sparse Regression (MiSpaR) and analytically compute the GE curves for $l_2$ and $l_1$ penalties.We show that the overfitting peak arising in the interpolation limit is dissociated from the regime of good generalization.The analytical expressions are obtained in the so called "thermodynamic" limit.We find an additional interesting phenomenon: increasing overparametrization in the fitting model increases sparsity, which should intuitively improve performance of $l_1$ penalized regression.However, at the same time, the relative number of measurements decrease compared to the number of fitting parameters, and eventually overparametrization does lead to poor generalization.Nevertheless, $l_1$ penalized regression can show good generalization performance under conditions of data interpolation even with a large amount of overparametrization.These results provide a theoretical avenue into studying inverse problems in the interpolating regime using overparametrized fitting functions such as deep nets.

Schlägt ein analytisch nachvollziehbares Modell und Inferenzverfahren vor (misparametrisierte spärliche Regression, abgeleitet mit L_1 Strafe und untersucht in der Daten-Interpolationsgrenze), um Deep-Net-verwandte Phänomene im Kontext von inversen Problemen zu untersuchen.


Hashing-based collaborative filtering learns binary vector representations (hash codes) of users and items, such that recommendations can be computed very efficiently using the Hamming distance, which is simply the sum of differing bits between two hash codes.A problem with hashing-based collaborative filtering using the Hamming distance, is that each bit is equally weighted in the distance computation, but in practice some bits might encode more important properties than other bits, where the importance depends on the user. To this end, we propose an end-to-end trainable variational hashing-based collaborative filtering approach that uses the novel concept of self-masking: the user hash code acts as a mask on the items (using the Boolean AND operation), such that it learns to encode which bits are important to the user, rather than the user's preference towards the underlying item property that the bits represent.This allows a binary user-level importance weighting of each item without the need to store additional weights for each user.We experimentally evaluate our approach against state-of-the-art baselines on 4 datasets, and obtain significant gains of up to 12% in NDCG.We also make available an efficient implementation of self-masking, which experimentally yields <4% runtime overhead compared to the standard Hamming distance.

Wir schlagen einen neuen variationalen Hashing-basierten kollaborativen Filteransatz vor, der für eine neuartige Selbstmasken-Variante der Hamming-Distanz optimiert ist und den Stand der Technik bei NDCG um bis zu 12 % übertrifft.


Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search.This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit that achieves performance equivalent to that of best fixed batch-size.At each epoch, the RMGD samples a batch size according to a certain probability distribution proportional to a batch being successful in reducing the loss function.Sampling from this probability provides a mechanism for exploring different batch size and exploiting batch sizes with history of success.  After obtaining the validation loss at each epoch with the sampled batch size, the probability distribution is updated to incorporate the effectiveness of the sampled batch size.Experimental results show that the RMGD achieves performance better than the best performing single batch size.It is surprising that the RMGD achieves better performance than grid search.Furthermore, it attains this performance in a shorter amount of time than grid search.

Ein Optimierungsalgorithmus, der verschiedene Chargengrößen auf der Grundlage der Wahrscheinlichkeit untersucht und automatisch die erfolgreichste Chargengröße nutzt, die den Validierungsverlust minimiert.


Knowledge graph has gained increasing attention in recent years for its successful applications of numerous tasks.Despite the rapid growth of knowledge construction, knowledge graphs still suffer from severe incompletion and inevitably involve various kinds of errors.Several attempts have been made to complete knowledge graph as well as to detect noise.However, none of them considers unifying these two tasks even though they are inter-dependent and can mutually boost the performance of each other.In this paper, we proposed to jointly combine these two tasks with a unified Generative Adversarial Networks (GAN) framework to learn noise-aware knowledge graph embedding.Extensive experiments have demonstrated that our approach is superior to existing state-of-the-art algorithms both in regard to knowledge graph completion and error detection.

Wir haben einen einheitlichen Rahmen für Generative Adversarial Networks (GAN) vorgeschlagen, um eine geräuschbewusste Wissensgrapheneinbettung zu erlernen.


Energy-based models (EBMs), a.k.a.un-normalized models, have had recent successes in continuous spaces.However, they have not been successfully applied to model text sequences.  While decreasing the energy at training samples is straightforward, mining (negative) samples where the energy should be increased is difficult.   In part, this is because standard gradient-based methods are not readily applicable when the input is high-dimensional and discrete.  Here, we side-step this issue by generating negatives using pre-trained auto-regressive language models.  The EBM then worksin the {\em residual} of the language model; and is trained to discriminate real text from text generated by the auto-regressive models.We  investigate the generalization ability of residual EBMs, a pre-requisite for using them in other applications.  We extensively analyze generalization for the task of classifying whether an input is machine or human generated, a natural task given the training loss and how we mine negatives.Overall, we observe that EBMs can generalize remarkably well to changes in the architecture of the generators producing negatives.However, EBMs exhibit more sensitivity to the training set used by such generators.

Eine Rest-EBM für Text, deren Formulierung der Unterscheidung zwischen menschlichem und maschinell generiertem Text entspricht. Wir untersuchen sein Generalisierungsverhalten.


Semi-supervised learning, i.e. jointly learning from labeled an unlabeled samples, is an active research topic due to its key role on relaxing human annotation constraints.In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different  perturbations of unlabeled samples.We, conversely, propose to learn from unlabeled data by generating soft pseudo-labels using the network predictions.We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it.The proposed approach achieves state-of-the-art results in CIFAR-10/100 and Mini-ImageNet despite being much simpler than other state-of-the-art.These results demonstrate that pseudo-labeling can outperform consistency regularization methods, while the opposite was supposed in previous work.Code will be made available.

Pseudo-Labeling hat sich als schwache Alternative für semi-supervised learning erwiesen. Umgekehrt zeigen wir, dass Pseudo-Labeling ein geeigneter Ansatz ist, um Bestätigungsfehler mit verschiedenen Regularisierungen zu beseitigen.


Model-free reinforcement learning (RL) has been proven to be a powerful, general tool for learning complex behaviors.However, its sample efficiency is often impractically large for solving challenging real-world problems, even for off-policy algorithms such as Q-learning.A limiting factor in classic model-free RL is that the learning signal consists only of scalar rewards, ignoring much of the rich information contained in state transition tuples.Model-based RL uses this information, by training a predictive model, but often does not achieve the same asymptotic performance as model-free RL due to model bias.We introduce temporal difference models (TDMs), a family of goal-conditioned value functions that can be trained with model-free learning and used for model-based control.TDMs combine the benefits of model-free and model-based RL: they leverage the rich information in state transitions to learn very efficiently, while still attaining asymptotic performance that exceeds that of direct model-based RL methods.Our experimental results show that, on a range of continuous control tasks, TDMs provide a substantial improvement in efficiency compared to state-of-the-art model-based and model-free methods.

Wir zeigen, dass eine spezielle, mit modellfreien Methoden trainierte Zielzustandswertfunktion im Rahmen der modellbasierten Steuerung verwendet werden kann, was zu einer wesentlich besseren Stichprobeneffizienz und Leistung führt.


We introduce a neural architecture to perform amortized approximate Bayesian inference over latent random permutations of two sets of objects.The method involves approximating permanents of matrices of pairwise probabilities using recent ideas on functions defined over sets.Each sampled permutation comes with a probability estimate, a quantity unavailable in MCMC approaches.We illustrate the method in sets of 2D points and MNIST images.

Eine neuartige neuronale Architektur zur effizienten Amortisierung von Schlussfolgerungen über latente Permutationen.


Machine learned large-scale retrieval systems require a large amount of training data representing query-item relevance.However, collecting users' explicit feedback is costly.In this paper, we propose to leverage user logs and implicit feedback as auxiliary objectives to improve relevance modeling in retrieval systems.Specifically, we adopt a two-tower neural net architecture to model query-item relevance given both collaborative and content information.By introducing auxiliary tasks trained with much richer implicit user feedback data, we improve the quality and resolution for the learned representations of queries and items.Applying these learned representations to an industrial retrieval system has delivered significant improvements.

Wir schlagen eine neuartige Zwei-Turm-Shared-Bottom-Modellarchitektur vor, um Wissen aus reichhaltigen impliziten Rückmeldungen zu übertragen, um Relevanz für groß angelegte Retrievalsysteme vorherzusagen.


The ability to autonomously explore and navigate a physical space is a fundamental requirement for virtually any mobile autonomous agent, from household robotic vacuums to autonomous vehicles.Traditional SLAM-based approaches for exploration and navigation largely focus on leveraging scene geometry, but fail to model dynamic objects (such as other agents) or semantic constraints (such as wet floors or doorways).Learning-based RL agents are an attractive alternative because they can incorporate both semantic and geometric information, but are notoriously sample inefficient, difficult to generalize to novel settings, and are difficult to interpret.In this paper, we combine the best of both worlds with a modular approach that {\em learns} a spatial representation of a scene that is trained to be effective when coupled with traditional geometric planners.Specifically, we design an agent that learns to predict a spatial affordance map that elucidates what parts of a scene are navigable through active self-supervised experience gathering.In contrast to most simulation environments that assume a static world, we evaluate our approach in the VizDoom simulator, using large-scale randomly-generated maps containing a variety of dynamic actors and hazards.We show that learned affordance maps can be used to augment traditional approaches for both exploration and navigation, providing significant improvements in performance.

Wir befassen uns mit der Aufgabe der autonomen Exploration und Navigation unter Verwendung von räumlichen Affordanzzuordnungen, die auf selbstüberwachte Weise erlernt werden können. Diese übertreffen die klassischen geometrischen Grundlinien und sind gleichzeitig effizienter als zeitgenössische RL-Algorithmen.