Deep Learning NLP domain lacks procedures for the analysis of model robustness. In this paper we propose a framework which validates robustness of any Question Answering model through model explainers. We propose that output of a robust model should be invariant to alterations that do not change its semantics. We test this property by manipulating question in two ways: swapping important question word for 1) its semantically correct synonym and 2) for word vector that is close in embedding space. We estimate importance of words in asked questions with Locally Interpretable Model Agnostic Explanations method (LIME). With these two steps we compare state-of-the-art Q&A models. We show that although accuracy of state-of-the-art models is high, they are very fragile to changes in the input. We can choose architecture that is more immune to attacks and thus more robust and stable in production environment. Morevoer, we propose 2 adversarial training scenarios which raise model sensitivity to true synonyms by up to 7% accuracy measure. Our findings help to understand which models are more stable and how they can be improved. In addition, we have created and published a new dataset that may be used for validation of robustness of a Q&A model.

Proponiamo un approccio model agnostic per la convalida della robustezza del sistema Q&A e dimostriamo i risultati su modelli Q&A allo stato dell'arte.

Affronta il problema della robustezza alle informazioni adversarial nel question answering.

Migliorare la robustezza della comprensione automatica/question answering.


In this paper, we propose a mix-generator generative adversarial networks (PGAN) model that works in parallel by mixing multiple disjoint generators to approximate a complex real distribution. In our model, we propose an adjustment component that collects all the generated data points from the generators, learns the boundary between each pair of generators, and provides error to separate the support of each of the generated distributions. To overcome the instability in a multiplayer game, a shrinkage adjustment component method is introduced to gradually reduce the boundary between generators during the training procedure. To address the linearly growing training time problem in a multiple generators model, we propose a method to train the generators in parallel. This means that our work can be scaled up to large parallel computation frameworks. We present an efficient loss function for the discriminator, an effective adjustment component, and a suitable generator. We also show how to introduce the decay factor to stabilize the training procedure. We have performed extensive experiments on synthetic datasets, MNIST, and CIFAR-10. These experiments reveal that the error provided by the adjustment component could successfully separate the generated distributions and each of the generators can stably learn a part of the real distribution even if only a few modes are contained in the real distribution.

Multi generatore per catturare Pdata, risolvere la concorrenza e il problema di one-beat-all

Propone GAN parallele per evitare il mode collapse nelle GAN attraverso una combinazione di più generatori deboli.


We capitalize on the natural compositional structure of images in order to learn object segmentation with weakly labeled images. The intuition behind our approach is that removing objects from images will yield natural images, however removing random patches will yield unnatural images. We leverage this signal to develop a generative model that decomposes an image into layers, and when all layers are combined, it reconstructs the input image. However, when a layer is removed, the model learns to produce a different image that still looks natural to an adversary, which is possible by removing objects. Experiments and visualizations suggest that this model automatically learns object segmentation on images labeled only by scene better than baselines.

Segmentazione di immagini debolmente supervisionata usando la struttura compositiva delle immagini e modelli generativi.

Questo articolo crea una rappresentazione a layer per imparare meglio la segmentazione da immagini senza label.

Questo articolo propone un modello generativo basato su GAN che decompone le immagini in layer multipli, dove l'obiettivo della GAN è quello di distinguere le immagini reali dalle immagini formate combinando i layer.

Questo articolo propone un'architettura di rete neurale intorno all'idea della composizione stratificata della scena


Adversarial examples are a pervasive phenomenon of machine learning models where seemingly imperceptible perturbations to the input lead to misclassifications for otherwise statistically accurate models. We propose a geometric framework, drawing on tools from the manifold reconstruction literature, to analyze the high-dimensional geometry of adversarial examples. In particular, we highlight the importance of codimension: for low-dimensional data manifolds embedded in high-dimensional space there are many directions off the manifold in which to construct adversarial examples. Adversarial examples are a natural consequence of learning a decision boundary that classifies the low-dimensional data manifold well, but classifies points near the manifold incorrectly. Using our geometric framework we prove (1) a tradeoff between robustness under different norms, (2) that adversarial training in balls around the data is sample inefficient, and (3) sufficient sampling conditions under which nearest neighbor classifiers and ball-based adversarial training are robust.

Presentiamo un framework geometrico per dimostrare le garanzie di robustezza ed evidenziare l'importanza della codimensione negli adversarial example.

Questo articolo fornisce un'analisi teorica degli adversarial example, mostrando che esiste un compromesso tra la robustezza in diverse norme, l'adversarial training è sample inefficient, e il classificatore nearest neighbor può essere robusto in certe condizioni.


Character-based neural machine translation (NMT) models alleviate out-of-vocabulary issues, learn morphology, and move us closer to completely end-to-end translation systems.   Unfortunately, they are also very brittle and easily falter when presented with noisy data. In this paper, we confront NMT models with synthetic and natural sources of noise. We find that state-of-the-art models fail to translate even moderately noisy texts that humans have no trouble comprehending. We explore two approaches to increase model robustness: structure-invariant word representations and robust training on noisy texts. We find that a model based on a character convolutional neural network is able to simultaneously learn representations robust to multiple kinds of noise.

Questo articolo studia l'impatto del rumore a livello di carattere su 4 diversi sistemi di traduzione automatica neurale

Questo articolo studia empiricamente le prestazioni dei sistemi NMT a livello di carattere di fronte al rumore a livello di carattere, sia sintetizzato che naturale.

Questo articolo studia l'impatto dell'input rumoroso sulla traduzione automatica e prova modi per rendere i modelli NMT più robusti


As neural networks grow deeper and wider, learning networks with hard-threshold activations is becoming increasingly important, both for network quantization, which can drastically reduce time and energy requirements, and for creating large integrated systems of deep networks, which may have non-differentiable components and must avoid vanishing and exploding gradients for effective learning. However, since gradient descent is not applicable to hard-threshold functions, it is not clear how to learn them in a principled way. We address this problem by observing that setting targets for hard-threshold hidden units in order to minimize loss is a discrete optimization problem, and can be solved as such. The discrete optimization goal is to find a set of targets such that each unit, including the output, has a linearly separable problem to solve. Given these targets, the network decomposes into individual perceptrons, which can then be learned with standard convex approaches. Based on this, we develop a recursive mini-batch algorithm for learning deep hard-threshold networks that includes the popular but poorly justified straight-through estimator as a special case. Empirically, we show that our algorithm improves classification accuracy in a number of settings, including for AlexNet and ResNet-18 on ImageNet, when compared to the straight-through estimator.

Impariamo deep network con unità hard-threshold impostando gli obiettivi delle hidden unit utilizzando l'ottimizzazione combinatoria e i pesi tramite l'ottimizzazione convessa, con conseguente miglioramento delle prestazioni su ImageNet.

L'articolo spiega e generalizza gli approcci per l'apprendimento delle reti neurali con attivazione hard.

Questo articolo esamina il problema dell'ottimizzazione delle deep network con unità hard-threshold.

L'articolo discute il problema dell'ottimizzazione delle reti neurali con hard-threshold e propone una nuova soluzione con un insieme di euristiche/approssimazioni.


The robust and efficient recognition of visual relations in images is a hallmark of biological vision. Here, we argue that, despite recent progress in visual recognition, modern machine vision algorithms are severely limited in their ability to learn visual relations. Through controlled experiments, we demonstrate that visual-relation problems strain convolutional neural networks (CNNs). The networks eventually break altogether when rote memorization becomes impossible such as when the intra-class variability exceeds their capacity. We further show that another type of feedforward network, called a relational network (RN), which was shown to successfully solve seemingly difficult visual question answering (VQA) problems on the CLEVR datasets, suffers similar limitations. Motivated by the comparable success of biological vision, we argue that feedback mechanisms including working memory and attention are the key computational components underlying abstract visual reasoning.

Utilizzando una nuova challenge controllata di visual-relation, dimostriamo che i task same-different mettono a dura prova la capacità delle CNN; sosteniamo che le visual-relation possono essere risolte meglio utilizzando strategie attentivo-mnemoniche.

Dimostra che le reti neurali convoluzionali e relazionali non riescono a risolvere i problemi di visual-relation addestrando le reti su dati di visual-relation generati artificialmente.

Questo articolo esplora come le attuali CNN e le reti relazionali non riescano a riconoscere le visual-relation nelle immagini.


Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations. Previous work has shown that the tracker can be trained in a simulator via reinforcement learning and deployed in real-world scenarios. However, during training, such a method requires manually specifying the moving path of the target object to be tracked, which cannot ensure the tracker’s generalization on the unseen object moving patterns. To learn a robust tracker for VAT, in this paper, we propose a novel adversarial RL method which adopts an Asymmetric Dueling mechanism, referred to as AD-VAT. In AD-VAT, both the tracker and the target are approximated by end-to-end neural networks, and are trained via RL in a dueling/competitive manner: i.e., the tracker intends to lockup the target, while the target tries to escape from the tracker. They are asymmetric in that the target is aware of the tracker, but not vice versa. Specifically, besides its own observation, the target is fed with the tracker’s observation and action, and learns to predict the tracker’s reward as an auxiliary task. We show that such an asymmetric dueling mechanism produces a stronger target, which in turn induces a more robust tracker. To stabilize the training, we also propose a novel partial zero-sum reward for the tracker/target. The experimental results, in both 2D and 3D environments, demonstrate that the proposed method leads to a faster convergence in training and yields more robust tracking behaviors in different testing scenarios. For supplementary videos, see: https://www.youtube.com/playlist?list=PL9rZj4Mea7wOZkdajK1TsprRg8iUf51BS   The code is available at https://github.com/zfw1226/active_tracking_rl

Proponiamo AD-VAT, dove il tracker e l'oggetto target, visti come due agenti apprendibili, sono avversari e possono migliorarsi reciprocamente durante l'allenamento.

Questo lavoro mira ad affrontare il problema dell'active tracking visivo con un meccanismo di training in cui l'inseguitore e il bersaglio sono avversari reciproci

Questo articolo presenta un semplice task di deep RL multi-agente in cui un inseguitore in movimento cerca di seguire un bersaglio in movimento.

Propone una nuova funzione di reward "a somma parziale zero", che incoraggia solo la competizione inseguitore-obiettivo quando sono vicini e penalizza quando sono troppo lontani.


Identifying the hypernym relations that hold between words is a fundamental task in NLP. Word embedding methods have recently shown some capability to encode hypernymy. However, such methods tend not to explicitly encode the hypernym hierarchy that exists between words. In this paper, we propose a method to learn a hierarchical word embedding in a speciﬁc order to capture the hypernymy. To learn the word embeddings, the proposed method considers not only the hypernym relations that exists between words on a taxonomy, but also their contextual information in a large text corpus. The experimental results on a supervised hypernymy detection and a newly-proposed hierarchical path completion tasks show the ability of the proposed method to encode the hierarchy. Moreover, the proposed method outperforms previously proposed methods for learning word and hypernym-speciﬁc word embeddings on multiple benchmarks.

Abbiamo presentato un metodo per imparare congiuntamente un Hierarchical Word Embedding (HWE) usando un corpus e una tassonomia per identificare le relazioni di iperonimia tra le parole.

L'articolo presenta un metodo per imparare congiuntamente word embedding usando statistiche di co-occorrenza e incorporando informazioni gerarchiche dalle reti semantiche.

Questo articolo ha proposto un metodo di apprendimento congiunto degli iperonimi sia dal testo grezzo che dai dati della tassonomia supervisionata.

Questo articolo propone di aggiungere una misura di differenza di "inclusione distributiva" all'obiettivo GloVE allo scopo di rappresentare le relazioni di iperonimia.


While self-organizing principles have motivated much of early learning models, such principles have rarely been included in deep learning architectures. Indeed, from a supervised learning perspective it seems that topographic constraints are rather decremental to optimal performance. Here we study a network model that incorporates self-organizing maps into a supervised network and show how gradient learning results in a form of a self-organizing learning rule. Moreover, we show that such a model is robust in the sense of its application to a variety of  areas, which is believed to be a hallmark of biological learning systems.

Integrazione di auto-organizzazione e apprendimento supervisionato in una rete neurale gerarchica

L'articolo discute l'apprendimento in una rete neurale a tre layer, dove il layer intermedio è organizzato topograficamente e studia l'interazione tra l'apprendimento unsupervised e quello gerarchico supervisionato nel contesto biologico.

Una variante supervisionata della mappa auto-organizzante di Kohonen, ma dove il layer di output lineare è sostituito con errore quadratico da un layer softmax con cross-entropy.

Propone un modello che utilizza hidden neuron con funzioni di attivazione auto-organizzanti, i cui output alimentano un classificatore con funzione di output softmax.


Quantization of a neural network has an inherent problem called accumulated quantization error, which is the key obstacle towards ultra-low precision, e.g., 2- or 3-bit precision. To resolve this problem, we propose precision highway, which forms an end-to-end high-precision information flow while performing the ultra-low-precision computation. First, we describe how the precision highway reduce the accumulated quantization error in both convolutional and recurrent neural networks. We also provide the quantitative analysis of the benefit of precision highway and evaluate the overhead on the state-of-the-art hardware accelerator. In the experiments, our proposed method outperforms the best existing quantization methods while offering 3-bit weight/activation quantization with no accuracy loss and 2-bit quantization with a 2.45 % top-1 accuracy loss in ResNet-50. We also report that the proposed method significantly outperforms the existing method in the 2-bit quantization of an LSTM for language modeling.

Precision highway: un concetto generalizzato di flusso di informazioni ad alta precisione per la quantizzazione sub 4-bit

Indaga il problema della quantizzazione delle reti neurali impiegando una precision highway end-to-end per ridurre l'errore di quantizzazione accumulato e consentire una precisione ultra-bassa nelle deep neural network.

Questo articolo studia i metodi per migliorare le prestazioni delle reti neurali quantizzate

Questo articolo propone di mantenere un alto flusso di attivazione/gradiente in due tipi di strutture di reti, ResNet e LSTM.


The vast majority of natural sensory data is temporally redundant. For instance, video frames or audio samples which are sampled at nearby points in time tend to have similar values.   Typically, deep learning algorithms take no advantage of this redundancy to reduce computations.   This can be an obscene waste of energy.   We present a variant on backpropagation for neural networks in which computation scales with the rate of change of the data - not the rate at which we process the data.   We do this by implementing a form of Predictive Coding wherein neurons communicate a combination of their state, and their temporal change in state, and quantize this signal using Sigma-Delta modulation.   Intriguingly, this simple communication rule give rise to units that resemble biologically-inspired leaky integrate-and-fire neurons, and to a spike-timing-dependent weight-update similar to Spike-Timing Dependent Plasticity (STDP), a synaptic learning rule observed in the brain.   We demonstrate that on MNIST, on a temporal variant of MNIST, and on Youtube-BB, a dataset with videos in the wild, our algorithm performs about as well as a standard deep network trained with backpropagation, despite only communicating discrete values between layers.

Un algoritmo per addestrare reti neurali in modo efficiente su dati temporalmente ridondanti.

L'articolo descrive uno schema di codifica neurale per l'apprendimento basato sugli spike nelle deep neural network

Questo articolo presenta un metodo per l'apprendimento basato sugli spike che mira a ridurre il calcolo necessario durante l'apprendimento e il test quando si classificano dati temporalmente ridondanti.

Questo articolo applica una versione di codifica predittiva dello schema di encoding Sigma-Delta per ridurre il carico computazionale su una rete di deep learning, combinando i tre componenti in un modo non visto in precedenza.


Information bottleneck (IB) is a method for extracting information from one random variable X that is relevant for predicting another random variable Y. To do so, IB identifies an intermediate "bottleneck" variable T that has low mutual information I(X;T) and high mutual information I(Y;T). The "IB curve" characterizes the set of bottleneck variables that achieve maximal I(Y;T) for a given I(X;T), and is typically explored by maximizing the "IB Lagrangian", I(Y;T) - βI(X;T). In some cases, Y is a deterministic function of X, including many classification problems in supervised learning where the output class Y is a deterministic function of the input X. We demonstrate three caveats when using IB in any situation where Y is a deterministic function of X: (1) the IB curve cannot be recovered by maximizing the IB Lagrangian for different values of β; (2) there are "uninteresting" trivial solutions at all points of the IB curve; and (3) for multi-layer classifiers that achieve low prediction error, different layers cannot exhibit a strict trade-off between compression and prediction, contrary to a recent proposal. We also show that when Y is a small perturbation away from being a deterministic function of X, these three caveats arise in an approximate way. To address problem (1), we propose a functional that, unlike the IB Lagrangian, can recover the IB curve in all cases. We demonstrate the three caveats on the MNIST dataset.

L'information bottleneck si comporta in modi sorprendenti ogni volta che l'output è una funzione deterministica dell'input.

Sostiene che la maggior parte dei problemi reali di classificazione mostra una relazione deterministica tra le label di classe e gli input X ed esplora diversi problemi che derivano da tali patologie.

Esplora i problemi che sorgono quando si applicano i concetti di information bottleneck ai modelli deterministici di apprendimento supervisionato

Gli autori chiariscono diversi comportamenti controintuitivi del metodo dell'information bottleneck per l'apprendimento supervisionato di una regola deterministica.


We prove, under two sufficient conditions, that idealised models can have no adversarial examples. We discuss which idealised models satisfy our conditions, and show that idealised Bayesian neural networks (BNNs) satisfy these. We continue by studying near-idealised BNNs using HMC inference, demonstrating the theoretical ideas in practice. We experiment with HMC on synthetic data derived from MNIST for which we know the ground-truth image density, showing that near-perfect epistemic uncertainty correlates to density under image manifold, and that adversarial images lie off the manifold in our setting. This suggests why MC dropout, which can be seen as performing approximate inference, has been observed to be an effective defence against adversarial examples in practice; We highlight failure-cases of non-idealised BNNs relying on dropout, suggesting a new attack for dropout models and a new defence as well. Lastly, we demonstrate the defence on a cats-vs-dogs image classification task with a VGG13 variant.

Dimostriamo che le reti neurali bayesiane idealizzate non possono avere adversarial example, e diamo prove empiriche con BNN del mondo reale.

L'articolo studia l'adversarial robustness dei classificatori bayesiani e stabilisce due condizioni che dimostrano essere provatamente sufficienti per "modelli idealizzati" su "dataset idealizzati" per non avere adversarial example

Il paper propone una classe di classificatori discriminativi bayesiani che non hanno adversarial example.


Deep neural networks are susceptible to adversarial attacks. In computer vision, well-crafted perturbations to images can cause neural networks to make mistakes such as confusing a cat with a computer. Previous adversarial attacks have been designed to degrade performance of models or cause machine learning models to produce specific outputs chosen ahead of time by the attacker. We introduce attacks that instead reprogram the target model to perform a task chosen by the attacker without the attacker needing to specify or compute the desired output for each test-time input. This attack finds a single adversarial perturbation, that can be added to all test-time inputs to a machine learning model in order to cause the model to perform a task chosen by the adversary—even if the model was not trained to do this task. These perturbations can thus be considered a program for the new task. We demonstrate adversarial reprogramming on six ImageNet classification models, repurposing these models to perform a counting task, as well as classification tasks: classification of MNIST and CIFAR-10 examples presented as inputs to the ImageNet model.

Introduciamo la prima istanza di adversarial attack che riprogramma il modello bersaglio per eseguire un task scelto dall'attaccante senza che l'attaccante abbia bisogno di specificare o calcolare l'output desiderato per ogni input al test-time.

Gli autori presentano un nuovo schema di adversarial attack in cui una rete neurale viene riprogrammata per svolgere un task diverso da quello su cui è stata originariamente addestrata

Questo articolo ha proposto la "riprogrammazione adversarial" di reti neurali ben addestrate e fissate e mostra che la riprogrammazione adversarial è meno efficace sulle reti non addestrate.

L'articolo estende l'idea degli "adversarial attack" nell'apprendimento supervisionato delle NN a una completa riprogrammazione della soluzione di una rete addestrata.


As shown in recent research, deep neural networks can perfectly fit randomly labeled data, but with very poor accuracy on held out data. This phenomenon indicates that loss functions such as cross-entropy are not a reliable indicator of generalization. This leads to the crucial question of how generalization gap should be predicted from the training data and network parameters. In this paper, we propose such a measure, and conduct extensive empirical studies on how well it can predict the generalization gap. Our measure is based on the concept of margin distribution, which are the distances of training points to the decision boundary. We find that it is necessary to use margin distributions at multiple layers of a deep network. On the CIFAR-10 and the CIFAR-100 datasets, our proposed measure correlates very strongly with the generalization gap. In addition, we find the following other factors to be of importance: normalizing margin values for scale independence, using characterizations of margin distribution rather than just the margin (closest distance to decision boundary), and working in log space instead of linear space (effectively using a product of margins rather than a sum).  Our measure can be easily applied to feedforward deep networks with any architecture and may point towards new training loss functions that could enable better generalization.

Sviluppiamo un nuovo schema per prevedere il gap di generalizzazione nelle deep network con alta precisione.

Gli autori suggeriscono di usare un margine geometrico e una distribuzione dei margini layer-wise per prevedere il generalization gap.

Mostra empiricamente un'interessante connessione tra le statistiche di margine proposte e il gap di generalizzazione, che può essere utilizzato per fornire alcune intuizioni prescrittive verso la comprensione della generalizzazione nelle deep neural network.


We propose a new algorithm to learn a one-hidden-layer convolutional neural network where both the convolutional weights and the outputs weights are parameters to be learned. Our algorithm works for a general class of (potentially overlapping) patches, including commonly used structures for computer vision tasks. Our algorithm draws ideas from (1) isotonic regression for learning neural networks and (2) landscape analysis of non-convex matrix factorization problems. We believe these findings may inspire further development in designing provable algorithms for learning neural networks and other complex models. While our focus is theoretical, we also present experiments that illustrate our theoretical findings.

Proponiamo un algoritmo per recuperare in modo dimostrabile i parametri (pesi di convoluzione e di output) di una rete convoluzionale con patch sovrapposte.

Questo articolo studia l'apprendimento teorico delle reti neurali convoluzionali ad un hidden layer, risultando in un algoritmo di apprendimento e garanzie dimostrabili utilizzando l'algoritmo.

Questo articolo fornisce un nuovo algoritmo per l'apprendimento di una rete neurale a due layer che coinvolge un singolo filtro convoluzionale e un vettore di peso per diverse posizioni.


Since their invention, generative adversarial networks (GANs) have become a popular approach for learning to model a distribution of real (unlabeled) data. Convergence problems during training are overcome by Wasserstein GANs which minimize the distance between the model and the empirical distribution in terms of a different metric, but thereby introduce a Lipschitz constraint into the optimization problem. A simple way to enforce the Lipschitz constraint on the class of functions, which can be modeled by the neural network, is weight clipping. Augmenting the loss by a regularization term that penalizes the deviation of the gradient norm of the critic (as a function of the network's input) from one, was proposed as an alternative that improves training. We present theoretical arguments why using a weaker regularization term enforcing the Lipschitz constraint is preferable. These arguments are supported by experimental results on several data sets.

Un nuovo termine di regolarizzazione può migliorare l'allenamento delle Wasserstein GAN

L'articolo propone uno schema di regolarizzazione per Wasserstein GAN basato sul rilassamento dei vincoli sulla costante di Lipschitz pari ad 1.

L'articolo tratta della regolarizzazione/penalizzazione nel fitting delle GAN, quando si basa su una metrica L_1 di Wasserstein.


We introduce a new method for training GANs by applying the Wasserstein-2 metric proximal on the generators.   The approach is based on the gradient operator induced by optimal transport, which connects the geometry of sample space and parameter space in implicit deep generative models. From this theory, we obtain an easy-to-implement regularizer for the parameter updates. Our experiments demonstrate that this method improves the speed and stability in training GANs in terms of wall-clock time and Fr\'echet Inception Distance (FID) learning curves.

Proponiamo il metodo prossimale di Wasserstein per il training delle GAN.

Propone una nuova procedura per le GAN che tiene conto dei punti generati nell'iterazione precedente e aggiorna il generatore per l volte.

Considera l'apprendimento a gradiente naturale nel learning delle GAN, dove viene impiegata la struttura Riemanniana indotta dalla distanza Wasserstein-2.

L'articolo intende utilizzare il gradiente naturale indotto dalla distanza Wasserstein-2 per addestrare il generatore nelle GAN e gli autori propongono l'operatore prossimale di Wasserstein come regolarizzazione.


Influence diagrams provide a modeling and inference framework for sequential decision problems, representing the probabilistic knowledge by a Bayesian network and the preferences of an agent by utility functions over the random variables and decision variables.  MDPs and POMDPS, widely used for planning under uncertainty can also be represented by influence diagrams.  The time and space complexity of computing the maximum expected utility (MEU) and its maximizing policy is exponential in the induced width of the underlying graphical model, which is often prohibitively large due to the growth of the information set under the sequence of decisions.  In this paper, we develop a weighted mini-bucket approach for bounding the MEU.  These bounds can be used as a stand-alone approximation that can be improved as a function of a controlling i-bound parameter . They can also be used as heuristic  functions to guide search, especially for planning   such as MDPs and POMDPs.  We evaluate the scheme empirically against state-of-the-art, thus illustrating its potential.

Questo articolo introduce una funzione euristica basata sull'eliminazione per il processo decisionale sequenziale, adatta a guidare algoritmi di ricerca AND/OR per risolvere diagrammi di influenza.

Generalizza l'euristica di inferenza dei minibucket ai diagrammi di influenza.


Probabilistic Neural Networks deal with various sources of stochasticity: input noise, dropout, stochastic neurons, parameter uncertainties modeled as random variables, etc.  In this paper we revisit a feed-forward propagation approach that allows one to estimate for each neuron its mean and variance w.r.t. all mentioned sources of stochasticity. In contrast, standard NNs propagate only point estimates, discarding the uncertainty.  Methods propagating also the variance have been proposed by several authors in different context. The view presented here attempts to clarify the assumptions and derivation behind such methods, relate them to classical NNs and broaden their scope of applicability.  The main technical contributions are new approximations for the distributions of argmax and max-related transforms, which allow for fully analytic uncertainty propagation in networks with softmax and max-pooling layers as well as leaky ReLU activations.  We evaluate the accuracy of the approximation and suggest a simple calibration. Applying the method to networks with dropout allows for faster training and gives improved test likelihoods without the need of sampling.

Approssimazione della media e della varianza dell'output di una NN nel caso di input rumorosi / dropout / parametri incerti. Approssimazioni analitiche per i layer argmax, softmax e max.

Gli autori si concentrano sul problema della propagazione dell'incertezza nelle DNN

Questo articolo rivisita la propagazione feed-forward della media e della varianza nei neuroni, affrontando il problema della propagazione dell'incertezza attraverso layer di max-pooling e softmax.


Generative Adversarial Networks are one of the leading tools in generative modeling, image editing and content creation.   However, they are hard to train as they require a delicate balancing act between two deep networks fighting a never ending duel. Some of the most promising adversarial models today minimize a Wasserstein objective. It is smoother and more stable to optimize. In this paper, we show that the Wasserstein distance is just one out of a large family of objective functions that yield these properties. By making the discriminator of a GAN robust to adversarial attacks we can turn any GAN objective into a smooth and stable loss. We experimentally show that any GAN objective, including Wasserstein GANs, benefit from adversarial robustness both quantitatively and qualitatively. The training additionally becomes more robust to suboptimal choices of hyperparameters, model architectures, or objective functions.

Un discriminatore che non è facilmente ingannato dall'adversarial example rende il training GAN più robusto e porta ad un obiettivo più smooth.

Questo articolo propone un nuovo modo per stabilizzare il processo di training delle GAN regolarizzando il discriminatore per essere robusto agli adversarial example.

L'articolo propone un modo sistematico di addestrare le GAN con termini di regolarizzazione della robustezza, permettendo un training più fluido.

Presenta l'idea che rendendo un discriminatore robusto alle adversarial perturbation l'obiettivo GAN può essere reso smooth, il che si traduce in risultati migliori sia visivamente che in termini di FID.


We propose a method to learn stochastic activation functions for use in probabilistic neural networks.  First, we develop a framework to embed stochastic activation functions based on Gaussian processes in probabilistic neural networks.  Second, we analytically derive expressions for the propagation of means and covariances in such a network, thus allowing for an efficient implementation and training without the need for sampling.  Third, we show how to apply variational Bayesian inference to regularize and efficiently train this model.  The resulting model can deal with uncertain inputs and implicitly provides an estimate of the confidence of its predictions.  Like a conventional neural network it can scale to datasets of arbitrary size and be extended with convolutional and recurrent connections, if desired.

Modelliamo la funzione di attivazione di ogni neurone come un processo gaussiano e la impariamo insieme al peso con l'inferenza variazionale.

Propone di mettere prior di processo gaussiano sulla forma funzionale di ogni funzione di attivazione nella rete neurale per imparare la forma delle funzioni di attivazione.


Recent results from linear algebra stating that any matrix can be decomposed into products of diagonal and circulant matrices has lead to the design of compact deep neural network architectures that perform well in practice. In this paper, we bridge the gap between these good empirical results   and the theoretical approximation capabilities of Deep diagonal-circulant ReLU networks. More precisely, we first demonstrate  that a Deep diagonal-circulant ReLU networks of  bounded width and small depth can approximate a deep ReLU network in which the dense matrices are  of low rank. Based on this result, we provide new bounds on the expressive power and universal approximativeness of this type of networks. We support our experimental results with thorough experiments on a large, real world video classification problem.

Forniamo uno studio teorico delle proprietà delle reti ReLU circolanti-diagonali profonde e dimostriamo che sono approssimatori universali a larghezza limitata.

L'articolo propone di utilizzare matrici circolanti e diagonali per accelerare il calcolo e ridurre i requisiti di memoria nelle reti neurali.

Questo articolo dimostra che le reti ReLU diagonali-circolari di larghezza limitata (DC-ReLU) sono approssimatori universali.


Camera drones, a rapidly emerging technology, offer people the ability to remotely inspect an environment with a high degree of mobility and agility. However, manual remote piloting of a drone is prone to errors. In contrast, autopilot systems can require a significant degree of environmental knowledge and are not necessarily designed to support flexible visual inspections. Inspired by camera manipulation techniques in interactive graphics, we designed StarHopper, a novel touch screen interface for efficient object-centric camera drone navigation, in which a user directly specifies the navigation of a drone camera relative to a specified object of interest. The system relies on minimal environmental information and combines both manual and automated control mechanisms to give users the freedom to remotely explore an environment with efficiency and accuracy. A lab study shows that StarHopper offers an efficiency gain of 35.4% over manual piloting, complimented by an overall user preference towards our object-centric navigation system.

StarHopper è una nuova interfaccia touch screen per la navigazione object-centric, efficiente e flessibile dei droni con telecamera

Gli autori delineano una nuova interfaccia di controllo del drone chiamata StarHopper, combinando il pilotaggio automatico e manuale in una nuova interfaccia di navigazione ibrida che si sbarazza del presupposto che l'oggetto di destinazione è già nel FOV del drone utilizzando una telecamera supplementare sopra la testa.

Questo articolo presenta StarHopper, un sistema per la navigazione semiautomatica dei droni nel contesto dell'ispezione remota.

Presenta StarHopper, un'applicazione che utilizza tecniche di computer vision con input tattile per supportare il pilotaggio dei droni con un approccio centrato sull'oggetto.


Recurrent neural networks (RNN), convolutional neural networks (CNN) and self-attention networks (SAN) are commonly used to produce context-aware representations. RNN can capture long-range dependency but is hard to parallelize and not time-efficient. CNN focuses on local dependency but does not perform well on some tasks. SAN can model both such dependencies via highly parallelizable computation, but memory requirement grows rapidly in line with sequence length. In this paper, we propose a model, called "bi-directional block self-attention network (Bi-BloSAN)", for RNN/CNN-free sequence encoding. It requires as little memory as RNN but with all the merits of SAN. Bi-BloSAN splits the entire sequence into blocks, and applies an intra-block SAN to each block for modeling local context, then applies an inter-block SAN to the outputs for all blocks to capture long-range dependency. Thus, each SAN only needs to process a short sequence, and only a small amount of memory is required. Additionally, we use feature-level attention to handle the variation of contexts around the same word, and use forward/backward masks to encode temporal order information. On nine benchmark datasets for different NLP tasks, Bi-BloSAN achieves or improves upon state-of-the-art accuracy, and shows better efficiency-memory trade-off than existing RNN/CNN/SAN.

Una rete di self-attention per la codifica di sequenze senza RNN/CNN con un piccolo consumo di memoria, calcolo altamente parallelizzabile e prestazioni allo stato dell'arte su diversi task NLP

Propone di applicare self-attention a due livelli per limitare la richiesta di memoria nei modelli basati sull'attention con un impatto trascurabile sulla velocità.

Questo articolo introduce il modello di self-attention bidirezionale a blocchi come un encoder generale per vari task di modellazione di sequenze in NLP


End-to-end neural models have made significant progress in question answering, however recent studies show that these models implicitly assume that the answer and evidence appear close together in a single document. In this work, we propose the Coarse-grain Fine-grain Coattention Network (CFC), a new question answering model that combines information from evidence across multiple documents. The CFC consists of a coarse-grain module that interprets documents with respect to the query then finds a relevant answer, and a fine-grain module which scores each candidate answer by comparing its occurrences across all of the documents with the query. We design these modules using hierarchies of coattention and self-attention, which learn to emphasize different parts of the input. On the Qangaroo WikiHop multi-evidence question answering task, the CFC obtains a new state-of-the-art result of 70.6% on the blind test set, outperforming the previous best by 3% accuracy despite not using pretrained contextual encoders.

Un nuovo modello allo stato dell'arte per la question answering multi-evidenza utilizzando l'attention gerarchica a grana grossa e fine.

Propone un metodo per il QA multi-hop basato su due moduli separati (moduli coarse-grained e fine-grained).

Questo articolo propone un'interessante architettura di rete di co-attention coarse-grained e fine-grained per affrontare question answering multi-evidence

Si concentra su QA multi-choice e propone un framework di punteggio coarse-to-fine.


Generative adversarial networks (GANs) are an expressive class of neural generative models with tremendous success in modeling high-dimensional continuous measures. In this paper, we present a scalable method for unbalanced optimal transport (OT) based on the generative-adversarial framework. We formulate unbalanced OT as a problem of simultaneously learning a transport map and a scaling factor that push a source measure to a target measure in a cost-optimal manner. We provide theoretical justification for this formulation, showing that it is closely related to an existing static formulation by Liero et al. (2018). We then propose an algorithm for solving this problem based on stochastic alternating gradient updates, similar in practice to GANs, and perform numerical experiments demonstrating how this methodology can be applied to population modeling.

Proponiamo una nuova metodologia per il trasporto ottimale sbilanciato utilizzando generative adversarial network.

Gli autori considerano il problema del trasporto ottimale sbilanciato tra due misure con massa totale diversa usando un algoritmo stocastico min-max e uno scaling locale

Gli autori propongono un approccio per stimare il trasporto ottimale sbilanciato tra misure campionate che scala bene nella dimensione e nel numero di sample.

L'articolo introduce una formulazione statica per il trasporto ottimale sbilanciato imparando simultaneamente una mappa di trasporto T e un fattore di scala xi.


Extracting saliency maps, which indicate parts of the image important to classification, requires many tricks to achieve satisfactory performance when using classifier-dependent methods. Instead, we propose classifier-agnostic saliency map extraction, which finds all parts of the image that any classifier could use, not just one given in advance. We observe that the proposed approach extracts higher quality saliency maps and outperforms existing weakly-supervised localization techniques, setting the new state of the art result on the ImageNet dataset.

Proponiamo un nuovo metodo di estrazione delle mappe di saliency che risulta nell'estrazione di mappe di qualità superiore.

Propone un metodo indipendente dal classificatore per l'estrazione delle mappe di saliency.

Questo articolo introduce un nuovo estrattore di mappe di saliency che sembra migliorare i risultati allo stato dell'arte.

Gli autori sostengono che quando una mappa di saliency estratta dipende direttamente da un modello, allora potrebbe non essere utile per un classificatore diverso, e suggerisce uno schema per approssimare la soluzione.


Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs). However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images. To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration. The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances. To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances. We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances. Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases. Code and results are available in https://github.com/sangwoomo/instagan

Proponiamo un nuovo metodo per incorporare l'insieme degli attributi di istanza per la traduzione image-to-image.

Questo articolo propone un metodo - InstaGAN - che si basa su CycleGAN prendendo in considerazione le informazioni dell'istanza sotto forma di maschere di segmentazione per istanza, con risultati che si confrontano favorevolmente con CycleGAN e altre baseline.

Propone di aggiungere maschere di segmentazione instance-aware per il problema della traduzione image-to-image non abbinata.


Deep neural networks (DNNs) generalize remarkably well without explicit regularization even in the strongly over-parametrized regime  where classical learning theory would instead predict that they would severely overfit.   While many proposals for some kind of implicit regularization have been made to rationalise this success, there is no consensus for the fundamental reason why DNNs do not strongly overfit.   In this paper, we provide a new explanation. By applying a very general probability-complexity bound recently derived from  algorithmic information theory (AIT), we argue that the parameter-function map of many DNNs should be exponentially biased towards simple functions. We then provide clear evidence for this strong simplicity bias in a model DNN for Boolean functions, as well as in much larger fully connected and convolutional networks trained on CIFAR10 and MNIST.  As the target functions in many real problems are expected to be highly structured, this intrinsic simplicity bias helps explain why deep networks generalize well on real world problems.  This picture also facilitates a novel PAC-Bayes approach where the prior is taken over the DNN input-output function space, rather than  the more conventional prior over parameter space.   If we assume that the training algorithm samples parameters close to uniformly within the zero-error region then the PAC-Bayes theorem can be used to guarantee good expected generalization for target functions producing high-likelihood training sets.   By exploiting recently discovered connections between DNNs and Gaussian processes to estimate the marginal likelihood,  we produce relatively tight generalization PAC-Bayes error bounds which correlate well with the true error on realistic datasets such as MNIST and CIFAR10 and for architectures including convolutional and fully connected networks.

La mappa parametro-funzione delle deep network è enormemente biased; questo può spiegare perché generalizzano. Usiamo PAC-Bayes e processi gaussiani per ottenere bound non banali.

L'articolo studia le capacità di generalizzazione delle deep neural network, con l'aiuto della teoria dell'apprendimento PAC-Bayesiana e delle intuizioni empiricamente sostenute.

Questo articolo propone una spiegazione dei comportamenti di generalizzazione delle grandi reti neurali iper-parametrizzate sostenendo che la mappa parametro-funzione nelle reti neurali è biased verso funzioni "semplici" e il comportamento di generalizzazione sarà buono se il concetto target è anche "semplice".


We establish a theoretical link between evolutionary algorithms and variational parameter optimization of probabilistic generative models with binary hidden variables.  While the novel approach is independent of the actual generative model, here we use two such models to investigate its applicability and scalability: a noisy-OR Bayes Net (as a standard example of binary data) and Binary Sparse Coding (as a model for continuous data).   Learning of probabilistic generative models is first formulated as approximate maximum likelihood optimization using variational expectation maximization (EM).  We choose truncated posteriors as variational distributions in which discrete latent states serve as variational parameters. In the variational E-step,  the latent states are then   optimized according to a tractable free-energy objective . Given a data point, we can show that evolutionary algorithms can be used for the variational optimization loop by (A)~considering the bit-vectors of the latent states as genomes of individuals, and by (B)~defining the fitness of the  individuals as the (log) joint probabilities given by the used generative model.   As a proof of concept, we apply the novel evolutionary EM approach to the optimization of the parameters of noisy-OR Bayes nets and binary sparse coding on artificial and real data (natural image patches). Using point mutations and single-point cross-over for the evolutionary algorithm, we find that scalable variational EM algorithms are obtained which efficiently improve the data likelihood. In general we believe that, with the link established here, standard as well as recent results in the field of evolutionary optimization can be leveraged to address the difficult problem of parameter optimization in generative models.

Presentiamo Evolutionary EM come un nuovo algoritmo per il training unsupervised di modelli generativi con variabili latenti binarie che collega intimamente l'EM variazionale con l'ottimizzazione evolutiva

L'articolo presenta una combinazione di calcolo evolutivo ed EM variazionale per modelli con variabili latenti binarie rappresentate tramite un'approssimazione particle-based

L'articolo fa un tentativo di integrare strettamente gli algoritmi di training di expectation-maximization con gli algoritmi evolutivi.


While deep neural networks have achieved groundbreaking prediction results in many tasks, there is a class of data where existing architectures are not optimal -- sequences of probability distributions. Performing forward prediction on sequences of distributions has many important applications. However, there are two main challenges in designing a network model for this task. First, neural networks are unable to encode distributions compactly as each node encodes just a real value. A recent work of Distribution Regression Network (DRN) solved this problem with a novel network that encodes an entire distribution in a single node, resulting in improved accuracies while using much fewer parameters than neural networks. However, despite its compact distribution representation, DRN does not address the second challenge, which is the need to model time dependencies in a sequence of distributions. In this paper, we propose our Recurrent Distribution Regression Network (RDRN) which adopts a recurrent architecture for DRN. The combination of compact distribution representation and shared weights architecture across time steps makes RDRN suitable for modeling the time dependencies in a distribution sequence. Compared to neural networks and DRN, RDRN achieves the best prediction performance while keeping the network compact.

Proponiamo un modello efficiente di rete ricorrente per la forward prediction su distribuzioni variabili nel tempo.

Questo articolo propone un metodo per creare reti neurali che mappano le distribuzioni storiche sulle distribuzioni e applica il metodo a diversi task di predizione della distribuzione.

Propone una Recurent Distribution Regression Network che utilizza un'architettura ricorrente su un modello precedente di Distribution Regression Network.

Questo articolo riguarda la regressione su distribuzioni di probabilità studiando distribuzioni variabili nel tempo in una rete neurale ricorrente


We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of computationally intensive matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, we address several key challenges of spectral-based graph neural networks simultaneously, and make our model readily applicable to inductive as well as transductive problems. Our GAT models have achieved or matched state-of-the-art results across four established transductive and inductive graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, as well as a protein-protein interaction dataset (wherein test graphs remain unseen during training).

Un nuovo approccio all'elaborazione dei dati strutturati a grafo da parte delle reti neurali, sfruttando l'attention sul neighborhood di un nodo. Raggiunge risultati allo stato dell'arte su task di citation network trasduttiva e un task di interazione proteina-proteina induttiva.

Questo articolo propone un nuovo metodo per classificare i nodi di un grafo, che può essere utilizzato in scenari semi-supervisionati e su un grafo completamente nuovo.

L'articolo introduce un'architettura di rete neurale per operare su dati strutturati a grafo chiamata Graph Attention Network.

Fornisce una discussione corretta e quasi completa dello stato dell'arte degli approcci all'apprendimento di rappresentazioni vettoriali per i nodi di un grafo.


While bigger and deeper neural network architectures continue to advance the state-of-the-art for many computer vision tasks, real-world adoption of these networks is impeded by hardware and speed constraints. Conventional model compression methods attempt to address this problem by modifying the architecture manually or using pre-defined heuristics. Since the space of all reduced architectures is very large, modifying the architecture of a deep neural network in this way is a difficult task. In this paper, we tackle this issue by introducing a principled method for learning reduced network architectures in a data-driven way using reinforcement learning. Our approach takes a larger 'teacher' network as input and outputs a compressed 'student' network derived from the 'teacher' network. In the first stage of our method, a recurrent policy network aggressively removes layers from the large 'teacher' model. In the second stage, another  recurrent policy network carefully reduces the size of each remaining layer. The resulting network is then evaluated to obtain a reward -- a score based on the accuracy and compression of the network. Our approach uses this reward signal with policy gradients to train the policies to find a locally optimal student network. Our experiments show that we can achieve compression rates of more than 10x for models such as ResNet-34 while maintaining similar performance to the input 'teacher' network. We also present a valuable transfer learning result which shows that policies which are pre-trained on smaller 'teacher' networks can be used to rapidly speed up training on larger 'teacher' networks.

Un nuovo approccio basato sul reinforcement learning per comprimere le deep neural network con la knowledge distillation

Questo articolo propone di usare il reinforcement learning invece di euristiche predefinite per determinare la struttura del modello compresso nel processo di knowledge distillation

Introduce un metodo per la compressione da rete a rete, che utilizza i policy gradient per ottimizzare due policy che comprimono un teacher model forte in uno student model forte ma più piccolo.


Recent advances in conditional image generation tasks, such as image-to-image translation and image inpainting, are largely accounted to the success of conditional GAN models, which are often optimized by the joint use of the GAN loss with the reconstruction loss. However, we reveal that this training recipe shared by almost all existing methods causes one critical side effect: lack of diversity in output samples. In order to accomplish both training stability and multimodal output generation, we propose novel training schemes with a new set of losses named moment reconstruction losses that simply replace the reconstruction loss. We show that our approach is applicable to any conditional generation tasks by performing thorough experiments on image-to-image translation, super-resolution and image inpainting using Cityscapes and CelebA dataset. Quantitative evaluations also confirm that our methods achieve a great diversity in outputs while retaining or even improving the visual fidelity of generated samples.

Dimostriamo che il mode collapse nelle GAN condizionali è in gran parte attribuito a una mancata corrispondenza tra la loss di ricostruzione e la loss GAN e introduciamo una serie di nuove funzioni di loss come alternative alla loss di ricostruzione.

L'articolo propone una modifica al tradizionale obiettivo GAN condizionale per promuovere una generazione di immagini diverse e multimodali.

Questo articolo propone un'alternativa agli errori L1/L2 che sono usati per aumentare le loss adversarial quando si addestrano le GAN condizionali.


Generative models are important tools to capture and investigate the properties of complex empirical data. Recent developments such as Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) use two very similar, but \textit{reverse}, deep convolutional architectures, one to generate and one to extract information from data. Does learning the parameters of both architectures obey the same rules? We exploit the causality principle of independence of mechanisms to quantify how the weights of successive layers adapt to each other. Using the recently introduced Spectral Independence Criterion, we quantify the dependencies between the kernels of successive convolutional layers and show that those are more independent for the generative process than for information extraction, in line with results from the field of causal inference. In addition, our experiments on generation of human faces suggest that more independence between successive layers of generators results in improved performance of these architectures.

Usiamo l'inferenza causale per caratterizzare l'architettura dei modelli generativi

Questo articolo esamina la natura dei filtri convoluzionali nell'encoder e nel decoder di un VAE, e nel generatore e nel discriminatore di una GAN.

Questo lavoro sfrutta il principio di causalità per quantificare come i pesi dei layer successivi si adattino gli uni agli altri.


Many deep reinforcement learning approaches use graphical state representations,  this means visually distinct games that share the same underlying structure cannot  effectively share knowledge. This paper outlines a new approach for learning  underlying game state embeddings irrespective of the visual rendering of the game  state. We utilise approaches from multi-task learning and domain adaption in  order to place visually distinct game states on a shared embedding manifold. We  present our results in the context of deep reinforcement learning agents.

Un approccio all'apprendimento di uno spazio di embedding condiviso tra giochi visivamente distinti.

Un nuovo approccio per l'apprendimento della struttura di base dei giochi visivamente distinti che combina layer convoluzionali per l'elaborazione delle immagini di input, Asynchronous Advantage Actor Critic per deep reinforcement learning e l'approccio adversarial per forzare la rappresentazione di embedding ad essere indipendente dalla rappresentazione visiva dei giochi

Introduce un metodo per imparare una policy su giochi visivamente distinti adattando il deep reinforcement learning.

Questo articolo discute un'architettura di un agente che usa una rappresentazione condivisa per addestrarsi su task multipli con diverse statistiche visive a livello di sprite


We study discrete time dynamical systems governed by the state equation $h_{t+1}=ϕ(Ah_t+Bu_t)$. Here A,B are weight matrices, ϕ is an activation function, and $u_t$ is the input data. This relation is the backbone of recurrent neural networks (e.g. LSTMs) which have broad applications in sequential learning tasks. We utilize stochastic gradient descent to learn the weight matrices from a finite input/state trajectory $(u_t,h_t)_{t=0}^N$. We prove that SGD estimate linearly converges to the ground truth weights while using near-optimal sample size. Our results apply to increasing activations whose derivatives are bounded away from zero. The analysis is based on i) an SGD convergence result with nonlinear activations and ii) careful statistical characterization of the state vector. Numerical experiments verify the fast convergence of SGD on ReLU and leaky ReLU in consistence with our theory.

Studiamo l'equazione di stato di una rete neurale ricorrente. Mostriamo che SGD può apprendere in modo efficiente la dinamica sconosciuta da poche osservazioni di input/output sotto opportune ipotesi.

L'articolo studia sistemi dinamici a tempo discreto con un'equazione di stato non lineare, dimostrando che l'esecuzione di SGD su una traiettoria di lunghezza fissa dà una convergenza logaritmica.

Questo lavoro considera il problema dell'apprendimento di un sistema dinamico non lineare in cui l'output è uguale allo stato.

Questo articolo studia la capacità di SGD di apprendere le dinamiche di un sistema lineare e di un'attivazione non lineare.


Although deep neural networks show their extraordinary power in various tasks, they are not feasible for deploying such large models on embedded systems due to high computational cost and storage space limitation. The recent work knowledge distillation (KD) aims at transferring model knowledge from a well-trained teacher model to a small and fast student model which can significantly help extending the usage of large deep neural networks on portable platform. In this paper, we show that, by properly defining the neuron manifold of deep neuron network (DNN), we can significantly improve the performance of student DNN networks through approximating neuron manifold of powerful teacher network. To make this, we propose several novel methods for learning neuron manifold from DNN model. Empowered with neuron manifold knowledge, our experiments show the great improvement across a variety of DNN architectures and training data. Compared with other KD methods, our Neuron Manifold Transfer (NMT) has best transfer ability of the learned features.

Un nuovo metodo di knowledge distillation per transfer learning

Il lavoro introduce un metodo di knowledge distillation utilizzando il concetto di manifold neuronale proposto.

Propone un metodo di knowledge distillation in cui il manifold neurale è considerato come la conoscenza trasferita.


We present a simple and general method to train a single neural network executable at different widths (number of channels in a layer), permitting instant and adaptive accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve similar (and in many cases better) ImageNet classification accuracy than individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths respectively. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. Lastly we visualize and discuss the learned features of slimmable networks. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks

Presentiamo un metodo semplice e generale per addestrare una singola rete neurale eseguibile a diverse larghezze (numero di canali in un layer), permettendo trade-off di precisione-efficienza istantanei e adattivi al runtime.

L'articolo propone un'idea di combinare insieme modelli di dimensioni diverse in una rete condivisa, migliorando notevolmente le prestazioni per il rilevamento

Questo documento allena una singola rete eseguibile a diverse larghezze.


Measuring visual (dis)similarity between two or more instances within a data distribution is a fundamental task in many applications, specially in image retrieval. Theoretically, non-metric distances are able to generate a more complex and accurate similarity model than metric distances, provided that the non-linear data distribution is precisely captured by the similarity model. In this work, we analyze a simple approach for deep learning networks to be used as an approximation of non-metric similarity functions and we study how these models generalize across different image retrieval datasets.

Similarity network per imparare una stima di somiglianza visiva non metrica tra una coppia di immagini

Gli autori propongono l'apprendimento di misure di somiglianza per la somiglianza visiva e ottengono con questo un miglioramento in dataset molto noti di Oxford e Parigi per image retrieval.

L'articolo sostiene che è più adatto usare le distanze non metriche invece di quelle metriche.


Training a model to perform a task typically requires a large amount of data from the domains in which the task will be applied.  However, it is often the case that data are abundant in some domains but scarce in others. Domain adaptation deals with the challenge of adapting a model trained from a data-rich source domain to perform well in a data-poor target domain. In general, this requires learning plausible mappings between domains. CycleGAN is a powerful framework that efficiently learns to map inputs from one domain to another using adversarial training and a cycle-consistency constraint. However, the conventional approach of enforcing cycle-consistency via reconstruction may be overly restrictive in cases where one or more domains have limited training data. In this paper, we propose an augmented cyclic adversarial learning model that enforces the cycle-consistency constraint via an external task specific model, which encourages the preservation of task-relevant content as opposed to exact reconstruction. We explore digit classification in a low-resource setting in supervised, semi and unsupervised situation, as well as high resource unsupervised. In low-resource supervised setting, the results show that our approach improves absolute performance by 14% and 4% when adapting SVHN to MNIST and vice versa, respectively, which outperforms unsupervised domain adaptation methods that require high-resource unlabeled target domain.   Moreover, using only few unsupervised target data, our approach can still outperforms many high-resource unsupervised models. Our model also outperforms on USPS to MNIST and synthetic digit to SVHN for high resource unsupervised adaptation. In speech domains, we similarly adopt a speech recognition model from each domain as the task specific model. Our approach improves absolute performance of speech recognition by 2% for female speakers in the TIMIT dataset, where the majority of training samples are from male voices.

Un nuovo adversarial learning ciclico arricchito con un modello di task ausiliario che migliora le prestazioni di domain adaptation in setting supervisionati e non supervisionati con poche risorse

Propone un'estensione dei metodi di adattamento cycle-consistent adversarial al fine di affrontare la domain adaptation quando i dati target supervisionati disponibili sono limitati.

Questo articolo introduce un approccio di domain adaptation basato sull'idea di GAN ciclica e propone due diversi algoritmi.


Nodes residing in different parts of a graph can have similar structural roles within their local network topology. The identification of such roles provides key insight into the organization of networks and can also be used to inform machine learning on graphs. However, learning structural representations of nodes is a challenging unsupervised-learning task, which typically involves manually specifying and tailoring topological features for each node. Here we develop GraphWave, a method that represents each node’s local network neighborhood via a low-dimensional embedding by leveraging spectral graph wavelet diffusion patterns. We prove that nodes with similar local network neighborhoods will have similar GraphWave embeddings even though these nodes may reside in very different parts of the network. Our method scales linearly with the number of edges and does not require any hand-tailoring of topological features. We evaluate performance on both synthetic and real-world datasets, obtaining improvements of up to 71% over state-of-the-art baselines.

Sviluppiamo un metodo per l'apprendimento delle firme strutturali nelle reti basato sulla diffusione delle wavelet spettrali del grafo.

Si utilizzano i modelli di diffusione di wavelet spettrali del grafo del neighborhood locale di un nodo per fare l'embedding del nodo in uno spazio bidimensionale

L'articolo ha derivato un modo per confrontare i nodi nel grafo basato sull'analisi wavelet del laplaciano del grafo.


Driving simulators play an important role in vehicle research. However, existing virtual reality simulators do not give users a true sense of presence. UniNet is our driving simulator, designed to allow users to interact with and visualize simulated traffic in mixed reality. It is powered by SUMO and Unity. UniNet's modular architecture allows us to investigate interdisciplinary research topics such as vehicular ad-hoc networks, human-computer interaction, and traffic management. We accomplish this by giving users the ability to observe and interact with simulated traffic in a high fidelity driving simulator. We present a user study that subjectively measures user's sense of presence in UniNet. Our findings suggest that our novel mixed reality system does increase this sensation.

Un simulatore di guida in realtà mista con telecamere stereo e VR passthrough valutato in uno studio utente con 24 partecipanti.

Propone un sistema complicato per la simulazione di guida.

Questo articolo presenta una configurazione di simulatore di guida in realtà mista per migliorare la sensazione di presenza

Propone un simulatore di guida in realtà mista che incorpora la generazione di traffico e sostiene una "presenza" migliorata grazie a un sistema MR.


We consider the problem of improving kernel approximation via feature maps. These maps arise as Monte Carlo approximation to integral representations of kernel functions and scale up kernel methods for larger datasets. We propose to use more efficient numerical integration technique to obtain better estimates of the integrals compared to the state-of-the-art methods. Our approach allows to use information about the integrand to enhance approximation and facilitates fast computations. We derive the convergence behavior and conduct an extensive empirical study that supports our hypothesis.

Regole di quadratura per l'approssimazione del kernel.

L'articolo propone di migliorare l'approssimazione del kernel delle feature casuali utilizzando regole di quadratura come le regole sferico-radiali stocastiche.

Gli autori propongono una nuova versione dell'approccio random feature map per risolvere approssimativamente problemi di kernel su larga scala.

Questo articolo mostra che le tecniche dovute a Genz & Monahan (1998) possono essere utilizzate per ottenere un basso errore di approssimazione del kernel nel framework della feature di Fourier casuali, un nuovo modo di applicare le regole di quadratura per migliorare l'approssimazione del kernel.


Human world knowledge is both structured and flexible. When people see an object, they represent it not as a pixel array but as a meaningful arrangement of semantic parts. Moreover, when people refer to an object, they provide descriptions that are not merely true but also relevant in the current context. Here, we combine these two observations in order to learn fine-grained correspondences between language and contextually relevant geometric properties of 3D objects. To do this, we employed an interactive communication task with human participants to construct a large dataset containing natural utterances referring to 3D objects from ShapeNet in a wide variety of contexts. Using this dataset, we developed neural listener and speaker models with strong capacity for generalization. By performing targeted lesions of visual and linguistic input, we discovered that the neural listener depends heavily on part-related words and associates these words correctly with the corresponding geometric properties of objects, suggesting that it has learned task-relevant structure linking the two input modalities. We further show that a neural speaker that is `listener-aware' --- that plans its utterances according to how an imagined listener would interpret its words in context --- produces more discriminative referring expressions than an `listener-unaware' speaker, as measured by human performance in identifying the correct object.

Come costruire parlanti/ascoltatori neurali che apprendono feature fine-grained di oggetti 3D, dal linguaggio referenziale.

Gli autori forniscono uno studio sull'apprendimento dei riferimenti agli oggetti 3D, raccogliendo un dataset di espressioni referenziali e addestrando diversi modelli sperimentando una serie di scelte di architetture


Object-based factorizations provide a useful level of abstraction for interacting with the world. Building explicit object representations, however, often requires supervisory signals that are difficult to obtain in practice. We present a paradigm for learning object-centric representations for physical scene understanding without direct supervision of object properties. Our model, Object-Oriented Prediction and Planning (O2P2), jointly learns a perception function to map from image observations to object representations, a pairwise physics interaction function to predict the time evolution of a collection of objects, and a rendering function to map objects back to pixels. For evaluation, we consider not only the accuracy of the physical predictions of the model, but also its utility for downstream tasks that require an actionable representation of intuitive physics. After training our model on an image prediction task, we can use its learned representations to build block towers more complicated than those observed during training.

Presentiamo un framework per l'apprendimento di rappresentazioni centrate sull'oggetto adatte alla pianificazione in task che richiedono una comprensione della fisica.

L'articolo presenta una piattaforma per prevedere immagini di oggetti che interagiscono tra loro sotto l'effetto di forze gravitazionali.

L'articolo presenta un metodo che impara a riprodurre "block tower" da un'immagine data.

Propone un metodo che impara a ragionare sull'interazione fisica di diversi oggetti senza supervisione delle proprietà degli oggetti.


We study the error landscape of deep linear and nonlinear neural networks with the squared error loss. Minimizing the loss of a deep linear neural network is a nonconvex problem, and despite recent progress, our understanding of this loss surface is still incomplete. For deep linear networks, we present necessary and sufficient conditions for a critical point of the risk function to be a global minimum. Surprisingly, our conditions provide an efficiently checkable test for global optimality, while such tests are typically intractable in nonconvex optimization. We further extend these results to deep nonlinear neural networks and prove similar sufficient conditions for global optimality, albeit in a more limited function space setting.

Forniamo condizioni necessarie e sufficienti verificabili in modo efficiente per l'ottimalità globale nelle deep linear neural network, con alcune estensioni iniziali ai setting non lineari.

L'articolo fornisce condizioni per l'ottimalità globale della loss delle deep linear neural network

L'articolo fornisce risultati teorici sull'esistenza di minimi locali nella funzione obiettivo delle deep neural network.

Studia alcune proprietà teoriche delle deep linear network.


Recurrent auto-encoder model can summarise sequential data through an encoder structure into a fixed-length vector and then reconstruct into its original sequential form through the decoder structure. The summarised information can be used to represent time series features. In this paper, we propose relaxing the dimensionality of the decoder output so that it performs partial reconstruction. The fixed-length vector can therefore represent features only in the selected dimensions. In addition, we propose using rolling fixed window approach to generate samples. The change of time series features over time can be summarised as a smooth trajectory path. The fixed-length vectors are further analysed through additional visualisation and unsupervised clustering techniques.    This proposed method can be applied in large-scale industrial processes for sensors signal analysis purpose where clusters of the vector representations can be used to reflect the operating states of selected aspects of the industrial system.

Si utilizza il modello auto-encoder ricorrente per estrarre le feature multidimensionali delle serie temporali

Questo articolo descrive un'applicazione di autoencoder ricorrente per analizzare serie temporali multidimensionali

L'articolo descrive un modello di auto-encoder sequence to sequence che viene utilizzato per imparare rappresentazioni di sequenze, mostrando che, per la loro applicazione, si ottengono prestazioni migliori quando la rete viene addestrata solo per ricostruire un sottoinsieme delle misure dei dati.

Propone una strategia ispirata al modello auto-encoder ricorrente in modo che il clustering di dati multidimensionali di serie temporali possa essere eseguito sulla base di vettori di contesto.


We view molecule optimization as a graph-to-graph translation problem. The goal is to learn to map from one molecular graph to another with better properties based on an available corpus of paired molecules. Since molecules can be optimized in different ways, there are multiple viable translations for each input graph. A key challenge is therefore to model diverse translation outputs. Our primary contributions include a junction tree encoder-decoder for learning diverse graph translations along with a novel adversarial training method for aligning distributions of molecules. Diverse output distributions in our model are explicitly realized by low-dimensional latent vectors that modulate the translation process. We evaluate our model on multiple molecule optimization tasks and show that our model outperforms previous state-of-the-art baselines by a significant margin.

Introduciamo un framework di encoder-decoder graph-to-graph per imparare diverse traduzioni di grafi.

Propone un modello di traduzione graph-to-graph per l'ottimizzazione delle molecole ispirato all'analisi delle coppie molecolari abbinate.

Estensione di JT-VAE nello scenario di traduzione graph-to-graph aggiungendo la variabile latente per catturare la multi-modalità e una regolarizzazione adversarial nello spazio latente

Propone un sistema abbastanza complesso, che coinvolge molte scelte e componenti diverse, per ottenere composti chimici con proprietà migliorate a partire da un dato corpora.


Partial differential equations (PDEs) are widely used across the physical and computational sciences. Decades of research and engineering went into designing fast iterative solution methods. Existing solvers are general purpose, but may be sub-optimal for specific classes of problems. In contrast to existing hand-crafted solutions, we propose an approach to learn a fast iterative solver tailored to a specific domain. We achieve this goal by learning to modify the updates of an existing solver using a deep neural network. Crucially, our approach is proven to preserve strong correctness and convergence guarantees. After training on a single geometry, our model generalizes to a wide variety of geometries and boundary conditions, and achieves 2-3 times speedup compared to state-of-the-art solvers.

Impariamo un solutore neurale veloce per le PDE che ha garanzie di convergenza.

Sviluppa un metodo per accelerare il metodo delle differenze finite nella risoluzione delle PDE e propone un framework rivisto per l'iterazione del punto fisso dopo la discretizzazione.

Gli autori propongono un metodo lineare per accelerare i solutori di PDE.


Variational Bayesian neural networks (BNN) perform variational inference over weights, but it is difficult to specify meaningful priors and approximating posteriors in a high-dimensional weight space. We introduce functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund (ELBO) defined directly on stochastic processes, i.e. distributions over functions. We prove that the KL divergence between stochastic processes is equal to the supremum of marginal KL divergences over all finite sets of inputs. Based on this, we introduce a practical training objective which approximates the functional ELBO using finite measurement sets and the spectral Stein gradient estimator. With fBNNs, we can specify priors which entail rich structure, including Gaussian processes and implicit stochastic processes. Empirically, we find that fBNNs extrapolate well using various structured priors, provide reliable uncertainty estimates, and can scale to large datasets.

Eseguiamo l'inferenza funzionale variazionale sui processi stocastici definiti dalle reti neurali bayesiane.

Adattamento di approssimazioni variazionali di reti neurali bayesiane in forma funzionale tenendo in considerazione la corrispondenza a un processo stocastico a priori implicitamente tramite sample.

Presenta un nuovo obiettivo ELBO per il training delle BNN che permette di codificare nel modello prior più significativi piuttosto che i prior di peso meno informativi presenti in letteratura.

Presenta un nuovo algoritmo di inferenza variazionale per modelli di reti neurali bayesiani in cui il prior è specificato in modo funzionale piuttosto che attraverso un prior sui pesi.


Words are not created equal. In fact, they form an aristocratic graph with a latent hierarchical structure that the next generation of unsupervised learned word embeddings should reveal. In this paper, justified by the notion of delta-hyperbolicity or tree-likeliness of a space, we propose to embed words in a Cartesian product of hyperbolic spaces which we theoretically connect to the Gaussian word embeddings and their Fisher geometry. This connection allows us to introduce a novel principled hypernymy score for word embeddings. Moreover, we adapt the well-known Glove algorithm to learn unsupervised word embeddings in this type of Riemannian manifolds. We further explain how to solve the analogy task using the Riemannian parallel transport that generalizes vector arithmetics to this new type of geometry. Empirically, based on extensive experiments, we prove that our embeddings, trained unsupervised, are the first to simultaneously outperform strong and popular baselines on the tasks of similarity, analogy and hypernymy detection. In particular, for word hypernymy, we obtain new state-of-the-art on fully unsupervised WBLESS classification accuracy.

Mappiamo le parole nello spazio iperbolico e facciamo la connessione con i word embedding gaussiani.

Questo articolo adatta i word embedding di Glove a uno spazio iperbolico dato dal modello del semipiano di Poincare

Questo articolo propone un approccio per implementare un modello di word embedding iperbolico basato su GLOVE, che è ottimizzato attraverso i metodi di ottimizzazione Riemanniana.


Answering questions about a text frequently requires aggregating information from multiple places in that text. End-to-end neural network models, the dominant approach in the current literature, can theoretically learn how to distill and manipulate representations of the text without explicit supervision about how to do so. We investigate a canonical architecture for this task, the memory network, and analyze how effective it really is in the context of three multi-hop reasoning settings. In a simple synthetic setting, the path-finding task of the bAbI dataset, the model fails to learn the correct reasoning without additional supervision of its attention mechanism. However, with this supervision, it can perform well. On a real text dataset, WikiHop, the memory network gives nearly state-of-the-art performance, but does so without using its multi-hop capabilities. A tougher anonymized version of the WikiHop dataset is qualitatively similar to bAbI: the model fails to perform well unless it has additional supervision. We hypothesize that many "multi-hop" architectures do not truly learn this reasoning as advertised, though they could learn this reasoning if appropriately supervised.

Le memory network non imparano il ragionamento multi-hop a meno che non le supervisioniamo.

Il paper dichiara che il ragionamento multi-hop non è facile da imparare direttamente e richiede una supervisione diretta, inoltre fare bene su WikiHop non significa necessariamente che il modello stia effettivamente imparando a fare ragionamenti multi-hop.

L'articolo si propone di indagare il ben noto problema dell'apprendimento delle memory network e più precisamente la difficoltà della supervisione dell'apprendimento con attention di tali modelli.

Questo articolo sostiene che le memory network non riescono ad apprendere un ragionevole ragionamento multi-hop.


Generative Adversarial Nets (GANs) and Variational Auto-Encoders (VAEs) provide impressive image generations from Gaussian white noise, but the underlying mathematics are not well understood. We compute deep convolutional network generators by inverting a fixed embedding operator. Therefore, they do not require to be optimized with a discriminator or an encoder. The embedding is Lipschitz continuous to deformations so that generators transform linear interpolations between input white noise vectors into deformations between output images. This embedding is computed with a wavelet Scattering transform. Numerical experiments demonstrate that the resulting Scattering generators have similar properties as GANs or VAEs, without learning a discriminative network or an encoder.

Introduciamo reti generative che non richiedono di essere apprese con un discriminatore o un encoder; si ottengono invertendo uno speciale operatore di embedding definito da una trasformata wavelet Scattering.

Introduce le trasformate di scattering come modelli generativi di immagini nel contesto delle Generative Adversarial Network e suggerisce perché potrebbero essere viste come trasformate di Gaussianizzazione con loss di informazione controllata e invertibilità.

L'articolo propone un modello generativo per le immagini che non richiede l'apprendimento di un discriminatore (come nelle GAN) o di un embedding appreso.


Recurrent neural networks (RNNs) can model natural language by sequentially ''reading'' input tokens and outputting a distributed representation of each token. Due to the sequential nature of RNNs, inference time is linearly dependent on the input length, and all inputs are read regardless of their importance. Efforts to speed up this inference, known as ''neural speed reading'', either ignore or skim over part of the input. We present Structural-Jump-LSTM: the first neural speed reading model to both skip and jump text during inference. The model consists of a standard LSTM and two agents: one capable of skipping single words when reading, and one capable of exploiting punctuation structure (sub-sentence separators (,:), sentence end symbols (.!?), or end of text markers) to jump ahead after reading a word.  A comprehensive experimental evaluation of our model against all five state-of-the-art neural reading models shows that   Structural-Jump-LSTM achieves the best overall floating point operations (FLOP) reduction (hence is faster), while keeping the same accuracy or even improving it compared to a vanilla LSTM that reads the whole text.

Proponiamo un nuovo modello per neural speed reading che utilizza la struttura intrinseca della punteggiatura di un testo per definire un comportamento efficace di jumping e skipping.

L'articolo propone un modello Structural-Jump-LSTM per accelerare il machine reading con due agenti invece di uno

Propone un nuovo modello per neural speed reading in cui il nuovo reader ha la capacità di saltare una parola o una sequenza di parole.

L'articolo propone un metodo di lettura veloce usando azioni di skip e di jump, mostrando che il metodo proposto è accurato come le LSTM ma usa molto meno computazione.


One of the key challenges of session-based recommender systems is to enhance users’ purchase intentions. In this paper, we formulate the sequential interactions between user sessions and a recommender agent as a Markov Decision Process (MDP). In practice, the purchase reward is delayed and sparse, and may be buried by clicks, making it an impoverished signal for policy learning. Inspired by the prediction error minimization (PEM) and embodied cognition, we propose a simple architecture to augment reward, namely Imagination Reconstruction Network (IRN). Speciﬁcally, IRN enables the agent to explore its environment and learn predictive representations via three key components. The imagination core generates predicted trajectories, i.e., imagined items that users may purchase. The trajectory manager controls the granularity of imagined trajectories using the planning strategies, which balances the long-term rewards and short-term rewards. To optimize the action policy, the imagination-augmented executor minimizes the intrinsic imagination error of simulated trajectories by self-supervised reconstruction, while maximizing the extrinsic reward using model-free algorithms. Empirically, IRN promotes quicker adaptation to user interest, and shows improved robustness to the cold-start scenario and ultimately higher purchase performance compared to several baselines. Somewhat surprisingly, IRN using only the purchase reward achieves excellent next-click prediction performance, demonstrating that the agent can "guess what you like" via internal planning.

Proponiamo l'architettura IRN per aumentare la reward di acquisto sparsa e ritardata per la raccomandazione basata sulla sessione.

L'articolo propone di migliorare le prestazioni dei sistemi di raccomandazione attraverso reinforcement learning utilizzando una Imagination Reconstruction Network.

L'articolo presenta un approccio di raccomandazione basato sulla sessione, concentrandosi sugli acquisti degli utenti invece che sui clic.


The question why deep learning algorithms generalize so well has attracted increasing  research interest. However, most of the well-established approaches,  such as hypothesis capacity, stability or sparseness, have not provided complete  explanations (Zhang et al., 2016; Kawaguchi et al., 2017). In this work, we focus  on the robustness approach (Xu & Mannor, 2012), i.e., if the error of a hypothesis  will not change much due to perturbations of its training examples, then it  will also generalize well. As most deep learning algorithms are stochastic (e.g.,  Stochastic Gradient Descent, Dropout, and Bayes-by-backprop), we revisit the robustness  arguments of Xu & Mannor, and introduce a new approach – ensemble  robustness – that concerns the robustness of a population of hypotheses. Through  the lens of ensemble robustness, we reveal that a stochastic learning algorithm can  generalize well as long as its sensitiveness to adversarial perturbations is bounded  in average over training examples. Moreover, an algorithm may be sensitive to  some adversarial examples (Goodfellow et al., 2015) but still generalize well. To  support our claims, we provide extensive simulations for different deep learning  algorithms and different network architectures exhibiting a strong correlation between  ensemble robustness and the ability to generalize.

Spiegare la generalizzazione degli algoritmi di deep learning stocastico, teoricamente ed empiricamente, attraverso la robustezza dell'ensemble

Questo articolo presenta un adattamento della robustezza algoritmica di Xu&Mannor'12 e presenta bound di apprendimento e un esperimento che mostra la correlazione tra la robustezza empirica dell'ensemble e l'errore di generalizzazione.

Propone uno studio della capacità di generalizzazione degli algoritmi di deep learning utilizzando un'estensione della nozione di stabilità chiamata robustezza dell'ensemble e fornisce bound all'errore di generalizzazione di un algoritmo randomizzato in termini del parametro di stabilità e fornisce uno studio empirico che cerca di collegare la teoria con la pratica.

L'articolo ha studiato la capacità di generalizzazione degli algoritmi di apprendimento dal punto di vista della robustezza in un contesto di deep learning


Deep autoregressive models have shown state-of-the-art performance in density estimation for natural images on large-scale datasets such as ImageNet.   However, such models require many thousands of gradient-based weight updates and unique image examples for training. Ideally, the models would rapidly learn visual concepts from only a handful of examples, similar to the manner in which humans learns across many vision tasks.   In this paper, we show how 1) neural attention and 2) meta learning techniques can be used in combination with autoregressive models to enable effective few-shot density estimation. Our proposed modifications to PixelCNN result in state-of-the art few-shot density estimation on the Omniglot dataset.   Furthermore, we visualize the learned attention policy and find that it learns intuitive algorithms for simple tasks such as image mirroring on ImageNet and handwriting on Omniglot without supervision. Finally, we extend the model to natural images and demonstrate few-shot image generation on the Stanford Online Products dataset.

L'articolo propone di utilizzare la stima della densità quando la disponibilità di dati di training è bassa, utilizzando un modello di meta-learning.

Questo articolo considera il problema della stima della densità in un setting one/few-shot, usando tecniche di metalearning che sono state applicate all'apprendimento supervisionato one/few-shot

L'articolo si concentra sul few shot learning con la stima della densità autoregressiva e migliora PixelCNN con l'attention neurale e le tecniche di meta learning.


Neural networks exhibit good generalization behavior in the  over-parameterized regime, where the number of network parameters  exceeds the number of observations. Nonetheless,  current generalization bounds for neural networks fail to explain this  phenomenon. In an attempt to bridge this gap, we study the problem of  learning a two-layer over-parameterized neural network, when the data is generated by a linearly separable function. In the case where the network has Leaky  ReLU activations, we provide both optimization and generalization guarantees for over-parameterized networks.  Specifically, we prove convergence rates of SGD to a global  minimum and provide generalization guarantees for this global minimum  that are independent of the network size.   Therefore, our result clearly shows that the use of SGD for optimization both finds a global minimum, and avoids overfitting despite the high capacity of the model. This is the first theoretical demonstration that SGD can avoid overfitting, when learning over-specified neural network classifiers.

Mostriamo che SGD apprende reti neurali a due layer iperparametrizzate con attivazioni Leaky ReLU che generalizzano in modo dimostrabile su dati linearmente separabili.

L'articolo studia i modelli iperparametrizzati in grado di apprendere soluzioni che ben generalizzano utilizzando una rete a 1 hidden layer con un layer di output fisso.

Questo articolo mostra che, su dati linearmente separabili, SGD su una rete iperparametrizzata può ancora appoggiare un classificatore che generalizza in modo dimostrabile.