Incremental class learning involves sequentially learning classes in bursts of examples from the same class. This violates the assumptions that underlie  methods for training standard deep neural networks, and will cause them to suffer from catastrophic forgetting. Arguably, the best method for incremental class learning is iCaRL, but it requires storing  training examples for each class, making it challenging to scale. Here, we propose FearNet for incremental class learning. FearNet is a generative model that does not store previous examples, making it memory efficient. FearNet uses a brain-inspired dual-memory system in which new memories are consolidated from a network for recent memories inspired by the mammalian hippocampal complex to a network for long-term storage inspired by medial prefrontal cortex. Memory consolidation is inspired by mechanisms that occur during sleep. FearNet also uses a module inspired by the basolateral amygdala for determining which memory system to use for recall.   FearNet achieves state-of-the-art performance at incremental class learning on image (CIFAR-100, CUB-200) and audio classification (AudioSet) benchmarks.

FearNet è una rete neurale efficiente in termini di memoria, ispirata alla formazione della memoria nel cervello dei mammiferi, che è in grado di apprendere classi incrementali senza catastrophic forgetting.

Questo articolo presenta una nuova soluzione a un problema di classificazione incrementale basato su un sistema a doppia memoria.


Multi-view learning can provide self-supervision when different views are available of the same data. Distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we present two multi-view frameworks for learning sentence representations in an unsupervised fashion. One framework uses a generative objective and the other a discriminative one. In both frameworks, the final representation is an ensemble of two views, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model. We show that, after learning, the vectors produced by our multi-view frameworks provide improved representations over their single-view learnt counterparts, and the combination of different views gives representational improvement over each view and demonstrates solid transferability on standard downstream tasks.

L'apprendimento multi-view migliora l'apprendimento unsupervised di sentence embedding

L'approccio utilizza diversi encoder complementari della frase in input e la massimizzazione del consenso.

L'articolo presenta un quadro multi-view per migliorare la rappresentazione delle frasi nei task NLP usando architetture generative e discriminative.

Questo articolo mostra che le strutture multi-view sono più efficaci dell'uso di encoder individuali per l'apprendimento di rappresentazioni di frasi.


We show how discrete objects can be learnt in an unsupervised fashion from pixels, and how to perform reinforcement learning using this object representation.   More precisely, we construct a differentiable mapping from an image to a discrete tabular list of objects, where each object consists of a differentiable position, feature vector, and scalar presence value that allows the representation to be learnt using an attention mechanism.   Applying this mapping to Atari games, together with an interaction net-style architecture for calculating quantities from objects, we construct agents that can play Atari games using objects learnt in an unsupervised fashion. During training, many natural objects emerge, such as the ball and paddles in Pong, and the submarine and fish in Seaquest.   This gives the first reinforcement learning agent for Atari with an interpretable object representation, and opens the avenue for agents that can conduct object-based exploration and generalization.

Mostriamo come gli oggetti discreti possono essere appresi in modo unsupervised dai pixel, e come eseguire reinforcement learning usando questa rappresentazione degli oggetti.

Un metodo per imparare rappresentazioni di oggetti dai pixel per fare reinforcement learning.

L'articolo propone un'architettura neurale per mappare i flussi video su una collezione discreta di oggetti, senza annotazioni umane, utilizzando una loss di ricostruzione dei pixel non supervisionata.


Most recent gains in visual recognition have originated from the inclusion of attention mechanisms in deep convolutional networks (DCNs). Because these networks are optimized for object recognition, they learn where to attend using only a weak form of supervision derived from image class labels. Here, we demonstrate the benefit of using stronger supervisory signals by teaching DCNs to attend to image regions that humans deem important for object recognition. We first describe a large-scale online experiment (ClickMe) used to supplement ImageNet with nearly half a million human-derived "top-down" attention maps. Using human psychophysics, we confirm that the identified top-down features from ClickMe are more diagnostic than "bottom-up" saliency features for rapid image categorization. As a proof of concept, we extend a state-of-the-art attention network and demonstrate that adding ClickMe supervision significantly improves its accuracy and yields visual features that are more interpretable and more similar to those used by human observers.

Un dataset su larga scala per training di modelli di attention per il riconoscimento degli oggetti porta a un riconoscimento degli oggetti più accurato, interpretabile e simile a quello umano.

Il paper sostiene che i recenti guadagni nel riconoscimento visivo derivino dall'utilizzo di meccanismi di attention visiva nelle deep convolutional network, che imparano dove concentrarsi attraverso una forma debole di supervisione basata sulle label di classe delle immagini.

Presenta un nuovo approccio all'attention in cui un grande dataset sull'attention viene raccolto e utilizzato per addestrare una NN in modo supervisionato per sfruttare l'attention umana self-reported.

Questo articolo propone un nuovo approccio per utilizzare segnali più informativi, in particolare, le regioni che gli esseri umani ritengono importanti nelle immagini, per migliorare le deep convolutional neural network.


In recent years, deep neural networks have demonstrated outstanding performancein many machine learning tasks. However, researchers have discovered that thesestate-of-the-art models are vulnerable to adversarial examples:  legitimate examples added by small perturbations which are unnoticeable to human eyes. Adversarial training, which augments the training data with adversarial examples duringthe training process,  is a well known defense to improve the robustness of themodel against adversarial attacks.   However, this robustness is only effective tothe same attack method used for adversarial training.   Madry et al. (2017) suggest that effectiveness of iterative multi-step adversarial attacks and particularlythat projected gradient descent (PGD) may be considered the universal first order adversary and applying the adversarial training with PGD implies resistanceagainst many other first order attacks.    However,  the computational cost of theadversarial training with PGD and other multi-step adversarial examples is muchhigher than that of the adversarial training with other simpler attack techniques. In this paper, we show how strong adversarial examples can be generated only ata cost similar to that of two runs of the fast gradient sign method (FGSM), allowing defense against adversarial attacks with a robustness level comparable to thatof the adversarial training with multi-step adversarial examples.   We empiricallydemonstrate the effectiveness of the proposed two-step defense approach againstdifferent attack methods and its improvements over existing defense strategies.

Abbiamo proposto un metodo di difesa efficiente in termini di tempo contro gli adversarial attack one-step e iterativi.

Proporre un nuovo metodo efficiente dal punto di vista computazionale chiamato e2SAD che genera insiemi di due sample di adversarial training per ogni sample di training pulito.

L'articolo introduce un metodo di adversarial defense in due fasi, per generare due adversarial example per ogni sample pulito e includerli nel ciclo di training attuale per raggiungere la robustezza e affermare che può superare i metodi iterativi più costosi.

L'articolo presenta un approccio in 2 fasi per generare forti adversarial example ad un costo molto inferiore rispetto ai recenti adversarial attack iterativi in più fasi.


Recently several different deep learning architectures have been proposed that take a string of characters as the raw input signal and automatically derive features for text classification. Little studies are available that compare the effectiveness of these approaches for character based text classification with each other. In this paper we perform such an empirical comparison for the important cybersecurity problem of DGA detection: classifying domain names as either benign vs. produced by malware (i.e., by a Domain Generation Algorithm). Training and evaluating on a dataset with 2M domain names shows that there is surprisingly little difference between various convolutional neural network (CNN) and recurrent neural network (RNN) based architectures in terms of accuracy, prompting a preference for the simpler architectures, since they are faster to train and less prone to overfitting.

Un confronto tra cinque architetture di deep neural network per il rilevamento di domain name malevoli mostra sorprendentemente poche differenze.

Gli autori propongono di utilizzare cinque architetture di deep learning per il task di cybersicurezza relativo al rilevamento degli algoritmi di generazione del dominio.

Applica diverse architetture NN per classificare gli URL tra quelli benigni e quelli relativi a malware.

Questo articolo propone di riconoscere automaticamente i nomi di dominio come malevoli o benigni tramite deep network addestrate per classificare direttamente la sequenza di caratteri.


Recognizing the relationship between two texts is an important aspect of natural language understanding (NLU), and a variety of neural network models have been proposed for solving NLU tasks. Unfortunately, recent work showed that the datasets these models are trained on often contain biases that allow models to achieve non-trivial performance without possibly learning the relationship between the two texts. We propose a framework for building robust models by using adversarial learning to encourage models to learn latent, bias-free representations. We test our approach in a Natural Language Inference (NLI) scenario, and show that our adversarially-trained models learn robust representations that ignore known dataset-specific biases. Our experiments demonstrate that our models are more robust to new NLI datasets.

I metodi di adversarial training incoraggiano i modelli di NLI a ignorare i bias specifici del dataset e aiutano i modelli a fare transfer attraverso i dataset.

L'articolo propone un adversarial setup per mitigare gli artefatti di annotazione nei dati di natural language inference

Questo articolo presenta un metodo per rimuovere i bias di un modello di entailment testuale attraverso un obiettivo di adversarial training.


We study the problem of learning representations of entities and relations in knowledge graphs for predicting missing links. The success of such a task heavily relies on the ability of modeling and inferring the patterns of (or between) the relations. In this paper, we present a new approach for knowledge graph embedding called RotatE, which is able to model and infer various relation patterns including: symmetry/antisymmetry, inversion, and composition. Specifically, the RotatE model defines each relation as a rotation from the source entity to the target entity in the complex vector space. In addition, we propose a novel self-adversarial negative sampling technique for efficiently and effectively training the RotatE model. Experimental results on multiple benchmark knowledge graphs show that the proposed RotatE model is not only scalable, but also able to infer and model various relation patterns and significantly outperform existing state-of-the-art models for link prediction.

Un nuovo approccio allo stato dell'arte per l'embedding dei knowledge graph.

Presenta una funzione di scoring per neural link prediction che può dedurre modelli di simmetria, antisimmetria, inversione e composizione delle relazioni in una knowledge base.

Questo articolo propone un approccio all'embedding dei knowledge graph modellando le relazioni come rotazioni nello spazio vettoriale complesso.

Propone un metodo per l'embedding di grafi da utilizzare per la link prediction


Deep learning algorithms have been known to be vulnerable to adversarial perturbations in various tasks such as image classification. This problem was addressed by employing several defense methods for detection and rejection of particular types of attacks. However, training and manipulating networks according to particular defense schemes increases computational complexity of the learning algorithms. In this work, we propose a simple yet effective method to improve robustness of convolutional neural networks (CNNs) to adversarial attacks by using data dependent adaptive convolution kernels. To this end, we propose a new type of HyperNetwork in order to employ statistical properties of input data and features for computation of statistical adaptive maps. Then, we filter convolution weights of CNNs with the learned statistical maps to compute dynamic kernels. Thereby, weights and kernels are collectively optimized for learning of image classification models robust to  adversarial attacks without employment of additional target detection and rejection algorithms.  We empirically demonstrate that the proposed method enables CNNs to spontaneously defend against different types of attacks, e.g. attacks generated by Gaussian noise, fast gradient sign methods (Goodfellow et al., 2014) and a black-box attack (Narodytska & Kasiviswanathan, 2016).

Abbiamo modificato la CNN usando HyperNetworks e abbiamo osservato una migliore robustezza contro gli adversarial example.

Migliorare la robustezza e l'affidabilità delle deep convolutional neural network usando kernel di convoluzione dipendenti dai dati


Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures.  Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent.  Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently.  In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning.  The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data.  This requires back-propagating errors through the solver steps.  While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage.  We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components.  Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.

Proponiamo un approccio di meta-learning per la classificazione few shot che raggiunge forti prestazioni ad alta velocità facendo la back-propagation attraverso la soluzione di risolutori veloci, come la regressione ridge o la regressione logistica.

L'articolo propone un algoritmo per il meta-learning che equivale a fissare le feature (cioè tutti gli hidden layer di una deep NN), e trattare ogni task come se avesse il proprio layer finale come una regressione ridge o una regressione logistica.

Questo articolo propone un approccio di meta-learning per il problema della classificazione few shot, usano un metodo basato sulla parametrizzazione del learner per ogni task con un risolutore in forma chiusa.


While many active learning papers assume that the learner can simply ask for a label and receive it, real annotation often presents a mismatch between the form of a label (say, one among many classes), and the form of an annotation (typically yes/no binary feedback). To annotate examples corpora for multiclass classification, we might need to ask multiple yes/no questions, exploiting a label hierarchy if one is available. To address this more realistic setting, we propose active learning with partial feedback (ALPF), where the learner must actively choose both which example to label and which binary question to ask. At each step, the learner selects an example, asking if it belongs to a chosen (possibly composite) class. Each answer eliminates some classes, leaving the learner with a partial label. The learner may then either ask more questions about the same example (until an exact label is uncovered) or move on immediately, leaving the first example partially labeled. Active learning with partial labels requires (i) a sampling strategy to choose (example, class) pairs, and (ii) learning from partial labels between rounds. Experiments on Tiny ImageNet demonstrate that our most effective method improves 26% (relative) in top-1 classification accuracy compared to i.i.d. baselines and standard active learners given 30% of the annotation budget that would be required (naively) to annotate the dataset. Moreover, ALPF-learners fully annotate TinyImageNet at 42% lower cost. Surprisingly, we observe that accounting for per-example annotation costs can alter the conventional wisdom that active learners should solicit labels for hard examples.

Forniamo una nuova prospettiva sul training di un modello di apprendimento automatico da zero nel setting delle label gerarchiche, cioè pensandolo come una comunicazione a due vie tra l'uomo e gli algoritmi, e studiamo come possiamo sia misurare che migliorare l'efficienza.

Introduce un nuovo setting di Active Learning in cui l'oracolo offre una label parziale o debole invece di cercare la label di un particolare esempio, portando a un recupero più semplice delle informazioni.

Questo articolo propone un metodo di active learning con feedback parziale che supera le baseline esistenti sotto un budget limitato.

L'articolo considera un problema di classificazione multiclasse in cui le label sono raggruppate in un dato numero M di sottoinsiemi, che contengono tutte le label individuali come singleton.


Despite their prevalence, Euclidean embeddings of data are fundamentally limited in their ability to capture latent semantic structures, which need not conform to Euclidean spatial assumptions. Here we consider an alternative, which embeds data as discrete probability distributions in a Wasserstein space, endowed with an optimal transport metric. Wasserstein spaces are much larger and more flexible than Euclidean spaces, in that they can successfully embed a wider variety of metric structures. We propose to exploit this flexibility by learning an embedding that captures the semantic information in the Wasserstein distance between embedded distributions. We examine empirically the representational capacity of such learned Wasserstein embeddings, showing that they can embed a wide variety of complex metric structures with smaller distortion than an equivalent Euclidean embedding. We also investigate an application to word embedding, demonstrating a unique advantage of Wasserstein embeddings: we can directly visualize the high-dimensional embedding, as it is a probability distribution on a low-dimensional space. This obviates the need for dimensionality reduction techniques such as t-SNE for visualization.

Mostriamo che gli spazi di Wasserstein sono buoni obiettivi per incorporare dati con una struttura semantica complessa.

Impara gli embedding in uno spazio discreto di distribuzioni di probabilità, utilizzando una versione minimizzata e regolarizzata delle distanze di Wasserstein.

L'articolo descrive un nuovo metodo di embedding che incorpora i dati nello spazio delle misure di probabilità dotato della distanza di Wasserstein.

L'articolo propone di incorporare i dati in spazi Wasserstein a bassa dimensione, che possono catturare la struttura sottostante ai dati in modo più accurato.


Clustering high-dimensional datasets is hard because interpoint distances become less informative in high-dimensional spaces. We present a clustering algorithm that performs nonlinear dimensionality reduction and clustering jointly. The data is embedded into a lower-dimensional space by a deep autoencoder. The autoencoder is optimized as part of the clustering process. The resulting network produces clustered data. The presented approach does not rely on prior knowledge of the number of ground-truth clusters. Joint nonlinear dimensionality reduction and clustering are formulated as optimization of a global continuous objective. We thus avoid discrete reconfigurations of the objective that characterize prior clustering algorithms. Experiments on datasets from multiple domains demonstrate that the presented algorithm outperforms state-of-the-art clustering schemes, including recent methods that use deep networks.

Un algoritmo di clustering che esegue la riduzione non lineare della dimensionalità e il clustering congiunto ottimizzando un obiettivo globale continuo.

Presenta un algoritmo di clustering risolvendo congiuntamente l'autoencoder profondo e il clustering come obiettivo continuo globale, mostrando risultati migliori rispetto agli schemi di clustering allo stato dell'arte.

Deep Continuous Clustering è un metodo di clustering che integra l'obiettivo dell'autoencoder con l'obiettivo del clustering e poi si addestra usando SGD.


Deep convolutional neural networks (CNNs) are deployed in various applications but demand immense computational requirements. Pruning techniques and Winograd convolution are two typical methods to reduce the CNN computation. However, they cannot be directly combined because Winograd transformation fills in the sparsity resulting from pruning. Li et al. (2017) propose sparse Winograd convolution in which weights are directly pruned in the Winograd domain, but this technique is not very practical because Winograd-domain retraining requires low learning rates and hence significantly longer training time. Besides, Liu et al. (2018) move the ReLU function into the Winograd domain, which can help increase the weight sparsity but requires changes in the network structure. To achieve a high Winograd-domain weight sparsity without changing network structures, we propose a new pruning method, spatial-Winograd pruning. As the first step, spatial-domain weights are pruned in a structured way, which efficiently transfers the spatial-domain sparsity into the Winograd domain and avoids Winograd-domain retraining. For the next step, we also perform pruning and retraining directly in the Winograd domain but propose to use an importance factor matrix to adjust weight importance and weight gradients. This adjustment makes it possible to effectively retrain the pruned Winograd-domain network without changing the network structure. For the three models on the datasets of CIFAR-10, CIFAR-100, and ImageNet, our proposed method can achieve the Winograd-domain sparsities of 63%, 50%, and 74%, respectively.

Per accelerare il calcolo delle reti neurali convoluzionali, proponiamo una nuova tecnica di pruning in due fasi che raggiunge una maggiore sparsità dei pesi nel dominio di Winograd senza cambiare la struttura della rete.

Propone un quadro di pruning spatial-Winograd che permette di mantenere il peso pruned dal dominio spaziale nel dominio di Winograd e migliora la sparsità del dominio di Winograd.

Propone due tecniche per il pruning dei layer convoluzionali che utilizzano l'algoritmo di Winograd


In federated learning problems, data is scattered across different servers and exchanging or pooling it is often impractical or prohibited. We develop a Bayesian nonparametric framework for federated learning with neural networks. Each data server is assumed to train local neural network weights, which are modeled through our framework. We then develop an inference approach that allows us to synthesize a more expressive global network without additional supervision or data pooling. We then demonstrate the efficacy of our approach on federated learning problems simulated from two popular image classification datasets.

Proponiamo un modello bayesiano non parametrico per federated learning con reti neurali.

Utilizza il processo beta per fare neural federated matching.

L'articolo considera il federated learning delle reti neurali, dove i dati sono distribuiti su più macchine e l'allocazione dei data point è potenzialmente disomogenea e sbilanciata.


We present a general-purpose method to train Markov chain Monte Carlo kernels, parameterized by deep neural networks, that converge and mix quickly to their target distribution. Our method generalizes Hamiltonian Monte Carlo and is trained to maximize expected squared jumped distance, a proxy for mixing speed. We demonstrate large empirical gains on a collection of simple but challenging distributions, for instance achieving a 106x improvement in effective sample size in one case, and mixing when standard HMC makes no measurable progress in a second. Finally, we show quantitative and qualitative gains on a real-world task: latent-variable generative modeling. Python source code will be open-sourced with the camera-ready paper.

Metodo generale per addestrare kernel MCMC espressivi parametrizzati con deep neural network. Data una distribuzione di destinazione p, il nostro metodo fornisce un sampler veloce, in grado di esplorare in modo efficiente lo spazio di stato.

Propone un HMC generalizzato modificando l'integratore leapfrog utilizzando reti neurali per far convergere e mixare rapidamente il sampler.


This paper addresses the problem of evaluating learning systems in safety critical domains such as autonomous driving, where failures can have catastrophic consequences. We focus on two problems: searching for scenarios when learned agents fail and assessing their probability of failure. The standard method for agent evaluation in reinforcement learning, Vanilla Monte Carlo, can miss failures entirely, leading to the deployment of unsafe agents. We demonstrate this is an issue for current agents, where even matching the compute used for training is sometimes insufficient for evaluation. To address this shortcoming, we draw upon the rare event probability estimation literature and propose an adversarial evaluation approach. Our approach focuses evaluation on adversarially chosen situations, while still providing unbiased estimates of failure probabilities. The key difficulty is in identifying these adversarial situations -- since failures are rare there is little signal to drive optimization. To solve this we propose a continuation approach that learns failure modes in related but less robust agents. Our approach also allows reuse of data already collected for training the agent. We demonstrate the efficacy of adversarial evaluation on two standard domains: humanoid control and simulated driving. Experimental results show that our methods can find catastrophic failures and estimate failures rates of agents multiple orders of magnitude faster than standard evaluation schemes, in minutes to hours rather than days.

Mostriamo che le failure rare ma catastrofiche possono essere interamente non rilevate dai test casuali, il che pone problemi per un deployment sicuro. Il nostro approccio proposto per gli adversarial test risolve questo problema.

Propone un metodo che impara un predittore di probabilità di fallimento per un learned agent, portando a predire quali stati iniziali causano il fallimento di un sistema.

Questo articolo propone un approccio di importance sampling per campionare i casi di fallimento per gli algoritmi RL basati su una funzione appresa tramite una rete neurale sui fallimenti che si verificano durante il training dell'agente

Questo articolo ha proposto un approccio adversarial per identificare i casi di fallimento catastrofico nel reinforcement learning.


The variational autoencoder (VAE) is a popular combination of deep latent variable model and accompanying variational learning technique. By using a neural inference network to approximate the model's posterior on latent variables, VAEs efficiently parameterize a lower bound on marginal data likelihood that can be optimized directly via gradient methods. In practice, however, VAE training often results in a degenerate local optimum known as "posterior collapse" where the model learns to ignore the latent variable and the approximate posterior mimics the prior. In this paper, we investigate posterior collapse from the perspective of training dynamics. We find that during the initial stages of training the inference network fails to approximate the model's true posterior, which is a moving target. As a result, the model is encouraged to ignore the latent encoding and posterior collapse occurs. Based on this observation, we propose an extremely simple modification to VAE training to reduce inference lag: depending on the model's current mutual information between latent variable and observation, we aggressively optimize the inference network before performing each model update. Despite introducing neither new model components nor significant complexity over basic VAE, our approach is able to avoid the problem of collapse that has plagued a large amount of previous work. Empirically, our approach outperforms strong autoregressive baselines on text and image benchmarks in terms of held-out likelihood, and is competitive with more complex techniques for avoiding collapse while being substantially faster.

Per affrontare il posterior collapse nei VAE, proponiamo una procedura di training nuova ma semplice che ottimizza in modo aggressivo la rete di inferenza con più aggiornamenti. Questa nuova procedura di training attenua il posterior collapse e porta a un modello VAE migliore.

Esamina il fenomeno del posterior collapse, mostrando che un maggiore training della rete di inferenza può ridurre il problema e portare a ottimi migliori.

Gli autori propongono di cambiare la procedura di training delle VAE solo come soluzione al posterior collapse, lasciando il modello e l'obiettivo intatti.


Online healthcare services can provide the general public with ubiquitous access to medical knowledge and reduce the information access cost for both individuals and societies. To promote these benefits, it is desired to effectively expand the scale of high-quality yet novel relational medical entity pairs that embody rich medical knowledge in a structured form. To fulfill this goal, we introduce a generative model called Conditional Relationship Variational Autoencoder (CRVAE), which can discover meaningful and novel relational medical entity pairs without the requirement of additional external knowledge. Rather than discriminatively identifying the relationship between two given medical entities in a free-text corpus, we directly model and understand medical relationships from diversely expressed medical entity pairs. The proposed model introduces the generative modeling capacity of variational autoencoder to entity pairs, and has the ability to discover new relational medical entity pairs solely based on the existing entity pairs. Beside entity pairs, relationship-enhanced entity representations are obtained as another appealing benefit of the proposed method. Both quantitative and qualitative evaluations on real-world medical datasets demonstrate the effectiveness of the proposed method in generating relational medical entity pairs that are meaningful and novel.

Scoprire generativamente nuove coppie di entità significative con una certa relazione medica imparando puramente dalle coppie di entità significative esistenti, senza il requisito di un corpus di testo aggiuntivo per l'estrazione discriminante.

Presenta un autoencoder variazionale per generare coppie di entità data una relazione in un ambiente medico.

Nel contesto medico, questo articolo descrive il classico problema del "completamento della knowledge base" dai soli dati strutturati.


Although variational autoencoders (VAEs) represent a widely influential deep generative model, many aspects of the underlying energy function remain poorly understood.  In particular, it is commonly believed that Gaussian encoder/decoder assumptions reduce the effectiveness of VAEs in generating realistic samples .  In this regard, we rigorously analyze the VAE objective, differentiating situations where this belief is and is not actually true .  We then leverage the corresponding insights to develop a simple VAE enhancement that requires no additional hyperparameters or sensitive tuning .  Quantitatively, this proposal produces crisp samples and stable FID scores that are actually competitive with a variety of GAN models, all while retaining desirable attributes of the original VAE architecture . The code for our model is available at \url{https://github.com/daib13/TwoStageVAE}.

Analizziamo da vicino la funzione obiettivo VAE e traiamo nuove conclusioni che portano a semplici miglioramenti.

Propone un metodo VAE a due stadi per generare sample di alta qualità ed evitare la blurriness.

Questo articolo analizza i VAE gaussiani.

L'articolo fornisce una serie di risultati teorici sugli autoencoder variazionali gaussiani "vanilla", che sono poi utilizzati per costruire un nuovo algoritmo chiamato "2 stage VAEs".


We give a simple, fast algorithm for hyperparameter optimization inspired by techniques from the analysis of Boolean functions.   We focus on the high-dimensional regime where the canonical example is training a neural network with a large number of hyperparameters. The algorithm --- an iterative application of compressed sensing techniques for orthogonal polynomials --- requires only uniform sampling of the hyperparameters and is thus easily parallelizable.    Experiments for training deep neural networks on Cifar-10 show that compared to state-of-the-art tools (e.g., Hyperband and Spearmint), our algorithm finds significantly improved solutions, in some cases better than what is attainable by hand-tuning.   In terms of overall running time (i.e., time required to sample various settings of hyperparameters plus additional computation time), we are at least an order of magnitude faster than Hyperband and Bayesian Optimization.   We also outperform Random Search $8\times$.     Our method is inspired by provably-efficient algorithms for learning decision trees using the discrete Fourier transform.   We obtain improved sample-complexty bounds for learning decision trees while matching state-of-the-art bounds on running time (polynomial and quasipolynomial, respectively).

Un algoritmo di hyperparameter tuning utilizzando l'analisi di Fourier discreta e il rilevamento compresso

Indaga il problema dell'ottimizzazione degli iperparametri sotto l'ipotesi che la funzione sconosciuta possa essere approssimata, mostrando che la minimizzazione approssimata può essere eseguita sull'ipercubo booleano.

L'articolo esplora l'ottimizzazione degli iperparametri assumendo una struttura nella funzione sconosciuta che mappa gli iperparametri all'accuratezza della classificazione


Permutations and matchings are core building blocks in a variety of latent variable models, as they allow us to align, canonicalize, and sort data. Learning in such models is difficult, however, because exact marginalization over these combinatorial objects is intractable. In response, this paper introduces a collection of new methods for end-to-end learning in such models that approximate discrete maximum-weight matching using the continuous Sinkhorn operator.   Sinkhorn iteration is attractive because it functions as a simple, easy-to-implement analog of the softmax operator. With this, we can define the Gumbel-Sinkhorn method, an extension of the Gumbel-Softmax method (Jang et al. 2016, Maddison2016 et al. 2016) to distributions over latent matchings. We demonstrate the effectiveness of our method by outperforming competitive baselines on a range of qualitatively different tasks: sorting numbers, solving jigsaw puzzles, and identifying neural signals in worms.

Un nuovo metodo per l'inferenza delle permutazioni con gradient-descent, con applicazioni al latent matching inference e all'apprendimento supervisionato di permutazioni con reti neurali

L'articolo utilizza un'approssimazione finita dell'operatore Sinkhorn per descrivere come si può costruire una rete neurale per l'apprendimento da dati di training valutati in permutazione.

L'articolo propone un nuovo metodo che approssima il peso massimo discreto per l'apprendimento delle permutazioni latenti


Recent work in network quantization has substantially reduced the time and space complexity of neural network inference, enabling their deployment on embedded and mobile devices with limited computational and memory resources. However, existing quantization methods often represent all weights and activations with the same precision (bit-width). In this paper, we explore a new dimension of the design space: quantizing different layers with different bit-widths. We formulate this problem as a neural architecture search problem and propose a novel differentiable neural architecture search (DNAS) framework to efficiently explore its exponential search space with gradient-based optimization. Experiments show we surpass the state-of-the-art compression of ResNet on CIFAR-10 and ImageNet. Our quantized models with 21.1x smaller model size or 103.9x lower computational cost can still outperform baseline quantized or even full precision models.

Un nuovo framework di ricerca dell'architettura neurale differenziabile per la quantizzazione mista di ConvNet.

Gli autori introducono un nuovo metodo per la ricerca dell'architettura neurale che seleziona la quantizzazione della precisione dei pesi in ogni layer della rete neurale, e lo usano nel contesto della compressione della rete.

L'articolo presenta un nuovo approccio nella quantizzazione della rete quantizzando diversi layer con diverse larghezze di bit e introduce un nuovo framework di ricerca dell'architettura neurale differenziabile.


The top-$k$ error is a common measure of performance in machine learning and computer vision. In practice, top-$k$ classification is typically performed with deep neural networks trained with the cross-entropy loss. Theoretical results indeed suggest that cross-entropy is an optimal learning objective for such a task in the limit of infinite data. In the context of limited and noisy data however, the use of a loss function that is specifically designed for top-$k$ classification can bring significant improvements.  Our empirical evidence suggests that the loss function must be smooth and have non-sparse gradients in order to work well with deep neural networks. Consequently, we introduce a family of smoothed loss functions that are suited to top-$k$ optimization via deep learning. The widely used cross-entropy is a special case of our family. Evaluating our smooth loss functions is computationally challenging: a na{\"i}ve algorithm would require $\mathcal{O}(\binom{n}{k})$ operations, where $n$ is the number of classes. Thanks to a connection to polynomial algebra and a divide-and-conquer approach, we provide an algorithm with a time complexity of $\mathcal{O}(k n)$. Furthermore, we present a novel approximation to obtain fast and stable algorithms on GPUs with single floating point precision. We compare the performance of the cross-entropy loss and our margin-based losses in various regimes of noise and data size, for the predominant use case of $k=5$. Our investigation reveals that our loss is more robust to noise and overfitting than cross-entropy.

Una loss smooth per la minimizzazione dell'errore Top-k

Propone di utilizzare la loss top-k con deep model per affrontare il problema della confusione di classi con classi simili presenti o assenti nel dataset di training.

Questo articolo introduce una loss surrogata smooth per la SVM top-k, allo scopo di collegare gli SVM alle deep neural network.


Designing a molecule with desired properties is one of the biggest challenges in drug development, as it requires optimization of chemical compound structures with respect to many complex properties. To augment the compound design process we introduce Mol-CycleGAN -- a CycleGAN-based model that generates optimized compounds with a chemical scaffold of interest. Namely, given a molecule our model generates a structurally similar one with an optimized value of the considered property. We evaluate the performance of the model on selected optimization objectives related to structural properties (presence of halogen groups, number of aromatic rings) and to a physicochemical property (penalized logP). In the task of optimization of penalized logP of drug-like molecules our model significantly outperforms previous results.

Introduciamo Mol-CycleGAN - un nuovo modello generativo per l'ottimizzazione delle molecole per aiutare la progettazione dei farmaci.

L'articolo presenta un approccio per l'ottimizzazione delle proprietà molecolari basato sull'applicazione di CycleGAN agli autoencoder variazionali per le molecole e impiega un VAE specifico del dominio chiamato Junction Tree VAE (JT-VAE).

Questo articolo usa un autoencoder variazionale per imparare una funzione di traduzione, dall'insieme delle molecole senza la proprietà interessata all'insieme delle molecole con tale proprietà.


Knowledge distillation is a potential solution for model compression. The idea is to make a small student network imitate the target of a large teacher network, then the student network can be competitive to the teacher one. Most previous studies focus on model distillation in the classification task, where they propose different architectures and initializations for the student network. However, only the classification task is not enough, and other related tasks such as regression and retrieval are barely considered. To solve the problem, in this paper, we take face recognition as a breaking point and propose model distillation with knowledge transfer from face classification to alignment and verification. By selecting appropriate initializations and targets in the knowledge transfer, the distillation can be easier in non-classification tasks. Experiments on the CelebA and CASIA-WebFace datasets demonstrate that the student network can be competitive to the teacher one in alignment and verification, and even surpasses the teacher network under specific compression rates. In addition, to achieve stronger knowledge transfer, we also use a common initialization trick to improve the distillation performance of classification. Evaluations on the CASIA-Webface and large-scale MS-Celeb-1M datasets show the effectiveness of this simple trick.

Prendiamo il riconoscimento dei volti come punto di rottura e proponiamo la distillazione dei modelli con il transfer di conoscenze dalla classificazione dei volti all'allineamento e alla verifica

Questo articolo propone di trasferire il classificatore dal modello per la classificazione dei volti al task di allineamento e verifica.

Il manoscritto presenta esperimenti sulla knowledge distillation da un modello di classificazione dei volti a modelli di student per l'allineamento e la verifica dei volti.


RNNs have been shown to be excellent models for sequential data and in particular for session-based user behavior. The use of RNNs provides impressive performance benefits over classical methods in session-based recommendations. In this work we introduce a novel ranking loss function tailored for RNNs in recommendation settings. The better performance of such loss over alternatives, along with further tricks and improvements described in this work, allow to achieve an overall improvement of up to 35% in terms of MRR and Recall@20 over previous session-based RNN solutions and up to 51% over classical collaborative filtering approaches. Unlike data augmentation-based improvements, our method does not increase training times significantly.

Miglioramento delle raccomandazioni basate sulla sessione con RNN (GRU4Rec) del 35% utilizzando funzioni di loss e sampling di nuova concezione.

Questo articolo analizza le funzioni di loss esistenti per le raccomandazioni basate sulla sessione e propone due nuove funzioni di loss che aggiungono una ponderazione alle funzioni di loss esistenti basate sul ranking

Presenta modifiche sulla base di lavori precedenti per la raccomandazione basata sulla sessione usando RNN e pesando gli esempi negativi per la loro "rilevanza".

Questo articolo discute i problemi di ottimizzazione delle funzioni di loss in GRU4Rec, propone dei trucchi per l'ottimizzazione e suggerisce una versione migliorata.


In representational lifelong learning an agent aims to continually learn to solve novel tasks while updating its representation in light of previous tasks. Under the assumption that future tasks are related to previous tasks, representations should be learned in such a way that they capture the common structure across learned tasks, while allowing the learner sufficient flexibility to adapt to novel aspects of a new task. We develop a framework for lifelong learning in deep neural networks that is based on generalization bounds, developed within the PAC-Bayes framework. Learning takes place through the construction of a distribution over networks based on the tasks seen so far, and its utilization for learning a new task. Thus, prior knowledge is incorporated through setting a history-dependent prior for novel tasks. We develop a gradient-based algorithm implementing these ideas, based on minimizing an objective function motivated by generalization bounds, and demonstrate its effectiveness through numerical examples.

Sviluppiamo un approccio di lifelong learning per transfer learning basato sulla teoria PAC-Bayes, in cui i prior vengono aggiustati quando si incontrano nuovi task, facilitando così l'apprendimento di nuovi task.

Un nuovo bound sul rischio PAC-Bayesiano che serve come funzione obiettivo per l'apprendimento automatico multi-task, e un algoritmo per minimizzare una versione semplificata di quella funzione obiettivo.

Estende i bound PAC-Bayes esistenti al multi-task learning, per consentire l'adattamento del prior a diversi task.


Optimization algorithms for training deep models not only affects the convergence rate and stability of the training process, but are also highly related to the generalization performance of trained models. While adaptive algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in many scenarios, they often lead to worse generalization performance than SGD, when used for training deep neural networks (DNNs). In this work, we identify two problems regarding the direction and step size for updating the weight vectors of hidden units, which may degrade the generalization performance of Adam. As a solution, we propose the normalized direction-preserving Adam (ND-Adam) algorithm, which controls the update direction and step size more precisely, and thus bridges the generalization gap between Adam and SGD. Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits. By bridging the gap between SGD and Adam, we also shed some light on why certain optimization algorithms generalize better than others.

Una versione su misura di Adam per il training delle DNN, che colma il divario di generalizzazione tra Adam e SGD.

Propone una variante dell'algoritmo di ottimizzazione ADAM che normalizza i pesi di ogni hidden unit usando la batch normalization

Estensione dell'algoritmo di ottimizzazione di Adam per preservare la direzione di aggiornamento adattando il learning rate per i pesi in entrata a una hidden unit utilizzando congiuntamente la norma L2 del vettore del gradiente


Options in reinforcement learning allow agents to hierarchically decompose a task into subtasks, having the potential to speed up learning and planning. However, autonomously learning effective sets of options is still a major challenge in the field. In this paper we focus on the recently introduced idea of using representation learning methods to guide the option discovery process. Specifically, we look at eigenoptions, options obtained from representations that encode diffusive information flow in the environment. We extend the existing algorithms for eigenoption discovery to settings with stochastic transitions and in which handcrafted features are not available.   We propose an algorithm that discovers eigenoptions while learning non-linear state representations from raw pixels. It exploits recent successes in the deep reinforcement learning literature and the equivalence between proto-value functions and the successor representation. We use traditional tabular domains to provide intuition about our approach and Atari 2600 games to demonstrate its potential.

Mostriamo come possiamo usare la rappresentazione del successore per scoprire eigenoption in domini stocastici, a partire da pixel grezzi. Le eigenoption sono opzioni apprese per navigare nelle dimensioni latenti di una rappresentazione appresa.

Estende l'idea di eigenoption a domini con transizioni stocastiche e dove le feature dello stato vengono apprese.

Mostra l'equivalenza tra le proto value function e le rappresentazioni dei successori e deriva l'idea delle eigenoption come meccanismo nella scoperta delle opzioni

L'articolo è un seguito del precedente lavoro di Machado et al. (2017) che mostra come le funzioni di proto-valore possono essere usate per definire opzioni chiamate "eigenoption".


One form of characterizing the expressiveness of a piecewise linear neural network is by the number of linear regions, or pieces, of the function modeled. We have observed substantial progress in this topic through lower and upper bounds on the maximum number of linear regions and a counting procedure. However, these bounds only account for the dimensions of the network and the exact counting may take a prohibitive amount of time, therefore making it infeasible to benchmark the expressiveness of networks. In this work, we approximate the number of linear regions of specific rectifier networks with an algorithm for probabilistic lower bounds of mixed-integer linear sets. In addition, we present a tighter upper bound that leverages network coefficients. We test both on trained networks. The algorithm for probabilistic lower bounds is several orders of magnitude faster than exact counting and the values reach similar orders of magnitude, hence making our approach a viable method to compare the expressiveness of such networks. The refined upper bound is particularly stronger on networks with narrow layers.

Forniamo limiti superiori migliorati per il numero di regioni lineari usate nell'espressività della rete, e un algoritmo altamente efficiente (rispetto al conteggio esatto) per ottenere limiti inferiori probabilistici sul numero effettivo di regioni lineari.

Contribuisce allo studio del numero di regioni lineari nelle reti neurali RELU utilizzando un algoritmo di conteggio probabilistico approssimato e l'analisi

Costruisce il lavoro precedente che studia il conteggio delle regioni lineari nelle deep neural network, e migliora il limite superiore precedentemente proposto cambiando il vincolo di dimensionalità

L'articolo si occupa dell'espressività di una rete neurale piecewise linear, caratterizzata dal numero di regioni lineari della funzione modellata, e sfrutta algoritmi probabilistici per calcolare i limiti più velocemente, e dimostra limiti più stretti.


The ability to look multiple times through a series of pose-adjusted glimpses is fundamental to human vision. This critical faculty allows us to understand highly complex visual scenes. Short term memory plays an integral role in aggregating the information obtained from these glimpses and informing our interpretation of the scene. Computational models have attempted to address glimpsing and visual attention but have failed to incorporate the notion of memory. We introduce a novel, biologically inspired visual working memory architecture that we term the Hebb-Rosenblatt memory. We subsequently introduce a fully differentiable Short Term Attentive Working Memory model (STAWM) which uses transformational attention to learn a memory over each image it sees. The state of our Hebb-Rosenblatt memory is embedded in STAWM as the weights space of a layer. By projecting different queries through this layer we can obtain goal-oriented latent representations for tasks including classification and visual reconstruction. Our model obtains highly competitive classification performance on MNIST and CIFAR-10. As demonstrated through the CelebA dataset, to perform reconstruction the model learns to make a sequence of updates to a canvas which constitute a parts-based representation. Classification with the self supervised representation obtained from MNIST is shown to be in line with the state of the art models (none of which use a visual attention mechanism). Finally, we show that STAWM can be trained under the dual constraints of classification and reconstruction to provide an interpretable visual sketchpad which helps open the `black-box' of deep learning.

Una working memory biologicamente ispirata che può essere integrata in modelli di attention visiva ricorrente per prestazioni allo stato dell'arte

Introduce una nuova architettura di rete ispirata alla working memory visiva con attention e la applica a task di classificazione usandola come modello generativo

L'articolo aumenta il modello di attention ricorrente con un nuovo modello di working memory di Hebb-Rosenblatt e raggiunge risultati competitivi su MNIST


Generative models have been successfully applied to image style transfer and domain translation. However, there is still a wide gap in the quality of results when learning such tasks on musical audio. Furthermore, most translation models only enable one-to-one or one-to-many transfer by relying on separate encoders or decoders and complex, computationally-heavy models. In this paper, we introduce the Modulated Variational auto-Encoders (MoVE) to perform musical timbre transfer. First, we define timbre transfer as applying parts of the auditory properties of a musical instrument onto another. We show that we can achieve and improve this task by conditioning existing domain translation techniques with Feature-wise Linear Modulation (FiLM). Then, by replacing the usual adversarial translation criterion by a Maximum Mean Discrepancy (MMD) objective, we alleviate the need for an auxiliary pair of discriminative networks. This allows a faster and more stable training, along with a controllable latent space encoder. By further conditioning our system on several different instruments, we can generalize to many-to-many transfer within a single variational architecture able to perform multi-domain transfers. Our models map inputs to 3-dimensional representations, successfully translating timbre from one instrument to another and supporting sound synthesis on a reduced set of control parameters. We evaluate our method in reconstruction and generation tasks while analyzing the auditory descriptor distributions across transferred domains. We show that this architecture incorporates generative controls in multi-domain transfer, yet remaining rather light, fast to train and effective on small datasets.

L'articolo utilizza l'autoencoding variazionale e il condizionamento di rete per il transfer del timbro musicale, sviluppiamo e generalizziamo la nostra architettura per trasferimenti many-to-many tra strumenti insieme a visualizzazioni e valutazioni.

Propone un auto-Encoder Variazionale Modulato per eseguire il transfer del timbro musicale sostituendo il solito criterio di traduzione avversaria con una Maximum Mean Discrepancy

Descrive un modello many-to-many per il transfer del timbro musicale che si basa sui recenti sviluppi nel transfer del dominio e dello stile

Propone un modello ibrido basato su VAE per eseguire il transfer del timbro su registrazioni di strumenti musicali.


We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena when the depth becomes large. This, in particular, provides quantitative answers and insights to three questions that were yet fully understood in the literature. Firstly, we provide a precise answer on how the random deep weight-tied autoencoder model performs “approximate inference” as posed by Scellier et al. (2018), and its connection to reversibility considered by several theoretical studies. Secondly, we show that deep autoencoders display a higher degree of sensitivity to perturbations in the parameters, distinct from the shallow counterparts. Thirdly, we obtain insights on pitfalls in training initialization practice, and demonstrate experimentally that it is possible to train a deep autoencoder, even with the tanh activation and a depth as large as 200 layers, without resorting to techniques such as layer-wise pre-training or batch normalization. Our analysis is not specific to any depths or any Lipschitz activations, and our analytical techniques may have broader applicability.

Studiamo il comportamento degli autoencoder multilayer con pesi vincolati sotto l'ipotesi di pesi casuali. Attraverso una caratterizzazione esatta nel limite di grandi dimensioni, la nostra analisi rivela interessanti fenomeni di transizione di fase.

Un'analisi teorica degli autoencoder con pesi vincolati tra encoder e decoder (weight-tied) tramite l'analisi del campo medio

Analizza le prestazioni degli autoencoder con pesi vincolati basandosi sui recenti progressi nell'analisi dei problemi di statistica ad alta dimensione e in particolare, l'algoritmo di message passing

Questo articolo studia gli autoencoder sotto diverse ipotesi, e sottolinea che questo modello di autoencoder casuale può essere analizzato in modo elegante e rigoroso con equazioni monodimensionali.


Assessing distance betweeen the true and the sample distribution is a key component of many state of the art generative models, such as Wasserstein Autoencoder (WAE). Inspired by prior work on Sliced-Wasserstein Autoencoders (SWAE) and  kernel smoothing we construct a new generative model – Cramer-Wold AutoEncoder (CWAE). CWAE cost function, based on introduced Cramer-Wold distance between samples, has a simple closed-form in the case of normal prior. As a consequence, while simplifying the optimization procedure (no need of sampling necessary to evaluate the distance function in the training loop), CWAE performance matches quantitatively and qualitatively that of WAE-MMD (WAE using maximum mean discrepancy based distance function) and often improves upon SWAE.

Ispirati da lavori precedenti su Sliced-Wasserstein Autoencoders (SWAE) e kernel smoothing costruiamo un nuovo modello generativo - Cramer-Wold AutoEncoder (CWAE).

Questo articolo propone una variante WAE basata su una nuova distanza statistica tra la distribuzione dei dati codificati e la distribuzione a priori latente

Introduce una variazione dell'autoencoder di Wasserstein che è una nuova architettura di autoencoder regolarizzato che propone una scelta specifica della penalità di divergenza

Questo articolo propone l'autoencoder Cramer-Wold, che utilizza la distanza Cramer-Wold tra due distribuzioni basata sul teorema Cramer-Wold.


We propose a rejection sampling scheme using the discriminator of a GAN to  approximately correct errors in the GAN generator distribution. We show that  under quite strict assumptions, this will allow us to recover the data distribution  exactly. We then examine where those strict assumptions break down and design a  practical algorithm—called Discriminator Rejection Sampling (DRS)—that can be  used on real data-sets. Finally, we demonstrate the efficacy of DRS on a mixture of  Gaussians and on the state of the art SAGAN model. On ImageNet, we train an  improved baseline that increases the best published Inception Score from 52.52 to  62.36 and reduces the Frechet Inception Distance from 18.65 to 14.79. We then use  DRS to further improve on this baseline, improving the Inception Score to 76.08  and the FID to 13.75.

Usiamo un discriminatore GAN per eseguire uno schema di rejection sampling approssimato sull'output del generatore GAN.

Propone un algoritmo di rejection sampling per il sampling dal generatore GAN.

Questo articolo ha proposto uno schema di rejection sampling post-elaborazione per le GAN, chiamato Discriminator Rejection Sampling, per aiutare a filtrare i sample "buoni" dal generatore delle GAN.


The quality of the features used in visual recognition is of fundamental importance for the overall system. For a long time, low-level hand-designed feature algorithms as SIFT and HOG have obtained the best results on image recognition. Visual features have recently been extracted from trained convolutional neural networks. Despite the high-quality results, one of the main drawbacks of this approach, when compared with hand-designed features, is the training time required during the learning process. In this paper, we propose a simple and fast way to train supervised convolutional models to feature extraction while still maintaining its high-quality. This methodology is evaluated on different datasets and compared with state-of-the-art approaches.

Un metodo semplice e veloce per estrarre le feature visive dalle reti neurali convoluzionali

Propone un modo veloce per imparare feature convoluzionali che in seguito possono essere utilizzate con qualsiasi classificatore utilizzando un numero ridotto di epoch di allenamento e schedule delay del learning rate

Utilizzare uno schema di learning rate decay che è fisso rispetto al numero di epoche utilizzate nel training ed estrarre l'output del penultimo layer come feature per addestrare un classificatore convenzionale.


We develop a framework for understanding and improving recurrent neural networks (RNNs) using max-affine spline operators (MASOs). We prove that RNNs using piecewise affine and convex nonlinearities can be written as a simple piecewise affine spline operator. The resulting representation provides several new perspectives for analyzing RNNs, three of which we study in this paper. First, we show that an RNN internally partitions the input space during training and that it builds up the partition through time. Second, we show that the affine slope parameter of an RNN corresponds to an input-specific template, from which we can interpret an RNN as performing a simple template matching (matched filtering) given the input. Third, by carefully examining the MASO RNN affine mapping, we prove that using a random initial hidden state corresponds to an explicit L2 regularization of the affine parameters, which can mollify exploding gradients and improve generalization. Extensive experiments on several datasets of various modalities demonstrate and validate each of the above conclusions. In particular, using a random initial hidden states elevates simple RNNs to near state-of-the-art performers on these datasets.

Forniamo nuove intuizioni e interpretazioni delle RNN dal punto di vista degli operatori spline max-affine.

Riscrive le equazioni di Elman RNN in termini dei cosiddetti operatori spline max-affine

Fornire un nuovo approccio verso la comprensione delle RNN utilizzando operatori spline max-affline (MASO) riscrivendoli con attivazioni affini e convesse piecewise MASO

Gli autori si basano sull'interpetazione dell'operatore spline max-affine di una classe sostanziale di reti profonde, concentrandosi sulle reti neurali ricorrenti che utilizzano il rumore nell'hidden state iniziale come regolarizzazione


Reasoning over text and Knowledge Bases (KBs) is a major challenge for Artificial Intelligence, with applications in machine reading, dialogue, and question answering.  Transducing text to logical forms which can be operated on is a brittle and error-prone process . Operating directly on text by jointly learning representations and transformations thereof by means of neural architectures that lack the ability to learn and exploit general rules can be very data-inefficient and not generalise correctly . These issues are addressed by Neural Theorem Provers (NTPs) (Rocktäschel & Riedel, 2017), neuro-symbolic systems based on a continuous relaxation of Prolog’s backward chaining algorithm, where symbolic unification between atoms is replaced by a differentiable operator computing the similarity between their embedding representations . In this paper, we first propose Neighbourhood-approximated Neural Theorem Provers (NaNTPs) consisting of two extensions toNTPs, namely a) a method for drastically reducing the previously prohibitive time and space complexity during inference and learning, and b) an attention mechanism for improving the rule learning process, deeming them usable on real-world datasets. Then, we propose a novel approach for jointly reasoning over KB facts and textual mentions, by jointly embedding them in a shared embedding space. The proposed method is able to extract rules and provide explanations—involving both textual patterns and KB relations—from large KBs and text corpora. We show that NaNTPs perform on par with NTPs at a fraction of a cost, and can achieve competitive link prediction results on challenging large-scale datasets, including WN18, WN18RR, and FB15k-237 (with and without textual mentions) while being able to provide explanations for each prediction and extract interpretable rules.

Scaliamo i Neural Theorem Prover a grandi insiemi di dati, miglioriamo il processo di apprendimento delle regole e lo estendiamo per ragionare insieme su testo e knowledge base.

Propone un'estensione del sistema Neural Theorem Provers che affronta i problemi principali di questo modello riducendo la complessità temporale e spaziale del modello

Scala gli NTP usando la ricerca approssimativa dei nearest neighbour su fatti e regole durante l'unificazione e suggerisce di parametrizzare i predicati usando l'attention su predicati conosciuti

Migliora l'approccio Neural Theorem Prover proposto in precedenza utilizzando la nearest neighbor search.


We investigate the methods by which a Reservoir Computing Network (RCN) learns concepts such as 'similar' and 'different' between pairs of images using a small training dataset and generalizes these concepts to previously unseen types of data. Specifically, we show that an RCN trained to identify relationships between image-pairs drawn from a subset of digits from the MNIST database or the depth maps of subset of visual scenes from a moving camera generalizes the learned transformations to images of digits unseen during training or depth maps of different visual scenes. We infer, using Principal Component Analysis, that the high dimensional reservoir states generated from an input image pair with a specific transformation converge over time to a unique relationship. Thus, as opposed to training the entire high dimensional reservoir state, the reservoir only needs to train on these unique relationships, allowing the reservoir to perform well with very few training examples. Thus, generalization of learning to unseen images is interpretable in terms of clustering of the reservoir state onto the attractor corresponding to the transformation in reservoir space. We find that RCNs can identify and generalize linear and non-linear transformations, and combinations of transformations, naturally and be a robust and effective image classifier. Additionally, RCNs perform significantly better than state of the art neural network classification techniques such as deep Siamese Neural Networks (SNNs) in generalization tasks both on the MNIST dataset and more complex depth maps of visual scenes from a moving camera. This work helps bridge the gap between explainable machine learning and biological learning through analogies using small datasets, and points to new directions in the investigation of learning processes.

Generalizzazione delle relazioni apprese tra coppie di immagini utilizzando una piccola quantità di dati di training a tipi di immagini precedentemente non visti utilizzando un modello di sistemi dinamici spiegabili, Reservoir Computing, e una tecnica di apprendimento biologicamente plausibile basata sulle analogie.

Propone dei risultati di "combining transformations" nel contesto di RC utilizzando una echo-state network con accattivazioni tanh standard con la differenza che i pesi ricorrenti non sono addestrati

Nuovo metodo per classificare diverse distorsioni dei dati MNIST

L'articolo utilizza una echo-state network per imparare a classificare le trasformazioni di immagini tra coppie di immagini in una delle cinque classi.


We present Generative Adversarial Privacy and Fairness (GAPF), a data-driven framework for learning private and fair representations of the data. GAPF leverages recent advances in adversarial learning to allow a data holder to learn "universal" representations that decouple a set of sensitive attributes from the rest of the dataset. Under GAPF, finding the optimal decorrelation scheme is formulated as a constrained minimax game between a generative decorrelator and an adversary. We show that for appropriately chosen adversarial loss functions, GAPF provides privacy guarantees against strong information-theoretic adversaries and enforces demographic parity. We also evaluate the performance of GAPF on multi-dimensional Gaussian mixture models and real datasets, and show how a designer can certify that representations learned under an adversary with a fixed architecture perform well against more complex adversaries.

Presentiamo Generative Adversarial Privacy and Fairness (GAPF), un framework data-driven per l'apprendimento di rappresentazioni private e fair con garanzie certificate di privacy/fairness

Questo articolo usa un modello GAN per fornire una panoramica del lavoro correlato all'apprendimento di rappresentazioni private/fair (PRL).

Questo articolo presenta un approccio adversarial per rappresentazioni private e fair attraverso la distorsione appresa dei dati che minimizza la dipendenza dalle variabili sensibili mentre il grado di distorsione è vincolato.

Gli autori descrivono un framework relativo all'imparare una rappresentazione di parità demografica che può essere utilizzata per addestrare alcuni classificatori.


Current machine learning algorithms can be easily fooled by adversarial examples. One possible solution path is to make models that use confidence thresholding to avoid making mistakes. Such models refuse to make a prediction when they are not confident of their answer. We propose to evaluate such models in terms of tradeoff curves with the goal of high success rate on clean examples and low failure rate on adversarial examples. Existing untargeted attacks developed for models that do not use confidence thresholding tend to underestimate such models' vulnerability. We propose the MaxConfidence family of attacks, which are optimal in a variety of theoretical settings, including one realistic setting: attacks against linear models. Experiments show the attack attains good results in practice. We show that simple defenses are able to perform well on MNIST but not on CIFAR, contributing further to previous calls that MNIST should be retired as a benchmarking dataset for adversarial robustness research.   We release code for these evaluations as part of the cleverhans (Papernot et al 2018) library  (ICLR reviewers should be careful not to look at who contributed these features to cleverhans to avoid de-anonymizing this submission).

Presentiamo una metrica e un attacco ottimale per valutare i modelli che si difendono dagli adversarial example usando confidence thresholding

Questo articolo introduce una famiglia di attacchi agli algoritmi di confidence thresholding, concentrandosi principalmente sulle metodologie di valutazione.

Propone un metodo di valutazione per i modelli di difesa di confidence thresholding e un approccio per generare adversarial example scegliendo la classe sbagliata con più fiducia quando si usano attacchi mirati

L'articolo presenta una metodologia di valutazione degli attacchi ai metodi di confidence thresholding e propone un nuovo tipo di attacco.


Deep learning has achieved remarkable successes in solving challenging reinforcement learning (RL) problems when dense reward function is provided. However, in sparse reward environment it still often suffers from the need to carefully shape reward function to guide policy optimization. This limits the applicability of RL in the real world since both reinforcement learning and domain-specific knowledge are required. It is therefore of great practical importance to develop algorithms which can learn from a binary signal indicating successful task completion or other unshaped, sparse reward signals. We propose a novel method called competitive experience replay, which efficiently supplements a sparse reward by placing learning in the context of an exploration competition between a pair of agents. Our method complements the recently proposed hindsight experience replay (HER) by inducing an automatic exploratory curriculum. We evaluate our approach on the tasks of reaching various goal locations in an ant maze and manipulating objects with a robotic arm. Each task provides only binary rewards indicating whether or not the goal is achieved. Our method asymmetrically augments these sparse rewards for a pair of agents each learning the same task, creating a competitive game designed to drive exploration. Extensive experiments demonstrate that this method leads to faster converge and improved task performance.

Un nuovo metodo per imparare con reward sparse usando l'adversarial re-labeling della reward

Propone di utilizzare un ambiente competitivo multi-agente per incoraggiare l'esplorazione e mostra che CER + HER > HER ~ CER

Proporre un nuovo metodo per imparare da reward sparse in setting di reinforcement learning model-free e rendendo la reward densa

Per affrontare i problemi di reward sparsa e incoraggiare l'esplorazione negli algoritmi RL, gli autori propongono una strategia di relabeling chiamata Competitive Experience Reply (CER).


This paper proposes a neural end-to-end text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions. The model is formulated as a conditional generative model with two levels of hierarchical latent variables. The first level is a categorical variable, which represents attribute groups (e.g. clean/noisy) and provides interpretability. The second level, conditioned on the first, is a multivariate Gaussian variable, which characterizes specific attribute configurations (e.g. noise level, speaking rate) and enables disentangled fine-grained control over these attributes. This amounts to using a Gaussian mixture model (GMM) for the latent distribution. Extensive evaluation demonstrates its ability to control the aforementioned attributes. In particular, it is capable of consistently synthesizing high-quality clean speech regardless of the quality of the training data for the target speaker.

Costruire un modello TTS con Gaussian Mixture VAE permette un controllo preciso dello stile di conversazione, delle condizioni di rumore e altro.

Descrive il modello GAN condizionato per generare gli spettri di Mel condizionati dal parlante aumentando lo spazio z corrispondente all'identificazione

Questo articolo propone un modello con variabili latenti a due layer per ottenere una rappresentazione latente disentangled, facilitando così un preciso controllo su vari attributi

Questo articolo propone un modello che può controllare gli attributi non annotati, come lo stile di conversazione, l'accento, il rumore di fondo, ecc.


Visual Question Answering (VQA) models have struggled with counting objects in natural images so far. We identify a fundamental problem due to soft attention in these models as a cause. To circumvent this problem, we propose a neural network component that allows robust counting from object proposals. Experiments on a toy task show the effectiveness of this component and we obtain state-of-the-art accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model. On a difficult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.

Abilitare i modelli di Visual Question Answering a contare gestendo le proposte di oggetti sovrapposti.

Questo articolo propone un'architettura di rete progettata a mano su un grafo di proposte di oggetti per eseguire una non-maximum suppression per ottenere il conteggio degli oggetti.

Si concentra su un problema di conteggio nel visual question answering utilizzando il meccanismo dell'attention e propone un componente di conteggio differenziabile che conta esplicitamente il numero di oggetti

Questo articolo affronta il problema del conteggio degli oggetti nella visual question answering, propone molte euristiche per trovare il conteggio corretto.


We propose a simple and robust training-free approach for building sentence representations. Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and its surrounding context in a sentence. We model the semantic meaning of a word in a sentence based on two aspects. One is its relatedness to the word vector subspace already spanned by its contextual words. The other is its novel semantic meaning which shall be introduced as a new basis vector perpendicular to this existing subspace.   Following this motivation, we develop an innovative method based on orthogonal basis to combine pre-trained word embeddings into sentence representation. This approach requires zero training and zero parameters, along with efficient inference performance. We evaluate our approach on 11 downstream NLP tasks. Experimental results show that our model outperforms all existing zero-training alternatives in all the tasks and it is competitive to other approaches relying on either large amounts of labelled data or prolonged training time.

Un approccio semplice e senza training per l'embedding delle frasi con prestazioni competitive rispetto ai modelli sofisticati che richiedono una grande quantità di dati di training o un tempo di training prolungato.

Presentato un nuovo modo senza training di generare sentence embedding con un'analisi sistematica

Propone un nuovo metodo basato sulla geometria per l'embedding delle frasi dai word embedding, quantificando la novità, il significato e l'unicità nel corpus di ogni parola

Questo articolo esplora l'embedding delle frasi basato sulla decomposizione ortogonale dello spazio coperto dai word embedding


In few-shot classification, we are interested in learning algorithms that train a classifier from only a handful of labeled examples. Recent progress in few-shot classification has featured meta-learning, in which a parameterized model for a learning algorithm is defined and trained on episodes representing different classification problems, each with a small labeled training set and its corresponding test set. In this work, we advance this few-shot classification paradigm towards a scenario where unlabeled examples are also available within each episode. We consider two situations: one where all unlabeled examples are assumed to belong to the same set of classes as the labeled examples of the episode, as well as the more challenging situation where examples from other distractor classes are also provided. To address this paradigm, we propose novel extensions of Prototypical Networks (Snell et al., 2017) that are augmented with the ability to use unlabeled examples when producing prototypes. These models are trained in an end-to-end way on episodes, to learn to leverage the unlabeled examples successfully. We evaluate these methods on versions of the Omniglot and miniImageNet benchmarks, adapted to this new framework augmented with unlabeled examples. We also propose a new split of ImageNet, consisting of a large set of classes, with a hierarchical structure. Our experiments confirm that our Prototypical Networks can learn to improve their predictions due to unlabeled examples, much like a semi-supervised algorithm would.

Proponiamo nuove estensioni delle Prototypical Network che sono aumentate con la capacità di utilizzare esempi non annotati quando si producono prototipi.

Questo documento è un'estensione di una Prototypical Network che considera l'impiego degli esempi non annotati disponibili per aiutare il training in ogni episodio

Studia il problema della classificazione semi-supervisionata few-shot estendendo le reti prototipiche al setting dell'apprendimento semi-supervised con esempi di classi distrattrici

Estende la Prototypical Network al setting semi-supervisionato aggiornando i prototipi usando pseudo-label assegnate, trattando i distrattori e pesando i sample usando la distanza dai prototipi originali.


We investigate the properties of multidimensional probability distributions in the context of latent space prior distributions of implicit generative models. Our work revolves around the phenomena arising while decoding linear interpolations between two random latent vectors -- regions of latent space in close proximity to the origin of the space are oversampled, which restricts the usability of linear interpolations as a tool to analyse the latent space. We show that the distribution mismatch can be eliminated completely by a proper choice of the latent probability distribution or using non-linear interpolations. We prove that there is a trade off between the interpolation being linear, and the latent distribution having even the most basic properties required for stable training, such as finite mean. We use the multidimensional Cauchy distribution as an example of the prior distribution, and also provide a general method of creating non-linear interpolations, that is easily applicable to a large family of commonly used latent distributions.

Proviamo teoricamente che le interpolazioni lineari sono inadatte all'analisi dei modelli generativi impliciti addestrati.

Studia il problema di quando l'interpolante lineare tra due variabili casuali segue la stessa distribuzione, in relazione alla distribuzione a priori di un modello generativo implicito

Questo lavoro chiede come interpolare nello spazio latente dato un modello di variabile latente.


Deep neural networks (DNN) have shown promising performance in computer vision. In medical imaging, encouraging results have been achieved with deep learning for applications such as segmentation, lesion detection and classification. Nearly all of the deep learning based image analysis methods work on reconstructed images, which are obtained from original acquisitions via solving inverse problems (reconstruction). The reconstruction algorithms are designed for human observers, but not necessarily optimized for DNNs which can often observe features that are incomprehensible for human eyes. Hence, it is desirable to train the DNNs directly from the original data which lie in a different domain with the images. In this paper, we proposed an end-to-end DNN for abnormality detection in medical imaging. To align the acquisition with the annotations made by radiologists in the image domain, a DNN was built as the unrolled version of iterative reconstruction algorithms to map the acquisitions to images, and followed by a 3D convolutional neural network (CNN) to detect the abnormality in the reconstructed images. The two networks were trained jointly in order to optimize the entire DNN for the detection task from the original acquisitions. The DNN was implemented for lung nodule detection in low-dose chest computed tomography (CT), where a numerical simulation was done to generate acquisitions from 1,018 chest CT images with radiologists' annotations. The proposed end-to-end DNN demonstrated better sensitivity and accuracy for the task compared to a two-step approach, in which the reconstruction and detection DNNs were trained separately. A significant reduction of false positive rate on suspicious lesions were observed, which is crucial for the known over-diagnosis in low-dose lung CT imaging. The images reconstructed by the proposed end-to-end network also presented enhanced details in the region of interest.

Rilevamento del nodulo polmonare a partire da dati di proiezione piuttosto che da immagini.

Le DNN sono utilizzate per il rilevamento dei noduli polmonari basato su patch nei dati di proiezione CT.

Modellazione congiunta della ricostruzione della tomografia computerizzata e del rilevamento delle lesioni nel polmone addestrando la mappatura dal sinogramma grezzo agli output di rilevamento in modo end-to-end

Presenta un training end-to-end di un'architettura CNN che combina l'elaborazione del segnale dell'immagine CT e l'analisi dell'immagine.


Deep reinforcement learning (DRL) algorithms have demonstrated progress in learning to find a goal in challenging environments. As the title of the paper by Mirowski et al. (2016) suggests, one might assume that DRL-based algorithms are able to “learn to navigate” and are thus ready to replace classical mapping and path-planning algorithms, at least in simulated environments. Yet, from experiments and analysis in this earlier work, it is not clear what strategies are used by these algorithms in navigating the mazes and finding the goal. In this paper, we pose and study this underlying question: are DRL algorithms doing some form of mapping and/or path-planning? Our experiments show that the algorithms are not memorizing the maps of mazes at the testing stage but, rather, at the training stage. Hence, the DRL algorithms fall short of qualifying as mapping or path-planning algorithms with any reasonable definition of mapping. We extend the experiments in Mirowski et al. (2016) by separating the set of training and testing maps and by a more ablative coverage of the space of experiments. Our systematic experiments show that the NavA3C-D1-D2-L algorithm, when trained and tested on the same maps, is able to choose the shorter paths to the goal. However, when tested on unseen maps the algorithm utilizes a wall-following strategy to find the goal without doing any mapping or path planning.

Valutiamo quantitativamente e qualitativamente i metodi di navigazione basati sul deep reinforcement learning in una varietà di condizioni per rispondere alla domanda su quanto siano vicini a sostituire i pianificatori di percorso classici e gli algoritmi di mapping.

Valutare un modello basato su Deep RL su labirinti di allenamento misurando la repeated latency to goal e il confronto con il percorso più breve


In many robotic applications, it is crucial to maintain a belief about the state of   a system, like the location of a robot or the pose of an object.  These state estimates serve as input for planning and decision making and   provide feedback during task execution.   Recursive Bayesian Filtering algorithms address the state estimation problem,  but they require a model of the process dynamics and the sensory observations as well as   noise estimates that quantify the accuracy of these models.   Recently, multiple works have demonstrated that the process and sensor models can be   learned by end-to-end training through differentiable versions of Recursive Filtering methods.  However, even if the predictive models are known, finding suitable noise models   remains challenging. Therefore, many practical applications rely on very simplistic noise   models.   Our hypothesis is that end-to-end training through differentiable Bayesian   Filters enables us to learn more complex heteroscedastic noise models for  the system dynamics. We evaluate learning such models with different types of   filtering algorithms and on two different robotic tasks. Our experiments show that especially   for sampling-based filters like the Particle Filter, learning heteroscedastic noise   models can drastically improve the tracking performance in comparison to using   constant noise models.

Valutiamo l'apprendimento di modelli di rumore eteroschedastici all'interno di diversi filtri di Bayes differenziabili

Propone di apprendere modelli di rumore eteroscedastici dai dati ottimizzando la probabilità di predizione end-to-end attraverso filtri bayesiani differenziabili e due diverse versioni del filtro di Kalman unscented

Rivede i filtri di Bayes e valuta il beneficio di addestrare i modelli di osservazione e di rumore di processo mantenendo fissi tutti gli altri modelli

Questo articolo presenta un metodo per imparare e utilizzare il rumore dipendente dallo stato e dall'osservazione nei tradizionali algoritmi di filtraggio bayesiano. L'approccio consiste nel costruire un modello di rete neurale che prende come input i dati grezzi di osservazione e produce una rappresentazione compatta e una covarianza diagonale associata.


Graph convolutional neural networks have recently shown great potential for the task of zero-shot learning. These models are highly sample efficient as related concepts in the graph structure share statistical strength allowing generalization to new classes when faced with a lack of data. However, we find that the extensive use of Laplacian smoothing at each layer in current approaches can easily dilute the knowledge from distant nodes and consequently decrease the performance in zero-shot learning. In order to still enjoy the benefit brought by the graph structure while preventing the dilution of knowledge from distant nodes, we propose a Dense Graph Propagation (DGP) module with carefully designed direct links among distant nodes. DGP allows us to exploit the hierarchical graph structure of the knowledge graph through additional connections. These connections are added based on a node's relationship to its ancestors and descendants. A weighting scheme is further used to weigh their contribution depending on the distance to the node. Combined with finetuning of the representations in a two-stage training approach our method outperforms state-of-the-art zero-shot learning approaches.

Ripensiamo al modo in cui le informazioni possono essere sfruttate in modo più efficiente nel grafo della conoscenza per migliorare le prestazioni nel task di apprendimento Zero-Shot e proponiamo un modulo DGP (dense graph propagation) per questo scopo.

Questi autori propongono una soluzione al problema dell'over-smoothing nelle graph convolutional neural network permettendo una propagazione densa tra tutti i nodi correlati, ponderata dalla distanza reciproca.

Propone una nuova graph convolutional neural network per affrontare il problema della classificazione zero-shot utilizzando strutture relazionali tra le classi come input delle graph convolutional neural network per imparare classificatori per classi non viste


In this paper, we propose a capsule-based neural network model to solve the semantic segmentation problem. By taking advantage of the extractable part-whole dependencies available in capsule layers, we derive the probabilities of the class labels for individual capsules through a recursive, layer-by-layer procedure. We model this procedure as a traceback pipeline and take it as a central piece to build an end-to-end segmentation network. Under the proposed framework, image-level class labels and object boundaries are jointly sought in an explicit manner, which poses a significant advantage over the state-of-the-art fully convolutional network (FCN) solutions. Experiments conducted on modified MNIST and neuroimages demonstrate that our model considerably enhance the segmentation performance compared to the leading FCN variant.

Una segmentazione semantica basata su capsule, in cui le probabilità delle label di classe sono rintracciate attraverso la pipeline di capsule.

Gli autori presentano un meccanismo di tracciamento per associare il livello più basso di Capsule con le loro rispettive classi

Propone un layer di tracciamento per capsule network per fare la segmentazione semantica e fa uso esplicito della relazione parte-intero nei capsule layer

Propone un metodo trace-back basato sul concetto CapsNet di Sabour per eseguire una segmentazione semantica in parallelo alla classificazione.


Studying the evolution of information theoretic quantities during Stochastic Gradient Descent (SGD) learning of Artificial Neural Networks (ANNs) has gained popularity in recent years.   Nevertheless, these type of experiments require estimating mutual information and entropy which becomes intractable for moderately large problems. In this work we propose a framework for understanding SGD learning in the information plane which consists of observing entropy and conditional entropy of the output labels of ANN. Through experimental results and theoretical justifications it is shown that, under some assumptions, the SGD learning trajectories appear to be similar for different ANN architectures. First, the SGD learning is modeled as a Hidden Markov Process (HMP) whose entropy tends to increase to the maximum. Then, it is shown that the SGD learning trajectory appears to move close to the shortest path between the initial and final joint distributions in the space of probability measures equipped with the total variation metric. Furthermore, it is shown that the trajectory of learning in the information plane can provide an alternative for observing the learning process, with potentially richer information about the learning than the trajectories in training and test error.

Consideriamo SGD come una traiettoria nello spazio delle misure di probabilità, mostriamo la sua connessione con i processi di Markov, proponiamo un semplice modello di Markov per l'apprendimento di SGD e lo confrontiamo sperimentalmente con SGD utilizzando quantità dalla teoria dell'informazione.

Costruisce una catena di Markov che segue un percorso corto nella metrica TV su P e mostra che le traiettorie di SGD e \alpha-SMLC hanno entropia condizionale simile

Studia la traiettoria di H(\hat{y}) contro H(\hat{y}|y) sul piano dell'informazione per i metodi di stochastic gradient descent per il training delle reti neurali

Descrive SGD dal punto di vista della distribuzione p(y',y) dove y è una vera label di classe (eventualmente corrotta) e y' una predizione del modello.


Stochastic gradient Markov chain Monte Carlo (SG-MCMC) has become increasingly popular for simulating posterior samples in large-scale Bayesian modeling. However, existing SG-MCMC schemes are not tailored to any specific probabilistic model, even a simple modification of the underlying dynamical system requires significant physical intuition. This paper presents the first meta-learning algorithm that allows automated design for the underlying continuous dynamics of an SG-MCMC sampler. The learned sampler generalizes Hamiltonian dynamics with state-dependent drift and diffusion, enabling fast traversal and efficient exploration of energy landscapes. Experiments validate the proposed approach on Bayesian fully connected neural network, Bayesian convolutional neural network and Bayesian recurrent neural network tasks, showing that the learned sampler outperforms generic, hand-designed SG-MCMC algorithms, and generalizes to different datasets and larger architectures.

Questo articolo propone un metodo per automatizzare la progettazione della proposta MCMC a gradiente stocastico usando un approccio di meta learning.

Propone un approccio di meta-learning per progettare automaticamente un sampler MCMC basato sulla dinamica hamiltoniana per fare mixin più velocemente su problemi simili a quelli di allenamento

Parametrizza le matrici di diffusione e di curl tramite reti neurali e meta-apprende e ottimizza un algoritmo sg-mcmc.


We propose a new, multi-component energy function for energy-based Generative Adversarial Networks (GANs) based on methods from the image quality assessment literature. Our approach expands on the Boundary Equilibrium Generative Adversarial Network (BEGAN) by outlining some of the short-comings of the original energy and loss functions. We address these short-comings by incorporating an l1 score, the Gradient Magnitude Similarity score, and a chrominance score into the new energy function. We then provide a set of systematic experiments that explore its hyper-parameters. We show that each of the energy function's components is able to represent a slightly different set of features, which require their own evaluation criteria to assess whether they have been adequately learned. We show that models using the new energy function are able to produce better image representations than the BEGAN model in predicted ways.

Le tecniche di valutazione della qualità dell'immagine migliorano il training e la valutazione delle generative adversarial network basate sull'energia

Propone una formulazione basata sull'energia per il modello BEGAN e la modifica per includere un termine basato sulla valutazione della qualità dell'immagine

Propone alcune nuove funzioni energetiche nel BEGAN (boundary equilibrium GAN framework), tra cui l_1 score, Gradient magnitude similarity score, e chrominance score.