Across numerous applications, forecasting relies on numerical solvers for partial differential equations (PDEs).Although the use of deep-learning techniques has been proposed, the uses have been restricted by the fact the training data are obtained using PDE solvers.Thereby, the uses were limited to domains, where the PDE solver was applicable, but no further. We present methods for training on small domains, while applying the trained models on larger domains, with consistency constraints ensuring the solutions are physically meaningful even at the boundary of the small domains.We demonstrate the results on an air-pollution forecasting model for Dublin, Ireland.

Presentiamo RNN per il training di modelli surrogati di PDE, in cui i vincoli di coerenza assicurano che le soluzioni siano fisicamente significative, anche quando il training utilizza domini molto più piccoli di quelli a cui viene applicato il modello addestrato.


We address the issue of limit cycling behavior in training Generative Adversarial Networks and propose the use of Optimistic Mirror Decent (OMD) for training Wasserstein GANs.Recent theoretical results have shown that optimistic mirror decent (OMD) can enjoy faster regret rates in the context of zero-sum games.WGANs is exactly a context of solving a zero-sum game with simultaneous no-regret dynamics.  Moreover, we show that optimistic mirror decent addresses the limit cycling problem in training WGANs.We formally show that in the case of bi-linear zero-sum games the last iterate of OMD dynamics converges to an equilibrium, in contrast to GD dynamics which are bound to cycle.We also portray the huge qualitative difference between GD and OMD dynamics with toy examples, even when GD is modified with many adaptations proposed in the recent literature, such as gradient penalty or momentum.We apply OMD WGAN training to a bioinformatics problem of generating DNA sequences.We observe that models trained with OMD achieve consistently smaller KL divergence with respect to the true underlying distribution, than models trained with GD variants.Finally, we introduce a new algorithm, Optimistic Adam, which is an optimistic variant of Adam.We apply it to WGAN training on CIFAR10 and observe improved performance in terms of inception score as compared to Adam.

Proponiamo l'uso di mirror ottimistici sufficienti per affrontare i problemi di cycling nel training delle GAN. Introduciamo anche l'algoritmo Optimistic Adam

Questo articolo propone l'uso della mirror descent ottimistica per addestrare le WGAN

L'articolo propone di usare la gradient descent ottimistica per il training GAN che evita il cycling osservato con SGD e le sue varianti e fornisce risultati promettenti nel training GAN.

Questo articolo propone una semplice modifica del gradient descent standard, sostenendo di migliorare la convergenza delle GAN e di altri problemi di ottimizzazione min-max.


Learning good representations of users and items is crucially important to recommendation with implicit feedback.Matrix factorization is the basic idea to derive the representations of users and items by decomposing the given interaction matrix.However, existing matrix factorization based approaches share the limitation in that the interaction between user embedding and item embedding is only weakly enforced by fitting the given individual rating value, which may lose potentially useful information.In this paper, we propose a novel Augmented Generalized Matrix Factorization (AGMF) approach that is able to incorporate the historical interaction information of users and items for learning effective representations of users and items.Despite the simplicity of our proposed approach, extensive experiments on four public implicit feedback datasets demonstrate that our approach outperforms state-of-the-art counterparts.Furthermore, the ablation study demonstrates that by using multi-hot encoding to enrich user embedding and item embedding for Generalized Matrix Factorization, better performance, faster convergence, and lower training loss can be achieved.

Una semplice estensione della fattorizzazione matriciale generalizzata può superare gli approcci state-of-the-art per recommendation.

Il lavoro presenta un framework di fattorizzazione della matrice per imporre l'effetto dei dati storici quando si imparano le preferenze degli utenti nei setting di filtraggio collaborativo.


We propose an unsupervised method for building dynamic representations of sequential data, particularly of observed interactions.The method simultaneously acquires representations of input data and its dynamics.It is based on a hierarchical generative model composed of two levels.In the first level, a model learns representations to generate observed data.In the second level, representational states encode the dynamics of the lower one.The model is designed as a Bayesian network with switching variables represented in the higher level, and which generates transition models.The method actively explores the latent space guided by its knowledge and the uncertainty about it.That is achieved by updating the latent variables from prediction error signals backpropagated to the latent space.So, no encoder or inference models are used since the generators also serve as their inverse transformations.The method is evaluated in two scenarios, with static images and with videos.The results show that the adaptation over time leads to better performance than with similar architectures without temporal dependencies, e.g., variational autoencoders.With videos, it is shown that the system extracts the dynamics of the data in states that highly correlate with the ground truth of the actions observed.

Un metodo che costruisce rappresentazioni di dati sequenziali e le sue dinamiche attraverso modelli generativi con un processo attivo

Combina reti neurali e distribuzioni gaussiane per creare un'architettura e un modello generativo per immagini e video che minimizza l'errore tra le immagini generate e quelle fornite.

L'articolo propone un modello di rete bayesiana, realizzato come una rete neurale, che apprende diversi dati sotto forma di un sistema dinamico lineare


Activation is a nonlinearity function that plays a predominant role in the convergence and performance of deep neural networks.While Rectified Linear Unit (ReLU) is the most successful activation function, its derivatives have shown superior performance on benchmark datasets.In this work, we explore the polynomials as activation functions (order â‰¥ 2) that can approximate continuous real valued function within a given interval.Leveraging this property, the main idea is to learn the nonlinearity, accepting that the ensuing function may not be monotonic.While having the ability to learn more suitable nonlinearity, we cannot ignore the fact that it is a challenge to achieve stable performance due to exploding gradients - which is prominent with the increase in order.To handle this issue, we introduce dynamic input scaling, output scaling, and lower learning rate for the polynomial weights.Moreover, lower learning rate will control the abrupt fluctuations of the polynomials between weight updates.In experiments on three public datasets, our proposed method matches the performance of prior activation functions, thus providing insight into a networkâ€™s nonlinearity preference.

Proponiamo polinomi come funzioni di attivazione.

Gli autori introducono funzioni di attivazione learnable che sono parametrizzate da funzioni polinomiali e mostrano risultati leggermente migliori di ReLU.


We introduce CBF, an exploration method that works in the absence of rewards or end of episode signal.CBF is based on intrinsic reward derived from the error of a dynamics model operating in feature space.It was inspired by (Pathak et al., 2017), is easy to implement, and can achieve results such as passing four levels of Super Mario Bros, navigating VizDoom mazes and passing two levels of SpaceInvaders.We investigated the effect of combining the method with several auxiliary tasks, but find inconsistent improvements over the CBF baseline.

Un semplice metodo di motivazione intrinseca utilizzando l'errore del modello di dinamica forward nello spazio delle feature della policy.


This paper is concerned with the robustness of VAEs to adversarial attacks.We highlight that conventional VAEs are brittle under attack but that methods recently introduced for disentanglement such as Î²-TCVAE (Chen et al., 2018) improve robustness, as demonstrated through a variety of previously proposed adversarial attacks (Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al.(2018)).This motivated us to develop Seatbelt-VAE, a new hierarchical disentangled VAE that is designed to be significantly more robust to adversarial attacks than existing approaches, while retaining high quality reconstructions.

Mostriamo che i VAE disentangled sono più robusti dei VAE standard agli adversarial attack che mirano a ingannarli nella decodifica dell'input avversario verso un obiettivo scelto. Sviluppiamo poi un VAE gerarchico disentangled ancora più robusto, Seatbelt-VAE.

Gli autori propongono un nuovo modello VAE chiamato seatbelt-VAE, che dimostra di essere più robusto per gli attacchi latenti rispetto ai benchmark.


The backpropagation algorithm is the de-facto standard for credit assignment in artificial neural networks due to its empirical results.Since its conception, variants of the backpropagation algorithm have emerged.More specifically, variants that leverage function changes in the backpropagation equations to satisfy their specific requirements.Feedback Alignment is one such example, which replaces the weight transpose matrix in the backpropagation equations with a random matrix in search of a more biologically plausible credit assignment algorithm.In this work, we show that function changes in the  backpropagation procedure is equivalent to adding an implicit learning rate to an artificial neural network.Furthermore, we learn activation function derivatives in the backpropagation equations to demonstrate early convergence in these artificial neural networks.Our work reports competitive performances with early convergence on MNIST and CIFAR10 on sufficiently large deep neural network architectures.

Dimostriamo che i cambiamenti di funzione nella backpropagation sono equivalenti a un learning rate implicito


Unsupervised text style transfer is the task of re-writing text of a given style into a target style without using a parallel corpus of source style and target style sentences for training.Style transfer systems are evaluated on their ability to generate sentences that1) possess the target style,2) are fluent and natural sounding, and3) preserve the non-stylistic parts (content) of the source sentence.We train a reinforcement learning (RL) based unsupervised style transfer system that incorporates rewards for the above measures, and describe novel rewards shaping methods for the same.Our approach does not attempt to disentangle style and content, and leverages the power of massively pre-trained language models as well as the Transformer.Our system significantly outperforms existing state-of-art systems based on human as well as automatic evaluations on target style, fluency and content preservation as well as on overall success of style transfer, on a variety of datasets.

Un approccio di reinforcement learning al text style transfer

Introduce un metodo basato su RL che fa leva su un language model pre-trained per trasferire lo stile del testo, senza un obiettivo di disentanglement, utilizzando style-transfer generation da un altro modello.

Gli autori propongono una reward combinata composta da fluidità, contenuto e stile per il text style transfer.


Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what networks have learned inside the deep generative representations and how photo-realistic images are able to be composed from random noises.In this work, we show that highly-structured semantic hierarchy emerges from the generative representations as the variation factors for synthesizing scenes.By probing the layer-wise representations with a broad set of visual concepts at different abstraction levels, we are able to quantify the causality between the activations and the semantics occurring in the output image.Such a quantification identifies the human-understandable variation factors learned by GANs to compose scenes.The qualitative and quantitative results suggest that the generative representations learned by GAN are specialized to synthesize different hierarchical semantics: the early layers tend to determine the spatial layout and configuration, the middle layers control the categorical objects, and the later layers finally render the scene attributes as well as color scheme.Identifying such a set of manipulatable latent semantics facilitates semantic scene manipulation.

Mostriamo che la gerarchia semantica altamente strutturata emerge nelle rappresentazioni generative deep come risultato per sintetizzare le scene.

Il paper studia gli aspetti codificati dalle variabili latenti in ingresso nei diversi layer di StyleGAN.

L'articolo presenta un'interpretazione visivamente guidata delle attivazioni dei layer di convoluzione nel generatore di StyleGAN su layout, categoria di scena, attributi di scena e colore.


Variational autoencoders (VAEs) defined over SMILES string and graph-based representations of molecules promise to improve the optimization of molecular properties, thereby revolutionizing the pharmaceuticals and materials industries.However, these VAEs are hindered by the non-unique nature of SMILES strings and the computational cost of graph convolutions.To efficiently pass messages along all paths through the molecular graph, we encode multiple SMILES strings of a single molecule using a set of stacked recurrent neural networks, harmonizing hidden representations of each atom between SMILES representations, and use attentional pooling to build a final fixed-length latent representation.By then decoding to a disjoint set of SMILES strings of the molecule, our All SMILES VAE learns an almost bijective mapping between molecules and latent representations near the high-probability-mass subspace of the prior.Our SMILES-derived but molecule-based latent representations significantly surpass the state-of-the-art in a variety of fully- and semi-supervised property regression and molecular property optimization tasks.

Mettiamo in comune i messaggi tra più stringhe SMILES della stessa molecola per passare le informazioni lungo tutti i percorsi attraverso il grafo molecolare, producendo rappresentazioni latenti che superano significativamente lo stato dell'arte in una varietà di task.

Il metodo utilizza input multipli di stringhe SMILES, la fusione delle feature in base ai caratteri attraverso queste stringhe e il training della rete attraverso target multipli di output di stringhe SMILES, creando una robusta rappresentazione latente a lunghezza fissa indipendente dalla variazione SMILES.

Gli autori descrivono un nuovo metodo di autoencoder variazionale per le molecole che codifica le molecole come stringhe per ridurre le operazioni necessarie per condividere le informazioni tra gli atomi nella molecola.


We propose a simple yet highly effective method that addresses the mode-collapse problem in the Conditional Generative  Adversarial  Network (cGAN).Although conditional distributions are multi-modal (i.e., having many modes) in practice, most cGAN approaches tend to learn an overly simplified distribution where an input is always mapped to a single output regardless of variations in latent code.To address such issue, we propose to explicitly regularize the generator to produce diverse outputs depending on latent codes.The proposed regularization is simple, general, and can be easily integrated into most conditional GAN objectives.Additionally, explicit regularization on generator allows our method to control a balance between visual quality and diversity.We demonstrate the effectiveness of our method on three conditional generation tasks: image-to-image translation, image inpainting, and future video prediction.We show that simple addition of our regularization to existing models leads to surprisingly diverse generations, substantially outperforming the previous approaches for multi-modal conditional generation specifically designed in each individual task.

Proponiamo un approccio semplice e generale che evita un mode collapse in varie GAN condizionali.

L'articolo propone un termine di regolarizzazione per l'obiettivo GAN condizionale al fine di promuovere una generazione multimodale diversa e prevenire il mode collapse.

L'articolo propone un metodo per generare diversi output per vari framework GAN condizionali tra cui la traduzione image-to-image, l'image-inpainting e la predizione video, che può essere applicato a vari framework di sintesi condizionale per vari task.


The transformer is a state-of-the-art neural translation model that uses attention to iteratively refine lexical representations with information drawn from the surrounding context.Lexical features are fed into the first layer and propagated through a deep network of hidden layers.We argue that the need to represent and propagate lexical features in each layer limits the modelâ€™s capacity for learning and representing other information relevant to the task.To alleviate this bottleneck, we introduce gated shortcut connections between the embedding layer and each subsequent layer within the encoder and decoder.This enables the model to access relevant lexical content dynamically, without expending limited resources on storing it within intermediate states.We show that the proposed modification yields consistent improvements on standard WMT translation tasks and reduces the amount of lexical information passed along the hidden layers.We furthermore evaluate different ways to integrate lexical connections into the transformer architecture and present ablation experiments exploring the effect of proposed shortcuts on model behavior.

Dotare il transformer di shortcut per il layer di embedding libera capacità del modello per imparare nuove informazioni.


Probability density estimation is a classical and well studied problem, but standard density estimation methods have historically lacked the power to model complex and high-dimensional image distributions.  More recent generative models leverage the power of neural networks to implicitly learn and represent probability models over complex images.  We describe methods to extract explicit probability density estimates from GANs, and explore the properties of these image density functions.  We perform sanity check experiments to provide evidence that these probabilities are reasonable.  However, we also show that density functions of natural images are difficult to interpret and thus limited in use.  We study reasons for this lack of interpretability, and suggest that we can get better interpretability by doing density estimation on latent representations of images.

Esaminiamo la relazione tra i valori di densità di probabilità e il contenuto dell'immagine nelle GAN non invertibili.

Gli autori cercano di stimare la distribuzione di probabilità dell'immagine con l'aiuto di GAN e sviluppano un'approssimazione adeguata alle PDF nello spazio latente.


Convolutional Neural Networks (CNNs) are composed of multiple convolution layers and show elegant performance in vision tasks.The design of the regular convolution is based on the Receptive Field (RF) where the information within a specific region is processed.In the view of the regular convolution's RF, the outputs of neurons in lower layers with smaller RF are bundled to create neurons in higher layers with larger RF. As a result, the neurons in high layers are able to capture the global context even though the neurons in low layers only see the local information.However, in lower layers of the biological brain, the information outside of the RF changes the properties of neurons.In this work, we extend the regular convolution and propose spatially shuffled convolution (ss convolution).In ss convolution, the regular convolution is able to use the information outside of its RF by spatial shuffling which is a simple and lightweight operation.We perform experiments on CIFAR-10 and ImageNet-1k dataset, and show that ss convolution improves the classification performance across various CNNs.

Proponiamo una convoluzione spaziale rimescolata che la convoluzione regolare incorpora le informazioni dall'esterno del suo campo recettivo.

Propone la convoluzione SS che utilizza informazioni al di fuori della sua RF, mostrando risultati migliori quando viene testata su più modelli CNN.

Gli autori hanno proposto una strategia di shuffling per i layer di convoluzione nelle reti neurali convoluzionali.


We propose a framework to model the distribution of sequential data coming froma set of entities connected in a graph with a known topology.The method isbased on a mixture of shared hidden Markov models (HMMs), which are trainedin order to exploit the knowledge of the graph structure and in such a way that theobtained mixtures tend to be sparse.Experiments in different application domainsdemonstrate the effectiveness and versatility of the method.

Un metodo per modellare la distribuzione generativa di sequenze provenienti da entità connesse a grafi.

Gli autori propongono un metodo per modellare i dati sequenziali da fonti multiple interconnesse utilizzando una miscela di pool comune di HMM.


To gain high rewards in muti-agent scenes, it is sometimes necessary to understand other agents and make corresponding optimal decisions.We can solve these tasks by first building models for other agents and then finding the optimal policy with these models.To get an accurate model, many observations are needed and this can be sample-inefficient.What's more, the learned model and policy can overfit to current agents and cannot generalize if the other agents are replaced by new agents.In many practical situations, each agent we face can be considered as a sample from a population with a fixed but unknown distribution.Thus we can treat the task against some specific agents as a task sampled from a task distribution.We apply meta-learning method to build models and learn policies.Therefore when new agents come, we can adapt to them efficiently.Experiments on grid games show that our method can quickly get high rewards.

Il nostro lavoro applica il meta learning al multi-agent Reinforcement Learning per aiutare il nostro agente ad adattarsi in modo efficiente ai nuovi avversari in arrivo.

Questo articolo si concentra sull'adattamento veloce al nuovo comportamento degli altri agenti dell'ambiente utilizzando un metodo basato su MAML

L'articolo presenta un approccio all'apprendimento multi-agente basato sul framework del meta learning agnostico per il task di modellazione dell'avversario per il RL multi-agente.


We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation.  This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball.We show that this is an effective regularizer;  for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2% to 5.3%.

Caratterizziamo i valori singolari della trasformata lineare associata a un layer convoluzionario standard 2D multicanale, permettendo il loro calcolo efficiente.

L'articolo è dedicato al calcolo dei valori singolari dei layer convoluzionali

Deriva formule esatte per il calcolo dei valori singolari dei layer di convoluzione delle deep neural network e mostra che il calcolo dei valori singolari può essere fatto molto più velocemente del calcolo del SVD completo della matrice di convoluzione facendo appello alle trasformazioni FFT veloci.


Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning.A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment.Computing a Bayes-optimal policy is however intractable for all but the smallest tasks.In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection.In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty.We also evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher return during training than existing methods.

VariBAD apre una strada all'esplorazione approssimativa Bayes-ottimale trattabile per il deep RL usando idee dal meta learning, del RL bayesiano e dall'inferenza variazionale approssimata.

Questo articolo presenta un nuovo metodo di deep reinforcement learning che può fare un trade-off in modo efficiente tra exploration e exploitation combinando meta learning, inferenza variazionale e RL bayesiana.


In a continual learning setting, new categories may be introduced over time, and an ideal learning system should perform well on both the original categories and the new categories.While deep neural nets have achieved resounding success in the classical setting, they are known to forget about knowledge acquired in prior episodes of learning if the examples encountered in the current episode of learning are drastically different from those encountered in prior episodes.This makes deep neural nets ill-suited to continual learning.In this paper, we propose a new model that can both leverage the expressive power of deep neural nets and is resilient to forgetting when new categories are introduced.We demonstrate an improvement in terms of accuracy on original classes compared to a vanilla deep neural net.

Mostriamo che il metric learning può aiutare a ridurre il catastrophic forgetting

Questo articolo applica il metric learning per ridurre il catastrophic forgetting sulle reti neurali migliorando l'espressività del layer finale, portando a migliori risultati nel continual learning.


Biomedical knowledge bases are crucial in modern data-driven biomedical sciences, but auto-mated biomedical knowledge base construction remains challenging.In this paper, we consider the problem of disease entity normalization, an essential task in constructing a biomedical knowledge base.  We present NormCo, a deep coherence model which considers the semantics of an entity mention, as well as the topical coherence of the mentions within a single document.NormCo mod-els entity mentions using a simple semantic model which composes phrase representations from word embeddings, and treats coherence as a disease concept co-mention sequence using an RNN rather than modeling the joint probability of all concepts in a document, which requires NP-hard inference.  To overcome the issue of data sparsity, we used distantly supervised data and synthetic data generated from priors derived from the BioASQ dataset.  Our experimental results show thatNormCo outperforms state-of-the-art baseline methods on two disease normalization corpora in terms of (1) prediction quality and (2) efficiency, and is at least as performant in terms of accuracy and F1 score on tagged documents.

Presentiamo NormCo, un modello di deep coherence che considera la semantica di un'entity mention, così come la topical coherence delle mention all'interno di un singolo documento per eseguire la normalizzazione delle disease entity.

Utilizza un autoencoder GRU per rappresentare il "contesto" (entità correlate di una data malattia nell'arco di una frase), risolvendo il task BioNLP con miglioramenti significativi rispetto ai metodi più noti.


We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others.Multiplicative interaction layers as primitive operations have a long-established presence in the literature, though this often not emphasized and thus under-appreciated.We begin by showing that such layers strictly enrich the representable function classes of neural networks.We conjecture that multiplicative interactions offer a particularly powerful inductive bias when fusing multiple streams of information or when conditional computation is required.We therefore argue that they should be considered in many situation where multiple compute or information paths need to be combined, in place of the simple and oft-used concatenation operation.Finally, we back up our claims and demonstrate the potential of multiplicative interactions by applying them in large-scale complex RL and sequence modelling tasks, where their use allows us to deliver state-of-the-art results, and thereby provides new evidence in support of multiplicative interactions playing a more prominent role when designing new neural network architectures.

Esploriamo il ruolo dell'interazione moltiplicativa come framework unificante per descrivere una serie di pattern architetturali di reti neurali classiche e moderne, come il gating, i layer di attention, le iperreti e le convoluzioni dinamiche.

Presenta l'interazione moltiplicativa come caratterizzazione unificata per rappresentare i componenti di progettazione dell'architettura del modello comunemente usati, mostrando prove empiriche di prestazioni superiori in task come la modellazione di RL e sequenze.

L'articolo esplora diversi tipi di interazioni moltiplicative e trova modelli MI in grado di raggiungere una performance allo stato dell'arte nel language modelling e nei problemi di reinforcement learning.


Developing conditional generative models for text-to-video synthesis is an extremely challenging yet an important topic of research in machine learning.In this work, we address this problem by introducing Text-Filter conditioning Generative Adversarial Network (TFGAN), a GAN model with novel conditioning scheme that aids improving the text-video associations.With a combination of this conditioning scheme and a deep GAN architecture, TFGAN generates photo-realistic videos from text on very challenging real-world video datasets.In addition, we construct a benchmark synthetic dataset of moving shapes to systematically evaluate our conditioning scheme.Extensive experiments demonstrate that TFGAN significantly outperforms the existing approaches, and can also generate videos of novel categories not seen during training.

Un'efficace framework GAN di text-conditioning per la generazione di video dal testo

Questo articolo presenta un metodo basato su GAN per la generazione di video condizionati dalla descrizione del testo, con un nuovo metodo di condizionamento che genera filtri di convoluzione dal testo codificato e li usa per una convoluzione nel discriminatore.

Questo articolo propone modelli GAN condizionali per la sintesi testo-video: sviluppando filtri CNN condizionati dalle feature del testo e costruendo dataset moving-shape con prestazioni migliorate nella generazione di video/immagini.


Over-parameterization is ubiquitous nowadays in training neural networks to benefit both optimization in seeking global optima and generalization in reducing prediction error.However, compressive networks are desired in many real world applications and direct training of small networks may be trapped in local optima.In this paper, instead of pruning or distilling over-parameterized models to compressive ones, we propose a new approach based on \emph{differential inclusions of inverse scale spaces}, that generates a family of models from simple to complex ones by coupling gradient descent and mirror descent to explore model structural sparsity.It has a simple discretization, called the Split Linearized Bregman Iteration (SplitLBI), whose global convergence analysis in deep learning is established that from any initializations, algorithmic iterations converge to a critical point of empirical risks.Experimental evidence shows that\ SplitLBI may achieve state-of-the-art performance in large scale training on ImageNet-2012 dataset etc., while with \emph{early stopping} it unveils effective subnet architecture with comparable test accuracies to dense models after retraining instead of pruning well-trained ones.

SplitLBI è applicato al deep learning per esplorare la sparsità strutturale del modello, raggiungendo prestazioni allo stato dell'arte in ImageNet-2012 e svelando un'efficace architettura di subnet.

Propone un algoritmo basato sull'ottimizzazione per trovare importanti strutture sparse di reti neurali su larga scala accoppiando l'apprendimento della matrice dei pesi e i vincoli di sparsità, offrendo una convergenza garantita su problemi di ottimizzazione non convessi.


In this paper, we study the learned iterative shrinkage thresholding algorithm (LISTA) for solving sparse coding problems.  Following assumptions made by prior works, we first discover that the code components in its estimations may be lower than expected, i.e., require gains, and to address this problem, a gated mechanism amenable to theoretical analysis is then introduced.Specific design of the gates is inspired by convergence analyses of the mechanism and hence its effectiveness can be formally guaranteed.In addition to the gain gates, we further introduce overshoot gates for compensating insufficient step size in LISTA.Extensive empirical results confirm our theoretical findings and verify the effectiveness of our method.

Proponiamo meccanismi gated per migliorare l'ISTA appreso per la codifica sparsa, con garanzie teoriche sulla superiorità del metodo.

Propone estensioni di LISTA che affrontano la sottostima introducendo "gain gates" e includendo il momento con "overshoot gates", mostrando tassi di convergenza migliorati.

Questo articolo si concentra sulla soluzione di problemi di codifica sparsa usando reti di tipo LISTA, proponendo una "gain gating function" per mitigare la debolezza dell'ipotesi "nessun falso positivo".


The learning of hierarchical representations for image classification has experienced an impressive series of successes due in part to the availability of large-scale labeled data for training.On the other hand, the trained classifiers have traditionally been evaluated on a handful of test images, which are deemed to be extremely sparsely distributed in the space of all natural images.It is thus questionable whether recent performance improvements on the excessively re-used test sets generalize to real-world natural images with much richer content variations.In addition, studies on adversarial learning show that it is effortless to construct adversarial examples that fool nearly all image classifiers, adding more complications to relative performance comparison of existing models.This work presents an efficient framework for comparing image classifiers, which we name the MAximum Discrepancy (MAD) competition.Rather than comparing image classifiers on fixed test sets, we adaptively sample a test set from an arbitrarily large corpus of unlabeled images so as to maximize the discrepancies between the classifiers, measured by the distance over WordNet hierarchy.Human labeling on the resulting small and model-dependent image sets reveals the relative performance of the competing classifiers and provides useful insights on potential ways to improve them.We report the MAD competition results of eleven ImageNet classifiers while noting that the framework is readily extensible and cost-effective to add future classifiers into the competition.

Presentiamo un framework efficiente e adattivo per confrontare i classificatori di immagini al fine di massimizzare le discrepanze tra i classificatori, invece di confrontarli su test set fissi.

Meccanismo di individuazione degli errori che confronta i classificatori di immagini creando un test set dove sono più in disaccordo, misurando il disaccordo attraverso una distanza semantica derivata dall'ontologia WordNet.


Robustness of neural networks has recently been highlighted by the adversarial examples, i.e., inputs added with well-designed  perturbations which are imperceptible to humans but can cause the network to give incorrect outputs.In this paper, we design a new CNN architecture that by itself has good robustness.We introduce a simple but powerful technique, Random Mask, to modify existing CNN structures.We show that CNN with Random Mask achieves state-of-the-art performance against black-box adversarial attacks without applying any adversarial training.We next investigate the adversarial examples which â€œfoolâ€ a CNN with Random Mask.Surprisingly, we find that these adversarial examples often â€œfoolâ€ humans as well.This raises fundamental questions on how to define adversarial examples and robustness properly.

Proponiamo una tecnica che modifica le strutture CNN per migliorare la robustezza mantenendo un'alta accuratezza di test, e solleviamo dubbi sull'adeguatezza dell'attuale definizione di adversarial example generando adversarial example in grado di ingannare gli umani.

Questo articolo propone una tecnica semplice per migliorare la robustezza delle reti neurali contro gli attacchi black-box.

Gli autori propongono un metodo semplice per aumentare la robustezza delle reti neurali convoluzionali contro gli adversarial example, con risultati sorprendentemente buoni.


Supervised deep learning methods require cleanly labeled large-scale datasets, but collecting such data is difficult and sometimes impossible.There exist two popular frameworks to alleviate this problem: semi-supervised learning and robust learning to label noise.Although these frameworks relax the restriction of supervised learning, they are studied independently.Hence, the training scheme that is suitable when only small cleanly-labeled data are available remains unknown.In this study, we consider learning from bi-quality data as a generalization of these studies, in which a small portion of data is cleanly labeled, and the rest is corrupt.Under this framework, we compare recent algorithms for semi-supervised and robust learning.The results suggest that semi-supervised learning outperforms robust learning with noisy labels.We also propose a training strategy for mixing mixup techniques to learn from such bi-quality data effectively.

Proponiamo di confrontare l'apprendimento semi-supervised e robusto per le label rumorose in un setting condiviso

Gli autori propongono una strategia basata sul mixup per il training di un modello in un setting formale che include i task di apprendimento semi-supervised e robusto come casi speciali.


Hierarchical Sparse Coding (HSC) is a powerful model to efficiently represent multi-dimensional, structured data such as images.The simplest solution to solve this computationally hard problem is to decompose it into independent layerwise subproblems.However, neuroscientific evidence would suggest inter-connecting these subproblems as in the Predictive Coding (PC) theory, which adds top-down connections between consecutive layers.In this study, a new model called Sparse Deep Predictive Coding (SDPC) is introduced to assess the impact of this inter-layer feedback connection.In particular, the SDPC is compared with a Hierarchical Lasso (Hi-La) network made out of a sequence of Lasso layers.A 2-layered SDPC and a Hi-La networks are trained on 3 different databases and with different sparsity parameters on each layer.First, we show that the overall prediction error generated by SDPC is lower thanks to the feedback mechanism as it transfers prediction error between layers.Second, we demonstrate that the inference stage of the SDPC is faster to converge than for the Hi-La model.Third, we show that the SDPC also accelerates the learning process.Finally, the qualitative analysis of both models dictionaries, supported by their activation probability, show that the SDPC features are more generic and informative.

Questo articolo dimostra sperimentalmente l'effetto benefico delle connessioni top-down nell'algoritmo Hierarchical Sparse Coding.

Questo articolo presenta uno studio che confronta le tecniche di Hierarchical Sparse Coding, mostrando che il termine top-down è vantaggioso nel ridurre l'errore predittivo e può imparare più velocemente.


Explaining a deep learning model can help users understand its behavior and allow researchers to discern its shortcomings.Recent work has primarily focused on explaining models for tasks like image classification or visual question answering.  In this paper, we introduce an explanation approach for image similarity models, where a model's output is a score measuring the similarity of two inputs rather than a classification.  In this task, an explanation depends on both of the input images, so standard methods do not apply.We propose an explanation method that pairs a saliency map identifying important image regions with an attribute that best explains the match.  We find that our explanations provide additional information not typically captured by saliency maps alone, and can also improve performance on the classic task of attribute recognition.Our approach's ability to generalize is demonstrated on two datasets from diverse domains, Polyvore Outfits and Animals with Attributes 2.

Un approccio a scatola nera per spiegare le previsioni di un modello di image similarity.

Introduce un metodo per la spiegazione del modello di image similarity che identifica gli attributi che contribuiscono positivamente al punteggio di somiglianza e li accoppia con una mappa generata di saliency.

L'articolo propone un meccanismo di explanation che accoppia le regioni tipiche della mappa di saliency insieme agli attributi per le deep neural network per similarity matching.


Adversarial examples have been shown to be an effective way of assessing the robustness of neural sequence-to-sequence (seq2seq) models, by applying perturbations to the input of a model leading to large degradation in performance.However, these perturbations are only indicative of a weakness in the model if they do not change the semantics of the input in a way that would change the expected output.Using the example of machine translation (MT), we propose a new evaluation framework for adversarial attacks on seq2seq models taking meaning preservation into account and demonstrate that existing methods may not preserve meaning in general.Based on these findings, we propose new constraints for attacks on word-based MT systems and show, via human and automatic evaluation, that they produce more semantically similar adversarial inputs.Furthermore, we show that performing adversarial training with meaning-preserving attacks is beneficial to the model in terms of adversarial robustness without hurting test performance.

Come dovreste valutare gli adversarial attack su seq2seq

Gli autori studiano modi di generare adversarial example, mostrando che l'adversarial training con l'attacco più coerente con i criteri meaning-preserving introdotti risulta in una migliore robustezza a questo tipo di attacco senza degradazione nel setting non-adversarial.

L'articolo riguarda le adversarial perturbation meaning-preserving che preservano il significato nel contesto dei modelli Seq2Seq


We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs.The proposed technique keeps the contribution of positive and negative weights to the layer output in equilibrium.We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.

Una tecnica di normalizzazione alternativa alla batch normalization

Introduce una tecnica di normalizzazione, che normalizza i pesi dei layer convoluzionali.

Questo manoscritto introduce una nuova trasformazione per i layer, EquiNorm, per migliorare la batch normalization che non modifica gli input ai layer ma piuttosto i pesi dei layer.


We present a framework for building unsupervised representations of entities and their compositions, where each entity is viewed as a probability distribution rather than a fixed length vector.In particular, this distribution is supported over the contexts which co-occur with the entity and are embedded in a suitable low-dimensional space.This enables us to consider the problem of representation learning with a perspective from Optimal Transport and take advantage of its numerous tools such as Wasserstein distance and Wasserstein barycenters.We elaborate how the method can be applied for obtaining unsupervised representations of text and illustrate the performance quantitatively as well as qualitatively on tasks such as measuring sentence similarity and word entailment, where we empirically observe significant gains (e.g., 4.1% relative improvement over Sent2vec and GenSen).The key benefits of the proposed approach include:(a) capturing uncertainty and polysemy via modeling the entities as distributions,(b) utilizing the underlying geometry of the particular task (with the ground cost),(c) simultaneously providing interpretability with the notion of optimal transport between contexts and(d) easy applicability on top of existing point embedding methods.In essence, the framework can be useful for any unsupervised or supervised problem (on text or other modalities); and only requires a co-occurrence structure inherent to many problems.The code, as well as pre-built histograms, are available under https://github.com/context-mover.

Rappresentare ogni entità come una distribuzione di probabilità su contesti codificati in un ground space.

Propone di costruire embedding di parole da un istogramma su parole di contesto, invece che come point vector, che permette di misurare le distanze tra due parole in termini di trasporto ottimale tra gli istogrammi attraverso un metodo che aumenta la rappresentazione di un'entità da "punto in uno spazio vettoriale" standard a un istogramma con bins situati in alcuni punti di quello spazio vettoriale.


Over the last few years, the phenomenon of adversarial examples --- maliciously constructed inputs that fool trained machine learning models --- has captured the attention of the research community, especially when the adversary is restricted to making small modifications of a correctly handled input.At the same time, less surprisingly, image classifiers lack human-level performance on randomly corrupted images, such as images with additive Gaussian noise.In this work, we show that these are two manifestations of the same underlying phenomenon.We establish this connection in several ways.First, we find that adversarial examples exist at the same distance scales we would expect from a linear model with the same performance on corrupted images.Next, we show that Gaussian data augmentation during training improves robustness to small adversarial perturbations and that adversarial training improves robustness to several types of image corruptions.Finally, we present a model-independent upper bound on the distance from a corrupted image to its nearest error given test performance and show that in practice we already come close to achieving the bound, so that improving robustness further for the corrupted image distribution requires significantly reducing test error.All of this suggests that improving adversarial robustness should go hand in hand with improving performance in the presence of more general and realistic image corruptions.This yields a computationally tractable evaluation metric for defenses to consider: test error in noisy image distributions.

Piccole perturbazioni adversarial dovrebbero essere attese dati i tassi di errore osservati nei modelli al di fuori della distribuzione naturale dei dati.

Questo articolo propone una visione alternativa per gli adversarial example in spazi ad alta dimensione, considerando il "tasso di errore" in una distribuzione gaussiana centrata su ogni test point.


Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training.Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019).However, surprisingly,  the simple baseline of just pre-training and fine-tuning compact models has been overlooked.In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work.Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation.The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements.Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data.One surprising observation is that they have a compound effect even when sequentially applied on the same data.To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.

Studia come l'apprendimento self-supervised e la knowledge distillation interagiscono nel contesto della costruzione di modelli compatti.

Studia il training di language model pre-trained tramite la distillation e mostra che l'uso di un teacher per distillare un modello compatto di student funziona meglio del pre-training diretto del modello.

Questa presentazione mostra che il pre-training su masked language modelling di uno student è meglio della distillation, e il meglio è combinare entrambi e distillare da quel modello uno student pretrained.


In this paper, we investigate lossy compression of deep neural networks (DNNs) by weight quantization and lossless source coding for memory-efficient deployment.Whereas the previous work addressed non-universal scalar quantization and entropy coding of DNN weights, we for the first time introduce universal DNN compression by universal vector quantization and universal source coding.In particular, we examine universal randomized lattice quantization of DNNs, which randomizes DNN weights by uniform random dithering before lattice quantization and can perform near-optimally on any source without relying on knowledge of its probability distribution.Moreover, we present a method of fine-tuning vector quantized DNNs to recover the performance loss after quantization.Our experimental results show that the proposed universal DNN compression scheme compresses the 32-layer ResNet (trained on CIFAR-10) and the AlexNet (trained on ImageNet) with compression ratios of $47.1$ and $42.5$, respectively.

Introduciamo lo schema di compressione universale delle deep neural network, che è applicabile universalmente per la compressione di qualsiasi modello e può funzionare in modo quasi ottimale indipendentemente dalla loro distribuzione dei pesi.

Introduce una pipeline per la compressione di rete che è simile alla deep compression e usa la quantizzazione a reticolo randomizzata invece della classica quantizzazione vettoriale, e usa la codifica universale della sorgente (bzip2) invece della codifica Huffman.


What would be learned by variational autoencoder(VAE) and what influence the disentanglement of VAE?This paper tries to preliminarily address VAE's intrinsic dimension, real factor, disentanglement and indicator issues theoretically in the idealistic situation and implementation issue practically through noise modeling perspective in the realistic case.  On intrinsic dimension issue, due to information conservation, the idealistic VAE learns and only learns intrinsic factor dimension.Besides, suggested by mutual information separation property, the constraint induced by Gaussian prior to the VAE objective encourages the information sparsity in dimension.On disentanglement issue,   subsequently, inspired by information conservation theorem the clarification on disentanglement in this paper is made.On real factor issue, due to factor equivalence, the idealistic VAE possibly learns any factor set in the equivalence class.  On indicator issue, the behavior of current disentanglement metric is discussed, and several performance indicators regarding the disentanglement and generating influence are subsequently raised to evaluate the performance of VAE model and to supervise the used factors.On implementation issue, the experiments under noise modeling and constraints empirically testify the theoretical analysis and also show their own characteristic in pursuing disentanglement.

Questo articolo cerca di affrontare preliminarmente il disentanglement nella situazione ideale teorica e in pratica attraverso la prospettiva di modellazione del rumore nel caso realistico.

Studia l'importanza della modellazione del rumore nei VAE gaussiani e propone di addestrare il rumore usando il metodo Empirical-Bayes.

Modificare il modo in cui i fattori di rumore sono trattati nello sviluppo dei modelli VAE


Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization.Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures.We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.

Studiamo la weight decay regularization per diversi ottimizzatori e identifichiamo tre meccanismi distinti attraverso i quali il weight decay migliora la generalizzazione.

Discute l'effetto del weight decay sul training dei modelli di deep network con e senza batch normalization e quando si usano metodi di ottimizzazione del primo/secondo ordine e ipotizza che un learning rate più grande abbia un effetto di regolarizzazione.


In this paper we present the first freely available dataset for the development and evaluation of domain adaptation methods, for the sound event detection task.The dataset contains 40 log mel-band energies extracted from $100$ different synthetic sound event tracks, with additive noise from nine different acoustic scenes (from indoor, outdoor, and vehicle environments), mixed at six different sound-to-noise ratios, SNRs, (from -12 to -27 dB with a step of -3 dB), and totaling to 5400 (9 * 100 * 6) sound files and a total length of 30 564 minutes.We provide the dataset as is, the code to re-create the dataset and remix the sound event tracks and the acoustic scenes with different SNRs, and a baseline method that tests the adaptation performance with the proposed dataset and establishes some first results.

Il primo dataset di domain adaptation liberamente disponibile per il rilevamento di eventi sonori.


This paper aims to address the limitations of mutual information estimators based on variational optimization.By redefining the cost using generalized functions from nonextensive statistical mechanics we raise the upper bound of previous estimators and enable the control of the bias variance trade off.Variational based estimators outperform previous methods especially in high dependence high dimensional scenarios found in machine learning setups.Despite their performance, these estimators either exhibit a high variance or are upper bounded by log(batch size).Our approach inspired by nonextensive statistical mechanics uses different generalizations for the logarithm and the exponential in the partition function.This enables the estimator to capture changes in mutual information over a wider range of dimensions and correlations of the input variables whereas previous estimators saturate them.

Stimatore di informazioni reciproche basato sulla meccanica statistica non estensiva

Questo articolo cerca di stabilire nuovi limiti inferiori variazionali per l'informazione reciproca introducendo il parametro q e definendo q-algebra, mostrando che i limiti inferiori hanno una varianza minore e raggiungono valori elevati.


Generative adversarial networks (GANs) are a widely used framework for learning generative models.Wasserstein GANs (WGANs), one of the most successful variants of GANs, require solving a minmax problem to global optimality, but in practice, are successfully trained with stochastic gradient descent-ascent.In this paper, we show that, when the generator is a one-layer network, stochastic gradient descent-ascent converges to a global solution in polynomial time and sample complexity.

Mostriamo che lo stochastic gradient descent ascent converge a un ottimo globale per WGAN con una rete di generatori a un layer.

Tenta di dimostrare che lo Stochastic Gradient Descent-Ascent potrebbe convergere ad una soluzione globale per il problema min-max di WGAN.


Classifiers such as deep neural networks have been shown to be vulnerable against adversarial perturbations on problems with high-dimensional input space.While adversarial training improves the robustness of classifiers against such adversarial perturbations, it leaves classifiers sensitive to them on a non-negligible fraction of the inputs.We argue that there are two different kinds of adversarial perturbations: shared perturbations which fool a classifier on many inputs and singular perturbations which only fool the classifier on a small fraction of the data.We find that adversarial training increases the robustness of classifiers against shared perturbations.Moreover, it is particularly effective in removing universal perturbations, which can be seen as an extreme form of shared perturbations.Unfortunately, adversarial training does not consistently increase the robustness against singular perturbations on unseen inputs.However, we find that adversarial training decreases robustness of the remaining perturbations against image transformations such as changes to contrast and brightness or  Gaussian blurring.It thus makes successful attacks on the classifier in the physical world less likely.Finally, we show that even singular perturbations can be easily detected and must thus exhibit generalizable patterns even though the perturbations are specific for certain inputs.

Mostriamo empiricamente che l'adversarial training è efficace per rimuovere le perturbazioni universali, rende gli adversarial example meno robusti alle trasformazioni dell'immagine, e li lascia rilevabili per un approccio di rilevamento.

Analizza l'adversarial training e il suo effetto sugli adversarial example universali così come sugli adversarial example standard (iterazione di base) e come l'adversarial training influisce sul rilevamento.

Gli autori mostrano che l'adversarial training è efficace nel proteggere contro l'adversarial perturbation "condivisa", in particolare contro la perturbazione universale, ma meno efficace per proteggere contro le perturbazioni singolari.


We address the challenging problem of efficient deep learning model deployment, where the goal is to design neural network architectures that can fit different hardware platform constraints.Most of the traditional approaches either manually design or use Neural Architecture Search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally expensive and unscalable.Our key idea is to decouple model training from architecture search to save the cost.To this end, we propose to train a once-for-all network (OFA) that supports diverse architectural settings (depth, width, kernel size, and resolution).Given a deployment scenario, we can then quickly get a specialized sub-network by selecting from the OFA network without additional training.To prevent interference between many sub-networks during training, we also propose a novel progressive shrinking algorithm, which can train a surprisingly large number of sub-networks ($> 10^{19}$) simultaneously.Extensive experiments on various hardware platforms (CPU, GPU, mCPU, mGPU, FPGA accelerator) show that OFA consistently outperforms SOTA NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3) while reducing orders of magnitude GPU hours and $CO_2$ emission.In particular, OFA achieves a new SOTA 80.0% ImageNet top1 accuracy under the mobile setting ($<$600M FLOPs).Code and pre-trained models are released at https://github.com/mit-han-lab/once-for-all.

Introduciamo tecniche per addestrare una singola rete una volta per tutte che si adatta a molte piattaforme hardware.

Il metodo si traduce in una rete da cui si possono estrarre sottoreti per vari vincoli di risorse (latenza, memoria) che funzionano bene senza bisogno di retraining.

Questo articolo cerca di affrontare il problema della ricerca delle migliori architetture per scenari distribuiti con vincoli di risorse specializzate con un metodo NAS basato sulla predizione.


A deep generative model is a powerful method of learning a data distribution, which has achieved tremendous success in numerous scenarios.However, it is nontrivial for a single generative model to faithfully capture the distributions of the complex data such as images with complicate structures.In this paper, we propose a novel approach of cascaded boosting for boosting generative models, where meta-models (i.e., weak learners) are cascaded together to produce a stronger model.Any hidden variable meta-model can be leveraged as long as it can support the likelihood evaluation.We derive a decomposable variational lower bound of the boosted model, which allows each meta-model to be trained separately and greedily.We can further improve the learning power of the generative models by combing our cascaded boosting framework with the multiplicative boosting framework.

Proporre un approccio per il boosting dei modelli generativi mediante modelli a cascata di variabili nascoste

Questo articolo ha proposto un nuovo approccio di boosting in cascata per il boosting dei modelli generativi che permette ad ogni meta-modello di essere addestrato separatamente e in modo greedy.


Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks.Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline.We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena.We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

Abbiamo analizzato la struttura della frase in ELMo e nei contextual embedding model. Troviamo che i modelli esistenti codificano efficientemente la sintassi e mostrano prove di saper gestire long-range dependency, ma offrono solo piccoli miglioramenti sui task semantici.

Propone il metodo "edge probing" e si concentra sulla relazione tra gli span piuttosto che sulle singole parole, permettendo agli autori di guardare alla costituzione sintattica, alle dipendenze, alle label delle entità e al semantic role labelling.

Fornisce nuove intuizioni su ciò che viene catturato dai contextualized word embedding compilando una serie di task di "edge probing".


Deep reinforcement learning has succeeded in sophisticated games such as Atari, Go, etc.Real-world decision making, however, often requires reasoning with partial information extracted from complex visual observations.This paper presents  Discriminative Particle Filter Reinforcement Learning (DPFRL), a new reinforcement learning framework for partial and complex observations.DPFRL encodes a differentiable particle filter with learned transition and observation models in a neural network, which allows for reasoning with partial observations over multiple time steps.While a standard particle filter relies on a generative observation model, DPFRL learns a discriminatively parameterized model that is training directly for decision making.We show that the discriminative parameterization results in significantly improved performance, especially for tasks with complex visual observations, because it circumvents the difficulty of modelling observations explicitly.In most cases, DPFRL outperforms state-of-the-art POMDP RL models in Flickering Atari Games, an existing POMDP RL benchmark, and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark that we introduce.We further show that DPFRL performs well for visual navigation with real-world data.

Introduciamo DPFRL, un framework per il reinforcement learning sotto osservazioni parziali e complesse con un fully differentiable discriminative particle filter

Introduce idee per il training di agenti DLR con variabili di stato latenti, modellate come una distribuzione di belief, in modo che possano gestire ambienti parzialmente osservati.

Questo articolo introduce un metodo di principio per POMDP RL: Discriminative Particle Filter Reinforcement Learning che permette di ragionare con osservazioni parziali su più time step, raggiungendo lo stato dell'arte sui benchmark.


Extending models with auxiliary latent variables is a well-known technique to in-crease model expressivity.Bachman & Precup (2015); Naesseth et al. (2018); Cremer et al. (2017); Domke & Sheldon (2018) show that Importance Weighted Autoencoders (IWAE) (Burda et al., 2015) can be viewed as extending the variational family with auxiliary latent variables.Similarly, we show that this view encompasses many of the recent developments in variational bounds (Maddisonet al., 2017; Naesseth et al., 2018; Le et al., 2017; Yin & Zhou, 2018; Molchanovet al., 2018; Sobolev & Vetrov, 2018).The success of enriching the variational family with auxiliary latent variables motivates applying the same techniques to the generative model.We develop a generative model analogous to the IWAE bound and empirically show that it outperforms the recently proposed Learned Accept/Reject Sampling algorithm (Bauer & Mnih, 2018), while being substantially easier to implement.Furthermore, we show that this generative process provides new insights on ranking Noise Contrastive Estimation (Jozefowicz et al.,2016; Ma & Collins, 2018) and Contrastive Predictive Coding (Oord et al., 2018).

Gli obiettivi di Monte Carlo sono analizzati utilizzando l'inferenza variazionale delle variabili ausiliarie, ottenendo una nuova analisi di CPC e NCE così come un nuovo modello generativo.

Propone una visione diversa sul miglioramento dei limiti variazionali con modelli ausiliari di variabili latenti ed esplora l'uso di questi modelli nel modello generativo.


Stochastic Gradient Descent or SGD is the most popular optimization algorithm for large-scale problems.SGD estimates the gradient by uniform sampling with sample size one.There have been several other works that suggest faster epoch wise convergence by using weighted non-uniform sampling for better gradient estimates.Unfortunately, the per-iteration cost of maintaining this adaptive distribution for gradient estimation is more than calculating the full gradient.As a result, the false impression of faster convergence in iterations leads to slower convergence in time, which we call a chicken-and-egg loop.In this paper, we break this barrier by providing the first demonstration of a sampling scheme, which leads to superior gradient estimation, while keeping the sampling cost per iteration similar to that of the uniform sampling.Such an algorithm is possible due to the sampling view of Locality Sensitive Hashing (LSH), which came to light recently.As a consequence of superior and fast estimation, we reduce the running time of all existing gradient descent algorithms.We demonstrate the benefits of our proposal on both SGD and AdaGrad.

Miglioriamo l'esecuzione di tutti gli algoritmi di gradient descent esistenti.

Gli autori propongono di campionare i gradienti stocastici da una funzione monotona proporzionale alla grandezza del gradiente utilizzando LSH.

Considera SGD su un obiettivo a forma di una somma su esempi di una loss quadratica.


In recent years we have made significant progress identifying computational principles that underlie neural function.While not yet complete, we have sufficient evidence that a synthesis of these ideas could result in an understanding of how neural computation emerges from a combination of innate dynamics and plasticity, and which could potentially be used to construct new AI technologies with unique capabilities.I discuss the relevant principles, the advantages they have for computation, and how they can benefit AI.Limitations of current AI are generally recognized, but fewer people are aware that we understand enough about the brain to immediately offer novel AI formulations.

Le limitazioni dell'attuale IA sono generalmente riconosciute, ma meno persone sono consapevoli che comprendiamo abbastanza del cervello per offrire immediatamente nuove formulazioni di IA.


Recent work has demonstrated how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments.We propose question-answering as a general paradigm to decode and understand the representations that such agents develop, applying our method to two recent approaches to predictive modeling â€“ action-conditional CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019).After training agents with these predictive objectives in a visually-rich, 3D environment with an assortment of objects, colors, shapes, and spatial configurations, we probe their internal state representations with a host of synthetic (English) questions, without backpropagating gradients from the question-answering decoder into the agent.The performance of different agents when probed in this way reveals that they learn to encode detailed, and seemingly compositional, information about objects, properties and spatial relations from their physical environment.Our approach is intuitive, i.e. humans can easily interpret the responses of the model as opposed to inspecting continuous vectors, and model-agnostic, i.e. applicable to any modeling approach.By revealing the implicit knowledge of objects, quantities, properties and relations acquired by agents as they learn, question-conditional agent probing can stimulate the design and development of stronger predictive learning objectives.

Utilizziamo question answering per valutare quanta conoscenza dell'environment può essere appresa dagli agenti attraverso la predizione self-supervised .

Propone QA come uno strumento per investigare ciò che gli agenti imparano nel mondo, sostenendo che questo è un metodo intuitivo per gli umani che permette una complessità arbitraria.

Gli autori propongono un framework per valutare le rappresentazioni costruite dai modelli predittivi che contengono informazioni sufficienti per rispondere alle domande sull'ambiente su cui sono addestrati, mostrando che quelli di SimCore contenevano informazioni sufficienti perché la LSTM possa rispondere accuratamente alle domande.


In most real-world scenarios, training datasets are highly class-imbalanced, where deep neural networks suffer from generalizing to a balanced testing criterion.In this paper, we explore a novel yet simple way to alleviate this issue via synthesizing less-frequent classes with adversarial examples of other classes.Surprisingly, we found this counter-intuitive method can effectively learn generalizable features of minority classes by transferring and leveraging the diversity of the majority information.Our experimental results on various types of class-imbalanced datasets in image classification and natural language processing show that the proposed method not only improves the generalization of minority classes significantly compared to other re-sampling or re-weighting methods, but also surpasses other methods of state-of-art level for the class-imbalanced classification.

Sviluppiamo un nuovo metodo per la classificazione imbalanced usando adversarial example

Propone un nuovo obiettivo di ottimizzazione che genera sample sintetici sovracampionando le classi di maggioranza invece di quelle di minoranza, risolvendo il problema dell'overfitting alle classi di minoranza.

Gli autori propongono di affrontare la classificazione imbalanced usando metodi di resampling, mostrando che gli adversarial example nella classe minoritaria aiuterebbero ad addestrare un nuovo modello che generalizza meglio.


Active matter consists of active agents which transform energy extracted from surroundings into momentum, producing a variety of collective phenomena.A model, synthetic active system composed of microtubule polymers driven by protein motors spontaneously forms a liquid-crystalline nematic phase.Extensile stress created by the protein motors precipitates continuous buckling and folding of the microtubules creating motile topological defects and turbulent fluid flows.Defect motion is determined by the rheological properties of the material; however, these remain largely unquantified.Measuring defects dynamics can yield fundamental insights into active nematics, a class of materials that include bacterial films and animal cells.Current methods for defect detection lack robustness and precision, and require fine-tuning for datasets with different visual quality.  In this study, we applied Deep Learning to train a defect detector to automatically analyze microscopy videos of the microtubule active nematic.  Experimental results indicate that our method is robust and accurate.It is expected to significantly increase the amount of video data that can be processed.

Un'interessante applicazione della CNN negli esperimenti di fisica della materia soft condensed.

Gli autori dimostrano che un approccio di deep learning offre un miglioramento sia nell'accuratezza dell'identificazione che al tasso di identificazione dei difetti dei cristalli liquidi nematici.

Applicare un modello neurale ben noto (YOLO) per rilevare le bounding box degli oggetti nelle immagini.


In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL). In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed.The results of our experiment show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning.

Un'analisi degli effetti della compositività e della località sul representation learning per lo zero-shot learning.

Propone un framework di valutazione per ZSL in cui il modello non può essere pretrained ed invece i parametri del modello sono inizializzati in modo casuale per una migliore comprensione di ciò che accade in ZSL.


It is becoming increasingly clear that many machine learning classifiers are vulnerable to adversarial examples.In attempting to explain the origin of adversarial examples, previous studies have typically focused on the fact that neural networks operate on high dimensional data, they overfit, or they are too linear.Here we show that distributions of logit differences have a universal functional form.This functional form is independent of architecture, dataset, and training protocol; nor does it change during training.This leads to adversarial error having a universal scaling, as a power-law, with respect to the size of the adversarial perturbation.We show that this universality holds for a broad range of datasets (MNIST, CIFAR10, ImageNet, and random data), models (including state-of-the-art deep networks, linear models, adversarially trained networks, and networks trained on randomly shuffled labels), and attacks (FGSM, step l.l., PGD).Motivated by these results, we study the effects of reducing prediction entropy on adversarial robustness.Finally, we study the effect of network architectures on adversarial sensitivity.To do this, we use neural architecture search with reinforcement learning to find adversarially robust architectures on CIFAR10.Our resulting architecture is more robust to white \emph{and} black box attacks compared to previous attempts.

L'adversarial error ha una forma simile alla power-law per tutti i dataset e i modelli studiati, e l'architettura ha un ruolo importante.


Reinforcement learning (RL) has led to increasingly complex looking behavior in recent years.However, such complexity can be misleading and hides over-fitting.We find that visual representations may be a useful metric of complexity, and both correlates well objective optimization and causally effects reward optimization.We then propose curious representation learning (CRL) which allows us to use better visual representation learning algorithms to correspondingly increase visual representation in policy through an intrinsic objective on both simulated environments and transfer to real images.Finally, we show better visual representations induced by CRL allows us to obtain better performance on Atari without any reward than other curiosity objectives.

Presentiamo una formulazione della curiosity come un problema di visual representation learning e mostriamo che ottiene buone rappresentazioni visive negli agenti.

Questo articolo formula il training RL basato sulla curiosity come apprendimento di un modello di visual representation, sostenendo che concentrandosi su una migliore LR e massimizzando la loss del modello per le scene nuove si otterrà una migliore performance complessiva.


This paper introduces the task of semantic instance completion: from an incomplete RGB-D scan of a scene, we aim to detect the individual object instances comprising the scene and infer their complete object geometry.This enables a semantically meaningful decomposition of a scanned scene into individual, complete 3D objects, including hidden and unobserved object parts.This will open up new possibilities for interactions with object in a scene, for instance for virtual or robotic agents.To address this task, we propose 3D-SIC, a new data-driven approach that jointly detects object instances and predicts their completed geometry.The core idea of 3D-SIC is a novel end-to-end 3D neural network architecture that leverages joint color and geometry feature learning.The fully-convolutional nature of our 3D network enables efficient inference of semantic instance completion for 3D scans at scale of large indoor environments in a single forward pass.In a series evaluation, we evaluate on both real and synthetic scan benchmark data, where we outperform state-of-the-art approaches by over 15 in mAP@0.5 on ScanNet, and over 18 in mAP@0.5 on SUNCG.

Da una scansione RGB-D incompleta di una scena, miriamo a rilevare le istanze individuali degli oggetti che compongono la scena e a dedurre la loro geometria completa.

Propone una CNN 3D end-to-end che combina feature di colore e feature 3D per prevedere la struttura 3D mancante di una scena da scansioni RGB-D.

Gli autori propongono una nuova rete di convoluzione 3D end-to-end che predice il completamento dell'istanza semantica 3D come bounding box dell'oggetto, le label di classe e la geometria completa dell'oggetto.


Style transfer usually refers to the task of applying color and texture information from a specific style image to a given content image while preserving the structure of the latter.Here we tackle the more generic problem of semantic style transfer: given two unpaired collections of images, we aim to learn a mapping between the corpus-level style of each collection, while preserving semantic content shared across the two domains.We introduce XGAN ("Cross-GAN"), a dual adversarial autoencoder, which captures a shared representation of the common domain semantic content in an unsupervised way, while jointly learning the domain-to-domain image translations in both directions.  We exploit ideas from the domain adaptation literature and define a semantic consistency loss which encourages the model to preserve semantics in the learned embedding space.We report promising qualitative results for the task of face-to-cartoon translation.The cartoon dataset we collected for this purpose will also be released as a new benchmark for semantic style transfer.

XGAN è un modello unsupervised per la traduzione image-to-image a livello di feature applicato a problemi di semantic style transfer come il face-to-cartoon task, per il quale introduciamo un nuovo dataset.

Questo articolo propone un nuovo modello basato su GAN per la image-to-image translation non accoppiata simile a DTN


Training neural networks on large datasets can be accelerated by distributing the workload over a network of machines.As datasets grow ever larger, networks of hundreds or thousands of machines become economically viable.The time cost of communicating gradients limits the effectiveness of using such large machine counts, as may the increased chance of network faults.We explore a particularly simple algorithm for robust, communication-efficient learning---signSGD.Workers transmit only the sign of their gradient vector to a server, and the overall update is decided by a majority vote.This algorithm uses 32x less communication per iteration than full-precision, distributed SGD.Under natural conditions verified by experiment, we prove that signSGD converges in the large and mini-batch settings, establishing convergence for a parameter regime of Adam as a byproduct.Aggregating sign gradients by majority vote means that no individual worker has too much power.We prove that unlike SGD, majority vote is robust when up to 50% of workers behave adversarially.The class of adversaries we consider includes as special cases those that invert or randomise their gradient estimate.On the practical side, we built our distributed training system in Pytorch.Benchmarking against the state of the art collective communications library (NCCL), our framework---with the parameter server housed entirely on one machine---led to a 25% reduction in time for training resnet50 on Imagenet when using 15 AWS p3.2xlarge machines.

I lavoratori inviano i segni del gradiente al server, e l'aggiornamento viene deciso secondo la maggioranza. Mostriamo che questo algoritmo è convergente, efficiente nella comunicazione e tollerante agli errori, sia in teoria che in pratica.

Presenta un'implementazione distribuita di signSGD con voto di maggioranza come aggregazione.


Profiling cellular phenotypes from microscopic imaging can provide meaningful biological information resulting from various factors affecting the cells.One motivating application is drug development: morphological cell features can be captured from images, from which similarities between different drugs applied at different dosages can be quantified.The general approach is to find a function mapping the images to an embedding space of manageable dimensionality whose geometry captures relevant features of the input images.An important known issue for such methods is separating relevant biological signal from nuisance variation.For example, the embedding vectors tend to be more correlated for cells that were cultured and imaged during the same week than for cells from a different week, despite having identical drug compounds applied in both cases.In this case, the particular batch a set of experiments were conducted in constitutes the domain of the data; an ideal set of image embeddings should contain only the relevant biological information (e.g. drug effects).We develop a general framework for adjusting the image embeddings in order to `forget' domain-specific information while preserving relevant biological information.To do this, we minimize a loss function based on distances between marginal distributions (such as the Wasserstein distance) of embeddings across domains for each replicated treatment.For the dataset presented, the replicated treatment is the negative control.We find that for our transformed embeddings (1) the underlying geometric structure is not only preserved but the embeddings also carry improved biological signal (2) less domain-specific information is present.

Correggiamo la nuisance variation per image embedding in diversi domini, conservando solo le informazioni rilevanti.

Discute un metodo per regolare gli image embedding al fine di separare la variazione tecnica dal segnale biologico.

Gli autori presentano un metodo per rimuovere l'informazione specifica del dominio preservando l'informazione biologica rilevante, addestrando una rete che minimizza la distanza di Wasserstein tra le distrbuzioni.


This paper presents a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size.MINE is  back-propable and we prove that it is strongly consistent.We illustrate a handful of applications in which MINE is succesfully applied  to enhance the property of generative models in both unsupervised and supervised settings.We apply our framework to estimate the information bottleneck, and apply it in tasks related to supervised classification problems.Our results  demonstrate substantial added flexibility and improvement in these settings.

Uno stimatore di informazione mutua scalabile per dimensione e sample size.


Reinforcement learning methods have recently achieved impressive results on a wide range of control problems.However, especially with complex inputs, they still require an extensive amount of training data in order to converge to a meaningful solution.This limitation largely prohibits  their usage for complex input spaces such as video signals, and it is still impossible to use it for a number of complex problems in a real world environments, including many of those for video based control.Supervised learning, on the contrary, is capable of learning on a relatively small number of samples, however it does not take into account reward-based control policies and is not capable to provide independent control policies.  In this article we propose a model-free control method, which uses a combination of reinforcement and supervised learning for autonomous control and paves the way towards policy based control in real world environments.We use SpeedDreams/TORCS video game to demonstrate that our approach requires much less samples (hundreds of thousands against millions or tens of millions) comparing to the state-of-the-art reinforcement learning techniques on similar data, and at the same time overcomes both supervised and reinforcement learning approaches in terms of quality.Additionally, we demonstrate the applicability of the method to MuJoCo control problems.

La nuova combinazione di reinforcement e supervised learning diminuisce drasticamente il numero di sample richiesti per il training su video

Questo articolo propone di sfruttare i dati controllati annotati per accelerare il reinforcement learning di una policy di controllo


A typical experiment to study cognitive function is to train animals to perform tasks, while the researcher records the electrical activity of the animals neurons.The main obstacle faced, when using this type of electrophysiological experiment to uncover the circuit mechanisms underlying complex behaviors, is our incomplete access to relevant circuits in the brain.One promising approach is to model neural circuits using an artificial neural network (ANN), which can provide complete access to the â€œneural circuitsâ€ responsible for a behavior.More recently, reinforcement learning models have been adopted to understand the functions of cortico-basal ganglia circuits as reward-based learning has been found in mammalian brain.In this paper, we propose a Biologically-plausible Actor-Critic with Episodic Memory (B-ACEM) framework to model a prefrontal cortex-basal ganglia-hippocampus (PFC-BG) circuit, which is verified to capture the behavioral findings from a well-known perceptual decision-making task, i.e., random dots motion discrimination.This B-ACEM framework links neural computation to behaviors, on which we can explore how episodic memory should be considered to govern future decision.Experiments are conducted using different settings of the episodic memory and results show that all patterns of episodic memories can speed up learning.In particular, salient events are prioritized to propagate reward information and guide decisions.Our B-ACEM framework and the built-on experiments give inspirations to both designs for more standard decision-making models in biological system and a more biologically-plausible ANN.

Apprendimento veloce attraverso la memoria episodica verificata da un framework biologicamente plausibile per il prefrontal cortex-basal ganglia-hippocampus (PFC-BG) circuit.


Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet challenging question in deep learning and approximation theory.In a seminal paper, Telgarsky high- lighted the benefits of depth by presenting a family of functions (based on sim- ple triangular waves) for which DNNs achieve zero classification error, whereas shallow networks with fewer than exponentially many nodes incur constant error.Even though Telgarskyâ€™s work reveals the limitations of shallow neural networks, it doesnâ€™t inform us on why these functions are difficult to represent and in fact he states it as a tantalizing open question to characterize those functions that cannot be well-approximated by smaller depths.In this work, we point to a new connection between DNNs expressivity and Sharkovskyâ€™s Theorem from dynamical systems, that enables us to characterize the depth-width trade-offs of ReLU networks for representing functions based on the presence of a generalized notion of fixed points, called periodic points (a fixed point is a point of period 1).Motivated by our observation that the triangle waves used in Telgarskyâ€™s work contain points of period 3 â€“ a period that is special in that it implies chaotic behaviour based on the celebrated result by Li-Yorke â€“ we proceed to give general lower bounds for the width needed to represent periodic functions as a function of the depth.Technically, the crux of our approach is based on an eigenvalue analysis of the dynamical systems associated with such functions.

In questo lavoro indichiamo una nuova connessione tra l'espressività delle DNN e il teorema di Sharkovsky dei sistemi dinamici, che ci permette di caratterizzare il trade-off tra profondità e larghezza delle reti ReLU

Mostra come la potenza espressiva di NN dipende dalla sua profondità e larghezza, approfondendo la comprensione del beneficio delle deep neural network per rappresentare certe classi di funzioni.

Gli autori derivano condizioni di trade-off depth-width per quando le reti relu sono in grado di rappresentare funzioni periodiche usando l'analisi dei sistemi dinamici.


We investigate low-bit quantization to reduce computational cost of deep neural network (DNN) based keyword spotting (KWS).We propose approaches to further reduce quantization bits via integrating quantization into keyword spotting model training, which we refer to as quantization-aware training.Our experimental results on large dataset indicate that quantization-aware training can recover performance models quantized to lower bits representations.By combining quantization-aware training and weight matrix factorization, we are able to significantly reduce model size and computation for small-footprint keyword spotting, while maintaining performance.

Studiamo il training informato sulla quantizzazione nei keyword spotter low-bit quantized per ridurre il costo on-device del keyword spotting.

Questa presentazione propone una combinazione di decomposizione low-rank e un approccio di quantizzazione per comprimere i modelli DNN per keyword spotting.


Single-cell RNA-sequencing (scRNA-seq) is a powerful tool for analyzing biological systems.However, due to biological and technical noise, quantifying the effects of multiple experimental conditions presents an analytical challenge.To overcome this challenge, we developed MELD: Manifold Enhancement of Latent Dimensions.MELD leverages tools from graph signal processing to learn a latent dimension within the data scoring the prototypicality of each datapoint with respect to experimental or control conditions.We call this dimension the Enhanced Experimental Signal (EES).MELD learns the EES by filtering the noisy categorical experimental label in the graph frequency domain to recover a smooth signal with continuous values.This method can be used to identify signature genes that vary between conditions and identify which cell types are most affected by a given perturbation.We demonstrate the advantages of MELD analysis in two biological datasets, including T-cell activation in response to antibody-coated beads and treatment of human pancreatic islet cells with interferon gamma.

Un nuovo framework di elaborazione dei graph signal per quantificare gli effetti delle perturbazioni sperimentali nei dati biomedici delle singole cellule.

Questo articolo introduce diversi metodi per elaborare i risultati sperimentali sulle cellule biologiche e propone un algoritmo MELD che mappa le assegnazioni di gruppo hard in assegnazioni soft, permettendo di raggruppare i gruppi di cellule rilevanti.


Models of user behavior are critical inputs in many prescriptive settings and can be viewed as decision rules that transform state information available to the user into actions.Gaussian processes (GPs), as well as nonlinear extensions thereof, provide a flexible framework to learn user models in conjunction with approximate Bayesian inference.However, the resulting models may not be interpretable in general.We propose decision-rule GPs (DRGPs) that apply GPs in a transformed space defined by decision rules that have immediate interpretability to practitioners.We illustrate this modeling tool on a real application and show that structural variational inference techniques can be used with DRGPs.We find that DRGPs outperform the direct use of GPs in terms of out-of-sample performance.

Proponiamo una classe di modelli utente basati sull'utilizzo di processi gaussiani applicati a uno spazio trasformato definito da regole decisionali


While Bayesian optimization (BO) has achieved great success in optimizing expensive-to-evaluate black-box functions, especially tuning hyperparameters of neural networks, methods such as random search (Li et al., 2016) and multi-fidelity BO (e.g. Klein et al. (2017)) that exploit cheap approximations, e.g. training on a smaller training data or with fewer iterations, can outperform standard BO approaches that use only full-fidelity observations.In this paper, we propose a novel Bayesian optimization algorithm, the continuous-fidelity knowledge gradient (cfKG) method, that can be used when fidelity is controlled by one or more continuous settings such as training data size and the number of training iterations.cfKG characterizes the value of the information gained by sampling a point at a given fidelity, choosing to sample at the point and fidelity with the largest value per unit cost.Furthermore, cfKG can be generalized, following Wu et al. (2017), to settings where derivatives are available in the optimization process, e.g. large-scale kernel learning, and where more than one point can be evaluated simultaneously.Numerical experiments show that cfKG outperforms state-of-art algorithms when optimizing synthetic functions, tuning convolutional neural networks (CNNs) on CIFAR-10 and SVHN, and in large-scale kernel learning.

Proponiamo un algoritmo di ottimizzazione bayesiana Bayes-optimal per la scelta degli iperparametri sfruttando approssimazioni economiche.

Studia l'ottimizzazione degli iperparametri tramite l'ottimizzazione bayesiana, usando il framework del Knowledge Gradient e permettendo all'ottimizzatore bayesiano di fare tuning della fidelity rispetto al costo.