A number of recent methods to understand neural networks have focused on quantifying the role of individual features.  One such method, NetDissect identifies interpretable features of a model using the Broden dataset of visual semantic labels (colors, materials, textures, objects and scenes).  Given the recent rise of a number of action recognition datasets, we propose extending the Broden dataset to include actions to better analyze learned action models.  We describe the annotation process, results from interpreting action recognition models on the extended Broden dataset and examine interpretable feature paths to help us understand the conceptual hierarchy used to classify an action.

Espandiamo la Network Dissection per includere l'interpretazione delle azioni ed esaminiamo i percorsi delle feature interpretabili per capire la gerarchia concettuale usata per classificare un'azione.


Automatic melody generation for pop music has been a long-time aspiration forboth AI researchers and musicians.However, learning to generate euphoniousmelody has turned out to be highly challenging due to a number of factors.Representationof multivariate property of notes has been one of the primary challenges.It is also difficult to remain in the permissible spectrum of musical variety, outsideof which would be perceived as a plain random play without auditory pleasantness.Observing the conventional structure of pop music poses further challenges.In this paper, we propose to represent each note and its properties as a uniqueâ€˜word,â€™ thus lessening the prospect of misalignments between the properties, aswell as reducing the complexity of learning.We also enforce regularization policieson the range of notes, thus encouraging the generated melody to stay closeto what humans would find easy to follow.Furthermore, we generate melodyconditioned on song part information, thus replicating the overall structure of afull song.Experimental results demonstrate that our model can generate auditorilypleasant songs that are more indistinguishable from human-written ones thanprevious models.

Proponiamo un nuovo modello per rappresentare le note e le loro proprietà, che può migliorare la generazione automatica della melodia.

Questo articolo propone un modello generativo di melodia simbolica (MIDI) nella musica popolare occidentale che codifica congiuntamente i simboli delle note con informazioni di tempo e durata per formare "parole" musicali.

L'articolo propone di facilitare la generazione di melodia rappresentando le note come "parole", rappresentando tutte le proprietà della nota e permettendo così la generazione di "frasi" musicali.


Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts.We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth.We propose robust growing and stopping policies to generalize to different network architectures and datasets.Our experiments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet.For example, in terms of accuracy-computation trade-off, AutoGrow discovers a better depth combination in ResNets than human experts.Our AutoGrow is efficient.It discovers depth within similar time of training a single DNN.

Un metodo che fa crescere automaticamente i layer nelle reti neurali per scoprire la profondità ottimale.

Un framework per interlacciare il training di una rete meno profonda e l'aggiunta di nuovi layer che fornisce intuizioni sul paradigma delle "growing network".


Given the importance of remote sensing, surprisingly little attention has been paid to it by the representation learning community.To address it and to speed up innovation in this domain, we provide simplified access to 5 diverse remote sensing datasets in a standardized form.We specifically explore in-domain representation learning and address the question of "what characteristics should a dataset have to be a good source for remote sensing representation learning".The established baselines achieve state-of-the-art performance on these datasets.

Esplorazione del representation learning in-domain per dataset di remote sensing.

Questo articolo ha fornito diversi dataset standardizzati per il remote sensing e ha dimostrato che la rappresentazione in-domain potrebbe produrre migliori baseline per il remote sensing rispetto al fine-tuning su ImageNet o all'apprendimento da zero.


Generative seq2seq dialogue systems are trained to predict the next word in dialogues that have already occurred.They can learn from large unlabeled conversation datasets, build a deep understanding of conversational context, and generate a wide variety of responses.This flexibility comes at the cost of control.Undesirable responses in the training data will be reproduced by the model at inference time, and longer generations often donâ€™t make sense.Instead of generating responses one word at a time, we train a classifier to choose from a predefined list of full responses.The classifier is trained on (conversation context, response class) pairs, where each response class is a noisily labeled group of interchangeable responses.At inference, we generate the exemplar response associated with the predicted response class.Experts can edit and improve these exemplar responses over time without retraining the classifier or invalidating old training data.Human evaluation of 775 unseen doctor/patient conversations shows that this tradeoff improves responses.Only 12% of our discriminative approachâ€™s responses are worse than the doctorâ€™s response in the same conversational context, compared to 18% for the generative model.A discriminative model trained without any manual labeling of response classes achieves equal performance to the generative model.

Evitare di generare risposte una parola alla volta utilizzando una supervisione debole per addestrare un classificatore a scegliere una risposta completa.

Un modo per generare risposte per il dialogo medico usando un classificatore per selezionare da un insieme di risposte curate da esperti in base al contesto della conversazione.


There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs).This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP.In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels.We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible. Surprisingly, in the absence of pooling layers, the GPs corresponding to CNNs with and without weight sharing are identical.As a consequence, translation equivariance, beneficial in finite channel CNNs trained with stochastic gradient descent (SGD), is guaranteed to play no role in the Bayesian treatment of the infinite channel limit - a qualitative difference between the two regimes that is not present in the FCN case.We confirm experimentally, that while in some scenarios the performance of SGD-trained finite CNNs approaches that of the corresponding GPs as the channel count increases, with careful tuning SGD-trained CNNs can significantly outperform their corresponding GPs, suggesting advantages from SGD training compared to fully Bayesian parameter estimation.

CNN con width finita addestrate con SGD contro CNN completamente bayesiane con width infinita. Chi vince?

L'articolo stabilisce una connessione tra la rete neurale convoluzionale bayesiana a canale infinito e i processi gaussiani.


Bayesian inference promises to ground and improve the performance of deep neural networks.It promises to be robust to overfitting, to simplify the training procedure and the space of hyperparameters, and to provide a calibrated measure of uncertainty that can enhance decision making, agent exploration and prediction fairness.Markov Chain Monte Carlo (MCMC) methods enable Bayesian inference by generating samples from the posterior distribution over model parameters.Despite the theoretical advantages of Bayesian inference and the similarity between MCMC and optimization methods, the performance of sampling methods has so far lagged behind  optimization methods for large scale deep learning tasks.We aim to fill this gap and introduce ATMC, an adaptive noise MCMC algorithm that estimates and is able to sample from the posterior of a neural network.ATMC dynamically adjusts the amount of momentum and noise applied to each parameter update in order to compensate for the use of stochastic gradients.We use a ResNet architecture without batch normalization to test ATMC on the Cifar10 benchmark and the large scale ImageNet benchmark and show that, despite the  absence of batch normalization, ATMC outperforms a strong optimization baseline in terms of both classification accuracy and test log-likelihood.We show that ATMC is intrinsically robust to overfitting on the training data and that ATMC provides a better calibrated measure of uncertainty compared to the optimization baseline.

Scaliamo l'inferenza bayesiana alla classificazione ImageNet e raggiungiamo risultati competitivi di accuratezza e calibrazione dell'incertezza.

Un algoritmo MCMC adattivo per la classificazione delle immagini che regola dinamicamente il momentum e il rumore applicato ad ogni aggiornamento dei parametri, ed è robusto all'overfitting e fornisce una misura di incertezza con le previsioni.


Now GANs can generate more and more realistic face images that can easily fool human beings.  In contrast, a common convolutional neural network(CNN), e.g. ResNet-18, can achieve more than 99.9% accuracy in discerning fake/real faces if training and testing faces are from the same source.In this paper, we performed both human studies and CNN experiments, which led us to two important findings.One finding is that the textures of fake faces are substantially different from real ones.CNNs can capture local image texture information for recognizing fake/real face, while such cues are easily overlooked by humans.The other finding is that global image texture information is more robust to image editing and generalizable to fake faces from different GANs and datasets.Based on the above findings, we propose  a  novel  architecture  coined  as  Gram-Net,  which  incorporates  â€œGram Blockâ€ in multiple semantic levels to extract global image texture representations.Experimental results demonstrate that our Gram-Net performs better than existing approaches for fake face detection.  Especially, our Gram-Net is more robust to image editing, e.g.  downsampling, JPEG compression, blur, and noise.  More importantly, our Gram-Net generalizes significantly better in detecting fake faces from GAN models not seen in the training phase.

Uno studio empirico sulle immagini false rivela che la texture è un importante indizio che le immagini false attuali differiscono dalle immagini reali. Il nostro modello migliorato che cattura le statistiche globali delle texture mostra migliori prestazioni di rilevamento delle immagini false cross-GAN.

L'articolo propone un modo per migliorare le prestazioni del modello per il rilevamento di volti falsi in immagini generate da una GAN per essere più generalizzabile in base alle informazioni sulla texture.


The Wasserstein probability metric has received much attention from the machine learning community.Unlike the Kullback-Leibler divergence, which strictly measures change in probability, the Wasserstein metric reflects the underlying geometry between outcomes.The value of being sensitive to this geometry has been demonstrated, among others, in ordinal regression and generative modelling, and most recently in reinforcement learning.In this paper we describe three natural properties of probability divergences that we believe reflect requirements from machine learning: sum invariance, scale sensitivity, and unbiased sample gradients.The Wasserstein metric possesses the first two properties but, unlike the Kullback-Leibler divergence, does not possess the third.We provide empirical evidence suggesting this is a serious issue in practice.Leveraging insights from probabilistic forecasting we propose an alternative to the Wasserstein metric, the CramÃ©r distance.We show that the CramÃ©r distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences.We give empirical results on a number of domains comparing these three divergences.To illustrate the practical relevance of the CramÃ©r distance we design a new algorithm, the CramÃ©r Generative Adversarial Network (GAN), and show that it has a number of desirable properties over the related Wasserstein GAN.

La distanza di Wasserstein è difficile da minimizzare con la stochastic gradient descent, mentre la distanza di Cramer può essere ottimizzata facilmente e funziona altrettanto bene.

Il manoscritto propone di usare la distanza di Cramer come loss quando si ottimizza una funzione obiettivo usando stochastic gradient descent perché ha unbiased sample gradient.

Il contributo dell'articolo è legato ai criteri di performance, in particolare alla metrica di Wasserstein/Mallows


We humans have an innate understanding of the asymmetric progression of time, which we use to efficiently and safely perceive and manipulate our environment.Drawing inspiration from that, we approach the problem of learning an arrow of time in a Markov (Decision) Process.We illustrate how a learned arrow of time can capture salient information about the environment, which in turn can be used to measure reachability, detect side-effects and to obtain an intrinsic reward signal.Finally, we propose a simple yet effective algorithm to parameterize the problem at hand and learn an arrow of time with a function approximator (here, a deep neural network).Our empirical results span a selection of discrete and continuous environments, and demonstrate for a class of stochastic processes that the learned arrow of time agrees reasonably well with a well known notion of an arrow of time due to Jordan, Kinderlehrer and Otto (1998).

Impariamo la freccia del tempo per gli MDP e la usiamo per misurare la raggiungibilità, rilevare gli effetti collaterali e ottenere un segnale di ricompensa della curiosity.

Questo lavoro propone l'h-potenziale come soluzione a un obiettivo che misura l'asimmetria stato-transizione in un MDP.


We formulate stochastic gradient descent (SGD) as a novel factorised Bayesian filtering problem, in which each parameter is inferred separately, conditioned on the corresopnding backpropagated gradient.  Inference in this setting naturally gives rise to BRMSprop and BAdam: Bayesian variants of RMSprop and Adam.  Remarkably, the Bayesian approach recovers many features of state-of-the-art adaptive SGD methods, including amongst others root-mean-square normalization, Nesterov acceleration and AdamW.  As such, the Bayesian approach provides one explanation for the empirical effectiveness of state-of-the-art adaptive SGD algorithms.  Empirically comparing BRMSprop and BAdam with naive RMSprop and Adam on MNIST, we find that Bayesian methods have the potential to considerably reduce test loss and classification error.

Abbiamo formulato SGD come un problema di filtraggio bayesiano, e mostriamo che questo dà origine a RMSprop, Adam, AdamW, NAG e altre feature dei metodi adattativi allo stato dell'arte

L'articolo analizza la stochastic gradient descent attraverso il filtraggio bayesiano come framework per analizzare i metodi adattivi.

Gli autori tentano di unificare i metodi di gradiente adattivo esistenti nel framework del filtraggio bayesiano con un prior dinamico


Data augmentation (DA) has been widely utilized to improve generalization in training deep neural networks.Recently, human-designed data augmentation has been gradually replaced by automatically learned augmentation policy.Through finding the best policy in well-designed search space of data augmentation, AutoAugment (Cubuk et al., 2019) can significantly improve validation accuracy on image classification tasks.However, this approach is not computationally practical for large-scale problems.In this paper, we develop an adversarial method to arrive at a computationally-affordable solution called Adversarial AutoAugment, which can simultaneously optimize target related object and augmentation policy search loss.The augmentation policy network attempts to increase the training loss of a target network through generating adversarial augmentation policies, while the target network can learn more robust features from harder examples to improve the generalization.In contrast to prior work, we reuse the computation in target network training for policy evaluation, and dispense with the retraining of the target network.Compared to AutoAugment, this leads to about 12x reduction in computing cost and 11x shortening in time overhead on ImageNet.We show experimental results of our approach on CIFAR-10/CIFAR-100, ImageNet, and demonstrate significant performance improvements over state-of-the-art.On CIFAR-10, we achieve a top-1 test error of 1.36%, which is the currently best performing single model.On ImageNet, we achieve a leading performance of top-1 accuracy 79.40% on ResNet-50 and 80.00% on ResNet-50-D without extra data.

Introduciamo l'idea dell'adversarial learning nella data augmentation automatica per migliorare la generalizzazione di una rete target.

Una tecnica chiamata Adversarial AutoAugment che impara dinamicamente buone policy di data augmentation durante il training usando un approccio adversarial.


In this study we focus on first-order meta-learning algorithms that aim to learn a parameter initialization of a network which can quickly adapt to new concepts, given a few examples.We investigate two approaches to enhance generalization and speed of learning of such algorithms, particularly expanding on the Reptile (Nichol et al., 2018) algorithm.We introduce a novel regularization technique called meta-step gradient pruning and also investigate the effects of increasing the depth of network architectures in first-order meta-learning.We present an empirical evaluation of both approaches, where we match benchmark few-shot image classification results with 10 times fewer iterations using Mini-ImageNet dataset and with the use of deeper networks, we attain accuracies that surpass the current benchmarks of few-shot image classification using Omniglot dataset.

Lo studio introduce due approcci per migliorare la generalizzazione del meta learning del primo ordine e presenta una valutazione empirica sulla classificazione few-shot di immagini.

L'articolo presenta uno studio empirico dell'algoritmo Reptile di meta learning del primo ordine, investigando una tecnica di regolarizzazione proposta e reti più profonde


In this paper, we propose the use of in-training matrix factorization to reduce the model size for neural machine translation.Using in-training matrix factorization, parameter matrices may be decomposed into the products of smaller matrices, which can compress large machine translation architectures by vastly reducing the number of learnable parameters.We apply in-training matrix factorization to different layers of standard neural architectures and show that in-training factorization is capable of reducing nearly 50% of learnable parameters without any associated loss in BLEU score.Further, we find that in-training matrix factorization is especially powerful on embedding layers, providing a simple and effective method to curtail the number of parameters with minimal impact on model performance, and, at times, an increase in performance.

Questo articolo propone di usare la fattorizzazione della matrice al momento del training per la traduzione automatica neurale, che può ridurre la dimensione del modello e diminuire il tempo di training senza impattare le prestazioni.

Questo articolo propone di comprimere i modelli usando la fattorizzazione della matrice durante il training per le deep neural network per machine translation.


Though state-of-the-art sentence representation models can perform tasks requiring significant knowledge of grammar, it is an open question how best to evaluate their grammatical knowledge.We explore five experimental methods inspired by prior work evaluating pretrained sentence representation models.We use a single linguistic phenomenon, negative polarity item (NPI) licensing, as a case study for our experiments.NPIs like 'any' are grammatical only if they appear in a licensing environment like negation ('Sue doesn't have any cats' vs. '*Sue has any cats').This phenomenon is challenging because of the variety of NPI licensing environments that exist.We introduce an artificially generated dataset that manipulates key features of NPI licensing for the experiments.We find that BERT has significant knowledge of these features, but its success varies widely across different experimental methods.We conclude that a variety of methods is necessary to reveal all relevant aspects of a model's grammatical knowledge in a given domain.

Diversi metodi di analisi di BERT suggeriscono conclusioni diverse (ma compatibili) in un caso di studio su NPI.


The primate visual system builds robust, multi-purpose representations of the external world in order to support several diverse downstream cortical processes.Such representations are required to be invariant to the sensory inconsistencies caused by dynamically varying lighting, local texture distortion, etc.A key architectural feature combating such environmental irregularities is â€˜long-range horizontal connectionsâ€™ that aid the perception of the global form of objects.In this work, we explore the introduction of such horizontal connections into standard deep convolutional networks; we present V1Net -- a novel convolutional-recurrent unit that models linear and nonlinear horizontal inhibitory and excitatory connections inspired by primate visual cortical connectivity.We introduce the Texturized Challenge -- a new benchmark to evaluate object recognition performance under perceptual noise -- which we use to evaluate V1Net against an array of carefully selected control models with/without recurrent processing.Additionally, we present results from an ablation study of V1Net demonstrating the utility of diverse neurally inspired horizontal connections for state-of-the-art AI systems on the task of object boundary detection from natural images.We also present the emergence of several biologically plausible horizontal connectivity patterns, namely center-on surround-off, association fields and border-ownership connectivity patterns in a V1Net model trained to perform boundary detection on natural images from the Berkeley Segmentation Dataset 500 (BSDS500).Our findings suggest an increased representational similarity between V1Net and biological visual systems, and highlight the importance of neurally inspired recurrent contextual processing principles for learning visual representations that are robust to perceptual noise and furthering the state-of-the-art in computer vision.

In questo lavoro, presentiamo V1Net - una nuova rete neurale ricorrente che modella le connessioni orizzontali corticali che danno origine a robuste rappresentazioni visive attraverso il raggruppamento percettivo.

Gli autori propongono di modificare una variante convoluzionale di LSTM per includere connessioni orizzontali ispirate alle interazioni note nella corteccia visiva.


Humans understand novel sentences by composing meanings and roles of core language components.In contrast, neural network models for natural language modeling fail when such compositional generalization is required.The main contribution of this paper is to hypothesize that language compositionality is a form of group-equivariance.Based on this hypothesis, we propose a set of tools for constructing equivariant sequence-to-sequence models.Throughout a variety of experiments on the SCAN tasks, we analyze the behavior of existing models under the lens of equivariance, and demonstrate that our equivariant architecture is able to achieve the type compositional generalization required in human language understanding.

Proponiamo un collegamento tra l'equivarianza di permutazione e la generalizzazione compositiva, e forniamo language model equivarianti

Questo lavoro si concentra sull'apprendimento di rappresentazioni e funzioni localmente equivarianti su parole di input/output ai fini del task SCAN.


Variational inference (VI) is a popular approach for approximate Bayesian inference that is particularly promising for highly parameterized models such as deep neural networks.  A key challenge of variational inference is to approximate the posterior over model parameters with a distribution that is simpler and tractable yet sufficiently expressive.In this work, we propose a method for training highly flexible variational distributions by starting with a coarse approximation and iteratively refining it.Each refinement step makes cheap, local adjustments and only requires optimization of simple variational families.We demonstrate theoretically that our method always improves a bound on the approximation (the Evidence Lower BOund) and observe this empirically across a variety of benchmark tasks.  In experiments, our method consistently outperforms recent variational inference methods for deep learning in terms of log-likelihood and the ELBO.  We see that the gains are further amplified on larger scale models, significantly outperforming standard VI and deep ensembles on residual networks on CIFAR10.

L'articolo propone un algoritmo per aumentare la flessibilità del posteriore variazionale nelle reti neurali bayesiane attraverso l'ottimizzazione iterativa.

Un metodo per addestrare distribuzioni posteriori variazionali flessibili, applicato alle reti neurali bayesiane per eseguire l'inferenza di variazione (VI) sui pesi.


In this paper, we propose a residual non-local attention network for high-quality image restoration.Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial- and channel-wise features.To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts.Specifically, we design trunk branch and (non-)local mask branch in each (non-)local attention block.The trunk branch is used to extract hierarchical features.Local and non-local mask branches aim to adaptively rescale these hierarchical features with mixed attentions.The local mask branch concentrates on more local structures with convolutional operations, while non-local attention considers more about long-range dependencies in the whole feature map.Furthermore, we propose residual local and non-local attention learning to train the very deep network, which further enhance the representation ability of the network.Our proposed method can be generalized for various image restoration applications, such as image denoising, demosaicing, compression artifacts reduction, and super-resolution.Experiments demonstrate that our method obtains comparable or better results compared with recently leading methods quantitatively and visually.

Nuovo framework allo stato dell'arte per il restauro delle immagini

L'articolo propone un'architettura di rete neurale convoluzionale che include blocchi per meccanismi di attention locale e non locale, che sono ritenuti responsabili del raggiungimento di risultati eccellenti in quattro applicazioni di restauro delle immagini.

Questo articolo propone una rete di attention residua non locale per il restauro delle immagini


Most approaches to learning action planning models heavily rely on a significantly large volume of training samples or plan observations.In this paper, we adopt a different approach based on deductive learning from domain-specific knowledge, specifically from logic formulae that specify constraints about the possible states of a given domain.The minimal input observability required by our approach is a single example composed of a full initial state and a partial goal state.We will show that exploiting specific domain knowledge enables to constrain the space of possible action models as well as to complete partial observations, both of which turn out helpful to learn good-quality action models.

Approccio ibrido all'acquisizione di modelli che compensa la mancanza di dati disponibili con la conoscenza specifica del dominio fornita da esperti

Un approccio di acquisizione del dominio che considera l'uso di una rappresentazione diversa per il modello di dominio parziale utilizzando relazioni mutex schematiche al posto delle condizioni pre/post.


We release the largest public ECG dataset of continuous raw signals for representation learning containing over 11k patients and 2 billion labelled beats.Our goal is to enable semi-supervised ECG models to be made as well as to discover unknown subtypes of arrhythmia and anomalous ECG signal events.To this end, we propose an unsupervised representation learning task, evaluated in a semi-supervised fashion.  We provide a set of baselines for different feature extractors that can be built upon.  Additionally, we perform qualitative evaluations on results from PCA embeddings, where we identify some clustering of known subtypes indicating the potential for representation learning in arrhythmia sub-type discovery.

Rilasciamo un dataset costruito a partire dai dati ECG single-lead di 11.000 pazienti a cui è stato prescritto l'uso del dispositivo {DEVICENAME}(TM).

Questo articolo descrive un dataset ECG su larga scala che gli autori intendono pubblicare e fornisce un'analisi e una visualizzazione non supervisionata del dataset.


As the basic building block of Convolutional Neural Networks (CNNs), the convolutional layer is designed to extract local patterns and lacks the ability to model global context in its nature.Many efforts have been recently made to complement CNNs with the global modeling ability, especially by a family of works on global feature interaction.In these works, the global context information is incorporated into local features before they are fed into convolutional layers.However, research on neuroscience reveals that, besides influences changing the inputs to our neurons, the neurons' ability of modifying their functions dynamically according to context is essential for perceptual tasks, which has been overlooked in most of CNNs.Motivated by this, we propose one novel Context-Gated Convolution (CGC) to explicitly modify the weights of convolutional layers adaptively under the guidance of global context.As such, being aware of the global context, the modulated convolution kernel of our proposed CGC can better extract representative local patterns and compose discriminative features.Moreover, our proposed CGC is lightweight, amenable to modern CNN architectures, and consistently improves the performance of CNNs according to extensive experiments on image classification, action recognition, and machine translation.

Un nuovo Context-Gated Convolution che incorpora informazioni globali sul contesto nelle CNN modulando esplicitamente i kernel di convoluzione, e quindi cattura modelli locali più rappresentativi ed estrae feature discriminanti.

Questo articolo usa il contesto globale per modulare i pesi dei layer convoluzionali e aiutare le CNN a catturare più feature discriminanti con alte prestazioni e meno parametri rispetto alla modulazione della feature map.


We analyze the trade-off between quantization noise and clipping distortion in low precision networks.We identify the statistics of various tensors, and derive exact expressions for the mean-square-error degradation due to clipping.By optimizing these expressions, we show marked improvements over standard quantization schemes that normally avoid clipping.For example, just by choosing the accurate clipping values, more than 40\% accuracy improvement is obtained for the quantization of VGG-16 to 4-bits of precision.Our results have many applications for the quantization of neural networks at both training and inference time.

Analizziamo il trade-off tra il rumore di quantizzazione e la distorsione di clipping nelle reti a bassa precisione, e mostriamo miglioramenti marcati rispetto agli schemi di quantizzazione standard che normalmente evitano il clipping

Deriva una formula per trovare i valori minimi e massimi di clipping per la quantizzazione uniforme che minimizzano l'errore quadratico risultante dalla quantizzazione, per una distribuzione Laplace o Gaussiana sul valore pre-quantizzato.


Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field.But its performance can awfully degrade with insufficient batch size.This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of memory consumption.Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and increase huge consumption.In this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before.The extra batch statistics associated with gradients also can severely affect the training of deep neural network.Based on our analysis, we propose a novel normalization method, named Moving Average Batch Normalization (MABN).MABN can completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations in inference procedure.We prove the benefits of MABN by both theoretical analysis and experiments.Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO.The code has been released in https://github.com/megvii-model/MABN.

Proponiamo un nuovo metodo di normalizzazione per gestire i casi aventi piccole batch size.

Un metodo per affrontare il problema della piccola batch size di BN che applica l'operazione di media mobile senza troppo overhead e riduce il numero di statistiche di BN per una migliore stabilità.


We present a simple proof for the benefit of depth in multi-layer feedforward network with rectifed activation (``"depth separation").Specifically we present a sequence of classification problems f_i such that(a) for any fixed depth rectified network we can find an index m such that problems with index > m require exponential network width to fully represent the function f_m; and(b) for any problem f_m in the family, we present a concrete neural network with linear depth and bounded width that fully represents it.While there are several previous work showing similar results, our proof uses substantially simpler tools and techniques, and should be accessible to undergraduate students in computer science and people with similar backgrounds.

Dimostrazione di separazione della profondità per ReLU MLP con argomenti geometrici

Una prova che le reti più profonde hanno bisogno di meno unità di quelle meno profonde per una famiglia di problemi.


The rich and accessible labeled data fuel the revolutionary success of deep learning.Nonetheless, massive supervision remains a luxury for many real applications, boosting great interest in label-scarce techniques such as few-shot learning (FSL).An intuitively feasible approach to FSL is to conduct data augmentation via synthesizing additional training samples.The key to this approach is how to guarantee both discriminability and diversity of the synthesized samples.In this paper, we propose a novel FSL model, called $\textrm{D}^2$GAN, which synthesizes Diverse and Discriminative features based on Generative Adversarial Networks (GAN).$\textrm{D}^2$GAN secures discriminability of the synthesized features by constraining them to have high correlation with real features of the same classes while low correlation with those of different classes.  Based on the observation that noise vectors that are closer in the latent code space are more likely to be collapsed into the same mode when mapped to feature space, $\textrm{D}^2$GAN incorporates a novel anti-collapse regularization term, which encourages feature diversity by penalizing the ratio of the logarithmic similarity of two synthesized features and the logarithmic similarity of the latent codes generating them.Experiments on three common benchmark datasets verify the effectiveness of $\textrm{D}^2$GAN by comparing with the state-of-the-art.

Un nuovo algoritmo di apprendimento few-shot basato su GAN sintetizzando feature diverse e discriminanti

Un metodo di meta learning che impara un modello generativo che può aumentare il support set di un few-shot learner che ottimizza una combinazione di loss.


The lack of crisp mathematical models that capture the structure of real-worlddata sets is a major obstacle to the detailed theoretical understanding of deepneural networks.Here, we first demonstrate the effect of structured data setsby experimentally comparing the dynamics and the performance of two-layernetworks trained on two different data sets:(i) an unstructured synthetic dataset containing random i.i.d. inputs, and(ii) a simple canonical data set suchas MNIST images.Our analysis reveals two phenomena related to the dynamics ofthe networks and their ability to generalise that only appear when training onstructured data sets.Second, we introduce a generative model for data sets,where high-dimensional inputs lie on a lower-dimensional manifold and havelabels that depend only on their position within this manifold.We call it the*hidden manifold model* and we experimentally demonstrate that trainingnetworks on data sets drawn from this model reproduces both the phenomena seenduring training on MNIST.

Dimostriamo come la struttura nei dataset abbia un impatto sulle reti neurali e introduciamo un modello generativo per dataset sintetici che riproduce questo impatto.

L'articolo studia come diversi setting della struttura dei dati influenzino l'apprendimento delle reti neurali e come imitare il comportamento su dataset reali quando si effettua il learning su uno sintetico.


In this paper, we study deep diagonal circulant neural networks, that is deep neural networks in which weight matrices are the product of diagonal and circulant ones.Besides making a theoretical analysis of their expressivity, we introduced principled techniques for training these models: we devise an initialization scheme and proposed a smart use of non-linearity functions in order to train deep diagonal circulant networks. Furthermore, we show that these networks outperform recently introduced deep networks with other types of structured layers.We conduct a thorough experimental study to compare the performance of deep diagonal circulant networks with state of the art models based on structured matrices and with dense models.We show that our models achieve better accuracy than other structured approaches while required 2x fewer weights as the next best approach.Finally we train deep diagonal circulant networks to build a compact and accurate models on a real world video classification dataset with over 3.8 million training examples.

Addestriamo deep neural network basate su matrici diagonali e circolanti, e dimostriamo che questo tipo di reti sono sia compatte che accurate in applicazioni del mondo reale.

Gli autori forniscono un'analisi teorica della potenza espressiva delle reti neurali diagonali circolanti (DCNN) e propongono uno schema di inizializzazione per le DCNN deep.


Interpretability has largely focused on local explanations, i.e. explaining why a model made a particular prediction for a sample.These explanations are appealing due to their simplicity and local fidelity.However, they do not provide information about the general behavior of the model.We propose to leverage model distillation to learn global additive explanations that describe the relationship between input features and model predictions.These global explanations take the form of feature shapes, which are more expressive than feature attributions.Through careful experimentation, we show qualitatively and quantitatively that global additive explanations are able to describe model behavior and yield insights about models such as neural nets.A visualization of our approach applied to a neural net as it is trained is available at https://youtu.be/ErQYwNqzEdc

Proponiamo di sfruttare la model distillation per imparare spiegazioni additive globali sotto forma di forme di feature (che sono più espressive delle attribuzioni di feature) per modelli come le reti neurali addestrate su dati tabulari.

Questo articolo incorpora i Generalized Additive Model (GAM) con la model distillation per fornire spiegazioni globali delle reti neurali.


A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner.These representations are typically used as general purpose features for words across a range of NLP problems.However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem.Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations.In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences.Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods.We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

Una framework di apprendimento multi-task su larga scala con diversi obiettivi di training per imparare rappresentazioni di frasi di lunghezza fissa

Questo articolo riguarda l'apprendimento di embeddings di frasi combinando diversi segnali di training: skip-thought, predizione della traduzione, classificazione delle relazioni di implicazione e predizione del parsing dei costituenti.


In a time where neural networks are increasingly adopted in sensitive applications, algorithmic bias has emerged as an issue with moral implications.While there are myriad ways that a system may be compromised by bias, systematically isolating and evaluating existing systems on such scenarios is non-trivial, i.e., bias may be subtle, natural and inherently difficult to quantify.To this end, this paper proposes the first systematic study of benchmarking state-of-the-art neural models against biased scenarios.More concretely, we postulate that the bias annotator problem can be approximated with neural models, i.e., we propose generative models of latent bias to deliberately and unfairly associate latent features to a specific class.All in all, our framework provides a new way for principled quantification and evaluation of models against biased datasets.Consequently, we find that state-of-the-art NLP models (e.g., BERT, RoBERTa, XLNET) are readily compromised by biased data.

Proponiamo un annotatore neurale di bias per confrontare i modelli sulla loro robustezza ai dataset di testo distorti.

Un metodo per generare dataset biased per NLP, basandosi su un autoencoder condizionato regolarizzato dall'avversario (CARA).


We consider the problem of topic modeling in a weakly semi-supervised setting.In this scenario, we assume that the user knows a priori a subset of the topics she wants the model to learn and is able to provide a few exemplar documents for those topics.In addition, while each document may typically consist of multiple topics, we do not assume that the user will identify all its topics exhaustively.      Recent state-of-the-art topic models such as NVDM, referred to herein as Neural Topic Models (NTMs), fall under the variational autoencoder framework.We extend NTMs to the weakly semi-supervised setting by using informative priors in the training objective.After analyzing the effect of informative priors, we propose a simple modification of the NVDM model using a logit-normal posterior that we show achieves better alignment to user-desired topics versus other NTM models.

Proponiamo di supervisionare i topic model in stile VAE regolando in modo intelligente il prior per ogni documento. Troviamo che un posterior logit-normale fornisce le migliori prestazioni.

Un metodo flessibile di supervisionare debolmente un topic model per ottenere un migliore allineamento con l'intuizione dell'utente.


Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently as a tool to gain insight into, among others, their generalization ability.However, it is by no means obvious how to estimate mutual information (MI) between each hidden layer and the input/desired output, to construct the IP.For instance, hidden layers with many neurons require MI estimators with robustness towards the high dimensionality associated with such layers.MI estimators should also be able to naturally handle convolutional layers, while at the same time being computationally tractable to scale to large networks.None of the existing IP methods to date have been able to study truly deep Convolutional Neural Networks (CNNs), such as the e.g.\ VGG-16.In this paper, we propose an IP analysis using the new matrix--based R\'enyi's entropy coupled with tensor kernels over convolutional layers, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data.The obtained results shed new light on the previous literature concerning small-scale DNNs, however using a completely new approach.Importantly, the new framework enables us to provide the first comprehensive IP analysis of contemporary large-scale DNNs and CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.

Prima analisi completa del piano di informazione delle deep neural network su larga scala utilizzando l'entropia basata sulla matrice e i kernel tensori.

Gli autori propongono uno stimatore basato su tensor-kernel per la stima dell'informazione mutua tra layer ad alta dimensionalità in una rete neurale.


Developing agents that can learn to follow natural language instructions has been an emerging research direction.While being accessible and flexible, natural language instructions can sometimes be ambiguous even to humans.To address this, we propose to utilize programs, structured in a formal language, as a precise and expressive way to specify tasks.We then devise a modular framework that learns to perform a task specified by a program â€“ as different circumstances give rise to diverse ways to accomplish the task, our framework can perceive which circumstance it is currently under, and instruct a multitask policy accordingly to fulfill each subtask of the overall task.Experimental results on a 2D Minecraft environment not only demonstrate that the proposed framework learns to reliably accomplish program instructions and achieves zero-shot generalization to more complex instructions but also verify the efficiency of the proposed modulation mechanism for learning the multitask policy.We also conduct an analysis comparing various models which learn from programs and natural language instructions in an end-to-end fashion.

Proponiamo un framework modulare che può eseguire task specificati da programmi e raggiungere una generalizzazione zero-shot per task più complessi.

Questo articolo studia il training di agenti RL con istruzioni e decomposizioni di task formalizzati come programmi, proponendo un modello per un agente guidato dal programma che interpreta un programma e propone sotto-obiettivi a un modulo di azione.


We analyze the convergence of (stochastic) gradient descent algorithm for learning a convolutional filter with Rectified Linear Unit (ReLU) activation function.Our analysis does not rely on any specific form of the input distribution and our proofs only use the definition of ReLU, in contrast with previous works that are restricted to standard Gaussian input.We show that (stochastic) gradient descent with random initialization can learn the convolutional filter in polynomial time and the convergence rate depends on the smoothness of the input distribution and the closeness of patches.To the best of our knowledge, this is the first recovery guarantee of gradient-based algorithms for convolutional filter on non-Gaussian input distributions.Our theory also justifies the two-stage learning rate strategy in deep neural networks.While our focus is theoretical, we also present experiments that justify our theoretical findings.

Dimostriamo che la gradient descent inizializzata in modo casuale (stocastico) impara un filtro convoluzionale in tempo polinomiale.

Studia il problema dell'apprendimento di un singolo filtro convoluzionale usando SGD e mostra che sotto certe condizioni, SGD impara un singolo filtro convoluzionale.

Questo articolo estende l'ipotesi di distribuzione gaussiana a un'ipotesi di smoothness angolare più generale, che copre una famiglia più ampia di distribuzioni di input


Deep neural networks (DNNs) are widely adopted in real-world cognitive applications because of their high accuracy.The robustness of DNN models, however, has been recently challenged by adversarial attacks where small disturbance on input samples may result in misclassification.State-of-the-art defending algorithms, such as adversarial training or robust optimization, improve DNNs' resilience to adversarial attacks by paying high computational costs.Moreover, these approaches are usually designed to defend one or a few known attacking techniques only.The effectiveness to defend other types of attacking methods, especially those that have not yet been discovered or explored, cannot be guaranteed.This work aims for a general approach of enhancing the robustness of DNN models under adversarial attacks.In particular, we propose Bamboo -- the first data augmentation method designed for improving the general robustness of DNN without any hypothesis on the attacking algorithms.Bamboo augments the training data set with a small amount of data uniformly sampled on a fixed radius ball around each training data and hence, effectively increase the distance between natural data points and decision boundary.Our experiments show that Bamboo substantially improve the general robustness against arbitrary types of attacks and noises, achieving better results comparing to previous adversarial training methods, robust optimization methods and other data augmentation methods with the same amount of data points.

Il primo metodo di data augmentation appositamente progettato per migliorare la robustezza generale di DNN senza alcuna ipotesi sugli algoritmi di attacco.

Propone un metodo di training di data augmentation per guadagnare la robustezza del modello contro le adversarial perturbation, aumentando i sample casuali in modo uniforme da una sfera a raggio fisso centrata sui dati di training.


The ability to synthesize realistic patterns of neural activity is crucial for studying neural information processing.Here we used the Generative Adversarial Networks (GANs) framework to simulate the concerted activity of a population of neurons.We adapted the Wasserstein-GAN variant to facilitate the generation of unconstrained neural population activity patterns while still benefiting from parameter sharing in the temporal domain.We demonstrate that our proposed GAN, which we termed Spike-GAN, generates spike trains that match accurately the first- and second-order statistics of datasets of tens of neurons and also approximates well their higher-order statistics.We applied Spike-GAN to a real dataset recorded from salamander retina and showed that it performs as well as state-of-the-art approaches based on the maximum entropy and the dichotomized Gaussian frameworks.Importantly, Spike-GAN does not require to specify a priori the statistics to be matched by the model, and so constitutes a more flexible method than these alternative approaches.Finally, we show how to exploit a trained Spike-GAN  to construct 'importance maps' to detect the most relevant statistical structures present in a spike train. Spike-GAN provides a powerful, easy-to-use technique for generating realistic spiking neural activity and for describing the most relevant features of the large-scale neural population recordings studied in modern systems neuroscience.

Utilizzo di Wasserstein-GANs per generare attività neurale realistica e per rilevare le feature più rilevanti presenti nei pattern di popolazione neurale.

Un metodo per simulare spike train da popolazioni di neuroni che corrispondono a dati empirici utilizzando una GAN semi-convoluzionale.

L'articolo propone di usare le GAN per sintetizzare modelli realistici di attività neurale


Deep latent variable models have become a popular model choice due to the scalable learning algorithms introduced by (Kingma & Welling 2013, Rezende et al. 2014).These approaches maximize a variational lower bound on the intractable log likelihood of the observed data.Burda et al. (2015) introduced a multi-sample variational bound, IWAE, that is at least as tight as the standard variational lower bound and becomes increasingly tight as the number of samples increases.Counterintuitively, the typical inference network gradient estimator for the IWAE bound performs poorly as the number of samples increases (Rainforth et al. 2018, Le et al. 2018).Roeder et a.(2017) propose an improved gradient estimator, however, are unable to show it is unbiased.We show that it is in fact biased and that the bias can be estimated efficiently with a second application of the reparameterization trick.The doubly reparameterized gradient (DReG) estimator does not suffer as the number of samples increases, resolving the previously raised issues.The same idea can be used to improve many recently introduced training techniques for latent variable models.In particular, we show that this estimator reduces the variance of the IWAE gradient, the reweighted wake-sleep update (RWS) (Bornschein & Bengio 2014), and the jackknife variational inference (JVI) gradient (Nowozin 2018).Finally, we show that this computationally efficient, drop-in estimator translates to improved performance for all three objectives on several modeling tasks.

Gli stimatori di gradiente doppiamente riparametrizzati forniscono una riduzione imparziale della varianza che porta a migliori prestazioni.

L'autore ha trovato sperimentalmente che lo stimatore del lavoro esistente è biased e propone di ridurre il bias per migliorare lo stimatore del gradiente dell'ELBO.


Zeroth-order optimization is the process of minimizing an objective $f(x)$, given oracle access to evaluations at adaptively chosen inputs $x$.In this paper, we present two simple yet powerful GradientLess Descent (GLD) algorithms that do not rely on an underlying gradient estimate and are numerically stable.We analyze our algorithm from a novel geometric perspective and we show that for {\it any monotone transform} of a smooth and strongly convex objective with latent dimension $k \ge n$, we present a novel analysis that shows convergence within an $\epsilon$-ball of the optimum in $O(kQ\log(n)\log(R/\epsilon))$ evaluations, where the input dimension is $n$, $R$ is the diameter of the input space and $Q$ is the condition number.Our rates are the first of its kind to be both1) poly-logarithmically dependent on dimensionality and2) invariant under monotone transformations.We further leverage our geometric perspective to show that our analysis is optimal.Both monotone invariance and its ability to utilize a low latent dimensionality are key to the empirical success of our algorithms, as demonstrated on synthetic and MuJoCo benchmarks.

Gradientless Descent è un algoritmo senza gradiente provatamente efficiente che è monotono-invariante e veloce per l'ottimizzazione di ordine zero ad alta dimensionalità.

Questo articolo propone algoritmi GradientLess Descent (GLD) stabili che non si basano sulla stima del gradiente.


Many processes can be concisely represented as a sequence of events leading from a starting state to an end state.Given raw ingredients, and a finished cake, an experienced chef can surmise the recipe.Building upon this intuition,  we propose a new class of visual generative models: goal-conditioned predictors (GCP).Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video.GCP instead treats videos as start-goal transformations, making video generation easier by conditioning on the more informative context provided by the first and final frames. Not only do existing forward prediction approaches synthesize better and longer videos when modified to become goal-conditioned,  but GCP models can also utilize structures that are not linear in time, to accomplish hierarchical prediction. To this end, we study both auto-regressive GCP models and novel tree-structured GCP models that generate frames recursively, splitting the video iteratively into finer and finer segments delineated by subgoals. In experiments across simulated and real datasets, our GCP methods generate high-quality sequences over long horizons.  Tree-structured GCPs are also substantially easier to parallelize than auto-regressive GCPs, making training  and  inference  very  efficient, and allowing the model to train on sequences that are thousands of frames in length.Finally, we demonstrate the utility of GCP approaches for imitation learning in the setting without access to expert actions. Videos are on the supplementary website: https://sites.google.com/view/video-gcp

Proponiamo una nuova classe di modelli generativi visivi: i predittori condizionati dall'obiettivo. Mostriamo sperimentalmente che il condizionamento sull'obiettivo permette di ridurre l'incertezza e produrre previsioni su orizzonti molto più lunghi.

Questo articolo riformula il problema della predizione video come interpolazione invece di estrapolazione, condizionando la predizione sul fotogramma iniziale e finale (obiettivo), ottenendo predizioni di qualità superiore.


Recent advances in computing technology and sensor design have made it easier to collect longitudinal or time series data from patients, resulting in a gigantic amount of available medical data.Most of the medical time series lack annotations or even when the annotations are available they could be subjective and prone to human errors.Earlier works have developed natural language processing techniques to extract concept annotations and/or clinical narratives from doctor notes.However, these approaches are slow and do not use the accompanying medical time series data.To address this issue, we introduce the problem of concept annotation for the medical time series data, i.e., the task of predicting and localizing medical concepts by using the time series data as input.We propose Relational Multi-Instance Learning (RMIL) - a deep Multi Instance Learning framework based on recurrent neural networks, which uses pooling functions and attention mechanisms for the concept annotation tasks.Empirical results on medical datasets show that our proposed models outperform various multi-instance learning models.

Proponiamo un framework profondo di Multi Instance Learning basato su reti neurali ricorrenti che utilizza funzioni di pooling e meccanismi di attention per i task di concept annotation.

L'articolo affronta la classificazione dei dati delle serie temporali mediche e propone di modellare la relazione temporale tra le istanze di ogni serie utilizzando un'architettura di rete neurale ricorrente.

Propone una nuova formulazione di Multiple Instance Learning (MIL) chiamata Relation MIL (RMIL), e discute una serie di sue varianti con LSTM, Bi-LSTM, S2S, ecc. ed esplora l'integrazione di RMIL con vari meccanismi di attention, e dimostra il suo utilizzo nella predizione di concetti medici da dati di serie temporali.


The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing.However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting.We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance.  We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.

I layer di embedding sono fattorizzati con la decomposizione Tensor Train per ridurre il loro impatto sulla memoria.

Questo articolo propone un modello di decomposizione tensoriale low-rank per parametrizzare la matrice di embedding nel Natural Language Processing (NLP), che comprime la rete e talvolta aumenta la precisione sul test set.


We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor. We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function.We provide empirical evidence that our proposed modification(i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and(ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter).We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence.Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32. Our source code will become available after the review process.

Weight decay regularization nei metodi a gradiente adattivo come Adam

Propone l'idea di disaccoppiare il weight decay dal numero di step del processo di ottimizzazione.

L'articolo presenta un modo alternativo per implementare il weight decay in Adam con risultati empirici

Studia i problemi di weight decay nelle varianti SGD e propone il metodo di disaccoppiamento tra il weight decay e l'aggiornamento basato sul gradiente.


Lifelong learning is the problem of learning multiple consecutive tasks in a sequential manner where knowledge gained from previous tasks is retained and used for future learning.It is essential towards the development of intelligent machines that can adapt to their surroundings.In this work we focus on a lifelong learning approach to generative modeling where we continuously incorporate newly observed streaming distributions into our learnt model.We do so through a student-teacher architecture which allows us to learn and preserve all the distributions seen so far without the need to retain the past data nor the past models.Through the introduction of a novel cross-model regularizer, the student model leverages the information learnt by the teacher, which acts as a summary of everything seen till now.The regularizer has the additional benefit of reducing the effect of catastrophic interference that appears when we learn over streaming data.We demonstrate its efficacy on streaming distributions as well as its ability to learn a common latent representation across a complex transfer learning scenario.

Lifelong distributional learning attraverso un'architettura teacher-student accoppiata con un posterior regularizer cross model.


Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling.In this paper, we look at geometric data represented as point clouds.We introduce a deep autoencoder (AE) network with excellent reconstruction quality and generalization ability.The learned representations outperform the state of the art in 3D recognition tasks and enable basic shape editing applications via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation.We also perform a thorough study of different generative models including GANs operating on the raw point clouds, significantly improved GANs trained in the fixed latent space our AEs and, Gaussian mixture models (GMM).Interestingly, GMMs trained in the latent space of our AEs produce samples of the best fidelity and diversity.To perform our quantitative evaluation of generative models, we propose simple measures of fidelity and diversity based on optimally matching between sets point clouds.

Deep autoencoder per imparare una buona rappresentazione per i dati geometrici delle nuvole di punti 3D; Modelli generativi per le nuvole di punti.

Approcci per apprendere modelli generativi di tipo GAN usando l'architettura PointNet e il latent-space GAN.


Despite the remarkable performance of deep neural networks (DNNs) on various tasks, they are susceptible to adversarial perturbations which makes it difficult to deploy them in real-world safety-critical applications.In this paper, we aim to obtain robust networks by sparsifying DNN's latent features sensitive to adversarial perturbation.Specifically, we define vulnerability at the latent feature space and then propose a Bayesian framework to prioritize/prune features based on their contribution to both the original and adversarial loss.We also suggest regularizing the features' vulnerability during training to improve robustness further.While such network sparsification has been primarily studied in the literature for computational efficiency and regularization effect of DNNs, we confirm that it is also useful to design a defense mechanism through quantitative evaluation and qualitative analysis.We validate our method, \emph{Adversarial Neural Pruning (ANP)} on multiple benchmark datasets, which results in an improvement in test accuracy and leads to state-of-the-art robustness.ANP also tackles the practical problem of obtaining sparse and robust networks at the same time, which could be crucial to ensure adversarial robustness on lightweight networks deployed to computation and memory-limited devices.

Proponiamo un nuovo metodo per sopprimere la vulnerabilità dello spazio delle feature latenti per ottenere reti robuste e compatte.

Questo articolo propone un metodo di "adversarial neural pruning" per addestrare una maschera di pruning e una nuova loss di soppressione della vulnerabilità per migliorare la precisione e la adversarial robustness.


In anomaly detection (AD), one seeks to identify whether a test sample is abnormal,  given a data set of normal samples.   A recent and promising approach to AD relies on deep generative models, such as variational autoencoders (VAEs),for unsupervised learning of the normal data distribution.In semi-supervised AD (SSAD), the data also includes a small sample of labeled anomalies.In this work,we propose two variational methods for training VAEs for SSAD.The intuitive idea in both methods is to train the encoder to â€˜separateâ€™ between latent vectors for normal and outlier data.We show that this idea can be derived from principled probabilistic formulations of the problem, and propose simple and effective algorithms.  Our methods can be applied to various data types, as we demonstrate on SSAD datasets ranging from natural images to astronomy and medicine, and can be combined with any VAE model architecture.When comparing to state-of-the-art SSAD methods that are not specific to particular data types, we obtain marked improvement in outlier detection.

Abbiamo proposto due modifiche VAE che tengono conto dei data example negativi, e li abbiamo utilizzati per il rilevamento semi-supervised delle anomalie.

Gli articoli propongono due metodi di approccio simili a VAE per il rilevamento semi-supervised della novlety, MML-VAE e DP-VAE.


We introduce dynamic instance hardness (DIH) to facilitate the training of machine learning models.DIH is a property of each training sample and is computed as the running mean of the sample's instantaneous hardness as measured over the training history.We use DIH to evaluate how well a model retains knowledge about each training sample over time.We find that for deep neural nets (DNNs), the DIH of a sample in relatively early training stages reflects its DIH in later stages and as a result, DIH can be effectively used to reduce the set of training samples in future epochs.Specifically, during each epoch, only samples with high DIH are trained (since they are historically hard) while samples with low DIH can be safely ignored.DIH is updated each epoch only for the selected samples, so it does not require additional computation.Hence, using DIH during training leads to an appreciable speedup.Also, since the model is focused on the historically more challenging samples, resultant models are more accurate.The above, when formulated as an algorithm, can be seen as a form of curriculum learning, so we call our framework DIH curriculum learning (or DIHCL).The advantages of DIHCL, compared to other curriculum learning approaches, are: (1) DIHCL does not require additional inference steps over the data not selected by DIHCL in each epoch, (2) the dynamic instance hardness, compared to static instance hardness (e.g., instantaneous loss), is more stable as it integrates information over the entire training history up to the present time.Making certain mathematical assumptions, we formulate the problem of DIHCL as finding a curriculum that maximizes a multi-set function $f(\cdot)$, and derive an approximation bound for a DIH-produced curriculum relative to the optimal curriculum.Empirically, DIHCL-trained DNNs significantly outperform random mini-batch SGD and other recently developed curriculum learning methods in terms of efficiency, early-stage convergence, and final performance, and this is shown in training several state-of-the-art DNNs on 11 modern datasets.

La nuova comprensione delle dinamiche di training e le metriche di hardness della memorizzazione portano ad un efficiente e dimostrabile curriculum learning.

Questo articolo formula DIH come un problema di curriculum learning che può utilizzare più efficacemente i dati per addestrare le DNN, e deriva la teoria sul bound di approssimazione.


This paper explores many immediate connections between adaptive control and machine learning, both through common update laws as well as common concepts.Adaptive control as a field has focused on mathematical rigor and guaranteed convergence.The rapid advances in machine learning on the other hand have brought about a plethora of new techniques and problems for learning.This paper elucidates many of the numerous common connections between both fields such that results from both may be leveraged together to solve new problems.In particular, a specific problem related to higher order learning is solved through insights obtained from these intersections.

Storia degli sviluppi paralleli delle regole di update e dei concetti tra controllo adattivo e ottimizzazione nell'apprendimento automatico.


Recurrent convolution (RC) shares the same convolutional kernels and unrolls them multiple times, which is originally proposed to model time-space signals.We suggest that RC can be viewed as a model compression strategy for deep convolutional neural networks.RC reduces the redundancy across layers and is complementary to most existing model compression approaches.However, the performance of an RC network can't match the performance of its corresponding standard one, i.e. with the same depth but independent convolutional kernels.  This reduces the value of RC for model compression.In this paper, we propose a simple variant which improves RC networks: The batch normalization layers of an RC module are learned independently (not shared) for different unrolling steps.We provide insights on why this works.Experiments on CIFAR show that unrolling a convolutional layer several steps can improve the performance, thus indirectly plays a role in model compression.

Convoluzione ricorrente per la compressione del modello e un trucco per addestrarlo, cioè l'apprendimento di layer BN indipendenti sugli step.

L'autore modifica la rete neurale di convoluzione ricorrente (RCNN) con una batch normalization indipendente, con i risultati sperimentali su RCNN comparabili con l'architettura della rete neurale ResNet quando contiene lo stesso numero di layer.


The visual world is vast and varied, but its variations divide into structured and unstructured factors.Structured factors, such as scale and orientation, admit clear theories and efficient representation design.Unstructured factors, such as what it is that makes a cat look like a cat, are too complicated to model analytically, and so require free-form representation learning.We compose structured Gaussian filters and free-form filters, optimized end-to-end, to factorize the representation for efficient yet general learning.Our experiments on dynamic structure, in which the structured filters vary with the input, equal the accuracy of dynamic inference with more degrees of freedom while improving efficiency.(Please see https://arxiv.org/abs/1904.11487 for the full edition.)

I campi recettivi dinamici con struttura gaussiana spaziale sono accurati ed efficienti.

Questo articolo propone un operatore di convoluzione strutturato per modellare le deformazioni delle regioni locali di un'immagine, che ha ridotto significativamente il numero di parametri.


It is widely known that well-designed perturbations can cause state-of-the-art machine learning classifiers to mis-label an image, with sufficiently small perturbations that are imperceptible to the human eyes.However, by detecting the inconsistency between the image and wrong label, the human observer would be alerted of the attack.In this paper, we aim to design attacks that not only make classifiers generate wrong labels, but also make the wrong labels imperceptible to human observers.To achieve this, we propose an algorithm called LabelFool which identifies a target label similar to the ground truth label and finds a perturbation of the image for this target label.We first find the target label for an input image by a probability model, then move the input in the feature space towards the target label.Subjective studies on ImageNet show that in the label space, our attack is much less recognizable by human observers, while objective experimental results on ImageNet show that we maintain similar performance in the image space as well as attack rates to state-of-the-art attack algorithms.

Un trucco sugli adversarial sample in modo che le label mal classificate siano impercettibili nello spazio delle label agli osservatori umani

Un metodo per costruire adversarial attack che sono meno rilevabili dall'uomo senza costi nello spazio dell'immagine cambiando la classe di destinazione per essere simile alla classe originale dell'immagine.


This paper presents noise type/position classification of various impact noises generated in a building which is a serious conflict issue in apartment complexes.For this study, a collection of floor impact noise dataset is recorded with a single microphone.Noise types/positions are selected based on a report by the Floor Management Center under Korea Environmental Corporation.Using a convolutional neural networks based classifier, the impact noise signals converted to log-scaled Mel-spectrograms are classified into noise types or positions.Also, our model is evaluated on a standard environmental sound dataset ESC-50 to show extensibility on environmental sound classification.

Questo documento presenta la classificazione del tipo/posizione di vari rumori d'impatto generati in un edificio, che è un serio problema di conflitto nei complessi di appartamenti

Questo lavoro descrive l'uso delle reti neurali convoluzionali in una nuova area di applicazione relativa alla classificazione del tipo di rumore degli edifici e della posizione del rumore.


Recordings of neural circuits in the brain reveal extraordinary dynamical richness and high variability.At the same time, dimensionality reduction techniques generally uncover low-dimensional structures underlying these dynamics.What determines the dimensionality of activity in neural circuits?What is the functional role of dimensionality in behavior and task learning?In this work we address these questions using recurrent neural network (RNN) models.We find that, depending on the dynamics of the initial network, RNNs learn to increase and reduce dimensionality in a way that matches task demands.These findings shed light on fundamental dynamical mechanisms by which neural networks solve tasks with robust representations that generalize to new cases.

Le reti neurali ricorrenti imparano ad aumentare e ridurre la dimensionalità della loro rappresentazione interna in un modo che corrisponde al task, a seconda della dinamica della rete iniziale.


Domain adaptation addresses the common problem when the target distribution generating our test data drifts from the source (training) distribution.While absent assumptions, domain adaptation is impossible, strict conditions, e.g. covariate or label shift, enable principled algorithms.Recently-proposed domain-adversarial approaches consist of aligning source and target encodings, often motivating this approach as minimizing two (of three) terms in a theoretical bound on target error.Unfortunately, this minimization can cause arbitrary increases in the third term, e.g. they can break down under shifting label distributions.We propose asymmetrically-relaxed distribution alignment, a new approach that overcomes some limitations of standard domain-adversarial algorithms.Moreover, we characterize precise assumptions under which our algorithm is theoretically principled and demonstrate empirical benefits on both synthetic and real datasets.

Invece dei rigidi allineamenti di distribuzione nei tradizionali obiettivi di deep domain adaptation, che falliscono quando la distribuzione delle label target si sposta, proponiamo di ottimizzare un obiettivo rilassato con nuove analisi, nuovi algoritmi e convalida sperimentale.

Questo articolo suggerisce metriche rilassate per domain adaptation che danno nuovi limiti teorici sull'errore di destinazione.


In this paper, we explore \textit{summary-to-article generation}: the task of generating long articles given a short summary, which provides finer-grained content control for the generated text.To prevent sequence-to-sequence (seq2seq) models from degenerating into language models and better controlling the long text to be generated, we propose a hierarchical generation approach which first generates a sketch of intermediate length based on the summary and then completes the article by enriching the generated sketch.To mitigate the discrepancy between the ``oracle'' sketch used during training and the noisy sketch generated during inference, we propose an end-to-end joint training framework based on multi-agent reinforcement learning.For evaluation, we use text summarization corpora by reversing their inputs and outputs, and introduce a novel evaluation method that employs a summarization system to summarize the generated article and test its match with the original input summary.Experiments show that our proposed hierarchical generation approach can generate a coherent and relevant article based on the given summary, yielding significant improvements upon conventional seq2seq models.

esploriamo il task di summary-to-article generation e proponiamo uno schema di generazione gerarchico insieme a un framework di reinforcement learning congiunto end-to-end per addestrare il modello gerarchico.

Per affrontare il problema della degenerazione in summary-to-article generation, questo articolo propone un approccio di generazione gerarchica che genera prima uno schizzo intermedio dell'articolo e poi l'articolo completo.


When training a deep neural network for supervised image classification, one can broadly distinguish between two types of latent features of images that will drive the classification of class Y. Following the notation of Gong et al. (2016), we can divide features broadly into the classes of(i) â€œcoreâ€ or â€œconditionally invariantâ€ features X^ci whose distribution P(X^ci | Y) does not change substantially across domains and(ii) â€œstyleâ€ or â€œorthogonalâ€ features X^orth whose distribution P(X^orth | Y) can change substantially across domains.These latter orthogonal features would generally include features such as position, rotation, image quality or brightness but also more complex ones like hair color or posture for images of persons.We try to guard against future adversarial domain shifts by ideally just using the â€œconditionally invariantâ€ features for classification.In contrast to previous work, we assume that the domain itself is not observed and hence a latent variable.We can hence not directly see the distributional change of features across different domains. We do assume, however, that we can sometimes observe a so-called identifier or ID variable.We might know, for example, that two images show the same person, with ID referring to the identity of the person.In data augmentation, we generate several images from the same original image, with ID referring to the relevant original image.The method requires only a small fraction of images to have an ID variable.We provide a causal framework for the problem by adding the ID variable to the model of Gong et al. (2016).However, we are interested in settings where we cannot observe the domain directly and we treat domain as a latent variable.If two or more samples share the same class and identifier, (Y, ID)=(y,i), then we treat those samples as counterfactuals under different style interventions on the orthogonal or style features.Using this grouping-by-ID approach, we regularize the network to provide near constant output across samples that share the same ID by penalizing with an appropriate graph Laplacian.This is shown to substantially improve performance in settings where domains change in terms of image quality, brightness, color changes, and more complex changes such as changes in movement and posture.We show links to questions of interpretability, fairness and transfer learning.

Proponiamo una regolarizzazione controfattuale per difendersi dagli adversarial domain shift che si verificano attraverso spostamenti nella distribuzione delle "style feature" latenti delle immagini.

L'articolo discute i modi per difendersi dagli adversarial domain shift con la regolarizzazione controfattuale imparando un classificatore che è invariante ai cambiamenti superficiali (o feature di "stile") nelle immagini.

Questo documento mira a una classificazione robusta delle immagini contro gli adversarial domain shift e l'obiettivo è raggiunto evitando di usare le feature di stile mutevoli.


Gradient-based meta-learning algorithms require several steps of gradient descent to adapt to newly incoming tasks.This process becomes more costly as the number  of  samples  increases.  Moreover,  the  gradient  updates  suffer  from  several sources of noise leading to a degraded performance.   In this work,  we propose a meta-learning algorithm equipped with the GradiEnt Component COrrections, aGECCO cell for short, which generates a multiplicative corrective low-rank matrix which (after vectorization) corrects the estimated gradients. GECCO contains a simple decoder-like network with learnable parameters, an attention module and a so-called context input parameter. The context parameter of GECCO is updated to  generate  a  low-rank  corrective  term  for  the  network  gradients.   As  a  result, meta-learning requires only a few of gradient updates to absorb new task (often, a single update is sufficient in the few-shot scenario). While previous approaches address this problem by altering the learning rates, factorising network parameters or directly learning feature corrections from features and/or gradients, GECCO is an off-the-shelf generator-like unit that performs element-wise gradient corrections without the need to â€˜observeâ€™ the features and/or the gradients directly.  We show that our GECCO(i) accelerates learning,(ii) performs robust corrections of the gradients corrupted by a noise, and(iii) leads to notable improvements over existing gradient-based meta-learning algorithms.

Proponiamo un meta-learner per adattarsi rapidamente su più task anche con un solo step in un setting few-shot.

Questo articolo propone un metodo per meta-learning di un modulo di correzione del gradiente in cui il precondizionamento è parametrizzato da una rete neurale, e costruisce un processo di aggiornamento del gradiente in due fasi durante l'adattamento.


Discriminative  question  answering  models  can  overfit  to  superficial  biases  in datasets,  because their loss function saturates when any clue makes the answer likely.  We introduce generative models of the joint distribution of questions and answers, which are trained to explain the whole question, not just to answer it.Our  question  answering  (QA)  model  is  implemented  by  learning  a  prior  over answers,  and  a  conditional  language  model  to  generate  the  question  given  the answerâ€”allowing scalable and interpretable many-hop reasoning as the question is generated word-by-word.  Our model achieves competitive performance with specialised discriminative models on the SQUAD and CLEVR benchmarks, indicating that it is a more general architecture for language understanding and reasoning than previous work.The model greatly improves generalisation both from biased training data and to adversarial testing data, achieving a new state-of-the-art on ADVERSARIAL SQUAD.We will release our code.

I modelli di question answering che modellano la distribuzione congiunta di domande e risposte possono imparare di più dei modelli discriminativi

Questo articolo propone un approccio generativo al QA testuale e visuale, dove viene appresa una distribuzione congiunta sullo spazio delle domande e delle risposte dato il contesto, che cattura relazioni più complesse.

Questo articolo introduce un modello generativo per question answering e propone di modellare p(q,a|c), fattorizzato come p(a|c) * p(q|a,c).

Gli autori propongono un modello generativo di QA, che ottimizza congiuntamente la distribuzione delle domande e delle risposte date da un documento/contesto.


In this paper, we turn our attention to the interworking between the activation functions and the batch normalization, which is a virtually mandatory technique to train deep networks currently.We propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization.Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of standardized VGG and Residual Networks state-of-the-art models.These convolutional neural networks were trained on CIFAR-100 and CIFAR-10, the most commonly used deep learning computer vision datasets.The results showed DReLU speeded up learning in all models and datasets.Besides, statistical significant performance assessments (p<0.05) showed DReLU enhanced the test accuracy presented by ReLU in all scenarios.Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception, in which case it presented the second best performance.Therefore, this work demonstrates that it is possible to increase performance replacing ReLU by an enhanced activation function.

Viene proposta una nuova funzione di attivazione chiamata Displaced Rectifier Linear Unit. Ha dimostrato di migliorare le prestazioni di training e inferenza delle reti neurali convoluzionali con batch normalization.

Il documento confronta e dà indicazioni contro l'uso della batch normalization dopo l'uso di ReLU

Questo articolo propone una funzione di attivazione, chiamata shifted ReLU, per migliorare le prestazioni delle CNN che usano la normalizzazione batch.


Encoding the input scale information explicitly into the representation learned by a convolutional neural network (CNN) is beneficial for many vision tasks especially when dealing with multiscale input signals.We study, in this paper, a scale-equivariant CNN architecture with joint convolutions across the space and the scaling group, which is shown to be both sufficient and necessary to achieve scale-equivariant representations.To reduce the model complexity and computational burden, we decompose the convolutional filters under two pre-fixed separable bases and truncate the expansion to low-frequency components.A further benefit of the truncated filter expansion is the improved deformation robustness of the equivariant representation.Numerical experiments demonstrate that the proposed scale-equivariant neural network with decomposed convolutional filters (ScDCFNet) achieves significantly improved performance in multiscale image classification and better interpretability than regular CNNs at a reduced model size.

Costruiamo reti neurali convoluzionali scale-equivariant nella forma più generale con efficienza computazionale e dimostrata robustezza alla deformazione.

Gli autori propongono un'architettura CNN che è teoricamente equivariante a scalature e traslazioni isotrope aggiungendo una dimensione di scala extra ai tensori di attivazione.


In this paper, we diagnose deep neural networks for 3D point cloud processing to explore the utility of different network architectures.We propose a number of hypotheses on the effects of specific network architectures on the representation capacity of DNNs.In order to prove the hypotheses, we design five metrics to diagnose various types of DNNs from the following perspectives, information discarding, information concentration, rotation robustness, adversarial robustness, and neighborhood inconsistency.We conduct comparative studies based on such metrics to verify the hypotheses, which may shed new lights on the architectural design of neural networks.Experiments demonstrated the effectiveness of our method.The code will be released when this paper is accepted.

Abbiamo diagnosticato deep neural network per l'elaborazione di nuvole di punti 3D per esplorare l'utilità di diverse architetture di rete.

L'articolo studia diverse architetture di reti neurali per l'elaborazione di nuvole di punti 3D e propone metriche per la adversarial robustness, la robustezza rotazionale e la coerenza di vicinato.


In this work we construct flexible joint distributions from low-dimensional conditional semi-implicit distributions.Explicitly defining the structure of the approximation allows to make the variational lower bound tighter, resulting in more accurate inference.

L'utilizzo della struttura delle distribuzioni migliora l'inferenza variazionale semi-implicita


Imitation learning from human-expert demonstrations has been shown to be greatly helpful for challenging reinforcement learning problems with sparse environment rewards.However, it is very difficult to achieve similar success without relying on expert demonstrations.Recent works on self-imitation learning showed that imitating the agent's own past good experience could indirectly drive exploration in some environments, but these methods often lead to sub-optimal and myopic behavior.To address this issue, we argue that exploration in diverse directions by imitating diverse trajectories, instead of focusing on limited good trajectories, is more desirable for the hard-exploration tasks.We propose a new method of learning a trajectory-conditioned policy to imitate diverse trajectories from the agent's own past experiences and show that such self-imitation helps avoid myopic behavior and increases the chance of finding a globally optimal solution for hard-exploration tasks, especially when there are misleading rewards.Our method significantly outperforms existing self-imitation learning and count-based exploration methods on various hard-exploration tasks with local optima.In particular, we report a state-of-the-art score of more than 20,000 points on Montezumas Revenge without using expert demonstrations or resetting to arbitrary states.

Self-imitation learning di traiettorie diverse con policy condizionata dalla traiettoria

Questo articolo affronta task di esplorazione difficili applicando la self-imitation a una diversa selezione di traiettorie dall'esperienza passata, per guidare un'esplorazione più efficiente in problemi a ricompensa sparsa, ottenendo risultati SOTA.


We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost.This is achieved by gating the deep-learning architecture on a fine-grained-level.Individual convolutional maps are turned on/off conditionally on features in the network.To achieve this, we introduce a new residual block architecture that gates convolutional channels in a fine-grained manner.We also introduce a generally applicable tool batch-shaping that matches the marginal aggregate posteriors of features in a neural network to a pre-specified prior distribution.We use this novel technique to force gates to be more conditional on the data.We present results on CIFAR-10 and ImageNet datasets for image classification, and Cityscapes for semantic segmentation.Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy.In particular, on ImageNet, our ResNet50 and ResNet34 gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity.We also show that the resulting networks automatically learn to use more features for difficult examples and fewer features for simple examples.

Un metodo che addestra reti neurali di grande capacità con una precisione significativamente migliorata e un costo computazionale dinamico inferiore

Un metodo per addestrare una rete con grande capacità, di cui solo parti sono usate al momento dell'inferenza dipendente dall'input, usando una selezione condizionale fine-grained e un nuovo metodo di regolarizzazione, "batch shaping".


With a view to bridging the gap between deep learning and symbolic AI, we present a novel end-to-end neural network architecture that learns to form propositional representations with an explicitly relational structure from raw pixel data.In order to evaluate and analyse the architecture, we introduce a family of simple visual relational reasoning tasks of varying complexity.We show that the proposed architecture, when pre-trained on a curriculum of such tasks, learns to generate reusable representations that better facilitate subsequent learning on previously unseen tasks when compared to a number of baseline architectures.The workings of a successfully trained model are visualised to shed some light on how the architecture functions.

Presentiamo un'architettura differenziabile end-to-end che impara a mappare i pixel ai predicati, e la valutiamo su una serie di semplici task di ragionamento relazionale

Un'architettura di rete basata sul modulo di self-attention a più head per imparare una nuova forma di rappresentazioni relazionali, che migliora l'efficienza dei dati e la capacità di generalizzazione nel curriculum learning.


In natural language inference, the semantics of some words do not affect the inference.Such information is considered superficial and brings overfitting.How can we represent and discard such superficial information?In this paper, we use first order logic (FOL) - a classic technique from meaning representation language â€“ to explain what information is superficial for a given sentence pair.Such explanation also suggests two inductive biases according to its properties.We proposed a neural network-based approach that utilizes the two inductive biases.We obtain substantial improvements over extensive experiments.

Usiamo le reti neurali per proiettare le informazioni superficiali per natural language inference, definendo e identificando le informazioni superficiali dalla prospettiva della logica del primo ordine.

Questo articolo cerca di ridurre le informazioni superficiali per natural language inference per prevenire l'overfitting, e introduce una graph neural network per modellare la relazione tra premessa e ipotesi.

Un approccio per trattare natural language inference usando la logica del primo ordine e per infondere i modelli NLI con informazioni logiche per essere più robusti nell'inferenza.