A central challenge in reinforcement learning is discovering effective policies for tasks where rewards are sparsely distributed. We postulate that in the absence of useful reward signals, an effective exploration strategy should seek out {\it decision states}. These states lie at critical junctions in the state space from where the agent can transition to new, potentially unexplored regions. We propose to learn about decision states from prior experience. By training a goal-conditioned model with an information bottleneck, we can identify decision states by examining where the model accesses the goal state through the bottleneck. We find that this simple mechanism effectively identifies decision states, even in partially observed settings. In effect, the model learns the sensory cues that correlate with potential subgoals. In new environments, this model can then identify novel subgoals for further exploration, guiding the agent through a sequence of potential decision  states and through new regions of the state space.

Il training degli agenti con goal-policy information bottleneck promuove il transfer e produce un potente bonus di esplorazione

Propone la regolarizzazione delle loss di RL standard con l'informazione mutua condizionata negativa per la ricerca della policy in un ambiente di RL multi-goal.

Questo articolo propone il concetto di stato di decisione e propone una regolarizzazione di divergenza KL per apprendere la struttura dei task e usare questa informazione per incoraggiare la policy a visitare gli stati di decisione.

L'articolo propone un metodo per regolarizzare le policy goal-conditioned con un termine di informazione mutua.


Many applications in machine learning require optimizing a function whose true gradient is unknown, but where surrogate gradient information (directions that may be correlated with, but not necessarily identical to, the true gradient) is available instead. This arises when an approximate gradient is easier to compute than the full gradient (e.g. in meta-learning or unrolled optimization), or when a true gradient is intractable and is replaced with a surrogate (e.g. in certain reinforcement learning applications or training networks with discrete variables). We propose Guided Evolutionary Strategies, a method for optimally using surrogate gradient directions along with random search. We define a search distribution for evolutionary strategies that is elongated along a subspace spanned by the surrogate gradients. This allows us to estimate a descent direction which can then be passed to a first-order optimizer. We analytically and numerically characterize the tradeoffs that result from tuning how strongly the search distribution is stretched along the guiding subspace, and use this to derive a setting of the hyperparameters that works well across problems. Finally, we apply our method to example problems including truncated unrolled optimization and training neural networks with discrete variables, demonstrating improvement over both standard evolutionary strategies and first-order methods (that directly follow the surrogate gradient). We provide a demo of Guided ES at: redacted URL

Proponiamo un metodo di ottimizzazione per quando sono disponibili solo gradienti biased - definiamo un nuovo stimatore di gradiente per questo scenario, deriviamo il bias e la varianza di questo stimatore e lo applichiamo a problemi di esempio.

Gli autori propongono un approccio che combina la ricerca casuale con l'informazione del gradiente surrogato e fanno una discussione sul trade-off variance-bias così come sull'ottimizzazione degli iperparametri.

L'articolo propone un metodo per migliorare la ricerca casuale costruendo un sottospazio dei precedenti k gradienti surrogati.

Questo articolo cerca di accelerare l'evoluzione del tipo OpenAI introducendo una distribuzione non isotropica con una matrice di covarianza nella forma I + UU^t e informazioni esterne come un gradiente surrogato per determinare U


Point clouds are an important type of geometric data and have widespread use in computer graphics and vision. However, learning representations for point clouds is particularly challenging due to their nature as being an unordered collection of points irregularly distributed in 3D space. Graph convolution, a generalization of the convolution operation for data defined over graphs, has been recently shown to be very successful at extracting localized features from point clouds in supervised or semi-supervised tasks such as classification or segmentation. This paper studies the unsupervised problem of a generative model exploiting graph convolution. We focus on the generator of a GAN and define methods for graph convolution when the graph is not known in advance as it is the very output of the generator. The proposed architecture learns to generate localized features that approximate graph embeddings of the output geometry. We also study the problem of defining an upsampling layer in the graph-convolutional generator, such that it learns to exploit a self-similarity prior on the data distribution to sample more effectively.

Una GAN che usa operazioni di convoluzione dei grafi con grafi calcolati dinamicamente da feature nascoste

L'articolo propone una versione delle GAN specificamente progettata per la generazione di nuvole di punti con il contributo centrale del lavoro costituito dall'operazione di upsampling.

Questo articolo propone delle GAN grafo-convoluzionali per nuvole di punti 3D irregolari che imparano il dominio e le feature allo stesso tempo.


Memorization in over-parameterized neural networks can severely hurt generalization in the presence of mislabeled examples. However, mislabeled examples are to hard avoid in extremely large datasets. We address this problem using the implicit regularization effect of stochastic gradient descent with large learning rates, which we find to be able to separate clean and mislabeled examples with remarkable success using loss statistics. We leverage this to identify and on-the-fly discard mislabeled examples using a threshold on their losses. This leads to On-the-fly Data Denoising (ODD), a simple yet effective algorithm that is robust to mislabeled examples, while introducing almost zero computational overhead. Empirical results demonstrate the effectiveness of ODD on several datasets containing artificial and real-world mislabeled examples.

Introduciamo un algoritmo veloce e facile da implementare che è robusto al rumore del dataset.

Il paper mira a rimuovere i potenziali esempi con rumore sulle label scartando quelli con grandi loss nella procedura di training.


Binarized Neural Networks (BNNs) have recently attracted significant interest due to their computational efficiency. Concurrently, it has been shown that neural networks may be overly sensitive to ``attacks" -- tiny adversarial changes in the input -- which may be detrimental to their use in safety-critical domains. Designing attack algorithms that effectively fool trained models is a key step towards learning robust neural networks.  The discrete, non-differentiable nature of BNNs, which distinguishes them from their full-precision counterparts, poses a challenge to gradient-based attacks. In this work, we study the problem of attacking a BNN through the lens of combinatorial and integer optimization. We propose a Mixed Integer Linear Programming (MILP) formulation of the problem. While exact and flexible, the MILP quickly becomes intractable as the network and perturbation space grow. To address this issue, we propose IProp, a decomposition-based algorithm that solves a sequence of much smaller MILP problems. Experimentally, we evaluate both proposed methods against the standard gradient-based attack (PGD) on MNIST and Fashion-MNIST, and show that IProp performs favorably compared to PGD, while scaling beyond the limits of the MILP.

Gli attacchi basati sul gradiente sulle reti neurali binarizzate non sono efficaci a causa della non differenziabilità di tali reti; il nostro algoritmo IPROP risolve questo problema utilizzando l'ottimizzazione intera

Propone un nuovo algoritmo in stile propagazione degli obiettivi per generare forti adversarial attack sulle reti neurali binarizzate.

Questo articolo ha proposto un nuovo algoritmo di attacco basato su MILP su reti neurali binarie.

Questo articolo presenta un algoritmo per trovare adversarial attack alle reti neurali binarie che trova iterativamente le rappresentazioni desiderate layer per layer dall'alto all'input ed è più efficiente che risolvere il risolutore completo di programmazione lineare intera mista.


Highly regularized LSTMs achieve impressive results on several benchmark datasets in language modeling. We propose a new regularization method based on decoding the last token in the context using the predicted distribution of the next token. This biases the model towards retaining more contextual information, in turn improving its ability to predict the next token. With negligible overhead in the number of parameters and training time, our Past Decode Regularization (PDR) method achieves a word level perplexity of 55.6 on the Penn Treebank and 63.5 on the WikiText-2 datasets using a single softmax. We also show gains by using PDR in combination with a mixture-of-softmaxes, achieving a word level perplexity of 53.8 and 60.5 on these datasets. In addition, our method achieves 1.169 bits-per-character on the Penn Treebank Character dataset for character level language modeling. These results constitute a new state-of-the-art in their respective settings.

La decodifica dell'ultimo token nel contesto usando la distribuzione del token successivo previsto agisce come un regolarizzatore e migliora il language modeling.

Gli autori introducono l'idea del past decoding allo scopo di regolarizzare la perplexity su Penn Treebank

Propone un termine di loss aggiuntivo da usare quando si addestra un LM basato su LSTM e mostra che aggiungendo questo termine di loss si può raggiungere la perplexity SOTA su una serie di benchmark LM.

Suggerisce una nuova tecnica di regolarizzazione che può essere aggiunta in cima a quelle usate in AWD-LSTM di Merity et al. (2017) con poco overhead.


The assumption that data samples are independently identically distributed is the backbone of many learning algorithms. Nevertheless, datasets often exhibit rich structures in practice, and we argue that there exist some unknown orders within the data instances. Aiming to find such orders, we introduce a novel Generative Markov Network (GMN) which we use to extract the order of data instances automatically. Specifically, we assume that the instances are sampled from a Markov chain. Our goal is to learn the transitional operator of the chain as well as the generation order by maximizing the generation probability under all possible data permutations. One of our key ideas is to use neural networks as a soft lookup table for approximating the possibly huge, but discrete transition matrix. This strategy allows us to amortize the space complexity with a single model and make the transitional operator generalizable to unseen instances. To ensure the learned Markov chain is ergodic, we propose a greedy batch-wise permutation scheme that allows fast training.   Empirically, we evaluate the learned Markov chain by showing that GMNs are able to discover orders among data instances and also perform comparably well to state-of-the-art methods on the one-shot recognition benchmark task.

Propone di osservare gli ordini impliciti nei dataset nell'ottica di un modello generativo.

Gli autori affrontano il problema dell'ordinamento implicito in un dataset e la sfida di recuperarlo e propongono di imparare un modello senza distanza metrica che assume una catena di Markov come meccanismo generativo dei dati

L'articolo propone Generative Markov Network - un approccio basato sul deep learning per modellare le sequenze e scoprire l'ordine nei dataset.

Propone di imparare l'ordine di un sample di dati non ordinato imparando una catena di Markov.


We present a Neural Program Search, an algorithm to generate programs from natural language description and a small number of input / output examples. The algorithm combines methods from Deep Learning and Program Synthesis fields by designing rich domain-specific language (DSL) and defining efficient search algorithm guided by a Seq2Tree model on it. To evaluate the quality of the approach we also present a semi-synthetic dataset of descriptions with test examples and corresponding programs. We show that our algorithm significantly outperforms sequence-to-sequence model with attention baseline.

Program synthesis dalla descrizione in linguaggio naturale e dagli esempi di input/output tramite la ricerca Tree-Beam sul modello Seq2Tree

Presenta un modello seq2Tree per tradurre una dichiarazione di un problema in linguaggio naturale al corrispondente programma funzionale in DSL, che ha mostrato un miglioramento rispetto all'approccio di base seq2seq.

Questo articolo affronta il problema di fare la sintesi di un programma quando viene data una descrizione del problema e un piccolo numero di esempi di input-output.

L'articolo introduce una tecnica per la sintesi di programmi che coinvolge una grammatica ristretta di problemi che è esplorata tramite beam search usando una rete encoder-decoder con attention.


Generative adversarial training can be generally understood as minimizing certain moment matching loss defined by a set of discriminator functions, typically  neural networks. The discriminator set should be large enough to be able to uniquely identify the true distribution (discriminative), and also be small enough to go beyond memorizing samples (generalizable). In this paper, we show that a discriminator set is guaranteed to be discriminative whenever its linear span is dense in the set of bounded continuous functions. This is a very mild condition satisfied even by neural networks with a single neuron. Further, we develop generalization bounds between the learned distribution and true distribution under different evaluation metrics. When evaluated with neural distance, our bounds show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set. When evaluated with KL divergence, our bound provides an explanation on the counter-intuitive behaviors of testing likelihood in GAN training. Our analysis sheds lights on understanding the practical performance of GANs.

Questo articolo studia le proprietà di discriminazione e generalizzazione delle GAN quando l'insieme discriminatore è una classe di funzioni ristretta come le reti neurali.

Bilancia le capacità delle classi di generatori e discriminatori nelle GAN garantendo che le IPM indotte siano metriche e non pseudo metriche

Questo articolo fornisce un'analisi matematica del ruolo della dimensione dell'insieme adversarial/discriminatore nelle GAN


Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization. Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization. We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers. Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation.

Tutto ciò di cui avete bisogno per deep residual network è una buona inizializzazione; i layer di normalizzazione non sono necessari.

Viene presentato un metodo per l'inizializzazione e la normalizzazione delle deep residual network. Questo si basa sulle osservazioni dell'esplosione forward e backward in tali reti. La performance del metodo è alla pari con i migliori risultati ottenuti da altre reti con una normalizzazione più esplicita.

Gli autori propongono un nuovo modo di inizializzare le residual network, che è motivato dalla necessità di evitare gli exploding/vanishing gradient.

Propone un nuovo metodo di inizializzazione usato per addestrare ResNet molto profonde senza usare batch-norm.


Designing a metric manually for unsupervised sequence generation tasks, such as text generation, is essentially difficult. In a such situation, learning a metric of a sequence from data is one possible solution. The previous study, SeqGAN, proposed the framework for unsupervised sequence generation, in which a metric is learned from data, and a generator is optimized with regard to the learned metric with policy gradient, inspired by generative adversarial nets (GANs) and reinforcement learning. In this paper, we make two proposals to learn better metric than SeqGAN's: partial reward function and expert-based reward function training. The partial reward function is a reward function for a partial sequence of a certain length. SeqGAN employs a reward function for completed sequence only. By combining long-scale and short-scale partial reward functions, we expect a learned metric to be able to evaluate a partial correctness as well as a coherence of a sequence, as a whole. In expert-based reward function training, a reward function is trained to discriminate between an expert (or true) sequence and a fake sequence that is produced by editing an expert sequence. Expert-based reward function training is not a kind of GAN frameworks. This makes the optimization of the generator easier. We examine the effect of the partial reward function and expert-based reward function training on synthetic data and real text data, and show improvements over SeqGAN and the model trained with MLE. Specifically, whereas SeqGAN gains 0.42 improvement of NLL over MLE on synthetic data, our best model gains 3.02 improvement, and whereas SeqGAN gains 0.029 improvement of BLEU over MLE, our best model gains 0.250 improvement.

Questo articolo mira a imparare una metrica migliore per l'apprendimento unsupervised, come la generazione di testo, e mostra un miglioramento significativo rispetto a SeqGAN.

Descrive un approccio alla generazione di sequenze temporali attraverso l'apprendimento di valori stato-azione, dove lo stato è la sequenza generata finora, e l'azione è la scelta del valore successivo.

Questo articolo considera il problema di migliorare la generazione di sequenze imparando metriche migliori, in particolare il problema del bias di esposizione


One of the most successful techniques in generative models has been decomposing a complicated generation task into a series of simpler generation tasks.   For example, generating an image at a low resolution and then learning to refine that into a high resolution image often improves results substantially.   Here we explore a novel strategy for decomposing generation for complicated objects in which we first generate latent variables which describe a subset of the observed variables, and then map from these latent variables to the observed space.   We show that this allows us to achieve decoupled training of complicated generative models and present both theoretical and experimental results supporting the benefit of such an approach.

Decompone il task di apprendere un modello generativo nell'apprendimento di fattori latenti disgiunti per sottoinsiemi di dati per poi apprendere la distribuzione congiunta su quei fattori latenti.

Fattori Localmente Disentangled per un modello generativo gerarchico a variabili latenti, che può essere visto come una variante gerarchica di Adversarially Learned Inference

L'articolo indaga il potenziale dei modelli gerarchici a variabili latenti per generare immagini e sequenze di immagini e propone di addestrare diversi modelli ALI impilati uno sull'altro per creare una rappresentazione gerarchica dei dati.

L'articolo mira ad apprendere le gerarchie per il training delle GAN direttamente in un programma di ottimizzazione gerarchica invece di essere progettate da un umano


Visual grounding of language is an active research field aiming at enriching text-based representations with visual information. In this paper, we propose a new way to leverage visual knowledge for sentence representations. Our approach transfers the structure of a visual representation space to the textual space by using two complementary sources of information: (1) the cluster information: the implicit knowledge that two sentences associated with the same visual content describe the same underlying reality and (2) the perceptual information contained within the structure of the visual space. We use a joint approach to encourage beneficial interactions during training between textual, perceptual, and cluster information. We demonstrate the quality of the learned representations on semantic relatedness, classification, and cross-modal retrieval tasks.

Proponiamo un modello congiunto per incorporare la conoscenza visiva nelle rappresentazioni delle frasi

L'articolo propone un metodo per utilizzare i video abbinati a caption per migliorare i sentence embedding

Questo paper propone un modello per l'apprendimento di rappresentazioni di frasi che sono fondate e basate su dati video associati.

Propone un metodo per migliorare i sentence embedding basati sul testo attraverso un framework multimodale congiunto.