Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL.However, the theoretical understanding of such methods has been rather limited.This paper introduces a novel algorithmic framework for designing and analyzing model-based RL algorithms with theoretical guarantees.We design a meta-algorithm with a theoretical guarantee of monotone improvement to a local maximum of the expected reward.The meta-algorithm iteratively builds a lower bound of the expected reward based on the estimated dynamical model and sample trajectories, and then maximizes the lower bound jointly over the policy and the model.The framework extends the optimism-in-face-of-uncertainty principle to non-linear dynamical models in a way that requires no explicit uncertainty quantification.Instantiating our framework with simplification gives a  variant of model-based RL algorithms Stochastic Lower Bounds Optimization (SLBO).Experiments demonstrate that SLBO achieves the state-of-the-art performance when only 1M or fewer samples are permitted on a range of continuous control benchmark tasks.

Progettiamo algoritmi di reinforcement learning basati su modelli con garanzie teoriche e raggiungiamo risultati allo stato dell'arte sui task di benchmark Mujuco quando sono ammessi un milione o meno di sample.

L'articolo ha proposto un framework per progettare algoritmi RL basati su modelli e OFU che raggiungono prestazioni SOTA sui task MuJoCo.


We study the use of knowledge distillation to compress the U-net architecture.We show that, while standard distillation is not sufficient to reliably train a compressed U-net, introducing other regularization methods, such as batch normalization and class re-weighting, in knowledge distillation significantly improves the training process.This allows us to compress a U-net by over 1000x, i.e., to 0.1% of its original number of parameters, at a negligible decrease in performance.

Presentiamo ulteriori tecniche per utilizzare la knowledge distillation per comprimere U-net di oltre 1000x.

Gli autori hanno introdotto una strategia di distillazione modificata per comprimere un'architettura U-net di oltre 1000x mantenendo una precisione vicina alla U-net originale.


Learning neural networks with gradient descent over a long sequence of tasks is problematic as their fine-tuning to new tasks overwrites the network weights that are important for previous tasks.This leads to a poor performance on old tasks â€“ a phenomenon framed as catastrophic forgetting.  While early approaches use task rehearsal and growing networks that both limit the scalability of the task sequence orthogonal approaches build on regularization.  Based on the Fisher information matrix (FIM) changes to parameters that are relevant to old tasks are penalized, which forces the task to be mapped into the available remaining capacity of the network.This requires to calculate the Hessian around a mode, which makes learning tractable.In this paper, we introduce Hessian-free curvature estimates as an alternative method to actually calculating the Hessian.  In contrast to previous work, we exploit the fact that most regions in the loss surface are flat and hence only calculate a Hessian-vector-product around the surface that is relevant for the current task.Our experiments show that on a variety of well-known task sequences we either significantly outperform or are en par with previous work.

Questo articolo fornisce un approccio per affrontare il catastrophic forgetting attraverso stime di curvatura senza Hessiana

L'articolo propone un metodo approssimato di Laplace per il training delle reti neurali nel setting di continual learning con una bassa complessità spaziale.


There has been recent interest in improving performance of simple models for multiple reasons such as interpretability, robust learning from small data, deployment in memory constrained settings as well as environmental considerations.In this paper, we propose a novel method SRatio that can utilize information from high performing complex models (viz. deep neural networks, boosted trees, random forests) to reweight a training dataset for a potentially low performing simple model such as a decision tree or a shallow network enhancing its performance.Our method also leverages the per sample hardness estimate of the simple model which is not the case with the prior works which primarily consider the complex model's confidences/predictions and is thus conceptually novel.Moreover, we generalize and formalize the concept of attaching probes to intermediate layers of a neural network, which was one of the main ideas in previous work \citep{profweight}, to other commonly used classifiers and incorporate this into our method.The benefit of these contributions is witnessed in the experiments where on 6 UCI datasets and CIFAR-10 we outperform competitors in a majority (16 out of 27) of the cases and tie for best performance in the remaining cases.In fact, in a couple of cases, we even approach the complex model's performance.We also conduct further experiments to validate assertions and intuitively understand why our method works.Theoretically, we motivate our approach by showing that the weighted loss minimized by simple models using our weighting upper bounds the loss of the complex model.

Metodo per migliorare le prestazioni dei modelli semplici dato un modello complesso e accurato.

L'articolo propone un metodo per migliorare le predizioni di un modello a bassa capacità che mostra vantaggi rispetto agli approcci esistenti.


We propose a principled method for kernel learning, which relies on a Fourier-analytic characterization of translation-invariant or rotation-invariant kernels.Our method produces a sequence of feature maps, iteratively refining the SVM margin.We provide rigorous guarantees for optimality and generalization, interpreting our algorithm as online equilibrium-finding dynamics in a certain two-player min-max game.Evaluations on synthetic and real-world datasets demonstrate scalability and consistent improvements over related random features-based methods.

Un algoritmo semplice e pratico per l'apprendimento di un kernel dai training data che massimizza i margini ed è invariante alle traslazioni o è sfericamente simmetrico, utilizzando strumenti dell'analisi di Fourier e di regret minimization.

L'articolo propone di imparare un kernal custom invariante alla traslazione o alla rotazione nella rappresentazione di Fourier per massimizzare il margine di SVM.

Gli autori propongono un interessante algoritmo per apprendere insieme la l1-SVM e il kernel rappresentato da Fourier

Gli autori considerano l'apprendimento diretto di rappresentazioni di Fourier di kernel invarianti rispetto a shift o traslazioni per applicazioni di machine learning con l'allineamento del kernel ai dati come funzione obiettivo da ottimizzare.


We elaborate on using importance sampling for causal reasoning, in particular for counterfactual inference.We show how this can be implemented natively in probabilistic programming.By considering the structure of the counterfactual query, one can significantly optimise the inference process.We also consider design choices to enable further optimisations.We introduce MultiVerse, a probabilistic programming prototype engine for approximate causal reasoning.We provide experimental results and compare with Pyro, an existing probabilistic programming framework with some of causal reasoning tools.

Programmazione probabilistica che supporta nativamente l'inferenza causale e controfattuale


We consider the problem of representing collective behavior of large populations and predicting the evolution of a population distribution over a discrete state space.A discrete time mean field game (MFG) is motivated as an interpretable model founded on game theory for understanding the aggregate effect of individual actions and predicting the temporal evolution of population distributions.We achieve a synthesis of MFG and Markov decision processes (MDP) by showing that a special MFG is reducible to an MDP.This enables us to broaden the scope of mean field game theory and infer MFG models of large real-world systems via deep inverse reinforcement learning.Our method learns both the reward function and forward dynamics of an MFG from real data, and we report the first empirical test of a mean field game model of a real-world social media population.

Inferenza di un mean field game (MFG) del comportamento di grandi popolazioni attraverso una sintesi di MFG e processi decisionali di Markov.

Gli autori trattano l'inferenza nei modelli di comportamento collettivo usando il reinforcement learning inverso per imparare le funzioni di reward degli agenti nel modello.


We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as  offensive basketball gameplay.  When modeling such settings, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables.  Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulable way.We present a hierarchical framework that can effectively learn such sequential generative models.  Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime.In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods.We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts.

Mescoliamo deep generative model con una supervisione debole programmatica per generare traiettorie coordinate multi-agent di qualità significativamente più alta rispetto alle baseline precedenti.

Propone modelli generativi sequenziali multi-agente.

L'articolo propone il training di modelli generativi che producono traiettorie multi-agente usando funzioni euristiche che etichettano le variabili che altrimenti sarebbero latenti nei dati di training


Many automated machine learning methods, such as those for hyperparameter and neural architecture optimization, are computationally expensive because they involve training many different model configurations.In this work, we present a new method that saves computational budget by terminating poor configurations early on in the training.In contrast to existing methods, we consider this task as a ranking and transfer learning problem.We qualitatively show that by optimizing a pairwise ranking loss and leveraging learning curves from other data sets, our model is able to effectively rank learning curves without having to observe many or very long learning curves.We further demonstrate that our method can be used to accelerate a neural architecture search by a factor of up to 100 without a significant performance degradation of the discovered architecture.In further experiments we analyze the quality of ranking, the influence of different model components as well as the predictive behavior of the model.

Imparare a classificare le learning curve al fine di fermare prima i training job non promettenti. La novità è l'uso della loss di pairwise ranking per modellare direttamente la probabilità di migliorare e di trasferire l'apprendimento attraverso i dataset per ridurre i dati di training richiesti.

L'articolo propone un metodo per classificare le curve di apprendimento delle reti neurali che può modellare le curve di apprendimento su diversi dataset, ottenendo una maggiore velocità nei task di classificazione delle immagini.


Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster.Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distributions.This phenomenon, referred to as catastrophic forgetting, is considered a major hurdle to learning with non-stationary data or sequences of new tasks, and prevents networks from continually accumulating knowledge and skills.We examine this issue in the context of reinforcement learning, in a setting where an agent is exposed to tasks in a sequence.Unlike most other work, we do not provide an explicit indication to the model of task boundaries, which is the most general circumstance for a learning agent exposed to continuous experience.While various methods to counteract catastrophic forgetting have recently been proposed, we explore a straightforward, general, and seemingly overlooked solution - that of using experience replay buffers for all past events - with a mixture of on- and off-policy learning, leveraging behavioral cloning.We show that this strategy can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities.When buffer storage is constrained, we confirm that a simple mechanism for randomly discarding data allows a limited size buffer to perform almost as well as an unbounded one.

Mostriamo che, nei setting di continual learning, il catastrophic forgetting può essere evitato applicando la RL off-policy a una miscela di esperienze nuove e di replay, con una loss di behavioural cloning.

Propone una particolare variante di experience replay con behavioural cloning come metodo per continual learning.


We present a method which learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents.Our method is based on a graph-structured variational recurrent neural network, which is trained end-to-end to infer the current state of the (partially observed) world, as well as to forecast future states.We show that our method outperforms various baselines on two sports datasets, one based on real basketball trajectories, and one generated by a soccer game engine.

Presentiamo un metodo che impara a integrare informazioni temporali e informazioni visive ambigue nel contesto di agenti che interagiscono.

Gli autori propongono Graph VRNN che modella l'interazione di più agenti distribuendo una VRNN per ogni agente

Questo articolo presenta un'architettura basata su una graph neural network che è addestrata per localizzare e modellare le interazioni degli agenti in un ambiente direttamente dai pixel e mostra il vantaggio di questo modello per effettuare task tracking e prevedere le posizioni degli agenti.


In this paper we study the problem of learning the weights of a deep convolutional neural network.We consider a network where convolutions are carried out over non-overlapping patches with a single kernel in each layer.We develop an algorithm for simultaneously learning all the kernels from the training data.Our approach dubbed Deep Tensor Decomposition (DeepTD) is based on a rank-1 tensor decomposition.We theoretically investigate DeepTD under a realizable model for the training data where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to planted convolutional kernels.We show that DeepTD is data-efficient and provably works as soon as the sample size exceeds the total number of convolutional weights in the network.Our numerical experiments demonstrate the effectiveness of DeepTD and verify our theoretical findings.

Consideriamo un modello semplificato di deep convolutional neural network. Mostriamo che tutti i layer di questa rete possono essere appresi approssimativamente con una corretta applicazione della decomposizione tensoriale.

Fornisce garanzie teoriche per l'apprendimento di deep convolutional neural network usando la decomposizione tensoriale rank-one.

Questo articolo propone un metodo di apprendimento per un caso ristretto di deep convolutional neural netowrk, dove i layer sono limitati al caso non sovrapposto e hanno solo un canale di uscita per layer

Analizza il problema dell'apprendimento di una classe molto speciale di CNN: ogni layer consiste di un singolo filtro, applicato a patch non sovrapposte dell'input.


Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively.Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations.The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations.We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10.Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

Le reti neurali feedforward che possono avere pesi sottoposti a pruning dopo il training potrebbero avere gli stessi pesi pruned prima del training

Mostra che esistono sottoreti sparse che possono essere addestrate da zero con buone prestazioni di generalizzazione e propone una NNs non pruned e inizializzata in modo casuale che contiene sottoreti che possono essere addestrate da zero con un'accuratezza di generalizzazione simile.

L'articolo esamina l'ipotesi che le reti neurali inizializzate in modo casuale contengano sottoreti che convergono altrettanto velocemente o più velocemente e possono raggiungere la stessa o una migliore accuratezza di classificazione


We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime.Recent work of \citep{Gale2019, Liu2018} has shown that sparse ResNet-50 architectures trained on ImageNet-2012 dataset converge to solutions that are significantly worse than those found by pruning.We show that, despite the failure of optimizers, there is a linear path with a monotonically decreasing objective from the initialization to the ``good'' solution.Additionally, our attempts to find a decreasing objective path from ``bad'' solutions to the ``good'' ones in the sparse subspace fail.However, if we allow the path to traverse the dense subspace, then we consistently find a path between two solutions.These findings suggest traversing extra dimensions may be needed to escape stationary points found in the sparse subspace.

In questo articolo evidenziamo la difficoltà di addestrare reti neurali sparse facendo esperimenti di interpolazione nell'energy landscape.


Neural network training depends on the structure of the underlying loss landscape, i.e. local minima, saddle points, flat plateaus, and loss barriers.In relation to the structure of the landscape, we study the permutation symmetry of neurons in each layer of a deep neural network, which gives rise not only to multiple equivalent global minima of the loss function but also to critical points in between partner minima.In a network of $d-1$ hidden layers with $n_k$ neurons in layers $k = 1, \ldots, d$, we construct continuous paths between equivalent global minima that lead through a `permutation point' where the input and output weight vectors of two neurons in the same hidden layer $k$ collide and interchange.We show that such permutation points are critical points which lie inside high-dimensional subspaces of equal loss, contributing to the global flatness of the landscape.We also find that a permutation point for the exchange of neurons $i$ and $j$ transits into a flat high-dimensional plateau that enables all $n_k!$permutations of neurons in a given layer $k$ at the same loss value. Moreover, we introduce higher-order permutation points by exploiting the hierarchical structure in the loss landscapes of neural networks, and find that the number of $K$-th order permutation points is much larger than the (already huge) number of equivalent global minima -- at least by a polynomial factor of order $K$. In twotasks, we demonstrate numerically with our path finding method that continuous paths between partner minima exist: first, in a toy network with a single hidden layer on a function approximation task and, second, in a multilayer network on the MNIST task. Our geometricapproach yields a lower bound on the number of critical points generated by weight-space symmetries and provides a simple intuitive link between previous theoretical results and numerical observations.

La simmetria dello spazio dei pesi nei landscape delle reti neurali dà luogo ad un gran numero di punti di sella e sottospazi piatti ad alta dimensionalità.

L'articolo ha presentato un metodo low-loss per studiare la funzione di loss rispetto ai parametri in una rete neurale dal punto di vista della simmetria nel weight space.


The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated.We derive, using mean field theory, a set of scalar equations describing how input signals propagate through surrogate networks.The equations reveal that depending on the choice of surrogate model, the networks may or may not exhibit an order to chaos transition, and the presence of depth scales that limit the maximum trainable depth.Specifically, in solving the equations for edge of chaos conditions, we show that surrogates derived using the Gaussian local reparameterisation trick have no critical initialisation, whereas a deterministic surrogates based on analytic Gaussian integration do.The theory is applied to a range of binary neuron and weight design choices, such as different neuron noise models, allowing the categorisation of algorithms in terms of their behaviour at initialisation.Moreover, we predict theoretically and confirm numerically, that common weight initialization schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance.This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to close to $\pm 1$ for deeper networks to be trainable.

teoria della propagazione del segnale applicata a surrogati continui di reti binarie; inizializzazione contro intuitiva; reparameterisation trick non utile

Gli autori studiano le dinamiche di allenamento delle reti neurali binarie quando si usano surrogati continui, studiano quali proprietà dovrebbero avere le reti all'inizializzazione per effettuare il training al meglio e forniscono consigli concreti sui pesi stocastici all'inizializzazione.

Un'esplorazione approfondita delle reti binarie stocastiche, dei surrogati continui e delle loro dinamiche di training, con approfondimenti su come inizializzare i pesi per ottenere le migliori prestazioni.


Semantic dependency parsing, which aims to find rich bi-lexical relationships, allows words to have multiple dependency heads, resulting in graph-structured representations.We propose an approach to semi-supervised learning of semantic dependency parsers based on the CRF autoencoder framework.Our encoder is a discriminative neural semantic dependency parser that predicts the latent parse graph of the input sentence.Our decoder is a generative neural model that reconstructs the input sentence conditioned on the latent parse graph.Our model is arc-factored and therefore parsing and learning are both tractable.Experiments show our model achieves significant and consistent improvement over the supervised baseline.

Proponiamo un approccio all'apprendimento semi-supervised di semantic dependency parser basato sul framework CRF autoencoder.

Questo articolo si concentra sul parsing semi-supervised delle dipendenze semantiche usando il CRF-autoencoder per addestrare il modello in uno stile semi-supervised, indicando l'efficacia su task di dati annotati con poche risorse.


For sequence models with large word-level vocabularies, a majority of network parameters lie in the input and output layers.In this work, we describe a new method, DeFINE, for learning deep word-level representations efficiently.Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods.DeFINE can be incorporated easily in new or existing sequence models.Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity.On WikiText-103, DeFINE reduces total parameters of Transformer-XL by half with minimal impact on performance.On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters,  achieving comparable performance to state-of-the-art methods with fewer parameters.For machine translation, DeFINE improves a Transformer model by 2% while simultaneously reducing total parameters by 26%

DeFINE utilizza una rete profonda, gerarchica e sparsa con nuove skip connection per imparare in modo efficiente i word embedding.

Questo articolo descrive un nuovo metodo per l'apprendimento di rappresentazioni deep a livello di parola in modo efficiente utilizzando una struttura gerarchica con skip-connection per l'uso di layer di input e output a bassa dimensione.


In this paper, we present a reproduction of the paper of Bertinetto et al. [2019] "Meta-learning with differentiable closed-form solvers" as part of the ICLR 2019 Reproducibility Challenge.In successfully reproducing the most crucial part of the paper, we reach a performance that is comparable with or superior to the original paper on two benchmarks for several settings.We evaluate new baseline results, using a new dataset presented in the paper.Yet, we also provide multiple remarks and recommendations about reproducibility and comparability.  After we brought our reproducibility work to the authorsâ€™ attention, they have updated the original paper on which this work is based and released code as well.Our contributions mainly consist in reproducing the most important results of their original paper, in giving insight in the reproducibility and in providing a first open-source implementation.

Riproduciamo con successo e diamo osservazioni sul confronto con le baseline di un approccio di meta learning per la classificazione few-shot che funziona tramite la backpropagation attraverso la soluzione di un risolutore in forma chiusa.


Network pruning has emerged as a powerful technique for reducing the size of deep neural networks.Pruning uncovers high-performance subnetworks by taking a trained dense network and gradually removing unimportant connections.Recently, alternative techniques have emerged for training sparse networks directly without having to train a large dense model beforehand, thereby achieving small memory footprints during both training and inference.These techniques are based on dynamic reallocation of non-zero parameters during training.Thus, they are in effect executing a training-time search for the optimal subnetwork.We investigate a most recent one of these techniques and conduct additional experiments to elucidate its behavior in training sparse deep convolutional networks.Dynamic parameter reallocation converges early during training to a highly trainable subnetwork.We show that neither the structure, nor the initialization of the discovered high-performance subnetwork is sufficient to explain its good performance.Rather, it is the dynamics of parameter reallocation that are responsible for successful learning.Dynamic parameter reallocation thus improves the trainability of deep convolutional networks, playing a similar role as overparameterization, without incurring the memory and computational cost of the latter.

La riallocazione dinamica dei parametri permette il successo del training diretto di reti sparse compatte, e gioca un ruolo indispensabile anche quando conosciamo la rete sparsa ottimale a-priori


n this paper we present a thrust in three directions of visual development us- ing supervised and semi-supervised techniques.The first is an implementation of semi-supervised object detection and recognition using the principles of Soft At- tention and Generative Adversarial Networks (GANs).The second and the third are supervised networks that learn basic concepts of spatial locality and quantity respectively using Convolutional Neural Networks (CNNs).The three thrusts to- gether are based on the approach of Experiential Robot Learning, introduced in previous publication.While the results are unripe for implementation, we believe they constitute a stepping stone towards autonomous development of robotic vi- sual modules.

3 spinte che servono come trampolini di lancio per l'experiential learning dei robot del modulo di visione

Indaga sulle prestazioni dei classificatori di immagini e dei rilevatori di oggetti esistenti.


Characterization of the representations learned in intermediate layers of deep networks can provide valuable insight into the nature of a task and can guide the development of well-tailored learning strategies.Here we study convolutional neural network-based acoustic models in the context of automatic speech recognition.Adapting a method proposed by Yosinski et al. [2014], we measure the transferability of each layer between German and English to assess the their language-specifity.We observe three distinct regions of transferability: (1) the first two layers are entirely transferable between languages, (2) layers 2â€“8 are also highly transferable but we find evidence of some language specificity, (3) the subsequent fully connected layers are more language specific but can be successfully finetuned to the target language.To further probe the effect of weight freezing, we performed follow-up experiments using freeze-training [Raghu et al., 2017].Our results are consistent with the observation that CCNs converge 'bottom up' during training and demonstrate the benefit of freeze training, especially for transfer learning.

Tutti i nostri modelli acustici basati su CNN, tranne per i primi due layer, hanno dimostrato un certo grado di specificità per la lingua, ma il freeze training ha permesso un transfer di successo tra le lingue.

L'articolo misura la trasferibilità delle feature per ogni layer nei modelli acustici basati su CNN attraverso le lingue, concludendo che gli AM addestrati con la tecnica "freeze training" hanno superato gli altri modelli di transfer learning.


Policy gradients methods often achieve better performance when the change in policy is limited to a small Kullback-Leibler divergence.We derive policy gradients where the change in policy is limited to a small Wasserstein distance (or trust region).This is done in the discrete and continuous multi-armed bandit settings with entropy regularisation.We show that in the small steps limit with respect to the Wasserstein distance $W_2$, policy dynamics are governed by the heat equation, following the Jordan-Kinderlehrer-Otto result.This means that policies undergo diffusion and advection, concentrating near actions with high reward.This helps elucidate the nature of convergence in the probability matching setup, and provides justification for empirical practices such as Gaussian policy priors and additive gradient noise.

Si collegano i gradienti di policy entropica della regione di Wasserstein-trust e l'equazione del calore.

L'articolo esplora le connessioni tra reinforcement learning e la teoria del trasporto ottimale quadratico

Gli autori hanno studiato il gradiente della policy con il cambiamento delle policy limitato da una regione di fiducia della distanza di Wasserstein nel setting del multi-armed bandit, mostrando che nel limite degli small step, la dinamica della policy è governata dall'equazione del calore (equazione di Fokker-Planck).


The softmax function is widely used to train deep neural networks for multi-class classification.Despite its outstanding performance in classification tasks, the features derived from the supervision of softmax are usually sub-optimal in some scenarios where Euclidean distances apply in feature spaces.To address this issue, we propose a new loss, dubbed the isotropic loss, in the sense that the overall distribution of data points is regularized to approach the isotropic normal one.Combined with the vanilla softmax, we formalize a novel criterion called the isotropic softmax, or isomax for short, for supervised learning of deep neural networks.By virtue of the isomax, the intra-class features are penalized by the isotropic loss while inter-class distances are well kept by the original softmax loss.Moreover, the isomax loss does not require any additional modifications to the network, mini-batches or the training process.Extensive experiments on classification and clustering are performed to demonstrate the superiority and robustness of the isomax loss.

La capacità discriminativa di softmax per l'apprendimento di vettori di feature di oggetti è efficacemente migliorata dalla normalizzazione isotropica sulla distribuzione globale dei punti dati.


A fundamental question in reinforcement learning is whether model-free algorithms are sample efficient.Recently,  Jin et al. (2018) proposed a Q-learning algorithm with UCB exploration policy, and proved it has nearly optimal regret bound for finite-horizon episodic MDP.In this paper, we adapt Q-learning with UCB-exploration bonus to infinite-horizon MDP with discounted rewards \emph{without} accessing a generative model.We show that the \textit{sample complexity of exploration} of our algorithm is bounded by $\tilde{O}({\frac{SA}{\epsilon^2(1-\gamma)^7}})$.This improves the previously best known result of $\tilde{O}({\frac{SA}{\epsilon^4(1-\gamma)^8}})$ in this setting achieved by delayed Q-learning (Strehlet al., 2006),, and matches the lower bound in terms of $\epsilon$ as well as $S$ and $A$ up to logarithmic factors.

Adattiamo Q-learning con bonus UCB-exploration a MDP infinite-horizon con reward scontate senza accedere a un modello generativo, e migliora il miglior risultato precedentemente conosciuto.

Questo articolo ha considerato un algoritmo di Q learning con una policy di esplorazione UCB per MDP infinite-horizon.


Backpropagation is driving today's artificial neural networks (ANNs).However, despite extensive research, it remains unclear if the brain implements this algorithm.Among neuroscientists, reinforcement learning (RL) algorithms are often seen as a realistic alternative: neurons can randomly introduce change, and use unspecific feedback signals to observe their effect on the cost and thus approximate their gradient.However, the convergence rate of such learning scales poorly with the number of involved neurons.Here we propose a hybrid learning approach.Each neuron uses an RL-type strategy to learn how to approximate the gradients that backpropagation would provide.We provide proof that our approach converges to the true gradient for certain classes of networks.In both feedforward and convolutional networks, we empirically show that our approach learns to approximate the gradient, and can match the performance of gradient-based learning.Learning feedback weights provides a biologically plausible mechanism of achieving good performance, without the need for precise, pre-specified learning rules.

Le perturbazioni possono essere usate per addestrare i pesi di feedback per imparare in reti neurali fully connected e convoluzionali

Questo articolo propone un metodo che affronta il problema del "trasporto dei pesi" stimando i pesi per il backward pass usando uno stimatore basato sul rumore


This paper proposes and demonstrates a surprising pattern in the training of neural networks: there is a one to one relation between the values of any pair of losses (such as cross entropy, mean squared error, 0/1 error etc.) evaluated for a model arising at (any point of) a training run.This pattern is universal in the sense that this one to one relationship is identical across architectures (such as VGG, Resnet, Densenet etc.), algorithms (SGD and SGD with momentum) and training loss functions (cross entropy and mean squared error).

Identifichiamo alcuni modelli universali nel comportamento di diverse loss surrogate (CE, MSE, loss 0-1) durante il training delle reti neurali e presentiamo prove empiriche di supporto.