This paper introduces the concept of continuous convolution to neural networks and deep learning applications in general. Rather than directly using discretized information, input data is first projected into a high-dimensional Reproducing Kernel Hilbert Space (RKHS), where it can be modeled as a continuous function using a series of kernel bases. We then proceed to derive a closed-form solution to the continuous convolution operation between two arbitrary functions operating in different RKHS. Within this framework, convolutional filters also take the form of continuous functions, and the training procedure involves learning the RKHS to which each of these filters is projected, alongside their weight parameters. This results in much more expressive filters, that do not require spatial discretization and benefit from properties such as adaptive support and non-stationarity. Experiments on image classification are performed, using classical datasets, with results indicating that the proposed continuous convolutional neural network is able to achieve competitive accuracy rates with far fewer parameters and a faster convergence rate.

Questo articolo propone un nuovo layer di convoluzione che opera in uno spazio di Hilbert continuo a kernel riproducente.

Proiettare gli esempi in uno spazio RK Hilbert ed eseguire la convoluzione e il filtraggio in quello spazio.

Questo articolo formula una variante delle reti neurali convoluzionali che modella sia le attivazioni che i filtri come funzioni continue composte da basi di kernel


Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes. Some recent studies suggest a more important role of image textures. We here put these conflicting hypotheses to a quantitative test by evaluating CNNs and human observers on images with a texture-shape cue conflict. We show that ImageNet-trained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence and reveals fundamentally different classification strategies. We then demonstrate that the same standard architecture (ResNet-50) that learns a texture-based representation on ImageNet is able to learn a shape-based representation instead when trained on 'Stylized-ImageNet', a stylized version of ImageNet. This provides a much better fit for human behavioural performance in our well-controlled psychophysical lab setting (nine experiments totalling 48,560 psychophysical trials across 97 observers) and comes with a number of unexpected emergent benefits such as improved object detection performance and previously unseen robustness towards a wide range of image distortions, highlighting advantages of a shape-based representation.

Le CNN addestrate in ImageNet sono orientate verso la texture dell'oggetto (invece che verso la forma come gli umani). Il superamento di questa grande differenza tra la visione umana e la visione della macchina produce prestazioni di rilevamento migliorate e una robustezza mai vista prima alle distorsioni dell'immagine.

Usare lo stylizaton delle immagini per aumentare i dati di training per le CNN addestrate in ImageNet per far apparire le reti risultanti più allineate ai giudizi umani

Questo articolo studia le CNN come AlexNet, VGG, GoogleNet e ResNet50, mostra che questi modelli sono biased verso le texture quando addestrati su ImageNet, e propone un nuovo dataset ImageNet.


In this work, we exploited different strategies to provide prior knowledge to commonly used generative modeling approaches aiming to obtain speaker-dependent low dimensional representations from short-duration segments of speech data, making use of available information of speaker identities. Namely, convolutional variational autoencoders are employed, and statistics of its learned posterior distribution are used as low dimensional representations of fixed length short-duration utterances. In order to enforce speaker dependency in the latent layer, we introduced a variation of the commonly used prior within the variational autoencoders framework, i.e. the model is simultaneously trained for reconstruction of inputs along with a discriminative task performed on top of latent layers outputs. The effectiveness of both triplet loss minimization and speaker recognition are evaluated as implicit priors on the challenging cross-language NIST SRE 2016 setting and compared against fully supervised and unsupervised baselines.

Valutiamo l'efficacia di avere task discriminatori ausiliari eseguiti sopra le statistiche della distribuzione posteriore appresa dagli autoencoder variazionali per imporre la speaker dependency.

Proporre un modello di autoencoder per imparare una rappresentazione per la speaker verification utilizzando finestre di analisi di breve durata.

Una versione modificata del modello di autoencoder variazionale che affronta il problema dello speaker recognition nel contesto di segmenti di breve durata


The importance-weighted autoencoder (IWAE) approach of Burda et al. defines a sequence of increasingly tighter bounds on the marginal likelihood of latent variable models. Recently, Cremer et al. reinterpreted the IWAE bounds as ordinary variational evidence lower bounds (ELBO) applied to increasingly accurate variational distributions. In this work, we provide yet another perspective on the IWAE bounds. We interpret each IWAE bound as a biased estimator of the true marginal likelihood where for the bound defined on $K$ samples we show the bias to be of order O(1/K). In our theoretical analysis of the IWAE objective we derive asymptotic bias and variance expressions. Based on this analysis we develop jackknife variational inference (JVI),  a family of bias-reduced estimators reducing the bias to $O(K^{-(m+1)})$ for any given m < K while retaining computational efficiency. Finally, we demonstrate that JVI leads to improved evidence estimates in variational autoencoders. We also report first results on applying JVI to learning variational autoencoders.   Our implementation is available at https://github.com/Microsoft/jackknife-variational-inference

L'inferenza variazionale è biased, facciamo debiasing.

Introduce l'inferenza variazionale jackknife, un metodo per il debiasing di obiettivi Monte Carlo come l'auto-encoder importance weighted.

Gli autori analizzano il bias e la varianza del limite IWAE e derivano un approccio jacknife per stimare i momenti come un modo per fare debiasing dell'IWAE per sample pesati con importanza finita.


In this paper, we consider the problem of autonomous lane changing for self driving vehicles in a multi-lane, multi-agent setting. We present a framework that demonstrates a more structured and data efficient alternative to end-to-end complete policy learning on problems where the high-level policy is hard to formulate using traditional optimization or rule based methods but well designed low-level controllers are available. Our framework uses deep reinforcement learning solely to obtain a high-level policy for tactical decision making, while still maintaining a tight integration with the low-level controller, thus getting the best of both worlds. We accomplish this with Q-masking, a technique with which we are able to incorporate prior knowledge, constraints, and information from a low-level controller, directly in to the learning process thereby simplifying the reward function and making learning faster and data efficient. We provide preliminary results in a simulator and show our approach to be more efficient than a greedy baseline, and more successful and safer than human driving.

Un framework che fornisce una policy per il cambio di corsia autonomo imparando a prendere decisioni tattiche di alto livello con il deep reinforcement learning, e mantenendo una stretta integrazione con un controller di basso livello per prendere azioni di basso livello.

Considera il problema del cambio di corsia autonomo per le auto a guida autonoma nel setting multi-lane multi-agente slot car, propone una nuova strategia di apprendimento Q-masking - accoppiando un controller definito di basso livello con una policy decisionale tattica di alto livello.

Questo articolo propone un approccio di deep Q-learning al problema del cambio di corsia usando il "Q-masking", che riduce lo spazio d'azione in base a vincoli o conoscenze precedenti.

Gli autori propongono un metodo che utilizza una policy di alto livello basata sul Q-learning che è combinata con una maschera contestuale derivata dai vincoli di sicurezza e dai controllori di basso livello, che disabilitano certe azioni dall'essere selezionabili in certi stati.


Despite the recent successes in robotic locomotion control, the design of robot relies heavily on human engineering. Automatic robot design has been a long studied subject, but the recent progress has been slowed due to the large combinatorial search space and the difficulty in evaluating the found candidates. To address the two challenges, we formulate automatic robot design as a graph search problem and perform evolution search in graph space. We propose Neural Graph Evolution (NGE), which performs selection on current candidates and evolves new ones iteratively. Different from previous approaches, NGE uses graph neural networks to parameterize the control policies, which reduces evaluation cost on new candidates with the help of skill transfer from previously evaluated designs. In addition, NGE applies Graph Mutation with Uncertainty (GM-UC) by incorporating model uncertainty, which reduces the search space by balancing exploration and exploitation. We show that NGE significantly outperforms previous methods by an order of magnitude. As shown in experiments, NGE is the first algorithm that can automatically discover kinematically preferred robotic graph structures, such as a fish with two symmetrical flat side-fins and a tail, or a cheetah with athletic front and back legs. Instead of using thousands of cores for weeks, NGE efficiently solves searching problem within a day on a single 64 CPU-core Amazon EC2  machine.

Ricerca automatica di design robotico con graph neural network

Propone un approccio per la progettazione automatica di robot basato sull'evoluzione del grafo neurale. Gli esperimenti dimostrano che ottimizzare sia il controller che l'hardware è meglio che ottimizzare solo il controller.

Gli autori propongono uno schema basato su una rappresentazione a grafo della struttura del robot, e una graph neural network come controllore per ottimizzare le strutture del robot, combinate con i loro controllori.


Deep learning on graphs has become a popular research topic with many applications. However, past work has concentrated on learning graph embedding tasks only, which is in contrast with advances in generative models for images and text. Is it possible to transfer this progress to the domain of graphs? We propose to sidestep hurdles associated with linearization of such discrete structures by having a decoder output a probabilistic fully-connected graph of a predefined maximum size directly at once. Our method is formulated as a variational autoencoder. We evaluate on the challenging task of conditional molecule generation.

Imparare a generare grafi usando metodi di deep learning in "one shot", producendo direttamente le probabilità di esistenza di nodi e archi e i vettori di attributi dei nodi.

Un encoder automatico variazionale per generare grafi


Long Short-Term Memory (LSTM) is one of the most widely used recurrent structures in sequence modeling. Its goal is to use gates to control the information flow (e.g., whether to skip some information/transformation or not) in the recurrent computations, although its practical implementation based on soft gates only partially achieves this goal and is easy to overfit. In this paper, we propose a new way for LSTM training, which pushes the values of the gates towards 0 or 1. By doing so, we can (1) better control the information flow: the gates are mostly open or closed, instead of in a middle state; and (2) avoid overfitting to certain extent: the gates operate at their flat regions, which is shown to correspond to better generalization ability. However, learning towards discrete values of the gates is generally difficult. To tackle this challenge, we leverage the recently developed Gumbel-Softmax trick from the field of variational methods, and make the model trainable with standard backpropagation. Experimental results on language modeling and machine translation show that (1) the values of the gates generated by our method are more reasonable and intuitively interpretable, and (2) our proposed method generalizes better and achieves better accuracy on test sets in all tasks. Moreover, the learnt models are not sensitive to low-precision approximation and low-rank approximation of the gate parameters due to the flat loss surface.

Proponiamo un nuovo algoritmo per il training LSTM forzando il learning verso gate a valore binario che abbiamo dimostrato avere molte proprietà desiderabili.

Proporre una nuova funzione "gate" per LSTM per abilitare i valori dei gate verso 0 o 1.

L'articolo mira a spingere le porte LSTM ad essere binarie impiegando il recente trucco del Gumbel-Softmax per ottenere una distribuzione categorica addestrabile end-to-end.


We present a personalized recommender system using neural network for recommending  products, such as eBooks, audio-books, Mobile Apps, Video and Music.  It produces recommendations based on customer’s implicit feedback history such  as purchases, listens or watches. Our key contribution is to formulate recommendation  problem as a model that encodes historical behavior to predict the future  behavior using soft data split, combining predictor and auto-encoder models. We  introduce convolutional layer for learning the importance (time decay) of the purchases  depending on their purchase date and demonstrate that the shape of the time  decay function can be well approximated by a parametrical function. We present  offline experimental results showing that neural networks with two hidden layers  can capture seasonality changes, and at the same time outperform other modeling  techniques, including our recommender in production. Most importantly, we  demonstrate that our model can be scaled to all digital categories, and we observe  significant improvements in an online A/B test. We also discuss key enhancements  to the neural network model and describe our production pipeline. Finally  we open-sourced our deep learning library which supports multi-gpu model parallel  training. This is an important feature in building neural network based recommenders  with large dimensionality of input and output data.

Migliorare le raccomandazioni utilizzando la modellazione sensibile al tempo con reti neurali in più categorie di prodotti su un sito web di vendita al dettaglio

L'articolo propone un nuovo metodo basato su reti neurali per la raccomandazione.

Gli autori descrivono una procedura di costruzione del loro sistema di production recommender system da zero e integrano il decadimento temporale degli acquisti nel framework di apprendimento.


Deep Learning (DL) algorithms based on Generative Adversarial Network (GAN) have demonstrated great potentials in computer vision tasks such as image restoration. Despite the rapid development of image restoration algorithms using DL and GANs, image restoration for specific scenarios, such as medical image enhancement and super-resolved identity recognition, are still facing challenges. How to ensure visually realistic restoration while avoiding hallucination or mode- collapse? How to make sure the visually plausible results do not contain hallucinated features jeopardizing downstream tasks such as pathology identification and subject identification?  Here we propose to resolve these challenges by coupling the GAN based image restoration framework with another task-specific network. With medical imaging restoration as an example, the proposed model conducts additional pathology recognition/classification task to ensure the preservation of detailed structures that are important to this task. Validated on multiple medical datasets, we demonstrate the proposed method leads to improved deep learning based image restoration while preserving the detailed structure and diagnostic features. Additionally, the trained task network show potentials to achieve super-human level performance in identifying pathology and diagnosis.  Further validation on super-resolved identity recognition tasks also show that the proposed method can be generalized for diverse image restoration tasks.

Accoppiare il framework di restauro dell'immagine basata su GAN con un'altra rete specifica per il task per generare un'immagine realistica preservando le feature specifiche del task.

Un nuovo metodo di Task-GAN di accoppiamento dell'immagine che accoppia GAN e una rete specifica del task, che aiuta ad evitare l'allucinazione o il mode collapse.

Gli autori propongono di aumentare il restauro di immagini basato su GAN con un altro ramo specifico, come i task di classificazione, per un ulteriore miglioramento.


Unsupervised anomaly detection on multi- or high-dimensional data is of great importance in both fundamental machine learning research and industrial applications, for which density estimation lies at the core. Although previous approaches based on dimensionality reduction followed by density estimation have made fruitful progress, they mainly suffer from decoupled model learning with inconsistent optimization goals and incapability of preserving essential information in the low-dimensional space. In this paper, we present a Deep Autoencoding Gaussian Mixture Model (DAGMM) for unsupervised anomaly detection. Our model utilizes a deep autoencoder to generate a low-dimensional representation and reconstruction error for each input data point, which is further fed into a Gaussian Mixture Model (GMM). Instead of using decoupled two-stage training and the standard Expectation-Maximization (EM) algorithm, DAGMM jointly optimizes the parameters of the deep autoencoder and the mixture model simultaneously in an end-to-end fashion, leveraging a separate estimation network to facilitate the parameter learning of the mixture model. The joint optimization, which well balances autoencoding reconstruction, density estimation of latent representation, and regularization, helps the autoencoder escape from less attractive local optima and further reduce reconstruction errors, avoiding the need of pre-training. Experimental results on several public benchmark datasets show that, DAGMM significantly outperforms state-of-the-art anomaly detection techniques, and achieves up to 14% improvement based on the standard F1 score.

Una deep neural network addestrata end-to-end che sfrutta la Gaussian Mixture Modeling per eseguire la stima della densità e il rilevamento unsupervised delle anomalie in uno spazio bidimensionale appreso da un deep autoencoder.

L'articolo presenta un framework di deep learning congiunto per la riduzione delle dimensioni e il clustering, che porta a un rilevamento delle anomalie con prestazioni competitive.

Una nuova tecnica per il rilevamento delle anomalie in cui la riduzione delle dimensioni e la stima della densità sono ottimizzate congiuntamente.


Generalization from limited examples, usually studied under the umbrella of meta-learning, equips learning techniques with the ability to adapt quickly in dynamical environments and proves to be an essential aspect of lifelong learning. In this paper, we introduce the Projective Subspace Networks (PSN), a deep learning paradigm that learns non-linear embeddings from limited supervision. In contrast to previous studies, the embedding in PSN deems samples of a given class to form an affine subspace. We will show that such modeling leads to robust solutions, yielding competitive results on supervised and semi-supervised few-shot classification. Moreover, our PSN approach has the ability of end-to-end learning. In contrast to previous works, our projective subspace can be thought of as a richer representation capturing higher-order information datapoints for modeling new concepts.

Abbiamo proposto le reti a sottospazio proiettivo per few shot learning e semi-supervised few shot learning

Questo articolo propone un nuovo approccio basato sull'embedding per il problema del few shot learning e un'estensione di questo modello al setting dell'apprendimento semi-supervised few shot.

Nuovo metodo per la classificazione completamente e semi-supervisionata few shot basato sull'apprendimento di un'embedding generale e poi sull'apprendimento di un sottospazio di esso per ogni classe


This paper investigates whether learning contingency-awareness and controllable aspects of an environment can lead to better exploration in reinforcement learning. To investigate this question, we consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE). In this study, we develop an attentive dynamics model (ADM) that discovers controllable elements of the observations, which are often associated with the location of the character in Atari games. The ADM is trained in a self-supervised fashion to predict the actions taken by the agent. The learned contingency information is used as a part of the state representation for exploration purposes. We demonstrate that combining actor-critic algorithm with count-based exploration using our representation achieves impressive results on a set of notoriously challenging Atari games due to sparse rewards. For example, we report a state-of-the-art score of >11,000 points on Montezuma's Revenge without using expert demonstrations, explicit high-level information (e.g., RAM states), or supervisory data. Our experiments confirm that contingency-awareness is indeed an extremely powerful concept for tackling exploration problems in reinforcement learning and opens up interesting research questions for further investigations.

Studiamo la consapevolezza della contingenza e gli aspetti controllabili nell'esplorazione e raggiungiamo prestazioni allo stato dell'arte su Montezuma's Revenge senza dimostrazioni di esperti.

Questo articolo indaga il problema dell'estrazione di una rappresentazione di stato significativa per aiutare l'esplorazione quando ci si confronta con un task con reward sparsa, identificando le feature controllabili (apprese) dello stato

Questo articolo propone l'idea nuova di usare la consapevolezza della contingenza per aiutare l'esplorazione in task di reinforcement learning a reward sparsa, ottenendo risultati allo stato dell'arte.


Disentangling factors of variation has always been a challenging problem in representation learning. Existing algorithms suffer from many limitations, such as unpredictable disentangling factors, bad quality of generated images from encodings, lack of identity information, etc. In this paper, we proposed a supervised algorithm called DNA-GAN trying to disentangle different attributes of images. The latent representations of images are DNA-like, in which each individual piece represents an independent factor of variation. By annihilating the recessive piece and swapping a certain piece of two latent representations, we obtain another two different representations which could be decoded into images. In order to obtain realistic images and also disentangled representations, we introduced the discriminator for adversarial training. Experiments on Multi-PIE and CelebA datasets demonstrate the effectiveness of our method and the advantage of overcoming limitations existing in other methods.

Abbiamo proposto un algoritmo supervisionato, DNA-GAN, per distinguere più attributi delle immagini.

Questo articolo indaga il problema della generazione di immagini condizionata dagli attributi usando generative adversarial network, e propone di generare immagini da attributi e codice latente come rappresentazione di alto livello.

Questo articolo ha proposto un nuovo metodo per distinguere diversi attributi delle immagini utilizzando un nuovo framework di DNA GAN


Representations learnt through deep neural networks tend to be highly informative, but opaque in terms of what information they learn to encode. We introduce an approach to probabilistic modelling that learns to represent data with two separate deep representations: an invariant representation that encodes the information of the class from which the data belongs, and an equivariant representation that encodes the symmetry transformation defining the particular data point within the class manifold (equivariant in the sense that the representation varies naturally with symmetry transformations). This approach to representation learning is conceptually transparent, easy to implement, and in-principle generally applicable to any data comprised of discrete classes of continuous distributions (e.g. objects in images, topics in language, individuals in behavioural data). We demonstrate qualitatively compelling representation learning and competitive quantitative performance, in both supervised and semi-supervised settings, versus comparable modelling approaches in the literature with little fine tuning.

Questo articolo presenta una nuova tecnica di modellazione generativa di variabili latenti che permette la rappresentazione di informazioni globali in una variabile latente e di informazioni locali in un'altra variabile latente.

L'articolo presenta un VAE che usa le label per separare la rappresentazione appresa in una parte invariante e una covariante.


Convolutional neural networks (CNNs) have been successfully applied to many recognition and learning tasks using a universal recipe;  training a deep model on a very large dataset of supervised examples. However, this approach is rather restrictive in practice since collecting a large set of labeled images is very expensive. One way to ease this problem is coming up with smart ways for choosing images to be labelled from a  very large collection (i.e. active learning).   Our empirical study suggests that many of the active learning heuristics in the literature are not effective when applied to CNNs when applied in batch setting. Inspired by these limitations, we define the problem of active learning as core-set selection, i.e. choosing set of points such that a model learned over the selected subset is competitive for the remaining data points. We further present a theoretical result characterizing the performance of any selected subset using the geometry of the datapoints. As an active learning algorithm, we choose the subset which is expected to yield best result according to our characterization. Our experiments show that the proposed method significantly outperforms existing approaches in image classification experiments by a large margin.

Ci avviciniamo al problema dell'active learning come un problema di selezione del core-set e mostriamo che questo approccio è particolarmente utile nel setting del batch active learning che è cruciale quando si addestrano le CNN.

Gli autori forniscono un algoritmo di active learning indipendente dall'algoritmo per la classificazione multiclasse

L'articolo propone un algoritmo di active learning in modalità batch per CNN come un problema di core-set che supera il sampling casuale e il sampling di incertezza.

Studia l'active learning per le reti neurali convoluzionali e formula il problema dell'active learning come selezione di core-set e presenta una nuova strategia


Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP). This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because EVGP prevents important gradient components from being back-propagated adequately over a large number of steps. We introduce a simple stochastic algorithm (\textit{h}-detach) that is specific to LSTM optimization and targeted towards addressing this problem. Specifically, we show that when the LSTM weights are large, the gradient components through the linear path (cell state) in the LSTM computational graph get suppressed. Based on the hypothesis that these components carry information about long term dependencies (which we show empirically), their suppression can prevent LSTMs from capturing them. Our algorithm\footnote{Our code is available at https://github.com/bhargav104/h-detach. } prevents gradients flowing through this path from getting suppressed, thus allowing the LSTM to capture such dependencies better. We show significant improvements over vanilla LSTM gradient based training in terms of convergence speed, robustness to seed and learning rate, and generalization using our modification of LSTM gradient on various benchmark datasets.

Un semplice algoritmo per migliorare l'ottimizzazione e la gestione delle dipendenze a lungo termine in LSTM

L'articolo introduce un semplice algoritmo stocastico chiamato h-detach che è specifico per l'ottimizzazione LSTM e mirato ad affrontare questo problema.

Propone una semplice modifica al processo di training dell'LSTM per facilitare la propagazione del gradiente lungo gli stati delle cellule, o il "percorso temporale lineare".


Convolutional Neural Networks (CNNs) significantly improve the state-of-the-art for many applications, especially in computer vision. However, CNNs still suffer from a tendency to confidently classify out-distribution samples from unknown classes into pre-defined known classes. Further, they are also vulnerable to adversarial examples. We are relating these two issues through the tendency of CNNs to over-generalize for areas of the input space not covered well by the training set. We show that a CNN augmented with an extra output class can act as a simple yet effective end-to-end model for controlling over-generalization. As an appropriate training set for the extra class, we introduce two resources that are computationally efficient to obtain: a representative natural out-distribution set and interpolated in-distribution samples. To help select a representative natural out-distribution set among available ones, we propose a simple measurement to assess an out-distribution set's fitness. We also demonstrate that training such an augmented CNN with representative out-distribution natural datasets and some interpolated samples allows it to better handle a wide range of unseen out-distribution samples and black-box adversarial examples without training it on any adversaries. Finally, we show that generation of white-box adversarial attacks using our proposed augmented CNN can become harder, as the attack algorithms have to get around the rejection regions when generating actual adversaries.

Addestrare correttamente le CNN con la classe dustbin aumenta la loro robustezza agli adversarial attack e la loro capacità di gestire i sample fuori distribuzione.

Questo articolo propone di aggiungere una label aggiuntiva per rilevare i sample OOD e gli adversarial example nei modelli CNN.

L'articolo propone una classe aggiuntiva che incorpora immagini naturali out-distribution e immagini interpolate per gli adversarial e out-distribution sample CNN


Modern deep artificial neural networks have achieved impressive results through models with very large capacity---compared to the number of training examples---that control overfitting with the help of different forms of regularization. Regularization can be implicit, as is the case of stochastic gradient descent or parameter sharing in convolutional layers, or explicit. Most common explicit regularization techniques, such as dropout and weight decay, reduce the effective capacity of the model and typically require the use of deeper and wider architectures to compensate for the reduced capacity. Although these techniques have been proven successful in terms of results, they seem to waste capacity. In contrast, data augmentation techniques reduce the generalization error by increasing the number of training examples and without reducing the effective capacity. In this paper we systematically analyze the effect of data augmentation on some popular architectures and conclude that data augmentation alone---without any other explicit regularization techniques---can achieve the same performance or higher as regularized models, especially when training with fewer examples.

In una deep convolutional neural network con un livello sufficiente di data augmentation, ottimizzata da SGD, i regolarizzatori espliciti (weight decay e dropout) potrebbero non fornire alcun miglioramento aggiuntivo della generalizzazione.

Questo articolo propone data augmentation come alternativa alle tecniche di regolarizzazione comunemente usate, e mostra che per alcuni modelli/task di riferimento la stessa performance di generalizzazione può essere ottenuta usando solo data augmentation.

Questo articolo presenta uno studio sistematico della data augmentation nella classificazione delle immagini con deep neural network, suggerendo che data augmentation possa replicare alcuni regolarizzatori comuni come il weight decay e il dropout.


Text editing on mobile devices can be a tedious process. To perform various editing operations, a user must repeatedly move his or her fingers between the text input area and the keyboard, making multiple round trips and breaking the flow of typing. In this work, we present Gedit, a system of on-keyboard gestures for convenient mobile text editing. Our design includes a ring gesture and flicks for cursor control, bezel gestures for mode switching, and four gesture shortcuts for copy, paste, cut, and undo. Variations of our gestures exist for one and two hands. We conducted an experiment to compare Gedit with the de facto touch+widget based editing interactions. Our results showed that Gedit’s gestures were easy to learn, 24% and 17% faster than the de facto interactions for one- and two-handed use, respectively, and preferred by participants.

In questo lavoro, presentiamo Gedit, un sistema di gesture sulla tastiera per un comodo editing di testo su mobile.

Riporta la progettazione e la valutazione delle tecniche di interazione di Gedit.

Presenta una nuova serie di gesture tattili per eseguire la transizione senza soluzione di continuità tra l'inserimento e la modifica del testo nei dispositivi mobili


Deep learning achieves remarkable generalization capability with overwhelming number of model parameters. Theoretical understanding of deep learning generalization receives recent attention yet remains not fully explored. This paper attempts to provide an alternative understanding from the perspective of maximum entropy. We first derive two feature conditions that softmax regression strictly apply maximum entropy principle. DNN is then regarded as approximating the feature conditions with multilayer feature learning, and proved to be a recursive solution towards maximum entropy principle. The connection between DNN and maximum entropy well explains why typical designs such as shortcut and regularization improves model generalization, and provides instructions for future model development.

Dimostriamo che una DNN è una soluzione approssimata ricorsivamente al principio di massima entropia.

Presenta una derivazione che collega una DNN all'applicazione ricorsiva dell'adattamento del modello di massima entropia.

L'articolo mira a fornire una visione del deep learning dalla prospettiva del principio di massima entropia.


As people learn to navigate the world, autonomic nervous system (e.g., ``fight or flight) responses provide intrinsic feedback about the potential consequence of action choices (e.g., becoming nervous when close to a cliff edge or driving fast around a bend.) Physiological changes are correlated with these biological preparations to protect one-self from danger. We present a novel approach to reinforcement learning that leverages a task-independent intrinsic reward function trained on peripheral pulse measurements that are correlated with human autonomic nervous system responses. Our hypothesis is that such reward functions can circumvent the challenges associated with sparse and skewed rewards in reinforcement learning settings and can help improve sample efficiency. We test this in a simulated driving environment and show that it can increase the speed of learning and reduce the number of collisions during the learning stage.

Presentiamo un nuovo approccio al reinforcement learning che sfrutta una funzione di reward intrinseca indipendente dal task, addestrata sulle misurazioni delle pulsazioni periferiche che sono correlate alle risposte del sistema nervoso autonomo dell'essere umano.

Propone un framework di reinforcement learning basato sulla reazione emotiva umana nel contesto della guida autonoma.

Gli autori propongono di utilizzare i segnali, come le risposte viscerali autonome di base che influenzano il processo decisionale, all'interno del framework di RL aumentando le funzioni di reward RL con un modello appreso direttamente dalle risposte del sistema nervoso umano.

Propone di utilizzare i segnali fisiologici per migliorare le prestazioni degli algoritmi di reinforcement learning e costruire una funzione di reward intrinseca che è meno sparsa misurando l'ampiezza delle pulsazioni cardiache


Deep convolutional neural networks (CNNs) are known to be robust against label noise on extensive datasets. However, at the same time, CNNs are capable of memorizing all labels even if they are random, which means they can memorize corrupted labels. Are CNNs robust or fragile to label noise? Much of researches focusing on such memorization uses class-independent label noise to simulate label corruption, but this setting is simple and unrealistic. In this paper, we investigate the behavior of CNNs under class-dependently simulated label noise, which is generated based on the conceptual distance between classes of a large dataset (i.e., ImageNet-1k). Contrary to previous knowledge, we reveal CNNs are more robust to such class-dependent label noise than class-independent label noise. We also demonstrate the networks under class-dependent noise situations learn similar representation to the no noise situation, compared to class-independent noise situations.

Le CNN sono robuste o fragili al rumore delle label? Nella pratica, robuste.

Gli autori sfidano la robustezza delle CNN al rumore delle label usando l'albero ImageNet 1k di WordNet.

Un'analisi delle prestazioni del modello di rete neurale convoluzionale quando viene introdotto rumore dipendente dalla classe e indipendente dalla classe

Dimostra che le CNN sono più robuste al rumore delle label relativo alla classe e sostiene che il rumore del mondo reale dovrebbe essere relativo alla classe


Efficient audio synthesis is an inherently difficult machine learning task, as human perception is sensitive to both global structure and fine-scale waveform coherence. Autoregressive models, such as WaveNet, model local structure at the expense of global latent structure and slow iterative sampling, while Generative Adversarial Networks (GANs), have global latent conditioning and efficient parallel sampling, but struggle to generate locally-coherent audio waveforms. Herein, we demonstrate that GANs can in fact generate high-fidelity and locally-coherent audio by modeling log magnitudes and instantaneous frequencies  with sufficient frequency resolution in the spectral domain. Through extensive empirical investigations on the NSynth dataset, we demonstrate that GANs are able to outperform strong WaveNet baselines on automated and human evaluation metrics, and efficiently generate audio several orders of magnitude faster than their autoregressive counterparts.

Sintesi audio di alta qualità con le GAN

Propone un approccio che utilizza il framework delle GAN per generare l'audio attraverso la modellazione di log magnitudini e frequenze istantanee con una risoluzione di frequenza sufficiente nel dominio spettrale.

Una strategia per generare sample audio dal rumore con le GAN, con modifiche all'architettura e alla rappresentazione necessarie per generare un audio convincente che contenga un codice latente interpretabile.

Presenta una semplice idea per rappresentare meglio i dati audio in modo da poter applicare modelli convoluzionali come le generative adversarial network


In this work we propose a novel approach for learning graph representation of the data using gradients obtained via backpropagation. Next we build a neural network architecture compatible with our optimization approach and motivated by graph filtering in the vertex domain. We demonstrate that the learned graph has richer structure than often used nearest neighbors graphs constructed based on features similarity. Our experiments demonstrate that we can improve prediction quality for several convolution on graphs architectures, while others appeared to be insensitive to the input graph.

Ottimizzazione dei grafi con filtraggio del segnale nel dominio dei vertici.

L'articolo studia l'apprendimento della matrice di adiacenza di un grafo con un grafo indiretto scarsamente connesso con pesi non negativi sugli archi e utilizza un algoritmo di projected sub-gradient descent.

Sviluppa un nuovo schema per fare backpropagation sulla matrice di adiacenza di un grafo di una rete neurale


The use of AR in an industrial context could help for the training of new operators. To be able to use an AR guidance system, we need a tool to quickly create a 3D representation of the assembly line and of its AR annotations. This tool should be very easy to use by an operator who is not an AR or VR specialist: typically the manager of the assembly line. This is why we proposed WAAT, a 3D authoring tool allowing user to quickly create 3D models of the workstations, and also test the AR guidance placement. WAAT makes on-site authoring possible, which should really help to have an accurate 3D representation of the assembly line. The verification of AR guidance should also be very useful to make sure everything is visible and doesn't interfere with technical tasks. In addition to these features, our future work will be directed in the deployment of WAAT into a real boiler assembly line to assess the usability of this solution.

Questo articolo descrive uno strumento di authoring 3D per fornire AR nelle linee di assemblaggio dell'industria 4.0

L'articolo affronta il modo in cui gli strumenti di authoring AR supportino il training dei sistemi della catena di montaggio e propone un approccio

Un sistema di guida AR per linee di assemblaggio industriali che permette l'authoring in loco di contenuti AR.

Presenta un sistema che permette agli operai di una fabbrica di essere addestrati in modo più efficiente usando un sistema di realtà aumentata.


Generative adversarial network (GAN) is one of the best known unsupervised learning techniques these days due to its superior ability to learn data distributions. In spite of its great success in applications, GAN is known to be notoriously hard to train. The tremendous amount of time it takes to run the training algorithm and its sensitivity to hyper-parameter tuning have been haunting researchers in this area. To resolve these issues, we need to first understand how GANs work. Herein, we take a step toward this direction by examining the dynamics of GANs. We relate a large class of GANs including the Wasserstein GANs to max-min optimization problems with the coupling term being linear over the discriminator. By developing new primal-dual optimization tools, we show that, with a proper stepsize choice, the widely used first-order iterative algorithm in training GANs would in fact converge to a stationary solution with a sublinear rate. The same framework also applies to multi-task learning and distributional robust learning problems. We verify our analysis on numerical examples with both synthetic and real data sets. We hope our analysis shed light on future studies on the theoretical properties of relevant machine learning problems.

Mostriamo che, con una scelta appropriata della stepsize, l'algoritmo iterativo del primo ordine ampiamente utilizzato nel training delle GAN convergerebbe a una soluzione stazionaria con un tasso sublineare.

Questo articolo usa le GAN e multi-task learning per fornire una garanzia di convergenza per algoritmi primali-duali su certi problemi min-max.

Analizza la dinamica di apprendimento delle GAN formulando il problema come un problema di ottimizzazione primale-duale assumendo una classe limitata di modelli


Social dilemmas, where mutual cooperation can lead to high payoffs but participants face incentives to cheat, are ubiquitous in multi-agent interaction. We wish to construct agents that cooperate with pure cooperators, avoid exploitation by pure defectors, and incentivize cooperation from the rest. However, often the actions taken by a partner are (partially) unobserved or the consequences of individual actions are hard to predict. We show that in a large class of games good strategies can be constructed by conditioning one's behavior solely on outcomes (ie. one's past rewards). We call this consequentialist conditional cooperation. We show how to construct such strategies using deep reinforcement learning techniques and demonstrate, both analytically and experimentally, that they are effective in social dilemmas beyond simple matrix games. We also show the limitations of relying purely on consequences and discuss the need for understanding both the consequences of and the intentions behind an action.

Mostriamo come usare deep RL per costruire agenti che possono risolvere dilemmi sociali al di là dei matrix game.

Imparare a giocare giochi a somma generale a due giocatori con uno stato avente informazioni imperfette

Specifica una strategia di innesco (CCC) e l'algoritmo corrispondente, dimostrando la convergenza verso risultati efficienti nei dilemmi sociali senza bisogno che gli agenti osservino le azioni degli altri.


In distributed training, the communication cost due to the transmission of gradients  or the parameters of the deep model is a major bottleneck in scaling up the number  of processing nodes. To address this issue, we propose dithered quantization for  the transmission of the stochastic gradients and show that training with Dithered  Quantized Stochastic Gradients (DQSG) is similar to the training with unquantized  SGs perturbed by an independent bounded uniform noise, in contrast to the other  quantization methods where the perturbation depends on the gradients and hence,  complicating the convergence analysis. We study the convergence of training  algorithms using DQSG and the trade off between the number of quantization  levels and the training time. Next, we observe that there is a correlation among the  SGs computed by workers that can be utilized to further reduce the communication  overhead without any performance loss. Hence, we develop a simple yet effective  quantization scheme, nested dithered quantized SG (NDQSG), that can reduce the  communication significantly without requiring the workers communicating extra  information to each other. We prove that although NDQSG requires significantly  less bits, it can achieve the same quantization variance bound as DQSG. Our  simulation results confirm the effectiveness of training using DQSG and NDQSG  in reducing the communication bits or the convergence time compared to the  existing methods without sacrificing the accuracy of the trained model.

L'articolo propone e analizza due schemi di quantizzazione per comunicare i gradienti stocastici nell'apprendimento distribuito che ridurrebbero i costi di comunicazione rispetto allo stato dell'arte, mantenendo la stessa precisione.

Gli autori propongono di applicare la quantizzazione dithered ai gradienti stocastici calcolati durante il processo di training, che migliora l'errore di quantizzazione e raggiunge risultati superiori rispetto alle baseline, e propongono uno schema annidato per ridurre i costi di comunicazione.

Gli autori stabiliscono una connessione tra la riduzione della comunicazione nell'ottimizzazione distribuita e la quantizzazione dithered e sviluppano due nuovi algoritmi di training distribuiti in cui l'overhead della comunicazione è significativamente ridotto.


Deep neural networks have been shown to perform well in many classical machine learning problems, especially in image classification tasks. However, researchers have found that neural networks can be easily fooled, and they are surprisingly sensitive to small perturbations imperceptible to humans.   Carefully crafted input images (adversarial examples) can force a well-trained neural network to provide arbitrary outputs.   Including adversarial examples during training is a popular defense mechanism against adversarial attacks. In this paper we propose a new defensive mechanism under the generative adversarial network~(GAN) framework. We model the adversarial noise using a generative network, trained jointly with a classification discriminative network as a minimax game. We show empirically that our adversarial network approach works well against black box attacks, with performance on par with state-of-art methods such as ensemble adversarial training and adversarial training with projected gradient descent.

Addestrare congiuntamente una rete generatrice di rumore adversarial con una rete di classificazione per fornire una migliore robustezza agli adversarial attack.

Una soluzione GAN per deep model per la classificazione, di fronte agli attacchi white e black box, che produce modelli robusti.

L'articolo propone un meccanismo difensivo contro gli adversarial attack utilizzando GAN con perturbazioni generate e poi usate come adversarial example e un discriminatore usato per distinguerli


Deep learning has become the state of the art approach in many machine learning problems such as classification. It has recently been shown that deep learning is highly vulnerable to adversarial perturbations. Taking the camera systems of self-driving cars as an example, small adversarial perturbations can cause the system to  make errors in important tasks, such as classifying traffic signs or detecting pedestrians. Hence, in order to use deep learning without safety concerns a proper defense strategy is required. We propose to use ensemble methods as a defense strategy against adversarial perturbations. We find that an attack leading one model to misclassify does not imply the same for other networks performing the same task. This makes ensemble methods an attractive defense strategy against adversarial attacks. We empirically show for the MNIST and the CIFAR-10 data sets that ensemble methods not only improve the accuracy of neural networks on test data but also increase their robustness against adversarial perturbations.

Utilizzo di metodi d'ensemble come difesa dalle adversarial perturbation prodotte contro le deep neural network.

Questo articolo propone di utilizzare l'ensembling come un meccanismo di adversarial defense.

Ha studiato empiricamente la robustezza di diversi ensemble di deep neural network ai due tipi di attacchi, FGSM e BIM, su due dataset popolari, MNIST e CIFAR10


In this paper, we propose the Associative Conversation Model that generates visual information from textual information and uses it for generating sentences in order to utilize visual information in a dialogue system without image input. In research on Neural Machine Translation, there are studies that generate translated sentences using both images and sentences, and these studies show that visual information improves translation performance. However, it is not possible to use sentence generation algorithms using images for the dialogue systems since many text-based dialogue systems only accept text input. Our approach generates (associates) visual information from input text and generates response text using context vector  fusing associative visual information and sentence textual information. A comparative experiment between our proposed model and a model without association showed that our proposed model is generating useful sentences by associating visual information related to sentences. Furthermore, analysis experiment of visual association showed that our proposed model generates (associates) visual information effective for sentence generation.

Proposta di un metodo di generazione di frasi basato sulla fusione tra informazioni testuali e informazioni visive associate alle informazioni testuali

Questo lavoro descrive un modello di deep learning per sistemi di dialogo che sfrutta le informazioni visive.

Questo articolo propone un nuovo dataset per il grounded dialog e fa un'osservazione computazionale che potrebbe aiutare a ragionare sulla visione anche quando si esegue un dialogo basato sul testo.

Propone di arricchire i tradizionali approcci di generazione di frasi/dialogo basati sul testo incorporando informazioni visive, raccogliendo un dataset che consiste sia di testo che di immagini o video associati


Feedforward convolutional neural network has achieved a great success in many computer vision tasks. While it validly imitates the hierarchical structure of biological visual system, it still lacks one essential architectural feature: contextual recurrent connections with feedback, which widely exists in biological visual system. In this work, we designed a Contextual Recurrent Convolutional Network with this feature embedded in a standard CNN structure. We found that such feedback connections could enable lower layers to ``rethink" about their representations given the top-down contextual information. We carefully studied the components of this network, and showed its robustness and superiority over feedforward baselines in such tasks as noise image classification, partially occluded object recognition and fine-grained image classification. We believed this work could be an important step to help bridge the gap between computer vision models and real biological visual system.

Abbiamo proposto una nuova rete convoluzionale ricorrente contestuale con proprietà robuste di apprendimento visivo

Questo articolo introduce la connessione di feedback per migliorare l'apprendimento delle feature attraverso l'embedding di informazioni del contesto.

L'articolo propone di aggiungere connessioni "ricorrenti" all'interno di una rete di convoluzione con meccanismo di gating.


Deep neural networks have led to a series of breakthroughs, dramatically improving the state-of-the-art in many domains. The techniques driving these advances, however, lack a formal method to account for model uncertainty. While the Bayesian approach to learning provides a solid theoretical framework to handle uncertainty, inference in Bayesian-inspired deep neural networks is difficult. In this paper, we provide a practical approach to Bayesian learning that relies on a regularization technique found in nearly every modern network, batch normalization. We show that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models, and we demonstrate how this finding allows us to make useful estimates of the model uncertainty. Using our approach, it is possible to make meaningful uncertainty estimates using conventional architectures without modifying the network or the training procedure. Our approach is thoroughly validated in a series of empirical experiments on different tasks and using various measures, showing it to outperform baselines on a majority of datasets with strong statistical significance.

Mostriamo che il training di una deep network usando la batch normalization è equivalente all'inferenza approssimativa nei modelli bayesiani, e dimostriamo come questa scoperta ci permette di fare stime utili dell'incertezza del modello nelle reti convenzionali.

Questo articolo propone di usare la batch normalization al momento del test per ottenere l'incertezza predittiva, e mostra che la predizione Monte Carlo al momento del test usando batch norm è migliore del dropout.

Propone che la procedura di regolarizzazione chiamata batch normalization possa essere intesa come l'esecuzione di un'inferenza bayesiana approssimata, che si comporta in modo simile al dropout MC in termini delle stime di incertezza che produce.


Data-parallel neural network training is network-intensive, so gradient dropping was designed to exchange only large gradients.   However, gradient dropping has been shown to slow convergence.   We propose to improve convergence by having each node combine its locally computed gradient with the sparse global gradient exchanged over the network. We empirically confirm with machine translation tasks that gradient dropping with local gradients approaches convergence 48% faster than non-compressed multi-node training and 28% faster compared to vanilla gradient dropping. We also show that gradient dropping with a local gradient update does not reduce the model's final quality.

Miglioriamo il gradient dropping (una tecnica per scambiare solo gradienti di grandi dimensioni nel training distribuito) incorporando i gradienti locali mentre facciamo un aggiornamento dei parametri per ridurre la loss di qualità e migliorare ulteriormente il tempo di training.

Questo articolo propone 3 modalità per combinare i gradienti locali e globali per utilizzare meglio più nodi di calcolo

Si occupa del problema della riduzione dei requisiti di comunicazione per l'implementazione delle tecniche di ottimizzazione distribuita, in particolare SGD


We establish the relation between Distributional RL and the Upper Confidence Bound (UCB) approach to exploration.      In this paper we show that the density of the Q function estimated by Distributional RL can be successfully used for the estimation of UCB. This approach does not require counting and, therefore, generalizes well to the Deep RL. We also point to the asymmetry of the empirical densities estimated by the Distributional RL algorithms like QR-DQN. This observation leads to the reexamination of the variance's performance in the UCB type approach to exploration. We introduce truncated variance as an alternative estimator of the UCB and a novel algorithm based on it. We empirically show that newly introduced algorithm achieves better performance in multi-armed bandits setting. Finally, we extend this approach to high-dimensional setting and test it on the Atari 2600 games. New approach achieves better performance compared to QR-DQN in 26 of games, 13 ties out of 49 games.

Esplorazione con RL distributivo e varianza troncata.

Presenta un metodo RL per gestire i trade-off di exploration-exploitation tramite tecniche UCB.

Un metodo per utilizzare la distribuzione appresa dalla Quantile Regression DQN per l'esplorazione, al posto della solita strategia epsilon-greedy.

Propone nuovi algoritmi (QUCB e QUCB+) per gestire il tradeoff di esplorazione in Multi-Armed Bandit e più in generale nel Reinforcement Learning


Good representations facilitate transfer learning and few-shot learning. Motivated by theories of language and communication that explain why communities with large number of speakers have, on average, simpler languages with more regularity, we cast the representation learning problem in terms of learning to communicate. Our starting  point sees traditional autoencoders as  a single encoder with a fixed decoder partner that must learn to communicate. Generalizing from there, we introduce community-based autoencoders in which multiple encoders and decoders collectively learn representations by being randomly paired up on successive training iterations. Our experiments show that increasing community sizes reduce idiosyncrasies in the learned codes, resulting in more invariant representations with increased reusability and structure.

Motivati dalle teorie del linguaggio e della comunicazione, introduciamo autoencoder basati sulla comunità, in cui encoder e decoder multipli imparano collettivamente rappresentazioni strutturate e riutilizzabili.

Gli autori affrontano il problema del representation learning, mirano a costruire una rappresentazione riutilizzabile e strutturata, sostengono che il co-adattamento tra encoder e decoder nell'AE tradizionale produce una scarsa rappresentazione, e introducono autoencoder basati sulla comunità.

L'articolo presenta un framework di autoencoder basato sulla comunità per affrontare il co-adattamento di codificatori e decodificatori e mira a costruire rappresentazioni migliori.


Humans are experts at high-fidelity imitation -- closely mimicking a demonstration, often in one attempt. Humans use this ability to quickly solve a  task instance, and to bootstrap learning of new tasks. Achieving these abilities in autonomous agents is an open problem. In this paper, we introduce an off-policy RL algorithm (MetaMimic) to narrow this gap. MetaMimic can learn both (i) policies for high-fidelity one-shot imitation of diverse novel skills, and (ii) policies that enable the agent to solve tasks more efficiently than the demonstrators. MetaMimic relies on the principle of storing all experiences in a memory and replaying these to learn massive deep neural network policies by off-policy RL. This paper introduces, to the best of our knowledge, the largest existing neural networks for deep RL and shows that larger networks with normalization are needed to achieve one-shot high-fidelity imitation on a challenging manipulation task.  The results also show that both types of policy can be learned from vision, in spite of the task rewards being sparse, and without access to demonstrator actions.

Presentiamo MetaMimic, un algoritmo che prende come input un dataset di dimostrazioni e produce (i) una policy di imitazione ad alta fedeltà one-shot e (ii) una policy di task incondizionata.

L'articolo esamina il problema dell'imitazione one-shot con alta precisione di imitazione, estendendo DDPGfD per usare solo traiettorie di stato.

Questo articolo propone un approccio per l'imitazione one-shot con alta precisione, e affronta il problema comune dell'esplorazione nell'imitation learning.

Presenta un metodo di RL per l'apprendimento dalla dimostrazione video senza accesso alle azioni degli esperti


Normalization methods are a central building block in the deep learning toolbox. They accelerate and stabilize training, while decreasing the dependence on manually tuned learning rate schedules. When learning from multi-modal distributions, the effectiveness of batch normalization (BN), arguably the most prominent normalization method, is reduced. As a remedy, we propose a more flexible approach: by extending the normalization to more than a single mean and variance, we detect modes of data on-the-fly, jointly normalizing samples that share common features. We demonstrate that our method outperforms BN and other widely used normalization techniques in several experiments, including single and multi-task datasets.

Presentiamo un nuovo metodo di normalizzazione per le deep neural network che è robusto alle multi-modalità nelle distribuzioni di feature intermedie.

Metodo di normalizzazione che impara la distribuzione multi-modale nello spazio delle feature

Propone una generalizzazione della Batch Normalization sotto l'ipotesi che la statistica delle attivazioni unitarie sulle batch e sulle dimensioni spaziali non sia unimodale


Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving. However, traditional multilingual translation usually yields inferior accuracy compared with the counterpart using individual models for each language pair, due to language diversity and model capacity limitations. In this paper, we propose a distillation-based approach to boost the accuracy of multilingual machine translation. Specifically, individual models are first trained and regarded as teachers, and then the multilingual model is trained to fit the training data and match the outputs of individual models simultaneously through knowledge distillation. Experiments on IWSLT, WMT and Ted talk translation datasets demonstrate the effectiveness of our method. Particularly, we show that one model is enough to handle multiple languages (up to 44 languages in our experiment), with comparable or even better accuracy than individual models.

Abbiamo proposto un metodo basato sulla knowledge distillation per aumentare la precisione della traduzione automatica neurale multilingue.

Un modello di traduzione automatica multilingue many-to-one che prima addestra modelli separati per ogni coppia di lingue e poi esegue la distillation.

L'articolo mira al training di un modello di traduzione automatica aumentando la loss standard di cross-entropy con una componente di distillation basata su teacher model individuali (single-language-pair).


What makes humans so good at solving seemingly complex video games?   Unlike computers, humans bring in a great deal of prior knowledge about the world, enabling efficient decision making. This paper investigates the role of human priors for solving video games. Given a sample game, we conduct a series of ablation studies to quantify the importance of various priors. We do this by modifying the video game environment to systematically mask different types of visual information that could be used by humans as priors. We find that removal of some prior knowledge causes a drastic degradation in the speed with which human players solve the game, e.g. from 2 minutes to over 20 minutes. Furthermore, our results indicate that general priors, such as the importance of objects and visual consistency, are critical for efficient game-play.

Indaghiamo i vari tipi di conoscenza preliminare che aiutano l'apprendimento umano e troviamo che i prior generali sugli oggetti giocano il ruolo più critico nel guidare il gioco umano.

Gli autori studiano tramite un esperimento quali aspetti dei prior umani sono importanti per reinforcement learning nei videogiochi.

Gli autori presentano uno studio sui prior impiegati dagli esseri umani nel giocare ai videogiochi e dimostrano l'esistenza di una tassonomia di feature che influenzano la capacità di completare i task nel gioco in vari gradi.


Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated. Examples include grid search, uniform random search, low discrepancy sequences, and other sampling distributions.  In particular, we propose the use of $k$-determinantal point processes in  hyperparameter optimization via random search. Compared to conventional uniform random search where hyperparameter settings are sampled independently, a $k$-DPP promotes diversity.   We describe an approach that transforms hyperparameter search spaces for efficient use with a $k$-DPP. In addition, we introduce a novel Metropolis-Hastings algorithm which can sample from $k$-DPPs defined over spaces with a mixture of discrete and continuous dimensions. Our experiments show significant benefits over uniform random search  in realistic scenarios with a limited budget for training supervised learners, whether in serial or parallel.

Spinti dalla necessità di metodi di ottimizzazione degli iperparametri parallelizzabili e open-loop, proponiamo l'uso di k-determinantal point process nell'ottimizzazione degli iperparametri tramite ricerca casuale.

Propone di usare il k-DPP per selezionare i punti candidati nelle ricerche di iperparametri.

Gli autori propongono k-DPP come un metodo open-loop per l'ottimizzazione degli iperparametri e forniscono il suo studio empirico e il confronto con altri metodi.

Considera la ricerca degli iperparametri non sequenziale e non informata usando determinantal point process, che sono distribuzioni di probabilità su sottoinsiemi di un ground set con la proprietà che i sottoinsiemi con elementi più 'diversi' hanno una probabilità più alta


In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch.  When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task.  However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task.  In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model.  We eventually recommend a simple $L^2$ penalty using the pre-trained model as a reference, and we show that this approach behaves much better than the standard scheme using weight decay on a partially frozen network.

Nel transfer learning induttivo, il fine-tuning di reti convoluzionali pre-addestrate supera sostanzialmente il training da zero.

Affronta il problema del transfer learning nelle deep network e propone di avere un termine di regolarizzazione che penalizza la divergenza dall'inizializzazione.

Propone un'analisi su diverse tecniche di regolarizzazione adattiva per il deep transfer learning, concentrandosi in particolare sull'uso di una condizione L@-SP


Artificial neural networks have opened up a world of possibilities in data science and artificial intelligence, but neural networks are cumbersome tools that grow with the complexity of the learning problem. We make contributions to this issue by considering a modified version of the fully connected layer we call a block diagonal inner product layer. These modified layers have weight matrices that are block diagonal, turning a single fully connected layer into a set of densely connected neuron groups. This idea is a natural extension of group, or depthwise separable, convolutional layers applied to the fully connected layers. Block diagonal inner product layers can be achieved by either initializing a purely block diagonal weight matrix or by iteratively pruning off diagonal block entries. This method condenses network storage and speeds up the run time without significant adverse effect on the testing accuracy, thus offering a new approach to improve network computation efficiency.

Studiamo le reti neurali con layer di prodotti interni diagonali a blocchi per l'efficienza.

Questo articolo propone che i layer interni di una rete neurale siano diagonali a blocchi, e discute sul fatto che le matrici diagonali a blocchi siano più efficienti del pruning e che i layer diagonali a blocchi portino a reti più efficienti.

Sostituzione di layer fully connected con layer fully connected block-diagonal


One of the challenges in the study of generative adversarial networks is the instability of its training.   In this paper, we propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator.  Our new normalization technique is computationally light and easy to incorporate into existing implementations.   We tested the efficacy of spectral normalization on CIFAR10, STL-10, and ILSVRC2012 dataset, and we experimentally confirmed that spectrally normalized GANs (SN-GANs) is capable of generating images of better or equal quality relative to the previous training stabilization techniques.

Proponiamo una nuova tecnica di normalizzazione dei pesi chiamata normalizzazione spettrale per stabilizzare il training del discriminatore delle GAN.

Questo articolo usa la regolarizzazione spettrale per normalizzare gli obiettivi GAN, e la GAN che ne risulta, chiamato SN-GAN, assicura essenzialmente la proprietà di Lipschitz del discriminatore.

Questo articolo propone la "normalizzazione spettrale", facendo un bel passo in avanti nel miglioramento del training delle GAN.


Humans acquire complex skills by exploiting previously learned skills and making transitions between them. To empower machines with this ability, we propose a method that can learn transition policies which effectively connect primitive skills to perform sequential tasks without handcrafted rewards. To efficiently train our transition policies, we introduce proximity predictors which induce rewards gauging proximity to suitable initial states for the next skill. The proposed method is evaluated on a set of complex continuous control tasks in bipedal locomotion and robotic arm manipulation which traditional policy gradient methods struggle at. We demonstrate that transition policies enable us to effectively compose complex skills with existing primitive skills. The proposed induced rewards computed using the proximity predictor further improve training efficiency by providing more dense information than the sparse rewards from the environments. We make our environments, primitive skills, and code public for further research at https://youngwoon.github.io/transition .

Le policy di transizione permettono agli agenti di comporre skill complesse collegando in modo fluido le skill primitive precedentemente acquisite.

Propone uno schema per la transizione a stati di strating favorevoli per l'esecuzione di opzioni date in domini continui. Utilizza due processi di apprendimento eseguiti simultaneamente.

Presenta un metodo per l'apprendimento delle policy di transizione da un task ad un altro con l'obiettivo di completare task complessi utilizzando lo stimatore di prossimità di stato per premiare la policy di transizione.

Propone un nuovo schema di training con una funzione di reward ausiliaria appresa per ottimizzare le policy di transizione che collegano lo stato finale di una precedente macro azione/opzione con buoni stati iniziali della seguente macro azione/opzione


Gated recurrent units (GRUs) were inspired by the common gated recurrent unit, long short-term memory (LSTM), as a means of capturing temporal structure with less complex memory unit architecture. Despite their incredible success in tasks such as natural and artificial language processing, speech, video, and polyphonic music, very little is understood about the specific dynamic features representable in a GRU network. As a result, it is difficult to know a priori how successful a GRU-RNN will perform on a given data set. In this paper, we develop a new theoretical framework to analyze one and two dimensional GRUs as a continuous dynamical system, and classify the dynamical features obtainable with such system.  We found rich repertoire that includes stable limit cycles over time (nonlinear oscillations), multi-stable state transitions with various topologies, and homoclinic orbits. In addition, we show that any finite dimensional GRU cannot precisely replicate the dynamics of a ring attractor, or more generally, any continuous attractor, and is limited to finitely many isolated fixed points in theory. These findings were then experimentally verified in two dimensions by means of time series prediction.

Classifichiamo le feature dinamiche che una e due celle GRU possono e non possono catturare in tempo continuo, e verifichiamo sperimentalmente i nostri risultati con la predizione di serie temporali in k-step.

Gli autori analizzano le GRU con hidden size di uno e due come sistemi dinamici a tempo continuo, sostenendo che la potenza espressiva della rappresentazione degli hidden state può fornire una conoscenza preliminare su quanto bene una GRU si comporterà su un dato dataset

Questo articolo analizza le GRU da una prospettiva di sistemi dinamici, e mostra che le GRU 2d possono essere addestrate ad adottare una varietà di punti fissi e possono approssimare gli attrattori di linea, ma non possono imitare un attrattore ad anello.

Converte le equazioni di GRU in tempo continuo e usa la teoria e gli esperimenti per studiare reti GRU a 1 e 2 dimensioni e mostrare ogni varietà di topologia dinamica disponibile in questi sistemi


Stacked hourglass network has become an important model for Human pose estimation. The estimation of human body posture depends on the global information of the keypoints type and the local information of the keypoints location. The consistent processing of inputs and constraints makes it difficult to form differentiated and determined collaboration mechanisms for each stacked hourglass network. In this paper, we propose a Multi-Scale Stacked Hourglass (MSSH) network to high-light the differentiation capabilities of each Hourglass network for human pose estimation.   The pre-processing network forms feature maps of different scales,and dispatch them to various locations of the stack hourglass network, where the small-scale features reach the front of stacked hourglass network, and large-scale features reach the rear of stacked hourglass network.    And a new loss function is proposed for multi-scale stacked hourglass network.   Different keypoints have different weight coefficients of loss function at different scales, and the keypoints weight coefficients are dynamically adjusted from the top-level hourglass network to the bottom-level hourglass network.   Experimental results show that the pro-posed method is competitive with respect to the comparison algorithm on MPII and LSP datasets.

Gli input differenziati causano una differenziazione funzionale della rete, e l'interazione delle funzioni di loss tra le reti può influenzare il processo di ottimizzazione.

Una modifica alla rete originale a clessidra per la single pose estimation che produce miglioramenti rispetto alla baseline originale.

Gli autori estendono una stacked hourglass network con moduli inception-resnet-A e propongono un approccio multiscala per la stima della posa umana in immagini RGB fisse.


We present a new unsupervised method for learning general-purpose sentence embeddings.  Unlike existing methods which rely on local contexts, such as words  inside the sentence or immediately neighboring sentences, our method selects, for  each target sentence, influential sentences in the entire document based on a document  structure. We identify a dependency structure of sentences using metadata  or text styles. Furthermore, we propose a novel out-of-vocabulary word handling  technique to model many domain-specific terms, which were mostly discarded by  existing sentence embedding methods. We validate our model on several tasks  showing 30% precision improvement in coreference resolution in a technical domain,  and 7.5% accuracy increase in paraphrase detection compared to baselines.

Per addestrare un modello di sentence embedding utilizzando documenti tecnici, il nostro approccio considera la struttura del documento per trovare un contesto più ampio e gestire le parole fuori dal vocabolario.

Presenta idee per migliorare i sentence embedding attingendo a più contesti.

Apprendimento di rappresentazioni di frasi con informazioni sulle dipendenze delle frasi

Estende l'idea di formare una rappresentazione non supervisionata delle frasi usata nell'approccio SkipThough utilizzando un insieme più ampio di prove per formare la rappresentazione di una frase


Neural network training relies on our ability to find ````````"good" minimizers of highly non-convex loss functions. It is well known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effect on the underlying loss landscape, is not well understood.   In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple ``"filter normalization" method that helps us visualize loss function curvature, and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture effects the loss landscape, and how training parameters affect the shape of minimizers.

Esploriamo la struttura delle funzioni di loss neurali e l'effetto dei landscape delle loss sulla generalizzazione, usando una serie di metodi di visualizzazione.

Questo articolo propone un metodo per visualizzare la loss di una NN e fornisce approfondimenti sull'addestrabilità e la generalizzazione delle NN.

Studia la non convessità della superficie di loss e dei percorsi di ottimizzazione.


Deep models are state-of-the-art for many computer vision tasks including image classification and object detection. However, it has been shown that deep models are vulnerable to adversarial examples. We highlight how one-hot encoding directly contributes to this vulnerability and propose breaking away from this widely-used, but highly-vulnerable mapping. We demonstrate that by leveraging a different output encoding, multi-way encoding, we can make models more robust. Our approach makes it more difficult for adversaries to find useful gradients for generating adversarial attacks. We present state-of-the-art robustness results for black-box, white-box attacks, and achieve higher clean accuracy on four benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN when combined with adversarial training. The strength of our approach is also presented in the form of an attack for model watermarking, raising challenges in detecting stolen models.

Dimostriamo che sfruttando una codifica dell'output a più vie, piuttosto che la codifica one-hot ampiamente utilizzata, possiamo rendere i deep model più robusti agli adversarial attack.

Questo articolo propone di sostituire il layer finale di cross-entropy addestrato sulle label one-hot nei classificatori codificando ogni label come un vettore ad alta densità e addestrando il classificatore per minimizzare la distanza L2 dalla codifica della classe corretta.

Gli autori propongono un nuovo metodo contro gli adversarial attack che mostra una quantità significativa di guadagni rispetto alle baseline


Existing approaches to neural machine translation condition each output word on previously generated outputs. We introduce a model that avoids this autoregressive property and produces its outputs in parallel, allowing an order of magnitude lower latency during inference. Through knowledge distillation, the use of input token fertilities as a latent variable, and policy gradient fine-tuning, we achieve this at a cost of as little as 2.0 BLEU points relative to the autoregressive Transformer network used as a teacher. We demonstrate substantial cumulative improvements associated with each of the three aspects of our training strategy, and validate our approach on IWSLT 2016 English–German and two WMT language pairs. By sampling fertilities in parallel at inference time, our non-autoregressive model achieves near-state-of-the-art performance of 29.8 BLEU on WMT 2016 English–Romanian.

Introduciamo il primo modello NMT con decodifica completamente parallela, riducendo la latenza di inferenza di 10 volte.

Questo lavoro propone un decoder non autoregressivo per il framework encoder-decoder in cui la decisione di generare una parola non dipende dalle decisioni precedenti sulle parole generate

Questo articolo descrive un approccio alla decodifica non autoregressiva per la traduzione automatica neurale con la possibilità di una decodifica più parallela che può risultare in un significativo aumento della velocità.

Propone l'introduzione di un insieme di variabili latenti per rappresentare la fertilità di ogni parola sorgente per rendere la generazione della frase target non autoregressiva


While neural networks have achieved high accuracy on standard image classification benchmarks, their accuracy drops to nearly zero in the presence of small adversarial perturbations to test inputs. Defenses based on regularization and adversarial training have been proposed, but often followed by new, stronger attacks that defeat these defenses. Can we somehow end this arms race? In this work, we study this problem for neural networks with one hidden layer. We first propose a method based on a semidefinite relaxation that outputs a certificate that for a given network and test input, no attack can force the error to exceed a certain value. Second, as this certificate is differentiable, we jointly optimize it with the network parameters, providing an adaptive regularizer that encourages robustness against all attacks. On MNIST, our approach produces a network and a certificate that no that perturbs each pixel by at most $\epsilon = 0.1$ can cause more than $35\%$ test error.

Dimostriamo un metodo certificabile, addestrabile e scalabile per difendersi dagli adversarial example.

Propone una nuova difesa contro gli attacchi di sicurezza alle reti neurali con il modello attaccante che produce un certificato di sicurezza sull'algoritmo.

Deriva un limite superiore all'adversarial perturbation per le reti neurali con un hidden layer


We formulate an information-based optimization problem for supervised classification. For invertible neural networks, the control of these information terms is passed down to the latent features and parameter matrix in the last fully connected layer, given that mutual information is invariant under invertible map.   We propose an objective function and prove that it solves the optimization problem. Our framework allows us to learn latent features in an more interpretable form while improving the classification performance. We perform extensive quantitative and qualitative experiments in comparison with the existing state-of-the-art classification models.

Proponiamo un regolarizzatore che migliora le prestazioni di classificazione delle reti neurali

Gli autori propongono di addestrare un modello da un punto di massimizzazione dell'informazione reciproca tra le predizioni e gli output veri, con un termine di regolarizzazione che minimizza l'informazione irrilevante durante l'apprendimento.

Propone di decomporre i parametri in una mappa di feature invertibile F e una trasformazione lineare w nell'ultimo layer per massimizzare l'informazione reciproca I(Y, \hat{T}) vincolando l'informazione irrilevante


Powerful generative models, particularly in Natural Language Modelling, are commonly trained by maximizing a variational lower bound on the data log likelihood. These models often suffer from poor use of their latent variable, with ad-hoc annealing factors used to encourage retention of information in the latent variable. We discuss an alternative and general approach to latent variable modelling, based on an objective that encourages a perfect reconstruction by tying a stochastic autoencoder with a variational autoencoder (VAE). This ensures by design that the latent variable captures information about the observations, whilst retaining the ability to generate well. Interestingly, although our model is fundamentally different to a VAE, the lower bound attained is identical to the standard VAE bound but with the addition of a simple pre-factor; thus, providing a formal interpretation of the commonly used, ad-hoc pre-factors in training VAEs.

Questo articolo introduce un nuovo framework di modellazione generativa che evita il collasso delle variabili latenti e chiarisce l'uso di alcuni fattori ad-hoc nel training degli autoencoder variazionali.

L'articolo propone di risolvere il problema di un auto-encoder variazionale che ignora le variabili latenti.

Questo articolo propone di aggiungere un autoencoder stocastico al modello VAE originale per affrontare il problema che il decoder LSTM di un language model potrebbe essere troppo forte per ignorare le informazioni della variabile latente.

Questo articolo presenta AutoGen, che combina un autoencoder variazionale generativo con un modello di ricostruzione ad alta fedeltà basato su autoencoder per utilizzare meglio la rappresentazione latente


This paper studies the problem of domain division which aims to segment instances drawn from different probabilistic distributions. This problem exists in many previous recognition tasks, such as Open Set Learning (OSL) and Generalized Zero-Shot Learning (G-ZSL), where the testing instances come from either seen or unseen/novel classes with different probabilistic distributions. Previous works only calibrate the conﬁdent prediction of classiﬁers of seen classes (WSVM Scheirer et al. (2014)) or taking unseen classes as outliers Socher et al. (2013). In contrast, this paper proposes a probabilistic way of directly estimating and ﬁne-tuning the decision boundary between seen and unseen classes. In particular, we propose a domain division algorithm to split the testing instances into known, unknown and uncertain domains, and then conduct recognition tasks in each domain. Two statistical tools, namely, bootstrapping and KolmogorovSmirnov (K-S) Test, for the ﬁrst time, are introduced to uncover and ﬁne-tune the decision boundary of each domain. Critically, the uncertain domain is newly introduced in our framework to adopt those instances whose domain labels cannot be predicted conﬁdently. Extensive experiments demonstrate that our approach achieved the state-of-the-art performance on OSL and G-ZSL benchmarks.

Questo articolo studia il problema della divisione del dominio segmentando le istanze tratte da diverse distribuzioni di probabilità.

Questo articolo si occupa del problema del riconoscimento della novità nell'open set learning e nello zero-shot learning generalizzato e propone una possibile soluzione

Un approccio alla separazione dei domini basato sul bootstrapping per identificare le soglie di cutoff di somiglianza per le classi note, seguito da un test di Kolmogorov-Smirnoff per raffinare le zone in-distribution bootstrapped.

Propone di introdurre un nuovo dominio, il dominio incerto, per gestire meglio la divisione tra domini visti/non visti nell'open set learning e nel zero-shot learning generalizzato


Stochastic gradient descent (SGD) is widely believed to perform implicit regularization when used to train deep neural networks, but the precise manner in which this occurs has thus far been elusive. We prove that SGD minimizes an average potential over the posterior distribution of weights along with an entropic regularization term. This potential is however not the original loss function in general. So SGD does perform variational inference, but for a different loss than the one used to compute the gradients. Even more surprisingly, SGD does not even converge in the classical sense: we show that the most likely trajectories of SGD for deep networks do not behave like Brownian motion around critical points. Instead, they resemble closed loops with deterministic components. We prove that such out-of-equilibrium behavior is a consequence of highly non-isotropic gradient noise in SGD; the covariance matrix of mini-batch gradients for deep networks has a rank as small as 1% of its dimension. We provide extensive empirical validation of these claims, proven in the appendix.

SGD esegue implicitamente l'inferenza variazionale; il rumore del gradiente è altamente non isotropo, quindi SGD non converge nemmeno ai punti critici della loss originale

Questo articolo fornisce un'analisi variazionale di SGD come processo di non-equilibrio.

Questo articolo discute la funzione obiettivo regolarizzata minimizzata da SGD standard nel contesto delle reti neurali, e fornisce una prospettiva di inferenza variazionale utilizzando l'equazione di Fokker-Planck.

Sviluppa una teoria per studiare l'impatto del rumore del gradiente stocastico per SGD, specialmente per i modelli di deep neural network


The current dominant paradigm for imitation learning relies on strong supervision of expert actions to learn both 'what' and 'how' to imitate. We pursue an alternative paradigm wherein an agent first explores the world without any expert supervision and then distills its experience into a goal-conditioned skill policy with a novel forward consistency loss. In our framework, the role of the expert is only to communicate the goals (i.e., what to imitate) during inference. The learned policy is then employed to mimic the expert (i.e., how to imitate) after seeing just a sequence of images demonstrating the desired task. Our method is 'zero-shot' in the sense that the agent never has access to expert actions during training or for the task demonstration at inference. We evaluate our zero-shot imitator in two real-world settings: complex rope manipulation with a Baxter robot and navigation in previously unseen office environments with a TurtleBot. Through further experiments in VizDoom simulation, we provide evidence that better mechanisms for exploration lead to learning a more capable policy which in turn improves end task performance. Videos, models, and more details are available at https://pathak22.github.io/zeroshot-imitation/.

Gli agenti possono imparare a imitare solo dimostrazioni visive (senza azioni) al momento del test dopo aver imparato dalla propria esperienza senza alcuna forma di supervisione al momento del training.

Questo articolo propone un approccio per visual learning zero-shot tramite l'apprendimento di skill function parametriche.

Un articolo sull'imitazione di un task presentato proprio durante l'inferenza, dove l'apprendimento viene eseguito in modo self-supervised e durante il training l'agente esplora task correlati ma diversi.

Propone un metodo per aggirare il problema della costose dimostrazioni da parte di esperti utilizzando l'esplorazione casuale di un agente per imparare abilità generalizzabili che possono essere applicate senza uno specifico pretraining


Distributional Semantics Models(DSM) derive word space from linguistic items  in context. Meaning is obtained by defining a distance measure between vectors  corresponding to lexical entities. Such vectors present several problems. This  work concentrates on quality of word embeddings, improvement of word embedding  vectors, applicability of a novel similarity metric used ‘on top’ of the  word embeddings. In this paper we provide comparison between two methods  for post process improvements to the baseline DSM vectors. The counter-fitting  method which enforces antonymy and synonymy constraints into the Paragram  vector space representations recently showed improvement in the vectors’ capability  for judging semantic similarity. The second method is our novel RESM  method applied to GloVe baseline vectors. By applying the hubness reduction  method, implementing relational knowledge into the model by retrofitting synonyms  and providing a new ranking similarity definition RESM that gives maximum  weight to the top vector component values we equal the results for the ESL  and TOEFL sets in comparison with our calculations using the Paragram and Paragram  + Counter-fitting methods. For SIMLEX-999 gold standard since we cannot  use the RESM the results using GloVe and PPDB are significantly worse compared  to Paragram. Apparently, counter-fitting corrects hubness. The Paragram  or our cosine retrofitting method are state-of-the-art results for the SIMLEX-999  gold standard. They are 0.2 better for SIMLEX-999 than word2vec with sense  de-conflation (that was announced to be state-of the-art method for less reliable  gold standards). Apparently relational knowledge and counter-fitting is more important  for judging semantic similarity than sense determination for words. It is to  be mentioned, though that Paragram hyperparameters are fitted to SIMLEX-999  results. The lesson is that many corrections to word embeddings are necessary  and methods with more parameters and hyperparameters perform better.

L'articolo fornisce una descrizione di una procedura per migliorare il modello di spazio vettoriale delle parole con una valutazione dei modelli Paragram e GloVe per i benchmark di similarità.

Questo articolo suggerisce un nuovo algoritmo che regola i word vector di GloVe e poi usa una funzione di somiglianza non euclidea tra loro.

Gli autori presentano osservazioni sulle debolezze dei modelli di spazio vettoriale esistenti ed elencano un approccio in 6 fasi per raffinare i word vector esistenti


Recurrent neural networks have achieved excellent performance in many applications. However, on portable devices with limited resources, the models are often too large to deploy. For applications on the server with large scale concurrent requests, the latency during inference can also be very critical for costly computing resources. In this work, we address these problems by quantizing the network, both weights and activations, into multiple binary codes {-1,+1}. We formulate the quantization as an optimization problem. Under the key observation that once the quantization coefficients are fixed the binary codes can be derived efficiently by binary search tree, alternating minimization is then applied.   We test the quantization for two well-known RNNs, i.e., long short term memory (LSTM) and gated recurrent unit (GRU), on the language models. Compared with the full-precision counter part, by 2-bit quantization we can achieve ~16x memory saving and  ~6x real inference acceleration on CPUs, with only a reasonable loss in the accuracy. By 3-bit quantization, we can achieve almost no loss in the accuracy or even surpass the original model, with ~10.5x memory saving and ~3x real inference acceleration. Both results beat the exiting quantization works with large margins.   We extend our alternating quantization to image classification tasks. In both RNNs and feedforward neural networks, the method also achieves  excellent performance.

Proponiamo un nuovo metodo di quantizzazione e lo applichiamo per quantizzare le RNN sia per la compressione che per l'accelerazione

Questo articolo propone un metodo di quantizzazione multi-bit per reti neurali ricorrenti.

Una tecnica per quantizzare le matrici di peso delle reti neurali e una procedura di ottimizzazione alternata per stimare l'insieme di k vettori binari e coefficienti che rappresentino meglio il vettore originale.