We demonstrate how to learn efficient heuristics for automated reasoning algorithms through deep reinforcement learning. We focus on backtracking search algorithms for quantified Boolean logics, which already can solve formulas of impressive size - up to 100s of thousands of variables. The main challenge is to find a representation of these formulas that lends itself to making predictions in a scalable way. For challenging problems, the heuristic learned through our approach reduces execution time by a factor of 10 compared to the existing handwritten heuristics.

Il RL trova migliori euristiche per algoritmi di ragionamento automatico.

Mira ad apprendere un'euristica per un algoritmo di ricerca backtracking utilizzando il Reinforcement learning e propone un modello che fa uso di Graph Neural Network per produrre l'embedding di letterali e clausole, e li usa per prevedere la qualità di ogni letterale per decidere la probabilità di ogni azione.

L'articolo propone un approccio all'apprendimento automatico di euristiche di selezione delle variabili per QBF utilizzando il deep learning


We consider the question of how to assess generative adversarial networks, in particular with respect to whether or not they generalise beyond memorising the training data. We propose a simple procedure for assessing generative adversarial network performance based on a principled consideration of what the actual goal of generalisation is. Our approach involves using a test set to estimate the Wasserstein distance between the generative distribution produced by our procedure, and the underlying data distribution. We use this procedure to assess the performance of several modern generative adversarial network architectures. We find that this procedure is sensitive to the choice of ground metric on the underlying data space, and suggest a choice of ground metric that substantially improves performance.   We finally suggest that attending to the ground metric used in Wasserstein generative adversarial network training may be fruitful, and outline a concrete pathway towards doing so.

Valutate se la vostra GAN sta effettivamente facendo qualcosa di diverso dalla memorizzazione dei dati di training.

Mira a fornire una misura/test di qualità per le GAN e propone di valutare l'approssimazione attuale di una distribuzione appresa da una GAN utilizzando la distanza di Wasserstein tra due distribuzioni composte da una somma di Dirac come baseline di performance.

Questo articolo ha proposto una procedura per valutare le prestazioni delle GAN riconsiderando la chiave di osservazione, utilizzando la procedura per testare e migliorare le GAN attuali


The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various hand-designed alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose to leverage automatic search techniques to discover new activation functions. Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions. We verify the effectiveness of the searches by conducting an empirical evaluation with the best discovered activation function. Our experiments show that the best discovered activation function, f(x) = x * sigmoid(beta * x), which we name Swish, tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.

Usiamo tecniche di ricerca per scoprire nuove funzioni di attivazione, e la nostra migliore funzione di attivazione scoperta, f(x) = x * sigmoid(beta * x), supera ReLU su una serie di task impegnativi come ImageNet.

Propone un approccio basato sul reinforcement learning per trovare le non linearità cercando attraverso combinazioni da un insieme di operatori unari e binari.

Questo articolo utilizza reinforcement learning per cercare la combinazione di un insieme di funzioni unarie e binarie che risultino in una nuova funzione di attivazione

L'autore usa il reinforcement learning per trovare nuove potenziali funzioni di attivazione da un ricco insieme di possibili candidati.


Successful training of convolutional neural networks is often associated with suffi-  ciently deep architectures composed of high amounts of features. These networks  typically rely on a variety of regularization and pruning techniques to converge  to less redundant states. We introduce a novel bottom-up approach to expand  representations in fixed-depth architectures. These architectures start from just a  single feature per layer and greedily increase width of individual layers to attain  effective representational capacities needed for a specific task. While network  growth can rely on a family of metrics, we propose a computationally efficient  version based on feature time evolution and demonstrate its potency in determin-  ing feature importance and a networks’ effective capacity. We demonstrate how  automatically expanded architectures converge to similar topologies that benefit  from lesser amount of parameters or improved accuracy and exhibit systematic  correspondence in representational complexity with the specified task. In contrast  to conventional design patterns with a typical monotonic increase in the amount of  features with increased depth, we observe that CNNs perform better when there is  more learnable parameters in intermediate, with falloffs to earlier and later layers.

Un algoritmo bottom-up che espande le CNN a partire da una feature per layer fino ad architetture con sufficiente capacità di rappresentazione.

Propone di regolare dinamicamente la profondità della feature map di una rete neurale completamente convoluzionale, formulando una misura di autosomiglianza e aumentando le prestazioni.

Introduce una semplice metrica basata sulla correlazione per misurare se i filtri nelle reti neurali sono usati efficacemente, come proxy della capacità effettiva.

Mira ad affrontare il problema della ricerca dell'architettura di deep learning attraverso l'aggiunta e la rimozione incrementale di canali nei layer intermedi della rete.


Deep neural networks are almost universally trained with reverse-mode automatic differentiation (a.k.a. backpropagation). Biological networks, on the other hand, appear to lack any mechanism for sending gradients back to their input neurons, and thus cannot be learning in this way. In response to this, Scellier & Bengio (2017) proposed Equilibrium Propagation - a method for gradient-based train- ing of neural networks which uses only local learning rules and, crucially, does not rely on neurons having a mechanism for back-propagating an error gradient. Equilibrium propagation, however, has a major practical limitation: inference involves doing an iterative optimization of neural activations to find a fixed-point, and the number of steps required to closely approximate this fixed point scales poorly with the depth of the network. In response to this problem, we propose Initialized Equilibrium Propagation, which trains a feedforward network to initialize the iterative inference procedure for Equilibrium propagation. This feed-forward network learns to approximate the state of the fixed-point using a local learning rule. After training, we can simply use this initializing network for inference, resulting in a learned feedforward network. Our experiments show that this network appears to work as well or better than the original version of Equilibrium propagation. This shows how we might go about training deep networks without using backpropagation.

Addestriamo una rete feedforward senza backprop utilizzando un modello basato sull'energia per fornire obiettivi locali

Questo articolo mira a velocizzare la procedura di inferenza iterativa nei modelli basati sull'energia addestrati con Equilibrium Propagation (EP), proponendo di addestrare una rete feedforward per prevedere un punto fisso della "rete equilibrante".

Addestrare una rete separata per inizializzare le reti ricorrenti addestrate utilizzando la propagazione dell'equilibrio


We propose a novel generative model architecture designed to learn representations for images that factor out a single attribute from the rest of the representation. A single object may have many attributes which when altered do not change the identity of the object itself. Consider the human face; the identity of a particular person is independent of whether or not they happen to be wearing glasses. The attribute of wearing glasses can be changed without changing the identity of the person. However, the ability to manipulate and alter image attributes without altering the object identity is not a trivial task. Here, we are interested in learning a representation of the image that separates the identity of an object (such as a human face) from an attribute (such as 'wearing glasses'). We demonstrate the success of our factorization approach by using the learned representation to synthesize the same face with and without a chosen attribute. We refer to this specific synthesis process as image attribute manipulation. We further demonstrate that our model achieves competitive scores, with state of the art, on a facial attribute classification task.

Imparare le rappresentazioni per le immagini che possono fattorizzare un singolo attributo.

Questo paper si basa su GAN con VAE condizionali per consentire la manipolazione degli attributi nel processo di sintesi.

Questo articolo propone un modello generativo per imparare la rappresentazione che può separare l'identità di un oggetto da un attributo, ed estende l'adversarial autoencoder aggiungendo una rete ausiliaria.


Stochastic video prediction models take in a sequence of image frames, and generate a sequence of consecutive future image frames. These models typically generate future frames in an autoregressive fashion, which is slow and requires the input and output frames to be consecutive. We introduce a model that overcomes these drawbacks by generating a latent representation from an arbitrary set of frames that can then be used to simultaneously and efficiently sample temporally consistent frames at arbitrary time-points. For example, our model can "jump" and directly sample frames at the end of the video, without sampling intermediate frames. Synthetic video evaluations confirm substantial gains in speed and functionality without loss in fidelity. We also apply our framework to a 3D scene reconstruction dataset. Here, our model is conditioned on camera location and can sample consistent sets of images for what an occluded region of a 3D scene might look like, even if there are multiple possibilities for what that region might contain. Reconstructions and videos are available at https://bit.ly/2O4Pc4R.

Presentiamo un modello per la ricostruzione 3D coerente e la jumpy video prediction, ad esempio la produzione di frame di immagini a più passi temporali nel futuro senza generare frame intermedi.

Questo articolo propone un metodo generale per la modellazione dei dati indicizzati codificando le informazioni dell'indice insieme all'osservazione in una rete neurale, e poi decodificare la condizione di osservazione sull'indice di destinazione.

Propone di utilizzare un VAE che codifica il video in input in modo invariante alla permutazione per prevedere i futuri frame di un video.


The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn’t. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of the stochastic gradient, whereas the update magnitude is solely determined by an estimate of its relative variance. We  disentangle these two aspects and analyze them in isolation, shedding light on ADAM ’s inner workings. Transferring the "variance adaptation” to momentum- SGD gives rise to a novel method, completing the practitioner’s toolbox for problems where ADAM fails.

Si analizza il popolare ottimizzatore Adam

L'articolo cerca di migliorare Adam basandosi sull'adattamento della varianza con momentum proponendo due algoritmi

Questo articolo analizza l'invarianza di scala e la forma particolare del learning rate utilizzato in Adam, sostenendo che l'aggiornamento di Adam è una combinazione di un sign-update e un learning rate basato sulla varianza.

L'articolo divide l'algoritmo ADAM in due componenti: direzione stocastica nel segno del gradiente e step size adattiva con varianza relativa, e due algoritmi vengono proposti per testare ciascuno di essi.


We propose a novel framework to adaptively adjust the dropout rates for the deep neural network based on a Rademacher complexity bound. The state-of-the-art deep learning algorithms impose dropout strategy to prevent feature co-adaptation. However, choosing the dropout rates remains an art of heuristics or relies on empirical grid-search over some hyperparameter space. In this work, we show the network Rademacher complexity is bounded by a function related to the dropout rate vectors and the weight coefficient matrices. Subsequently, we impose this bound as a regularizer and provide a theoretical justified way to trade-off between model complexity and representation power. Therefore, the dropout rates and the empirical loss are unified into the same objective function, which is then optimized using the block coordinate descent algorithm. We discover that the adaptively adjusted dropout rates converge to some interesting distributions that reveal meaningful patterns.Experiments on the task of image and document classification also show our method achieves better performance compared to the state-of the-art dropout algorithms.

Proponiamo un nuovo framework per regolare in modo adattivo i dropout rate per le deep neural network basato su un bound di complessità di Rademacher.

Gli autori collegano i parametri di dropout a un bound della complessità di Rademacher della rete

Mette in relazione la complessità dell'apprendibilità delle reti con i dropout rate nella backpropagation.


Sensor fusion is a key technology that integrates various sensory inputs to allow for robust decision making in many applications such as autonomous driving and robot control. Deep neural networks have been adopted for sensor fusion in a body of recent studies. Among these, the so-called netgated architecture was proposed, which has demonstrated improved performances over the conventional convolu- tional neural networks (CNN). In this paper, we address several limitations of the baseline negated architecture by proposing two further optimized architectures: a coarser-grained gated architecture employing (feature) group-level fusion weights and a two-stage gated architectures leveraging both the group-level and feature- level fusion weights. Using driving mode prediction and human activity recogni- tion datasets, we demonstrate the significant performance improvements brought by the proposed gated architectures and also their robustness in the presence of sensor noise and failures.

Vengono proposte architetture di deep learning gated ottimizzate per la fusione dei sensori.

Gli autori migliorano diverse limitazioni della baseline dell'architettura negata proponendo un'architettura di fusione gated più grossolana e un'architettura di fusione gated a due stadi

Propone due architetture di deep learning gated per la fusione dei sensori e, avendo le feature raggruppate, dimostra prestazioni migliori, soprattutto in presenza di rumore casuale dei sensori e di guasti.


We develop a mean field theory for batch normalization in fully-connected feedforward neural networks. In so doing, we provide a precise characterization of signal propagation and gradient backpropagation in wide batch-normalized networks at initialization. Our theory shows that gradient signals grow exponentially in depth and that these exploding gradients cannot be eliminated by tuning the initial weight variances or by adjusting the nonlinear activation function. Indeed, batch normalization itself is the cause of gradient explosion. As a result, vanilla batch-normalized networks without skip connections are not trainable at large depths for common initialization schemes, a prediction that we verify with a variety of empirical simulations. While gradient explosion cannot be eliminated, it can be reduced by tuning the network close to the linear regime, which improves the trainability of deep batch-normalized networks without residual connections. Finally, we investigate the learning dynamics of batch-normalized networks and observe that after a single step of optimization the networks achieve a relatively stable equilibrium in which gradients have dramatically smaller dynamic range. Our theory leverages Laplace, Fourier, and Gegenbauer transforms and we derive new identities that may be of independent interest.

La batch normalization causa l'esplosione dei gradienti nelle reti feedforward standard.

Sviluppa una teoria del campo medio per la batch normalization (BN) in reti completamente connesse con pesi inizializzati in modo casuale.

Fornisce una prospettiva dinamica sulle deep neural network utilizzando l'evoluzione della matrice di covarianza insieme ai layer.


We present NeuroSAT, a message passing neural network that learns to solve SAT problems after only being trained as a classifier to predict satisfiability.   Although it is not competitive with state-of-the-art SAT solvers, NeuroSAT can solve problems that are substantially larger and more difficult than it ever saw during training by simply running for more iterations. Moreover, NeuroSAT generalizes to novel distributions; after training only on random SAT problems, at test time it can solve SAT problems encoding graph coloring, clique detection, dominating set, and vertex cover problems, all on a range of distributions over small random graphs.

Addestriamo una graph network per prevedere la soddisfacibilità booleana e mostriamo che impara a cercare soluzioni, e che le soluzioni che trova possono essere decodificate dalle sue attivazioni.

L'articolo descrive un'architettura generale di rete neurale per prevedere la soddisfacibilità

Questo articolo presenta l'architettura NeuroSAT che utilizza una deep neural network con message passing per prevedere la soddisfacibilità delle istanze CNF


Spatiotemporal forecasting has various applications in neuroscience, climate and transportation domain. Traffic forecasting is one canonical example of such learning task. The task is challenging due to (1) complex spatial dependency on road networks, (2) non-linear temporal dynamics with changing road conditions and (3) inherent difficulty of long-term forecasting. To address these challenges, we propose to model the traffic flow as a diffusion process on a directed graph and introduce Diffusion Convolutional Recurrent Neural Network (DCRNN), a deep learning framework for traffic forecasting that incorporates both spatial and temporal dependency in the traffic flow. Specifically, DCRNN captures the spatial dependency using bidirectional random walks on the graph, and the temporal dependency using the encoder-decoder architecture with scheduled sampling. We evaluate the framework on two real-world large-scale road network traffic datasets and observe consistent improvement of 12% - 15% over state-of-the-art baselines

Un modello neurale per sequenze che impara a prevedere su un grafo diretto.

L'articolo propone l'architettura Diffusion Convolutional Recurrent Neural Network per il problema della predizione del traffico spazio-temporale

Propone di costruire un modello di predizione del traffico usando un processo di diffusione per reti neurali ricorrenti convoluzionali per affrontare l'autocorrelazione spazio-temporale.


Obtaining reliable uncertainty estimates of neural network predictions is a long standing challenge. Bayesian neural networks have been proposed as a solution, but it remains open how to specify their prior. In particular, the common practice of a standard normal prior in weight space imposes only weak regularities, causing the function posterior to possibly generalize in unforeseen ways on inputs outside of the training distribution. We propose noise contrastive priors (NCPs) to obtain reliable uncertainty estimates. The key idea is to train the model to output high uncertainty for data points outside of the training distribution. NCPs do so using an input prior, which adds noise to the inputs of the current mini batch, and an output prior, which is a wide distribution given these inputs. NCPs are compatible with any model that can output uncertainty estimates, are easy to scale, and yield reliable uncertainty estimates throughout training. Empirically, we show that NCPs prevent overfitting outside of the training distribution and result in uncertainty estimates that are useful for active learning. We demonstrate the scalability of our method on the flight delays data set, where we significantly improve upon previously published results.

Addestriamo le reti neurali ad essere incerte su input rumorosi per evitare predizioni troppo sicure al di fuori della distribuzione di training.

Presenta un approccio per ottenere stime di incertezza per le predizioni delle reti neurali che ha buone prestazioni quando si quantifica l'incertezza predittiva in punti che sono al di fuori della distribuzione di training.

L'articolo considera il problema della stima dell'incertezza delle reti neurali e propone di utilizzare l'approccio bayesiano con prior noise contrastive


Convolutional neural networks (CNNs) were inspired by human vision and, in some settings, achieve a performance comparable to human object recognition. This has lead to the speculation that both systems use similar mechanisms to perform recognition. In this study, we conducted a series of simulations that indicate that there is a fundamental difference between human vision and CNNs: while object recognition in humans relies on analysing shape, CNNs do not have such a shape-bias. We teased apart the type of features selected by the model by modifying the CIFAR-10 dataset so that, in addition to containing objects with shape, the images concurrently contained non-shape features, such as a noise-like mask. When trained on these modified set of images, the model did not show any bias towards selecting shapes as features. Instead it relied on whichever feature allowed it to perform the best prediction -- even when this feature was a noise-like mask or a single predictive pixel amongst 50176 pixels. We also found that regularisation methods, such as batch normalisation or Dropout, did not change this behaviour and neither did past or concurrent experience with images from other datasets.

Questo studio mette in evidenza una differenza chiave tra la visione umana e le CNN: mentre il riconoscimento degli oggetti negli esseri umani si basa sull'analisi della forma, le CNN non hanno una tale bias per la forma.

Cerca di stabilire attraverso una serie di esperimenti ben progettati che le CNN addestrate per la classificazione delle immagini non codificano i bias di forma come nella visione umana.

Questo articolo mette in evidenza il fatto che le CNN non imparano necessariamente a riconoscere gli oggetti in base alla loro forma e mostra che si fidano troppo delle feature basate sul rumore.


The development of high-dimensional generative models has recently gained a great surge of interest with the introduction of variational auto-encoders and generative adversarial neural networks. Different variants have been proposed where the underlying latent space is structured, for example, based on attributes describing the data to generate. We focus on a particular problem where one aims at generating samples corresponding to a number of objects under various views. We assume that the distribution of the data is driven by two independent latent factors: the content, which represents the intrinsic features of an object, and the view, which stands for the settings of a particular observation of that object. Therefore, we propose a generative model and a conditional variant built on such a disentangled latent space. This approach allows us to generate realistic samples corresponding to various objects in a high variety of views. Unlike many multi-view approaches, our model doesn't need any supervision on the views but only on the content. Compared to other conditional generation approaches that are mostly based on binary or categorical attributes, we make no such assumption about the factors of variations. Our model can be used on problems with a huge, potentially infinite, number of categories. We experiment it on four images datasets on which we demonstrate the effectiveness of the model and its ability to generalize.

Descriviamo un nuovo modello generativo multi-view che può generare più view dello stesso oggetto, o più oggetti nella stessa view senza bisogno di label sulle view.

Questo articolo presenta un metodo basato su GAN per la generazione di immagini che cerca di separare le variabili latenti che descrivono il contenuto dell'immagine da quelle che descrivono le proprietà della view.

Questo articolo propone un'architettura GAN che mira a decomporre la distribuzione sottostante di una particolare classe in "contenuto" e "view".

Propone un nuovo modello generativo basato sulla Generative Adversarial Network (GAN) che separa il contenuto e la view degli oggetti senza supervisione della view ed estende GMV in un modello generativo condizionale che prende un'immagine di input e genera diverse view dell'oggetto nell'immagine di input.


The huge size of deep networks hinders their use in small computing devices. In this paper, we consider compressing the network by weight quantization. We extend a recently proposed loss-aware weight binarization scheme to ternarization, with possibly different scaling parameters for the positive and negative weights, and m-bit (where m > 2) quantization. Experiments on feedforward and recurrent neural networks show that the proposed scheme outperforms state-of-the-art weight quantization algorithms, and is as accurate (or even more accurate) than the full-precision network.

Viene proposto un algoritmo di quantizzazione del peso consapevole della loss che considera direttamente il suo effetto sulla loss.

Propone un metodo di compressione della rete per mezzo della ternarizzazione dei pesi.

L'articolo propone un nuovo metodo per addestrare DNN con pesi quantizzati, includendo la quantizzazione come vincolo in un algoritmo quasi-Newton prossimale, che apprende simultaneamente uno scaling per i valori quantizzati.

L'articolo estende lo schema di binarizzazione del peso consapevole della loss alla ternarizzazione e alla quantizzazione arbitraria a m-bit e dimostra le sue promettenti prestazioni.


In the pursuit of increasingly intelligent learning systems, abstraction plays a vital role in enabling sophisticated decisions to be made in complex environments. The options framework provides formalism for such abstraction over sequences of decisions.   However most models require that options be given a priori, presumably specified by hand, which is neither efficient, nor scalable. Indeed, it is preferable to learn options directly from interaction with the environment. Despite several efforts, this remains a difficult problem: many approaches require access to a model of the environmental dynamics, and inferred options are often not interpretable, which limits our ability to explain the system behavior for verification or debugging purposes.   In this work we develop a novel policy gradient method for the automatic learning of policies with options.   This algorithm uses inference methods to simultaneously improve all of the options available to an agent, and thus can be employed in an off-policy manner, without observing option labels. Experimental results show that the options learned can be interpreted. Further, we find that the method presented here is more sample efficient than existing methods, leading to faster and more stable learning of policies with options.

Sviluppiamo un nuovo metodo di policy gradient per l'apprendimento automatico di policy con opzioni utilizzando una fase di inferenza differenziabile.

L'articolo presenta una nuova tecnica di policy gradient per l'apprendimento delle opzioni, dove un singolo sample può essere usato per aggiornare tutte le opzioni.

Propone un metodo off-policy per l'apprendimento di opzioni in problemi continui complessi.


The paper, interested in unsupervised feature selection, aims to retain the features best accounting for the local patterns in the data. The proposed approach, called Locally Linear Unsupervised Feature Selection, relies on a dimensionality reduction method to characterize such patterns; each feature is thereafter assessed according to its compliance w.r.t. the local patterns, taking inspiration from Locally Linear Embedding (Roweis and Saul, 2000). The experimental validation of the approach on the scikit-feature benchmark suite demonstrates its effectiveness compared to the state of the art.

Selezione non supervisionata delle feature attraverso la cattura della struttura lineare e locale dei dati

Propone una selezione delle feature non supervisionata localmente lineare.

L'articolo propone il metodo LLUFS per la selezione delle feature.


Humans can understand and produce new utterances effortlessly, thanks to their systematic compositional skills. Once a person learns the meaning of a new verb "dax," he or she can immediately understand the meaning of "dax twice" or "sing and dax." In this paper, we introduce the SCAN domain, consisting of a set of simple compositional navigation commands paired with the corresponding action sequences. We then test the zero-shot generalization capabilities of a variety of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence methods. We find that RNNs can generalize well when the differences between training and test commands are small, so that they can apply "mix-and-match" strategies to solve the task. However, when generalization requires systematic compositional skills (as in the "dax" example above), RNNs fail spectacularly. We conclude with a proof-of-concept experiment in neural machine translation, supporting the conjecture that lack of systematicity is an important factor explaining why neural networks need very large training sets.

Utilizzando un semplice task di navigazione guidato dal linguaggio, studiamo le capacità compositive delle moderne reti ricorrenti seq2seq.

Questo articolo si concentra sulle capacità compositive di zero-shot learning delle moderne RNN sequence-to-sequence ed espone i difetti delle attuali architetture RNN seq2seq.

L'articolo analizza le capacità di composizione delle RNN, in particolare, la capacità di generalizzazione delle RNN su un sottoinsieme casuale di comandi SCAN, su comandi SCAN più lunghi, e sulla composizione di comandi primitivi.

Gli autori introducono un nuovo dataset che facilita l'analisi di un caso di apprendimento Seq2Seq


This paper addresses the challenging problem of retrieval and matching of graph structured objects, and makes two key contributions. First, we demonstrate how  Graph Neural Networks (GNN), which have emerged as an effective model for various supervised prediction problems defined on structured data, can be trained to produce embedding of graphs in vector spaces that enables efficient similarity reasoning. Second, we propose a novel Graph Matching Network model that, given a pair of graphs as input, computes a similarity score between them by jointly reasoning on the pair through a new cross-graph attention-based matching mechanism. We demonstrate the effectiveness of our models on different domains including the challenging problem of control-flow-graph based function similarity search that plays an important role in the detection of vulnerabilities in software systems. The experimental analysis demonstrates that our models are not only able to exploit structure in the context of similarity learning but they can also outperform domain-specific baseline systems that have been carefully hand-engineered for these problems.

Affrontiamo il problema del similarity learning per oggetti strutturati con applicazioni in particolare nella sicurezza informatica, e proponiamo un nuovo modello di graph matching network che eccelle in questo task.

Gli autori introducono una Graph Matching Network per il recupero e la corrispondenza di oggetti strutturati a grafo.

Gli autori attaccano il problema del matching dei grafi proponendo un'estensione delle reti di embedding dei grafi

Gli autori presentano due metodi per l'apprendimento di un punteggio di somiglianza tra coppie di grafi e mostrano l'utilità di introdurre idee dal graph matching alle graph neural network.


Context information plays an important role in human language understanding, and it is also useful for machines to learn vector representations of language. In this paper, we explore an asymmetric encoder-decoder structure for unsupervised context-based sentence representation learning. As a result, we build an encoder-decoder architecture with an RNN encoder and a CNN decoder, and we show that neither an autoregressive decoder nor an RNN decoder is required.   We further combine a suite of effective designs to significantly improve model efficiency while also achieving better performance. Our model is trained on two different large unlabeled corpora, and in both cases transferability is evaluated on a set of downstream language understanding tasks. We empirically show that our model is simple and fast while producing rich sentence representations that excel in downstream tasks.

Abbiamo proposto un modello RNN-CNN encoder-decoder per un veloce apprendimento unsupervised di sentence embedding.

Modifiche al framework di skip-thought per l'apprendimento di embedding di frasi.

Questo articolo presenta un nuovo design ibrido costituito da un encoder RNN e un decoder CNN per l'uso nel pretraining, che non richiede un decoder autoregressivo quando si fa pretraining degli encoder.

Gli autori estendono Skip-thought decodificando solo una frase target utilizzando un decoder CNN.


Building on the success of deep learning, two modern approaches to learn a probability model of the observed data are Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs). VAEs consider an explicit probability model for the data and compute a generative distribution by maximizing a variational lower-bound on the log-likelihood function. GANs, however, compute a generative model by minimizing a distance between observed and generated probability distributions without considering an explicit model for the observed data. The lack of having explicit probability models in GANs prohibits computation of sample likelihoods in their frameworks and limits their use in statistical inference problems. In this work, we show that an optimal transport GAN with the entropy regularization can be viewed as a generative model that maximizes a lower-bound on average sample likelihoods, an approach that VAEs are based on. In particular, our proof constructs an explicit probability model for GANs that can be used to compute likelihood statistics within GAN's framework. Our numerical results on several datasets demonstrate consistent trends with the proposed theory.

Un approccio statistico per calcolare le verosimiglianze del sample nelle generative adversarial network

Mostrare che WGAN con regolarizzazione entropica massimizza un lower bound sulla probabilità della distribuzione dei dati osservati.

Gli autori sostengono che è possibile sfruttare l'upper bound di un trasporto ottimale regolarizzato dall'entropia per ottenere una misura di "likelihood del sample".


We introduce geomstats, a Python package for Riemannian modelization and optimization over manifolds such as hyperspheres, hyperbolic spaces, SPD matrices or Lie groups of transformations. Our contribution is threefold. First, geomstats allows the flexible modeling of many a machine learning problem through an efficient and extensively unit-tested implementations of these manifolds, as well as the set of useful Riemannian metrics, exponential and logarithm maps that we provide. Moreover, the wide choice of loss functions and our implementation of the corresponding gradients allow fast and easy optimization over manifolds. Finally, geomstats is the only package to provide a unified framework for Riemannian geometry, as the operations implemented in geomstats are available with different computing backends (numpy,tensorflow and keras), as well as with a GPU-enabled mode–-thus considerably facilitating the application of Riemannian geometry in machine learning. In this paper, we present geomstats through a review of the utility and advantages of manifolds in machine learning, using the concrete examples that they span to show the efficiency and practicality of their implementation using our package

Introduciamo geomstats, un efficiente pacchetto Python per la modellazione e l'ottimizzazione Riemanniana su manifold compatibile sia con numpy che con tensorflow.

L'articolo introduce il pacchetto software geomstats, che fornisce un uso semplice dei manifold e delle metriche Riemanniane all'interno dei modelli di apprendimento automatico

Propone un pacchetto Python per l'ottimizzazione e le applicazioni su manifold Riemanniani ed evidenzia le differenze tra il pacchetto Geomstats e altri pacchetti.

Introduce un toolbox geometrico, Geomstats, per machine learning su manifold Riemanniani.


We propose to execute deep neural networks (DNNs) with dynamic and sparse graph (DSG) structure for compressive memory and accelerative execution during both training and inference. The great success of DNNs motivates the pursuing of lightweight models for the deployment onto embedded devices. However, most of the previous studies optimize for inference while neglect training or even complicate it. Training is far more intractable, since (i) the neurons dominate the memory cost rather than the weights in inference; (ii) the dynamic activation makes previous sparse acceleration via one-off optimization on fixed weight invalid; (iii) batch normalization (BN) is critical for maintaining accuracy while its activation reorganization damages the sparsity. To address these issues, DSG activates only a small amount of neurons with high selectivity at each iteration via a dimensionreduction search and obtains the BN compatibility via a double-mask selection. Experiments show significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x) with little accuracy loss on various benchmarks.

Costruiamo il grafo dinamico sparso tramite la ricerca di dimension-reduction per ridurre i costi di calcolo e di memoria sia nel training di DNN che nell'inferenza.

Gli autori propongono di utilizzare il grafo computazionale dinamico e sparso per ridurre la memoria di calcolo e il costo temporale nelle deep neural network (DNN).

Questo articolo propone un metodo per accelerare il training e l'inferenza delle deep neural network usando il pruning dinamico del grafo computazionale.


Efficient exploration remains a major challenge for reinforcement learning. One reason is that the variability of the returns often depends on the current state and action, and is therefore heteroscedastic. Classical exploration strategies such as upper confidence bound algorithms and Thompson sampling fail to appropriately account for heteroscedasticity, even in the bandit setting. Motivated by recent findings that address this issue in bandits, we propose to use Information-Directed Sampling (IDS) for exploration in reinforcement learning. As our main contribution, we build on recent advances in distributional reinforcement learning and propose a novel, tractable approximation of IDS for deep Q-learning. The resulting exploration strategy explicitly accounts for both parametric uncertainty and heteroscedastic observation noise. We evaluate our method on Atari games and demonstrate a significant improvement over alternative approaches.

Sviluppiamo un'estensione pratica dell'Information-Directed Sampling per Reinforcement Learning, che tiene conto dell'incertezza parametrica e dell'eteroscedasticità nella distribuzione di ritorno per l'esplorazione.

Gli autori propongono un modo di estendere l'Information-Directed Sampling al reinforcement learning combinando due tipi di incertezza per ottenere una semplice strategia di esplorazione basata su IDS.

Questo articolo studia approcci sofisticati di esplorazione per reinforcement learning costruiti su Information Direct Sampling e su Distributional Reinforcement Learning


We address the problem of learning structured policies for continuous control. In traditional reinforcement learning, policies of agents are learned by MLPs which take the concatenation of all observations from the environment as input for predicting actions. In this work, we propose NerveNet to explicitly model the structure of an agent, which naturally takes the form of a graph. Specifically, serving as the agent's policy network, NerveNet first propagates information over the structure of the agent and then predict actions for different parts of the agent. In the experiments, we first show that our NerveNet is comparable to state-of-the-art methods on standard MuJoCo environments. We further propose our customized reinforcement learning environments for benchmarking two types of structure transfer learning tasks, i.e., size and disability transfer. We demonstrate that policies learned by NerveNet are significantly better than policies learned by other models and are able to transfer even in a zero-shot setting.

Si utilizzano le graph neural network per modellare le informazioni strutturali degli agenti per migliorare la policy e la trasferibilità

Un metodo per la rappresentazione e l'apprendimento di una policy strutturata per task di controllo continuo utilizzando Graph Neural Network

Il paper propone l'aggiunta di un ulteriore struttura nei problemi di reinforcement learning, in particolare la struttura della morfologia dell'agente

Si propone un'applicazione delle Graph Neural Network all'apprendimento di policy per il controllo di robot "centipede" di diverse lunghezze.


Real-world tasks are often highly structured. Hierarchical reinforcement learning (HRL) has attracted research interest as an approach for leveraging the hierarchical structure of a given task in reinforcement learning (RL). However, identifying the hierarchical policy structure that enhances the performance of RL is not a trivial task. In this paper, we propose an HRL method that learns a latent variable of a hierarchical policy using mutual information maximization. Our approach can be interpreted as a way to learn a discrete and latent representation of the state-action space. To learn option policies that correspond to modes of the advantage function, we introduce advantage-weighted importance sampling.    In our HRL method, the gating policy learns to select option policies based on an option-value function, and these option policies are optimized based on the deterministic policy gradient method. This framework is derived by leveraging the analogy between a monolithic policy in standard RL and a hierarchical policy in HRL by using a deterministic option policy.   Experimental results indicate that our HRL approach can learn a diversity of options and that it can enhance the performance of RL in continuous control tasks.

Questo articolo presenta un framework di reinforcement learning gerarchico basato su policy di opzioni deterministiche e massimizzazione dell'informazione mutua.

Propone un algoritmo HRL che cerca di imparare le opzioni che massimizzano la loro mutua informazione con la densità di stato-azione sotto la policy ottimale.

Questo articolo propone un sistema HRL in cui l'informazione mutua della variabile latente e delle coppie stato-azione è approssimativamente massimizzata.

Propone un criterio che mira a massimizzare l'informazione mutua tra le opzioni e le coppie stato-azione e mostra empiricamente che le opzioni apprese decompongono lo spazio stato-azione ma non lo spazio di stato.


Deep neural networks (DNNs) have achieved impressive predictive performance due to their ability to learn complex, non-linear relationships between variables. However, the inability to effectively visualize these relationships has led to DNNs being characterized as black boxes and consequently limited their applications. To ameliorate this problem, we introduce the use of hierarchical interpretations to explain DNN predictions through our proposed method: agglomerative contextual decomposition (ACD). Given a prediction from a trained DNN, ACD produces a hierarchical clustering of the input features, along with the contribution of each cluster to the final prediction. This hierarchy is optimized to identify clusters of features that the DNN learned are predictive. We introduce ACD using examples from Stanford Sentiment Treebank and ImageNet, in order to diagnose incorrect predictions, identify dataset bias, and extract polarizing phrases of varying lengths. Through human experiments, we demonstrate that ACD enables users both to identify the more accurate of two DNNs and to better trust a DNN's outputs. We also find that ACD's hierarchy is largely robust to adversarial perturbations, implying that it captures fundamental aspects of the input and ignores spurious noise.

Introduciamo e convalidiamo le interpretazioni locali gerarchiche, la prima tecnica per cercare e visualizzare automaticamente le interazioni importanti per le predizioni individuali fatte da LSTM e CNN.

Un nuovo approccio per spiegare le predizioni delle reti neurali imparando rappresentazioni gerarchiche di gruppi di feature di input e il loro contributo alla predizione finale

Estende un metodo di interpretazione delle feature esistente per LSTM alle DNN più generiche e introduce un clustering gerarchico delle feature di input e i contributi di ogni cluster alla predizione finale.

Questo articolo propone un'estensione gerarchica della decomposizione contestuale.


Principal Filter Analysis (PFA) is an easy to implement, yet effective method for neural network compression. PFA exploits the intrinsic correlation between filter responses within network layers to recommend a smaller network footprint. We propose two compression algorithms: the first allows a user to specify the proportion of the original spectral energy that should be preserved in each layer after compression, while the second is a heuristic that leads to a parameter-free approach that automatically selects the compression used at each layer. Both algorithms are evaluated against several architectures and datasets, and we show considerable compression rates without compromising accuracy, e.g., for VGG-16 on CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x, and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively. In our tests we also demonstrate that networks compressed with PFA achieve an accuracy that is very close to the empirical upper bound for a given compression ratio. Finally, we show how PFA is an effective tool for simultaneous compression and domain adaptation.

Proponiamo un metodo facile da implementare ma efficace per la compressione delle reti neurali. PFA sfrutta la correlazione intrinseca tra le risposte dei filtri all'interno dei layer della rete per fornire una rete più piccola.

Propone di fare pruning delle reti convoluzionali analizzando la correlazione osservata tra i filtri di uno stesso layer come espresso dallo spettro di autovalori della loro matrice di covarianza.

Questo articolo introduce un approccio alla compressione delle reti neurali guardando la correlazione delle risposte dei filtri in ogni layer attraverso due strategie.

Questo documento propone un metodo di compressione basato sull'analisi spettrale


We propose a method to efficiently learn diverse strategies in reinforcement learning for query reformulation in the tasks of document retrieval and question answering. In the proposed framework an agent consists of multiple specialized sub-agents and a meta-agent that learns to aggregate the answers from sub-agents to produce a final answer. Sub-agents are trained on disjoint partitions of the training data, while the meta-agent is trained on the full training set. Our method makes learning faster, because it is highly parallelizable, and has better generalization performance than strong baselines, such as an ensemble of agents trained on the full data. We show that the improved performance is due to the increased diversity of reformulation strategies.

Agenti di riformulazione di query multiple e diversificate addestrati con reinforcement learning per migliorare i motori di ricerca.

Parellelizzazione del metodo dell'ensemble nel reinforcement learning per la riformulazione delle query, accelerando il training e migliorando la diversità delle riformulazioni apprese

Gli autori propongono di addestrare più agenti distinti, ciascuno su un diverso sottoinsieme del training set.

Gli autori propongono un approccio ensemble per la riformulazione delle query


Network Embeddings (NEs) map the nodes of a given network into $d$-dimensional Euclidean space $\mathbb{R}^d$. Ideally, this mapping is such that 'similar' nodes are mapped onto nearby points, such that the NE can be used for purposes such as link prediction (if 'similar' means being 'more likely to be connected') or classification (if 'similar' means 'being more likely to have the same label'). In recent years various methods for NE have been introduced, all following a similar strategy: defining a notion of similarity between nodes (typically some distance measure within the network), a distance measure in the embedding space, and a loss function that penalizes large distances for similar nodes and small distances for dissimilar nodes.   A difficulty faced by existing methods is that certain networks are fundamentally hard to embed due to their structural properties: (approximate) multipartiteness, certain degree distributions, assortativity, etc. To overcome this, we introduce a conceptual innovation to the NE literature and propose to create \emph{Conditional Network Embeddings} (CNEs); embeddings that maximally add information with respect to given structural properties (e.g. node degrees, block densities, etc.). We use a simple Bayesian approach to achieve this, and propose a block stochastic gradient descent algorithm for fitting it efficiently.   We demonstrate that CNEs are superior for link prediction and multi-label classification when compared to state-of-the-art methods, and this without adding significant mathematical or computational complexity. Finally, we illustrate the potential of CNE for network visualization.

Introduciamo un metodo di embedding della rete che tiene conto delle informazioni del prior sulla rete, ottenendo prestazioni empiriche superiori.

L'articolo ha proposto di utilizzare una distribuzione a priori per vincolare l'embedding della rete, per la formulazione questo articolo ha utilizzato distribuzioni gaussiane molto ristrette.

Propone l'apprendimento unsupervised degli embedding dei nodi considerando le proprietà strutturali delle reti.


This paper studies a class of adaptive gradient based momentum algorithms that update the  search directions and learning rates simultaneously using past gradients. This class, which we refer to as the ''``Adam-type'', includes the popular algorithms such as Adam, AMSGrad, AdaGrad. Despite their popularity in training deep neural networks (DNNs), the convergence of these algorithms for solving  non-convex problems remains an open question. In this paper, we develop an analysis framework and a set of mild sufficient conditions that guarantee the convergence of the Adam-type methods, with a convergence rate of order   $O(\log{T}/\sqrt{T})$ for non-convex stochastic optimization. Our convergence analysis applies to a new algorithm called AdaFom (AdaGrad with First Order Momentum). We show that the conditions are essential, by identifying concrete examples in which violating the conditions makes an algorithm diverge. Besides providing one of the first comprehensive analysis for Adam-type methods in the non-convex setting, our results can also help the practitioners to easily  monitor the progress of algorithms and determine their convergence behavior.

Analizziamo la convergenza degli algoritmi di tipo Adam e forniamo lievi condizioni sufficienti per garantire la loro convergenza, mostriamo anche che la violazione delle condizioni può far divergere un algoritmo.

Presenta un'analisi della convergenza in un ambiente non convesso per una famiglia di algoritmi di ottimizzazione.

Questo articolo studia la condizione di convergenza degli ottimizzatori di tipo Adam nei problemi di ottimizzazione non vincolati non convessi.


This research paper describes a simplistic architecture named as AANN: Absolute Artificial Neural Network, which can be used to create highly interpretable representations of the input data. These representations are generated by penalizing the learning of the network in such a way that those learned representations correspond to the respective labels present in the labelled dataset used for supervised training; thereby, simultaneously giving the network the ability to classify the input data. The network can be used in the reverse direction to generate data that closely resembles the input by feeding in representation vectors as required. This research paper also explores the use of mathematical abs (absolute valued) functions as activation functions which constitutes the core part of this neural network architecture. Finally the results obtained on the MNIST dataset by using this technique are presented and discussed in brief.

L'auto-encoder a pesi vincolati con la funzione di valore assoluto come funzione di attivazione, impara a fare la classificazione nella forward direction e la regressione nella backward direction grazie alla funzione di costo appositamente definita.

L'articolo propone di utilizzare la funzione di attivazione del valore assoluto in un'architettura autoencoder con un termine aggiuntivo di apprendimento supervisionato nella funzione obiettivo

Questo paper introduce una rete reversibile con il valore assoluto usato come funzione di attivazione


Current state-of-the-art relation extraction methods typically rely on a set of lexical, syntactic, and semantic features, explicitly computed in a pre-processing step. Training feature extraction models requires additional annotated language resources, which severely restricts the applicability and portability of relation extraction to novel languages. Similarly, pre-processing introduces an additional source of error. To address these limitations, we introduce TRE, a Transformer for Relation Extraction, extending the OpenAI Generative Pre-trained Transformer [Radford et al., 2018]. Unlike previous relation extraction models, TRE uses pre-trained deep language representations instead of explicit linguistic features to inform the relation classification and combines it with the self-attentive Transformer architecture to effectively model long-range dependencies between entity mentions. TRE allows us to learn implicit linguistic features solely from plain text corpora by unsupervised pre-training, before fine-tuning the learned language representations on the relation extraction task. TRE obtains a new state-of-the-art result on the TACRED and SemEval 2010 Task 8 datasets, achieving a test F1 of 67.4 and 87.1, respectively. Furthermore, we observe a significant increase in sample efficiency. With only 20% of the training examples, TRE matches the performance of our baselines and our model trained from scratch on 100% of the TACRED dataset. We open-source our trained models, experiments, and source code.

Proponiamo un modello di relation extraction basato su Transformer che usa rappresentazioni linguistiche pre-trained invece di feature linguistiche esplicite.

Presenta un modello di relation extraction basato su trasformer che sfrutta il pre-training su testo non annotato con un obiettivo di language modeling.

Questo articolo descrive una nuova applicazione di reti di trasformer per relation extraction.

L'articolo presenta un'architettura basata su Transformer per relation extraction, valutandola su due dataset.


Neural networks have recently had a lot of success for many tasks. However, neural  network architectures that perform well are still typically designed manually  by experts in a cumbersome trial-and-error process. We propose a new method  to automatically search for well-performing CNN architectures based on a simple  hill climbing procedure whose operators apply network morphisms, followed  by short optimization runs by cosine annealing. Surprisingly, this simple method  yields competitive results, despite only requiring resources in the same order of  magnitude as training a single network. E.g., on CIFAR-10, our method designs  and trains networks with an error rate below 6% in only 12 hours on a single GPU;  training for one day reduces this error further, to almost 5%.

Proponiamo un metodo semplice ed efficace per la ricerca di architetture per le reti neurali convoluzionali.

Propone un metodo di ricerca dell'architettura neurale che raggiunge una precisione vicina allo stato dell'arte su CIFAR10 e richiede molte meno risorse computazionali.

Presenta un metodo per cercare architetture di reti neurali contemporanemente al training che risparmia drasticamente sul tempo di training e sul tempo di ricerca dell'architettura.

Propone una variante di ricerca dell'architettura neurale usando morfismi di rete per definire uno spazio di ricerca usando architetture CNN che completano il task di classificazione delle immagini CIFAR


We propose GraphGAN - the first implicit generative model for graphs that enables to mimic real-world networks.  We pose the problem of graph generation as learning the distribution of biased random walks over a single input graph.  Our model is based on a stochastic neural network that generates discrete output samples, and is trained using the Wasserstein GAN objective. GraphGAN enables us to generate sibling graphs, which have similar properties yet are not exact replicas of the original graph. Moreover, GraphGAN learns a semantic mapping from the latent input space to the generated graph's properties. We discover that sampling from certain regions of the latent space leads to varying properties of the output graphs, with smooth transitions between them. Strong generalization properties of GraphGAN are highlighted by its competitive performance in link prediction as well as promising results on node classification, even though not specifically trained for these tasks.

Si utilizzano le GAN per generare grafi tramite random walk.

Gli autori hanno proposto un modello generativo di random walk su grafi che permette l'apprendimento model-agnostic, il fitting controllabile, la generazione di ensemble di grafi

Propone una formulazione WGAN per generare grafi basati su random walk usando gli embedding dei nodi e un'architettura LSTM per la modellazione.


The ability of a classifier to recognize unknown inputs is important for many classification-based systems. We discuss the problem of simultaneous classification and novelty detection, i.e. determining whether an input is from the known set of classes and from which specific class, or from an unknown domain and does not belong to any of the known classes. We propose a method based on the Generative Adversarial Networks (GAN) framework. We show that a multi-class discriminator trained with a generator that generates samples from a mixture of nominal and novel data distributions is the optimal novelty detector. We approximate that generator with a mixture generator trained with the Feature Matching loss and empirically show that the proposed method outperforms conventional methods for novelty detection. Our findings demonstrate a simple, yet powerful new application of the GAN framework for the task of novelty detection.

Proponiamo di risolvere un problema di classificazione simultanea e novelty detection nel framework GAN.

Propone una GAN per unificare la classificazione e la novelty detection.

L'articolo presenta un metodo per la novelty detection basato su una GAN multiclasse che è addestrata a produrre immagini generate da una mistura di distribuzioni nominali e nuove.

L'articolo propone una GAN per la novelty detection utilizzando un generatore di misture con loss di feature matching


Verifying a person's identity based on their voice is a challenging, real-world problem in biometric security. A crucial requirement of such speaker verification systems is to be domain robust. Performance should not degrade even if speakers are talking in languages not seen during training. To this end, we present a flexible and interpretable framework for learning domain invariant speaker embeddings using Generative Adversarial Networks. We combine adversarial training with an angular margin loss function, which encourages the speaker embedding model to be discriminative by directly optimizing for cosine similarity between classes. We are able to beat a strong baseline system using a cosine distance classifier and a simple score-averaging strategy. Our results also show that models with adversarial adaptation perform significantly better than unadapted models. In an attempt to better understand this behavior, we quantitatively measure the degree of invariance induced by our proposed methods using Maximum Mean Discrepancy and Frechet distances. Our analysis shows that our proposed adversarial speaker embedding models significantly reduce the distance between source and target data distributions, while performing similarly on the former and better on the latter.

Le prestazioni di speaker verification possono essere migliorate significativamente adattando il modello ai dati in-domain utilizzando le generative adversarial network. Inoltre, l'adattamento può essere eseguito in modo unsupervised.

Propone una serie di varianti di GAN per il task di speaker verification nella condizione di dominio non corrispondente.


Learning disentangling representations of the independent factors of variations that explain the data in an unsupervised setting is still a major challenge. In the following paper we address the task of disentanglement and introduce a new state-of-the-art approach called Non-synergistic variational Autoencoder (Non-Syn VAE). Our model draws inspiration from population coding, where the notion of synergy arises when we describe the encoded information by neurons in the form of responses from the stimuli. If those responses convey more information together than separate as independent sources of encoding information, they are acting synergetically. By penalizing the synergistic mutual information within the latents we encourage information independence and by doing that disentangle the latent factors. Notably, our approach could be added to the VAE framework easily, where the new ELBO function is still a lower bound on the log likelihood. In addition, we qualitatively compare our model with Factor VAE and show that this one implicitly minimises the synergy of the latents.

Minimizzare l'informazione mutua sinergica tra i latenti e i dati per il task di disentanglement usando il framework VAE.

Propone una nuova funzione obiettivo per l'apprendimento di rappresentazioni disentangled in un framework variazionale minimizzando la sinergia delle informazioni fornite.

Gli autori mirano ad addestrare un VAE che ha dissociato le rappresentazioni latenti in modo "sinergicamente" massimo.

Questo articolo propone un nuovo approccio per imporre il disentanglement nei VAE utilizzando un termine che penalizza l'informazione mutua sinergica tra le variabili latenti.


Metric embeddings are   immensely useful representations of associations between entities   (images, users, search queries, words, and more).   Embeddings are learned by  optimizing a loss objective of the general form of a sum over example associations. Typically, the optimization uses stochastic gradient updates over minibatches of examples that are arranged  independently at random. In this work, we propose the use of {\em structured arrangements} through randomized {\em microbatches} of examples that are more likely to include similar ones. We make a principled argument for the properties of our arrangements  that accelerate the training and present efficient algorithms to generate microbatches that respect the marginal  distribution of training examples.   Finally, we observe experimentally that our structured arrangements accelerate training by 3-20\%. Structured arrangements emerge as a powerful and novel performance knob for SGD that is independent and complementary to other SGD  hyperparameters and thus is a candidate for wide deployment.

Accelerare SGD disponendo gli esempi in modo diverso

L'articolo presenta un metodo per migliorare il tasso di convergenza di Stochastic Gradient Descent per l'apprendimento di embedding raggruppando sample di training simili.

Propone una strategia di sampling non uniforme per costruire minibatch in SGD per il task di imparare embedding per associazioni di oggetti.


Ubuntu dialogue corpus is the largest public available dialogue corpus to make it feasible to build end-to-end deep neural network models directly from the conversation data. One challenge of Ubuntu dialogue corpus is  the large number of out-of-vocabulary words. In this paper we proposed an algorithm which combines the general pre-trained word embedding vectors with those  generated on the task-specific training set to address this issue.   We integrated character embedding into Chen et al's Enhanced LSTM method (ESIM) and used it to evaluate the effectiveness of our proposed method. For the task of next utterance selection, the proposed method has demonstrated a significant performance improvement against original ESIM and the new model has achieved state-of-the-art results on both Ubuntu dialogue corpus and Douban conversation corpus. In addition, we investigated the performance impact of end-of-utterance and end-of-turn token tags.

Combina le informazioni tra i word embedding pre-costruiti e i word embedding specifici per il task per affrontare il problema dell'out-of-vocabulary

Questo articolo propone un approccio per migliorare la predizione di embedding out-of-vocabulary per il task di modeling dei dialoghi con guadagni considerevoli rispetto alle baseline.

Propone di combinare gli embedding di parole esterne preaddestrati e gli embedding di parole preaddestrati sui dati di training mantenendoli come due view separate.

Propone un metodo per estendere la copertura di word embedding pre-addestrati per affrontare il problema OOV che si presenta quando li si applica a dataset di conversazione e applica inoltre nuove varianti del modello basato su LSTM al task di response-selection nel dialogue modeling.


Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks. Analogously, this suggests that learned update functions may similarly outperform current hand-designed optimizers, especially for specific tasks. However, learned optimizers are notoriously difficult to train and have yet to demonstrate wall-clock speedups over hand-designed optimizers, and thus are rarely used in practice. Typically, learned optimizers are trained by truncated backpropagation through an unrolled optimization process. The resulting gradients are either strongly biased (for short truncations) or have exploding norm (for long truncations). In this work we propose a training scheme which overcomes both of these difficulties, by dynamically weighting two unbiased gradient estimators for a variational loss on optimizer performance. This allows us to train neural networks to perform optimization faster than well tuned first-order methods. Moreover, by training the optimizer against validation loss, as opposed to training loss, we are able to use it to train models which generalize better than those trained by first order methods. We demonstrate these results on problems where our learned optimizer trains convolutional networks in a fifth of the wall-clock time compared to tuned first-order methods, and with an improvement

Analizziamo i problemi durante il training degli ottimizzatori appresi, affrontiamo questi problemi tramite l'ottimizzazione variazionale utilizzando due stimatori di gradiente complementari, e addestriamo ottimizzatori che sono 5 volte più veloci in termini di wall-clock time rispetto agli ottimizzatori di base (per esempio Adam).

Questo articolo usa l'ottimizzazione un-rolled per imparare le reti neurali per l'ottimizzazione.

Questo articolo affronta il problema dell'apprendimento di un ottimizzatore, in particolare gli autori si concentrano sull'ottenimento di gradienti più puliti dalla procedura di training unrolled.

Presenta un metodo per "imparare un ottimizzatore" usando un'ottimizzazione variazionale per la loss "esterna" dell'ottimizzatore e propone l'idea di combinare sia il gradiente riparametrizzato che lo stimatore della funzione di scoring per l'obiettivo variazionale e li pondera usando un prodotto di Gaussiane per la media.


Asynchronous distributed gradient descent algorithms for training of deep neural  networks are usually considered as inefficient, mainly because of the Gradient delay  problem. In this paper, we propose a novel asynchronous distributed algorithm  that tackles this limitation by well-thought-out averaging of model updates, computed  by workers. The algorithm allows computing gradients along the process  of gradient merge, thus, reducing or even completely eliminating worker idle time  due to communication overhead, which is a pitfall of existing asynchronous methods.  We provide theoretical analysis of the proposed asynchronous algorithm,  and show its regret bounds. According to our analysis, the crucial parameter for  keeping high convergence rate is the maximal discrepancy between local parameter  vectors of any pair of workers. As long as it is kept relatively small, the  convergence rate of the algorithm is shown to be the same as the one of a sequential  online learning. Furthermore, in our algorithm, this discrepancy is bounded  by an expression that involves the staleness parameter of the algorithm, and is  independent on the number of workers. This is the main differentiator between  our approach and other solutions, such as Elastic Asynchronous SGD or Downpour  SGD, in which that maximal discrepancy is bounded by an expression that  depends on the number of workers, due to gradient delay problem. To demonstrate  effectiveness of our approach, we conduct a series of experiments on image  classification task on a cluster with 4 machines, equipped with a commodity communication  switch and with a single GPU card per machine. Our experiments  show a linear scaling on 4-machine cluster without sacrificing the test accuracy,  while eliminating almost completely worker idle time. Since our method allows  using commodity communication switch, it paves a way for large scale distributed  training performed on commodity clusters.

Un metodo per un efficiente training distribuito asincrono di modelli di deep learning insieme ai bound teorici di regret.

L'articolo propone un algoritmo per limitare la staleness in SGD asincrono e fornisce un'analisi teorica

Propone un algoritmo ibrido per eliminare il ritardo del gradiente nei metodi asincroni.


Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs. However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks. This paper aims to raise people's awareness about the security of the quantized models, and we designed a novel quantization methodology to jointly optimize the efficiency and robustness of deep learning models. We first conduct an empirical study to show that vanilla quantization suffers more from adversarial attacks. We observe that the inferior robustness comes from the error amplification effect, where the quantization operation further enlarges the distance caused by amplified noise. Then we propose a novel Defensive Quantization (DQ) method by controlling the Lipschitz constant of the network during quantization, such that the magnitude of the adversarial noise remains non-expansive during inference. Extensive experiments on CIFAR-10 and SVHN datasets demonstrate that our new quantization method can defend neural networks against adversarial examples, and even achieves superior robustness than their full-precision counterparts, while maintaining the same hardware efficiency as vanilla quantization approaches. As a by-product, DQ can also improve the accuracy of quantized models without adversarial attack.

Abbiamo progettato una nuova metodologia di quantizzazione per ottimizzare congiuntamente l'efficienza e la robustezza dei modelli di deep learning.

Propone uno schema di regolarizzazione per proteggere le reti neurali quantizzate dagli adversarial attack usando un filtraggio costante di Lipschitz dell'input-output dei layer interni.


Recurrent Neural Networks (RNNs) continue to show  outstanding performance in sequence modeling tasks. However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models. Source code is publicly available at https://imatge-upc.github.io/skiprnn-2017-telecombcn/.

Una modifica per le architetture RNN esistenti che permette loro di saltare gli aggiornamenti di stato mantenendo le prestazioni delle architetture originali.

Propone il modello Skip RNN che permette a una rete ricorrente di saltare selettivamente l'aggiornamento del suo hidden state per alcuni input, portando a una riduzione dei calcoli in fase di test.

Propone un nuovo modello RNN in cui sia l'input che l'aggiornamento dello stato delle celle ricorrenti sono saltati in modo adattivo in alcuni time step.


We propose a fast second-order method that can be used as a drop-in replacement for current deep learning solvers. Compared to stochastic gradient descent (SGD), it only requires two additional forward-mode automatic differentiation operations per iteration, which has a computational cost comparable to two standard forward passes and is easy to implement. Our method addresses long-standing issues with current second-order solvers, which invert an approximate Hessian matrix every iteration exactly or by conjugate-gradient methods, procedures that are much slower than a SGD step. Instead, we propose to keep a single estimate of the gradient projected by the inverse Hessian matrix, and update it once per iteration with just two passes over the network. This estimate has the same size and is similar to the momentum variable that is commonly used in SGD. No estimate of the Hessian is maintained.  We first validate our method, called CurveBall, on small problems with known solutions (noisy Rosenbrock function and degenerate 2-layer linear networks), where current deep learning solvers struggle. We then train several large models on CIFAR and ImageNet, including ResNet and VGG-f networks, where we demonstrate faster convergence with no hyperparameter tuning. We also show our optimiser's generality by testing on a large set of randomly-generated architectures.

Un veloce solutore del secondo ordine per il deep learning che funziona su problemi di scala ImageNet senza tuning degli iperparametri

Scegliere la direzione usando un singolo passo di gradient descent "verso il passo di Newton" da una stima originale, e poi prendere questa direzione invece del gradiente originale

Un nuovo metodo di ottimizzazione approssimata del secondo ordine a basso costo computazionale che sostituisce il calcolo della matrice Hessiana con un singolo gradient step e una strategia di warm start.


The recently presented idea to learn heuristics for combinatorial optimization problems is promising as it can save costly development. However, to push this idea towards practical implementation, we need better models and better ways of training. We contribute in both directions: we propose a model based on attention layers with benefits over the Pointer Network and we show how to train this model using REINFORCE with a simple baseline based on a deterministic greedy rollout, which we find is more efficient than using a value function. We significantly improve over recent learned heuristics for the Travelling Salesman Problem (TSP), getting close to optimal results for problems up to 100 nodes. With the same hyperparameters, we learn strong heuristics for two variants of the Vehicle Routing Problem (VRP), the Orienteering Problem (OP) and (a stochastic variant of) the Prize Collecting TSP (PCTSP), outperforming a wide range of baselines and getting results close to highly optimized and specialized algorithms.

Modello basato sull'attention addestrato con REINFORCE con la greedy rollout baseline per imparare euristiche con risultati competitivi su TSP e altri problemi di routing

Presenta un approccio basato sull'attention per imparare una policy per risolvere TSP e altri problemi di ottimizzazione combinatoria di routing.

Questo articolo cerca di imparare l'euristica per risolvere problemi di ottimizzazione combinatoria


We propose an efficient online hyperparameter optimization method which uses a joint dynamical system to evaluate the gradient with respect to the hyperparameters. While similar methods are usually limited to hyperparameters with a smooth impact on the model, we show how to apply it to the probability of dropout in neural networks. Finally, we show its effectiveness on two distinct tasks.

Un algoritmo per ottimizzare gli iper-parametri di regolarizzazione durante il training

L'articolo propone un modo per reinizializzare y ad ogni aggiornamento di lambda e una procedura di clipping di y per mantenere la stabilità del sistema dinamico.

Propone un algoritmo per l'ottimizzazione degli iperparametri che può essere visto come un'estensione di Franceschi 2017 in cui alcune stime vengono riavviate a caldo per aumentare la stabilità del metodo.

Propone un'estensione di un metodo esistente per ottimizzare gli iperparametri di regolarizzazione.


Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks. However, these have been evaluated using differing codebases and limited computational resources, which represent uncontrolled sources of experimental variation. We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models. We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the Hutter Prize dataset.

Mostrare che le LSTM sono altrettanto buone o migliori delle recenti innovazioni per LM e che la valutazione dei modelli è spesso inaffidabile.

Questo articolo descrive una convalida completa dei language model a livello di parola e carattere basati su LSTM, portando a un risultato significativo nel language modeling e ad una pietra miliare nel deep learning.


Residual and skip connections play an important role in many current   generative models. Although their theoretical and numerical advantages   are understood, their role in speech enhancement systems has not been   investigated so far. When performing spectral speech enhancement,   residual connections are very similar in nature to spectral subtraction,   which is the one of the most commonly employed speech enhancement approaches.    Highway networks, on the other hand, can be seen as a combination of spectral   masking and spectral subtraction. However, when using deep neural networks, such operations would   normally happen in a transformed spectral domain, as opposed to traditional speech   enhancement where all operations are often done directly on the spectrum.    In this paper, we aim to investigate the role of residual and highway   connections in deep neural networks for speech enhancement, and verify whether   or not they operate similarly to their traditional, digital signal processing   counterparts. We visualize the outputs of such connections, projected back to   the spectral domain, in models trained for speech denoising, and show that while   skip connections do not necessarily improve performance with regards to the   number of parameters, they make speech enhancement models more interpretable.

Mostriamo come l'uso delle skip connection può rendere i modelli di speech enhancement più interpretabili, poiché li fa usare meccanismi simili a quelli che sono stati esplorati nella letteratura DSP.

Gli autori propongono di incorporare i blocchi Residual, Highway e Masking all'interno di una pipeline completamente convoluzionale per capire come viene eseguita l'inferenza iterativa dell'output e il mascheramento in un task di speech enhancement

Gli autori interpretano le connessioni highway, residual e di masking.

Gli autori generano il loro discorso rumoroso aggiungendo artificialmente del rumore da un dataset sul rumore ben noto ad un dataset sul discorso senza rumore meno conosciuto.


Bayesian neural networks (BNNs) hold great promise as a flexible and principled solution to deal with uncertainty when learning from finite data. Among approaches to realize probabilistic inference in deep neural networks, variational Bayes (VB) is theoretically grounded, generally applicable, and computationally efficient. With wide recognition of potential advantages, why is it that variational Bayes has seen very limited practical use for BNNs in real applications? We argue that variational inference in neural networks is fragile: successful implementations require careful initialization and tuning of prior variances, as well as controlling the variance of Monte Carlo gradient estimates. We provide two innovations that aim to turn VB into a robust inference tool for Bayesian neural networks: first, we introduce a novel deterministic method to approximate moments in neural networks, eliminating gradient variance; second, we introduce a hierarchical prior for parameters and a novel Empirical Bayes procedure for automatically selecting prior variances. Combining these two innovations, the resulting method is highly efficient and robust. On the application of heteroscedastic regression we demonstrate good predictive performance over alternative approaches.

Un metodo per eliminare la varianza del gradiente e fare automaticamente tuning dei prior per un training efficace delle reti neurali bayesiane

Propone un nuovo approccio per eseguire l'inferenza variazionale deterministica per BNN feed-forward con specifiche funzioni di attivazione non lineari approssimando i momenti layerwise.

L'articolo considera un approccio puramente deterministico all'apprendimento di approssimazioni posteriori variazionali per le reti neurali bayesiane.


Skills learned through (deep) reinforcement learning often generalizes poorly  across tasks and re-training is necessary when presented with a new task. We  present a framework that combines techniques in formal methods with reinforcement  learning (RL) that allows for the convenient specification of complex temporal  dependent tasks with logical expressions and construction of new skills from existing  ones with no additional exploration. We provide theoretical results for our  composition technique and evaluate on a simple grid world simulation as well as  a robotic manipulation task.

Un metodo formale per la composizione di skill in task di reinforcement learning

L'articolo combina il RL e vincoli espressi da formule logiche impostando un'automazione da formule scTLTL.

Propone un metodo che aiuta a costruire la policy dai sottotask appresi sul tema della combinazione di task RL con formule logiche temporali lineari.


The application of multi-modal generative models by means of a Variational Auto Encoder (VAE) is an upcoming research topic for sensor fusion and bi-directional modality exchange.  This contribution gives insights into the learned joint latent representation and shows that expressiveness and coherence are decisive properties for multi-modal datasets.  Furthermore, we propose a multi-modal VAE derived from the full joint marginal log-likelihood that is able to learn the most meaningful representation for ambiguous observations.  Since the properties of multi-modal sensor setups are essential for our approach but hardly available, we also propose a technique to generate correlated datasets from uni-modal ones.

Derivare una formulazione generale di un VAE multimodale dalla log-likelihood marginale congiunta.

Propone un VAE multimodale con un bound variazionale derivato dalla regola della catena.

Questo articolo propone un obiettivo, M^2VAE, per i VAE multimodali, che dovrebbe imparare una rappresentazione dello spazio latente più significativa.


We build on auto-encoding sequential Monte Carlo (AESMC): a method for model and proposal learning based on maximizing the lower bound to the log marginal likelihood in a broad family of structured probabilistic models. Our approach relies on the efficiency of sequential Monte Carlo (SMC) for performing inference in structured probabilistic models and the flexibility of deep neural networks to model complex conditional probability distributions. We develop additional theoretical insights and introduce a new training procedure which improves both model and proposal learning. We demonstrate that our approach provides a fast, easy-to-implement and scalable means for simultaneous model learning and proposal adaptation in deep generative models.

Ci basiamo sull'autoencoding sequenziale Monte Carlo, otteniamo nuove intuizioni teoriche e sviluppiamo una procedura di training migliorata basata su queste intuizioni.

L'articolo propone una versione del training in stile IWAE che usa SMC invece del classico importance sampling.

Questo lavoro propone l'autoencoding sequenziale Monte Carlo, estendendo il framework VAE a un nuovo obiettivo Monte Carto basato su SMC.


A key component for many reinforcement learning agents is to learn a value function, either for policy evaluation or control. Many of the algorithms for learning values, however, are designed for linear function approximation---with a fixed basis or fixed representation. Though there have been a few sound extensions to nonlinear function approximation, such as nonlinear gradient temporal difference learning, these methods have largely not been adopted, eschewed in favour of simpler but not sound methods like temporal difference learning and Q-learning. In this work, we provide a two-timescale network (TTN) architecture that enables linear methods to be used to learn values, with a nonlinear representation learned at a slower timescale. The approach facilitates the use of algorithms developed for the linear setting, such as data-efficient least-squares methods, eligibility traces and the myriad of recently developed linear policy evaluation algorithms, to provide nonlinear value estimates. We prove convergence for TTNs, with particular care given to ensure convergence of the fast linear component under potentially dependent features provided by the learned representation. We empirically demonstrate the benefits of TTNs, compared to other nonlinear value function approximation algorithms, both for policy evaluation and control.

Proponiamo un'architettura per l'apprendimento delle funzioni di valore che permette l'uso di qualsiasi algoritmo lineare di valutazione della policy in tandem con l'apprendimento non lineare delle feature.

L'articolo propone un framework a due scale temporali per l'apprendimento della funzione di valore e della rappresentazione dello stato con approssimazioni non lineari.

Questo articolo propone le reti a due scale temporali e prova la convergenza di questo metodo usando metodi di approssimazione stocastica a due scale temporali.

Questo articolo presenta una rete a due scale temporali che permette di utilizzare metodi lineari per imparare i valori.


Large-scale Long Short-Term Memory (LSTM) cells are often the building blocks of many state-of-the-art algorithms for tasks in Natural Language Processing (NLP). However, LSTMs are known to be computationally inefficient because the memory capacity of the models depends on the number of parameters, and the inherent recurrence that models the temporal dependency is not parallelizable. In this paper, we propose simple, but effective, low-rank matrix factorization (MF) algorithms to compress network parameters and significantly speed up LSTMs with almost no loss of performance (and sometimes even gain). To show the effectiveness of our method across different tasks, we examine two settings: 1) compressing core LSTM layers in Language Models, 2) compressing biLSTM layers of ELMo~\citep{ELMo} and evaluate in three downstream NLP tasks (Sentiment Analysis, Textual Entailment, and Question Answering). The latter is particularly interesting as embeddings from large pre-trained biLSTM Language Models are often used as contextual word representations. Finally, we discover that matrix factorization performs better in general, additive recurrence is often more important than multiplicative recurrence, and we identify an interesting correlation between matrix norms and compression performance.

Proponiamo semplici, ma efficaci, algoritmi di fattorizzazione di matrice a basso rango (MF) per accelerare il tempo di esecuzione, risparmiare memoria e migliorare le prestazioni delle LSTM.

Propone di accelerare le LSTM usando MF come strategia di compressione post-elaborazione e conduce ampi esperimenti per mostrare le prestazioni.


Manipulation and re-use of images in scientific publications is a recurring problem, at present lacking a scalable solution.   Existing tools for detecting image duplication are mostly manual or semi-automated, despite the fact that generating data for a learning-based approach is straightforward, as we here illustrate. This paper addresses the problem of determining if, given two images, one is a manipulated version of the other by means of certain geometric and statistical manipulations, e.g. copy, rotation, translation, scale, perspective transform, histogram adjustment, partial erasing, and compression artifacts. We propose a solution based on a 3-branch Siamese Convolutional Neural Network. The ConvNet model is trained to map images into a 128-dimensional space, where the Euclidean distance between duplicate (respectively, unique) images is no greater (respectively, greater) than 1. Our results suggest that such an approach can serve as tool to improve surveillance of the published and in-peer-review literature for image manipulation. We also show that as a byproduct the network learns useful representations for semantic segmentation, with performance comparable to that of domain-specific models.

Una metrica forense per determinare se una data immagine è una copia (con possibile manipolazione) di un'altra immagine da un dato dataset.

Introduce la siamese network per identificare le immagini duplicate e copiate/modificate, che può essere usata per migliorare la sorveglianza della letteratura pubblicata e in peer-review.

L'articolo presenta un'applicazione di deep convolutional network per il task di rilevamento di immagini duplicate

Questo lavoro affronta il problema di trovare immagini duplicate/near duplicate da pubblicazioni biomediche e propone una CNN standard e funzioni di loss, applicandole a questo campo.


Training generative adversarial networks is unstable in high-dimensions as the true data distribution tends to be concentrated in a small fraction of the ambient space. The discriminator is then quickly able to classify nearly all generated samples as fake, leaving the generator without meaningful gradients and causing it to deteriorate after a point in training. In this work, we propose training a single generator simultaneously against an array of discriminators, each of which looks at a different random low-dimensional projection of the data. Individual discriminators, now provided with restricted views of the input, are unable to reject generated samples perfectly and continue to provide meaningful gradients to the generator throughout training.   Meanwhile, the generator learns to produce samples consistent with the full data distribution to satisfy all discriminators simultaneously. We  demonstrate the practical utility of this approach experimentally, and show that it is able to produce image samples with higher quality than traditional training with a single discriminator.

Training stabile delle GAN ad alte dimensioni utilizzando un array di discriminatori, ciascuno con una visione a bassa dimensione dei sample generati

L'articolo propone di stabilizzare il training delle GAN utilizzando un insieme di discriminatori, ognuno dei quali lavora su una proiezione casuale dei dati di input, per fornire il segnale di training per il modello generatore.

L'articolo propone un metodo di training delle GAN per migliorare la stabilità del training.

L'articolo propone un nuovo approccio al training delle GAN, che fornisce gradienti stabili per addestrare il generatore.


We present a novel method to precisely impose tree-structured category information onto word-embeddings, resulting in ball embeddings in higher dimensional spaces (N-balls for short). Inclusion relations among N-balls implicitly encode subordinate relations among categories. The similarity measurement in terms of the cosine function is enriched by category information. Using a geometric construction method instead of back-propagation, we create large N-ball embeddings that satisfy two conditions: (1) category trees are precisely imposed onto word embeddings at zero energy cost; (2) pre-trained word embeddings are well preserved. A new benchmark data set is created for validating the category of unknown words. Experiments show that N-ball embeddings, carrying category information, significantly outperform word embeddings in the test of nearest neighborhoods, and demonstrate surprisingly good performance in validating categories of unknown words. Source codes and data-sets are free for public access \url{https://github.com/gnodisnait/nball4tree.git} and \url{https://github.com/gnodisnait/bp94nball.git}.

Mostriamo un metodo geometrico per codificare perfettamente le informazioni dell'albero delle categorie in word-embedding pre-addestrati.

L'articolo propone N-ball embedding per i dati tassonomici dove una N-ball è una coppia di un vettore centroide e il raggio dal centro.

L'articolo presenta un metodo per modificare gli embedding vettoriali esistenti di oggetti categorici (come le parole), per convertirle in embedding a sfera che seguono le gerarchie.

Si concentra sulla regolazione dei pretrained word embedding in modo che rispettino la relazione ipernimia/iponimia mediante un appropriato incapsulamento n-ball.


For the challenging semantic image segmentation task the best performing models  have traditionally combined the structured modelling capabilities of Conditional  Random Fields (CRFs) with the feature extraction power of CNNs. In more recent  works however, CRF post-processing has fallen out of favour. We argue that this  is mainly due to the slow training and inference speeds of CRFs, as well as the  difficulty of learning the internal CRF parameters. To overcome both issues we  propose to add the assumption of conditional independence to the framework of  fully-connected CRFs. This allows us to reformulate the inference in terms of  convolutions, which can be implemented highly efficiently on GPUs.Doing so  speeds up inference and training by two orders of magnitude. All parameters of  the convolutional CRFs can easily be optimized using backpropagation. Towards  the goal of facilitating further CRF research we have made our implementations  publicly available.

Proponiamo Convolutional CRF, un'alternativa veloce, potente e allenabile ai Fully Connected CRF.

Gli autori sostituiscono il grande passo di filtraggio nel lattice permutoedrico con un kernel convoluzionario spazialmente variabile e mostrano che l'inferenza è più efficiente e il training più facile.

Propone di eseguire il message passing su un CRF a kernel gaussiano troncato usando un kernel definito e il message passing parallelizzato su GPU.