To backpropagate the gradients through stochastic binary layers, we propose the augment-REINFORCE-merge (ARM) estimator that is unbiased, exhibits low variance, and has low computational complexity. Exploiting variable augmentation, REINFORCE, and reparameterization, the ARM estimator achieves adaptive variance reduction for Monte Carlo integration by merging two expectations via common random numbers. The variance-reduction mechanism of the ARM estimator can also be attributed to either antithetic sampling in an augmented space, or the use of an optimal anti-symmetric "self-control" baseline function together with the REINFORCE estimator in that augmented space. Experimental results show the ARM estimator provides state-of-the-art performance in auto-encoding variational inference and maximum likelihood estimation, for discrete latent variable models with one or multiple stochastic binary layers. Python code for reproducible research is publicly available.

Uno stimatore di gradiente unbiased e a bassa varianza per modelli di variabili latenti discreti

Propone una nuova tecnica di riduzione della varianza da usare quando si calcola un gradiente di expected loss dove l'aspettativa è rispetto a variabili casuali binarie indipendenti.

Un algoritmo che combina Rao-Blackwellization e numeri casuali comuni per abbassare la varianza dello stimatore del gradiente della funzione di punteggio nel caso speciale delle reti binarie stocastiche

Uno stimatore unbiased e a bassa varianza augment-REINFORCE-merge (ARM) per il calcolo e la backpropagation dei gradienti nelle reti neurali binarie


Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training. The scheme can reach a linear speed-up with respect to the number of workers, but this is rarely seen in practice as the scheme often suffers from large network delays and bandwidth limits. To overcome this communication bottleneck recent works propose to reduce the communication frequency. An algorithm of this type is local SGD that runs SGD independently in parallel on different workers and averages the sequences only once in a while. This scheme shows promising results in practice, but eluded thorough theoretical analysis.       We prove concise convergence rates for local SGD on convex problems and show that it converges at the same rate as mini-batch SGD in terms of number of evaluated gradients, that is, the scheme achieves linear speed-up in the number of workers and mini-batch size. The number of  communication rounds can be reduced up to a factor of T^{1/2}---where T denotes the number of total steps---compared to mini-batch SGD. This also holds for asynchronous implementations.   Local SGD can also be used for large scale training of deep learning models. The results shown here aim serving as a guideline to further explore the theoretical and practical aspects of local SGD in these applications.

Dimostriamo che SGD locale parallelo raggiunge uno speedup lineare con una comunicazione molto minore rispetto a SGD parallelo in mini-batch.

Fornisce una prova di convergenza per SGD locale, e dimostra che SGD locale può fornire gli stessi guadagni di accelerazione del minibatch, ma può essere in grado di comunicare molto meno.

Questo articolo presenta un'analisi di SGD locale e dei bound sulla frequenza con cui gli stimatori ottenuti dall'esecuzione di SGD devono essere mediati per ottenere un aumento lineare della velocità di parallelizzazione.

Gli autori analizzano l'algoritmo SGD locale, dove vengono eseguite $K$ catene parallele di SGD, e le iterazioni sono occasionalmente sincronizzate tra le macchine facendo la media


Extracting relevant information, causally inferring and predicting the future states with high accuracy is a crucial task for modeling complex systems. The endeavor to address these tasks is made even more challenging when we have to deal with high-dimensional heterogeneous data streams. Such data streams often have higher-order inter-dependencies across spatial and temporal dimensions. We propose to perform a soft-clustering of the data and learn its dynamics to produce a compact dynamical model while still ensuring the original objectives of causal inference and accurate predictions. To efficiently and rigorously process the dynamics of soft-clustering, we advocate for an information theory inspired approach that incorporates stochastic calculus and seeks to determine a trade-off between the predictive accuracy and compactness of the mathematical representation. We cast the model construction as a maximization of the compression of the state variables such that the predictive ability and causal interdependence (relatedness) constraints between the original data streams and the compact model are closely bounded. We provide theoretical guarantees concerning the convergence of the proposed learning algorithm. To further test the proposed framework, we consider a high-dimensional Gaussian case study and describe an iterative scheme for updating the new model parameters. Using numerical experiments, we demonstrate the benefits on compression and prediction accuracy for a class of dynamical systems. Finally, we apply the proposed algorithm to the real-world dataset of multimodal sentiment intensity and show improvements in prediction with reduced dimensions.

Percezione compatta del processo dinamico

Studia il problema di rappresentare in modo compatto il modello di un sistema dinamico complesso preservando l'informazione usando un metodo di information bottleneck.

Questo articolo studia la dinamica lineare gaussiana e propone un algoritmo per calcolare l'Information Bottleneck Hierarchy (IBH).


We propose the dense RNN, which has the fully connections from each hidden state to multiple preceding hidden states of all layers directly. As the density of the connection increases, the number of paths through which the gradient flows can be increased. It increases the magnitude of gradients, which help to prevent the vanishing gradient problem in time. Larger gradients, however, can also cause exploding gradient problem. To complement the trade-off between two problems, we propose an attention gate, which controls the amounts of gradient flows. We describe the relation between the attention gate and the gradient flows by approximation. The experiment on the language modeling using Penn Treebank corpus shows dense connections with the attention gate improve the model’s performance.

RNN densa che ha connessioni complete da ogni stato nascosto direttamente a più stati nascosti precedenti di tutti i layer.

Propone una nuova architettura RNN che modella meglio le dipendenze a lungo termine, può imparare la rappresentazione multiscala dei dati sequenziali e aggirare il problema dei gradienti usando unità di gating parametrizzate.

Questo articolo propone un'architettura RNN densa fully connected con connessioni gated per ogni layer e connessioni del layer precedente, e i suoi risultati sul task di modellazione a livello di caratteri PTB.


We propose a new algorithm for training generative adversarial networks to jointly learn latent codes for both identities (e.g. individual humans) and observations (e.g. specific photographs). In practice, this means that by fixing the identity portion of latent codes, we can generate diverse images of the same subject, and by fixing the observation portion we can traverse the manifold of subjects while maintaining contingent aspects such as lighting and pose. Our algorithm features a pairwise training scheme in which each sample from the generator consists of two images with a common identity code. Corresponding samples from the real dataset consist of two distinct photographs of the same subject. In order to fool the discriminator, the generator must produce images that are both photorealistic, distinct, and appear to depict the same person. We augment both the DCGAN and BEGAN approaches with Siamese discriminators to accommodate pairwise training. Experiments with human judges and an off-the-shelf face verification system demonstrate our algorithm’s ability to generate convincing, identity-matched photographs.

Le SD-GAN distinguono i codici latenti in base alle comunanze note in un dataset (ad esempio, fotografie che ritraggono la stessa persona).

Questo articolo indaga il problema della generazione controllata di immagini e propone un algoritmo che produce una coppia di immagini con la stessa identità.

Questo articolo propone SD-GAN, un metodo di training delle GAN per distinguere le informazioni di identità e non identità nel vettore latente di input Z.


The goal of unpaired cross-domain translation is to learn useful mappings between two domains, given unpaired sets of datapoints from these domains. While this formulation is highly underconstrained, recent work has shown that it is possible to learn mappings useful for downstream tasks by encouraging approximate cycle consistency in the mappings between the two domains [Zhu et al., 2017]. In this work, we propose AlignFlow, a framework for unpaired cross-domain translation that ensures exact cycle consistency in the learned mappings. Our framework uses a normalizing flow model to specify a single invertible mapping between the two domains. In contrast to prior works in cycle-consistent translations, we can learn AlignFlow via adversarial training, maximum likelihood estimation, or a hybrid of the two methods. Theoretically, we derive consistency results for AlignFlow which guarantee recovery of desirable mappings under suitable assumptions. Empirically, AlignFlow demonstrates significant improvements over relevant baselines on image-to-image translation and unsupervised domain adaptation tasks on benchmark datasets.

Proponiamo un learning framework per le traduzioni cross-domain che è esattamente cycle-consistent e può essere appreso tramite adversarial training, stima di massima verosimiglianza, o un ibrido dei due.

Propone AlignFlow, un modo efficiente di implementare il principio di coerenza del ciclo usando flussi invertibili.

Modelli di flusso per la traduzione da immagine a immagine non accoppiata


Program synthesis is a class of regression problems where one seeks a solution, in the form of a source-code program, that maps the inputs to their corresponding outputs exactly. Due to its precise and combinatorial nature, it is commonly formulated as a constraint satisfaction problem, where input-output examples are expressed constraints, and solved with a constraint solver. A key challenge of this formulation is that of scalability: While constraint solvers work well with few well-chosen examples, constraining the entire set of example constitutes a significant overhead in both time and memory. In this paper we address this challenge by constructing a representative subset of examples that is both small and is able to constrain the solver sufficiently. We build the subset one example at a time, using a trained discriminator to predict the probability of unchosen input-output examples conditioned on the chosen input-output examples, adding the least probable example to the subset. Experiment on a diagram drawing domain shows our approach produces subset of examples that are small and representative for the constraint solver.

In un contesto di program synthesis in cui l'input è un insieme di esempi, riduciamo il costo calcolando un sottoinsieme di esempi rappresentativi

Propone un metodo per identificare esempi rappresentativi per program synthesis per aumentare la scalabilità delle soluzioni di programmazione vincolata esistenti.

Un metodo per scegliere un sottoinsieme di esempi su cui eseguire un risolutore di vincoli per risolvere problemi di program synthesis.

Questo articolo propone un metodo per accelerare i sintetizzatori di programmi general-purpose.


Humans possess an ability to abstractly reason about objects and their interactions, an ability not shared with state-of-the-art deep learning models. Relational networks, introduced by Santoro et al. (2017), add the capacity for relational reasoning to deep neural networks, but are limited in the complexity of the reasoning tasks they can address. We introduce recurrent relational networks which increase the suite of solvable tasks to those that require an order of magnitude more steps of relational reasoning. We use recurrent relational networks to solve Sudoku puzzles and achieve state-of-the-art results by solving 96.6% of the hardest Sudoku puzzles, where relational networks fail to solve any. We also apply our model to the BaBi textual QA dataset solving 19/20 tasks which is competitive with state-of-the-art sparse differentiable neural computers. The recurrent relational network is a general purpose module that can augment any neural network model with the capacity to do many-step relational reasoning.

Introduciamo Recurrent Relational Networks, un modulo di rete neurale potente e generale per il ragionamento relazionale, e lo usiamo per risolvere il 96,6% dei task Sudoku più difficili e 19/20 dei task BaBi.

Introdotta la rete relazionale ricorrente (RRNs) che può essere aggiunta a qualsiasi rete neurale per aggiungere capacità di ragionamento relazionale.

Introduzione di una deep neural network per structured prediction che raggiunge prestazioni allo stato dell'arte sui puzzle Soduku e sul task BaBi.

Questo articolo descrive un metodo chiamato rete relazionale per aggiungere capacità di ragionamento relazionale alle deep neural network.


Empirical risk minimization (ERM), with proper loss function and regularization, is the common practice of supervised classification. In this paper, we study training arbitrary (from linear to deep) binary classifier from only unlabeled (U) data by ERM. We prove that it is impossible to estimate the risk of an arbitrary binary classifier in an unbiased manner given a single set of U data, but it becomes possible given two sets of U data with different class priors. These two facts answer a fundamental question---what the minimal supervision is for training any binary classifier from only U data. Following these findings, we propose an ERM-based learning method from two sets of U data, and then prove it is consistent. Experiments demonstrate the proposed method could train deep models and outperform state-of-the-art methods for learning from two sets of U data.

Tre prior di classe sono tutti necessari per addestrare deep model da soli dati U, mentre due qualsiasi non dovrebbero essere sufficienti.

Propone uno stimatore unbiased che permette di addestrare modelli con supervisione debole su due dataset non annotati con prior di classe noti e discute le proprietà teoriche degli stimatori.

Una metodologia per il training di qualsiasi classificatore binario da soli dati non annotati, e un metodo di empirical risk minimization per due insiemi di dati non annotati in cui sono dati i prior di classe.


Deep Neural Networks (DNNs) excel on many complex perceptual tasks but it has proven notoriously difficult to understand how they reach their decisions. We here introduce a high-performance DNN architecture on ImageNet whose decisions are considerably easier to explain. Our model, a simple variant of the ResNet-50 architecture called BagNet, classifies an image based on the occurrences of small local image features without taking into account their spatial ordering. This strategy is closely related to the bag-of-feature (BoF) models popular before the onset of deep learning and reaches a surprisingly high accuracy on ImageNet (87.6% top-5 for 32 x 32 px features and Alexnet performance for 16 x16 px features). The constraint on local features makes it straight-forward to analyse how exactly each part of the image influences the classification. Furthermore, the BagNets behave similar to state-of-the art deep neural networks such as VGG-16, ResNet-152 or DenseNet-169 in terms of feature sensitivity, error distribution and interactions between image parts. This suggests that the improvements of DNNs over previous bag-of-feature classifiers in the last few years is mostly achieved by better fine-tuning rather than by qualitatively different decision strategies.

L'aggregazione di prove di classe da molte piccole patch di immagini è sufficiente per risolvere ImageNet, produce modelli più interpretabili e può spiegare aspetti del processo decisionale delle DNN popolari.

Questo articolo suggerisce un'architettura di rete neurale nuova e compatta che utilizza le informazioni all'interno delle feature di bag-of-words. L'algoritmo proposto utilizza solo le informazioni delle patch in modo indipendente ed esegue il voto di maggioranza utilizzando patch classificate in modo indipendente.


Somatic cancer mutation detection at ultra-low variant allele frequencies (VAFs) is an unmet challenge that is intractable with current state-of-the-art mutation calling methods. Specifically, the limit of VAF detection is closely related to the depth of coverage, due to the requirement of multiple supporting reads in extant methods, precluding the detection of mutations at VAFs that are orders of magnitude lower than the depth of coverage. Nevertheless, the ability to detect cancer-associated mutations in ultra low VAFs is a fundamental requirement for low-tumor burden cancer diagnostics applications such as early detection, monitoring, and therapy nomination using liquid biopsy methods (cell-free DNA). Here we defined a spatial representation of sequencing information adapted for convolutional architecture that enables variant detection at VAFs, in a manner independent of the depth of sequencing. This method enables the detection of cancer mutations even in VAFs as low as 10x-4^, >2 orders of magnitude below the current state-of-the-art. We validated our method on both simulated plasma and on clinical cfDNA plasma samples from cancer patients and non-cancer controls. This method introduces a new domain within bioinformatics and personalized medicine – somatic whole genome mutation calling for liquid biopsy.

Gli attuali metodi di mutazione somatica non funzionano con le biopsie liquide (cioè il sequenziamento a bassa copertura), applichiamo un'architettura CNN a una rappresentazione unica di una lettura e il suo alignment, mostriamo un miglioramento significativo rispetto ai metodi precedenti nel setting a bassa frequenza.

Propone una soluzione basata su CNN chiamata Kittyhawk per somatic mutation calling a frequenze alleliche ultra basse.

Un nuovo algoritmo per rilevare le mutazioni del cancro dal sequenziamento del DNA libero delle cellule che identificherà il contesto di sequenza che distingue gli errori di sequenziamento dalle vere mutazioni.

Questo articolo propone un framework di deep learning per prevedere le mutazioni somatiche a frequenze estremamente basse che si verificano nel rilevamento del tumore dal DNA privo di cellule


This paper presents the formal release of {\em MedMentions}, a new manually annotated resource for the recognition of biomedical concepts. What distinguishes MedMentions from other annotated biomedical corpora is its size (over 4,000 abstracts and over 350,000 linked mentions), as well as the size of the concept ontology (over 3 million concepts from UMLS 2017) and its broad coverage of biomedical disciplines. In addition to the full corpus, a sub-corpus of MedMentions is also presented, comprising annotations for a subset of UMLS 2017 targeted towards document retrieval. To encourage research in Biomedical Named Entity Recognition and Linking, data splits for training and testing are included in the release, and a baseline model and its metrics for entity linking are also described.

L'articolo introduce un nuovo corpus gold-standard di letteratura scientifica biomedica annotato manualmente con le menzioni dei concetti UMLS.

Dettaglia la costruzione di un dataset annotato manualmente che copre concetti biomedici che è più grande e coperto da un'ontologia più grande dei dataset precedenti.

Questo articolo utilizza MedMentions, un modello TaggerOne semi-Markov per il concept recognition e linking end-to-end su un insieme di abstract di Pubmed per etichettare i documenti con concetti/entità biomediche


In this paper we propose a Deep Autoencoder Mixture Clustering (DAMIC) algorithm. It is based on a mixture of deep autoencoders where each cluster is represented by an autoencoder. A clustering network transforms the data into another space and then selects one of the clusters. Next, the autoencoder associated with this cluster is used to reconstruct the data-point. The clustering algorithm jointly learns the nonlinear data representation and the set of autoencoders. The optimal clustering is found by minimizing the reconstruction loss of the mixture of autoencoder network. Unlike other deep clustering algorithms, no regularization term is needed to avoid data collapsing to a single point. Our experimental evaluations on image and text corpora show significant improvement over state-of-the-art methods.

Proponiamo un metodo di deep clustering dove al posto di un centroide ogni cluster è rappresentato da un autoencoder

Presenta un deep clustering basato su una miscela di autoencoder, dove i datapoint sono assegnati a un cluster basato sull'errore di rappresentazione se la rete di autoencoder fosse usata per rappresentarlo.

Un approccio di deep clustering che utilizza un framework con autoencoder per imparare un'embedding bidimensionale dei dati mentre si raggruppano i dati utilizzando una deep neural network.

Un metodo di deep clustering che rappresenta ogni cluster con diversi autoencoder, funziona in modo end-to-end, e può anche essere usato per clusterizzare nuovi dati in arrivo senza rifare l'intera procedura di clustering.


We propose a new Integral Probability Metric (IPM) between distributions: the Sobolev IPM. The Sobolev IPM compares the mean discrepancy of two distributions for functions (critic) restricted to a Sobolev ball defined with respect to a dominant measure mu. We show that the Sobolev IPM compares two distributions in high dimensions based on weighted conditional Cumulative Distribution Functions (CDF) of each coordinate on a leave one out basis. The Dominant measure mu plays a crucial role as it defines the support on which conditional CDFs are compared. Sobolev IPM can be seen as an extension of the one dimensional Von-Mises Cramer statistics to high dimensional distributions. We show how Sobolev IPM can be used to train Generative Adversarial Networks (GANs). We then exploit the intrinsic conditioning implied by Sobolev IPM in text generation. Finally we show that a variant of Sobolev GAN achieves competitive results in semi-supervised learning on CIFAR-10, thanks to the smoothness enforced on the critic by Sobolev GAN which relates to Laplacian regularization.

Definiamo una nuova metrica di probabilità integrale (Sobolev IPM) e mostriamo come può essere utilizzata per il training di GAN per la generazione di testi e l'apprendimento semi-supervisionato.

Suggerisce un nuovo schema di regolarizzazione per le GAN basato sulla norma di Sobolev, che misura le deviazioni tra le norme L2 delle derivate.

Gli autori forniscono un altro tipo di GAN usando il setting tipico di una GAN ma con una classe di funzione diversa, e forniscono una procedura per addestrare le GAN con quel tipo di classe di funzione.

L'articolo propone una diversa penalità del gradiente per i GAN critic che forza la norma quadratica attesa del gradiente ad essere uguale a 1


We propose in this paper a new approach to train the Generative Adversarial Nets (GANs) with a mixture of generators to overcome the mode collapsing problem. The main intuition is to employ multiple generators, instead of using a single one as in the original GAN. The idea is simple, yet proven to be extremely effective at covering diverse data modes, easily overcoming the mode collapsing problem and delivering state-of-the-art results. A minimax formulation was able to establish among a classifier, a discriminator, and a set of generators in a similar spirit with GAN. Generators create samples that are intended to come from the same distribution as the training data, whilst the discriminator determines whether samples are true data or generated by generators, and the classifier specifies which generator a sample comes from. The distinguishing feature is that internal samples are created from multiple generators, and then one of them will be randomly selected as final output similar to the mechanism of a probabilistic mixture model. We term our method Mixture Generative Adversarial Nets (MGAN). We develop theoretical analysis to prove that, at the equilibrium, the Jensen-Shannon divergence (JSD) between the mixture of generators’ distributions and the empirical data distribution is minimal, whilst the JSD among generators’ distributions is maximal, hence effectively avoiding the mode collapsing problem. By utilizing parameter sharing, our proposed model adds minimal computational cost to the standard GAN, and thus can also efficiently scale to large-scale datasets. We conduct extensive experiments on synthetic 2D data and natural image databases (CIFAR-10, STL-10 and ImageNet) to demonstrate the superior performance of our MGAN in achieving state-of-the-art Inception scores over latest baselines, generating diverse and appealing recognizable objects at different resolutions, and specializing in capturing different types of objects by the generators.

Proponiamo un nuovo approccio per addestrare le GAN con una mistura di generatori per superare il problema del mode collapse.

Affrontare il problema del mode collapse nelle GAN utilizzando una mistura vincolata di distribuzioni per il generatore e un classificatore ausiliario che predice la componente d'origine della mistura.

L'articolo propone una mistura di generatori per addestrare le GAN senza costi computazionali aggiuntivi

Gli autori presentano l'uso di MGAN, che mira a superare il problema del mode collapse mediante misture di generatori e che raggiunge risultati allo stato dell'arte


Allowing humans to interactively train artificial agents to understand language instructions is desirable for both practical and scientific reasons.   Though, given  the lack of sample efficiency in current learning methods, reaching this goal may require substantial research efforts. We introduce the BabyAI research platform, with the goal of supporting investigations towards including humans in the loop for grounded language learning. The BabyAI platform comprises an extensible suite of 19 levels of increasing difficulty. Each level gradually leads the agent towards acquiring a combinatorially rich synthetic language, which is a proper subset of English. The platform also provides a hand-crafted bot agent, which simulates a human teacher.   We report estimated amount of supervision required for training neural reinforcement and behavioral-cloning agents on some BabyAI levels. We put forward strong evidence that current deep learning methods are not yet sufficiently sample-efficient in the context of learning a language with compositional properties.

Presentiamo la piattaforma BabyAI per studiare l'efficienza dei dati dell'apprendimento del linguaggio con human in the loop

Presenta una piattaforma di ricerca con un bot nel loop per imparare a eseguire istruzioni linguistiche in cui il linguaggio ha strutture compositive

Introduce una piattaforma per l'apprendimento delle lingue grounded che sostituisce qualsiasi human in the loop con un insegnante euristico e utilizza una lingua sintetica mappata in un 2D grid world


Recently, there has been growing interest in methods that perform neural network compression, namely techniques that attempt to substantially reduce the size of a neural network without significant reduction in performance. However, most existing methods are post-processing approaches in that they take a learned neural network as input and output a compressed network by either forcing several parameters to take the same value (parameter tying via quantization) or pruning irrelevant edges (pruning) or both. In this paper, we propose a novel algorithm that jointly learns and compresses a neural network. The key idea in our approach is to change the optimization criteria by adding $k$ independent Gaussian priors over the parameters and a sparsity penalty. We show that our approach is easy to implement using existing neural network libraries, generalizes L1 and L2 regularization and elegantly enforces parameter tying as well as pruning constraints. Experimentally, we demonstrate that our new algorithm yields state-of-the-art compression on several standard benchmarks with minimal loss in accuracy while requiring little to no hyperparameter tuning as compared with related, competing approaches.

Un prior k-means combinato con la regolarizzazione L1 produce risultati di compressione allo stato dell'arte.

Questo articolo esplora la soft parameter tying e la compressione di DNN/CNN


The application of stochastic variance reduction to optimization has shown remarkable recent theoretical and practical success. The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem. We show that naive application of the SVRG technique and related approaches fail, and explore why.

Il metodo SVRG fallisce sui moderni problemi di deep learning

Questo articolo presenta un'analisi dei metodi in stile SVRG, mostrando che dropout, batch norm, data augmention (crop/rotazioni/traslazioni casuali) tendono ad aumentare il bias e/o la varianza degli update.

Questo articolo studia l'applicabilità di SVGD alle moderne reti neurali e mostra che la semplice applicazione di SVGD tipicamente fallisce.


The ground-breaking performance obtained by deep convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks. One of the main challenge in applying CNNs to 3D shape analysis is how to define a natural convolution operator on non-euclidean surfaces. In this paper, we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere. A cascade set of geodesic disk filters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks. We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance, and its effectiveness is evaluated in applications of non-rigid/ rigid shape classification and shape retrieval.

Un metodo per applicare il deep learning alle superfici 3D usando i loro descrittori sferici e la convoluzione anisotropa alt-az su 2-sfere.

Presenta uno schema di convoluzione polare anisotropa su una sfera unitaria sostituendo la traslazione del filtro con la rotazione del filtro.

Questo articolo esplora il deep learning di forme 3D utilizzando la convoluzione anisotropa a 2 sfere di alt-az


Recent breakthroughs in computer vision make use of large deep neural networks, utilizing the substantial speedup offered by GPUs. For applications running on limited hardware, however, high precision real-time processing can still be a challenge.   One approach to solving this problem is training networks with binary or ternary weights, thus removing the need to calculate multiplications and significantly reducing memory size. In this work, we introduce LR-nets (Local reparameterization networks), a new method for training neural networks with discrete weights using stochastic parameters. We show how a simple modification to the local reparameterization trick, previously used to train Gaussian distributed weights, enables the training of discrete weights. Using the proposed training we test both binary and ternary models on MNIST, CIFAR-10 and ImageNet benchmarks and reach state-of-the-art results on most experiments.

Il training di reti binarie/ternarie usando la riparametrizzazione locale con l'approssimazione CLT

Addestra le reti di distribuzione dei pesi binarie e ternarie usando la backpropagation per campionare le preattivazioni dei neuroni con il reparameterization trick

Questo articolo suggerisce l'uso di parametri stocastici in combinazione con il reparameterization trick locale per addestrare reti neurali con pesi binari o ternari, che porta a risultati allo stato dell'arte.


We present Optimal Completion Distillation (OCD), a training procedure for optimizing sequence to sequence models based on edit distance. OCD is efficient, has no hyper-parameters of its own, and does not require pre-training or joint optimization with conditional log-likelihood. Given a partial sequence generated by the model, we first identify the set of optimal suffixes that minimize the total edit distance, using an efficient dynamic programming algorithm.   Then, for each position of the generated sequence, we use a target distribution which puts equal probability on the first token of all the optimal suffixes. OCD achieves the state-of-the-art performance on end-to-end speech recognition, on both Wall Street Journal and Librispeech datasets, achieving $9.3\%$ WER and $4.5\%$ WER, respectively.

Optimal Completion Distillation (OCD) è una procedura di training per ottimizzare i modelli sequence-to-sequence basati sulla edit distance che raggiunge lo stato dell'arte nei task di riconoscimento vocale end-to-end.

Approccio alternativo al training di modelli seq2seq utilizzando un programma dinamico per calcolare le prosecuzioni ottimali dei prefissi previsti

Un algoritmo di training per modelli auto-regressivi che non richiede alcun pre-training MLE e può ottimizzare direttamente dal sampling.

L'articolo considera una lacuna dei modelli sequence to sequence addestrati usando la stima di massima verosimiglianza e propone un approccio basato sulle edit distance e l'uso implicito di sequenze di label date durante il training.


As an emerging field, federated learning has recently attracted considerable attention.  Compared to distributed learning in the datacenter setting, federated learning  has more strict constraints on computate efficiency of the learned model and communication  cost during the training process. In this work, we propose an efficient  federated learning framework based on variational dropout. Our approach is able  to jointly learn a sparse model while reducing the amount of gradients exchanged  during the iterative training process. We demonstrate the superior performance  of our approach on achieving significant model compression and communication  reduction ratios with no accuracy loss.

Un modello congiunto e un metodo di sparsificazione del gradiente per federated learning

Applica il dropout variazionale per ridurre il costo di comunicazione nel training distribuito delle reti neurali, e fa esperimenti sui dataset mnist, cifar10 e svhn.

Gli autori propongono un algoritmo che riduce i costi di comunicazione nel federated learning inviando gradienti sparsi dal dispositivo al server e viceversa.

Combina l'algoritmo di ottimizzazione distribuita con il dropout variazionale per sparsificare i gradienti inviati al server principale dai learner locali.


We prove a multiclass boosting theory for the ResNet architectures which simultaneously creates a new technique for multiclass boosting and provides a new algorithm for ResNet-style architectures.   Our proposed training algorithm, BoostResNet, is particularly suitable in non-differentiable architectures.   Our method only requires the relatively inexpensive sequential training of T "shallow ResNets". We prove that the training error decays exponentially with the depth T if the weak module classifiers that we train perform slightly better than some weak baseline.   In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition.   A generalization error bound based on margin theory is proved and suggests that ResNet could be resistant to overfitting using a network with l_1 norm bounded weights.

Dimostriamo una teoria di boosting multiclasse per le architetture ResNet che crea contemporaneamente una nuova tecnica per il boosting multiclasse e fornisce un nuovo algoritmo per le architetture in stile ResNet.

Presenta un algoritmo in stile boosting per il training di deep residual network, un'analisi di convergenza per l'errore di training e un'analisi della capacità di generalizzazione.

Un metodo di apprendimento per ResNet utilizzando il framework di boosting che decompone l'apprendimento di reti complesse e prevede meno costi computazionali.

Gli autori propongono la deep ResNet come algoritmo di boosting, e sostengono che questo è più efficiente della backpropagation standard end-to-end.


We consider the problem of learning a one-hidden-layer neural network: we assume the input x is from Gaussian distribution and the label $y = a \sigma(Bx) + \xi$, where a is a nonnegative vector and  $B$ is a full-rank weight matrix, and $\xi$ is a noise vector. We first give an analytic formula for the population risk of the standard squared loss and demonstrate that it implicitly attempts to decompose a sequence of low-rank tensors simultaneously.  	  Inspired by the formula, we design a non-convex objective function $G$ whose landscape is guaranteed to have the following properties:	  1. All local minima of $G$ are also global minima.  2. All global minima of $G$ correspond to the ground truth parameters.  3. The value and gradient of $G$ can be estimated using samples. 	  With these properties, stochastic gradient descent on $G$ provably converges to the global minimum and learn the ground-truth parameters. We also prove finite sample complexity results and validate the results by simulations.

L'articolo analizza il panorama di ottimizzazione delle reti neurali con un hidden layer e progetta un nuovo obiettivo che non ha un minimo locale spurio.

Questo articolo studia il problema dell'apprendimento delle reti neurali con un hidden layer, stabilisce una connessione tra la least square population loss e i polinomi di Hermite, e propone una nuova loss.

Un metodo di fattorizzazione tensoriale per il learning di una rete neurale a un hidden layer


Open information extraction (OIE) systems extract relations and their   arguments from natural language text in an unsupervised manner. The resulting   extractions are a valuable resource for downstream tasks such as knowledge   base construction, open question answering, or event schema induction. In this   paper, we release, describe, and analyze an OIE corpus called OPIEC, which was   extracted from the text of English Wikipedia. OPIEC complements the available   OIE resources: It is the largest OIE corpus publicly available to date (over   340M triples) and contains valuable metadata such as provenance information,   confidence scores, linguistic annotations, and semantic annotations including   spatial and temporal information. We analyze the OPIEC corpus by comparing its   content with knowledge bases such as DBpedia or YAGO, which are also based on   Wikipedia. We found that most of the facts between entities present in OPIEC   cannot be found in DBpedia and/or YAGO, that OIE facts    often differ in the level of specificity compared to knowledge base facts, and   that OIE open relations are generally highly polysemous. We believe that the   OPIEC corpus is a valuable resource for future research on automated knowledge   base construction.

Un Open Information Extraction Corpus e la sua analisi approfondita

Costruisce un nuovo corpus per information extraction che è più grande dei corpora pubblici precedenti e contiene informazioni non esistenti nei corpora attuali.

Presenta un dataset di triple open-IE che sono state raccolte da Wikipedia con l'aiuto di un sistema di estrazione recente.

L'articolo descrive la creazione di un corpus Open IE su Wikipedia inglese attraverso un modo automatico


The process of designing neural architectures requires expert knowledge and extensive trial and error.  While automated architecture search may simplify these requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in both flexibility and components.  We propose a domain-specific language (DSL) for use in automated architecture search which can produce novel RNNs of arbitrary depth and width.  The DSL is flexible enough to define standard architectures such as the Gated Recurrent Unit and Long Short Term Memory and allows the introduction of non-standard RNN components such as trigonometric curves and layer normalization.   Using two different candidate generation techniques, random search with a ranking function and reinforcement learning,  we explore the novel architectures produced by the RNN DSL for language modeling and machine translation domains.  The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed.

Definiamo un DSL flessibile per la generazione di architetture RNN che permette RNN di varie dimensioni e complessità e proponiamo una funzione di classificazione che rappresenta RNN come reti neurali ricorsive, simulando le loro prestazioni per decidere le architetture più promettenti.

Introduce un nuovo metodo per generare architetture RNN utilizzando un linguaggio specifico del dominio per due tipi di generatori (casuale e basato su RL) insieme a una funzione di ranking e un valutatore.

Questo articolo presenta la ricerca di buone architetture per RNN Cell come un problema di ottimizzazione black-box in cui gli esempi sono rappresentati come un albero di operatori e valutati sulla base di funzioni apprese o generate da un agente RL.

Questo articolo indaga la strategia di meta-learning per la ricerca automatica dell'architettura nel contesto di RNN utilizzando un DSL che specifica le operazioni ricorrenti RNN.


Researches on deep neural networks with discrete parameters and their deployment in embedded systems have been active and promising topics. Although previous works have successfully reduced precision in inference, transferring both training and inference processes to low-bitwidth integers has not been demonstrated simultaneously. In this work, we develop a new method termed as ``"WAGE" to discretize both training and inference, where weights (W), activations (A), gradients (G) and errors (E) among layers are shifted and linearly constrained to low-bitwidth integers. To perform pure discrete dataflow for fixed-point devices, we further replace batch normalization by a constant scaling layer and simplify other components that are arduous for integer implementation. Improved accuracies can be obtained on multiple datasets, which indicates that WAGE somehow acts as a type of regularization. Empirically, we demonstrate the potential to deploy training in hardware systems such as integer-based deep learning accelerators and neuromorphic chips with comparable accuracy and higher energy efficiency, which is crucial to future AI applications in variable scenarios with transfer and continual learning demands.

Applichiamo il training e l'inferenza solo con numeri interi a bassa larghezza di bit nelle DNN

Un metodo chiamato WAGE che quantizza tutti gli operandi e gli operatori in una rete neurale per ridurre il numero di bit di rappresentazione in una rete.

Gli autori propongono pesi, attivazioni, gradienti ed errori discretizzati sia in fase di allenamento che di test sulle reti neurali


Modern Convolutional Neural Networks (CNNs) are complex, encompassing millions of parameters. Their deployment exerts computational, storage and energy demands, particularly on embedded platforms. Existing approaches to prune or sparsify CNNs require retraining to maintain inference accuracy. Such retraining is not feasible in some contexts. In this paper, we explore the sparsification of CNNs by proposing three model-independent methods. Our methods are applied on-the-fly and require no retraining. We show that the state-of-the-art models' weights can be reduced by up to 73% (compression factor of 3.7x) without incurring more than 5% loss in Top-5 accuracy. Additional fine-tuning gains only 8% in sparsity, which indicates that our fast on-the-fly methods are effective.

In questo articolo, sviluppiamo metodi di sparsificazione veloci e privi di retraining che possono essere impiegati per la sparsificazione on-the-fly delle CNN in molti contesti industriali.

Questo articolo propone approcci per il pruning delle CNN senza retraining, introducendo tre schemi per determinare le soglie sui pesi per il pruning.

Questo articolo descrive un metodo per la sparsificazione delle CNN senza retraining.


Curriculum learning and Self paced learning are popular topics in the machine learning that suggest to put the training samples in order by considering their difficulty levels. Studies in these topics show that starting with a small training set and adding new samples according to difficulty levels improves the learning performance. In this paper we experimented that we can also obtain good results by adding the samples randomly without a meaningful order. We compared our method with classical training, Curriculum learning, Self paced learning and their reverse ordered versions. Results of the statistical tests show that the proposed method is better than classical method and similar with the others. These results point a new training regime that removes the process of difficulty level determination in Curriculum and Self paced learning and as successful as these methods.

Proponiamo che il training con set crescenti stage-by-stage fornisce un'ottimizzazione per le reti neurali.

Gli autori paragonano il curriculum learning all'apprendimento in un ordine casuale con fasi che aggiungono un nuovo sample di esempi all'insieme precedentemente costruito in modo casuale

Questo articolo studia l'influenza dell'ordinamento nel Curriculum e Self paced learning, e mostra che in una certa misura l'ordinamento delle istanze di training non è importante.


We study the problem of learning to map, in an unsupervised way, between domains $A$ and $B$, such that the samples $\vb \in B$ contain all the information that exists in samples $\va\in A$ and some additional information. For example, ignoring occlusions, $B$ can be people with glasses, $A$ people without, and the glasses, would be the added information. When mapping a sample $\va$ from the first domain to the other domain, the missing information is replicated from an independent reference sample $\vb\in B$. Thus, in the above example, we can create, for every person without glasses a version with the glasses observed in any face image.    Our solution employs a single two-pathway encoder and a single decoder for both domains. The common part of the two domains and the separate part are encoded as two vectors, and the separate part is fixed at zero for domain $A$. The loss terms are minimal and involve reconstruction losses for the two domains and a domain confusion term. Our analysis shows that under mild assumptions, this architecture, which is much simpler than the literature guided-translation methods, is enough to ensure disentanglement between the two domains. We present convincing results in a few visual domains, such as no-glasses to glasses, adding facial hair based on a reference image, etc.

Un metodo di traduzione image to image che aggiunge a un'immagine il contenuto di un'altra creando così una nuova immagine.

Questo articolo affronta il task del transfer di contenuti, con una nuova loss.


Mathematical reasoning---a core ability within human intelligence---presents some unique challenges as a domain: we do not come to understand and solve mathematical problems primarily on the back of experience and evidence, but on the basis of inferring, learning, and exploiting laws, axioms, and symbol manipulation rules. In this paper, we present a new challenge for the evaluation (and eventually the design) of neural architectures and similar system, developing a task suite of mathematics problems involving sequential questions and answers in a free-form textual input/output format. The structured nature of the mathematics domain, covering arithmetic, algebra, probability and calculus, enables the construction of training and test spits designed to clearly illuminate the capabilities and failure-modes of different architectures, as well as evaluate their ability to compose and relate knowledge and learned processes. Having described the data generation process and its potential future expansions, we conduct a comprehensive analysis of models from two broad classes of the most powerful sequence-to-sequence architectures and find notable differences in their ability to resolve mathematical problems and generalize their knowledge.

Un dataset per testare il ragionamento matematico (e la generalizzazione algebrica), e i risultati sugli attuali modelli sequence-to-sequence.

Presenta un nuovo dataset sintetico per valutare la capacità di ragionamento matematico dei modelli sequence-to-sequence, e lo usa per valutare diversi modelli.

Modello per risolvere problemi matematici di base.


Convolutional Neural Networks (CNNs) filter the input data using a series of spatial convolution operators with compactly supported stencils and point-wise nonlinearities.  Commonly, the convolution operators couple features from all channels.  For wide networks, this leads to immense computational cost in the training of and prediction with CNNs.  In this paper, we present novel ways to parameterize the convolution more efficiently, aiming to decrease the number of parameters in CNNs and their computational complexity.  We propose new architectures that use a sparser coupling between the channels and thereby reduce both the number of trainable weights and the computational cost of the CNN.  Our architectures arise as new types of residual neural network (ResNet) that can be seen as discretizations of a Partial Differential Equations (PDEs) and thus have predictable theoretical properties. Our first architecture involves a convolution operator with a special sparsity structure, and is applicable to a large class of CNNs. Next, we present an architecture that can be seen as a discretization of a diffusion reaction PDE, and use it with three different convolution operators. We outline in our experiments that the proposed architectures,  although considerably reducing the number of trainable weights, yield comparable accuracy to existing CNNs that are fully coupled in the channel dimension.

Questo articolo introduce parametrizzazioni efficienti ed economiche di reti neurali convoluzionali motivate da equazioni differenziali parziali

Introduce quattro alternative "a basso costo" all'operazione di convoluzione standard che possono essere usate al posto dell'operazione di convoluzione standard per ridurre la loro complessità computazionale.

Questo articolo introduce metodi per ridurre il costo computazionale delle implementazioni CNN, e introduce nuove parametrizzazioni di CNN come architetture che limitano il parameter coupling.

L'articolo propone una prospettiva basata su PDE per comprendere e parametrizzare le CNN


In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equivalently a lower bound on the negative log likelihood) that the model can assign to some data as one varies the prior and/or the likelihood function in a latent variable model. The core of our contribution is to formally show that the problem of optimizing priors in latent variable models is exactly an instance of the variational optimization problem that information theorists solve when computing rate-distortion functions, and then to use this to derive a lower bound on negative log likelihood. Moreover, we will show that if changing the prior can improve the log likelihood, then there is a way to change the likelihood function instead and attain the same log likelihood, and thus rate-distortion theory is of relevance to both optimizing priors as well as optimizing likelihood functions. We will experimentally argue for the usefulness of quantities derived from rate-distortion theory in latent variable modeling by applying them to a problem in image modeling.

Utilizzare la teoria della distorsione del tasso per dare un bound su quanto un modello con variabili latenti può essere migliorato

Affronta i problemi di ottimizzazione del prior nel modello con variabili latenti e la selezione della funzione di verosimiglianza proponendo criteri basati su un limite inferiore della log-likelihood negativa.

Presenta un teorema che dà un limite inferiore alla likelihood negativa del tasso di distorsione per la modellazione delle variabili latenti

Gli autori sostengono che la teoria della distorsione del tasso per la compressione lossy fornisce uno strumento naturale per studiare i modelli con variabili latenti e propongono un lower bound.


Backprop is the primary learning algorithm used in many machine learning algorithms. In practice, however, Backprop in deep neural networks is a highly sensitive learning algorithm and successful learning depends on numerous conditions and constraints. One set of constraints is to avoid weights that lead to saturated units. The motivation for avoiding unit saturation is that gradients vanish and as a result learning comes to a halt. Careful weight initialization and re-scaling schemes such as batch normalization ensure that input activity to the neuron is within the linear regime where gradients are not vanished and can flow. Here we investigate backpropagating error terms only linearly. That is, we ignore the saturation that arise by ensuring gradients always flow. We refer to this learning rule as Linear Backprop since in the backward pass the network appears to be linear. In addition to ensuring persistent gradient flow, Linear Backprop is also favorable when computation is expensive since gradients are never computed. Our early results suggest that learning with Linear Backprop is competitive with Backprop and saves expensive gradient computations.

Ignoriamo le non linearità e non calcoliamo i gradienti nel backward pass per risparmiare calcoli e per assicurare che i gradienti scorrano sempre.

L'autore ha proposto algoritmi di backprop lineare per garantire il flusso dei gradienti per tutte le parti durante la backpropagation.


Deep neural networks with discrete latent variables offer the promise of better symbolic reasoning, and learning  abstractions that are more useful to new tasks. There has been a surge in interest in discrete latent variable models,  however, despite several recent improvements, the training of discrete latent variable models has remained  challenging and their performance has mostly failed to match their continuous counterparts. Recent work on vector quantized autoencoders (VQ-VAE) has made substantial progress in this direction, with its perplexity almost matching that of a VAE on datasets such as CIFAR-10. In this work, we investigate an alternate training technique for VQ-VAE, inspired by its connection to the Expectation Maximization (EM) algorithm. Training the discrete autoencoder with EM and combining it with sequence  level knowledge distillation alows us to develop a non-autoregressive machine translation model whose accuracy almost matches a strong greedy autoregressive baseline Transformer, while being 3.3 times faster at inference.

Capire l'autoencoder discreto VQ-VAE sistematicamente usando EM e usarlo per progettare un modello di traduzione non autogressivo che corrisponde a una baseline autoregressiva forte.

Questo articolo introduce un nuovo modo di interpretare il VQ-VAE e propone un nuovo algoritmo di training basato sul soft EM clustering.

L'articolo presenta una visione alternativa sulla procedura di training per il VQ-VAE utilizzando l'algoritmo soft EM


Recent research about margin theory has proved that maximizing the minimum margin like support vector machines does not necessarily lead to better performance, and instead, it is crucial to optimize the margin distribution. In the meantime, margin theory has been used to explain the empirical success of deep network in recent studies. In this paper, we present ODN (the Optimal margin Distribution Network), a network which embeds a loss function in regard to the optimal margin distribution. We give a theoretical analysis for our method using the PAC-Bayesian framework, which confirms the significance of the margin distribution for classification within the framework of deep networks. In addition, empirical results show that the ODN model always outperforms the baseline cross-entropy loss model consistently across different regularization situations. And our ODN  model also outperforms the cross-entropy loss (Xent), hinge loss and soft hinge loss model in generalization task through limited training data.

Questo articolo presenta una deep neural network che incorpora una loss per quanto riguarda la distribuzione ottimale del margine, che allevia il problema dell'overfitting teoricamente ed empiricamente.

Presenta un limite PAC-bayesiano per una margin loss


Deep network compression seeks to reduce the number of parameters in the network while maintaining a certain level of performance.   Deep network distillation seeks to train a smaller network that matches soft-max performance of a larger network.   While both regimes have led to impressive performance for their respective goals, neither provide insight into the importance of a given layer in the original model, which is useful if we are to improve our understanding of these highly parameterized models.   In this paper, we present the concept of deep net triage, which individually assesses small blocks of convolution layers to understand their collective contribution to the overall performance, which we call \emph{criticality}.  We call it triage because we assess this criticality by answering the question: what is the impact to the health of the overall network if we compress a block of layers into a single layer.  We propose a suite of triage methods and compare them on problem spaces of varying complexity.   We ultimately show that, across these problem spaces, deep net triage is able to indicate the of relative importance of different layers.   Surprisingly, our local structural compression technique also leads to an improvement in overall accuracy when the final model is fine-tuned globally.

Cerchiamo di capire le rappresentazioni apprese nelle reti compresse attraverso un regime sperimentale che chiamiamo deep net triage

Confronta vari metodi di inizializzazione e di training per trasferire la conoscenza dalla rete VGG a una student network più piccola, sostituendo blocchi di layer con layer singoli.

Questo articolo presenta cinque metodi per fare il triage o la compressione dei blocchi di layer per le deep network.

L'articolo propone un metodo per comprimere un blocco di layer in una NN che valuta diversi sotto-approcci


In this paper, we show a phenomenon, which we named ``super-convergence'', where residual networks can be trained using an order of magnitude fewer iterations than is used with standard training methods.    The existence of super-convergence is relevant to understanding why deep networks generalize well.   One of the key elements of super-convergence is training with cyclical learning rates and a large maximum learning rate.   Furthermore, we present evidence that training with large learning rates improves performance by regularizing the network. In addition, we show that super-convergence provides a  greater boost in performance relative to standard training when the amount of labeled training data is limited.   We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate.   The architectures to replicate this work will be made available upon publication.

La prova empirica di un nuovo fenomeno richiede nuove intuizioni teoriche ed è rilevante per le discussioni attive nella letteratura su SGD e la comprensione della generalizzazione.

L'articolo discute un fenomeno per cui il training delle reti neurali in setting molto specifici può trarre molto profitto da una schedule che include grandi learning rate

Gli autori analizzano il training delle residual network usando grandi learning rate ciclici, e dimostrano una convergenza veloce con i learning rate ciclici e la prova empirica che grandi learning rate posso fornire regolarizzazione.


Infinite-width neural networks have been extensively used to study the theoretical properties underlying the extraordinary empirical success of standard, finite-width neural networks. Nevertheless, until now, infinite-width networks have been limited to at most two hidden layers. To address this shortcoming, we study the initialisation requirements of these networks and show that the main challenge for constructing them is defining the appropriate sampling distributions for the weights. Based on these observations, we propose a principled approach to weight initialisation that correctly accounts for the functional nature of the hidden layer activations and facilitates the construction of arbitrarily many infinite-width layers, thus enabling the construction of arbitrarily deep infinite-width networks. The main idea of our approach is to iteratively reparametrise the hidden-layer activations into appropriately defined reproducing kernel Hilbert spaces and use the canonical way of constructing probability distributions over these spaces for specifying the required weight distributions in a principled way. Furthermore, we examine the practical implications of this construction for standard, finite-width networks. In particular, we derive a novel weight initialisation scheme for standard, finite-width networks that takes into account the structure of the data and information about the task at hand. We demonstrate the effectiveness of this weight initialisation approach on the MNIST, CIFAR-10 and Year Prediction MSD datasets.

Proponiamo un metodo per la costruzione di reti di larghezza infinita arbitrariamente profonde, sulla base del quale deriviamo un nuovo schema di inizializzazione dei pesi per le reti di larghezza finita e dimostriamo le sue prestazioni competitive.

Propone un approccio di inizializzazione dei pesi per consentire reti infinitamente profonde e di larghezza infinita con risultati sperimentali su piccoli dataset.

Propone deep neural network di larghezza infinita.


Working memory requires information about external stimuli to be represented in the brain even after those stimuli go away. This information is encoded in the activities of neurons, and neural activities change over timescales of tens of milliseconds. Information in working memory, however, is retained for tens of seconds, suggesting the question of how time-varying neural activities maintain stable representations. Prior work shows that, if the neural dynamics are in the `  null space' of the representation - so that changes to neural activity do not affect the downstream read-out of stimulus information - then information can be retained for periods much longer than the time-scale of individual-neuronal activities. The prior work, however, requires precisely constructed synaptic connectivity matrices, without explaining how this would arise in a biological neural network. To identify mechanisms through which biological networks can self-organize to learn  memory function, we derived biologically plausible synaptic plasticity rules that dynamically modify the connectivity matrix to enable information storing. Networks implementing this plasticity rule can successfully learn to form memory representations even if only 10% of the synapses are plastic, they are robust to synaptic noise, and they can represent information about multiple stimuli.

Abbiamo derivato regole di apprendimento della synaptic plasticity biologicamente plausibili per una rete neurale ricorrente per memorizzare le rappresentazioni degli stimoli.

Un modello di rete neurale composto da neuroni connessi in modo ricorrente e uno o più redout che mira a mantenere un certo output nel tempo.

Questo articolo presenta un meccanismo di memoria auto-organizzante in un modello neurale, e introduce una funzione obiettivo che minimizza i cambiamenti nel segnale da memorizzare.


Generative Adversarial Networks (GANs) have been proposed as an approach to learning generative models. While GANs have demonstrated promising performance on multiple vision tasks, their learning dynamics are not yet well understood, neither in theory nor in practice. In particular, the work in this domain has been focused so far only on understanding the properties of the stationary solutions that this dynamics might converge to, and of the behavior of that dynamics in this solutions’ immediate neighborhood.   To address this issue, in this work we take a first step towards a principled study of the GAN dynamics itself. To this end, we propose a model that, on one hand, exhibits several of the common problematic convergence behaviors (e.g., vanishing gradient, mode collapse, diverging or oscillatory behavior), but on the other hand, is sufficiently simple to enable rigorous convergence analysis.   This methodology enables us to exhibit an interesting phenomena: a GAN with an optimal discriminator provably converges, while guiding the GAN training using only a first order approximation of the discriminator leads to unstable GAN dynamics and mode collapse. This suggests that such usage of the first order approximation of the discriminator, which is a de-facto standard in all the existing GAN dynamics, might be one of the factors that makes GAN training so challenging in practice. Additionally, our convergence result constitutes the first rigorous analysis of a dynamics of a concrete parametric GAN.

Per comprendere il training delle GAN, definiamo una semplice dinamica GAN e mostriamo le differenze quantitative tra gli aggiornamenti ottimali e quelli del primo ordine in questo modello.

Gli autori studiano l'impatto delle GAN in setting in cui ad ogni iterazione il discriminatore si allena fino alla convergenza e il generatore si aggiorna con gradient step, o dove pochi gradient step sono fatti per il disciminatore e il generatore.

Questo articolo studia la dinamica dell'adversarial training delle GAN su un modello a mistura gaussiana


The machine learning and computer vision community is witnessing an unprecedented rate of new tasks being proposed and addressed, thanks to the power of deep convolutional networks to find complex mappings from X to Y. The advent of each task often accompanies the release of a large-scale human-labeled dataset, for supervised training of the deep network. However, it is expensive and time-consuming to manually label sufficient amount of training data. Therefore, it is important to develop algorithms that can leverage off-the-shelf labeled dataset to learn useful knowledge for the target task. While previous works mostly focus on transfer learning from a single source, we study multi-source transfer across domains and tasks (MS-DTT), in a semi-supervised setting. We propose GradMix, a model-agnostic method applicable to any model trained with gradient-based learning rule. GradMix transfers knowledge via gradient descent, by weighting and mixing the gradients from all sources during training. Our method follows a meta-learning objective, by assigning layer-wise weights to the source gradients, such that the combined gradient follows the direction that can minimize the loss for a small set of samples from the target dataset. In addition, we propose to adaptively adjust the learning rate for each mini-batch based on its importance to the target task, and a pseudo-labeling method to leverage the unlabeled samples in the target domain. We perform experiments on two MS-DTT tasks: digit recognition and action recognition, and demonstrate the advantageous performance of the proposed method against multiple baselines.

Proponiamo un metodo basato sul gradiente per trasferire la conoscenza da più fonti attraverso diversi domini e task.

Questo articolo propone di combinare i gradienti dei domini di origine per aiutare l'apprendimento nel dominio target.


Bayesian phylogenetic inference is currently done via Markov chain Monte Carlo with simple mechanisms for proposing new states, which hinders exploration efficiency and often requires long runs to deliver accurate posterior estimates. In this paper we present an alternative approach: a variational framework for Bayesian phylogenetic analysis. We approximate the true posterior using an expressive graphical model for tree distributions, called a subsplit Bayesian network, together with appropriate branch length distributions. We train the variational approximation via stochastic gradient ascent and adopt multi-sample based gradient estimators for different latent variables separately to handle the composite latent space of phylogenetic models. We show that our structured variational approximations are flexible enough to provide comparable posterior estimation to MCMC, while requiring less computation due to a more efficient tree exploration mechanism enabled by variational inference. Moreover, the variational approximations can be readily used for further statistical analysis such as marginal likelihood estimation for model comparison via importance sampling. Experiments on both synthetic data and real data Bayesian phylogenetic inference problems demonstrate the effectiveness and efficiency of our methods.

La prima formulazione Bayes variazionale dell'inferenza filogenetica, un difficile problema di inferenza su strutture con componenti discrete e continue intrecciate

Esplora una soluzione di inferenza approssimata al problema dell'inferenza bayesiana degli alberi filogenetici sfruttando le reti bayesiane subsplit recentemente proposte e i moderni stimatori di gradiente per VI.

Propone un approccio variazionale all'inferenza posteriore bayesiana negli alberi filogenetici.


This paper introduces HybridNet, a hybrid neural network to speed-up autoregressive  models for raw audio waveform generation. As an example, we propose  a hybrid model that combines an autoregressive network named WaveNet and a  conventional LSTM model to address speech synthesis. Instead of generating  one sample per time-step, the proposed HybridNet generates multiple samples per  time-step by exploiting the long-term memory utilization property of LSTMs. In  the evaluation, when applied to text-to-speech, HybridNet yields state-of-art performance.  HybridNet achieves a 3.83 subjective 5-scale mean opinion score on  US English, largely outperforming the same size WaveNet in terms of naturalness  and provide 2x speed up at inference.

È un'architettura neurale ibrida per accelerare il modello autoregressivo.

Conclude che per scalare la dimensione del modello senza aumentare il tempo di inferenza per la predizione sequenziale si possa usare un modello che predice più timestep contemporaneamente.

Questo articolo presenta HybridNet, un sistema di sintesi neurale del parlato e di altri audio che combina il modello WaveNet con una LSTM con l'obiettivo di offrire un modello con una generazione audio avente un tempo di inferenza più veloce.


Visual Interpretation and explanation of deep models is critical towards wide adoption of systems that rely on them. In this paper, we propose a novel scheme for both interpretation as well as explanation in which, given a pretrained model, we automatically identify internal features relevant for the set of classes considered by the model, without relying on additional annotations. We interpret the model through average visualizations of this reduced set of features. Then, at test time, we explain the network prediction by accompanying the predicted class label with supporting visualizations derived from the identified features. In addition, we propose a method to address the artifacts introduced by strided operations in deconvNet-based visualizations. Moreover, we introduce an8Flower , a dataset specifically designed for objective quantitative evaluation of methods for visual explanation. Experiments on the MNIST , ILSVRC 12, Fashion 144k and an8Flower datasets show that our method produces detailed explanations with good coverage of relevant features of the classes of interest.

Interpretazione identificando le feature apprese dal modello che servono come indicatori per il task di interesse. Si spiegano le decisioni del modello evidenziando la risposta di queste feature nei dati di test. Si valutano obiettivamente le spiegazioni con un dataset controllato.

Questo articolo propone un metodo per produrre spiegazioni visive per gli output delle deep neural network e rilascia un nuovo dataset sintetico.

Un metodo per deep neural network che identifica automaticamente le feature rilevanti dell'insieme delle classi, supportando l'interpretazione e la spiegazione senza fare affidamento su annotazioni aggiuntive.


In this work we propose a simple and efficient framework for learning sentence representations from unlabelled data. Drawing inspiration from the distributional hypothesis and recent work on learning sentence representations, we reformulate the problem of predicting the context in which a sentence appears as a classification problem. Given a sentence and the context in which it appears, a classifier distinguishes context sentences from other contrastive sentences based on their vector representations. This allows us to efficiently learn different types of encoding functions, and we show that the model learns high-quality sentence representations. We demonstrate that our sentence representations outperform state-of-the-art unsupervised and supervised representation learning methods on several downstream NLP tasks that involve understanding sentence semantics while achieving an order of magnitude speedup in training time.

Una framework per l'apprendimento efficiente di rappresentazioni di frasi di alta qualità.

Propone un algoritmo più veloce per l'apprendimento di rappresentazioni di frasi in stile SkipThought da corpora di frasi ordinate che sostituisce il decoder a livello di parola con una contrastive loss di classificazione.

Questo articolo propone un framework per l'apprendimento unsupervised di rappresentazioni di frasi massimizzando un modello di probabilità di frasi appartenenti al contesto rispetto a frasi candidate casuali


Many regularization methods have been proposed to prevent overfitting in neural networks. Recently, a regularization method has been proposed to optimize the variational lower bound of the Information Bottleneck Lagrangian. However, this method cannot be generalized to regular neural network architectures. We present the activation norm penalty that is derived from the information bottleneck principle and is theoretically grounded in a variation dropout framework. Unlike in previous literature, it can be applied to any general neural network. We demonstrate that this penalty can give consistent improvements to different state of the art architectures both in language modeling and image classification. We present analyses on the properties of this penalty and compare it to other methods that also reduce mutual information.

Ricaviamo una norm penalty sull'output della rete neurale dal punto di vista dell'information bottleneck

Propone Activation Norm Penalty, una regolarizzazione di tipo L_2 sulle attivazioni, derivandola dal principio di Information Bottleneck

Questo articolo crea una mappatura tra le norm penalty di attivazione e il framework dell'information bottleneck usando il framework di dropout variazionale.


Unsupervised learning of timeseries data is a challenging problem in machine learning. Here,  we propose a novel algorithm, Deep Temporal Clustering (DTC), a fully unsupervised method, to naturally integrate dimensionality reduction and temporal clustering into a single end to end learning framework. The algorithm starts with an initial cluster estimates using an autoencoder for dimensionality reduction and a novel temporal clustering layer for cluster assignment. Then it jointly optimizes the clustering objective and the dimensionality reduction objective. Based on requirement and application, the temporal clustering layer can be customized with any temporal similarity metric. Several similarity metrics are considered and compared.   To gain insight into features that the network has learned for its clustering, we apply a visualization method that generates a heat map of regions of interest in the timeseries. The viability of the algorithm is demonstrated using timeseries data from diverse domains, ranging from earthquakes to sensor data from spacecraft. In each case, we show that our algorithm outperforms traditional methods. This performance is attributed to fully integrated temporal dimensionality reduction and clustering criterion.

Un metodo completamente unsupervised, per integrare naturalmente la riduzione della dimensionalità e il clustering temporale in un unico framework di apprendimento end-to-end.

Propone un algoritmo che integra l'autoencoder con il clustering di dati di serie temporali usando una struttura di rete che si adatta ai dati di serie temporali.

Un algoritmo per eseguire congiuntamente la riduzione della dimensionalità e il clustering temporale in un contesto di deep learning, utilizzando un autoencoder e un obiettivo di clustering.

Gli autori hanno proposto un metodo di clustering unsupervised di serie temporali costruito con deep neural network e dotato di un encoder-decoder e una modalità di clustering per accorciare le serie temporali, estrarre le feature temporali locali e ottenere le rappresentazioni codificate.


We study many-class few-shot (MCFS) problem in both supervised learning and meta-learning scenarios. Compared to the well-studied many-class many-shot and few-class few-shot problems, MCFS problem commonly occurs in practical applications but is rarely studied. MCFS brings new challenges because it needs to distinguish between many classes, but only a few samples per class are available for training. In this paper, we propose ``memory-augmented hierarchical-classification network (MahiNet)'' for MCFS learning. It addresses the ``many-class'' problem by exploring the class hierarchy, e.g., the coarse-class label that covers a subset of fine classes, which helps to narrow down the candidates for the fine class and is cheaper to obtain. MahiNet uses a convolutional neural network (CNN) to extract features, and integrates a memory-augmented attention module with a multi-layer perceptron (MLP) to produce the probabilities over coarse and fine classes. While the MLP extends the linear classifier, the attention module extends a KNN classifier, both together targeting the ''`few-shot'' problem. We design different training strategies of MahiNet for supervised learning and meta-learning. Moreover, we propose two novel benchmark datasets ''mcfsImageNet'' (as a subset of ImageNet) and ''mcfsOmniglot'' (re-splitted Omniglot) specifically for MCFS problem. In experiments, we show that MahiNet outperforms several state-of-the-art models on MCFS classification tasks in both supervised learning and meta-learning scenarios.

Una rete neurale con memoria che affronta il problema many-class few-shot sfruttando la gerarchia delle classi sia nell'apprendimento supervisionato che nel meta-learning.

Questo articolo presenta metodi per aggiungere un bias induttivo a un classificatore attraverso la predizione coarse-to-fine lungo una gerarchia di classi e l'apprendimento di un classificatore KNN memory-based che tiene traccia delle istanze mal annotate durante l'apprendimento.

Questo articolo formula il problema di classificazione many-class few-shot da una prospettiva di apprendimento supervisionato e una prospettiva di meta-learning.


Learning a better representation with neural networks is a challenging problem, which has been tackled from different perspectives in the past few years. In this work, we focus on learning a representation that would be useful in a clustering task. We introduce two novel loss components that substantially improve the quality of produced clusters, are simple to apply to arbitrary models and cost functions, and do not require a complicated training procedure. We perform an extensive set of experiments, supervised and unsupervised, and evaluate the proposed loss components on two most common types of models, Recurrent Neural Networks and Convolutional Neural Networks, showing that the approach we propose consistently improves the quality of KMeans clustering in terms of mutual information scores and outperforms previously proposed methods.

Un nuovo componente di loss che costringe la rete ad apprendere una rappresentazione adatta al clustering durante l'allenamento per un task di classificazione.

Questo articolo propone due termini di regolarizzazione basati su una hinge loss sulla divergenza KL tra due argomenti di input normalizzati con softmax per incoraggiare l'apprendimento di rappresentazioni disgiunte

Proposta di due regolarizzatori destinati a rendere le rappresentazioni apprese nel penultimo layer di un classificatore più conformi alla struttura intrinseca dei dati.


In high dimensions, the performance of nearest neighbor algorithms depends crucially on structure in the data.  While traditional nearest neighbor datasets consisted mostly of hand-crafted feature vectors, an increasing number of datasets comes from representations learned with neural networks.  We study the interaction between nearest neighbor algorithms and neural networks in more detail.  We find that the network architecture can significantly influence the efficacy of nearest neighbor algorithms even when the classification accuracy is unchanged.  Based on our experiments, we propose a number of training modifications that lead to significantly better datasets for nearest neighbor algorithms.  Our modifications lead to learned representations that can accelerate nearest neighbor queries by 5x.

Mostriamo come ottenere buone rappresentazioni dal punto di vista della Similarity Search.

Studia l'impatto del cambiamento della parte di classificazione delle immagini a capo della DNN sulla capacità di indicizzare i descrittori con un LSH o un algoritmo kd-tree.

Propone di utilizzare la softmax cross-entropy loss per imparare una rete che cerca di ridurre gli angoli tra gli input e i vettori di classe corrispondenti in un framework supervisionato.


Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid. These are subsequently relaxed to continuous surrogates that can allow for efficient gradient-based optimization. We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent. We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classification.

Introduciamo una tecnica che permette il training basato sul gradiente delle reti neurali quantizzate.

Propone un modo unificato e generale di addestrare reti neurali con pesi e attivazioni sinaptiche quantizzate a precisione ridotta.

Un nuovo approccio alla quantizzazione delle attivazioni che è allo stato dell'arte o competitivo su diversi problemi di immagini reali.

Un metodo per l'apprendimento di reti neurali con pesi e attivazioni quantizzate quantizzando stocasticamente i valori e sostituendo la distribuzione categoriale risultante con un rilassamento continuo della stessa


In most current formulations of adversarial training, the discriminators can be expressed as single-input operators, that is, the mapping they define is separable over observations. In this work, we argue that this property might help explain the infamous mode collapse phenomenon in adversarially-trained generative models. Inspired by discrepancy measures and two-sample tests between probability distributions, we propose distributional adversaries that operate on samples, i.e., on sets of multiple points drawn from a distribution, rather than on single observations. We show how they can be easily implemented on top of existing models. Various experimental results show that generators trained in combination with our distributional adversaries are much more stable and are remarkably less prone to mode collapse than traditional models trained with observation-wise prediction discriminators. In addition, the application of our framework to domain adaptation results in strong improvement over recent state-of-the-art.

Mostriamo che il problema del mode collapse nelle GAN può essere spiegato da una mancanza di condivisione delle informazioni tra le osservazioni in un batch di training, e proponiamo un framework basato sulla distribuzione per la condivisione globale delle informazioni tra i gradienti che porta a un adversarial training più stabile ed efficace.

Propone di sostituire i discriminatori a sample singolo nell'adversarial training con discriminatori che operano esplicitamente su distribuzioni di esempi.

Teoria sui test a due sample e MMD e come possono essere beneficamente incorporati nel framework GAN.


Chemical information extraction is to convert chemical knowledge in text into true chemical database, which is a text processing task heavily relying on chemical compound name identification and standardization. Once a systematic name for a chemical compound is given, it will naturally and much simply convert the name into the eventually required molecular formula. However, for many chemical substances, they have been shown in many other names besides their systematic names which poses a great challenge for this task. In this paper, we propose a framework to do the auto standardization from the non-systematic names to the corresponding systematic names by using the spelling error correction, byte pair encoding tokenization and neural sequence to sequence model. Our framework is trained end to end and is fully data-driven. Our standardization accuracy on the test dataset achieves 54.04% which has a great improvement compared to previous state-of-the-art result.

Abbiamo progettato un framework end-to-end utilizzando il modello sequence to sequence per fare la standardizzazione dei nomi chimici.

Standardizza i nomi non sistematici nella information extraction chimica creando un corpus parallelo di nomi non sistematici e sistematici e costruendo un modello seq2seq.

Questo lavoro presenta un metodo per tradurre nomi non sistematici di composti chimici nei loro equivalenti sistematici utilizzando una combinazione di meccanismi


The training of deep neural networks with Stochastic Gradient Descent (SGD) with a large learning rate or a small batch-size typically ends in flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. This was found to correlate with a good final generalization performance.   In this paper we extend previous work by investigating the curvature of the loss surface along the whole training trajectory, rather than only at the endpoint. We find that initially SGD visits increasingly sharp regions, reaching a maximum sharpness determined by both the learning rate and the batch-size of SGD. At this peak value SGD starts to fail to minimize the loss along directions in the loss surface corresponding to the largest curvature (sharpest directions). To further investigate the effect of these dynamics in the training process, we study a variant of SGD using a reduced learning rate along the sharpest directions which we show can improve training speed while finding both sharper and better generalizing solution, compared to vanilla SGD. Overall, our results show that the SGD dynamics in the subspace of the sharpest directions influence the regions that SGD steers to (where larger learning rate or smaller batch size result in wider regions visited), the overall training speed, and the generalization ability of the final model.

SGD viene indirizzato all'inizio dell'training verso una regione in cui il suo step è troppo grande rispetto alla curvatura, il che ha un impatto sul resto del training.

Analizza la relazione tra la convergenza/generalizzazione e l'aggiornamento sui più grandi autovettori dell'Hessiana delle loss delle DNN.

Questo lavoro studia la relazione tra la step size SGD e la curvatura della superficie di loss


We introduce a new approach to estimate continuous actions using actor-critic algorithms for reinforcement learning problems. Policy gradient methods usually predict one continuous action estimate or parameters of a presumed distribution (most commonly Gaussian) for any given state which might not be optimal as it may not capture the complete description of the target distribution. Our approach instead predicts M actions with the policy network (actor) and then uniformly sample one action during training as well as testing at each state. This allows the agent to learn a simple stochastic policy that has an easy to compute expected return. In all experiments, this facilitates better exploration of the state space during training and converges to a better policy.

Introduciamo un nuovo algoritmo di reinforcement learning, che predice azioni multiple e fa sampling da esse.

Questo lavoro introduce una miscela uniforme di policy deterministiche, e trova che questa parametrizzazione delle policy stocastiche supera DDPG su diversi benchmark della OpenAI gym.

Gli autori studiano un metodo per migliorare le prestazioni delle reti addestrate con DDPG, e mostrano prestazioni migliorate su un gran numero di ambienti di controllo continuo standard.


Recently convolutional neural networks (CNNs) achieve great accuracy in visual recognition tasks. DenseNet becomes one of the most popular CNN models due to its effectiveness in feature-reuse. However, like other CNN models, DenseNets also face overfitting problem if not severer. Existing dropout method can be applied but not as effective due to the introduced nonlinear connections. In particular, the property of feature-reuse in DenseNet will be impeded, and the dropout effect will be weakened by the spatial correlation inside feature maps. To address these problems, we craft the design of a specialized dropout method from three aspects, dropout location, dropout granularity, and dropout probability. The insights attained here could potentially be applied as a general approach for boosting the accuracy of other CNN models with similar nonlinear connections. Experimental results show that DenseNets with our specialized dropout method yield better accuracy compared to vanilla DenseNet and state-of-the-art CNN models, and such accuracy boost increases with the model depth.

Rendendoci conto degli inconvenienti nell'applicazione del dropout originale su DenseNet, abbiamo realizzato il design di un metodo di dropout da tre aspetti, la cui idea potrebbe essere applicata anche ad altri modelli CNN.

Applicazione di diverse strutture e schedule di dropout binario con l'obiettivo specifico di regolarizzare l'architettura DenseNet.

Propone una tecnica di pre-dropout per densenet che implementa il dropout prima della funzione di attivazione non lineare.


While extremely successful in several applications, especially with low-level representations; sparse, noisy samples and structured domains (with multiple objects and interactions) are some of the open challenges in most deep models. Column Networks, a deep architecture, can succinctly capture such domain structure and interactions, but may still be prone to sub-optimal learning from sparse and noisy samples. Inspired by the success of human-advice guided learning in AI, especially in data-scarce domains, we propose Knowledge-augmented Column Networks that leverage human advice/knowledge for better learning with noisy/sparse samples. Our experiments demonstrate how our approach leads to either superior overall performance or faster convergence.

Guidare i deep model consapevoli delle relazioni verso un migliore apprendimento con la conoscenza umana.

Questo lavoro propone una variante della rete a colonne basata sull'iniezione della guida umana modificando i calcoli nella rete.

Un metodo per incorporare consigli umani nel deep learning estendendo Column Network, una graph neural network per la classificazione collettiva.


Recent research has shown that one can train a neural network with binary weights and activations at train time by augmenting the weights with a high-precision continuous latent variable that accumulates small changes from stochastic gradient descent. However, there is a dearth of work to explain why one can effectively capture the features in data with binary weights and activations. Our main result is that the neural networks with binary weights and activations trained using the method of Courbariaux, Hubara et al. (2016) work because of the high-dimensional geometry of binary vectors. In particular, the ideal continuous vectors that extract out features in the intermediate representations of these BNNs are well-approximated by binary vectors in the sense that dot products are approximately preserved. Compared to previous research that demonstrated good classification performance with BNNs, our work explains why these BNNs work in terms of HD geometry.   Furthermore, the results and analysis used on BNNs are shown to generalize to neural networks with ternary weights and activations. Our theory serves as a foundation for understanding not only BNNs but a variety of methods that seek to compress traditional neural networks. Furthermore, a better understanding of multilayer binary neural networks serves as a starting point for generalizing BNNs to other neural network architectures such as recurrent neural networks.

I recenti successi delle Reti Neurali Binarie possono essere compresi sulla base della geometria dei vettori binari ad alta densità

Indaga numericamente e teoricamente le ragioni dietro il successo empirico delle reti neurali binarizzate.

Questo articolo analizza l'efficacia delle reti neurali binarie e perché la binarizzazione è in grado di preservare le prestazioni del modello.


In recent years Convolutional Neural Networks (CNN) have been used extensively for Superresolution (SR). In this paper, we use inverse problem and sparse representation solutions to form a mathematical basis for CNN operations. We show how a single neuron is able to provide the optimum solution for inverse problem, given a low resolution image dictionary as an operator. Introducing a new concept called Representation Dictionary Duality, we show that CNN elements (filters) are trained to be representation vectors and then, during reconstruction, used as dictionaries. In the light of theoretical work, we propose a new algorithm which uses two networks with different structures that are separately trained with low and high coherency image patches and show that it performs faster compared to the state-of-the-art algorithms while not sacrificing from performance.

Dopo aver dimostrato che un neurone agisce come un risolutore di problemi inversi per la super-risoluzione ed è garantito che una rete di neuroni sia in grado di fornire una soluzione, abbiamo proposto un'architettura di rete doppia che performa più velocemente dello stato dell'arte.

Discute l'uso delle reti neurali per la super-risoluzione

Una nuova architettura per risolvere task di super-risoluzione delle immagini, e un'analisi che mira a stabilire una connessione tra CNN per risolvere la super-risoluzione e risolvere problemi inversi regolarizzati sparsi.


We consider the learning of algorithmic tasks by mere observation of input-output  pairs. Rather than studying this as a black-box discrete regression problem with  no assumption whatsoever on the input-output mapping, we concentrate on tasks  that are amenable to the principle of divide and conquer, and study what are its  implications in terms of learning.  This principle creates a powerful inductive bias that we leverage with neural  architectures that are defined recursively and dynamically, by learning two scale-  invariant atomic operations: how to split a given input into smaller sets, and how  to merge two partially solved tasks into a larger partial solution. Our model can be  trained in weakly supervised environments, namely by just observing input-output  pairs, and in even weaker environments, using a non-differentiable reward signal.  Moreover, thanks to the dynamic aspect of our architecture, we can incorporate  the computational complexity as a regularization term that can be optimized by  backpropagation. We demonstrate the flexibility and efficiency of the Divide-  and-Conquer Network on several combinatorial and geometric tasks: convex hull,  clustering, knapsack and euclidean TSP. Thanks to the dynamic programming  nature of our model, we show significant improvements in terms of generalization  error and computational complexity.

Modello dinamico che impara strategie di divide et impera tramite una supervisione debole.

Propone di aggiungere un nuovo bias induttivo all'architettura delle reti neurali utilizzando una strategia divide et impera.

Questo articolo studia i problemi che possono essere risolti usando un approccio di programmazione dinamica, e propone un'architettura di rete neurale per risolvere tali problemi che batte le baseline della sequenza.

L'articolo propone un'architettura di rete unica che può imparare strategie divide et impera per risolvere task algoritmici.