We present Newtonian Monte Carlo (NMC), a method to improve Markov Chain Monte Carlo (MCMC) convergence by analyzing the first and second order gradients of the target density to determine a suitable proposal density at each point.Existing first order gradient-based methods suffer from the problem of determining an appropriate step size.Too small a step size and it will take a large number of steps to converge, while a very large step size will cause it to overshoot the high density region.NMC is similar to the Newton-Raphson update in optimization where the second order gradient is used to automatically scale the step size in each dimension.However, our objective is not to find a maxima but instead to find a parameterized density that can best match the local curvature of the target density.  This parameterized density is then used as a single-site Metropolis-Hastings proposal.As a further improvement on first order methods, we show that random variables with constrained supports don't need to be transformed before taking a gradient step.NMC directly matches constrained random variables to a proposal density with the same support thus keeping the curvature of the target density intact.We demonstrate the efficiency of NMC on a number of different domains.For statistical models where the prior is conjugate to the likelihood, our method recovers the posterior quite trivially in one step.However, we also show results on fairly large non-conjugate models, where NMC performs better than adaptive first order methods such as NUTS or other inexact scalable inference methods such as Stochastic Variational Inference or bootstrapping.

Sfruttare la curvatura per far convergere i metodi MCMC più velocemente dello stato dell'arte.


Neural Tangents is a library designed to enable research into infinite-width neural networks.It provides a high-level API for specifying complex and hierarchical neural network architectures.These networks can then be trained and evaluated either at finite-width as usual or in their infinite-width limit.Infinite-width networks can be trained analytically using exact Bayesian inference or using gradient descent via the Neural Tangent Kernel.Additionally, Neural Tangents provides tools to study gradient descent training dynamics of wide but finite networks in either function space or weight space. The entire library runs out-of-the-box on CPU, GPU, or TPU.All computations can be automatically distributed over multiple accelerators with near-linear scaling in the number of devices. Neural Tangents is availableathttps://www.github.com/google/neural-tangentsWe also provide an accompanying interactive Colab notebook athttps://colab.sandbox.google.com/github/google/neural-tangents/blob/master/notebooks/neural_tangents_cookbook.ipynb


Deep neural networks have achieved great success in classiï¬cation tasks during the last years.However, one major problem to the path towards artiï¬cial intelligence is the inability of neural networks to accurately detect samples from novel class distributions and therefore, most of the existent classiï¬cation algorithms assume that all classes are known prior to the training stage.In this work, we propose a methodology for training a neural network that allows it to efï¬ciently detect out-of-distribution (OOD) examples without compromising much of its classiï¬cation accuracy on the test examples from known classes.Based on the Outlier Exposure (OE) technique, we propose a novel loss function that achieves state-of-the-art results in out-of-distribution detection with OE both on image and text classiï¬cation tasks.Additionally, the way this method was constructed makes it suitable for training any classiï¬cation algorithm that is based on Maximum Likelihood methods.

Proponiamo una nuova funzione di loss che raggiunge risultati allo stato dell'arte nel rilevamento di out-of-distribution con Outlier Exposure sia su task di classificazione di immagini che di testo.

Questo articolo affronta i problemi del out-of-distribution detection e della calibrazione del modello adattando la funzione di loss della tecnica Outlier Exposure, con risultati che dimostrano un aumento delle prestazioni rispetto a OE su benchmark di visione e testo e una migliore calibrazione del modello.

Proposta di una nuova funzione di loss per addestrare la rete con Outlier Exposure che porta a una migliore individuazione di OOD rispetto alle semplici funzioni di loss che utilizzano la divergenza KL.


Navigation is crucial for animal behavior and is assumed to require an internal representation of the external environment, termed a cognitive map.The precise form of this representation is often considered to be a metric representation of space.An internal representation, however, is judged by its contribution to performance on a given task, and may thus vary between different types of navigation tasks.Here we train a recurrent neural network that controls an agent performing several navigation tasks in a simple environment.To focus on internal representations, we split learning into a task-agnostic pre-training stage that modifies internal connectivity and a task-specific Q learning stage that controls the network's output.We show that pre-training shapes the attractor landscape of the networks, leading to either a continuous attractor, discrete attractors or a disordered state.These structures induce bias onto the Q-Learning phase, leading to a performance pattern across the tasks corresponding to metric and topological regularities.Our results show that, in recurrent networks, inductive bias takes the form of attractor landscapes -- which can be shaped by pre-training and analyzed using dynamical systems methods.Furthermore, we demonstrate that non-metric representations are useful for navigation tasks.

Il pre-training agnostico può modellare la configurazione degli attrattori di RNN e formare diversi bias induttivi per diversi task di navigazione

Questo articolo studia le rappresentazioni interne delle reti neurali ricorrenti addestrate su task di navigazione, e trova che le RNN pre-addestrate a usare path integration contengono attrattori continui 2D mentre le RNN pre-addestrate per landmark memory contengono attrattori discreti.

Questo articolo esplora come il pre-training delle reti ricorrenti su diversi obiettivi di navigazione conferisce diversi benefici per la risoluzione dei task a valle, e mostra come il diverso pre-training si manifesta come diverse strutture dinamiche nelle reti dopo il pre-training.


Formal verification of machine learning models has attracted attention recently, and significant progress has been made on proving simple properties like robustness to small perturbations of the input features.In this context, it has also been observed that folding the verification procedure into training makes it easier to train verifiably robust models.In this paper, we extend the applicability of verified training by extending it to (1) recurrent neural network architectures and (2) complex specifications that go beyond simple adversarial robustness, particularly specifications that capture temporal properties like requiring that a robot periodically visits a charging station or that a language model always produces sentences of bounded length.Experiments show that while models trained using standard training often violate desired specifications, our verified training method produces models that both perform well (in terms of test error or reward) and can be shown to be provably consistent with specifications.

Verifica della rete neurale per proprietà temporali e modelli di generazione di sequenze

Questo articolo estende l'interval bound propagation alla computazione ricorrente e ai modelli auto-regressivi, introduce ed estende la Signal Temporal Logic per specificare i vincoli temporali, e fornisce la prova che STL con bound propagation può garantire che i modelli neurali siano conformi alle specifiche temporali.

Un modo per addestrare regressori di serie temporali in modo verificabile rispetto a un insieme di regole definite dalla logica temporale del segnale, e un lavoro di derivazione delle regole di bound propagation per il linguaggio STL.


Neural Network (NN) has achieved state-of-the-art performances in many tasks within image, speech, and text domains.Such great success is mainly due to special structure design to fit the particular data patterns, such as CNN capturing spatial locality and RNN modeling sequential dependency.Essentially, these specific NNs achieve good performance by leveraging the prior knowledge over corresponding domain data.Nevertheless, there are many applications with all kinds of tabular data in other domains.Since there are no shared patterns among these diverse tabular data, it is hard to design specific structures to fit them all.Without careful architecture design based on domain knowledge, it is quite challenging for NN to reach satisfactory performance in these tabular data domains.To fill the gap of NN in tabular data learning, we propose a universal neural network solution, called TabNN, to derive effective NN architectures for tabular data in all kinds of tasks automatically.Specifically, the design of TabNN follows two principles: \emph{to explicitly leverages expressive feature combinations} and \emph{to reduce model complexity}.Since GBDT has empirically proven its strength in modeling tabular data, we use GBDT to power the implementation of TabNN.Comprehensive experimental analysis on a variety of tabular datasets demonstrate that TabNN can achieve much better performance than many baseline solutions.

Proponiamo una soluzione di rete neurale universale per derivare automaticamente delle architetture NN efficaci per i dati tabulari.

Una nuova procedura di training delle reti neurali, progettata per i dati tabulari, che cerca di sfruttare i cluster di feature estratte dai GBDT.

Proposta di un algoritmo ibrido di machine learning che utilizza alberi decisionali Gradient Boosted potenziati dal gradiente e reti neurali deep, con una direzione di ricerca pensata per i dati tabulari.


Knowledge Bases (KBs) are becoming increasingly large, sparse and probabilistic.These KBs are typically used to perform query inferences and rule mining.But their efficacy is only as high as their completeness.Efficiently utilizing incomplete KBs remains a major challenge as the current KB completion techniques either do not take into account the inherent uncertainty associated with each KB tuple or do not scale to large KBs.Probabilistic rule learning not only considers the probability of every KB tuple but also tackles the problem of KB completion in an explainable way.For any given probabilistic KB, it learns probabilistic first-order rules from its relations to identify interesting patterns.But, the current probabilistic rule learning techniques perform grounding to do probabilistic inference for evaluation of candidate rules.It does not scale well to large KBs as the time complexity of inference using grounding is exponential over the size of the KB.In this paper, we present SafeLearner -- a scalable solution to probabilistic KB completion that performs probabilistic rule learning using lifted probabilistic inference -- as faster approach instead of grounding. We compared SafeLearner to the state-of-the-art probabilistic rule learner ProbFOIL+ and to its deterministic contemporary AMIE+ on standard probabilistic KBs of NELL (Never-Ending Language Learner) and Yago.Our results demonstrate that SafeLearner scales as good as AMIE+ when learning simple rules and is also significantly faster than ProbFOIL+.

Sistema di apprendimento probabilistico delle regole utilizzando Lifted Inference

Un modello per l'apprendimento di regole probabilistiche per automatizzare il completamento di database probabilistici che usa AMIE+ e lifted inference per migliorare l'efficienza computazionale.


Recent efforts in Dialogue State Tracking (DST) for task-oriented dialogues have progressed toward open-vocabulary or generation-based approaches where the models can generate slot value candidates from the dialogue history itself.These approaches have shown good performance gain, especially in complicated dialogue domains with dynamic slot values.However, they fall short in two aspects: (1) they do not allow models to explicitly learn signals across domains and slots to detect potential dependencies among \textit{(domain, slot)} pairs; and (2) existing models follow auto-regressive approaches which incur high time cost when the dialogue evolves over multiple domains and multiple turns.In this paper, we propose a novel framework of Non-Autoregressive Dialog State Tracking (NADST) which can factor in potential dependencies among domains and slots to optimize the models towards better prediction of dialogue states as a complete set rather than separate slots.In particular, the non-autoregressive nature of our method not only enables decoding in parallel to significantly reduce the latency of DST for real-time dialogue response generation, but also detect dependencies among slots at token level in addition to slot and domain level.Our empirical results show that our model achieves the state-of-the-art joint accuracy across all domains on the MultiWOZ 2.1 corpus, and the latency of our model is an order of magnitude lower than the previous state of the art as the dialogue history extends over time.

Proponiamo il primo modello neurale non autoregressivo per il Dialogue State Tracking (DST), raggiungendo la precisione SOTA (49,04%) sul benchmark MultiWOZ2.1, e riducendo la latenza di inferenza di un ordine di grandezza.

Un nuovo modello per il task DST che riduce la complessità del tempo di inferenza con un decoder non autoregressivo, ottiene una precisione DST competitiva e mostra miglioramenti rispetto ad altre baseline.

Proposta di un modello capace di seguire gli stati di dialogo in modo non ricorsivo.


The 3D-zoom operation is the positive translation of the camera in the Z-axis, perpendicular to the image plane.In contrast, the optical zoom changes the focal length and the digital zoom is used to enlarge a certain region of an image to the original image size.In this paper, we are the first to formulate an unsupervised 3D-zoom learning problem where images with an arbitrary zoom factor can be generated from a given single image.An unsupervised framework is convenient, as it is a challenging task to obtain a 3D-zoom dataset of natural scenes due to the need for special equipment to ensure camera movement is restricted to the Z-axis.Besides, the objects in the scenes should not move when being captured, which hinders the construction of a large dataset of outdoor scenes.We present a novel unsupervised framework to learn how to generate arbitrarily 3D-zoomed versions of a single image, not requiring a 3D-zoom ground truth, called the Deep 3D-Zoom Net.The Deep 3D-Zoom Net incorporates the following features:(i) transfer learning from a pre-trained disparity estimation network via a back re-projection reconstruction loss;(ii) a fully convolutional network architecture that models depth-image-based rendering (DIBR), taking into account high-frequency details without the need for estimating the intermediate disparity; and(iii) incorporating a discriminator network that acts as a no-reference penalty for unnaturally rendered areas.Even though there is no baseline to fairly compare our results, our method outperforms previous novel view synthesis research in terms of realistic appearance on large camera baselines.We performed extensive experiments to verify the effectiveness of our method on the KITTI and Cityscapes datasets.

Una nuova architettura di rete per eseguire Deep 3D Zoom o close-up.

Un metodo per creare una "immagine ingrandita" per una data immagine di input, e una nuova loss di ricostruzione della retroproiezione che permette alla rete di imparare la struttura 3D sottostante e mantenere un aspetto naturale.

Un algoritmo per sintetizzare il comportamento dello zoom 3D quando la telecamera si muove in avanti, una struttura di rete che incorpora la stima della disparità in un framework con GANs per sintetizzare nuove viste, e un nuovo task di computer vision viene proposto.


The universal approximation theorem, in one of its most general versions, says that if we consider only continuous activation functions Ïƒ, then a standard feedforward neural network with one hidden layer is able to approximate any continuous multivariate function f to any given approximation threshold Îµ, if and only if Ïƒ is non-polynomial.In this paper, we give a direct algebraic proof of the theorem.Furthermore we shall explicitly quantify the number of hidden units required for approximation.Specifically, if X in R^n is compact, then a neural network with n input units, m output units, and a single hidden layer with {n+d choose d} hidden units (independent of m and Îµ), can uniformly approximate any polynomial function f:X -> R^m whose total degree is at most d for each of its m coordinate functions.In the general case that f is any continuous function, we show there exists some N in O(Îµ^{-n}) (independent of m), such that N hidden units would suffice to approximate f.We also show that this uniform approximation property (UAP) still holds even under seemingly strong conditions imposed on the weights.We highlight several consequences:(i) For any Î´ > 0, the UAP still holds if we restrict all non-bias weights w in the last layer to satisfy |w| < Î´.(ii) There exists some Î»>0 (depending only on f and Ïƒ), such that the UAP still holds if we restrict all non-bias weights w in the first layer to satisfy |w|>Î».(iii) If the non-bias weights in the first layer are *fixed* and randomly chosen from a suitable range, then the UAP holds with probability 1.

Un perfezionamento quantitativo del teorema di approssimazione universale attraverso un approccio algebrico.

Gli autori derivano le dimostrazioni della proprietà di approssimazione universale in modo algebrico e affermano che i risultati sono generali per altri tipi di reti neurali e learner simili.

Una nuova dimostrazione della versione di Leshno della proprietà di approssimazione universale per le reti neurali, e nuove intuizioni sulla proprietà di approssimazione universale.


In this paper, we design a generic framework for learning a robust text classification model that achieves accuracy comparable to standard full models under test-timebudget constraints.We take a different approach from existing methods and learn to dynamically delete a large fraction of unimportant words by a low-complexity selector such that the high-complexity classifier only needs to process a small fraction of important words.In addition, we propose a new data aggregation method to train the classifier, allowing it to make accurate predictions even on fragmented sequence of words.Our end-to-end method achieves state-of-the-art performance while its computational complexity scales linearly with the small fraction of important words in the whole corpus.Besides, a single deep neural network classifier trained by our framework can be dynamically tuned to different budget levels at inference time.

Struttura modulare per la classificazione dei documenti e tecnica di aggregazione dei dati per rendere il framework robusto a varie distorsioni e rumore e concentrarsi solo sulle parole importanti.

Gli autori considerano il training di una text classification basata su RNN dove c'è una restrizione di risorse a test-time prediction, e forniscono un approccio che utilizza un meccanismo di mascheramento per ridurre le parole/frasi/sentenze utilizzate nella predizione seguito da un classificatore per gestire tali componenti.


Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-net and searching for an optimal architecture.In this paper, we present a novel approach, namely  Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance.In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut.This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels.We solve it by introducing  edge normalization, which adds a new set of edge-level hyper-parameters to reduce uncertainty in search.Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoy both faster speed and higher training stability.Experiment results demonstrate the effectiveness of the proposed method.Specifically, we achieve an error rate of 2.57% on CIFAR10 within merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet (under the mobile setting) within 3.8 GPU-days for search.Our code has been made available at https://www.dropbox.com/sh/on9lg3rpx1r6dkf/AABG5mt0sMHjnEJyoRnLEYW4a?dl=0.

Consentire la connessione parziale dei canali nelle super-reti per regolarizzare e accelerare la ricerca di architetture differenziabili

Un'estensione del metodo di ricerca dell'architettura neurale DARTS che affronta il suo difetto di immenso costo di memoria utilizzando un sottoinsieme casuale di canali e un metodo per normalizzare i bordi.

Questo articolo propone di migliorare DARTS in termini di efficienza di training, da grandi costi di memoria e di calcolo, e propone un DARTS parzialmente connesso con connessione parziale del canale e edge normalization.


Dialogue research tends to distinguish between chit-chat and goal-oriented tasks.While the former is arguably more naturalistic and has a wider use of language, the latter has clearer metrics and a more straightforward learning signal.Humans effortlessly combine the two, and engage in chit-chat for example with the goal of exchanging information or eliciting a specific response.Here, we bridge the divide between these two domains in the setting of a rich multi-player text-based fantasy environment where agents and humans engage in both actions and dialogue.Specifically, we train a goal-oriented model with reinforcement learning via self-play against an imitation-learned chit-chat model with two new approaches: the policy either learns to pick a topic or learns to pick an utterance given the top-k utterances.We show that both models outperform a strong inverse model baseline and can converse naturally with their dialogue partner in order to achieve goals.

Gli agenti interagiscono (parlano, agiscono) e possono raggiungere obiettivi in un mondo complesso con un linguaggio diverso, colmando il divario tra il chit-chat e il dialogo orientato agli obiettivi.

Questo articolo studia un task di dialogo multiagente in cui l'agente che apprende mira a generare azioni in linguaggio naturale che elicitano una particolare azione dall'altro agente, e mostra che gli agenti RL possono raggiungere livelli più alti di completamento del task rispetto alle baseline di imitation learning.

Questo articolo esplora il setting del dialogo goal-oriented con reinforcement learning in un Fantasy Text Adventure Game e osserva che gli approcci RL superano i modelli di supervised learning.


We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies.Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation.We propose estimated mixture policy (EMP), a novel class of partially policy-agnostic methods to accurately estimate those quantities.With careful analysis, we show that EMP gives rise to estimates with reduced variance for estimating the state stationary distribution correction while it also offers a useful induction bias for estimating the state-action stationary distribution correction.In extensive experiments with both continuous and discrete environments, we demonstrate that our algorithm offers significantly improved accuracy compared to the state-of-the-art methods.

Un nuovo metodo parzialmente indipendente dalla policy per la valutazione infinite-horizon off-policy della policy con più policy di comportamento conosciute o sconosciute.

Una stima della policy mista che prende le idee dagli stimatori infinite-horizon di valutazione della politica off-policy e dal regression importance sampling per l'importance weight, e li estende a molte policy e a policy sconosciute.

Un algoritmo per risolvere la valutazione infinite horizon off-policy con policy di comportamento multiple stimando una policy mista sotto regressione, e la prova teorica che un rapporto di policy stimato può ridurre la varianza.


We introduce a more efficient neural architecture for amortized inference, which combines continuous and conditional normalizing flows using a principled choice of structure.Our gradient flow derives its sparsity pattern from the minimally faithful inverse of its underlying graphical model.We find that this factorization reduces the necessary numbers both of parameters in the neural network and of adaptive integration steps in the ODE solver.Consequently, the throughput at training time and inference time is increased, without decreasing performance in comparison to unconstrained flows.By expressing the structural inversion and the flow construction as compilation passes of a probabilistic programming language, we demonstrate their applicability to the stochastic inversion of realistic models such as convolutional neural networks (CNN).

Introduciamo un'architettura neurale più efficiente per l'inferenza ammortizzata, che combina normalizing flows continui e condizionali utilizzando una scelta con principio di sparsità.


We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way.By defining the combinatorial search space of NAS to be the set of different edge-partitionings (colorings) into same-weight classes, we represent compact architectures via efficient learned edge-partitionings.For several RL tasks, we manage to learn colorings translating to effective policies parameterized by as few as 17 weight parameters, providing >90 % compression over vanilla policies and 6x compression over state-of-the-art compact policies based on Toeplitz matrices, while still maintaining good reward.We believe that our work is one of the first attempts to propose a rigorous approach to training structured neural network architectures for RL problems that are of interest especially in mobile robotics with limited storage and computational resources.

Mostriamo che ENAS con ottimizzazione ES in RL è altamente scalabile, e lo usiamo per compattare le policy delle reti neurali attraverso la condivisione dei pesi.

Gli autori costruiscono policy di reinforcement learning con pochissimi parametri comprimendo una rete neurale feed-forward, forzandola a condividere i pesi e usando un metodo di reinforcement learning per imparare la mappatura dei pesi condivisi.

Questo articolo combina idee dai metodi ENAS e ES per l'ottimizzazione, e introduce l'architettura della rete cromatica, che partiziona i pesi della rete RL in sottogruppi legati.


Deep approaches to anomaly detection have recently shown promising results over shallow methods on large and complex datasets.Typically anomaly detection is treated as an unsupervised learning problem.In practice however, one may have---in addition to a large set of unlabeled samples---access to a small pool of labeled samples, e.g. a subset verified by some domain expert as being normal or anomalous.Semi-supervised approaches to anomaly detection aim to utilize such labeled samples, but most proposed methods are limited to merely including labeled normal samples.Only a few methods take advantage of labeled anomalies, with existing deep approaches being domain-specific.In this work we present Deep SAD, an end-to-end deep methodology for general semi-supervised anomaly detection.Using an information-theoretic perspective on anomaly detection, we derive a loss motivated by the idea that the entropy of the latent distribution for normal data should be lower than the entropy of the anomalous distribution.We demonstrate in extensive experiments on MNIST, Fashion-MNIST, and CIFAR-10, along with other anomaly detection benchmark datasets, that our method is on par or outperforms shallow, hybrid, and deep competitors, yielding appreciable performance improvements even when provided with only little labeled data.

Introduciamo Deep SAD, un metodo deep per il rilevamento generale semi-supervised delle anomalie che sfrutta soprattutto le anomalie etichettate.

Un nuovo metodo per trovare dati anomali, quando alcune anomalie etichettate sono date, che applica la loss derivata dalla teoria dell'informazione basata sui dati normali che hanno solitamente un'entropia più bassa dei dati anomali.

Proposta per un framework di rilevamento delle anomalie in impostazioni in cui sono disponibili dati non etichettati, dati positivi etichettati e dati negativi etichettati, e proposta di approccio all'AD semi-supervised da una prospettiva della teoria dell'informazione.


To analyze deep ReLU network, we adopt a student-teacher setting in which an over-parameterized student network learns from the output of a fixed teacher network of the same depth, with Stochastic Gradient Descent (SGD).Our contributions are two-fold.First, we prove that when the gradient is zero (or bounded above by a small constant) at every data point in training, a situation called  \emph{interpolation setting}, there exists many-to-one \emph{alignment} between student and teacher nodes in the lowest layer under mild conditions.This suggests that generalization in unseen dataset is achievable, even the same condition often leads to zero training error.Second, analysis of noisy recovery and training dynamics in 2-layer network shows that strong teacher nodes (with large fan-out weights) are learned first and subtle teacher nodes are left unlearned until late stage of training.As a result, it could take a long time to converge into these small-gradient critical points.Our analysis shows that over-parameterization plays two roles: (1) it is a necessary condition for alignment to happen at the critical points, and (2) in training dynamics, it helps student nodes cover more teacher nodes with fewer iterations.Both improve generalization.Experiments justify our finding.

Questo articolo analizza le dinamiche di training e i punti critici della formazione della rete ReLU deep tramite SGD nel setting teacher-student.

Studio della sovra-parametrizzazione nelle reti ReLU multilayer teacher-student, una parte teorica sui punti critici SGD per l'impostazione teacher-student, e una parte euristica ed empirica sulla dinamica dell'algoritmo SDG in funzione delle teacher networks.


We study the convergence of gradient descent (GD) and stochastic gradient descent (SGD) for training $L$-hidden-layer linear residual networks (ResNets).We prove that for training deep residual networks with certain linear transformations at input and output layers, which are fixed throughout training, both GD and SGD with zero initialization on all hidden weights can converge to the global minimum of the training loss.Moreover, when specializing to appropriate Gaussian random linear transformations, GD and SGD provably optimize wide enough deep linear ResNets.Compared with the global convergence result of GD for training standard deep linear networks \citep{du2019width}, our condition on the neural network width is sharper by a factor of $O(\kappa L)$, where $\kappa$ denotes the condition number of the covariance matrix of the training data.In addition, for the first time we establish the global convergence of SGD for training deep linear ResNets and prove a linear convergence rate when the global minimum is $0$.

Sotto certe condizioni sulle trasformazioni lineari in input e output, sia GD sia SGD possono raggiungere la convergenza globale per il training di ResNet lineari deep.

Gli autori studiano la convergenza della discesa del gradiente nel training di residual networks lineari deep, e stabiliscono una convergenza globale di GD/SGD e tassi di convergenza lineare di SG/SGD.

Studio delle proprietà di convergenza di GD e SGD su reti lineari deep, e dimostrazione che sotto certe condizioni sulle trasformazioni di input e output e con inizializzazione zero, GD e SGD convergono ai minimi globali.


In this paper, we empirically investigate the training journey of deep neural networks relative to fully trained shallow machine learning models.We observe that the deep neural networks (DNNs) train by learning to correctly classify shallow-learnable examples in the early epochs before learning the harder examples.We build on this observation this to suggest a way for partitioning the dataset into hard and easy subsets that can be used for improving the overall training process.Incidentally, we also found evidence of a subset of intriguing examples across all the datasets we considered, that were shallow learnable but not deep-learnable.In order to aid reproducibility, we also duly release our code for this work at https://github.com/karttikeya/Shallow_to_Deep/

Analizziamo il processo di training delle reti deep e mostriamo che partono da un rapido apprendimento di esempi classificabili poco profondi e generalizzano lentamente a datapoint più difficili.


While much recent work has targeted learning deep discrete latent variable models with variational inference, this setting remains challenging, and it is often necessary to make use of potentially high-variance gradient estimators in optimizing the ELBO.As an alternative, we propose to optimize a non-ELBO objective derived from the Bethe free energy approximation to an MRF's partition function.This objective gives rise to a saddle-point learning problem, which we train inference networks to approximately optimize.The derived objective requires no sampling, and can be efficiently computed for many MRFs of interest.We evaluate the proposed approach in learning high-order neural HMMs on text, and find that it often outperforms other approximate inference schemes in terms of true held-out log likelihood.At the same time, we find that all the approximate inference-based approaches to learning high-order neural HMMs we consider underperform learning with exact inference by a significant margin.

Apprendimento di MRF latenti deep con un obiettivo saddle-point derivato dall'approssimazione della funzione di partizione di Bethe.

Un metodo per l'apprendimento di MRF deep a variabili latenti con un obiettivo di ottimizzazione che utilizza l'energia libera di Bethe, che risolve anche i vincoli sottostanti alle ottimizzazioni dell'energia libera di Bethe.

Un obiettivo per l'apprendimento di MRF a variabili latenti basato sull'energia libera di Bethe e sull'inferenza ammortizzata, diverso dall'ottimizzazione dell'ELBO standard.


In an explanation generation problem, an agent needs to identify and explain the reasons for its decisions to another agent.Existing work in this area is mostly confined to planning-based systems that use automated planning approaches to solve the problem.In this paper, we approach this problem from a new perspective, where we propose a general logic-based framework for explanation generation.In particular, given a knowledge base $KB_1$ that entails a formula $\phi$ and a second knowledge base $KB_2$ that does not entail $\phi$, we seek to identify an explanation $\epsilon$ that is a subset of $KB_1$ such that the union of $KB_2$ and $\epsilon$ entails $\phi$.We define two types of explanations, model- and proof-theoretic explanations, and use cost functions to reflect preferences between explanations.Further, we present our algorithm implemented for propositional logic that compute such explanations and empirically evaluate it in random knowledge bases and a planning domain.

Un framework generale per la generazione di spiegazioni utilizzando la logica.

Questo articolo studia la generazione di spiegazioni da un punto di vista KR e conduce esperimenti che misurano la dimensione delle spiegazioni e il tempo di esecuzione su formule casuali e formule da un'istanza di Blocksworld.

Questo articolo fornisce una prospettiva sulle spiegazioni tra due basi di conoscenza, ed è parallelo al lavoro sulla riconciliazione dei modelli nella letteratura sulla pianificazione.


Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem.This problem can be typically resolved by the rectified linear unit (ReLU) activation.However, here we show that even for such activation, deep and narrow neural networks (NNs) will converge to erroneous mean or median states of the target function depending on the loss with high probability.Deep and narrow NNs are encountered in solving partial differential equations with high-order derivatives.We demonstrate this collapse of such NNs both numerically and theoretically, and provide estimates of the probability of collapse.We also construct a diagram of a safe region for designing NNs that avoid the collapse to erroneous states.Finally, we examine different ways of initialization and normalization that may avoid the collapse problem.Asymmetric initializations may reduce the probability of collapse but do not totally eliminate it.

Le reti neurali deep e strette convergeranno verso stati medi o mediani errati della funzione obiettivo a seconda della loss con alta probabilità.

Questo articolo studia le modalità di fallimento delle reti deep e strette, concentrandosi sui modelli più piccoli possibili per i quali si verifica il comportamento indesiderato.

Questo articolo mostra che il training delle reti neurali ReLU deep convergerà a un classificatore costante con alta probabilità rispetto all'inizializzazione casuale se le larghezze degli hidden lazer sono troppo piccole.


We study adversarial robustness of neural networks from a margin maximization perspective, where margins are defined as the distances from inputs to a classifier's decision boundary.Our study shows that maximizing margins can be achieved by minimizing the adversarial loss on the decision boundary at the "shortest successful perturbation", demonstrating a close connection between adversarial losses and the margins.We propose Max-Margin Adversarial (MMA) training to directly maximize the margins to achieve adversarial robustness. Instead of adversarial training with a fixed $\epsilon$, MMA offers an improvement by enabling adaptive selection of the "correct" $\epsilon$ as the margin individually for each datapoint.In addition, we rigorously analyze adversarial training with the perspective of margin maximization, and provide an alternative interpretation for adversarial training, maximizing either a lower or an upper bound of the margins.Our experiments empirically confirm our theory and demonstrate MMA training's efficacy on the MNIST and CIFAR10 datasets w.r.t. $\ell_\infty$ and $\ell_2$ robustness.

Proponiamo il training MMA per massimizzare direttamente il margine dello spazio di input al fine di migliorare la robustezza avversaria principalmente rimuovendo il requisito di specificare un limite di distorsione fisso.

Un approccio di training adversariale basato sul margine adattivo per addestrare DNN robuste, massimizzando il margine più stretto degli input al decision boundary, che rende possibile il training adversariale con grandi perturbazioni.

Viene introdotto un metodo per l'apprendimento robusto contro gli attacchi avversari in cui il margine dello spazio di input è direttamente massimizzato e una variante softmax del max-margin.


Many anomaly detection methods exist that perform well on low-dimensional problems however there is a notable lack of effective methods for high-dimensional spaces, such as images.Inspired by recent successes in deep learning we propose a novel approach to anomaly detection using generative adversarial networks.Given a sample under consideration, our method is based on searching for a good representation of that sample in the latent space of the generator; if such a representation is not found, the sample is deemed anomalous.  We achieve state-of-the-art performance on standard image benchmark datasets and visual inspection of the most anomalous samples reveals that our method does indeed return anomalies.

Proponiamo un metodo per il rilevamento delle anomalie con le GAN cercando nello spazio latente del generatore delle buone rappresentazioni del campione.

Gli autori propongono di utilizzare GAN per il rilevamento delle anomalie, un metodo basato sulla discesa del gradiente per aggiornare iterativamente le rappresentazioni latenti, e un nuovo tipo di aggiornamento dei parametri dei generatori.

Un approccio basato su GAN per fare il rilevamento delle anomalie per immagini dove lo spazio latente del generatore viene esplorato per trovare una rappresentazione per un'immagine di test.


Variational inference (VI) and Markov chain Monte Carlo (MCMC) are approximate posterior inference algorithms that are often said to have complementary strengths, with VI being fast but biased and MCMC being slower but asymptotically unbiased.In this paper, we analyze gradient-based MCMC and VI procedures and find theoretical and empirical evidence that these procedures are not as different as one might think.In particular, a close examination of the Fokker-Planck equation that governs the Langevin dynamics (LD) MCMC procedure reveals that LD implicitly follows a gradient flow that corresponds to a variational inference procedure based on optimizing a nonparametric normalizing flow.This result suggests that the transient bias of LD (due to too few warmup steps) may track that of VI (due to too few optimization steps), up to differences due to VIâ€™s parameterization and asymptotic bias.Empirically, we find that the transient biases of these algorithms (and momentum-accelerated versions) do evolve similarly.This suggests that practitioners with a limited time budget may get more accurate results by running an MCMC procedure (even if itâ€™s far from burned in) than a VI procedure, as long as the variance of the MCMC estimator can be dealt with (e.g., by running many parallel chains).

Il comportamento transitorio degli algoritmi MCMC basati sul gradiente e di variational inference è più simile di quanto si possa pensare, mettendo in discussione l'affermazione che variational inference è più veloce di MCMC.


Graph Convolutional Networks (GCNs) have recently been shown to be quite successful in modeling graph-structured data.However, the primary focus has been on handling simple undirected graphs.Multi-relational graphs are a more general and prevalent form of graphs where each edge has a label and direction associated with it.Most of the existing approaches to handle such graphs suffer from over-parameterization and are restricted to learning representations of nodes only.In this paper, we propose CompGCN, a novel Graph Convolutional framework which jointly embeds both nodes and relations in a relational graph.CompGCN leverages a variety of entity-relation composition operations from Knowledge Graph Embedding techniques and scales with the number of relations.It also generalizes several of the existing multi-relational GCN methods.We evaluate our proposed method on multiple tasks such as node classification, link prediction, and graph classification, and achieve demonstrably superior results.We make the source code of CompGCN available to foster reproducible research.

Un framework di convoluzione dei grafi basato sulla composizione per i grafi multirelazionali.

Gli autori sviluppano GCN su grafi multi-relazionali e propongono CompGCN, che sfrutta le intuizioni dei knowledge graph embeddings e impara le rappresentazioni dei nodi e delle relazioni per alleviare il problema dell'iper-parametrizzazione.

Questo articolo introduce un framework GCN per i grafi multi-relazionali e generalizza diversi approcci esistenti per Knowledge Graph embeddings in un framework.


State-of-the-art neural machine translation methods employ massive amounts of parameters.Drastically reducing computational costs of such methods without affecting performance has been up to this point unsolved.In this work, we propose a quantization strategy tailored to the Transformer architecture.We evaluate our method on the WMT14 EN-FR and WMT14 EN-DE translation tasks and achieve state-of-the-art quantization results for the Transformer, obtaining no loss in BLEU scores compared to the non-quantized baseline.We further compress the Transformer by showing that, once the model is trained, a good portion of the nodes in the encoder can be removed without causing any loss in BLEU.

Quantizziamo completamente il transformer a 8 bit e miglioriamo la qualità della traduzione rispetto al modello a precisione completa.

Un metodo di quantizzazione a 8 bit per quantizzare il modello di traduzione automatica Transformer, proponendo di usare una quantizzazione uniforme min-max durante l'inferenza e di usare i bucketing weights prima della quantizzazione per ridurre l'errore di quantizzazione.

Un metodo per ridurre lo spazio di memoria richiesto da una tecnica di quantizzazione, focalizzato sulla riduzione per l'architettura Transformer.


Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems.However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes.We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space.The resulting approach, latent embedding optimization (LEO), decouples the gradient-based adaptation procedure from the underlying high-dimensional space of model parameters.Our evaluation shows that LEO can achieve state-of-the-art performance on the competitive miniImageNet and tieredImageNet few-shot classification tasks.Further analysis indicates LEO is able to capture uncertainty in the data, and can perform adaptation more effectively by optimizing in latent space.

Latent Embedding Optimization (LEO) è un nuovo meta-learner basato sul gradiente con prestazioni allo stato dell'arte sui difficili task di classificazione 5-way 1-shot e 5-shot miniImageNet e tieredImageNet.

Un nuovo framework di meta-learning che impara lo spazio latente dipendente dai dati, esegue un adattamento veloce nello spazio latente, è efficace per il few-shot learning, ha un'inizializzazione dipendente dal task per l'adattamento, e funziona bene per la distribuzione multimodale del task.

Questo articolo propone un metodo di ottimizzazione di embedding latente per il meta-learning, e sostiene che il contributo è quello di disaccoppiare le tecniche di meta-learning basate sull'ottimizzazione dallo spazio ad alta densità dei parametri del modello.


We introduce an approach for augmenting model-free deep reinforcement learning agents with a mechanism for relational reasoning over structured representations, which improves performance, learning efficiency, generalization, and interpretability.Our architecture encodes an image as a set of vectors, and applies an iterative message-passing procedure to discover and reason about relevant entities and relations in a scene.In six of seven StarCraft II Learning Environment mini-games, our agent achieved state-of-the-art performance, and surpassed human grandmaster-level on four.In a novel navigation and planning task, our agent's performance and learning efficiency far exceeded non-relational baselines, it was able to generalize to more complex scenes than it had experienced during training.Moreover, when we examined its learned internal representations, they reflected important structure about the problem and the agent's intentions.The main contribution of this work is to introduce techniques for representing and reasoning about states in model-free deep reinforcement learning agents via relational inductive biases.Our experiments show this approach can offer advantages in efficiency, generalization, and interpretability, and can scale up to meet some of the most challenging test environments in modern artificial intelligence.

I bias induttivi relazionali migliorano le capacità di generalizzazione fuori distribuzione negli agenti di reinforcement learning senza modello

Un'architettura di rete relazionale condivisa per parametrizzare la rete actor and critc, focalizzata su algoritmi actor-critic advantage distribuiti, che migliora le tecniche di deep reinforcement senza modello con la conoscenza relazionale dell'ambiente in modo che gli agenti possano imparare rappresentazioni di stato interpretabili.

Un'analisi e una valutazione quantitativa e qualitativa del meccanismo di self-attention combinato con le relation network nel contesto del RL senza modello.


Image translation between two domains is a class of problems aiming to learn mapping from an input image in the source domain to an output image in the target domain.It has been applied to numerous applications, such as data augmentation, domain adaptation, and unsupervised training.When paired training data is not accessible, image translation becomes an ill-posed problem.We constrain the problem with the assumption that the translated image needs to be perceptually similar to the original image and also appears to be drawn from the new domain, and propose a simple yet effective image translation model consisting of a single generator trained with a self-regularization term and an adversarial term.We further notice that existing image translation techniques are agnostic to the subjects of interest and often introduce unwanted changes or artifacts to the input.Thus we propose to add an attention module to predict an attention map to guide the image translation process.The module learns to attend to key parts of the image while keeping everything else unaltered, essentially avoiding undesired artifacts or changes.The predicted attention map also opens door to applications such as unsupervised segmentation and saliency detection.Extensive experiments and evaluations show that our model while being simpler, achieves significantly better performance than existing image translation methods.

Proponiamo un semplice modello generativo per unsupervised  image translation e il rilevamento della saliency.


Building deep neural networks to control autonomous agents which have to interact in real-time with the physical world, such as robots or automotive vehicles, requires a seamless integration of time into a networkâ€™s architecture.The central question of this work is, how the temporal nature of reality should be reflected in the execution of a deep neural network and its components.Most artificial deep neural networks are partitioned into a directed graph of connected modules or layers and the layers themselves consist of elemental building blocks, such as single units.For most deep neural networks, all units of a layer are processed synchronously and in parallel, but layers themselves are processed in a sequential manner.In contrast, all elements of a biological neural network are processed in parallel.In this paper, we define a class of networks between these two extreme cases.These networks are executed in a streaming or synchronous layerwise-parallel manner, unlocking the layers of such networks for parallel processing.Compared to the standard layerwise-sequential deep networks, these new layerwise-parallel networks show a fundamentally different temporal behavior and flow of information, especially for networks with skip or recurrent connections.We argue that layerwise-parallel deep networks are better suited for future challenges of deep neural network design, such as large functional modularized and/or recurrent architectures as well as networks allocating different network capacities dependent on current stimulus and/or task complexity.We layout basic properties and discuss major challenges for layerwise-parallel networks.Additionally, we provide a toolbox to design, train, evaluate, and online-interact with layerwise-parallel networks.

Definiamo un concetto di reti neurali deep a layer model-parallel, per le quali i layer operano in parallelo, e forniamo un toolbox per progettare, formare, valutare e interagire online con queste reti.

Un toolbox accelerato dalla GPU per l'aggiornamento parallelo dei neuroni, scritto in Theano, che supporta diversi ordini di aggiornamento in reti ricorrenti e reti con connessioni che saltano gli layer.

Un nuovo toolbox per l'apprendimento e la valutazione delle reti neurali deep, e proposta per un cambio di paradigma dalle reti sequenziali a layer alle reti parallele a layer.


Deep neural networks are known to be vulnerable to adversarial perturbations.In this paper, we bridge adversarial robustness of neural nets with Lyapunov stability of dynamical systems.From this viewpoint, training neural nets is equivalent to finding an optimal control of the discrete dynamical system, which allows one to utilize methods of successive approximations, an optimal control algorithm based on Pontryagin's maximum principle, to train neural nets.This decoupled training method allows us to add constraints to the optimization, which makes the deep model more robust.The constrained optimization problem can be formulated as a semi-definite programming problem and hence can be solved efficiently.Experiments show that our method effectively improves deep model's adversarial robustness.

Un metodo di difesa avversaria che collega la robustezza delle reti neurali deep con la stabilità di Lyapunov

Gli autori formulano il training delle NN come la ricerca di un controllore ottimale per un sistema dinamico discreto, permettendo loro di utilizzare il metodo delle approssimazioni successive per addestrare una NN in modo da essere più robusta agli attacchi avversari.

Questo articolo usa la visione teorica di una rete neurale come un ODE discretizzato per sviluppare una teoria del controllo robusto che mira al training della rete imponendo al tempo stesso la robustezza.


In this paper, we propose a method named Dimensional reweighting Graph Convolutional Networks (DrGCNs), to tackle the problem of variance between dimensional information in the node representations of GCNs.We prove that DrGCNs can reduce the variance of the node representations by connecting our problem to the theory of the mean field.However, practically, we find that the degrees DrGCNs help vary severely on different datasets.We revisit the problem and develop a new measure K to quantify the effect.This measure guides when we should use dimensional reweighting in GCNs and how much it can help.Moreover, it offers insights to explain the improvement obtained by the proposed DrGCNs.The dimensional reweighting block is light-weighted and highly flexible to be built on most of the GCN variants.Carefully designed experiments, including several fixes on duplicates, information leaks, and wrong labels of the well-known node classification benchmark datasets, demonstrate the superior performances of DrGCNs over the existing state-of-the-art approaches.Significant improvements can also be observed on a large scale industrial dataset.

Proponiamo uno schema di reweighting semplice ma efficace per le GCN, supportato teoricamente dalla teoria del campo medio.

Un metodo, noto come DrGCN, per ripesare le diverse dimensioni delle rappresentazioni dei nodi nelle graph convolutional networks riducendo la varianza tra le dimensioni.


Knowledge-grounded dialogue is a task of generating an informative response based on both discourse context and external knowledge.As we focus on better modeling the knowledge selection in the multi-turn knowledge-grounded dialogue, we propose a sequential latent variable model as the first approach to this matter.The model named sequential knowledge transformer (SKT) can keep track of the prior and posterior distribution over knowledge; as a result, it can not only reduce the ambiguity caused from the diversity in knowledge selection of conversation but also better leverage the response information for proper choice of knowledge.Our experimental results show that the proposed model improves the knowledge selection accuracy and subsequently the performance of utterance generation.We achieve the new state-of-the-art performance on Wizard of Wikipedia (Dinan et al., 2019) as one of the most large-scale and challenging benchmarks.We further validate the effectiveness of our model over existing conversation methods in another knowledge-based dialogue Holl-E dataset (Moghe et al., 2018).

Il nostro approccio è il primo tentativo di sfruttare un modello di variabile latente sequenziale per knowledge selection nel dialogo multi-turn knowledge-grounded. Raggiunge il nuovo stato dell'arte delle prestazioni sul benchmark Wizard of Wikipedia.

Un modello sequenziale con variabili latenti per knowledge selection in dialogue generation che estende il modello di attention posteriore al latent knowledge selection problem e raggiunge prestazioni più elevate rispetto ai precedenti modelli allo stato dell'arte.

Una nuova architettura per knowledge-grounded multi-turn dialogue selection che rende lo stato dell'arte sui benchmark di riferimento e ottiene punteggi più alti nelle valutazioni umane.


Meta-learning, or learning-to-learn, has proven to be a successful strategy in attacking problems in supervised learning and reinforcement learning that involve small amounts of data.State-of-the-art solutions involve learning an initialization and/or learning algorithm using a set of training episodes so that the meta learner can generalize to an evaluation episode quickly.These methods perform well but often lack good quantification of uncertainty, which can be vital to real-world applications when data is lacking.We propose a meta-learning method which efficiently amortizes hierarchical variational inference across tasks, learning a prior distribution over neural network weights so that a few steps of Bayes by Backprop will produce a good task-specific approximate posterior.We show that our method produces good uncertainty estimates on contextual bandit and few-shot learning benchmarks.

Proponiamo un metodo di meta-learning che ammortizza in modo efficiente la variational inference gerarchica attraverso gli episodi di allenamento.

Un adattamento ai modelli di tipo MAML che tiene conto dell'incertezza posteriore nelle variabili latenti specifiche del task impiegando la variational inference per i parametri specifici del task in una visione gerarchica bayesiana di MAML.

Gli autori considerano il meta-learning per imparare un prior sui pesi delle reti neurali, fatto tramite varitional inference ammortizzata.


Often we wish to transfer representational knowledge from one neural network to another.Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator.Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network.We demonstrate that this objective ignores important structural knowledge of the teacher network.This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data.We formulate this objective as contrastive learning.Experiments demonstrate that our resulting new objective outperforms knowledge distillation on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer.When combined with knowledge distillation, our method sets a state of the art in many transfer tasks, sometimes even outperforming the teacher network.

Representation/knowledge distillation massimizzando l'informazione mutua tra teacher e student

Questo articolo combina un obiettivo contrastive che misura l'informazione mutua tra le rappresentazioni apprese dalle reti di teacher e student per la distillazione dei modelli, e propone un modello con un miglioramento rispetto alle alternative esistenti sui task di distillazione.


Developing effective biologically plausible learning rules for deep neural networks is important for advancing connections between deep learning and neuroscience.To date, local synaptic learning rules like those employed by the brain have failed to match the performance of backpropagation in deep networks.In this work, we employ meta-learning to discover networks that learn using feedback connections and local, biologically motivated learning rules.Importantly, the feedback connections are not tied to the feedforward weights, avoiding any biologically implausible weight transport.It can be shown mathematically that this approach has sufficient expressivity to approximate any online learning algorithm.Our experiments show that the meta-trained networks effectively use feedback connections to perform online credit assignment in multi-layer architectures.Moreover, we demonstrate empirically that this model outperforms a state-of-the-art gradient-based meta-learning algorithm for continual learning on regression and classification benchmarks.This approach represents a step toward biologically plausible learning mechanisms that can not only match gradient descent-based learning, but also overcome its limitations.

Le reti che imparano con connessioni di feedback e regole di plasticità locale possono essere ottimizzate per l'uso del meta learning.


In the visual system, neurons respond to a patch of the input known as their classical receptive field (RF), and can be modulated by stimuli in the surround.These interactions are often mediated by lateral connections, giving rise to extra-classical RFs.We use supervised learning via backpropagation to learn feedforward connections, combined with an unsupervised learning rule to learn lateral connections between units within a convolutional neural network.These connections allow each unit to integrate information from its surround, generating extra-classical receptive fields for the units in our new proposed model (CNNEx).We demonstrate that these connections make the network more robust and achieve better performance on noisy versions of the MNIST and CIFAR-10 datasets.Although the image statistics of MNIST and CIFAR-10 differ greatly, the same unsupervised learning rule generalized to both datasets.Our framework can potentially be applied to networks trained on other tasks, with the learned lateral connections aiding the computations implemented by feedforward connections when the input is unreliable.

Le CNN con connessioni laterali biologicamente ispirate apprese in modo unsupervised sono più robuste agli input rumorosi.


Deep learning (DL) has in recent years been widely used in naturallanguage processing (NLP) applications due to its superiorperformance.However, while natural languages are rich ingrammatical structure, DL has not been able to explicitlyrepresent and enforce such structures.This paper proposes a newarchitecture to bridge this gap by exploiting tensor productrepresentations (TPR), a structured neural-symbolic frameworkdeveloped in cognitive science over the past 20 years, with theaim of integrating DL with explicit language structures and rules.We call it the Tensor Product Generation Network(TPGN), and apply it to image captioning.The keyideas of TPGN are:1) unsupervised learning ofrole-unbinding vectors of words via a TPR-based deep neuralnetwork, and2) integration of TPR with typical DL architecturesincluding Long Short-Term Memory (LSTM) models.The novelty of ourapproach lies in its ability to generate a sentence and extractpartial grammatical structure of the sentence by usingrole-unbinding vectors, which are obtained in an unsupervisedmanner.Experimental results demonstrate the effectiveness of theproposed approach.

Questo articolo ha lo scopo di sviluppare un approccio di rappresentazione del prodotto tensoriale per applicazioni di natural language processing con deep learning.


It is well-known that  classifiers are vulnerable to adversarial perturbations.To defend against adversarial perturbations, various certified robustness results have been derived.However, existing certified robustnesses are limited to top-1 predictions.In many real-world applications, top-$k$ predictions are more relevant.In this work, we aim to derive certified robustness for top-$k$ predictions.In particular, our certified robustness is based on randomized smoothing, which turns any classifier to a new classifier via adding noise to an input example.We adopt randomized smoothing because it is scalable to large-scale neural networks and applicable to any classifier.We derive a tight robustness in $\ell_2$ norm for top-$k$ predictions  when using randomized smoothing with Gaussian noise.We find that generalizing the certified robustness  from top-1 to top-$k$ predictions faces significant technical challenges.We also empirically evaluate our method on CIFAR10 and ImageNet.For example, our method can obtain an ImageNet classifier with a certified top-5 accuracy of 62.8\% when the $\ell_2$-norms of the adversarial perturbations are less than 0.5 (=127/255).Our code is publicly available at: \url{https://github.com/jjy1994/Certify_Topk}.

Studiamo la robustezza certificata per le previsioni top-k attraverso lo smoothing randomizzato sotto il rumore gaussiano e deriviamo un limite di robustezza stretto nella norma L_2.

Questo articolo estende il lavoro sulla deduzione di un raggio certificato usando lo smoothing randomizzato, e mostra il raggio al quale un classificatore smoothed sotto perturbazioni gaussiane è certificato per le prime k previsioni.

Questo articolo si basa sulla tecnica di random smoothing per la predizione top-1, e mira a fornire una certificazione sulle predizioni top-k.


Recent work has shown increased interest in using the Variational Autoencoder (VAE) framework to discover interpretable representations of data in an unsupervised way.These methods have focussed largely on modifying the variational cost function to achieve this goal.However, we show that methods like beta-VAE simplify the tendency of variational inference to underfit causing pathological over-pruning and over-orthogonalization of learned components.In this paper we take a complementary approach: to modify the probabilistic model to encourage structured latent variable representations to be discovered.Specifically, the standard VAE probabilistic model is unidentifiable: the likelihood of the parameters is invariant under rotations of the latent space.This means there is no pressure to identify each true factor of variation with a latent variable.We therefore employ a rich prior distribution, akin to the ICA model, that breaks the rotational symmetry.Extensive quantitative and qualitative experiments demonstrate that the proposed prior mitigates the trade-off introduced by modified cost functions like beta-VAE and TCVAE between reconstruction loss and disentanglement.The proposed prior allows to improve these approaches with respect to both disentanglement and reconstruction quality significantly over the state of the art.

Presentiamo prior strutturati per l'apprendimento unsupervised di rappresentazioni disentangled in VAEs che mitigano significativamente il trade-off tra disentanglement e loss di ricostruzione.

Un framework generale per utilizzare la famiglia di distribuzioni L^p-nested come prior per il code vector di un VAE, dimostrando un MIG superiore.

Gli autori sottolineano i problemi negli attuali approcci VAE e forniscono una nuova prospettiva sul compromesso tra ricostruzione e ortogonalizzazione per VAE, beta-VAE e beta-TCVAE.


Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks.The explanations in the literature for the apparent effectiveness of shortcuts are varied and often contradictory.We hypothesize that shortcuts work primarily because they act as linear counterparts to nonlinear layers.We test this hypothesis by using several variations on the standard residual block, with different types of linear connections, to build small (100k--1.2M parameter) image classification networks.Our experiments show that other kinds of linear connections can be even more effective than the identity shortcuts.Our results also suggest that the best type of linear connection for a given application may depend on both network width and depth.

Generalizziamo i residual blocks ai tandem blocks, che usano mappe lineari arbitrarie invece di shortcut, e migliorano le prestazioni rispetto alle ResNet.

Questo articolo esegue un'analisi delle shortcut connections nelle architetture ResNet-like, e propone di sostituire le identity shortcut con una alternativa con la convoluzione chiamata blocco tandem.

Questo articolo studia l'effetto della sostituzione delle skip connections con identità con skip connections convoluzionali addestrabili in ResNet e trova che le prestazioni migliorino.


Adam-typed optimizers, as a class of adaptive moment estimation methods with the exponential moving average scheme, have been successfully used in many applications of deep learning.Such methods are appealing for capability on large-scale sparse datasets.On top of that, they are computationally efficient and insensitive to the hyper-parameter settings.In this paper, we present a new framework for adapting Adam-typed methods, namely AdamT.Instead of applying a simple exponential weighted average, AdamT also includes the trend information when updating the parameters with the adaptive step size and gradients.The newly added term is expected to efficiently capture the non-horizontal moving patterns on the cost surface, and thus converge more rapidly.We show empirically the importance of the trend component, where AdamT outperforms the conventional Adam method constantly in both convex and non-convex settings.

Presentiamo un nuovo framework per adattare i metodi di tipo Adam, cioè AdamT, per includere le informazioni sulla tendenza quando si aggiornano i parametri con step size e i gradienti adattivi.

Un nuovo tipo di variante di Adam che usa il metodo lineare di Holt per calcolare il momento smoothed del primo ordine e del secondo ordine invece di usare la media pesata esponenziale.


As machine learning methods see greater adoption and implementation in high stakes applications suchas medical image diagnosis, the need for model interpretability and explanation has become morecritical.Classical approaches that assess feature importance (eg saliency maps) do not explain how and why a particular region of an image is relevant to the prediction.We proposea method that explains the outcome of a classification black-box by gradually exaggeratingthe semantic effect of a given class.Given a query input to a classifier, our method produces aprogressive set of plausible variations of that query, which gradually change the posterior probabilityfrom its original class to its negation.These counter-factually generated samples preserve featuresunrelated to the classification decision, such that a user can employ our method as a ``tuning knob'' to traverse a data manifold while crossing the decision boundary.  Our method is model agnostic and only requires the output value and gradient of the predictor with respect to its input.

Un metodo per spiegare un classificatore, generando una perturbazione visiva di un'immagine esagerando o diminuendo le feature semantiche che il classificatore associa a un'etichetta target.

Un modello che quando viene data una query in input a una black-box, mira a spiegare il risultato fornendo variazioni plausibili e progressive alla query che possono portare a un cambiamento dell'output.

Un metodo per spiegare l'output della classificazione black box delle immagini, che genera una perturbazione graduale degli output in risposta a richieste di input gradualmente perturbate.


We study the problem of explaining a rich class of behavioral properties of deep neural networks.Our influence-directed explanations approach this problem by peering inside the network to identify neurons with high influence on the property of interest using an axiomatically justified influence measure, and then providing an interpretation for the concepts these neurons represent.We evaluate our approach by training convolutional neural networks on Pubfig, ImageNet, and Diabetic Retinopathy datasets.  Our evaluation demonstrates that influence-directed explanations (1) localize features used by the network, (2) isolate features distinguishing related instances, (3) help extract the essence of what the network learned about the class, and (4) assist in debugging misclassifications.

Noi presentiamo un approccio influenzato alla costruzione di spiegazioni per il comportamento delle reti convoluzionali deep, e mostriamo come può essere usato per rispondere a un'ampia serie di domande che non potevano essere affrontate dal lavoro precedente.

Un modo di misurare l'influenza che soddisfa certi assiomi, e una nozione di influenza che può essere utilizzata per identificare quale parte di input è più influente per l'output di un neurone in una deep neural network.

Questo articolo propone di misurare l'influenza di singoli neuroni rispetto a una quantità di interesse rappresentata da un altro neurone.


Standard deep learning systems require thousands or millions of examples to learn a concept, and cannot integrate new concepts easily.By contrast, humans have an incredible ability to do one-shot or few-shot learning.For instance, from just hearing a word used in a sentence, humans can infer a great deal about it, by leveraging what the syntax and semantics of the surrounding words tells us.Here, we draw inspiration from this to highlight a simple technique by which deep recurrent networks can similarly exploit their prior knowledge to learn a useful representation for a new word from little data.This could make natural language processing systems much more flexible, by allowing them to learn continually from the new words they encounter.

Evidenziamo una tecnica con cui i sistemi di elaborazione del linguaggio naturale possono imparare una nuova parola dal contesto, permettendo loro di essere molto più flessibili.

Una tecnica per sfruttare la conoscenza precedente per imparare embeddings per nuove parole con dati minimi.


Recent research developing neural network architectures with external memory have often used the benchmark bAbI question and answering dataset which provides a challenging number of tasks requiring reasoning.Here we employed a classic associative inference task from the human neuroscience literature in order to more carefully probe the reasoning capacity of existing memory-augmented architectures.This task is thought to capture the essence of reasoning -- the appreciation of distant relationships among elements distributed across multiple facts or memories.Surprisingly, we found that current architectures struggle to reason over long distance associations.Similar results were obtained on a more complex task involving finding the shortest path between nodes in a path.We therefore developed a novel architecture, MEMO, endowed with the capacity to reason over longer distances.This was accomplished with the addition of two novel components.First, it introduces a separation between memories/facts stored in external memory and the items that comprise these facts in external memory.Second, it makes use of an adaptive retrieval mechanism, allowing a variable number of â€˜memory hopsâ€™ before the answer is produced.MEMO is capable of solving our novel reasoning tasks, as well as all 20 tasks in bAbI.

Un'architettura con memoria che supporta il ragionamento inferenziale.

Questo articolo propone modifiche all'architettura End2End Memory Network, introduce un nuovo task Paired Associative Inference che la maggior parte dei modelli esistenti fatica a risolvere, e mostra che l'architettura proposta risolve meglio il task.

Un nuovo task (paired associate inference) tratto dalla psicologia cognitiva, e la proposta di una nuova architettura con memoria con caratteristiche che permettono una migliore performance sul task paired associate.


Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency.They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures).Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results.In this work, we study how depthwise separable convolutions can be applied to neural machine translation.We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves better results.In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation.We also introduce a new super-separable convolution operation that further reduces the number of parameters and computational cost of the models.

Le convoluzioni separabili in profondità migliorano la traduzione automatica neurale: più sono separabili, meglio è.

Questo articolo propone di usare layer di convoluzione separabili in profondità in un modello di traduzione automatica neurale completamente convoluzionale, e introduce un nuovo layer di convoluzione super-separabile che riduce ulteriormente il costo computazionale.


Interpreting generative adversarial network (GAN) training as approximate divergence minimization has beentheoretically insightful, has spurred discussion, and has lead to theoretically and practically interestingextensions such as f-GANs and Wasserstein GANs.For both classic GANs and f-GANs, there is an original variant of training and a "non-saturating" variant which uses an alternative form of generator gradient.The original variant is theoretically easier to study, but for GANs the alternative variant performs better in practice.The non-saturating scheme is often regarded as a simple modification to deal with optimization issues, but we show that in fact the non-saturating scheme for GANs is effectively optimizing a reverse KL-like f-divergence.We also develop a number of theoretical tools to help compare and classify f-divergences.We hope these results may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.

Il training non saturante delle GAN minimizza efficacemente una f-divergenza inversa simile a KL.

Questo articolo propone un'utile espressione della classe delle f-divergenze, indaga le proprietà teoriche delle f-divergenze popolari in strumenti di recente sviluppo, e studia le GAN con lo schema di training non saturante.


We introduce a novel method for converting text data into abstract image representations, which allows image-based processing techniques (e.g. image classification networks) to be applied to text-based comparison problems.We apply the technique to entity disambiguation of inventor names in US patents.The method involves converting text from each pairwise comparison between two inventor name records into a 2D RGB (stacked) image representation.We then train an image classification neural network to discriminate between such pairwise comparison images, and use the trained network to label each pair of records as either matched (same inventor) or non-matched (different inventors), obtaining highly accurate results (F1: 99.09%, precision: 99.41%, recall: 98.76%).Our new text-to-image representation method could potentially be used more broadly for other NLP comparison problems, such as disambiguation of academic publications, or for problems that require simultaneous classification of both text and images.

Introduciamo un nuovo metodo di rappresentazione del testo che permette ai classificatori di immagini di essere applicati a problemi di text classification, e applichiamo il metodo alla disambiguazione dei nomi degli inventori.

Un metodo per mappare una coppia di informazioni testuali in un'immagine RGB 2D che può essere alimentata a reti neurali convoluzionali 2D (classificatori di immagini).

Gli autori considerano il problema della disambiguazione dei nomi per gli inventori di nomi di brevetti e propongono di costruire una rappresentazione di pagina come immagine delle due stringhe di nomi da confrontare e di applicare un classificatore di immagini.


We propose a novel algorithm, Difference-Seeking Generative Adversarial Network (DSGAN), developed from traditional GAN.DSGAN considers the scenario that the training samples of target distribution, $p_{t}$, are difficult to collect.Suppose there are two distributions  $p_{\bar{d}}$ and $p_{d}$ such that the density of the target distribution can be the differences between the densities of $p_{\bar{d}}$ and $p_{d}$.We show how to learn the target distribution $p_{t}$ only via samples from $p_{d}$ and $p_{\bar{d}}$ (relatively easy to obtain).DSGAN has the flexibility to produce samples from various target distributions (e.g. the out-of-distribution).Two key applications, semi-supervised learning and adversarial training, are taken as examples to validate the effectiveness of DSGAN.We also provide theoretical analyses about the convergence of DSGAN.

Abbiamo proposto il modello "Difference-Seeking Generative Adversarial Network" (DSGAN) per imparare la distribuzione target per la quale è difficile raccogliere dati di formazione.

Questo articolo presenta DS-GAN, che mira ad apprendere la differenza tra due distribuzioni qualsiasi i cui campioni sono difficili o impossibili da raccogliere, e mostra la sua efficacia nell'apprendimento semi-supervised e nei task di adversarial training.

Questo articolo considera il problema dell'apprendimento di una GAN per catturare una distribuzione target con solo pochi campioni di training disponibili per quella distribuzione .


Recently, Generative Adversarial Network (GAN) and numbers of its variants have been widely used to solve the image-to-image translation problem and achieved extraordinary results in both a supervised and unsupervised manner.However, most GAN-based methods suffer from the imbalance problem between the generator and discriminator in practice.Namely, the relative model capacities of the generator and discriminator do not match, leading to mode collapse and/or diminished gradients.To tackle this problem, we propose a GuideGAN based on attention mechanism.More specifically, we arm the discriminator with an attention mechanism so not only it estimates the probability that its input is real, but also does it create an attention map that highlights the critical features for such prediction.This attention map then assists the generator to produce more plausible and realistic images.We extensively evaluate the proposed GuideGAN framework on a  number of image transfer tasks.Both qualitative results and quantitative comparison demonstrate the superiority of our proposed approach.

Un metodo generale che migliora le prestazioni di image translation del framework GAN utilizzando un discriminatore integrato con attention

Un meccanismo di feedback nel framework GAN che migliora la qualità delle immagini generate nella image-to-image translation, e il cui discriminatore produce una mappa che indica dove il generatore dovrebbe concentrarsi per rendere i suoi risultati più convincenti.

Proposta di una GAN con un discriminatore basato sull'attention per la traduzione I2I che fornisce la probabilità di vero/falso e una mappa di attention che riflette la salienza per la generazione dell'immagine.


The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation.However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains unexplored.This paper specifically aims to study the fact verification given semi-structured data as evidence.To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED.TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning.To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA).Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification.LPA parses statements into LISP-like programs and executes them against the tables to obtain the returned binary value for verification.Both methods achieve similar accuracy but still lag far behind human performance.We also perform a comprehensive analysis to demonstrate great future opportunities.

Proponiamo un nuovo datset per investigare il problema di entailment sotto la tabella semi-strutturata come premessa

Questo articolo propone un nuovo dataset per fact verification basata sulle tabelle e introduce metodi per questo task.

Gli autori propongono il problema di fact verification con fonti di dati semi-strutturati come le tabelle, creano un nuovo set di dati e valutano le baseline con variazioni.


This work presents a two-stage neural architecture for learning and refining structural correspondences between graphs.First, we use localized node embeddings computed by a graph neural network to obtain an initial ranking of soft correspondences between nodes.Secondly, we employ synchronous message passing networks to iteratively re-rank the soft correspondences to reach a matching consensus in local neighborhoods between graphs.We show, theoretically and empirically, that our message passing scheme computes a well-founded measure of consensus for corresponding neighborhoods, which is then used to guide the iterative re-ranking process.Our purely local and sparsity-aware architecture scales well to large, real-world inputs while still being able to recover global correspondences consistently.We demonstrate the practical effectiveness of our method on real-world tasks from the fields of computer vision and entity alignment between knowledge graphs, on which we improve upon the current state-of-the-art.

Sviluppiamo un'architettura di deep graph matching che raffina le corrispondenze iniziali al fine di raggiungere il consenso del neighborhood.

Un framework per rispondere alle domande di graph matching che consiste in embeddings di nodi locali con una fase di raffinamento con message passing.

Un'architettura basata su GNN a due stadi per stabilire le corrispondenze tra due grafi che si comporta bene nei task reali di image matching e di entity alignment nei knowledge graph.


This paper extends the proof of density of neural networks in the space of continuous (or even measurable) functions on Euclidean spaces to functions on compact sets of probability measures.By doing so the work parallels a more then a decade old results on mean-map embedding of probability measures in reproducing kernel Hilbert spaces.  The work has wide practical consequences for multi-instance learning, where it theoretically justifies some recently proposed constructions.The result is then extended to Cartesian products, yielding universal approximation theorem for tree-structured domains, which naturally occur in data-exchange formats like JSON, XML, YAML, AVRO, and ProtoBuffer.This has important practical implications, as it enables to automatically create an architecture of neural networks for processing structured data (AutoML paradigms), as demonstrated by an accompanied library for JSON format.

Questo articolo estende la prova di densità delle reti neurali nello spazio delle funzioni continue (o anche misurabili) su spazi euclidei alle funzioni su insiemi compatti di misure di probabilità.

Questo articolo studia le proprietà di approssimazione di una famiglia di reti neurali progettate per affrontare problemi di learning multistanza, e mostra che i risultati per le architetture standard ad un layer si estendono a questi modelli.

Questo articolo generalizza il teorema di approssimazione universale alle funzioni reali sullo spazio delle misure.


Interactions such as double negation in sentences and scene interactions in images are common forms of complex dependencies captured by state-of-the-art machine learning models.We propose MahÃ©, a novel approach to provide Model-Agnostic Hierarchical Explanations of how powerful machine learning models, such as deep neural networks, capture these interactions as either dependent on or free of the context of data instances.Specifically, MahÃ© provides context-dependent explanations by a novel local interpretation algorithm that effectively captures any-order interactions, and obtains context-free explanations through generalizing context-dependent interactions to explain global behaviors.Experimental results show that MahÃ© obtains improved local interaction interpretations over state-of-the-art methods and successfully provides explanations of interactions that are context-free.

Un nuovo framework per le spiegazioni delle predizioni dipendenti dal contesto e senza contesto

Gli autori estendono il metodo di attribuzione locale lineare LIME per interpretare i modelli black box, e propongono un metodo per discernere tra interazioni dipendenti dal contesto e senza contesto.

Un metodo che può fornire spiegazioni gerarchiche per un modello, includendo sia spiegazioni dipendenti dal contesto che libere dal contesto tramite un algoritmo di interpretazione locale.


To realize the promise of ubiquitous embedded deep network inference, it is essential to seek limits of energy and area efficiency.  To this end, low-precision networks offer tremendous promise because both energy and area scale down quadratically with the reduction in precision.  Here, for the first time, we demonstrate ResNet-18, ResNet-34, ResNet-50, ResNet-152, Inception-v3, densenet-161, and VGG-16bn networks on the ImageNet classification benchmark that, at 8-bit precision exceed the accuracy of the full-precision baseline networks after one epoch of finetuning, thereby leveraging the availability of pretrained models.We also demonstrate ResNet-18, ResNet-34, and ResNet-50 4-bit models that match the accuracy of the full-precision baseline networks -- the highest scores to date.Surprisingly, the weights of the low-precision networks are very close (in cosine similarity) to the weights of the corresponding baseline networks, making training from scratch unnecessary.We find that gradient noise due to quantization during training increases with reduced precision, and seek ways to overcome this noise.The number of iterations required by stochastic gradient descent to achieve a given training error is related to the square of (a) the distance of the initial solution from the final plus (b) the maximum variance of the gradient estimates.  By drawing inspiration from this observation, we (a) reduce solution distance by starting with pretrained fp32 precision baseline networks and fine-tuning, and (b) combat noise introduced by quantizing weights and activations during training, by using larger batches along with matched learning rate annealing.  Sensitivity analysis indicates that these techniques, coupled with proper activation function range calibration, offer a promising heuristic to discover low-precision networks, if they exist, close to fp32 precision baseline networks.

Il finetuning dopo la quantizzazione corrisponde o supera le reti a piena precisione allo stato dell'arte sia a 8 che a 4 bit.

Questo articolo propone di migliorare le prestazioni dei modelli a bassa precisione facendo la quantizzazione sui modelli pre-trained, usando grandi batch size, e usando un adeguato annealing del learning rate con un tempo di training più lungo.

Un metodo per una bassa quantizzazione dei bit per permettere l'inferenza su un hardware efficiente che raggiunge la piena accuratezza su ResNet50 con pesi e attivazioni a 4 bit, basato sull'osservazione che il fine-tuning a bassa precisione introduce rumore nel gradiente.


Analysis methods which enable us to better understand the  representations and functioning of neural models of language are  increasingly needed as deep learning becomes the dominant approach  in NLP.Here we present two methods based on Representational  Similarity Analysis (RSA) and Tree Kernels (TK) which allow us to  directly quantify how strongly the information encoded in neural  activation patterns corresponds to information represented by  symbolic structures such as syntax trees.We first validate our  methods on the case of a simple synthetic language for arithmetic  expressions with clearly defined syntax and semantics, and show that  they exhibit the expected pattern of results.We then apply our methods to  correlate neural representations of English sentences with their  constituency parse trees.

Due metodi basati su Representational Similarity Analysis (RSA) e Tree Kernels (TK) che quantificano direttamente quanto l'informazione codificata nei modelli di attivazione neurale corrisponde all'informazione rappresentata da strutture simboliche.


Supervised deep learning requires a large amount of training samples with annotations (e.g. label class for classification task, pixel- or voxel-wised label map for segmentation tasks), which are expensive and time-consuming to obtain.During the training of a deep neural network, the annotated samples are fed into the network in a mini-batch way, where they are often regarded of equal importance.However, some of the samples may become less informative during training, as the magnitude of the gradient start to vanish for these samples.In the meantime, other samples of higher utility or hardness may be more demanded for the training process to proceed and require more exploitation.To address the challenges of expensive annotations and loss of sample informativeness, here we propose a novel training framework which adaptively selects informative samples that are fed to the training process.The adaptive selection or sampling is performed based on a hardness-aware strategy in the latent space constructed by a generative model.To evaluate the proposed training framework, we perform experiments on three different datasets, including MNIST and CIFAR-10 for image classification task and a medical image dataset IVUS for biophysical simulation task.On all three datasets, the proposed framework outperforms a random sampling method, which demonstrates the effectiveness of our framework.

Questo articolo introduce un framework per l'apprendimento data-efficient di rappresentazioni attraverso il campionamento adattivo nello spazio latente.

Un metodo per la selezione sequenziale e adattiva dei training examples da presentare all'algoritmo di training, dove la selezione avviene nello spazio latente basato sulla scelta dei campioni nella direzione del gradiente della loss.

Un metodo per selezionare in modo efficiente i campioni difficili durante il training delle reti neurali, ottenuto tramite un variational auto-encoder che codifica i campioni in uno spazio latente.


Existing methods for AI-generated artworks still struggle with generating high-quality stylized content, where high-level semantics are preserved, or separating fine-grained styles from various artists.We propose a novel Generative Adversarial Disentanglement Network which can disentangle two complementary factors of variations when only one of them is labelled in general, and fully decompose complex anime illustrations into style and content in particular.Training such model is challenging, since given a style, various content data may exist but not the other way round.Our approach is divided into two stages, one that encodes an input image into a style independent content, and one based on a dual-conditional generator.We demonstrate the ability to generate high-fidelity anime portraits with a fixed content and a large variety of styles from over a thousand artists, and vice versa, using a single end-to-end network and with applications in style transfer.We show this unique capability as well as superior output to the current state-of-the-art.

Un metodo basato sull'adversarial training per distinguere due insiemi complementari di variazioni in un dataset in cui solo uno di essi è etichettato, testato su stile vs. contenuto nelle illustrazioni di anime.

Un metodo di generazione di immagini che combina GAN condizionali e VAE condizionali che genera immagini di anime ad alta fedeltà con vari stili di vari artisti.

Proposta di un metodo per imparare rappresentazioni disgiunte di stile (artista) e contenuto negli anime.