We propose an approach to training machine learning models that are fair in the sense that their performance is invariant under certain perturbations to the features.For example, the performance of a resume screening system should be invariant under changes to the name of the applicant.We formalize this intuitive notion of fairness by connecting it to the original  notion of individual fairness put forth by Dwork et al and show that the proposed approach achieves this notion of fairness.We also demonstrate the effectiveness of the approach on two machine learning tasks that are susceptible to gender and racial biases.

Algoritmo per il training di classificatori individualmente fair utilizzando l'adversarial robustness

Questo articolo propone una nuova definizione di fairness algoritmica e un algoritmo per trovare in modo dimostrabile un modello ML che soddisfi il vincolo di fairness.


In this paper, we propose a Seed-Augment-Train/Transfer (SAT) framework that contains a synthetic seed image dataset generation procedure for languages with different numeral systems using freely available open font file datasets.This seed dataset of images is then augmented to create a purely synthetic training dataset, which is in turn used to train a deep neural network and test on held-out real world handwritten digits dataset spanning five Indic scripts, Kannada, Tamil, Gujarati, Malayalam, and Devanagari.We showcase the efficacy of this approach both qualitatively, by training a Boundary-seeking GAN (BGAN) that generates realistic digit images in the five languages, and also qualitatively by testing a CNN trained on the synthetic data on the real-world datasets.This establishes not only an interesting nexus between the font-datasets-world and transfer learning but also provides a recipe for universal-digit classification in any script.

Il seeding e l'augmentation sono tutto ciò che serve per classificare le cifre in qualsiasi lingua?

Questo articolo presenta nuovi dataset per cinque lingue e propone un nuovo framework (SAT) per la generazione di dataset di immagini di font per la classificazione universale delle cifre.


An important research direction in machine learning has centered around developing meta-learning algorithms to tackle few-shot learning.An especially successful algorithm has been Model Agnostic Meta-Learning (MAML), a method that consists of two optimization loops, with the outer loop finding a meta-initialization, from which the inner loop can efficiently learn new tasks.Despite MAML's popularity, a fundamental open question remains -- is the effectiveness of MAML due to the meta-initialization being primed for rapid learning (large, efficient changes in the representations) or due to feature reuse,  with the meta initialization already containing high quality features?We investigate this question, via ablation studies and analysis of the latent representations, finding that feature reuse is the dominant factor.This leads to the ANIL (Almost No Inner Loop) algorithm, a simplification of MAML where we remove the inner loop for all but the (task-specific) head of the underlying neural network.ANIL matches MAML's performance on benchmark few-shot image classification and RL and offers computational improvements over MAML.We further study the precise contributions of the head and body of the network, showing that performance on the test tasks is entirely determined by the quality of the learned features, and we can remove even the head of the network (the NIL algorithm).We conclude with a discussion of the rapid learning vs feature reuse question for meta-learning algorithms more broadly.

Il successo di MAML si basa sul riutilizzo delle feature dalla meta-inizializzazione, che produce anche una semplificazione naturale dell'algoritmo, con il ciclo interno rimosso per il corpo della rete, così come altri approfondimenti sulla testa e sul corpo.

L'articolo trova che il riutilizzo delle feature è il fattore dominante nel successo di MAML, e propone nuovi algoritmi che richiedono molto meno calcolo di MAML.


Model training remains a dominant financial cost and time investment in machine learning applications.Developing and debugging models often involve iterative training, further exacerbating this issue.With growing interest in increasingly complex models, there is a need for techniques that help to reduce overall training effort.While incremental training can save substantial time and cost by training an existing model on a small subset of data, little work has explored policies for determining when incremental training provides adequate model performance versus full retraining.We provide a method-agnostic algorithm for deciding when to incrementally train versus fully train.We call this setting of non-deterministic full- or incremental training ``Mixed Setting Training".Upon evaluation in slot-filling tasks, we find that this algorithm provides a bounded error, avoids catastrophic forgetting, and results in a significant speedup over a policy of always fully training.

Forniamo un algoritmo indipendente dal metodo per decidere quando addestrare in modo incrementale rispetto al training completo e fornisce un significativo aumento di velocità rispetto al training completo ed evita la catastrophic forgetting

Questo articolo propone un approccio per decidere quando incrementare o riqualificare completamente un modello nel setting dello sviluppo iterativo del modello nei task di slot filling.


Neural networks have succeeded in many reasoning tasks.Empirically, these tasks require specialized network structures, e.g., Graph Neural Networks (GNNs) perform well on many such tasks, while less structured networks fail.Theoretically, there is limited understanding of why and when a network structure generalizes better than other equally expressive ones.We develop a framework to characterize which reasoning tasks a network can learn well, by studying how well its structure aligns with the algorithmic structure of the relevant reasoning procedure.We formally define algorithmic alignment and derive a sample complexity bound that decreases with better alignment.This framework explains the empirical success of popular reasoning models and suggests their limitations.We unify seemingly different reasoning tasks, such as intuitive physics, visual question answering, and shortest paths, via the lens of a powerful algorithmic paradigm, dynamic programming (DP).We show that GNNs can learn DP and thus solve these tasks.On several reasoning tasks, our theory aligns with empirical results.

Sviluppiamo un framework teorico per caratterizzare quali task di ragionamento una rete neurale può imparare bene.

L'articolo propone una misura di classi di allineamento algoritmico che misura quanto le reti neurali sono "vicine" agli algoritmi conosciuti, dimostrando il legame tra diverse classi di algoritmi conosciuti e le architetture delle reti neurali.


Cell-cell interactions have an integral role in tumorigenesis as they are critical in governing immune responses.As such, investigating specific cell-cell interactions has the potential to not only expand upon the understanding of tumorigenesis, but also guide clinical management of patient responses to cancer immunotherapies.A recent imaging technique for exploring cell-cell interactions, multiplexed ion beam imaging by time-of-flight (MIBI-TOF), allows for cells to be quantified in 36 different protein markers at sub-cellular resolutions in situ as high resolution multiplexed images.To explore the MIBI images, we propose a GAN for multiplexed data with protein specific attention.By conditioning image generation on cell types, sizes, and neighborhoods through semantic segmentation maps, we are able to observe how these factors affect cell-cell interactions simultaneously in different protein channels.Furthermore, we design a set of metrics and offer the first insights towards cell spatial orientations, cell protein expressions, and cell neighborhoods.Our model, cell-cell interaction GAN (CCIGAN), outperforms or matches existing image synthesis methods on all conventional measures and significantly outperforms on biologically motivated metrics.To our knowledge, we are the first to systematically model multiple cellular protein behaviors and interactions under simulated conditions through image synthesis.

Esploriamo le interazioni cellula-cellula attraverso i contesti dell'ambiente tumorale osservati in immagini altamente multiplexed, attraverso la sintesi delle immagini utilizzando una nuova architettura GAN di attention.

Un nuovo metodo per modellare i dati generati dal multiplexed ion beam imaging by time-of-flight (MIBI-TOF) imparando la mappatura many-to-many tra i tipi di cellule e i livelli di espressione dei marcatori di proteine.


Machine learning models for question-answering (QA), where given a question and a passage, the learner must select some span in the passage as an answer, are known to be brittle.By inserting a single nuisance sentence into the passage, an adversary can fool the model into selecting the wrong span.A promising new approach for QA decomposes the task into two stages:(i) select relevant sentences from the passage; and(ii) select a span among those sentences.Intuitively, if the sentence selector excludes the offending sentence, then the downstream span selector will be robust.While recent work has hinted at the potential robustness of two-stage QA, these methods have never, to our knowledge, been explicitly combined with adversarial training.This paper offers a thorough empirical investigation of adversarial robustness, demonstrating that although the two-stage approach lags behind single-stage span selection, adversarial training improves its performance significantly, leading to an improvement of over 22 points in F1 score over the adversarially-trained single-stage model.

Un approccio a due stadi che consiste nella selezione della frase seguita dalla selezione dello span può essere reso più robusto agli adversarial attack rispetto a un modello a stadio singolo addestrato sul contesto completo.

Questo articolo esamina un modello esistente e scopre che un metodo di QA addestrato in due fasi non è più robusto agli adversarial attack rispetto ad altri metodi.


The aim of this study is to introduce a formal framework for analysis and synthesis of driver assistance systems.It applies formal methods to the verification of a stochastic human driver model built using the cognitive architecture ACT-R, and then bootstraps safety in semi-autonomous vehicles through the design of provably correct Advanced Driver Assistance Systems.The main contributions include the integration of probabilistic ACT-R models in the formal analysis of semi-autonomous systems and an abstraction technique that enables a finite representation of a large dimensional, continuous system in the form of a Markov model.The effectiveness of the method is illustrated in several case studies under various conditions.

Verifica di un modello di guidatore umano basato su un'architettura cognitiva e sintesi di un ADAS corretto per costruzione da esso.


In contrast to the older writing system of the 19th century, modern Hawaiian orthography employs characters for long vowels and glottal stops.These extra characters account for about one-third of the phonemes in Hawaiian, so including them makes a big difference to reading comprehension and pronunciation.However, transliterating between older and newer texts is a laborious task when performed manually.We introduce two related methods to help solve this transliteration problem automatically, given that there were not enough data to train an end-to-end deep learning model.One approach is implemented, end-to-end, using finite state transducers (FSTs).The other is a hybrid deep learning approach which approximately composes an FST with a recurrent neural network (RNN).We find that the hybrid approach outperforms the end-to-end FST by partitioning the original problem into one part that can be modelled by hand, using an FST, and into another part, which is easily solved by an RNN trained on the available data.

Un nuovo approccio ibrido di deep learning fornisce la migliore soluzione a un problema di dati limitati (che è importante per la conservazione della lingua hawaiana)


In many real-world settings, a learning model must perform few-shot classification: learn to classify examples from unseen classes using only a few labeled examples per class.Additionally, to be safely deployed, it should have the ability to detect out-of-distribution inputs: examples that do not belong to any of the classes.While both few-shot classification and out-of-distribution detection are popular topics,their combination has not been studied.In this work, we propose tasks for out-of-distribution detection in the few-shot setting and establish benchmark datasets, based on four popular few-shot classification datasets.  Then, we propose two new methods for this task and investigate their performance.In sum, we establish baseline out-of-distribution detection results using standard metrics on new benchmark datasets and show improved results with our proposed methods.

Studiamo quantitativamente il rilevamento di out-of-distribution nel setting few-shot, stabiliamo baseline con ProtoNet, MAML, ABML, e li miglioriamo.

L'articolo propone due nuovi punteggi di confidence che sono più adatti per il rilevamento di out-of-distribution della classificazione few-shot e mostra che un approccio basato sulla metrica della distanza migliora le prestazioni.


While modern  generative  models are able to synthesize high-fidelity, visually appealing images, successfully generating examples that are useful for recognition tasks remains an elusive goal.To this end, our key insight is that the examples should be synthesized to recover classifier decision boundaries that would be learned from a large amount of real examples.More concretely, we treat a classifier trained on synthetic examples as ''student'' and a classifier trained on real examples as ''teacher''.By introducing knowledge distillation into a meta-learning framework, we encourage the generative model to produce examples in a way that enables the student classifier to mimic the behavior of the teacher.To mitigate the potential gap between student and teacher classifiers, we further propose to distill the knowledge in a progressive manner, either by gradually strengthening the teacher or weakening the student.We demonstrate the use of our model-agnostic distillation approach to deal with data scarcity, significantly improving few-shot learning performance on miniImageNet and ImageNet1K benchmarks.

Questo articolo introduce la progressive knowledge distillation per l'apprendimento di modelli generativi che sono orientati a task di riconoscimento

Questo articolo dimostra il curriculum learning easy-to-hard per addestrare un modello generativo per migliorare la classificazione few-shot.


Deep neural networks provide state-of-the-art performance for many applications of interest.Unfortunately they are known to be vulnerable to adversarial examples, formed by applying small but malicious perturbations to the original inputs.Moreover, the perturbations can transfer across models: adversarial examples generated for a specific model will often mislead other unseen models.Consequently  the adversary can leverage it to attack against the deployed black-box systems. In this work, we demonstrate that the adversarial perturbation can be decomposed into two components: model-specific and data-dependent one, and it is the latter that mainly contributes to the transferability.Motivated by this understanding, we propose to craft adversarial examples by utilizing the noise reduced gradient (NRG) which approximates the data-dependent component.Experiments on various classification models trained on ImageNet demonstrates that the new approach enhances the transferability dramatically.We also find that low-capacity models have more powerful attack capability than high-capacity counterparts, under the condition that they have comparable test performance.  These insights give rise to a principled manner to construct adversarial examples with high success rates and could potentially provide us guidance for designing effective defense approaches against black-box attacks.

Proponiamo un nuovo metodo per migliorare la trasferibilità degli adversarial sample utilizzando il gradiente noise-reduced.

Questo articolo postula che una adversarial perturbation consiste in una componente specifica del modello e in una componente specifica dei dati, e che l'amplificazione di quest'ultima è più adatta per gli adversarial attack.

Questo articolo si concentra sul miglioramento della trasferibilità degli adversarial sample da un modello a un altro modello.


We present the iterative two-pass decomposition flow to accelerate existing convolutional neural networks (CNNs).  The proposed rank selection algorithm can effectively determine the proper ranks of the target convolutional layers for the low rank approximation.Our two-pass CP-decomposition helps prevent from the instability problem.The iterative flow makes the decomposition of the deeper networks systematic.The experiment results shows that VGG16 can be accelerated with a 6.2x measured speedup while the accuracy drop remains only 1.2%.

Presentiamo il flusso iterativo di decomposizione CP a due step per accelerare efficacemente le reti neurali convoluzionali esistenti (CNN).

L'articolo propone un nuovo flusso di lavoro per l'accelerazione e la compressione delle CNN e propone anche un modo per determinare il rango target di ogni layer data l'accelerazione globale target.

Questo articolo affronta il problema dell'apprendimento di un'operazione con filtro tensore a basso rango per i layer di filtraggio nelle deep neural network (DNN).


We introduce LiPopt, a polynomial optimization framework for computing increasingly tighter upper bound on the Lipschitz constant of neural networks.The underlying optimization problems boil down to either linear (LP) or semidefinite (SDP) programming.We show how to use the sparse connectivity of a network, to significantly reduce the complexity of computation.This is specially useful for convolutional as well as pruned neural networks.We conduct experiments on networks with random weights as well as networks trained on MNIST, showing that in the particular case of the $\ell_\infty$-Lipschitz constant, our approach yields superior estimates as compared to other baselines available in the literature.

Limiti superiori basati su LP sulla costante di Lipschitz delle reti neurali

Gli autori studiano il problema della stima della costante di Lipschitz di una deep neural network con funzione di attivazione ELO, formulandolo come un problema di ottimizzazione polinomiale.


Although few-shot learning research has advanced rapidly with the help of meta-learning, its practical usefulness is still limited because most of the researches assumed that all meta-training and meta-testing examples came from a single domain.We propose a simple but effective way for few-shot classification in which a task distribution spans multiple domains including previously unseen ones during meta-training.The key idea is to build a pool of embedding models which have their own metric spaces and to learn to select the best one for a particular task through multi-domain meta-learning.This simplifies task-specific adaptation over a complex task distribution as a simple selection problem rather than modifying the model with a number of parameters at meta-testing time.Inspired by common multi-task learning techniques, we let all models in the pool share a base network and add a separate modulator to each model to refine the base network in its own way.This architecture allows the pool to maintain representational diversity and each model to have domain-invariant representation as well. Experiments show that our selection scheme outperforms other few-shot classification algorithms when target tasks could come from many different domains.They also reveal that aggregating outputs from all constituent models is effective for tasks from unseen domains showing the effectiveness of our framework.

Affrontiamo la classificazione multi-dominio few-shot costruendo modelli multipli per rappresentare questa complessa distribuzione di task in modo collettivo e semplificando l'adattamento specifico del task come un problema di selezione da questi modelli pre-trained.

Questo articolo affronta la classificazione few-shot con molti domini diversi costruendo un pool di modelli di embedding per catturare feature invarianti e specifiche del dominio senza un aumento significativo del numero di parametri.


Still in 2019, many scanned documents come into businesses in non-digital format.Text to be extracted from real world documents is often nestled inside rich formatting, such as tabular structures or forms with fill-in-the-blank boxes or underlines whose ink often touches or even strikes through the ink of the text itself.Such ink artifacts can severely interfere with the performance of recognition algorithms or other downstream processing tasks.In this work, we propose DeepErase, a neural preprocessor to erase ink artifacts from text images.We devise a method to programmatically augment text images with real artifacts, and use them to train a segmentation network in an weakly supervised manner.In additional to high segmentation accuracy, we show that our cleansed images achieve a significant boost in downstream recognition accuracy by popular OCR software such as Tesseract 4.0.We test DeepErase on out-of-distribution datasets (NIST SDB) of scanned IRS tax return forms and achieve double-digit improvements in recognition accuracy over baseline for both printed and handwritten text.

Rimozione su base neurale degli artefatti di inchiostro dei documenti (sottolineature, sbavature, ecc.) senza dati di training annotati manualmente


Black-box adversarial attacks require a large number of attempts before finding successful adversarial examples that are visually indistinguishable from the original input.Current approaches relying on substitute model training, gradient estimation or genetic algorithms often require an excessive number of queries.Therefore, they are not suitable for real-world systems where the maximum query number is limited due to cost.We propose a query-efficient black-box attack which uses Bayesian optimisation in combination with Bayesian model selection to optimise over the adversarial perturbation and the optimal degree of search space dimension reduction.We demonstrate empirically that our method can achieve comparable success rates with 2-5 times fewer queries compared to previous state-of-the-art black-box attacks.

Proponiamo un attacco black-box query-efficiente che utilizza l'ottimizzazione bayesiana in combinazione con la selezione del modello bayesiano per ottimizzare l'adversarial perturbation e il grado ottimale di riduzione della dimensione dello spazio di ricerca.

Gli autori propongono di utilizzare l'ottimizzazione bayesiana con un surrogato GP per la generazione di immagini adversarial, sfruttando la struttura additiva e utilizzando la selezione bayesiana del modello per determinare una riduzione ottimale della dimensionalità.


Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information.Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data:1) models must learn the complex intra-modal and cross-modal interactions for prediction and2) models must be robust to unexpected missing or noisy modalities during testing.In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels.We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors.Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction.Modality-specific generative factors are unique for each modality and contain the information required for generating data.Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets.Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance.Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning.

Proponiamo un modello per imparare rappresentazioni multimodali fattorizzate che sono discriminative, generative e interpretabili.

Questo articolo presenta il "modello di fattorizzazione multimodale" che fattorizza le rappresentazioni in fattori discriminativi multimodali condivisi e fattori generativi specifici della modalità.


The successful application of flexible, general learning algorithms to real-world robotics applications is often limited by their poor data-efficiency.To address the challenge, domains with more than one dominant task of interest encourage the sharing of information across tasks to limit required experiment time.To this end, we investigate compositional inductive biases in the form of hierarchical policies as a mechanism for knowledge transfer across tasks in reinforcement learning (RL).We demonstrate that this type of hierarchy enables positive transfer while mitigating negative interference.Furthermore, we demonstrate the benefits of additional incentives to efficiently decompose task solutions.Our experiments show that these incentives are naturally given in multitask learning and can be easily introduced for single objectives.We design an RL algorithm that enables stable and fast learning of structured policies and the effective reuse of both behavior components and transition data across tasks in an off-policy setting.Finally, we evaluate our algorithm in simulated environments as well as physical robot experiments and  demonstrate substantial improvements in data data-efficiency over competitive baselines.

Sviluppiamo un algoritmo gerarchico e actor-critic per il transfer compositivo attraverso la condivisione di componenti di policy e dimostriamo la specializzazione dei componenti e i relativi benefici diretti nei domini multitask, così come il suo adattamento per task singoli.

Una combinazione di diverse tecniche di apprendimento per l'acquisizione della struttura e l'apprendimento con dati asimmetrici, utilizzati per addestrare una policy HRL.

Gli autori introducono un framework di policy gerarchica per l'uso nel reinforcement learning sia a task singolo che multitask, e valutano l'utilità della struttura su task robotici complessi.


In this paper, we study the representational power of deep neural networks (DNN) that belong to the family of piecewise-linear (PWL) functions, based on PWL activation units such as rectifier or maxout.We investigate the complexity of such networks by studying the number of linear regions of the PWL function.Typically, a PWL function from a DNN can be seen as a large family of linear functions acting on millions of such regions.We directly build upon the work of MontÂ´ufar et al. (2014), MontÂ´ufar (2017), and Raghu et al. (2017) by refining the upper and lower bounds on the number of linear regions for rectified and maxout networks.In addition to achieving tighter bounds, we also develop a novel method to perform exact numeration or counting of the number of linear regions with a mixed-integer linear formulation that maps the input space to output.We use this new capability to visualize how the number of linear regions change while training DNNs.

Contiamo empiricamente il numero di regioni lineari delle reti con ReLU e raffiniamo i limiti superiori e inferiori.

Questo articolo presenta dei limiti migliorati per il conteggio del numero di regioni lineari nelle reti ReLU.


Convolutional neural networks memorize part of their training data, which is why strategies such as data augmentation and drop-out are employed to mitigate over- fitting.This paper considers the related question of â€œmembership inferenceâ€, where the goal is to determine if an image was used during training.We con- sider membership tests over either ensembles of samples or individual samples.First, we show how to detect if a dataset was used to train a model, and in particular whether some validation images were used at train time.Then, we introduce a new approach to infer membership when a few of the top layers are not available or have been fine-tuned, and show that lower layers still carry information about the training samples.To support our findings, we conduct large-scale experiments on Imagenet and subsets of YFCC-100M with modern architectures such as VGG and Resnet.

Analizziamo le proprietà di memorizzazione tramite un convnet del set di allenamento e proponiamo diversi casi d'uso in cui possiamo estrarre alcune informazioni sul training set.

Mostra le proprietà di generalizzazione/memorizzazione delle Convnet grandi e deep e cerca di sviluppare procedure relative all'identificazione se un input di una ConvNet addestrata è stato effettivamente utilizzato per addestrare la rete.


While Generative Adversarial Networks (GANs) have empirically produced impressive results on learning complex real-world distributions, recent works have shown that they suffer from lack of diversity or mode collapse.The theoretical work of Arora et al. (2017a) suggests a dilemma about GANsâ€™ statistical properties: powerful discriminators cause overfitting, whereas weak discriminators cannot detect mode collapse.By contrast, we show in this paper that GANs can in principle learn distributions in Wasserstein distance (or KL-divergence in many cases) with polynomial sample complexity, if the discriminator class has strong distinguishing power against the particular generator class (instead of against all possible generators).For various generator classes such as mixture of Gaussians, exponential families, and invertible and injective neural networks generators, we design corresponding discriminators (which are often neural nets of specific architectures) such that the Integral Probability Metric (IPM) induced by the discriminators can provably approximate the Wasserstein distance and/or KL-divergence.This implies that if the training is successful, then the learned distribution is close to the true distribution in Wasserstein distance or KL divergence, and thus cannot drop modes.Our preliminary experiments show that on synthetic datasets the test IPM is well correlated with KL divergence or the Wasserstein distance, indicating that the lack of diversity in GANs may be caused by the sub-optimality in optimization instead of statistical inefficiency.

Le GAN possono in linea di principio imparare distribuzioni in modo efficiente dal punto di vista del sample, se la classe discriminante è compatta e ha un forte potere di distinzione rispetto alla particolare classe generatrice.

Propone la nozione di approssimabilità ristretta e fornisce un limite di complessità del sample, polinomiale nella dimensione, che è utile nello studio della mancanza di diversità nelle GAN.

Discute sul fatto che la metrica di probabilità integrale può essere una buona approssimazione della distanza di Wasserstein sotto alcune ipotesi lievi.


Understanding the optimization trajectory is critical to understand training of deep neural networks.We show how the hyperparameters of stochastic gradient descent influence the covariance of the gradients (K) and the Hessian of the training loss (H) along this trajectory.Based on a theoretical model, we predict that using a high learning rate or a small batch size in the early phase of training leads SGD to regions of the parameter space with (1) reduced spectral norm of K, and (2) improved conditioning of K and H. We show that the point on the trajectory after which these effects hold, which we refer to as the break-even point, is reached early during training.We demonstrate these effects empirically for a range of deep neural networks applied to multiple different tasks.Finally, we apply our analysis to networks with batch normalization (BN) layers and find that it is necessary to use a high learning rate to achieve loss smoothing effects attributed previously to BN alone.

Nella fase iniziale del training delle deep neural network esiste un "break-even point" che determina le proprietà dell'intera traiettoria di ottimizzazione.

Questo lavoro analizza l'ottimizzazione delle deep neural network considerando come gli iperparametri batch size e step size modifichino le traiettorie di apprendimento.


Graph Convolution Network (GCN) has been recognized as one of the most effective graph models for semi-supervised learning, but it extracts merely the first-order or few-order neighborhood information through information propagation, which suffers performance drop-off for deeper structure.Existing approaches that deal with the higher-order neighbors tend to take advantage of adjacency matrix power.In this paper, we assume a seemly trivial condition that the higher-order neighborhood information may be similar to that of the first-order neighbors.Accordingly, we present an unsupervised approach to describe such similarities and learn the weight matrices of higher-order neighbors automatically through Lasso that minimizes the feature loss between the first-order and higher-order neighbors, based on which we formulate the new convolutional filter for GCN to learn the better node representations.Our model, called higher-order weighted GCN (HWGCN), has achieved the state-of-the-art results on a number of node classification tasks over Cora, Citeseer and Pubmed datasets.

Proponiamo HWGCN per mescolare a diversi ordini le informazioni di neighborhood pertinenti per imparare meglio le rappresentazioni dei nodi.

Gli autori propongono una variante di GCN, HWGCN, per considerare la convoluzione oltre i vicini ad uno step, che è paragonabile ai metodi allo stato dell'arte.


The performance of deep neural networks is often attributed to their automated, task-related feature construction.It remains an open question, though, why this leads to solutions with good generalization, even in cases where the number of parameters is larger than the number of samples.Back in the 90s, Hochreiter and Schmidhuber observed that flatness of the loss surface around a local minimum correlates with low generalization error.For several flatness measures, this correlation has been empirically validated.However, it has recently been shown that existing measures of flatness cannot theoretically be related to generalization: if a network uses ReLU activations, the network function can be reparameterized without changing its output in such a way that flatness is changed almost arbitrarily.This paper proposes a natural modification of existing flatness measures that results in invariance to reparameterization.The proposed measures imply a robustness of the network to changes in the input and the hidden layers.Connecting this feature robustness to generalization leads to a generalized definition of the representativeness of data.With this, the generalization error of a model trained on representative data can be bounded by its feature robustness which depends on our novel flatness measure.

Introduciamo una nuova misura di planarità ai minimi locali della superficie di loss delle deep neural network che è invariante rispetto alle riparametrizzazioni a livello di layer e colleghiamo la planarità alla robustezza delle feature e alla generalizzazione.

Gli autori propongono una nozione di robustezza delle feature che è invariante rispetto al ridimensionamento dei pesi e discutono la relazione di questa nozione con la generalizzazione.

Questo articolo definisce una nozione di feature-robustness e la combina con la epsilon representativeness di una funzione per descrivere una connessione tra flatness dei minimi e generalizzazione nelle deep neural network.


Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e.g.neurons.We apply and further develop this approach for gated recurrent architectures.Specifically, in addition to sparsification of individual weights and neurons, we propose to sparsify preactivations of gates and information flow in LSTM.It makes some gates and information flow components constant, speeds up forward pass and improves compression.Moreover, the resulting structure of gate sparsity is interpretable and depends on the task.

Proponiamo di sparsificare le preattivazioni dei gate e il flusso di informazioni in LSTM per renderle costanti e aumentare il livello di sparsità dei neuroni

Questo articolo ha proposto un metodo di sparsificazione per le reti neurali ricorrenti eliminando i neuroni con preattivazioni zero per ottenere reti compatte.


Improving the accuracy of numerical methods remains a central challenge in many disciplines and is especially important for nonlinear simulation problems.A representative example of such problems is fluid flow, which has been thoroughly studied to arrive at efficient simulations of complex flow phenomena.This paper presents a data-driven approach that learns to improve the accuracy of numerical solvers.The proposed method utilizes an advanced numerical scheme with a fine simulation resolution to acquire reference data.We, then, employ a neural network that infers a correction to move a coarse thus quickly obtainable result closer to the reference data.We provide insights into the targeted learning problem with different learning approaches: fully supervised learning methods with a naive and an optimized data acquisition as well as an unsupervised learning method with a differentiable Navier-Stokes solver.While our approach is very general and applicable to arbitrary partial differential equation models, we specifically highlight gains in accuracy for fluid flow simulations.

Introduciamo un approccio di rete neurale per assistere i risolutori di equazioni differenziali parziali.

Gli autori mirano a migliorare la precisione dei risolutori numerici addestrando una rete neurale su dati di riferimento simulati che correggono il risolutore numerico.


A patientâ€™s health information is generally fragmented across silos.Though it is technically feasible to unite data for analysis in a manner that underpins a rapid learning healthcare system, privacy concerns and regulatory barriers limit data centralization.Machine learning can be conducted in a federated manner on patient datasets with the same set of variables, but separated across sites of care.But federated learning cannot handle the situation where different data types for a givenpatient are separated vertically across different organizations.We call methods that enable machine learning model training on data separated by two or more degrees â€œconfederated machine learning.â€We built and evaluated a confederated machinelearningmodel to stratify the risk of accidental falls among the elderly.

un metodo di confederated learning che allena il modello da dati medici separati orizzontalmente e verticalmente

Un metodo di confederated learning che impara attraverso le divisioni nei dati medici separati sia orizzontalmente che verticalmente.


Existing neural networks are vulnerable to "adversarial examples"---created by adding maliciously designed small perturbations in inputs to induce a misclassification by the networks.The most investigated defense strategy is adversarial training which augments training data with adversarial examples.However, applying single-step adversaries in adversarial training does not support the robustness of the networks, instead, they will even make the networks to be overfitted.In contrast to the single-step, multi-step training results in the state-of-the-art performance on MNIST and CIFAR10, yet it needs a massive amount of time.Therefore, we propose a method, Stochastic Quantized Activation (SQA) that solves overfitting problems in single-step adversarial training and fastly achieves the robustness comparable to the multi-step.SQA attenuates the adversarial effects by providing random selectivity to activation functions and allows the network to learn robustness with only single-step training.Throughout the experiment, our method demonstrates the state-of-the-art robustness against one of the strongest white-box attacks as PGD training, but with much less computational cost.Finally, we visualize the learning process of the network with SQA to handle strong adversaries, which is different from existing methods.

Questo articolo propone l'attivazione quantizzata stocastica che risolve i problemi di overfitting nell'adversarial training FGSM e raggiunge rapidamente la robustezza paragonabile al training multi-step.

L'articolo propone un modello per migliorare l'adversarial training introducendo perturbazioni casuali nelle attivazioni di uno degli hidden layer


Neural activity is highly variable in response to repeated stimuli.We used an open dataset, the Allen Brain Observatory, to quantify the distribution of responses to repeated natural movie presentations.A large fraction of responses are best fit by log-normal distributions or Gaussian mixtures with two components.These distributions are similar to those from units in deep neural networks with dropout.Using a separate set of electrophysiological recordings, we constructed a population coupling model as a control for state-dependent activity fluctuations and found that the model residuals also show non-Gaussian distributions.We then analyzed responses across trials from multiple sections of different movie clips and observed that the noise in cortex aligns better with in-clip versus out-of-clip stimulus variations.We argue that noise is useful for generalization when it moves along representations of different exemplars in-class, similar to the structure of cortical noise.

Studiamo la struttura del rumore nel cervello e scopriamo che può aiutare la generalizzazione spostando le rappresentazioni lungo le variazioni degli stimoli in-class.


Unsupervised domain adaptation has received significant attention in recent years.Most of existing works tackle the closed-set scenario, assuming that the source and target domains share the exactly same categories.In practice, nevertheless, a target domain often contains samples of classes unseen in source domain (i.e., unknown class).The extension of domain adaptation from closed-set to such open-set situation is not trivial since the target samples in unknown class are not expected to align with the source.In this paper, we address this problem by augmenting the state-of-the-art domain adaptation technique, Self-Ensembling, with category-agnostic clusters in target domain.Specifically, we present Self-Ensembling with Category-agnostic Clusters (SE-CC) --- a novel architecture that steers domain adaptation with the additional guidance of category-agnostic clusters that are specific to target domain.These clustering information provides domain-specific visual cues, facilitating the generalization of Self-Ensembling for both closed-set and open-set scenarios.Technically, clustering is firstly performed over all the unlabeled target samples to obtain the category-agnostic clusters, which reveal the underlying data space structure peculiar to target domain.A clustering branch is capitalized on to ensure that the learnt representation preserves such underlying structure by matching the estimated assignment distribution over clusters to the inherent cluster distribution for each target sample.Furthermore, SE-CC enhances the learnt representation with mutual information maximization.Extensive experiments are conducted on Office and VisDA datasets for both open-set and closed-set domain adaptation, and superior results are reported when comparing to the state-of-the-art approaches.

Presentiamo un nuovo design, cioè Self-Ensembling con Category-agnostic Clusters, per domain adaptation closed-set e open-set.

Un nuovo approccio a domain adaptation open set, dove le categorie del dominio di origine sono contenute nelle categorie del dominio target al fine di filtrare le categorie outlier e consentire l'adattamento all'interno delle classi condivise.


We present Spectral Inference Networks, a framework for learning eigenfunctions of linear operators by stochastic optimization.Spectral Inference Networks generalize Slow Feature Analysis to generic symmetric operators, and are closely related to Variational Monte Carlo methods from computational physics.As such, they can be a powerful tool for unsupervised representation learning from video or graph-structured data.We cast training Spectral Inference Networks as a bilevel optimization problem, which allows for online learning of multiple eigenfunctions.We show results of training Spectral Inference Networks on problems in quantum mechanics and feature learning for videos on synthetic datasets.Our results demonstrate that Spectral Inference Networks accurately recover eigenfunctions of linear operators and can discover interpretable representations from video in a fully unsupervised manner.

Mostriamo come imparare le decomposizioni spettrali degli operatori lineari con il deep learning, e lo usiamo per l'apprendimento unsupervised senza un modello generativo.

Gli autori propongono di utilizzare un framework di deep learning per risolvere il calcolo degli autovettori più grandi.

Questo articolo presenta una framework per imparare le autofunzioni attraverso un processo stocastico e propone di affrontare la sfida del calcolo delle autofunzioni in un contesto su larga scala approssimando e poi usando un processo di ottimizzazione stocastica a due fasi.


The Tensor-Train factorization (TTF) is an efficient way to compress large weight matrices of fully-connected layers and recurrent layers in recurrent neural networks (RNNs).However, high Tensor-Train ranks for all the core tensors of parameters need to be element-wise fixed, which results in an unnecessary redundancy of model parameters.This work applies Riemannian stochastic gradient descent (RSGD) to train core tensors of parameters in the Riemannian Manifold before finding vectors of lower Tensor-Train ranks for parameters.The paper first presents the RSGD algorithm with a convergence analysis and then tests it on more advanced Tensor-Train RNNs such as bi-directional GRU/LSTM and Encoder-Decoder RNNs with a Tensor-Train attention model.The experiments on digit recognition and machine translation tasks suggest the effectiveness of the RSGD algorithm for Tensor-Train RNNs.

Applicazione dell'algoritmo Riemannian SGD (RSGD) per il training di Tensor-Train RNNs per ridurre ulteriormente i parametri del modello.

L'articolo propone di utilizzare l'algoritmo dello stochastic gradient Riemanniano per l'apprendimento di tensori a basso rango nelle deep neural network.

Propone un algoritmo per l'ottimizzazione delle reti neurali parametrizzate dalla decomposizione Tensor Train basata sull'ottimizzazione Riemanniana e sull'adattamento del rango, e progetta un'architettura TT LSTM bidirezionale.


In this paper, we consider the problem of learning control policies that optimize areward function while satisfying constraints due to considerations of safety, fairness, or other costs.We propose a new algorithm - Projection Based ConstrainedPolicy Optimization (PCPO), an iterative method for optimizing policies in a two-step process - the first step performs an unconstrained update while the secondstep reconciles the constraint violation by projection the policy back onto the constraint set.We theoretically analyze PCPO and provide a lower bound on rewardimprovement, as well as an upper bound on constraint violation for each policy update.We further characterize the convergence of PCPO with projection basedon two different metrics - L2 norm and Kullback-Leibler divergence.Our empirical results over several control tasks demonstrate that our algorithm achievessuperior performance, averaging more than 3.5 times less constraint violation andaround 15% higher reward compared to state-of-the-art methods.

Proponiamo un nuovo algoritmo che impara policy che soddisfano i vincoli, e forniamo un'analisi teorica e una dimostrazione empirica nel contesto del reinforcement learning con vincoli.

Questo articolo introduce un algoritmo di ottimizzazione delle policy vincolate che utilizza un processo di ottimizzazione a due fasi, dove le policy che non soddisfano il vincolo possono essere riproiettate nell'insieme dei vincoli.


Deep networks face challenges of ensuring their robustness against inputs that cannot be effectively represented by information learned from training data.We attribute this vulnerability to the limitations inherent to activation-based representation.To complement the learned information from activation-based representation, we propose utilizing a gradient-based representation that explicitly focuses on missing information.In addition, we propose a directional constraint on the gradients as an objective during training to improve the characterization of missing information.To validate the effectiveness of the proposed approach, we compare the anomaly detection performance of gradient-based and activation-based representations.We show that the gradient-based representation outperforms the activation-based representation by 0.093 in CIFAR-10 and 0.361 in CURE-TSR datasets in terms of AUROC averaged over all classes.Also, we propose an anomaly detection algorithm that uses the gradient-based representation, denoted as GradCon, and validate its performance on three benchmarking datasets.The proposed method outperforms the majority of the state-of-the-art algorithms in CIFAR-10, MNIST, and fMNIST datasets with an average AUROC of 0.664, 0.973, and 0.934, respectively.

Proponiamo una rappresentazione basata sul gradiente per caratterizzare le informazioni che le deep neural network non hanno imparato.

Gli autori presentano la creazione di rappresentazioni basate sui gradienti rispetto ai pesi per integrare le informazioni mancanti dal dataset di training per le deep neural network.


Medical images may contain various types of artifacts with different patterns and mixtures, which depend on many factors such as scan setting, machine condition, patientsâ€™ characteristics, surrounding environment, etc.However, existing deep learning based artifact reduction methods are restricted by their training set with specific predetermined artifact type and pattern.As such, they have limited clinical adoption.In this paper, we introduce a â€œZero-Shotâ€ medical image Artifact Reduction (ZSAR) framework, which leverages the power of deep learning but without using general pre-trained networks or any clean image reference.Specifically, we utilize the low internal visual entropy of an image and train a light-weight image-specific artifact reduction network to reduce artifacts in an image at test-time.We use Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) as vehicles to show that ZSAR can reduce artifacts better than state-of-the-art both qualitatively and quantitatively, while using shorter execution time.To the best of our knowledge, this is the first deep learning framework that reduces artifacts in medical images without using a priori training set.

Introduciamo un framework zero-shot per la riduzione degli artefatti nelle immagini mediche, che sfrutta la potenza del deep learning, ma senza utilizzare reti generali pre-trained o qualsiasi riferimento ad un'immagine pulita.


Attribution methods provide insights into the decision-making of machine learning models like artificial neural networks.For a given input sample, they assign a relevance score to each individual input variable, such as the pixels of an image.In this work we adapt the information bottleneck concept for attribution.By adding noise to intermediate feature maps we restrict the flow of information and can quantify (in bits) how much information image regions provide.We compare our method against ten baselines using three different metrics on VGG-16 and ResNet-50, and find that our methods outperform all baselines in five out of six settings.The methodâ€™s information-theoretic foundation provides an absolute frame of reference for attribution values (bits) and a guarantee that regions scored close to zero are not necessary for the network's decision.

Applichiamo il concetto di informational bottleneck all'attribution.

L'articolo propone un nuovo metodo basato sulla perturbazione per il calcolo delle mappe di attribution/saliency per classificatori di immagini basati su deep neural network, iniettando rumore artificiale in un layer iniziale della rete.


Recurrent Neural Networks (RNNs) are used in state-of-the-art models in domains such as speech recognition, machine translation, and language modelling.Sparsity is a technique to reduce compute and memory requirements of deep learning models.Sparse RNNs are easier to deploy on devices and high-end server processors.Even though sparse operations need less compute and memory relative to their dense counterparts, the speed-up observed by using sparse operations is less than expected on different hardware platforms.In order to address this issue, we investigate two different approaches to induce block sparsity in RNNs: pruning blocks of weights in a layer and using group lasso regularization with pruning to create blocks of weights with zeros.Using these techniques, we can create block-sparse RNNs with sparsity ranging from 80% to 90% with a small loss in accuracy.This technique allows us to reduce the model size by roughly 10x.Additionally, we can prune a larger dense network to recover this loss in accuracy while maintaining high block sparsity and reducing the overall parameter count.Our technique works with a variety of block sizes up to 32x32.Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.

Mostriamo che le RNN possono essere pruned per indurre la sparsità dei blocchi che migliora la velocità per le operazioni sparse sull'hardware esistente.

Gli autori propongono un approccio di pruning di sparsità a blocchi per comprimere le RNN, usando il gruppo LASSO per promuovere la sparsità e per potare, ma con una schedule molto specializzata per quanto riguarda il peso di pruning e il pruning stesso.


Value iteration networks are an approximation of the value iteration (VI) algorithm implemented with convolutional neural networks to make VI fully differentiable.In this work, we study these networks in the context of robot motion planning, with a focus on applications to planetary rovers.The key challenging task in learning-based motion planning is to learn a transformation from terrain observations to a suitable navigation reward function.In order to deal with complex terrain observations and policy learning, we propose a value iteration recurrence, referred to as the soft value iteration network (SVIN).SVIN is designed to produce more effective training gradients through the value iteration network.It relies on a soft policy model, where the policy is represented with a probability distribution over all possible actions, rather than a deterministic policy that returns only the best action.We demonstrate the effectiveness of the proposed method in robot motion planning scenarios.In particular, we study the application of SVIN to very challenging problems in planetary rover navigation and present early training results on data gathered by the  Curiosity rover that is currently operating on Mars.

Proponiamo un miglioramento per le value iteration network, con applicazioni alla pianificazione del percorso del rover planetario.

Questo articolo apprende una funzione di reward basata sulle traiettorie degli esperti utilizzando un modulo di iterazione del valore per rendere il planning step differenziabile


Transformer networks have lead to important progress in language modeling and machine translation.These models include two consecutive modules, a feed-forward layer and a self-attention layer.The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers.Building upon this intuition, we propose a new model that solely consists of attention layers.More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer.Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer.Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.

Un nuovo layer di attention che combina la self-attention e i sub-layer feed-forward delle reti Transformer.

Questo articolo propone una modifica al modello Transformer incorporando l'attention sui vettori di memoria "persistenti" nel layer di self-attention, ottenendo prestazioni alla pari con i modelli esistenti e utilizzando meno parametri.


This work views neural networks as data generating systems and applies anomalous pattern detection techniques on that data in order to detect when a network is processing a group of anomalous inputs.  Detecting anomalies is a critical component for multiple machine learning problems including detecting the presence of adversarial noise added to inputs.More broadly, this work is a step towards giving neural networks the ability to detect groups of out-of-distribution samples.  This work introduces ``Subset Scanning methods from the anomalous pattern detection domain to the task of detecting anomalous inputs to neural networks.  Subset Scanning allows us to answer the question: "``Which subset of inputs have larger-than-expected activations at which subset of nodes?"  Framing the adversarial detection problem this way allows us to identify systematic patterns in the activation space that span multiple adversarially noised images.  Such images are ``"weird together".  Leveraging this common anomalous pattern, we show increased detection power as the proportion of noised images increases in a test set.   Detection power and accuracy results are provided for targeted adversarial noise added to CIFAR-10 images on a 20-layer ResNet using the Basic Iterative Method attack.

Troviamo efficacemente un sottoinsieme di immagini che hanno attivazioni più alte del previsto per alcuni sottoinsiemi di nodi. Queste immagini appaiono più anomale e più facili da rilevare se viste come un gruppo.

L'articolo ha proposto uno schema per rilevare la presenza di input anomali basato su un approccio di "subset scanning" per rilevare attivazioni anomale nella rete di deep learning.


Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks.In this work, we conduct a thorough investigation of stable recurrent models.Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent.Empirically, we demonstrate stable recurrent models often perform as well as their unstable counterparts on benchmark sequence tasks.Taken together, these findings shed light on the effective power of recurrent networks and suggest much of sequence learning happens, or can be made to happen, in the stable regime.Moreover, our results help to explain why in many cases practitioners succeed in replacing recurrent models by feed-forward models.

I modelli ricorrenti stabili possono essere approssimati da reti feed-forward e performano come i modelli instabili su task di riferimento.

Studia la stabilità delle RNN e l'indagine della normalizzazione spettrale alle previsioni sequenziali.


Weight-sharing plays a significant role in the success of many deep neural networks, by increasing memory efficiency and incorporating useful inductive priors about the problem into the network.But understanding how weight-sharing can be used effectively in general is a topic that has not been studied extensively.Chen et al. (2015) proposed HashedNets, which augments a multi-layer perceptron with a hash table, as a method for neural network compression.We generalize this method into a framework (ArbNets) that allows for efficient arbitrary weight-sharing, and use it to study the role of weight-sharing in neural networks.We show that common neural networks can be expressed as ArbNets with different hash functions.We also present two novel hash functions, the Dirichlet hash and the Neighborhood hash, and use them to demonstrate experimentally that balanced and deterministic weight-sharing helps with the performance of a neural network.

Si studia il ruolo del weight sharing nelle reti neurali usando le funzioni hash, scoprendo che una funzione hash equilibrata e deterministica aiuta le prestazioni della rete.

Si propone ArbNet per studiare weight sharing in modo più sistematico definendo la funzione di weight sharing come una funzione hash.


We introduce Neural Markov Logic Networks (NMLNs), a statistical relational learning system that borrows ideas from Markov logic.Like Markov Logic Networks (MLNs), NMLNs are an exponential-family model for modelling distributions over possible worlds, but unlike MLNs, they do not rely on explicitly specified first-order logic rules.Instead, NMLNs learn an implicit representation of such rules as a neural network that acts as a potential function on fragments of the relational structure.Interestingly, any MLN can be represented as an NMLN.Similarly to recently proposed Neural theorem provers (NTPs) (Rocktaschel at al. 2017), NMLNs can exploit embeddings of constants but, unlike NTPs, NMLNs work well also in their absence.This is extremely important for predicting in settings other than the transductive one.We showcase the potential of NMLNs on knowledge-base completion tasks and on generation of molecular (graph) data.

Introduciamo un sistema di apprendimento statistico relazionale che prende in prestito idee dalla logica di Markov ma impara una rappresentazione implicita delle regole come una rete neurale.

L'articolo fornisce un'estensione alle reti logiche di Markov rimuovendo la loro dipendenza da regole logiche di primo ordine predefinite per modellare più domini nei task di completamento delle knowledge base.


Using variational Bayes neural networks, we develop an algorithm capable of accumulating knowledge into a prior from multiple different tasks.This results in a rich prior capable of few-shot learning on new tasks.The posterior can go beyond the mean field approximation and yields good uncertainty on the performed experiments.Analysis on toy tasks show that it can learn from significantly different tasks while finding similarities among them.Experiments on Mini-Imagenet reach state of the art with 74.5% accuracy on 5 shot learning.Finally, we provide two new benchmarks, each showing a failure mode of existing meta learning algorithms such as MAML and prototypical Networks.

Un metodo scalabile per l'apprendimento di un prior espressivo sulle reti neurali in task multipli.

L'articolo presenta un metodo per addestrare un modello probabilistico per il Multitask Transfer Learning introducendo una variabile latente per task per catturare la comunanza nelle istanze del task.

Il lavoro propone un approccio variazionale al meta learning che impiega variabili latenti corrispondenti a dataset specifici del task.

Mira ad apprendere un prior sulle reti neurali per task multipli.


Sequential data often originates from diverse environments.Across them exist both shared regularities and environment specifics.To learn robust cross-environment descriptions of sequences we introduce disentangled state space models (DSSM).In the latent space of DSSM environment-invariant state dynamics is explicitly disentangled from environment-specific information governing that dynamics.We empirically show that such separation enables robust prediction, sequence manipulation and environment characterization.We also propose an unsupervised VAE-based training procedure to learn DSSM as Bayesian filters.In our experiments, we demonstrate state-of-the-art performance in controlled generation and prediction of bouncing ball video sequences across varying gravitational influences.

L'articolo presenta un modello generativo dello spazio di stato che utilizza una variabile latente globale E per catturare le informazioni specifiche dell'ambiente.


In this work, we approach one-shot and few-shot learning problems as methods for finding good prototypes for each class, where these prototypes are generalizable to new data samples and classes.We propose a metric learner that learns a Bregman divergence by learning its underlying convex function.Bregman divergences are a good candidate for this framework given they are the only class of divergences with the property that the best representative of a set of points is given by its mean.We propose a flexible extension to prototypical networks to enable joint learning of the embedding and the divergence, while preserving computational efficiency.Our preliminary results are comparable with the prior work on the Omniglot and Mini-ImageNet datasets, two standard benchmarks for one-shot and few-shot learning.We argue that our model can be used for other tasks that involve metric learning or tasks that require approximate convexity such as structured prediction and data completion.

Apprendimento della divergenza di Bregman per few-shot learning.


Motivated by the flexibility of biological neural networks whose connectivity structure changes significantly during their lifetime,we introduce the Unrestricted Recursive Network (URN) and demonstrate that it can exhibit similar flexibility during training via gradient descent.We show empirically that many of the different neural network structures commonly used in practice today (including fully connected, locally connected and residual networks of differ-ent depths and widths) can emerge dynamically from the same URN.These different structures can be derived using gradient descent on a single general loss function where the structure of the data and the relative strengths of various regulator terms determine the structure of the emergent network.We show that this loss function and the regulators arise naturally when considering the symmetries of the network as well as the geometric properties of the input data.

Introduciamo un framework di rete che può modificare la sua struttura durante il training e mostriamo che può convergere a vari archetipi di rete ML come MLP e LCN.


We present CROSSGRAD , a method to use multi-domain training data to learn a classifier that generalizes to new domains.CROSSGRAD does not need an adaptation phase via labeled or unlabeled data, or domain features in the new domain.Most existing domain adaptation methods attempt to erase domain signals using techniques like domain adversarial training.In contrast, CROSSGRAD is free to use domain signals for predicting labels, if it can prevent overfitting on training domains.We conceptualize the task in a Bayesian setting, in which a sampling step is implemented as data augmentation, based on domain-guided perturbations of input instances.CROSSGRAD jointly trains a label and a domain classifier on examples perturbed by loss gradients of each otherâ€™s objectives.This enables us to directly perturb inputs, without separating and re-mixing domain signals while making various distributional assumptions.Empirical evaluation on three different applications where this setting is natural establishes that (1) domain-guided perturbation provides consistently better generalization to unseen domains, compared to generic instance perturbation methods, and (2) data augmentation is a more stable and accurate method than domain adversarial training.

Data augmentation guidata dal dominio fornisce un metodo robusto e stabile di generalizzazione del dominio

Questo articolo propone un approccio di generalizzazione del dominio attraverso data augmentation dipendente dal dominio

Gli autori introducono il metodo CrossGrad, che allena sia un task di classificazione delle label che un task di classificazione del dominio.


We present sketch-rnn, a recurrent neural network able to construct stroke-based drawings of common objects.The model is trained on a dataset of human-drawn images representing many different classes.We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.

Studiamo un'alternativa ai tradizionali approcci di modellazione delle immagini tramite pixel, e proponiamo un modello generativo per le immagini vettoriali.

Questo articolo introduce un'architettura di rete neurale per la generazione di disegni ispirata all'autoencoder variazionale.


Wilson et al. (2017) showed that, when the stepsize schedule is properly designed, stochastic gradient generalizes better than ADAM (Kingma & Ba, 2014).In light of recent work on hypergradient methods (Baydin et al., 2018), we revisit these claims to see if such methods close the gap between the most popular optimizers.As a byproduct, we analyze the true benefit of these hypergradient methods compared to more classical schedules, such as the fixed decay of Wilson et al. (2017).In particular, we observe they are of marginal help since their performance varies significantly when tuning their hyperparameters.Finally, as robustness is a critical quality of an optimizer, we provide a sensitivity analysis of these gradient based optimizers to assess how challenging their tuning is.

Forniamo uno studio che cerca di vedere come il recente adattamento del learning rate online estende la conclusione fatta da Wilson et al. 2018 sui metodi con gradiente adattivo, insieme al confronto e all'analisi della sensibilità.

Riporta i risultati dei test di diversi metodi relativi alla regolazione dello step size, tra cui SGD standard, SGD con momento Nesterov e ADAM e confronta questi metodi con ipergradiente e senza.


Despite an ever growing literature on reinforcement learning algorithms and applications, much less is known about their statistical inference.In this paper, we investigate the large-sample behaviors of the Q-value estimates with closed-form characterizations of the asymptotic variances.This allows us to efficiently construct confidence regions for Q-value and optimal value functions, and to develop policies to minimize their estimation errors.This also leads to a policy exploration strategy that relies on estimating the relative discrepancies among the Q estimates.Numerical experiments show superior performances of our exploration strategy than other benchmark approaches.

Indaghiamo il comportamento su grandi sample delle stime dei valori Q e abbiamo proposto una strategia di esplorazione efficiente che si basa sulla stima delle discrepanze relative tra le stime dei valori Q.

Questo articolo presenta un algoritmo di pura esplorazione per il reinforcement learning basato su un'analisi asintotica dei valori Q e la loro convergenza alla distribuzione limite centrale, superando gli algoritmi di esplorazione di riferimento.


We perform completely unsupervised one-sided image to image translation between a source domain $X$ and a target domain $Y$ such that we preserve relevant underlying shared semantics (e.g., class, size, shape, etc). In particular, we are interested in a more difficult case than those typically addressed in the literature, where the source and target are ``far" enough that reconstruction-style or pixel-wise approaches fail.We argue that transferring (i.e., \emph{translating}) said relevant information should involve both discarding source domain-specific information while incorporate target domain-specific information, the latter of which we model with a noisy prior distribution. In order to avoid the degenerate case where the generated samples are only explained by the prior distribution, we propose to minimize an estimate of the mutual information between the generated sample and the sample from the prior distribution.We discover that the architectural choices are an important factor to consider in order to preserve the shared semantic between $X$ and $Y$. We show state of the art results on the MNIST to SVHN task for unsupervised image to image translation.

Addestriamo una rete di traduzione da immagine a immagine che prende come input l'immagine sorgente e un sample da una distribuzione prior per generare un sample dalla distribuzione target

Questo articolo formalizza il problema della traduzione non supervisionata e propone un framework GAN aumentato che utilizza l'informazione mutua per evitare il caso degenere

Questo articolo formula il problema della traduzione non supervisionata di immagini one-to-many e affronta il problema minimizzando l'informazione mututa.


Identifying salient points in images is a crucial component for visual odometry, Structure-from-Motion or SLAM algorithms.Recently, several learned keypoint methods have demonstrated compelling performance on challenging benchmarks.  However, generating consistent and accurate training data for interest-point detection in natural images still remains challenging, especially for human annotators.We introduce IO-Net (i.e. InlierOutlierNet), a novel proxy task for the self-supervision of keypoint detection, description and matching.By making the sampling of inlier-outlier sets from point-pair correspondences fully differentiable within the keypoint learning framework, we show that are able to simultaneously self-supervise keypoint description and improve keypoint matching.Second, we introduce KeyPointNet, a keypoint-network architecture that is especially amenable to robust keypoint detection and description.We design the network to allow local keypoint aggregation to avoid artifacts due to spatial discretizations commonly used for this task, and we improve fine-grained keypoint descriptor performance by taking advantage of efficient sub-pixel convolutions to upsample the descriptor feature-maps to a higher operating resolution.Through extensive experiments and ablative analysis, we show that the proposed self-supervised keypoint learning method greatly improves the quality of feature matching and homography estimation on challenging benchmarks over the state-of-the-art.

Si impara ad estrarre key point distinguibili da una proxy task, con rifiuto di outlier.

Questo articolo è dedicato al self-supervised learning di feature locali utilizzando Neural Guided RANSAC come un ulteriore fornitore di loss ausiliario per migliorare l'interpolazione dei descrittori.


We study the role of intrinsic motivation as an exploration bias for reinforcement learning in sparse-reward synergistic tasks, which are tasks where multiple agents must work together to achieve a goal they could not individually.Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own.Thus, we propose to incentivize agents to take (joint) actions whose effects cannot be predicted via a composition of the predicted effect for each individual agent.We study two instantiations of this idea, one based on the true states encountered, and another based on a dynamics model trained concurrently with the policy.While the former is simpler, the latter has the benefit of being analytically differentiable with respect to the action taken.We validate our approach in robotic bimanual manipulation tasks with sparse rewards; we find that our approach yields more efficient learning than both1) training with only the sparse reward and2) using the typical surprise-based formulation of intrinsic motivation, which does not bias toward synergistic behavior.Videos are available on the project webpage: https://sites.google.com/view/iclr2020-synergistic.

Proponiamo una formulazione di motivazione intrinseca che è adatta come bias di esplorazione in task sinergici a più agenti con reward sparse, incoraggiando gli agenti a influenzare l'ambiente in modi che non sarebbero stati possibili se avessero agito individualmente.

L'articolo si concentra sull'uso della motivazione intrinseca per migliorare il processo di esplorazione degli agenti di reinforcement learning in task che richiedono la presenza di più agenti.


A general graph-structured neural network architecture operates on graphs through two core components: (1) complex enough message functions; (2) a fixed information aggregation process.In this paper, we present the Policy Message Passing algorithm, which takes a probabilistic perspective and reformulates the whole information aggregation as stochastic sequential processes.The algorithm works on a much larger search space, utilizes reasoning history to perform inference, and is robust to noisy edges.We apply our algorithm to multiple complex graph reasoning and prediction tasks and show that our algorithm consistently outperforms state-of-the-art graph-structured models by a significant margin.

Un algoritmo di inferenza probabilistica guidato da una rete neurale per modelli strutturati come un grafo

Questo articolo introduce il policy message passing, una graph neural network con un meccanismo di inferenza che assegna messaggi agli archi in modo ricorrente, mostrando prestazioni competitive su task di ragionamento visivo.


Deep multitask networks, in which one neural network produces multiple predictive outputs, are more scalable and often better regularized than their single-task counterparts.Such advantages can potentially lead to gains in both speed and performance, but multitask networks are also difficult to train without finding the right balance between tasks.We present a novel gradient normalization (GradNorm) technique which automatically balances the multitask loss function by directly tuning the gradients to equalize task training rates.We show that for various network architectures, for both regression and classification tasks, and on both synthetic and real datasets, GradNorm improves accuracy and reduces overfitting over single networks, static baselines, and other adaptive multitask loss balancing techniques.GradNorm also matches or surpasses the performance of exhaustive grid search methods, despite only involving a single asymmetry hyperparameter $\alpha$.Thus, what was once a tedious search process which incurred exponentially more compute for each task added can now be accomplished within a few training runs, irrespective of the number of tasks.Ultimately, we hope to demonstrate that gradient manipulation affords us great control over the training dynamics of multitask networks and may be one of the keys to unlocking the potential of multitask learning.

Mostriamo come è possibile aumentare le prestazioni in una rete multitask facendo tuning di una funzione di loss adattiva multitask che viene appresa attraverso il bilanciamento diretto dei gradienti della rete.

Questo lavoro propone uno schema di update dinamico dei pesi che aggiorna i pesi per le diverse loss dei task durante il training, facendo uso dei rapporti di loss dei diversi task.


Image segmentation aims at grouping pixels that belong to the same object or region.At the heart of image segmentation lies the problem of determining whether a pixel is inside or outside a region, which we denote as the "insideness" problem.Many Deep Neural Networks (DNNs) variants excel in segmentation benchmarks, but regarding insideness, they have not been well visualized or understood: What representations do DNNs use to address the long-range relationships of insideness?How do architectural choices affect the learning of these representations?In this paper, we take the reductionist approach by analyzing DNNs solving the insideness problem in isolation, i.e. determining the inside of closed (Jordan) curves.We demonstrate analytically that state-of-the-art feed-forward and recurrent architectures can implement solutions of the insideness problem for any given curve.Yet, only recurrent networks could  learn these general solutions when the training enforced a specific "routine" capable of breaking down the long-range relationships.Our results highlights the need for new training strategies that decompose the learning into appropriate stages, and that lead to the general class of solutions necessary for DNNs to understand insideness.

Le DNN per la segmentazione delle immagini possono implementare soluzioni per il problema dell'insidie, ma solo alcune reti ricorrenti potrebbero impararle con un tipo specifico di supervisione.

Questo articolo introduce l'insideness per studiare la semantic segmentation nell'era del deep learning, e i risultati possono aiutare i modelli a generalizzare meglio.


We address the challenging problem of deep representation learning--the efficient adaption of a pre-trained deep network to different tasks.Specifically, we propose to explore gradient-based features.These features are gradients of the model parameters with respect to a task-specific loss given an input sample.Our key innovation is the design of a linear model that incorporates both gradient features and the activation of the network.We show that our model provides a local linear approximation to a underlying deep model, and discuss important theoretical insight.Moreover, we present an efficient algorithm for the training and inference of our model without computing the actual gradients.Our method is evaluated across a number of representation learning tasks on several datasets and using different network architectures.We demonstrate strong results in all settings.And our results are well-aligned with our theoretical insight.

Dato un modello pre-trained, abbiamo esplorato i gradienti per sample dei parametri del modello rispetto a una loss specifica del task, e costruito un modello lineare che combina i gradienti dei parametri del modello e l'attivazione del modello.

Questo articolo propone di utilizzare i gradienti di layer specifici di reti convoluzionali come feature in un modello linearizzato per il transfer learning e fast adaptation.


Recovering 3D geometry shape, albedo and lighting from a single image has wide applications in many areas, which is also a typical ill-posed problem.In order to eliminate the ambiguity, face prior knowledge like linear 3D morphable models (3DMM) learned from limited scan data are often adopted to the reconstruction process.However, methods based on linear parametric models cannot generalize well for facial images in the wild with various ages, ethnicity, expressions, poses, and lightings.Recent methods aim to learn a nonlinear parametric model using convolutional neural networks (CNN) to regress the face shape and texture directly.However, the models were only trained on a dataset that is generated from a linear 3DMM.Moreover, the identity and expression representations are entangled in these models, which hurdles many facial editing applications.In this paper, we train our model with adversarial loss in a semi-supervised manner on hybrid batches of unlabeled and labeled face images to exploit the value of large amounts of unlabeled face images from unconstrained photo collections.A novel center loss is introduced to make sure that different facial images from the same person have the same identity shape and albedo.Besides, our proposed model disentangles identity, expression, pose, and lighting representations, which improves the overall reconstruction performance and facilitates facial editing applications, e.g., expression transfer.Comprehensive experiments demonstrate that our model produces high-quality reconstruction compared to state-of-the-art methods and is robust to various expression, pose, and lighting conditions.

Addestriamo il nostro modello di ricostruzione facciale con loss adversarial in modo semi-supervised su batch ibride composte da immagini facciali non annotate e annotate per sfruttare il valore di grandi quantità di immagini facciali non annotate da collezioni fotografiche non vincolate.

Questo articolo propone un processo di training semi-supervised e adversarial per ottenere rappresentazioni disentangled non lineari da un'immagine del viso con funzioni di loss, raggiungendo prestazioni allo stato dell'arte nella ricostruzione del viso.


Human conversations naturally evolve around related entities and connected concepts, while may also shift from topic to topic.This paper presents ConceptFlow, which leverages commonsense knowledge graphs to explicitly model such conversation flows for better conversation response generation.ConceptFlow grounds the conversation inputs to the latent concept space and represents the potential conversation flow as a concept flow along the commonsense relations.The concept is guided by a graph attention mechanism that models the possibility of the conversation evolving towards different concepts.The conversation response is then decoded using the encodings of both utterance texts and concept flows, integrating the learned conversation structure in the concept space.Our experiments on Reddit conversations demonstrate the advantage of ConceptFlow over previous commonsense aware dialog models and fine-tuned GPT-2 models, while using much fewer parameters but with explicit modeling of conversation structures.

Questo articolo presenta ConceptFlow che modella esplicitamente il flusso di conversazione nel commonsense knowledge graph per una migliore generazione di conversazioni.

L'articolo propone un sistema per generare una risposta single-turn ad un enunciato in un ambiente di dialogo open-domain usando la diffiusione nei vicini dei grounded concept.


Biological neural networks face homeostatic and resource constraints that restrict the allowed configurations of connection weights.If a constraint is tight it defines a very small solution space, and the size of these constraint spaces determines their potential overlap with the solutions for computational tasks.We study the geometry of the solution spaces for constraints on neurons' total synaptic weight and on individual synaptic weights, characterizing the connection degrees (numbers of partners) that maximize the size of these solution spaces.We then hypothesize that the size of constraints' solution spaces could serve as a cost function governing neural circuit development.We develop analytical approximations and bounds for the model evidence of the maximum entropy degree distributions under these cost functions.We test these on a published electron microscopic connectome of an associative learning center in the fly brain, finding evidence for a developmental progression in circuit structure.

Esaminiamo l'ipotesi che l'entropia degli spazi di soluzione per i vincoli sui pesi sinaptici (la "flessibilità" del vincolo) potrebbe servire come funzione di costo per lo sviluppo dei circuiti neurali.


In  this  preliminary  work,  we  study  the  generalization  properties  of  infinite  ensembles  of infinitely-wide neural networks.  Amazingly, this model family admits tractable calculations for many information-theoretic quantities.  We report analytical and empirical investigations in the search for signals that correlate with generalization.

Gli insiemi infiniti di reti neurali infinitamente larghe sono una famiglia di modelli interessante dal punto di vista della teoria dell'informazione.


Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks.There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns unified multilingual representations using monolingual and cross-lingual objectives jointly.In this paper, we first conduct direct comparisons of representations learned using both of these methods across diverse cross-lingual tasks.Our empirical results reveal a set of pros and cons for both methods, and show that the relative performance of alignment versus joint training is task-dependent.Stemming from this analysis, we propose a simple and novel framework that combines these two previously mutually-exclusive approaches.Extensive experiments on various tasks demonstrate that our proposed framework alleviates limitations of both approaches, and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark.We further show that our proposed framework can generalize to contextualized representations and achieves state-of-the-art results on the CoNLL cross-lingual NER benchmark.

Conduciamo uno studio comparativo del cross-lingual alignment rispetto al joint training e uniamo questi due paradigmi precedentemente mantenuti separati in un nuovo framework.

Questo articolo confronta gli approcci al bilingual lexicon induction e mostra quale metodo si comporta meglio nel lessico, nell'induzione e nei task di NER e MT.


Large number of weights in deep neural networks make the models difficult to be deployed in low memory environments such as, mobile phones, IOT edge devices as well as "inferencing as a service" environments on the cloud. Prior work has considered reduction in the size of the models, through compression techniques like weight pruning, filter pruning, etc. or through low-rank decomposition of the convolution layers.In this paper, we demonstrate the use of multiple techniques to achieve not only higher model compression but also reduce the compute resources required during inferencing.We do filter pruning followed by low-rank decomposition using Tucker decomposition for model compression.We show that our approach achieves upto 57\% higher model compression when compared to either Tucker Decomposition or Filter pruning alone  at similar accuracy for GoogleNet.Also, it reduces the Flops by upto 48\% thereby making the inferencing faster.

Combinazione di tecniche di compressione del modello ortogonale per ottenere una riduzione significativa della dimensione del modello e del numero di flop richiesti durante l'inferenza.

Questo articolo propone di combinare la decomposizione di Tucker con il pruning dei filtri.


We review the limitations of BLEU and ROUGE -- the most popular metrics used to assess reference summaries against hypothesis summaries, and introduce JAUNE:  a set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries against hypothesis summaries.

Introduce JAUNE: una metodologia per sostituire il punteggio BLEU e ROUGE con valutatori multidimensionali e basati su modelli per la valutazione dei riassunti

Questo articolo propone una nuova metrica JAUNE per la valutazione di machine translation e dei sistemi di summarization, mostrando che il loro modello corrisponde meglio alla ground truth rispetto a BLEU.


This paper presents a new Graph Neural Network (GNN) type using feature-wise linear modulation (FiLM).Many standard GNN variants propagate information along the edges of a graph by computing ``messages'' based only on the representation of the source of each edge.In GNN-FiLM, the representation of the target node of an edge is additionally used to compute a transformation that can be applied to all incoming messages, allowing feature-wise modulation of the passed information.Results of experiments comparing different GNN architectures on three tasks from the literature are presented, based on re-implementations of baseline methods.Hyperparameters for all methods were found using extensive search, yielding somewhat surprising results: differences between baseline models are smaller than reported in the literature.Nonetheless, GNN-FiLM outperforms baseline methods on a regression task on molecular graphs and performs competitively on other tasks.

nuovo formalismo GNN + esperimenti estesi; dimostrando che le differenze tra GGNN/GCN/GAT sono minori di quanto si era pensato

L'articolo propone una nuova architettura Graph Neural Network che utilizza la Feature-wise Linear Modulation per condizionare il message passing tra i nodi di origine e di destinazione in base alla rappresentazione dei nodi di destinazione.


To deal simultaneously with both, the attributed network embedding and clustering, we propose a new model.It exploits both content and structure information, capitalising on their simultaneous use.The proposed model relies on the approximation of the relaxed continuous embedding solution by the true discrete clustering one.Thereby, we show that incorporating an embedding representation provides simpler and more interpretable solutions.Experiment results demonstrate that the proposed algorithm performs better, in terms of clustering and embedding, than the state-of-art algorithms, including deep learning methods devoted to similar tasks for attributed network datasets with different proprieties.

Questo articolo propone un nuovo framework di decomposizione della matrice per l'embedding simultaneo dei dati di rete e il clustering.

Questo articolo propone un algoritmo per eseguire insieme l'embedding della rete di attributi e il clustering.


We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis.The goal of our method is to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications (e.g., virtual showrooms, virtual tours and sightseeing, the digital inspection of historical artifacts).A core component of our work is the handling of view-dependent effects.Specifically, we directly train an object-specific deep neural network to synthesize the view-dependent appearance of an object.As input data we are using an RGB video of the object.This video is used to reconstruct a proxy geometry of the object via multi-view stereo.Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts.To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects.Based on these estimations, we are able to convert observed images to diffuse images.These diffuse images can be projected into other views.In the target view, our pipeline reinserts the new view-dependent effects.To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realistic results.Using this image-guided approach, the network does not have to allocate capacity on ``remembering'' object appearance, instead it learns how to combine the appearance of captured images.We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data.

Proponiamo una tecnica di rendering appresa guidata dalle immagini che combina i vantaggi del rendering basato sulle immagini e della sintesi delle immagini basata su GAN, considerando gli effetti dipendenti dalla vista.

Questa presentazione propone un metodo per gestire gli effetti dipendenti dalla vista nel rendering neurale, che migliora la robustezza dei metodi di rendering neurale esistenti.


We evaluate the distribution learning capabilities of generative adversarial networks by testing them on synthetic datasets.The datasets include common distributions of points in $R^n$ space and images containing polygons of various shapes and sizes.We find that by and large GANs fail to faithfully recreate point datasets which contain discontinous support or sharp bends with noise.Additionally, on image datasets, we find that GANs do not seem to learn to count the number of objects of the same kind in an image.We also highlight the apparent tension between generalization and learning in GANs.

Le GAN sono valutate su dataset sintetici


This paper proposes a new approach for step size adaptation in gradient methods.The proposed method called step size optimization (SSO) formulates the step size adaptation as an optimization problem which minimizes the loss function with respect to the step size for the given model parameters and gradients.Then, the step size is optimized based on alternating direction method of multipliers (ADMM).SSO does not require the second-order information or any probabilistic models for adapting the step size, so it is efficient and easy to implement.Furthermore, we also introduce stochastic SSO for stochastic learning environments.In the experiments, we integrated SSO to vanilla SGD and Adam, and they outperformed state-of-the-art adaptive gradient methods including RMSProp, Adam, L4-Adam, and AdaBound on extensive benchmark datasets.

Proponiamo un metodo di adattamento della step size efficiente ed efficace per i metodi basati sul gradiente.

Un nuovo adattamento della step size nei metodi basati sul gradiente del primo ordine che stabilisce un nuovo problema di ottimizzazione con l'espansione del primo ordine della funzione di loss e la regolarizzazione, dove la step size è trattata come variabile.


Despite the fact that generative models are extremely successful in practice, the theory underlying this phenomenon is only starting to catch up with practice.In this work we address the question of the universality of generative models: is it true that neural networks can approximate any data manifold arbitrarily well?We provide a positive answer to this question and show that under mild assumptions on the activation function one can always find a feedforward neural network that maps the latent space onto a set located within the specified Hausdorff distance from the desired data manifold.We also prove similar theorems for the case of multiclass generative models and cycle generative models, trained to map samples from one manifold to another and vice versa.

Abbiamo dimostrato che un'ampia classe di manifold può essere generata da reti ReLU e sigmoidi con precisione arbitraria.

Questo articolo fornisce alcune garanzie di base su quando i manifold possono essere scritti come l'immagine di una funzione approssimata da una rete neurale, e unisce insieme teoremi dalla geometria dei manifold e risultati standard di approssimazione universale.

Questo articolo mostra teoricamente che i modelli generativi basati sulle reti neurali possono approssimare i data manifold, e dimostra che, sotto blande ipotesi, le reti neurali possono mappare uno spazio latente su un insieme vicino al data manifold entro una piccola distanza di Hausdorff.