This result is higher than the neural model of the same architecture that was reported by #REFR (61.3%) but 0.4% lower than their best performing model (an HMM based).

We note that the development set is silver data (automatically annotated) and is likely to contain mistakes. We run the model and achieve 73.2% accuracy.

Next, to test how useful our data is to a real-world gold dataset we wish to inspect its usefulness with a transferlearning approach.

By first pre-training a simple model on our data and then fine-tune the model on PIG.

If our data is indeed of quality we'd expect to see performance improvements on PIG, especially as this data is relatively small.

We begin by training the LSTM model solely on PIG, 7 which results in 64.1% accuracy.

We continue by using the model trained on APFD and then fine-tune it on PIG, which leads to 66.8% accuracy, an improvement of 2.7 points over the previous SOTA.

We attribute this gain in performance to our dataset, which both increases the number of training examples and allows to train bigger neural models which excel with more training examples.

We also experiment in the opposite direction and fine-tune the model trained on PIG with our data, which result in 73.6% accuracy, which is better than training on our data alone, achieving 73.2% accuracy.

At Your Fingertips: Extracting Piano Fingering Instructions From Videos

Abstract: Piano fingering-knowing which finger to use to play each note in a musical piece, is a hard and important skill to master when learning to play the piano. While some sheet music is available with expert-annotated fingering information, most pieces lack this information, and people often resort to learning the fingering from demonstrations in online videos. We consider the AI task of automating the extraction of fingering information from videos. This is a non-trivial task as fingers are often occluded by other fingers, and it is often not clear from the video which of the keys were pressed, requiring the synchronization of hand position information and knowledge about the notes that were played. We show how to perform this task with high-accuracy using a combination of deeplearning modules, including a GAN-based approach for fine-tuning on out-of-domain data. We extract the fingering information with an f1 score of 97%. We run the resulting system on 90 videos, resulting in high-quality piano fingering information of 150K notes, the largest available dataset of piano-fingering to date.

Statistical Learning And Estimation Of Piano Fingering

Abstract: Automatic estimation of piano fingering is important for understanding the computational process of music performance and applicable to performance assistance and education systems. While a natural way to formulate the quality of fingerings is to construct models of the constraints/costs of performance, it is generally difficult to find appropriate parameter values for these models. Here we study an alternative data-driven approach based on statistical modeling in which the appropriateness of a given fingering is described by probabilities. Specifically, we construct two types of hidden Markov models (HMMs) and their higher-order extensions. We also study deep neural network (DNN)-based methods for comparison. Using a newly released dataset of fingering annotations, we conduct systematic evaluations of these models as well as a representative constraint-based method. We find that the methods based on high-order HMMs outperform the other methods in terms of estimation accuracies. We also quantitatively study individual difference of fingering and propose evaluation measures that can be used with multiple ground truth data. We conclude that the HMM-based methods are currently state of the art and generate acceptable fingerings in most parts and that they have certain limitations such as ignorance of phrase boundaries and interdependence of the two hands.


To improve generalization, SEMI #REFR exploits both synthetic and real-world rainy images to conduct supervised and unsupervised training respectively. In addition, rain presents notable diversity (e.g.

Recently, deep learning has been overwhelmingly successful in image restoration #OTHEREFR - #OTHEREFR , which also includes image deraining #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR - #OTHEREFR .

DerainNet #OTHEREFR and JORDER #OTHEREFR are two of the earliest convolution-based methods for deep single image deraining.

density, size, distribution, etc.), thus prior learning may be helpful for accurate rain removal.

Among these methods, DIDMDN #OTHEREFR guides the network to restore degraded image by estimating rain density.

Furthermore, UMRL #OTHEREFR and MSPFN #OTHEREFR utilize uncertainty and multi-scale information to obtain derained images.

Subsequently, the squeeze-excitation mechanism and progressive recursive learning are introduced in RESCAN #OTHEREFR , PReNet #OTHEREFR and DPENet #OTHEREFR to design networks.

MSPFN #OTHEREFR introduces multi-scale information to adapt to the distribution diversity of rain streaks.

Towards An Effective And Efficient Transformer For Rain-By-Snow Weather Removal

Abstract: Rain-by-snow weather removal is a specialized task in weather-degraded image restoration aiming to eliminate coexisting rain streaks and snow particles. In this paper, we propose RSFormer, an efficient and effective Transformer that addresses this challenge. Initially, we explore the proximity of convolution networks (ConvNets) and vision Transformers (ViTs) in hierarchical architectures and experimentally find they perform approximately at intra-stage feature learning. On this basis, we utilize a Transformer-like convolution block (TCB) that replaces the computationally expensive self-attention while preserving attention characteristics for adapting to input content. We also demonstrate that cross-stage progression is critical for performance improvement, and propose a global-local self-attention sampling mechanism (GLASM) that down-/up-samples features while capturing both global and local dependencies. Finally, we synthesize two novel rain-by-snow datasets, RSCityScape and RS100K, to evaluate our proposed RSFormer. Extensive experiments verify that RSFormer achieves the best trade-off between performance and time-consumption compared to other restoration methods. For instance, it outperforms Restormer with a 1.53% reduction in the number of parameters and a 15.6% reduction in inference time. Datasets, source code and pre-trained models are available at https://github.com/chdwyb/RSFormer.

Semi-Supervised Transfer Learning For Image Rain Removal

Abstract: Single image rain removal is a typical inverse problem in computer vision. The deep learning technique has been verified to be effective for this task and achieved state-of-the-art performance. However, previous deep learning methods need to pre-collect a large set of image pairs with/without synthesized rain for training, which tends to make the neural network be biased toward learning the specific patterns of the synthesized rain, while be less able to generalize to real test samples whose rain types differ from those in the training data. To this issue, this paper firstly proposes a semi-supervised learning paradigm toward this task. Different from traditional deep learning methods which only use supervised image pairs with/without synthesized rains, we further put real rainy images, without need of their clean ones, into the network training process. This is realized by elaborately formulating the residual between an input rainy image and its expected network output (clear image without rain) as a concise mixture of Gaussians distribution. The network is therefore trained to transfer to adapting the real rain pattern domain instead of only the synthesis rain domain, and thus both the short-of-training-sample and bias-to-supervised-sample issues can be evidently alleviated. Experiments on synthetic and real data verify the superiority of our model compared to the state-of-the-arts.


It is commonly believed for many machine learning researchers that the key to effectively construct a model is properly optimized feature extraction #REFR .

The extracted features should contain relevant information from the initial dataset so that this simplified representation can be used as the alteration of the full original dataset to perform the required task shown as figure 3.4 #OTHEREFR .

The main task involved by feature extraction is to reduce the resource which are necessary to describe large amounts of data.

When analyzing or learning with massive complex data, which usually requires a lot of memory and computational power, one of the main problems are caused by the number of variables involved.

Besides, with the original dataset, a classification algorithm or model is inclined to overfit training samples and may be generalized poorly afterwards.

In general, feature extraction is a generic term for methods combining features or variables to solve these problems with performing sufficient accuracy comparing with the initial dataset.

In the experiments, except for the classic dimensionality reduction technique Principal Component Analysis (PCA), several feature extraction methods are used such as permutation importance,feature importance and hierarchical clustering based on Spearman correlations.

Predictability Of Machine Learning Algorithms And Related Feature Extraction Techniques Master Thesis

Abstract: To implement machine learning, it is essential to first determine an appropriate algorithm for the dataset. Different algorithms may produce a large number of different models with different hyperparameter configurations, and it usually takes a lot of time to run the model on a large dataset when the model is relatively complex. Therefore, how to predict the performance of a model on a dataset is an fundamental problem to be solved. This thesis designs a prediction system based on matrix factorization to predict the classification accuracy of a specific model on a particular dataset. In this thesis, we conduct a comprehensive empirical research on more than fifty datasets that we collected from the openml web site. We study the performance prediciton of three fundamental machine learning algorithms, namely, random forest, XGBoost, and MultiLayer Perceptron(MLP). In particular, we obtain the following results: • Predictability of fine-tuned models using coarse-tuned variants: Usually, training and testing complex machine learning models are time-consuming. Thus, we hope to predict the complicate models by their simple ones. Three machine learning algorithms are compared in experiments. We find that random forest and XGBoost have good predictability on most datasets that is, as the model becomes more complex, the performance of the model becomes better, and thus the accuracy of the complex model can be foreseen directly from its simple model. Hence, we can decide efficiently which algorithm to utilize by comparing simple models. • Predictability of MLP using feature extraction techniques: Often, real datasets have quite numerous features, from a few hundred to a few thousand features. Training models on fully-featured datasets is a very time-consuming task. We explore the idea of training a model D on datasets that are projected on a few features and use this as a hint to predict the performance of the model D when we consider all features of the dataset. We try different feature extraction techniques including techniques based on permutation importance, gain-based feature importance, hierarchical clustering based on Spearman correlation, and principal component analysis. We study the performance of techniques on the multilayer perceptron (MLP) model and observe that feature extraction with permutation importance and hierarchical clustering based on Spearman correlation has better performance. That is, on most datasets, the accuracy of the MLP improves as the number of features extracted by these techniques increases. • Predict model performance using implicit feedback: After researching the predictability of three algorithms, our goal is to discover a method that can be used to predict the specific model performance on a particular dataset. In order to predict the classification accuracy of different algorithms on different datasets with different hyperparameters, a prediction system with matrix factorization is built to predict the performance of different models on different datasets. With this system, the input accuracy's can be seen as implicit feedback because there is no more information about the cause of these performance. This system can best achieve an mean absolute error of only 6.7% in the experiment. Predictability of Machine Learning Algorithms and Related Feature Extraction Techniques iii Acknowledgement I would like to thank my supervisor Morteza for his excellent guidance and support during this process. Thanks to my friends and families as well. You kept me motivated. And My parents deserve a particular note of appreciation for always encouraging me. Last but not the least, I am grateful to professor Meng Fang and Wouter Meulemans to be the committee members for me. I hope you enjoy your reading.

Theano: New Features And Speed Improvements

Abstract: Theano is a linear algebra compiler that optimizes a user's symbolically-specified mathematical computations to produce efficient low-level implementations. In this paper, we present new features and efficiency improvements to Theano, and benchmarks demonstrating Theano's performance relative to Torch7, a recently introduced machine learning library, and to RNNLM, a C++ library targeted at recurrent neural networks.


Comparison To Other Clustering Results For Bandit Problems

Gentile, Li, and Zappella #REFR consider a more structured setting, where users can be partitioned into m unknown clusters and the context vectors C t in each round t are generated i.i.d. (where the size can be arbitrary).

The graph provides structures to the parameters u i , i.e., they assume that (i,j)∈E u i − u j 2 are small compared to i∈V u i 2 . The learning proceeds in rounds.

For each round t, the user index i t and a set of arbitrary context vectors C it = {x t,1 , . . .

, x t,ct } is presented and the learner has to pick one actionx t ∈ C it and receives a reward of u T ix t with an additional sub-Gaussian noise.

Cesa-Bianchi et al #OTHEREFR maintain a set of n linear bandit algorithms and an inverse correlation matrix M t for feedback sharing between bandit algorithms.

They obtain a regret bound that depends on √ nT and log determinant of the matrix M t , which can be O(n).

We note that this setting is closely related to our work where m = f + 1.

Gentile et al #OTHEREFR give a regret bound that depends on √ mT with additional O(n + m) terms that are constant with T .

A recent result by Gentile et al #OTHEREFR considers various data-dependent assumptions to obtain sharper bounds that depend on √ T m and n • polylog(nT ).

Another line of work, by Maillard and Munos #OTHEREFR and Hong et al #OTHEREFR , considers latent bandits where there is a partition of context types B into C clusters C = {B c }, each with known reward distribution.

However, the learner, when receiving the context type b ∈ B, does not know the cluster B c containing b. This is a much harder problem.

Stochastic Contextual Bandits With Graph-Based Contexts

Abstract: We naturally generalize the on-line graph prediction problem to a version of stochastic contextual bandit problems where contexts are vertices in a graph and the structure of the graph provides information on the similarity of contexts. More specifically, we are given a graph G = (V, E), whose vertex set V represents contexts with unknown vertex label y. In our stochastic contextual bandit setting, vertices with the same label share the same reward distribution. The standard notion of instance difficulties in graph label prediction is the cutsize f defined to be the number of edges whose end points having different labels. For line graphs and trees we present an algorithm with regret bound ofÕ(T 2/3 K 1/3 f 1/3) where K is the number of arms. Our algorithm relies on the optimal stochastic bandit algorithm by Zimmert and Seldin [AISTAT'19, JMLR'21]. When the best arm outperforms the other arms, the regret improves toÕ(√ KT • f). The regret bound in the later case is comparable to other optimal contextual bandit results in more general cases, but our algorithm is easy to analyze, runs very efficiently, and does not require an i.i.d. assumption on the input context sequence. The algorithm also works with general graphs using a standard random spanning tree reduction.

Online Clustering Of Bandits

Abstract: We introduce a novel algorithmic approach to content recommendation based on adaptive clustering of exploration-exploitation "bandit") strategies. We provide a sharp regret analysis of this algorithm in a standard stochastic noise setting, demonstrate its scalability properties, and prove its effectiveness on a number of artificial and real-world datasets. Our experiments show a significant increase in prediction performance over state-of-the-art methods for bandit problems.


Prior Human-Designed Heuristics For Choosing The Ordering

It is perhaps surprising that the original heuristic of Brown is able to perform competitively or even beat these others despite having access to less information #REFR .

Once the importance of variable ordering was established, researchers began looking for strategies to choose a good variable ordering. The first attempts consisted of designing human-made heuristics.

In 2004 Brown #OTHEREFR documented a heuristic based on three simple hand-picked features of the set of polynomials, for the software QEPCAD. We call this heuristic Brown.

There have been other heuristics produced which can offer greater accuracy but at greater expense, performing increasing numbers of steps in the CAD algorithm: Dolzmann et al.

#OTHEREFR concluded it best to perform the projection stage of CAD and compare sums of the total degree of the polynomials produced (sotd); Bradford et al.

#OTHEREFR considered the initial decomposition of the real line; and Wilson et al. #OTHEREFR the open cells in the decomposition.

This suggests the necessary information to identify a good ordering may be available from the input alone.

Most recently del Río and England #OTHEREFR designed a new simple heuristic gmods based on properties of the input polynomials, selected by studying which features of the input polynomials have the greatest impact on the complexity analysis of CAD.

Explainable Ai Insights For Symbolic Computation: A Case Study On Selecting The Variable Ordering For Cylindrical Algebraic Decomposition

Abstract: In recent years there has been increased use of machine learning (ML) techniques within mathematics, including symbolic computation where it may be applied safely to optimise or select algorithms. This paper explores whether using explainable AI (XAI) techniques on such ML models can offer new insight for symbolic computation, inspiring new implementations within computer algebra systems that do not directly call upon AI tools. We present a case study on the use of ML to select the variable ordering for cylindrical algebraic decomposition. It has already been demonstrated that ML can make the choice well, but here we show how the SHAP tool for explainability can be used to inform new heuristics of a size and complexity similar to those human-designed heuristics currently commonly used in symbolic computation.

Using Machine Learning To Improve Cylindrical Algebraic Decomposition

Abstract: Cylindrical Algebraic Decomposition (CAD) is a key tool in computational algebraic geometry, best known as a procedure to enable Quantifier Elimination over real-closed fields. However, it has a worst case complexity doubly exponential in the size of the input, which is often encountered in practice. It has been observed that for many problems a change in algorithm settings or problem formulation can cause huge differences in runtime costs, changing problem instances from intractable to easy. A number of heuristics have been developed to help with such choices, but the complicated nature of the geometric relationships involved means these are imperfect and can sometimes make poor choices. We investigate the use of machine learning (specifically support vector machines) to make such choices instead. Machine learning is the process of fitting a computer model to a complex function based on properties learned from measured data. In this paper we apply it in two case studies: the first to select between heuristics for choosing a CAD variable ordering; the second to identify when a CAD problem instance would benefit from Grobner Basis preconditioning. These appear to be the first such applications of machine learning to Symbolic Computation. We demonstrate in both cases that the machine learned choice outperforms human developed heuristics.


The ML models and their corresponding functions such as clustering, classification and feature extraction, in the context of IoT are extensively investigated in #REFR .

Distributed data processing frameworks such as Hadoop MapReduce and in-memory alternatives such as Apache Spark, can be employed for this sensor data analytics on the cloud.

Moreover, since IoT mostly deals with big streaming data, message queues such as Apache Kafka can be used to buffer and feed the data into stream data processing systems such as Apache Storm and Apache Spark streaming #OTHEREFR .

Additionally, the edge analytics performed can be based on machine learning (ML). E.g. Drolia et al.

#OTHEREFR , proposed a Pre-Cog system on fog devices that recognizes images rapidly through catching and prefetching. Abdulkareem et al.

#OTHEREFR , provides a detailed review of approaches performing edge analytics using ML on fog infrastructure.

There are some generic distributable algorithms such as k-nearest neighbors (k-NN) and other special neural network methods, which can directly be used in resource constrained fog devices for performing ML tasks.

These studies lead to the development of frameworks such as CANTO #OTHEREFR , that can be used to train neural networks on fog nodes for performing edge analytics.

However, performing sophisticated ML algorithms in resource and power constrained fog nodes is still a major challenge.

Support for ML enabling hardware such as ENVISION, that can be used in fog networks, is summarized in 112 .

A Decade Of Research In Fog Computing: Relevance, Challenges, And Future Directions

Abstract: Recent developments in the Internet of Things (IoT) and real-time applications, have led to the unprecedented growth in the connected devices and their generated data. Traditionally, this sensor data is transferred and processed at the cloud, and the control signals are sent back to the relevant actuators, as part of the IoT applications. This cloud-centric IoT model, resulted in increased latencies and network load, and compromised privacy. To address these problems, Fog Computing was coined by Cisco in 2012, a decade ago, which utilizes proximal computational resources for processing the sensor data. Ever since its proposal, fog computing has attracted significant attention and the research fraternity focused at addressing different challenges such as fog frameworks, simulators, resource management, placement strategies, quality of service aspects, fog economics etc. However, after a decade of research, we still do not see large-scale deployments of public/private fog networks, which can be utilized in realizing interesting IoT applications. In the literature, we only see pilot case studies and small-scale testbeds, and utilization of simulators for demonstrating scale of the specified models addressing the respective technical challenges. There are several reasons for this, and most importantly, fog computing did not present a clear business case for the companies and participating individuals yet. This paper summarizes challenges, state-of-the-art and future research directions in realizing real-time fog computing applications. Contrary to other survey papers, that exhaustively address a specific set of aspects of fog computing, this work discusses the fog research challenges and solutions in much broader scope and thus provides a thorough opinion about progressing the research and quickly adapting fog computing in real-world applications.

Machine Learning For Internet Of Things Data Analysis: A Survey

Abstract: Rapid developments in hardware, software, and communication technologies have facilitated the emergence of Internet-connected sensory devices that provide observations and data measurements from the physical world. By 2020, it is estimated that the total number of Internet-connected devices being used will be between 25 and 50 billion. As these numbers grow and technologies become more mature, the volume of data being published will increase. The technology of Internet-connected devices, referred to as Internet of Things (IoT), continues to extend the current Internet by providing connectivity and interactions between the physical and cyber worlds. In addition to an increased volume, the IoT generates big data characterized by its velocity in terms of time and location dependency, with a variety of multiple modalities and varying data quality. Intelligent processing and analysis of this big data are the key to developing smart IoT applications. This article assesses the various machine learning methods that deal with the challenges presented by IoT data by considering smart cities as the main use case. The key contribution of this study is the presentation of a taxonomy of machine learning algorithms explaining how different techniques are applied to the data in order to extract higher level information. The potential and challenges of machine learning for IoT data analytics will also be discussed. A use case of applying a Support Vector Machine (SVM) to Aarhus smart city traffic data is presented for a more detailed exploration.


Sampling From The Posterior Distribution

Furthermore, this typically requires the calculation of higher-order derivatives #REFR which is computationally expensive.

Generating high-quality sample weights from the posterior distribution is usually done via the simulation of a Markov chain which converges in distribution to the posterior.

While several Markov chains methods have been constructed for neural network applications, we found them unsuitable for this use case.

The main difficulty we encountered was, that the scale of the gradients with respect to the different sets of parameters corresponding to the different layers of the neural network vary by several orders of magnitude.

This makes the use of a single step size for all parameters impossible, as it would cause the sets of parameters with smaller gradients to be frozen during the optimization.

While modern neural network optimizers circumvent this problem through an adaptive step size for each parameter #OTHEREFR , this is only possible to a very limited degree for Markov chains #OTHEREFR without changing the stationary distribution to which they converge.

To deal with these challenges, we develop here a new approach to sampling the posterior distribution based on the Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) algorithm introduced by Chen et al. #OTHEREFR .

In its basic form, the algorithm is given by the Markov chain

∆w t = − ∇ θt u(θ t ) + M −1 Cw t−1 ∆t+ √ 2C∆tN t (0, I), ∆θ t = M −1 w t ∆t.

u(θ) := − ln p(θ) − l i=1 ln p(y i |x i , θ),

High Accuracy Uncertainty-Aware Interatomic Force Modeling With Equivariant Bayesian Neural Networks

Abstract: Even though Bayesian neural networks offer a promising framework for modeling uncertainty, active learning and incorporating prior physical knowledge, few applications of them can be found in the context of interatomic force modeling. One of the main challenges in their application to learning interatomic forces is the lack of suitable Monte Carlo Markov chain sampling algorithms for the posterior density, as the commonly used algorithms do not converge in a practical amount of time for many of the state-of-the-art architectures. As a response to this challenge, we introduce a new Monte Carlo Markov chain sampling algorithm in this paper which can circumvent the problems of the existing sampling methods. In addition, we introduce a new stochastic neural network model based on the NequIP architecture and demonstrate that, when combined with our novel sampling algorithm, we obtain predictions with state-of-the-art accuracy as well as a good measure of uncertainty.

A Complete Recipe For Stochastic Gradient Mcmc

Abstract: Many recent Markov chain Monte Carlo (MCMC) samplers leverage continuous dynamics to define a transition kernel that efficiently explores a target distribution. In tandem, a focus has been on devising scalable variants that subsample the data and use stochastic gradients in place of full-data gradients in the dynamic simulations. However, such stochastic gradient MCMC samplers have lagged behind their full-data counterparts in terms of the complexity of dynamics considered since proving convergence in the presence of the stochastic gradient noise is non-trivial. Even with simple dynamics, significant physical intuition is often required to modify the dynamical system to account for the stochastic gradient noise. In this paper, we provide a general recipe for constructing MCMC samplers--including stochastic gradient versions--based on continuous Markov processes specified via two matrices. We constructively prove that the framework is complete. That is, any continuous Markov process that provides samples from the target distribution can be written in our framework. We show how previous continuous-dynamic samplers can be trivially"reinvented"in our framework, avoiding the complicated sampler-specific proofs. We likewise use our recipe to straightforwardly propose a new state-adaptive sampler: stochastic gradient Riemann Hamiltonian Monte Carlo (SGRHMC). Our experiments on simulated data and a streaming Wikipedia analysis demonstrate that the proposed SGRHMC sampler inherits the benefits of Riemann HMC, with the scalability of stochastic gradient methods.


In contrast to VAE that uses KL divergence and evidence lower bound, adversarial autoencoder (AAE) #REFR uses adversarial learning to impose a specific distribution on the latent variables, making itself superior to VAE in terms of imposing complicated distributions and shaping the latent space.

The variational autoencoder (VAE) #OTHEREFR provides a principled method for jointly learning deep latent-variable models and corresponding inference models using stochastic gradient descent #OTHEREFR , which is commonly used to generate samples in the target space from pre-defined latent distribution.

Training a VAE model consists of two kinds of loss: regularization and reconstruction.

The regularization is aimed to encode the input as certain distributions over the latent space using Kullback-Leibler (KL) divergence, while the reconstruction is to decode the latent variables to the target or original space.

In our work, we utilize an AAE architecture to model the semantics in the driving scenario.

Learning Representation For Anomaly Detection Of Vehicle Trajectories

Abstract: Predicting the future trajectories of surrounding vehicles based on their history trajectories is a critical task in autonomous driving. However, when small crafted perturbations are introduced to those history trajectories, the resulting anomalous (or adversarial) trajectories can significantly mislead the future trajectory prediction module of the ego vehicle, which may result in unsafe planning and even fatal accidents. Therefore, it is of great importance to detect such anomalous trajectories of the surrounding vehicles for system safety, but few works have addressed this issue. In this work, we propose two novel methods for learning effective and efficient representations for online anomaly detection of vehicle trajectories. Different from general time-series anomaly detection, anomalous vehicle trajectory detection deals with much richer contexts on the road and fewer observable patterns on the anomalous trajectories themselves. To address these challenges, our methods exploit contrastive learning techniques and trajectory semantics to capture the patterns underlying the driving scenarios for effective anomaly detection under supervised and unsupervised settings, respectively. We conduct extensive experiments to demonstrate that our supervised method based on contrastive learning and unsupervised method based on reconstruction with semantic latent space can significantly improve the performance of anomalous trajectory detection in their corresponding settings over various baseline methods. We also demonstrate our methods' generalization ability to detect unseen patterns of anomalies.

Adversarial Autoencoders

Abstract: In this paper, we propose the"adversarial autoencoder"(AAE), which is a probabilistic autoencoder that uses the recently proposed generative adversarial networks (GAN) to perform variational inference by matching the aggregated posterior of the hidden code vector of the autoencoder with an arbitrary prior distribution. Matching the aggregated posterior to the prior ensures that generating from any part of prior space results in meaningful samples. As a result, the decoder of the adversarial autoencoder learns a deep generative model that maps the imposed prior to the data distribution. We show how the adversarial autoencoder can be used in applications such as semi-supervised classification, disentangling style and content of images, unsupervised clustering, dimensionality reduction and data visualization. We performed experiments on MNIST, Street View House Numbers and Toronto Face datasets and show that adversarial autoencoders achieve competitive results in generative modeling and semi-supervised classification tasks.


For fair comparison, we select a subset from each user group to keep the interaction number for each user fixed, thus preventing the potential impact of node degree to results #REFR . Figure 8 presents the results.

To verify that our model can alleviate potential gradient conflict, we perform experiments on user groups with different behavior relevance levels.

In particular, we divide the test set into six user groups according to the average Pearson correlation #OTHEREFR among all behaviors.

The calculation of average Pearson correlation can be referred to Appendix A.4.

We omit the results on the IJCAI dataset due to space limitation, which have consistent trends.

For more rigorous results, we run each experiment 5 times and draw the mean and fluctuation range on the figure.

We find that MESI consistently outperforms all baselines among all user groups, which further demonstrates the superiority of MESI for MTL.

Besides, with the increase of behavior correlations, MESI gets better performances, while the performances of other baselines fluctuate or even decrease.

A possible reason is the negative transfer caused by potential gradient conflict when knowledge is transferred across different tasks.

Compressed Interaction Graph Based Framework For Multi-Behavior Recommendation

Abstract: Multi-types of user behavior data (e.g., clicking, adding to cart, and purchasing) are recorded in most real-world recommendation scenarios, which can help to learn users' multi-faceted preferences. However, it is challenging to explore multi-behavior data due to the unbalanced data distribution and sparse target behavior, which lead to the inadequate modeling of high-order relations when treating multi-behavior data "as features" and gradient conflict in multitask learning when treating multi-behavior data "as labels". In this paper, we propose CIGF, a Compressed Interaction Graph based Framework, to overcome the above limitations. Specifically, we design a novel Compressed Interaction Graph Convolution Network (CIGCN) to model instance-level high-order relations explicitly. To alleviate the potential gradient conflict when treating multi-behavior data "as labels", we propose a Multi-Expert with Separate Input (MESI) network with separate input on the top of CIGCN for multi-task learning. Comprehensive experiments on three large-scale real-world datasets demonstrate the superiority of CIGF. Ablation studies and in-depth analysis further validate * Both authors contributed equally to this research. † Work done when they were research interns at Huawei Noah's Ark Lab. ‡ Corresponding author. This work is licensed under a Creative Commons Attribution International 4.0 License.

Neural Graph Collaborative Filtering

Abstract: Learning vector representations (aka. embeddings) of users and items lies at the core of modern recommender systems. Ranging from early matrix factorization to recently emerged deep learning based methods, existing efforts typically obtain a user's (or an item's) embedding by mapping from pre-existing features that describe the user (or the item), such as ID and attributes. We argue that an inherent drawback of such methods is that, the collaborative signal, which is latent in user-item interactions, is not encoded in the embedding process. As such, the resultant embeddings may not be sufficient to capture the collaborative filtering effect. In this work, we propose to integrate the user-item interactions - more specifically the bipartite graph structure - into the embedding process. We develop a new recommendation framework Neural Graph Collaborative Filtering (NGCF), which exploits the user-item graph structure by propagating embeddings on it. This leads to the expressive modeling of high-order connectivity in user-item graph, effectively injecting the collaborative signal into the embedding process in an explicit manner. We conduct extensive experiments on three public benchmarks, demonstrating significant improvements over several state-of-the-art models like HOP-Rec [39] and Collaborative Memory Network [5]. Further analysis verifies the importance of embedding propagation for learning better user and item representations, justifying the rationality and effectiveness of NGCF. Codes are available at https://github.com/xiangwang1223/neural_graph_collaborative_filtering.


Generating Mnist And Fashionmnist Images

Table 2 also compares with several existing models, while Figure 3 visually compares results with DP-GAN #REFR , DP-MERF (Harder et al., 2021) and DP-HP (Vinaroz et al., 2022) .

This necessitates a trade-off in the feature dimensionality: large dimensions can lead to overwhelming amounts of added noise, while small dimensions may be inadequate to serve as a loss for image generation.

Figure 1 and Table 1 show that for both MNIST and Fashion-MNIST, changing the width does somewhat affect the final accuracy and image quality, but this effect is very minimal.

Therefore, for subsequent experiments, we choose a width of 800 as a good compromise.

Varying privacy levels Table 2 and Figure 2 show our model's performance under different levels of privacy.

Other than for FashionMNIST with = 0.2, the performance of our model does not degrade significantly as the privacy requirement becomes more stringent.

We see that even with simple architectures, DP-NTK broadly performs better than other high-accuracy models, and generates comprehensible images.

Differentially Private Neural Tangent Kernels For Privacy-Preserving Data Generation

Abstract: Maximum mean discrepancy (MMD) is a particularly useful distance metric for differentially private data generation: when used with finitedimensional features it allows us to summarize and privatize the data distribution once, which we can repeatedly use during generator training without further privacy loss. An important question in this framework is, then, what features are useful to distinguish between real and synthetic data distributions, and whether those enable us to generate quality synthetic data. This work considers the using the features of neural tangent kernels (NTKs), more precisely empirical NTKs (e-NTKs). We find that, perhaps surprisingly, the expressiveness of the untrained e-NTK features is comparable to that of the features taken from pre-trained perceptual features using public data. As a result, our method improves the privacy-accuracy trade-off compared to other state-of-the-art methods, without relying on any public data, as demonstrated on several tabular and image benchmark datasets.

Differentially Private Generative Adversarial Network

Abstract: Generative Adversarial Network (GAN) and its variants have recently attracted intensive research interests due to their elegant theoretical foundation and excellent empirical performance as generative models. These tools provide a promising direction in the studies where data availability is limited. One common issue in GANs is that the density of the learned generative distribution could concentrate on the training data points, meaning that they can easily remember training samples due to the high model complexity of deep networks. This becomes a major concern when GANs are applied to private or sensitive data such as patient medical records, and the concentration of distribution may divulge critical patient information. To address this issue, in this paper we propose a differentially private GAN (DPGAN) model, in which we achieve differential privacy in GANs by adding carefully designed noise to gradients during the learning procedure. We provide rigorous proof for the privacy guarantee, as well as comprehensive empirical evidence to support our analysis, where we demonstrate that our method can generate high quality data points at a reasonable privacy level.


The number of participants in internal datasets varied by orders of magnitude, with each study including 1 to 664 ovarian cancer patients, and one study including over 10,000 total patients across a range of 32 malignancies #REFR .

Only the five most common subtypes of ovarian carcinoma were used, with no study reporting the inclusion of less common carcinomas or non-epithelial ovarian cancers.

Only one study explicitly included any prospective data collection, and this was only for a small subset which was not used for external validation #OTHEREFR .

As shown in Figure 3 , the number of pathology slides used was often much greater than the number of patients included, with three studies using over 1000 slides from ovarian cancer patients #OTHEREFR .

Most of the studies used WSIs for model development (27/36) , with others using tissue microarrays (TMAs) (4/36) or pre-cropped digital pathology images (2/36).

Most studies used H&E-stained tissue (27/36) and the others used a variety of IHC stains (9/36), with no two papers reporting the use of the same IHC stains.

Artificial Intelligence In Ovarian Cancer Histopathology: A Systematic Review

Abstract: To characterise and assess the quality of published research evaluating artificial intelligence (AI) methods for ovarian cancer diagnosis or prognosis using histopathology data. Methods A search of PubMed, Scopus, Web of Science, Cochrane Central Register of Controlled Trials, and WHO International Clinical Trials Registry Platform was conducted up to 01/12/2022. The inclusion criteria required that research evaluated AI on histopathology images for diagnostic or prognostic inferences in ovarian cancer, including primary tumours of the ovaries, fallopian tubes, and peritoneum. Reviews and non-English language articles were excluded. The risk of bias was assessed for every model that met the inclusion criteria using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). Information about each model of interest was tabulated and summary statistics were reported. Based on the results, we provided recommendations to improve study design and reporting to reduce the risk of bias and improve the reproducibility of future research in the field. The study protocol was registered on PROSPERO (CRD42022334730). PRISMA 2020 reporting guidelines were followed. Results A total of 1434 research articles were identified, of which 36 were eligible for inclusion. These studies reported 62 models of interest, including 35 classifiers, 14 survival prediction models, 7 segmentation models, and 6 regression models. Models were developed using 1-1375 slides from 1-664 ovarian cancer patients. A wide array of outcomes were predicted, including overall survival (9/62), histological subtypes (7/62), stain quantity (6/62), malignancy (5/62), primary cancer (4/62), and tumour region (4/62). Older studies used traditional machine learning (ML) models with hand-crafted features, while newer studies typically employed deep learning (DL) to automatically learn features and predict the outcome(s) of interest. All models were found to be at high or unclear risk of bias overall, with most research having a high risk of bias in the analysis and a lack of clarity regarding participants and predictors in the study. Research was frequently limited by insufficient reporting, small sample sizes, and insufficient validation, with external validation being particularly rare. Conclusion Limited research has been conducted on the application of AI to histopathology images for diagnostic or prognostic purposes in ovarian cancer, and none of the associated models have been demonstrated to be ready for real-world implementation. Recommendations are provided addressing underlying biases and flaws in study design, which should help inform higher-quality reproducible future research. Key aspects to help ensure clinical translation include more transparent and comprehensive reporting of data provenance and modelling approaches, as well as improved quantitative performance evaluation using cross-validation and external validations.

Pan-Cancer Diagnostic Consensus Through Searching Archival Histopathology Images Using Artificial Intelligence

Abstract: The emergence of digital pathology has opened new horizons for histopathology. Artificial intelligence (AI) algorithms are able to operate on digitized slides to assist pathologists with different tasks. Whereas AI-involving classification and segmentation methods have obvious benefits for image analysis, image search represents a fundamental shift in computational pathology. Matching the pathology of new patients with already diagnosed and curated cases offers pathologists a new approach to improve diagnostic accuracy through visual inspection of similar cases and computational majority vote for consensus building. In this study, we report the results from searching the largest public repository (The Cancer Genome Atlas, TCGA) of whole-slide images from almost 11,000 patients. We successfully indexed and searched almost 30,000 high-resolution digitized slides constituting 16 terabytes of data comprised of 20 million 1000 × 1000 pixels image patches. The TCGA image database covers 25 anatomic sites and contains 32 cancer subtypes. High-performance storage and GPU power were employed for experimentation. The results were assessed with conservative "majority voting" to build consensus for subtype diagnosis through vertical search and demonstrated high accuracy values for both frozen section slides (e.g., bladder urothelial carcinoma 93%, kidney renal clear cell carcinoma 97%, and ovarian serous cystadenocarcinoma 99%) and permanent histopathology slides (e.g., prostate adenocarcinoma 98%, skin cutaneous melanoma 99%, and thymoma 100%). The key finding of this validation study was that computational consensus appears to be possible for rendering diagnoses if a sufficiently large number of searchable cases are available for each cancer subtype.


We observe that FMAS can reproduce the baseline accuracy of DL3+ #REFR , achieving MIoU errors of 23% (e.g., FMAS-FP1), compared to a reported error of 21% on the validation set.

Figures 2, 3, and 4 plot the MIoU error of the Pareto non-dominated front of the corresponding generation against the FLOPs count, network parameters count, and latency respectively.

We explore the capacity of FMAS to cut GPU time by evaluating a total of 240 modified Xception variants developed over 20 generations targeting FLOPs, and parameters; and a total of 300 MobileNetV2 variants developed over 25 generations targeting latency.

Table 3 reports the network structures, GPU time consumption, computational cost, accuracy evaluated on a subset of the validation set, and post-fine-tuning accuracy on the entire validation set of selected networks using the Xception backbone. It presents a #OTHEREFR .

Similar to Table 3 , Table 4 reports results when using the Mo-bileNetV2 backbone and searching for 25 generations.

In addition to FLOPs and parameters, we also report inference latency on the GAP8 for the original model, FCN-VGG16, and selected search results.

Note that while FCN-VGG16 uses only GAP8-supported operations, making it a suitable baseline, it requires more than 8× more RAM than the GAP8 has, and therefore cannot be deployed. Table 1 reports their hyperparameters.

FMAS-F1 cuts the number of FLOPs by 43% with respect to DL3+, and network parameters by 7.9%, for a relative increase of 5.2% in MIoU error; it was discovered in 0.68 GPU days (generation 17).

FMAS-F2 trades off only 2.5% of the MIoU error of DL3+ for reducing FLOPs by 10%, and network parameters by 20%, in 0.52 GPU days (generation 13).

Fmas: Fast Multi-Objective Supernet Architecture Search For Semantic Segmentation

Abstract: We present FMAS, a fast multi-objective neural architecture search framework for semantic segmentation. FMAS subsamples the structure and pre-trained parameters of DeepLabV3+, without finetuning, dramatically reducing training time during search. To further reduce candidate evaluation time, we use a subset of the validation dataset during the search. Only the final, Pareto non-dominated, candidates are ultimately fine-tuned using the complete training set. We evaluate FMAS by searching for models that effectively trade accuracy and computational cost on the PASCAL VOC 2012 dataset. FMAS finds competitive designs quickly, e.g., taking just 0.5 GPU days to discover a DeepLabV3+ variant that reduces FLOPs and parameters by 10% and 20% respectively, for less than 3% increased error. We also search on an edge device called GAP8 and use its latency as the metric. FMAS is capable of finding 2.2× faster network with 7.61% MIoU loss.

Encoder-Decoder With Atrous Separable Convolution For Semantic Image Segmentation

Abstract: Spatial pyramid pooling module or encode-decoder structure are used in deep neural networks for semantic segmentation task. The former networks are able to encode multi-scale contextual information by probing the incoming features with filters or pooling operations at multiple rates and multiple effective fields-of-view, while the latter networks can capture sharper object boundaries by gradually recovering the spatial information. In this work, we propose to combine the advantages from both methods. Specifically, our proposed model, DeepLabv3+, extends DeepLabv3 by adding a simple yet effective decoder module to refine the segmentation results especially along object boundaries. We further explore the Xception model and apply the depthwise separable convolution to both Atrous Spatial Pyramid Pooling and decoder modules, resulting in a faster and stronger encoder-decoder network. We demonstrate the effectiveness of the proposed model on PASCAL VOC 2012 and Cityscapes datasets, achieving the test set performance of 89.0\% and 82.1\% without any post-processing. Our paper is accompanied with a publicly available reference implementation of the proposed models in Tensorflow at \url{https://github.com/tensorflow/models/tree/master/research/deeplab}.


This behavior, consistant with findings by #REFR showing that highly predictive non-robust features in the data tend to reduce learner performance when presented with out-of-distribution examples.

In contrast, Figs.4b,4f ,4c,4g shows significant differences in performance for out-of-distribution (OOD) example testing.

ReI presents better behaved performance and outperforms by larger margins compared to the standard in all cases.

Note that even though the standard methods present a performance advantage over ReI for in-distribution examples, that this is not the case for out-of-distribution examples collected in the wild from Mars.

ReI learns representations with disentangled variables that show better generalizations against the tested out-of-distribution examples.

Table 1 provides addditional results comparing performance on Earth-to-Mars transfer on a variety of DL architectures and averaged over all elements y ∈ R n with n = 11.

Comparisons include fully connected (FC), multilayer perceptron (MLP), MLP Mixer #OTHEREFR , ResNet #OTHEREFR , U-Net #OTHEREFR , Transformers #OTHEREFR and again the standard VAE #OTHEREFR .

Note that some of the architectures do not produce a latent representation explicitly, these are however rather trained end-to-end for prediction.

The number in parenthesis next to each architecture name (e.g., FC(10)) expresses the corresponding depth of layers.

Representation Disentaglement Via Regularization By Identification

Abstract: This work focuses on the problem of learning disentangled representations from observational data. Given observations {x (i) } N i=1 drawn from p(x|y) with generative variables y admitting the distribution factorization p(y) = c p(y c), we ask whether learning disentangled representations matching the space of observations with identification guarantees on the posterior p(z|x,ŷ c) for each c, is plausible. We argue modern deep representation learning models are ill-posed with collider bias behaviour; a source of bias producing entanglement between generating variables. Under the rubric of causality, we show this issue can be explained and reconciled under the condition of identifiability; attainable under supervision or a weak-form of it. For this, we propose regularization by identification (ReI), a regularization framework defined by the identification of the causal queries involved in the learning problem. Empirical evidence shows that enforcing ReI in a variational framework results in disentangled representations equipped with generalization capabilities to out-of-distribution examples and that aligns nicely with the true expected effect between generating variables and measurement apparatus.

Robustness May Be At Odds With Accuracy

Abstract: We show that there exists an inherent tension between the goal of adversarial robustness and that of standard generalization. Specifically, training robust models may not only be more resource-consuming, but also lead to a reduction of standard accuracy. We demonstrate that this trade-off between the standard accuracy of a model and its robustness to adversarial perturbations provably exists even in a fairly simple and natural setting. These findings also corroborate a similar phenomenon observed in practice. Further, we argue that this phenomenon is a consequence of robust classifiers learning fundamentally different feature representations than standard classifiers. These differences, in particular, seem to result in unexpected benefits: the representations learned by robust models tend to align better with salient data characteristics and human perception.


This result was later developed in #REFR by counting the number of pairs of coprime polynomials with a nonzero constant term, and by providing a construction for maximal families of Mutually Orthogonal Latin Squares (MOLS) generated by linear bipermutive CA.

#OTHEREFR studied the analogies between the Lattice Gas Automata (LGA) model for fluids and the SPN paradigm.

In particular, the authors proposed to use the collision operator of the LGA model to implement the substitution layer of a block cipher.

We conclude this section with a brief outlook of the research line devoted to orthogonal cellular automata, which has been mostly investigated by the second author of this manuscript. Mariot et al.

first proved in #OTHEREFR a necessary and sufficient condition for a pair of linear bipermutive CA to generate orthogonal Latin squares.

The characterization is quite simple, since it consists in checking whether the polynomials associated to the local rules of the CA are relatively prime.

These results have been subsequently used by Gadouleau et al.

#OTHEREFR to devise a new construction of bent Boolean functions, which reach the highest possible nonlinearity.

Later, it turned out that the construction could be greatly simplified through the formalism of linear recurring sequences, instead of using orthogonal CA #OTHEREFR . Formenti et al.

#OTHEREFR devised a combinatorial algorithm to enumerate all pairs of coprime polynomials with nonzero constant term, and thus all linear OCA of a given diameter.

Finally, Mariot #OTHEREFR considered orthogonal CA as pseudorandom generators, and devised an algorithm to compute the period of the resulting sequences when the underlying CA are linear.

A Classification Of S-Boxes Generated By Orthogonal Cellular Automata

Abstract: Most of the approaches published in the literature to construct Sboxes via Cellular Automata (CA) work by either iterating a finite CA for several time steps, or by a one-shot application of the global rule. The main characteristic that brings together these works is that they employ a single CA rule to define the vectorial Boolean function of the S-box. In this work, we explore a different direction for the design of S-boxes that leverages on Orthogonal CA (OCA), i.e. pairs of CA rules giving rise to orthogonal Latin squares. The motivation stands on the facts that an OCA pair already defines a bijective transformation, and moreover the orthogonality property of the resulting Latin squares ensures a minimum amount of diffusion. We exhaustively enumerate all S-boxes generated by OCA pairs of diameter 4 ≤ d ≤ 6, and measure their nonlinearity. Interestingly, we observe that for d = 4 and d = 5 all S-boxes are linear, despite the underlying CA local rules being nonlinear. The smallest nonlinear S-boxes emerges for d = 6, but their nonlinearity is still too low to be used in practice. Nonetheless, we unearth an interesting structure of linear OCA S-boxes, proving that their Linear Components Space (LCS) is itself the image of a linear CA, or equivalently a polynomial code. We finally classify all linear OCA S-boxes in terms of their generator polynomials.

Mutually Orthogonal Latin Squares Based On Cellular Automata

Abstract: We investigate sets of mutually orthogonal latin squares (MOLS) generated by cellular automata (CA) over finite fields. After introducing how a CA defined by a bipermutive local rule of diameter d over an alphabet of q elements generates a Latin square of order \(q^{d-1}\), we study the conditions under which two CA generate a pair of orthogonal Latin squares. In particular, we prove that the Latin squares induced by two Linear Bipermutive CA (LBCA) over the finite field \(\mathbb {F}_q\) are orthogonal if and only if the polynomials associated to their local rules are relatively prime. Next, we enumerate all such pairs of orthogonal Latin squares by counting the pairs of coprime monic polynomials with nonzero constant term and degree n over \(\mathbb {F}_q\). Finally, we present a construction for families of MOLS based on LBCA, and prove that their cardinality corresponds to the maximum number of pairwise coprime polynomials with nonzero constant term. Although our construction does not yield all such families of MOLS, we show that the resulting lower bound is asymptotically close to their actual number.


C.3 Additional Qualitative Comparison With Baselines

In Figure 12 , even the GT label for semantic segmentation ("couch" class) is noisy as it is a pseudolabel generated by a pre-trained segmentation model #REFR , our model successfully segments two couches present in the figure.

We provide additional results on the qualitative evaluation of our model and the baselines.

Figure 11 -14 show visualizations on different query image and support set, where we vary the class of semantic segmentation task included in each support.

The result shows consistent trends of that we discussed in Section 5.

Ours is competitive to the fully supervised baselines (DPT and InvPT), while the other few-shot baselines (HSNet, VAT, DGPNet) fail to learn different dense prediction tasks.

This can be attributed to the task-agnostic architecture of VTM based on non-parametric matching.

Universal Few-Shot Learning Of Dense Predic-Tion Tasks With Visual Token Matching

Abstract: Dense prediction tasks are a fundamental class of problems in computer vision. As supervised methods suffer from high pixel-wise labeling cost, a few-shot learning solution that can learn any dense task from a few labeled images is desired. Yet, current few-shot learning methods target a restricted set of tasks such as semantic segmentation, presumably due to challenges in designing a general and unified model that is able to flexibly and efficiently adapt to arbitrary tasks of unseen semantics. We propose Visual Token Matching (VTM), a universal few-shot learner for arbitrary dense prediction tasks. It employs non-parametric matching on patchlevel embedded tokens of images and labels that encapsulates all tasks. Also, VTM flexibly adapts to any task with a tiny amount of task-specific parameters that modulate the matching algorithm. We implement VTM as a powerful hierarchical encoder-decoder architecture involving ViT backbones where token matching is performed at multiple feature hierarchies. We experiment VTM on a challenging variant of Taskonomy dataset and observe that it robustly few-shot learns various unseen dense prediction tasks. Surprisingly, it is competitive with fully supervised baselines using only 10 labeled examples of novel tasks (0.004% of full supervision) and sometimes outperforms using 0.1% of full supervision. Codes are available at https://github.com/GitGyun/visual_token_matching.

Taskonomy: Disentangling Task Transfer Learning

Abstract: Do visual tasks have a relationship, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a structure among visual tasks. Knowing this structure has notable values; it is the concept underlying transfer learning and provides a principled way for identifying redundancies across tasks, e.g., to seamlessly reuse supervision among related tasks or solve many tasks in one system without piling up the complexity. We proposes a fully computational approach for modeling the structure of space of visual tasks. This is done via finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space. The product is a computational taxonomic map for task transfer learning. We study the consequences of this structure, e.g. nontrivial emerged relationships, and exploit them to reduce the demand for labeled data. For example, we show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 (compared to training independently) while keeping the performance nearly the same. We provide a set of tools for computing and probing this taxonomical structure including a solver that users can employ to devise efficient supervision policies for their use cases.


To transform the models' output Mel-spectrograms into waveforms, we use MelGAN #REFR as our vocoder.

We utilize LibriTTS #OTHEREFR as our pre-training dataset and VCTK #OTHEREFR as our voice-cloning dataset.

The implementation of our FastSpeech 2 model and the training/pruning details can be found in our GitHub repository #OTHEREFR .

In our experiments, we mainly focus on 8-shot voice cloning, where only 8 audio samples of the target speaker are used for finetuning and pruning.

For each speaker in VCTK, we randomly sample 8 recordings for a voice cloning task.

We pre-train the TTS models for 40k steps with LibriTTS, followed by fine-tuning/pruning the model with 8-shot voice cloning tasks until convergence.

The remaining recordings and their corresponding transcripts are utilized for evaluation.

Personalized Lightweight Text-To-Speech: Voice Cloning With Adaptive Structured Pruning

Abstract: Personalized TTS is an exciting and highly desired application that allows users to train their TTS voice using only a few recordings. However, TTS training typically requires many hours of recording and a large model, making it unsuitable for deployment on mobile devices. To overcome this limitation, related works typically require fine-tuning a pre-trained TTS model to preserve its ability to generate high-quality audio samples while adapting to the target speaker's voice. This process is commonly referred to as "voice cloning." Although related works have achieved significant success in changing the TTS model's voice, they are still required to fine-tune from a large pre-trained model, resulting in a significant size for the voicecloned model. In this paper, we propose applying trainable structured pruning to voice cloning. By training the structured pruning masks with voice-cloning data, we can produce a unique pruned model for each target speaker. Our experiments demonstrate that using learnable structured pruning, we can compress the model size to 7 times smaller while achieving comparable voice-cloning performance.

Melgan: Generative Adversarial Networks For Conditional Waveform Synthesis

Abstract: Previous works (Donahue et al., 2018a; Engel et al., 2019a) have found that generating coherent raw audio waveforms with GANs is challenging. In this paper, we show that it is possible to train GANs reliably to generate high quality coherent waveforms by introducing a set of architectural changes and simple training techniques. Subjective evaluation metric (Mean Opinion Score, or MOS) shows the effectiveness of the proposed approach for high quality mel-spectrogram inversion. To establish the generality of the proposed techniques, we show qualitative results of our model in speech synthesis, music domain translation and unconditional music synthesis. We evaluate the various components of the model through ablation studies and suggest a set of guidelines to design general purpose discriminators and generators for conditional sequence synthesis tasks. Our model is non-autoregressive, fully convolutional, with significantly fewer parameters than competing models and generalizes to unseen speakers for mel-spectrogram inversion. Our pytorch implementation runs at more than 100x faster than realtime on GTX 1080Ti GPU and more than 2x faster than real-time on CPU, without any hardware specific optimization tricks.


Some expand on the ideas of influence prediction discussed previously to give an estimate of the level of activity a particular topic will receive, rather than a binary indicator for whether the campaign was successful (e.g. #REFR ).

Complementary to attempts to understand popular topics is the identification of individual user accounts that drive these trends.

Previous efforts have considered user metadata and posting habits online #OTHEREFR or social network position #OTHEREFR to predict user influence.

Most such efforts are hampered, however, by weak definitions (that use a partial substitute or proxy for influence) or incomplete data (since most data samples do not capture all interactions between individuals or content exposure due to practicalities and privacy constraints).

Thus the challenge of measuring influence remains unsolved and new efforts are required to offer a more complete picture of interpersonal and organisational influence in online spaces.

Attempts to quantify information contagion or influence campaigns on contact networks have seen success.

Many studies of information diffusion online rely on the use of specific parameters such as users, keywords, hashtags and URLs (see e.g. #OTHEREFR

2020 ; Cruickshank and Carley 2020)) but these necessarily miss relevant discussions that do not use the predetermined search parameters.

Work by #OTHEREFR is of particular relevance to the aims of this manuscript.

Their research presented a means of quantifying the flow of attention between different conversational formats, using averaged word co-occurrence of texts to measure the change in agreement after the publication of a reference text.

Regression analysis revealed that a small, but significant, flow of words emerged from news articles shared on Facebook to speeches made by MPs in the UK Houses of Parliament.

Using Semantic Similarity And Text Embedding To Measure The Social Media Echo Of Strategic Communications

Abstract: Online discourse covers a wide range of topics and many actors tailor their content to impact online discussions through carefully crafted messages and targeted campaigns. Yet the scale and diversity of online media content make it difficult to evaluate the impact of a particular message. In this paper, we present a new technique that leverages semantic similarity to quantify the change in the discussion after a particular message has been published. We use a set of press releases from environmental organisations and tweets from the climate change debate to show that our novel approach reveals a heavy-tailed distribution of response in online discourse to strategic communications.

Predicting Successful Memes Using Network And Community Structure

Abstract: We investigate the predictability of successful memes using their early spreading patterns in the underlying social networks. We propose and analyze a comprehensive set of features and develop an accurate model to predict future popularity of a meme given its early spreading patterns. Our paper provides the first comprehensive comparison of existing predictive frameworks. We categorize our features into three groups: influence of early adopters, community concentration, and characteristics of adoption time series. We find that features based on community structure are the most powerful predictors of future success. We also find that early popularity of a meme is not a good predictor of its future popularity, contrary to common belief. Our methods outperform other approaches, particularly in the task of detecting very popular or unpopular memes.


Finally, we combine NeRF #REFR by querying point features from TriVol at sampled locations to render the final images. An overview of our method is illustrated in Fig. 2 .

We aim to train a category-specific point renderer R to directly generate photo-realistic images I (the image height and width are denoted as H and W ) from the colored point cloud P , given camera parameters (intrinsic parameter K and extrinsic parameters R and t).

When rendering novel point clouds of the same category, no fine-tuning process is required. The rendering process can be represented as

where P is usually obtained from MVS #OTHEREFR , LiDAR scanners #OTHEREFR , or sampled from synthesized mesh models.

In this section, we first encode the point cloud as the proposed TriVol, then utilize three 3D UNet to decode it into the feature representation.

Trivol: Point Cloud Rendering Via Triple Volumes

Abstract: Figure 1. Given the colored point cloud of a category-specific scene or object, our TriVol can render photo-realistic images.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


Specifically, Google's #REFR and Meta's 3 RAI principles talk about "fairness and inclusion", Amazon's 4 RAI principles promote "diversity, equity, and inclusion" through "detecting bias".

That excludes the first part of the query, which tries to match terms such as wearable(s) or mobile(s) only in the papers' meta-data, as seen in Figure 3 .

For the definition of our query, we followed similar terminology with relevant review papers in the fairness literature #OTHEREFR .

Additionally, according to Fjeld et al.'s analysis of prominent AI principles documents, #OTHEREFR , "the fairness and non-discrimination theme is the most highly represented theme in our dataset, with every document referencing at least one of its six principles: "non-discrimination and the prevention of bias", "representative and high-quality data", "fairness", "equality", "inclusiveness in impact", and "inclusiveness in design", mostly included in our query's coverage.

To capture the industrial perspective, we consulted the Responsible Artificial Intelligence (RAI) white papers issued by large tech companies.

Similarly, Nokia's #OTHEREFR RAI fairness pillar talks about "fairness, non-discrimination, accessibility, and inclusivity", and Intel's RAI pillars mention "enabling ethical and equitable AI".

Thus, an iterative refinement process resulted in the query shown in Figure 3 .

To further validate our query, we manually inspected all publications from the latest IMWUT proceedings (Volume 6, Issue 4, published in January 2023) ( = 56) to identify eligible papers for inclusion (see inclusion and exclusion criteria below).

In total, we identified seven relevant publications, all of which were also returned by our query.

Beyond Accuracy: A Critical Review Of Fairness In Machine Learning For Mobile And Wearable Computing

Abstract: The field of mobile, wearable, and ubiquitous computing (UbiComp) is undergoing a revolutionary integration of machine learning. Devices can now diagnose diseases, predict heart irregularities, and unlock the full potential of human cognition. However, the underlying algorithms are not immune to biases with respect to sensitive attributes (e.g., gender, race), leading to discriminatory outcomes. The research communities of HCI and AI-Ethics have recently started to explore ways of reporting information about datasets to surface and, eventually, counter those biases. The goal of this work is to explore the extent to which the UbiComp community has adopted such ways of reporting and highlight potential shortcomings. Through a systematic review of papers published in the Proceedings of the ACM Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) journal over the past 5 years (2018-2022), we found that progress on algorithmic fairness within the UbiComp community lags behind. Our findings show that only a small portion (5%) of published papers adheres to modern fairness reporting, while the overwhelming majority thereof focuses on accuracy or error metrics. In light of these findings, our work provides practical guidelines for the design and development of ubiquitous technologies that not only strive for accuracy but also for fairness. CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing; • Applied computing → Consumer health; • Computing methodologies → Artificial intelligence; • Social and professional topics → Codes of ethics.

Fair Regression: Quantitative Definitions And Reduction-Based Algorithms

Abstract: In this paper, we study the prediction of a real-valued target, such as a risk score or recidivism rate, while guaranteeing a quantitative notion of fairness with respect to a protected attribute such as gender or race. We call this class of problems \emph{fair regression}. We propose general schemes for fair regression under two notions of fairness: (1) statistical parity, which asks that the prediction be statistically independent of the protected attribute, and (2) bounded group loss, which asks that the prediction error restricted to any protected group remain below some pre-determined level. While we only study these two notions of fairness, our schemes are applicable to arbitrary Lipschitz-continuous losses, and so they encompass least-squares regression, logistic regression, quantile regression, and many other tasks. Our schemes only require access to standard risk minimization algorithms (such as standard classification or least-squares regression) while providing theoretical guarantees on the optimality and fairness of the obtained solutions. In addition to analyzing theoretical properties of our schemes, we empirically demonstrate their ability to uncover fairness--accuracy frontiers on several standard datasets.


We confirmed that the pose is successfully tracked without drifting #REFR . Finally, the pose estimation error was (0.13°, 0.0003).

For the POC of the proposed idea, we applied TeGRA for tracking. We used seq0 from mix scene.

We use estimated pose & motion from the previous timestep to initialize (S ini t ,Ṡ ini t ) in the next timestep as discussed in Sec.3.3. The results are shown in Fig.6 .

The average number of events per pose update was 0.2% of the entire pixel.

Event-Based Camera Tracker By ∇ T Nerf

Abstract: When a camera travels across a 3D world, only a fraction of pixel value changes; an event-based camera observes the change as sparse events. How can we utilize sparse events for efficient recovery of the camera pose? We show that we can recover the camera pose by minimizing the error between sparse events and the temporal gradient of the scene represented as a neural radiance field (NeRF). To enable the computation of the temporal gradient of the scene, we augment NeRF's camera pose as a time function. When the input pose to the NeRF coincides with the actual pose, the output of the temporal gradient of NeRF equals the observed intensity changes on the event's points. Using this principle, we propose an event-based camera pose tracking framework called TeGRA which realizes the pose update by using the sparse event's observation. To the best of our knowledge, this is the first camera pose estimation algorithm using the scene's implicit representation and the sparse intensity change from events.

Sal: Sign Agnostic Learning Of Shapes From Raw Data

Abstract: Recently, neural networks have been used as implicit representations for surface reconstruction, modelling, learning, and generation. So far, training neural networks to be implicit representations of surfaces required training data sampled from a ground-truth signed implicit functions such as signed distance or occupancy functions, which are notoriously hard to compute. In this paper we introduce Sign Agnostic Learning (SAL), a deep learning approach for learning implicit shape representations directly from raw, unsigned geometric data, such as point clouds and triangle soups. We have tested SAL on the challenging problem of surface reconstruction from an un-oriented point cloud, as well as end-to-end human shape space learning directly from raw scans dataset, and achieved state of the art reconstructions compared to current approaches. We believe SAL opens the door to many geometric deep learning applications with real-world data, alleviating the usual painstaking, often manual pre-process.


DIP-VAE #REFR uses the concept of "moment" to predict the future posterior, and uses this extra information to move the prior towards the approximate posterior.

We only show two out of infinite true posteriors here to simplify the representation.

(iii) Reducing the size of Gap 3 with adaptable priors: GMVAE #OTHEREFR creates a finite set of candidate priors.

disentangling β-VAE #OTHEREFR creates an infinite set of candidate priors on an "equal KL-divergence" line (see Figure 3 row 2 column 1 all dashed curves for clarity).

(iv) Reducing the size of Gap 3 by adapting the prior to approximate the posterior: VAMP #OTHEREFR has a mixture prior, and uses aggregate posterior #OTHEREFR to move prior towards the approximated posterior during training.

β-TCVAE #OTHEREFR adds a regularization term to optimize the prior towards the posterior during training.

FactorVAE #OTHEREFR adds a regularization term to factorize prior to adapt to low-variance posterior.

(v) Reducing the size of Gap 2R and Gap 3R with a low-variance local prior: To achieve this goal without increasing Gap 2L and 3L, having many low-variance approximate posteriors is necessary to approximate the highvariance distribution.

These posteriors result in an inaccurate approximation of hte high-level factors when a model is small. The extreme case is Autoencoders #OTHEREFR .

WAE #OTHEREFR only maintains prior-like global approximate posteriors, but reduces the variance of local posteriors.

VQVAE #OTHEREFR nears the extreme with a no-variance quantized latent prior.

Encoding Binary Concepts In The Latent Space Of Generative Models For Enhancing Data Representation

Abstract: Binary concepts 1 are empirically used by humans to generalize efficiently. And they are based on Bernoulli distribution which is the building block of information. These concepts span both low-level and high-level features such as "large vs small" and "a neuron is active or inactive". Binary concepts are ubiquitous features and can be used to transfer knowledge to improve model generalization. We propose a novel binarized regularization to facilitate learning of binary concepts to improve the quality of data generation in autoencoders. We introduce a binarizing hyperparameter r in data generation process to disentangle the latent space symmetrically. We demonstrate that this method can be applied easily to existing variational autoencoder (VAE) variants to encourage symmetric disentanglement, improve reconstruction quality, and prevent posterior collapse without computation overhead. We also demonstrate that this method can boost existing models to learn more transferable representations and generate more representative samples for the input distribution which can alleviate catastrophic forgetting using generative replay under continual learning settings.

Variational Inference Of Disentangled Latent Concepts From Unlabeled Observations

Abstract: Disentangled representations, where the higher level data generative factors are reflected in disjoint latent dimensions, offer several benefits such as ease of deriving invariant representations, transferability to other tasks, interpretability, etc. We consider the problem of unsupervised learning of disentangled representations from large pool of unlabeled observations, and propose a variational inference based approach to infer disentangled latent factors. We introduce a regularizer on the expectation of the approximate posterior over observed data that encourages the disentanglement. We evaluate the proposed approach using several quantitative metrics and empirically observe significant gains over existing methods in terms of both disentanglement and data likelihood (reconstruction quality).


We used the SacreBLEU #REFR implementation 9 of the BLUE metric to evaluate the models.

We split our dataset into different size configurations and in each configuration, the model is trained in the directions Fr→Wo and Fr→En until it reaches convergence.

Convergence is considered to be reached when no improvement is observed on the validation set after 6 checkpoints.

For data subwording, we used SentencePiece #OTHEREFR with Byte-Pair Encoding (BPE) which offers interesting performance gains in agglutinative languages like Wolof #OTHEREFR .

We then generated a vocabulary on all segments of the considered size configuration's training set and performed an automatic model evaluation using BLEU #OTHEREFR .

BLEU is the most widely used metric in NMT in view of the fairly high correlation it has with human evaluations.

Low-Resourced Machine Translation For Senegalese Wolof Language

Abstract: Natural Language Processing (NLP) research has made great advancements in recent years with major breakthroughs that have established new benchmarks. However, these advances have mainly benefited a certain group of languages commonly referred to as resource-rich such as English and French. Majority of other languages with weaker resources are then left behind which is the case for most African languages including Wolof. In this work, we present a parallel Wolof/French corpus of 123,000 sentences on which we conducted experiments on machine translation models based on Recurrent Neural Networks (RNN) in different data configurations. We noted performance gains with the models trained on subworded data as well as those trained on the French-English language pair compared to those trained on the French-Wolof pair under the same experimental conditions.

A Call For Clarity In Reporting Bleu Scores

Abstract: The field of machine translation faces an under-recognized problem because of inconsistency in the reporting of scores from its dominant metric. Although people refer to"the"BLEU score, BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters. These parameters are often not reported or are hard to find, and consequently, BLEU scores between papers cannot be directly compared. I quantify this variation, finding differences as high as 1.8 between commonly used configurations. The main culprit is different tokenization and normalization schemes applied to the reference. Pointing to the success of the parsing community, I suggest machine translation researchers settle upon the BLEU scheme used by the annual Conference on Machine Translation (WMT), which does not allow for user-supplied reference processing, and provide a new tool, SacreBLEU, to facilitate this.


Plausibility Is A Misaligned Objective For Xai Evaluation And Optimization

This may be due to the fact that post-hoc XAI algorithms only summarize partial decision information from the model, whereas information on the full decision process is scattered throughout the model #REFR .

Doing so is misleading, harmful, and cannot achieve its expected explanation goals of understandability, trustworthiness, and transparency.

The reasons are as follows: First, using plausibility to evaluate XAI algorithms is based on the assumption that the explanation truthfully reflects the AI model decision process #OTHEREFR .

Users (including both technical users and non-technical users) hold this implicit assumption for AI explanations when interpreting them and assessing their plausibility.

But explanation truthfulness or faithfulness is not an intrinsic or de facto property of XAI algorithms, and this assumption may not hold true unless it is explicitly validated #OTHEREFR .

Indeed, prior systematic evaluations on posthoc XAI algorithms #OTHEREFR -algorithms that act as surrogate models to explain for a black-box AI model -show that post-hoc XAI algorithms do not truthfully reflect the decision process of the to-be-explained AI models #OTHEREFR .

Intrinsically interpretable AI models -models that incorporate interpretability into their decision processdo not guarantee the truthfulness assumption, and should be explicitly validated for truthfulness of its explanations as well #OTHEREFR .

If we select or optimize XAI algorithms solely for their plausibility while violating the truthfulness assumption, the resultant explanation can be misleading: without the truthfulness constraint, the XAI algorithm can be purposely optimized to generate explanations that are close to those of humans, at a time when the model's explanation is not even relevant to its underlying decision process.

Such explanation cannot provide users with any insightful information about the model decision process to make the AI model more transparent.

Furthermore, since users still hold the truthfulness assumption for AI explanation, the seemingly plausible explanation may deceitfully persuade users to trust the AI model and adopt its decisions, despite being potentially wrong #OTHEREFR .

In highstakes applications, this may even lead to harmful consequences.

Rethinking Ai Explainability And Plausibility

Abstract: Setting proper evaluation objectives for explainable artificial intelligence (XAI) is vital for making XAI algorithms follow human communication norms, support human reasoning processes, and fulfill human needs for AI explanations. In this article, we examine explanation plausibility, which is the most pervasive human-grounded concept in XAI evaluation. Plausibility measures how reasonable the machine explanation is compared to the human explanation. Plausibility has been conventionally formulated as an important evaluation objective for AI explainability tasks. We argue against this idea, and show how optimizing and evaluating XAI for plausibility is sometimes harmful, and always ineffective to achieve model understandability, transparency, and trustworthiness. Specifically, evaluating XAI algorithms for plausibility regularizes the machine explanation to express exactly the same content as human explanation, which deviates from the fundamental motivation for humans to explain: expressing similar or alternative reasoning trajectories while conforming to understandable forms or language. Optimizing XAI for plausibility regardless of the model decision correctness also jeopardizes model trustworthiness, as doing so breaks an important assumption in human-human explanation namely that plausible explanations typically imply correct decisions, and violating this assumption eventually leads to either undertrust or overtrust of AI models. Instead of being the end goal in XAI evaluation, plausibility can serve as an intermediate computational proxy for the human process of interpreting explanations to optimize the utility of XAI. We further highlight the importance of explainability-specific evaluation objectives by differentiating the AI explanation task from the object localization task.

Concept Whitening For Interpretable Image Recognition

Abstract: What does a neural network encode about a concept as we traverse through the layers? Interpretability in machine learning is undoubtedly important, but the calculations of neural networks are very challenging to understand. Attempts to see inside their hidden layers can either be misleading, unusable, or rely on the latent space to possess properties that it may not have. In this work, rather than attempting to analyze a neural network posthoc, we introduce a mechanism, called concept whitening (CW), to alter a given layer of the network to allow us to better understand the computation leading up to that layer. When a concept whitening module is added to a CNN, the axes of the latent space can be aligned with concepts of interest. By experiment, we show that CW can provide us a much clearer understanding for how the network gradually learns concepts over layers without hurting predictive performance.


It is based on the popular ResNet-50 network structure and trained on the VGGFace2 dataset #REFR with standard softmax loss.

(In light of the recent work of Albiero #OTHEREFR , we might now add that differing social conventions for hairstyle present another confounding effect.) In Issues Related to Face Recognition Accuracy Varying Based on Race and Skin Tone, Krishnapriya et al.

#OTHEREFR published the first experiment intended to isolate the effect of skin tone alone on accuracy.

To directly test the premise that face recognition is less accurate for darker skin tones, they examined a #OTHEREFR .) range of tones within the single demographic of African American male (AAM) using the MORPH dataset.

They used two matchers to produce AAM impostor distributions: ArcFace (as described in Section 0.2) and a publicly-available VGGFace2 model #OTHEREFR .

VGGFace2 is representative of the state-of-the-art in CNN matchers prior to ArcFace.

Faces are detected, aligned, and resized to 224×224 pixels, and a 2048-d feature vector is taken from the next-to-last layer.

As with ArcFace, cosine similarity is measured between feature vectors.

compared the frequency of images with darker skin tone in two regions of the AAM impostor distribution.

The high-similarity tail (HST) is the region containing the non-mated image pairs that are most likely to cause false matches.

Exploring Causes Of Demographic Variations In Face Recognition Accuracy

Abstract: FIGURE 1 Apple's Face ID ad campaign touted ease of use and improved security for face recognition over the prior fingerprint standard, and made bold promises. 0.1 Introduction Automated facial recognition (FR) technology dates back to the early 1970s, with Takeo Kanade's 1973 Ph.D. thesis Picture Processing System by Computer Complex and Recognition of Human Faces [1] often cited as an early landmark work. However, it was not until the late 2010s that increased availability and power of FR technology increased its routine usage. In 2017, Apple introduced the iPhone X as the "smartphone industry's benchmark" [2], with their new facial identification system, Face ID, as a primary innovation and selling point [3, 4] (Figure 1). The corresponding security guide [5] even claimed "the probability that a random person in the population could look at your iPhone X and unlock it using Face ID is approximately 1 in 1,000,000 (versus 1 in 50,000 for Touch ID)." Yet a security flaw was quickly publicized: some Chinese users reported that their iPhones opened for other, non-authorized individuals [6, 7]. The underpinnings of these incidents would soon be explored by the research community, leading to a major question: does face recognition perform equally across all demographics? In 2018, Buolamwini and Gebru explored a related question in Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification [8]. Their work evaluated the accuracy of commercial gender classification software. For one classifier, they reported that error rates for lighter-skinned males were only 0.8%, while error rates for darker-skinned females were dramatically higher, at 34.7%. Provocative headlines like "Facial Recognition Is Accurate, if You're a White Guy" [9] sparked public interest in Buolamwini and Gebru's work. Media coverage generally failed to make any distinction between gender classification, the task of assigning a gender label to one face image, and face recognition, the task of deciding whether or not two face images are from the same person. Government research organizations quickly addressed the growing public concern around possible "bias" in face recognition accuracy. As part of their 2018 biometric technology rally, the Department of Homeland Security assessed the effect of demographic factors on performance of commercial face biometric systems, as measured by transaction times and by similarity scores of pairs of images of the same person [10].

Vggface2: A Dataset For Recognising Faces Across Pose And Age

Abstract: In this paper, we introduce a new large-scale face dataset named VGGFace2. The dataset contains 3.31 million images of 9131 subjects, with an average of 362.6 images for each subject. Images are downloaded from Google Image Search and have large variations in pose, age, illumination, ethnicity and profession (e.g. actors, athletes, politicians). The dataset was collected with three goals in mind: (i) to have both a large number of identities and also a large number of images for each identity; (ii) to cover a large range of pose, age and ethnicity; and (iii) to minimize the label noise. We describe how the dataset was collected, in particular the automated and manual filtering stages to ensure a high accuracy for the images of each identity. To assess face recognition performance using the new dataset, we train ResNet-50 (with and without Squeeze-and-Excitation blocks) Convolutional Neural Networks on VGGFace2, on MS- Celeb-1M, and on their union, and show that training on VGGFace2 leads to improved recognition performance over pose and age. Finally, using the models trained on these datasets, we demonstrate state-of-the-art performance on the IJB-A and IJB-B face recognition benchmarks, exceeding the previous state-of-the-art by a large margin. Datasets and models are publicly available.