In terms of MRR and Hits@1, HIE is worse than that of several state-of-the-art models, while it obtains a 30% higher Hits@1 than R-GCN #REFR which uses more additional neighbor information.

Due to the massive inversion relation patterns existing in WN18 where the ratio reaches 94% #OTHEREFR .

To meet the inversion equation, the transformation matrix in HIE will be an identity matrix.

That's to say, more information on distance measurement is omitted.

Thus, the proposed HIE does not achieve best scores of MRR and Hits@1 on WN18.

• On WN18, the proposed HIE outperforms the compared algorithms and achieves the best MR, Hits@3 and Hits@10.

• On YAGO3-10, in terms of MRR, Hits@3 and Hits@10, HIE achieves the best score.

As for the MR and Hits@1, HIE is slightly worse than the state-of-the-art RotatE and InteractE in the corresponding aspects.

Notably, HIE obtains a 7% higher MRR than that of neural networkbased model M-DCN, 59% higher MRR than that of semantic matching-based model DistMult, and 9% higher than MRR that of translation based model RotatE.

Strikingly, our proposed HIE achieves better performance scores on Hits@3 and Hits@10 than that of ConvE, RotatE and DistMult.

Joint Embedding In Hierarchical Distance And Semantic Representation Learning For Link Prediction

Abstract: The link prediction task aims to predict missing links in the knowledge graph and is essential for the downstream application. Existing well-known models deal with this task by mainly focusing on representing knowledge graph triplets in the distance space or semantic space. However, they can not fully capture the information of head and tail entities, nor even make good use of hierarchical level information. Thus, in this paper, we propose a novel knowledge graph embedding model for the link prediction task, namely, HIE, which models each triplet (h, r, t) into distance measurement space and semantic measurement space simultaneously. Moreover, to leverage the hierarchical information of entities and relations, HIE is introduced into hierarchical-aware space for better representation learning. Specifically, we apply distance transformation operation on the head entity in distance space to obtain the tail entity instead of translation-based or rotation-based approaches. Experimental results of HIE on four real-world datasets show that HIE outperforms several existing state-of-the-art knowledge graph embedding methods on link prediction task and deals with complex relations accurately.

Modeling Relational Data With Graph Convolutional Networks

Abstract: Knowledge bases play a crucial role in many applications, for example question answering and information retrieval. Despite the great effort invested in creating and maintaining them, even the largest representatives (e.g., Yago, DBPedia or Wikidata) are highly incomplete. We introduce relational graph convolutional networks (R-GCNs) and apply them to two standard knowledge base completion tasks: link prediction (recovery of missing facts, i.e.~subject-predicate-object triples) and entity classification (recovery of missing attributes of entities). R-GCNs are a generalization of graph convolutional networks, a recent class of neural networks operating on graphs, and are developed specifically to deal with highly multi-relational data, characteristic of realistic knowledge bases. Our methods achieve competitive performance on standard benchmarks for both tasks, demonstrating especially promising results on the challenging FB15k-237 subset of Freebase.


We build COE on commonly used value decomposition methods, and used the hash-based counting technique #REFR to enable counting the visitations in continuous state-action domains.

At each node of the tree, the preceding agent's taken action can be considered as the parent node of the current agent.

Then we can perform optimism-based exploration by computing the upper confidence bounds of each action for the current agent, conditioned on its parent node's visitation count.

We disable exploration after training to obtain decentralized agents for execution.

We first review the basic background in the MARL setting and the UCT algorithm.

Then, we describe how the UCT exploration can be applied to the MARL setting to encourage cooperative exploration.

Our empirical results on various benchmark domains show that our method is more effective than well-known baselines in exploration-challenging tasks, and matches baseline performance in general MARL tasks.

Conditionally Optimistic Exploration For Cooperative Deep Multi-Agent Reinforcement Learning

Abstract: Efficient exploration is critical in cooperative deep Multi-Agent Reinforcement Learning (MARL). In this paper, we propose an exploration method that efficiently encourages cooperative exploration based on the idea of the theoretically justified tree search algorithm UCT (Upper Confidence bounds applied to Trees). The high-level intuition is that to perform optimism-based exploration, agents would achieve cooperative strategies if each agent's optimism estimate captures a structured dependency relationship with other agents. At each node (i.e., action) of the search tree, UCT performs optimismbased exploration using a bonus derived by conditioning on the visitation count of its parent node. We provide a perspective to view MARL as tree search iterations and develop a method called Conditionally Optimistic Exploration (COE). We assume agents take actions following a sequential order, and consider nodes at the same depth of the search tree as actions of one individual agent. COE computes each agent's state-action value estimate with an optimistic bonus derived from the visitation count of the state and joint actions taken by agents up to the current agent. COE is adaptable to any value decomposition method for centralized training with decentralized execution. Experiments across various cooperative MARL benchmarks show that COE outperforms current stateof-the-art exploration methods on hard-exploration tasks. Preprint. Under review.

Exploration: A Study Of Count-Based Exploration For Deep Reinforcement Learning

Abstract: Count-based exploration algorithms are known to perform near-optimally when used in conjunction with tabular reinforcement learning (RL) methods for solving small discrete Markov decision processes (MDPs). It is generally thought that count-based methods cannot be applied in high-dimensional state spaces, since most states will only occur once. ::: Recent deep RL exploration strategies are able to deal with high-dimensional continuous state spaces through complex heuristics, often relying on optimism in the face of uncertainty or intrinsic motivation. ::: ::: In this work, we describe a surprising finding: a simple generalization of the classic count-based approach can reach near state-of-the-art performance on various high-dimensional and/or continuous deep RL benchmarks. States are mapped to hash codes, which allows to count their occurrences with a hash table. These counts are then used to compute a reward bonus according to the classic count-based exploration theory. We find that simple hash functions can achieve surprisingly good results on many challenging tasks. Furthermore, we show that a domain-dependent learned hash code may further improve these results. ::: ::: Detailed analysis reveals important aspects of a good hash function: 1) having appropriate granularity and 2) encoding information relevant to solving the MDP. This exploration strategy achieves near state-of-the-art performance on both continuous control tasks and Atari 2600 games, hence providing a simple yet powerful baseline for solving MDPs that require considerable exploration.

continuous state-action domains
hash-based counting technique


Current workhorses to address transparency concerns in machine learning include black-box explanation and transparent design techniques #REFR .

The ever-increasing integration of machine learning models in high-stakes decision-making contexts (e.g., healthcare, justice, finance) has fostered a growing demand for transparency in recent years.

Black-box explanation techniques aim at explaining complex machine learning models in a post-hoc fashion with global explanations such as Trepan #OTHEREFR and BETA #OTHEREFR or local explanations such as LIME (Ribeiro et al., 2016) and SHAP (Lundberg and Lee, 2017).

On the other hand, transparent design concerns the development of inherently interpretable models such as rule lists #OTHEREFR , rule sets #OTHEREFR , decision trees #OTHEREFR , and scoring systems #OTHEREFR .

However, both black-box explanations and transparent design face performance and trustworthiness challenges that can prevent their wide adoption.

On the one hand, while inherently interpretable models can be more easily understood and adopted by non-domain experts, their out-of-the-box performance can be worst than non-transparent models.

Moreover, training such models to optimality is often NP-hard due to their discrete nature.

Learning Hybrid Interpretable Models: Theory, Taxonomy, And Methods

Abstract: A hybrid model involves the cooperation of an interpretable model and a complex black box. At inference, any input of the hybrid model is assigned to either its interpretable or complex component based on a gating mechanism. The advantages of such models over classical ones are twofold: 1) They grant users precise control over the level of transparency of the system and 2) They can potentially perform better than a standalone black box since redirecting some of the inputs to an interpretable model implicitly acts as regularization. Still, despite their high potential, hybrid models remain under-studied in the interpretability/explainability literature. In this paper, we remedy this fact by presenting a thorough investigation of such models from three perspectives: Theory, Taxonomy, and Methods. First, we explore the theory behind the generalization of hybrid models from the Probably-Approximately-Correct (PAC) perspective. A consequence of our PAC guarantee is the existence of a sweet spot for the optimal transparency of the system. When such a sweet spot is attained, a hybrid model can potentially perform better than a standalone black box. Secondly, we provide a general taxonomy for the different ways of training hybrid models: the Post-Black-Box and Pre-Black-Box paradigms. These approaches differ in the order in which the interpretable and complex components are trained. We show where the state-of-the-art hybrid models Hybrid-Rule-Set and Companion-Rule-List fall in this taxonomy. Thirdly, we implement the two paradigms in a single method: HybridCORELS, which extends the CORELS algorithm to hybrid modeling. By leveraging CORELS, Hy-bridCORELS provides a certificate of optimality of its interpretable component and precise control over transparency. We finally show empirically that HybridCORELS is competitive with existing hybrid models, and performs just as well as a standalone black box (or even better) while being partly transparent.

A Survey Of Methods For Explaining Black Box Models

Abstract: In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.


Noise2Noise #REFR is the first to use pairs of two different noisy images constructed from the same clean image to train a denoising model. Strictly speaking, Noise2Noise is not an unsupervised learning method. Collecting noisy pairs is still difficult.

Image denoising #OTHEREFR has been studied for many years.

Suppose x is a clean image, y is a noisy image of x, and p (y | x) is the noise model.

Supervised learning methods try to train a model representing a mapping from y to x.

Due to the difficulty of collecting paired clean and noisy images in practice, methods in unsupervised learning manner are focus of research.

Despite all this, many other methods #OTHEREFR are inspired from Noise2Noise or borrow the idea behind of it.

These methods achieve good performance in some simple noise models.

The main drawback of current methods is the constraint of application.

Once the noise model is complicated, they either are not applicable or have poor performance.

Nois-ier2Noise #OTHEREFR can only handle additive noise and pepper noise. Recorrupted-to-Recorrupted [17] is limited to Gaussian noise.

Unsupervised Image Denoising With Score Function

Abstract: Though achieving excellent performance in some cases, current unsupervised learning methods for single image denoising usually have constraints in applications. In this paper, we propose a new approach which is more general and applicable to complicated noise models. Utilizing the property of score function, the gradient of logarithmic probability, we define a solving system for denoising. Once the score function of noisy images has been estimated, the denoised result can be obtained through the solving system. Our approach can be applied to multiple noise models, such as the mixture of multiplicative and additive noise combined with structured correlation. Experimental results show that our method is comparable when the noise model is simple, and has good performance in complicated cases where other methods are not applicable or perform poorly.

Noise2Noise: Learning Image Restoration Without Clean Data

Abstract: We apply basic statistical reasoning to signal reconstruction by machine learning -- learning to map corrupted observations to clean signals -- with a simple and powerful conclusion: it is possible to learn to restore images by only looking at corrupted examples, at performance at and sometimes exceeding training using clean data, without explicit image priors or likelihood models of the corruption. In practice, we show that a single model learns photographic noise removal, denoising synthetic Monte Carlo images, and reconstruction of undersampled MRI scans -- all corrupted by different processes -- based on noisy data only.


First, it demonstrates the soundness of the GNN architecture used in the first experiment, which achieves results comparable to those of models carefully optimized on the benchmark datasets #REFR .

Since ASAPool and SAGPool implement an additional MP operation internally, we used only 1 GIN layer before them, rather than 2 as for the other pooling methods.

3 shows the average accuracy and the average run-time obtained on the seven benchmark datasets (the detailed results are in Appendix B.3).

These benchmarks are not designed to test the expressive power and, thus, a GNN equipped with a non-expressive pooling operator could achieve good performance.

For example, this happens in those datasets where all the necessary information is captured by the first two GIN layers that come before pooling or in datasets where only a small part of the graph determines the class. Nevertheless, this second experiment serves two purposes.

Second, and most importantly, it shows that the performances on the benchmark datasets and EXPWL1 are aligned; this underlines the relevance of our theoretical result on the expressivity in practical applications.

It is worth noting that on the benchmark datasets, it was not possible to obtain a pooling ratio of 0.1 for both Graclus and ECPool.

Using a pooling ratio of 0.5 gives Graclus and ECPool an advantage over other methods, which makes the comparison not completely fair and shows an important limitation of these methods.

As a concluding remark, we comment on the training time of the dense and sparse pooling methods.

A popular argument in favor of sparse pooling methods is their computational advantage compared to the dense ones. Our results show that this is not the case.

The Expressive Power Of Pooling In Graph Neural Networks

Abstract: In Graph Neural Networks (GNNs), hierarchical pooling operators generate a coarser representation of the input data by creating local summaries of the graph structure and its vertex features. Considerable attention has been devoted to studying the expressive power of message-passing (MP) layers in GNNs, while a study on how pooling operators affect the expressivity of a GNN is still lacking. Additionally, despite the recent advances in the design of effective pooling operators, there is not a principled criterion to compare them. Our work aims to fill this gap by providing sufficient conditions for a pooling operator to fully preserve the expressive power of the MP layers before it. These conditions serve as a universal and theoretically-grounded criterion for choosing among existing pooling operators or designing new ones. Based on our theoretical findings, we reviewed several existing pooling operators and identified those that fail to satisfy the expressiveness assumptions. Finally, we introduced an experimental setup to empirically measure the expressive power of a GNN equipped with pooling layers, in terms of its capability to perform a graph isomorphism test. * Equal contribution Preprint. Under review.

A Fair Comparison Of Graph Neural Networks For Graph Classification

Abstract: Experimental reproducibility and replicability are critical topics in machine learning. Authors have often raised concerns about their lack in scientific publications to improve the quality of the field. Recently, the graph representation learning field has attracted the attention of a wide research community, which resulted in a large stream of works. As such, several Graph Neural Network models have been developed to effectively tackle graph classification. However, experimental procedures often lack rigorousness and are hardly reproducible. Motivated by this, we provide an overview of common practices that should be avoided to fairly compare with the state of the art. To counter this troubling trend, we ran more than 47000 experiments in a controlled and uniform framework to re-evaluate five popular models across nine common benchmarks. Moreover, by comparing GNNs with structure-agnostic baselines we provide convincing evidence that, on some datasets, structural information has not been exploited yet. We believe that this work can contribute to the development of the graph learning field, by providing a much needed grounding for rigorous evaluations of graph classification models.


Inspired by FixMatch #REFR , a newly recent SOTA algorithm for semi-supervised image classification, we incorporate pseudo-labeling into consistencybased regularization and then can measure this similarity directly with cross-entropy.

They usually have different confidence levels even when predicted to be an identical class.

Hence, we can combinep t andp t−1 using a coefficient κ to form a final perturbed output:

How to Enforce Consistency Another important aspect of consistency-based regularization is to enforce consistency, which encourages a model to have close predictions on an unlabeled sample and its perturbed versions.

Such expected similarity can be computed using some popular distance measures like mean squared error (MSE), Kullback-Leiber (KL) divergence, and Jensen-Shannon (JS) divergence, which are not dependent on any true label.

Pseudo-labeling is not competitive against other modern semi-supervised learning algorithms on its own, but using it as a part of these modern ones could usually produce better results.

Given a signal sample x ∈ U and its corresponding model prediction after being perturbed, i.e.,p, we can convert p into a hard labelŷ (known as "one-hot" encoding), and it is retained only whenp enjoys high confidence:

where c ∈ {1, . . . , C} represents a possible class.

The predicted class probability ofp on c is Ω c (p), and τ is a preset threshold.

Semi-Supervised Rf Fingerprinting With Consistency-Based Regularization

Abstract: As a promising non-password authentication technology, radio frequency (RF) fingerprinting can greatly improve wireless security. Recent work has shown that RF fingerprinting based on deep learning can significantly outperform conventional approaches. The superiority, however, is mainly attributed to supervised learning using a large amount of labeled data, and it significantly degrades if only limited labeled data is available, making many existing algorithms lack practicability. Considering that it is often easier to obtain enough unlabeled data in practice with minimal resources, we leverage deep semi-supervised learning for RF fingerprinting, which largely relies on a composite data augmentation scheme designed for radio signals, combined with two popular techniques: consistency-based regularization and pseudo-labeling. Experimental results on both simulated and realworld datasets demonstrate that our proposed method for semisupervised RF fingerprinting is far superior to other competing ones, and it can achieve remarkable performance almost close to that of fully supervised learning with a very limited number of examples.

Fixmatch: Simplifying Semi-Supervised Learning With Consistency And Confidence

Abstract: Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model's predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 -- just 4 labels per class. Since FixMatch bears many similarities to existing SSL methods that achieve worse performance, we carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch's success. We make our code available at this https URL.


MegaAge-Asian #REFR is newly facial age dataset consisting of 40,000 Asian faces with ages ranging from 0 to 70 years old.

FG-Net #OTHEREFR contains 1002 facial images from 82 subjects, where the age ranges from 0 to 69.

In experiment, we adopt the setup of paper #OTHEREFR , which uses leaveone person-out (LOPO) cross-validation.

Hence, the average performance over 82 splits is reported, which makes the time of each training longer, and the MAEs fluctuate greatly in each split.

IMDB-Wiki #OTHEREFR consists of 523051 facial images of celebrities, crawled from IMDB and Wikipedia, and age ranges from 0 to 100.

Since there is much noise in this dataset, we selected about 300,000 images for training, where all non-face and severely occluded images were removed.

It helps to increase the diversity of human races and improve the generalization ability of the model.

Followed by the setting of #OTHEREFR , 3,945 images were reserved for testing, and the remains are treated as the training set.

This dataset applies the cumulative accuracy (CA) #OTHEREFR as the evaluation metric.

We take the Mean Absolute Error (MAE) in the experiments to calculate the discrepancy between the estimated age and the ground truth.

Daa: A Delta Age Adain Operation For Age Estimation Via Binary Code Transformer

Abstract: Naked eye recognition of age is usually based on comparison with the age of others. However, this idea is ignored by computer tasks because it is difficult to obtain representative contrast images of each age. Inspired by the transfer learning, we designed the Delta Age AdaIN (DAA) operation to obtain the feature difference with each age, which obtains the style map of each age through the learned values representing the mean and standard deviation. We let the input of transfer learning as the binary code of age natural number to obtain continuous age feature information. The learned two groups of values in Binary code mapping are corresponding to the mean and standard deviation of the comparison ages. In summary, our method consists of four parts: FaceEncoder, DAA operation, Binary code mapping, and AgeDecoder modules. After getting the delta age via AgeDecoder, we take the average value of all comparison ages and delta ages as the predicted age. Compared with state-of-the-art methods, our method achieves better performance with fewer parameters on multiple facial age datasets.

Quantifying Facial Age By Posterior Of Age Comparisons

Abstract: We introduce a novel approach for annotating large quantity of in-the-wild facial images with high-quality posterior age distribution as labels. Each posterior provides a probability distribution of estimated ages for a face. Our approach is motivated by observations that it is easier to distinguish who is the older of two people than to determine the person's actual age. Given a reference database with samples of known ages and a dataset to label, we can transfer reliable annotations from the former to the latter via human-in-the-loop comparisons. We show an effective way to transform such comparisons to posterior via fully-connected and SoftMax layers, so as to permit end-to-end training in a deep network. Thanks to the efficient and effective annotation approach, we collect a new large-scale facial age dataset, dubbed `MegaAge', which consists of 41,941 images. Data can be downloaded from our project page mmlab.ie.cuhk.edu.hk/projects/MegaAge and github.com/zyx2012/Age_estimation_BMVC2017. With the dataset, we train a network that jointly performs ordinal hyperplane classification and posterior distribution learning. Our approach achieves state-of-the-art results on popular benchmarks such as MORPH2, Adience, and the newly proposed MegaAge.


A GAN #REFR is a framework that trains a generator G and discriminator D in a two-player min-max game.

In the context of speech synthesis, a generator is identical to a speech synthesizer that synthesizes a speech waveform x from the input data s, where s indicates the intermediate representation (e.g., mel spectrogram) in a twostage approach and text in an end-to-end TTS system.

In GAN-based speech synthesis, the generator and discriminator are trained with two GAN-related losses: adversarial and feature-matching losses.

Wave-U-Net Discriminator: Fast And Lightweight Discriminator For Generative Adversarial Network-Based Speech Synthesis

Abstract: In speech synthesis, a generative adversarial network (GAN), training a generator (speech synthesizer) and a discriminator in a minmax game, is widely used to improve speech quality. An ensemble of discriminators is commonly used in recent neural vocoders (e.g., HiFi-GAN) and end-to-end text-to-speech (TTS) systems (e.g., VITS) to scrutinize waveforms from multiple perspectives. Such discriminators allow synthesized speech to adequately approach real speech; however, they require an increase in the model size and computation time according to the increase in the number of discriminators. Alternatively, this study proposes a Wave-U-Net discriminator, which is a single but expressive discriminator with Wave-U-Net architecture. This discriminator is unique; it can assess a waveform in a sample-wise manner with the same resolution as the input signal, while extracting multilevel features via an encoder and decoder with skip connections. This architecture provides a generator with sufficiently rich information for the synthesized speech to be closely matched to the real speech. During the experiments, the proposed ideas were applied to a representative neural vocoder (HiFi-GAN) and an end-to-end TTS system (VITS). The results demonstrate that the proposed models can achieve comparable speech quality with a 2.31 times faster and 14.5 times more lightweight discriminator when used in HiFi-GAN and a 1.90 times faster and 9.62 times more lightweight discriminator when used in VITS. 1

Generative Adversarial Nets

Abstract: We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to ½ everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.


Finally, we also utilize an advanced model, BERT-base #REFR , to extract word representations by averaging the outputs of the last layer.

We adopt fastText #OTHEREFR as our default gloss feature extractor.

Here we investigate other alternatives as shown in Table 15 .

Word2vec #OTHEREFR and GloVe #OTHEREFR are widely-adopted in NLP community.

They perform comparably to each other that GloVe achieves better results on the top-1 accuracy while word2vec is superior regarding to the top-5 accuracy.

As an improvement of word2vec, fastText leads to better results on all of the four metrics.

However, it performs worse than all the other methods since it is not dedicated to word representation learning.

Abstract: Sign languages are visual languages which convey information by signers' handshape, facial expression, body movement, and so forth. Due to the inherent restriction of combinations of these visual ingredients, there exist a significant number of visually indistinguishable signs (VISigns) in sign languages, which limits the recognition capacity of vision neural networks. To mitigate the problem, we propose the Natural Language-Assisted Sign Language Recognition (NLA-SLR) framework, which exploits semantic information contained in glosses (sign labels). First, for VISigns with similar semantic meanings, we propose language-aware label smoothing by generating soft labels for each training sign whose smoothing weights are computed from the normalized semantic similarities among the glosses to ease training. Second, for VISigns with distinct semantic meanings, we present an inter-modality mixup technique which blends vision and gloss features to further maximize the separability of different signs under the supervision of blended labels. Besides, we also introduce a novel backbone, video-keypoint network, which not only models both RGB videos and human body keypoints but also derives knowledge from sign videos of different temporal receptive fields. Empirically, our method achieves state-ofthe-art performance on three widely-adopted benchmarks: MSASL, WLASL, and NMFs-CSL. Codes are available at https://github.com/FangyunWei/SLRT.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


Iii. A New Randomized Algorithm For Frameproof

Our main tool is the celebrated algorithmic version of the Lovász Local Lemma for the symmetric case as given in #REFR .

We first need to recall the following well-known facts on binomial coefficients, where for positive integers c ≤ b ≤ a we have

and the following useful inequality, proved in #OTHEREFR , for positive integers

, E m be events in a probability space, where each event E i is mutually independent of the set of all the other events E j except for at most D, and

Bounds And Algorithms For Frameproof Codes And Related Combinatorial Structures

Abstract: In this paper, we study upper bounds on the minimum length of frameproof codes introduced by Boneh and Shaw [3] to protect copyrighted materials. A q-ary (k, n)-frameproof code of length t is a t×n matrix having entries in {0, 1,. .. , q−1} and with the property that for any column c and any other k columns, there exists a row where the symbols of the k columns are all different from the corresponding symbol (in the same row) of the column c. In this paper, we show the existence of q-ary (k, n)-frameproof codes of length t = O(k 2 q log n) for q ≤ k, using the Lovász Local Lemma, and of length t = O(k log(q/k) log(n/k)) for q > k using the expurgation method. Remarkably, for the practical case of q ≤ k our findings give codes whose length almost matches the lower bound Ω(k 2 q log k log n) on the length of any q-ary (k, n)-frameproof code and, more importantly, allow us to derive an algorithm of complexity O(tn 2) for the construction of such codes.

A Constructive Proof Of The General Lovász Local Lemma

Abstract: The Lovasz Local Lemma [EL75] is a powerful tool to non-constructively prove the existence of combinatorial objects meeting a prescribed collection of criteria. In his breakthrough paper [Bec91], Beck demonstrated that a constructive variant can be given under certain more restrictive conditions. Simplifications of his procedure and relaxations of its restrictions were subsequently exhibited in several publications [Alo91, MR98, CS00, Mos06, Sri08, Mos08]. In [Mos09], a constructive proof was presented that works under negligible restrictions, formulated in terms of the Bounded Occurrence Satisfiability problem. In the present paper, we reformulate and improve upon these findings so as to directly apply to almost all known applications of the general Local Lemma.

celebrated algorithmic version
Lovász Local Lemma


E. Detailed Results And More Architectures

The comparison results on WideResNet-28 have the same trend as that on ResNet-18 #REFR architecture.

Tab. 7 and Tab. 9 show the detailed results among all datasets.

Our method obtains consistently better results across all OOD detection metrics and all datasets.

Compared with other methods using extra OOD training data (MCD [32], OE #OTHEREFR , UDG #OTHEREFR ), our method boosts the OOD detection performance meanwhile maximally maintaining the ID classification performance and achieves the best results on ACC.

Following #OTHEREFR we also adopt another network architecture of WideResNet-28 #OTHEREFR to do experiments in Tab. 8 and Tab. 10 and compare the performance.

Our proposed method combining ET with L rep has advantages on almost all metrics, showing that our method enhances both the OOD detection and the ID classification ability.

Notably, the previous state-of-the-art approaches generally performed well on SVHN and Texture datasets, but in Tiny-ImageNet, LSUN, and Places365 suffered a defeat.

It can be explained that the images in the first two datasets have relatively flat backgrounds, which are quite different in style from those in CIFAR10/100, and the resulting covariate shifts make it easier for the model to identify them as OOD examples.

Our proposed method especially achieves performance advantages on Tiny-ImageNet, LSUN, and Places365, but its performance on CIFAR-100/10 (the styles between them are very similar) of the CIFAR-10/100 benchmark is still not outstanding.

This indicates that more exploration is needed to overcome the interference of covariate shifts in OOD detection tasks. Table 7 . Detailed results on CIFAR-10 benchmark using ResNet-18.

Uncertainty-Aware Optimal Transport For Semantically Coherent Out-Of-Distribution Detection

Abstract: Semantically coherent out-of-distribution (SCOOD) detection aims to discern outliers from the intended data distribution with access to unlabeled extra set. The coexistence of in-distribution and out-of-distribution samples will exacerbate the model overfitting when no distinction is made. To address this problem, we propose a novel uncertainty-aware optimal transport scheme. Our scheme consists of an energy-based transport (ET) mechanism that estimates the fluctuating cost of uncertainty to promote the assignment of semantic-agnostic representation, and an inter-cluster extension strategy that enhances the discrimination of semantic property among different clusters by widening the corresponding margin distance. Furthermore, a T-energy score is presented to mitigate the magnitude gap between the parallel transport and classifier branches. Extensive experiments on two standard SCOOD benchmarks demonstrate the above-par OOD detection performance, outperforming the state-of-the-art methods by a margin of 27.69% and 34.4% on FPR@95, respectively. Code is available at https://github.com/LuFan31/ET-OOD. * Co-first Author. †Corresponding Author. ‡Work done during an internship at Ant Group.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


The motivation for this paper is the work in #REFR which proposes a matrix completion approach for the Euclidean distance geometry problem (abbreviated EDG hereafter).

The problem is concerned with finding the configuration of points given partial information in terms of pairwise inter-point distances; it is of fundamental importance in many applications #OTHEREFR .

Consider a set of n points P = {p 1 , p 2 , ..., p n } T ∈ R n×r .

Let D ∈ R n×n denote the squared distance matrix with entries D i,j , and let Ω ⊂ {(i, j) | i, j = 1, ..., n, i < j}, |Ω| = m, be the set that consists of all the indices for which we know the corresponding entries of the distance matrix.

Given k > 0 and partial exact distance information, the EDG problem is to find a configuration of points

P = {p 1 , p 2 , ..., p n } T ∈ R n×k such that ||p i − p j || 2 2 = D i,j for (i, j) ∈ Ω.

A Dual Basis Approach To Multidimensional Scaling: Spectral Analysis And Graph Regularity

Abstract: Classical multidimensional scaling (CMDS) is a technique that aims to embed a set of objects in a Euclidean space given their pairwise Euclidean distance matrix. The main part of CMDS is based on double centering a squared distance matrix and employing a truncated eigendecomposition to recover the point coordinates. A central result in CMDS connects the squared Euclidean matrix to a Gram matrix derived from the set of points. In this paper, we study a dual basis approach to classical multidimensional scaling. We give an explicit formula for the dual basis and fully characterize the spectrum of an essential matrix in the dual basis framework. We make connections to a related problem in metric nearness.

Exact Reconstruction Of Euclidean Distance Geometry Problem Using Low-Rank Matrix Completion

Abstract: The Euclidean distance geometry problem arises in a wide variety of applications, from determining molecular conformations in computational chemistry to localization in sensor networks. When the distance information is incomplete, the problem can be formulated as a nuclear norm minimization problem. In this paper, this minimization program is recast as a matrix completion problem of a low-rank $r$ Gram matrix with respect to a suitable basis. The well-known restricted isometry property cannot be satisfied in this scenario. Instead, a dual basis approach is introduced to theoretically analyze the reconstruction problem. If the Gram matrix satisfies certain coherence conditions with parameter $\nu $ , the main result shows that the underlying configuration of $n$ points can be recovered with very high probability from $O(nr\nu \log ^{2}(n))$ uniformly random samples. Computationally, simple and fast algorithms are designed to solve the Euclidean distance geometry problem. Numerical tests on different 3-D data and protein molecules validate the effectiveness and efficiency of the proposed algorithms.


Generating Differentially Private Synthetic Data

In #REFR , the authors adopted a so-called variational autoencoder, DP-VAE, which assumes that the mapping from real data to the Gaussian distribution can be efficiently learned.

Although DP-based generative Artificial Neural Networks (ANNs) have been quite extensively researched in the major field of data science and machine learning, they have not been used in the context of process mining.

Thus, we mainly focus on some of the work outside the domain of process mining.

A different direction was then chosen by #OTHEREFR , where the authors used a Wasserstein GAN (WGAN) to generate differentially private mixed-type synthetic outputs employing a Wasserstein-distance-based loss function.

Finally, in #OTHEREFR , the concepts of WGAN and DP-VAE were combined to first learn a private data encoding and then generate respective encoded data.

We adapted this principle for our work to cope with the large dimensionality of event data.

Research in non-private generative models for process mining, primarily focuses on exploiting ANNs and GANs to predict the next state of processes such as #OTHEREFR , and #OTHEREFR .

Note that the approach in #OTHEREFR only provides synthetic event data without any privacy guarantees.

Travag: Differentially Private Trace Variant Generation Using Gans

Abstract: Process mining is rapidly growing in the industry. Consequently, privacy concerns regarding sensitive and private information included in event data, used by process mining algorithms, are becoming increasingly relevant. State-of-the-art research mainly focuses on providing privacy guarantees, e.g., differential privacy, for trace variants that are used by the main process mining techniques, e.g., process discovery. However, privacy preservation techniques for releasing trace variants still do not fulfill all the requirements of industry-scale usage. Moreover, providing privacy guarantees when there exists a high rate of infrequent trace variants is still a challenge. In this paper, we introduce TraVaG as a new approach for releasing differentially private trace variants based on Generative Adversarial Networks (GANs) that provides industry-scale benefits and enhances the level of privacy guarantees when there exists a high ratio of infrequent variants. Moreover, TraVaG overcomes shortcomings of conventional privacy preservation techniques such as bounding the length of variants and introducing fake variants. Experimental results on real-life event data show that our approach outperforms stateof-the-art techniques in terms of privacy guarantees, plain data utility preservation, and result utility preservation.

Differentially Private Data Generative Models

Abstract: Deep neural networks (DNNs) have recently been widely adopted in various applications, and such success is largely due to a combination of algorithmic breakthroughs, computation resource improvements, and access to a large amount of data. However, the large-scale data collections required for deep learning often contain sensitive information, therefore raising many privacy concerns. Prior research has shown several successful attacks in inferring sensitive training data information, such as model inversion, membership inference, and generative adversarial networks (GAN) based leakage attacks against collaborative deep learning. In this paper, to enable learning efficiency as well as to generate data with privacy guarantees and high utility, we propose a differentially private autoencoder-based generative model (DP-AuGM) and a differentially private variational autoencoder-based generative model (DP-VaeGM). We evaluate the robustness of two proposed models. We show that DP-AuGM can effectively defend against the model inversion, membership inference, and GAN-based attacks. We also show that DP-VaeGM is robust against the membership inference attack. We conjecture that the key to defend against the model inversion and GAN-based attacks is not due to differential privacy but the perturbation of training data. Finally, we demonstrate that both DP-AuGM and DP-VaeGM can be easily integrated with real-world machine learning applications, such as machine learning as a service and federated learning, which are otherwise threatened by the membership inference attack and the GAN-based attack, respectively.


Our observations reveal that all the models exhibit high confidence levels in their predictions, this is consistent with previous research on large models #REFR .

In this subsection, we first investigate the level of confidence for both the correct and incorrect samples.

Confidence is typically described in terms of a probability value, indicating the likelihood of belonging to a specific category.

To obtain prediction probabilities from ChatGPT, we ask it to output the probability (Confidence_Standard and Confidence_Open), as discussed in § 3.3.

Our aim is to investigate whether ChatGPT can provide a reasonable prediction confidence scores for its predictions, thus reducing the risk of misinterpretation.

In Table 6 , we present the confidence scores of correct and incorrect predictions from different models, referred to correct confidence and incorrect confidence, respectively.

Although ChatGPT performs worse than its BERT-based counterparts in Standard-IE setting, it displays overconfidence in both correct and incorrect predictions. Consequently, this overconfidence could lead to misguidance of users.

Furthermore, we note a significant confidence gap between correct and incorrect predictions, indicating the need for careful evaluation when ChatGPT's prediction has relatively low confidence.

We then focus on calibration, a critical property of LLMs as it could estimate the predictive uncertainty for the secure application of LLMs.

A well-calibrated model not only produces accurate predictions but also provides reliable and informative uncertainty estimates, necessary for sound decision-making.

In this research, we evaluate the calibration using the Expected Calibration Error (ECE) metric which measures the deviation be- tween predicted confidence and accuracy.

Evaluating Chatgpt'S Information Extraction Capabilities: An Assessment Of Performance, Explainability, Calibration, And Faithfulness

Abstract: The capability of Large Language Models (LLMs) like ChatGPT to comprehend user intent and provide reasonable responses has made them extremely popular lately. In this paper, we focus on assessing the overall ability of ChatGPT using 7 fine-grained information extraction (IE) tasks. Specially, we present the systematically analysis by measuring ChatGPT's performance, explainability, calibration, and faithfulness, and resulting in 15 keys from either the ChatGPT or domain experts. Our findings reveal that Chat-GPT's performance in Standard-IE setting is poor, but it surprisingly exhibits excellent performance in the OpenIE setting, as evidenced by human evaluation. In addition, our research indicates that ChatGPT provides high-quality and trustworthy explanations for its decisions. However, there is an issue of ChatGPT being overconfident in its predictions, which resulting in low calibration. Furthermore, ChatGPT demonstrates a high level of faithfulness to the original text in the majority of cases. We manually annotate and release the test sets of 7 finegrained IE tasks contains 14 datasets to further promote the research. The datasets and code are available at this url. 1

On Calibration Of Modern Neural Networks

Abstract: Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.


Though most results match previous work, we observe a significant improvement for the accuracy achieved by the SET algorithm compared to the implementation done in #REFR .

Instead, we aim to provide some baseline results using different training recipes and architectures.

For all experiments, we use the default hyper-parameters provided by the Scenic library.

ResNet-50 is a popular architecture in sparsity literature #OTHEREFR .

We train 80% sparse ResNet-50 models on ImageNet to reproduce previous results reported in the literature #OTHEREFR .

We use uniform sparsity across layers and leave the first convolutional layer dense as recommended by #OTHEREFR .

We use ERK sparsity distribution #OTHEREFR in our ViT experiments.

Sparse vision transformers achieve better generalization even for the shorter, 90 epoch, training runs (ViT-B16).

STE obtains the best results and exceeds the baseline performance by 0.2%.

Interestingly, when we increase the number of training epochs to 300 (ViT-B16+), this gap widens and sparse ViT-B/16 trained with STE obtains 1.2% higher accuracy, despite having worse (higher) training loss.

Dynamic sparse training methods (RigL and SET) performs poorly in shorter training runs, however with extended training, they achieve almost 5% higher accuracy and exceed the dense baseline.

A Concise Library For Sparsity Research

Abstract: This paper introduces JaxPruner, an open-source JAX-based pruning and sparse training library for machine learning research. JaxPruner aims to accelerate research on sparse neural networks by providing concise implementations of popular pruning and sparse training algorithms with minimal memory and latency overhead. Algorithms implemented in JaxPruner use a common API and work seamlessly with the popular optimization library Optax, which, in turn, enables easy integration with existing JAX based libraries. We demonstrate this ease of integration by providing examples in four different codebases: Scenic, t5x, Dopamine and FedJAX and provide baseline experiments on popular benchmarks. Jaxpruner is hosted at github.com/google-research/jaxpruner.

Rigging The Lottery: Making All Tickets Winners

Abstract: Sparse neural networks have been shown to be more parameter and compute efficient compared to dense networks and in some cases are used to decrease wall clock inference times. There is a large body of work on training dense networks to yield sparse networks for inference. This limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-to-sparse training methods. Our method updates the topology of the network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires fewer floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. Importantly, by adjusting the topology it can start from any initialization - not just "lucky" ones. We demonstrate state-of-the-art sparse training results with ResNet-50, MobileNet v1 and MobileNet v2 on the ImageNet-2012 dataset, WideResNets on the CIFAR-10 dataset and RNNs on the WikiText-103 dataset. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static.


An error occurs if the active user set fails at least one of the tests in #REFR or an incorrect set passes all the tests.

Then, we perform 2 k − 1 threshold tests as below.

for predetermined thresholds η s where s := |I 0 | ∈ {1, 2, . . . , k}.

We declareÃ as the active user set if it is the only set passing all threshold tests.

As shown in #OTHEREFR , using a set of thresholds given by η s = log

( ) e as P ( ) e ≤ k s=1 k s P log p z | x(I 0 ), x(I 1 ) p (z | x(I 1 )) ≤ log ρ k k s −k s + ρ.

Since preamble design uses i.i.d Bernoulli random variables, using concentration inequalities for the k = Θ(1) regime, we can achieve P

Active User Identification In Fast Fading Massive Random Access Channels

Abstract: Reliable and prompt identification of active users is critical for enabling random access in massive machine-tomachine type networks which typically operate within stringent access delay and energy constraints. In this paper, an energy efficient active user identification protocol is envisioned in which the active users simultaneously transmit On-Off Keying (OOK) modulated preambles whereas the base station uses non-coherent detection to avoid the channel estimation overheads. The minimum number of channel-uses required for active user identification in the asymptotic regime of total number of users when the number of active devices k scales as k = Θ(1) is characterized along with an achievability scheme relying on the equivalence of activity detection to a group testing problem. A practical scheme for active user identification based on a belief propagation strategy is also proposed and its performance is compared against the theoretical bounds.

Multi-Cell Sparse Activity Detection For Massive Random Access: Massive Mimo Versus Cooperative Mimo

Abstract: This paper considers sparse device activity detection for cellular machine-type communications with non-orthogonal signatures using the approximate message passing algorithm. This paper compares two network architectures, massive multiple-input-multiple-output (MIMO) and cooperative MIMO, in terms of their effectiveness in overcoming inter-cell interference. In the massive MIMO architecture, each base station (BS) detects only the users from its own cell while treating inter-cell interference as noise. In the cooperative MIMO architecture, each BS detects the users from neighboring cells as well; the detection results are then forwarded in the form of a log-likelihood ratio (LLR) to a central unit where final decisions are made. This paper analytically characterizes the probabilities of false alarm and missed detection for both architectures. The numerical results validate the analytic characterization and show that as the number of antennas increases, a massive MIMO system effectively drives the detection error to zero, while as the cooperation size increases, the cooperative MIMO architecture mainly improves the cell-edge user performance. Moreover, this paper studies the effect of LLR quantization to account for the finite-capacity fronthaul. The numerical simulations of a practical scenario suggest that in specific case cooperating three BSs in a cooperative MIMO system achieves about the same cell-edge detection reliability as a non-cooperative massive MIMO system with four times the number of antennas per BS.


One of the BERT variants that is suitable for sentence embedding is Sentence BERT #REFR .

Many research focuses on generating high-resolution images and semantically correct with the given text description.

But research on the text encoder and other languages is necessary.

CNN-LSTM has become popular text embedding since Reed et al. 2016 use that method.

2018 improve the text encoder with word-level attention to improve the generated image details. And then Tsue et al. 2020 use BERT as text encoder and word embedding.

This variant uses a siamese architecture network to produce text embedding that can then be compared with cosine similarity to find sentence with similar meanings.

Indonesian Text-To-Image Synthesis With Sentence-Bert And Fastgan

Abstract: Currently, text-to-image synthesis uses text encoder and image generator architecture. Research on this topic is challenging. This is because of the domain gap between natural language and vision. Nowadays, most research on this topic only focuses on producing a photo-realistic image, but the other domain, in this case, is the language, which is less concentrated. A lot of the current research uses English as the input text. Besides, there are many languages around the world. Bahasa Indonesia, as the official language of Indonesia, is quite popular. This language has been taught in Philipines, Australia, and Japan. Translating or recreating a new dataset into another language with good quality will cost a lot. Research on this domain is necessary because we need to examine how the image generator performs in other languages besides generating photo-realistic images. To achieve this, we translate the CUB dataset into Bahasa using google translate and manually by humans. We use Sentence BERT as the text encoder and FastGAN as the image generator. FastGAN uses lots of skip excitation modules and auto-encoder to generate an image with resolution 512 × 512 × 3, which is twice as bigger as the current state-of-the-art model (Zhang, Xu, Li, Zhang, Wang, Huang and Metaxas, 2019). We also get 4.76 ± 0.43 and 46.401 on Inception Score and Fréchet inception distance, respectively, and comparable with the current English text-to-image generation models. The mean opinion score also gives as 3.22 out of 5, which means the generated image is acceptable by humans. Link to source code: https://github.com/share424/Indonesian-Text-to-Imagesynthesis-with-Sentence-BERT-and-FastGAN

Sentence-Bert: Sentence Embeddings Using Siamese Bert-Networks

Abstract: BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. ::: In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. ::: We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.


The second baseline, Selective Simulation #REFR (SELSIM), is a search-based algorithm that interleaves a 'planning' phase and a 'tracking' phase.

We assume perfect knowledge of the initial workspace state and all object parameters.

We set a planning timeout of 120 s for 100 randomly generated MAMO problems at each level.

Our analysis includes two versions of our algorithm -M4M refers to Algorithm 1, and M4M refers to a version which only calls CBS once (after Line 5) and does not iterate between calling CBS and finding a valid push in simulation.

Baselines: We compare the performance of M4M against three types of baselines for solving MAMO problems with non-prehensile interactions.

The first are standard implementations of sampling-based algorithms KPIECE #OTHEREFR and RRT #OTHEREFR from OMPL #OTHEREFR that search the entire MAMO state space X by randomly sampling robot motions.

The former queries the physicsbased simulator for interactions with a set of 'relevant' movable objects identified so far.

The latter executes the solution found by the planning phase in the presence of all objects in simulation and, if any interaction constraints are violated, it adds the 'relevant' object to the set.

It only uses the simple motion primitives described in Section III.

Our final baseline is the work from Dogar et al.

#OTHEREFR (DOGAR) which introduced the idea of a negative goal region (NGR) we use in M4M.

Planning For Complex Non-Prehensile Manipulation Among Movable Objects By Interleaving Multi-Agent Pathfinding And Physics-Based Simulation

Abstract: Real-world manipulation problems in heavy clutter require robots to reason about potential contacts with objects in the environment. We focus on pick-and-place style tasks to retrieve a target object from a shelf where some 'movable' objects must be rearranged in order to solve the task. In particular, our motivation is to allow the robot to reason over and consider non-prehensile rearrangement actions that lead to complex robot-object and object-object interactions where multiple objects might be moved by the robot simultaneously, and objects might tilt, lean on each other, or topple. To support this, we query a physics-based simulator to forward simulate these interaction dynamics which makes action evaluation during planning computationally very expensive. To make the planner tractable, we establish a connection between the domain of Manipulation Among Movable Objects and Multi-Agent Pathfinding that lets us decompose the problem into two phases our M4M algorithm iterates over. First we solve a multiagent planning problem that reasons about the configurations of movable objects but does not forward simulate a physics model. Next, an arm motion planning problem is solved that uses a physics-based simulator but does not search over possible configurations of movable objects. We run simulated and realworld experiments with the PR2 robot and compare against relevant baseline algorithms. Our results highlight that M4M generates complex 3D interactions, and solves at least twice as many problems as the baselines with competitive performance.

Planning With Selective Physics-Based Simulation For Manipulation Among Movable Objects

Abstract: Use of physics-based simulation as a planning model enables a planner to reason and generate plans that involve non-trivial interactions with the world. For example, grasping a milk container out of a cluttered refrigerator may involve moving a robot manipulator in between other objects, pushing away the ones that are movable and avoiding interactions with certain fragile containers. A physics-based simulator allows a planner to reason about the effects of interactions with these objects and to generate a plan that grasps the milk container successfully. The use of physics-based simulation for planning however is underutilized. One of the reasons for it being that physics-based simulations are typically way too slow for being used within a planning loop that typically requires tens of thousands of actions to be evaluated within a matter of a second or two. In this work, we develop a planning algorithm that tries to address this challenge. In particular, it builds on the observation that only a small number of actions actually need to be simulated using physics, and the remaining set of actions, such as moving an arm around obstacles, can be evaluated using a much simpler internal planning model, e.g., a simple collision-checking model. Motivated by this, we develop an algorithm called Planning with Selective Physics-based Simulation that automatically discovers what should be simulated with physics and what can utilize an internal planning model for pick-and-place tasks.


The reconstructed errors on local anomalies are strengthened for anomaly detection, which is similar to #REFR .

With the local sensitive VQVAE, the image patches are reconstructed from a learned codebook of the normal data.

The reconstruction thus tends to be close to a normal sample.

Nevertheless, this separate model performs poorly at some points: (1) anomalies on global positions rather than on local textures, and (2) reconstructed anomalies due to the strong generalization of the VQVAE.

Therefore, we employ a full-attention transformer to model the global information of images and rectify abnormal patches based on the prior distribution, which is referred to as a global sensitive transformer as shown in Fig. 2 (right) .

Specifically, By the learned VQVAE, an image X is represented with a corresponding discrete encoding sequence

Unsupervised Anomaly Detection With Local-Sensitive Vqvae And Global-Sensitive Transformers

Abstract: Unsupervised anomaly detection (UAD) has been widely implemented in industrial and medical applications, which reduces the cost of manual annotation and improves efficiency in disease diagnosis. Recently, deep auto-encoder with its variants has demonstrated its advantages in many UAD scenarios. Training on the normal data, these models are expected to locate anomalies by producing higher reconstruction error for the abnormal areas than the normal ones. However, this assumption does not always hold because of the uncontrollable generalization capability. To solve this problem, we present LSGS, a method that builds on Vector Quantised-Variational Autoencoder (VQVAE) with a novel aggregated codebook and transformers with global attention. In this work, the VQVAE focus on feature extraction and reconstruction of images, and the transformers fit the manifold and locate anomalies in the latent space. Then, leveraging the generated encoding sequences that conform to a normal distribution, we can reconstruct a more accurate image for locating the anomalies. Experiments on various datasets demonstrate the effectiveness of the proposed method.

Memorizing Normality To Detect Anomaly: Memory-Augmented Deep Autoencoder For Unsupervised Anomaly Detection

Abstract: Deep autoencoder has been extensively used for anomaly detection. Training on the normal data, the autoencoder is expected to produce higher reconstruction error for the abnormal inputs than the normal ones, which is adopted as a criterion for identifying anomalies. However, this assumption does not always hold in practice. It has been observed that sometimes the autoencoder "generalizes" so well that it can also reconstruct anomalies well, leading to the miss detection of anomalies. To mitigate this drawback for autoencoder based anomaly detector, we propose to augment the autoencoder with a memory module and develop an improved autoencoder called memory-augmented autoencoder, i.e. MemAE. Given an input, MemAE firstly obtains the encoding from the encoder and then uses it as a query to retrieve the most relevant memory items for reconstruction. At the training stage, the memory contents are updated and are encouraged to represent the prototypical elements of the normal data. At the test stage, the learned memory will be fixed, and the reconstruction is obtained from a few selected memory records of the normal data. The reconstruction will thus tend to be close to a normal sample. Thus the reconstructed errors on anomalies will be strengthened for anomaly detection. MemAE is free of assumptions on the data type and thus general to be applied to different tasks. Experiments on various datasets prove the excellent generalization and high effectiveness of the proposed MemAE.


Our results for the weighted LIKE and weighted Balanced LIKE mechanisms match those obtained by Aleksandrov et al. #REFR perfectly.

These mechanisms have been developed as a first attempt to expand and adapt the online allocation with equal entitlements problem to a more general weighted condition.

Moreover, we have proven that the proposed algorithms adhere to fundamental principles such as Strategyproofness and Envy-freeness.

The results of this study provide a starting point for future research in the area of online fair allocation with weighted entitlements.

In particular, our future work will focus on improving the performance of the proposed mechanisms and extending them to more complex settings with additional constraints and preferences.

Overall, the proposed mechanisms can be valuable tools for ensuring fairness in various online allocation scenarios, such as resource allocation, job assignments, and scheduling.

Despite the fact that the outcomes are the same, our project's proof methodology differs from that of Aleksandrov et al.

The authors of the Aleksandrov et al study only offer proof sketches for the theorems pertaining to strategy-proofness and envy-freeness.

All of the proofs we did for strategy-proofness and envy-freeness would also apply in their context because our work is an extension of their study.

We hope that our project will make it easier for readers to grasp how the theorems in #OTHEREFR are proved.

Online Fair Division With Arbitrary Entitlements

Abstract: The division of goods in the online realm poses opportunities and challenges. While innovative mechanisms can be developed, uncertainty about the future may hinder effective solutions. This project aims to explore fair distribution models for goods among agents with arbitrary entitlements, specifically addressing food charity challenges in the real world. Building upon prior work in [AAGW15], which focuses on equal entitlements, our project seeks to better understand the proofs of the theorems mentioned in that paper, which currently only provide proof sketches. Our approach employs different proof techniques from those presented in [AAGW15].

Online Fair Division: Analysing A Food Bank Problem

Abstract: We study an online model of fair division designed to capture features of a real world charity problem. We consider two simple mechanisms for this model in which agents simply declare what items they like. We analyse axiomatic properties of these mechanisms such as strategy-proofness and envy freeness. Finally, we perform a competitive analysis and compute the price of anarchy.


Therefore, in order to save computational costs while retaining important information about the domain, we use a simple and efficient aggregator, the average aggregator in GraphSAGE #REFR , for message aggregation.

Previous works tend to use attention mechanisms #OTHEREFR or design weighting parameters #OTHEREFR for message aggregation.

This is the current work done by many popular GNN models who improve on the message passing and aggregation mechanisms to enhance GNN performance.

However, assuming that we have selected the most similar neighbors for each node, the attention coefficient or weighting parameter has no obvious meaning in neighbor information aggregation.

On the contrary, it will increase the computational cost significantly.

Compared with the attention mechanism, this method significantly reduces the model complexity. The formal description is as follows:

h (l) v = σ W • AGG h (l−1) v ∪ h (l−1) v , ∀v ∈ N (v) (12)

where σ is the activation function, W is the weight to be learned, and N (v) is the neighborhood of node v.

In implementation , we use M EAN () as aggregation function AGG().

M EAN () denotes the operation of averaging the embeddings of the nodes in the set.

Cost Sensitive Gnn-Based Imbalanced Learning For Mobile Social Network Fraud Detection

Abstract: With the rapid development of mobile networks, the people's social contacts have been considerably facilitated. However, the rise of mobile social network fraud upon those networks, has caused a great deal of distress, in case of depleting personal and social wealth, then potentially doing significant economic harm. To detect fraudulent users, call detail record (CDR) data, which portrays the social behavior of users in mobile networks, has been widely utilized. But the imbalance problem in the aforementioned data, which could severely hinder the effectiveness of fraud detectors based on graph neural networks(GNN), has hardly been addressed in previous work. In this paper, we are going to present a novel Cost-Sensitive Graph Neural Network (CS-GNN) by creatively combining cost-sensitive learning and graph neural networks. We conduct extensive experiments on two open-source realworld mobile network fraud datasets. The results show that CSGNN can effectively solve the graph imbalance problem and then achieve better detection performance than the state-of-the-art algorithms. We believe that our research can be applied to solve the graph imbalance problems in other fields. The CSGNN code and datasets are publicly available at https://github.com/xxhu94/CSGNN.

Inductive Representation Learning On Large Graphs

Abstract: Low-dimensional embeddings of nodes in large graphs have proved extremely useful in a variety of prediction tasks, from content recommendation to identifying protein functions. However, most existing approaches require that all nodes in the graph are present during training of the embeddings; these previous approaches are inherently transductive and do not naturally generalize to unseen nodes. Here we present GraphSAGE, a general, inductive framework that leverages node feature information (e.g., text attributes) to efficiently generate node embeddings for previously unseen data. Instead of training individual embeddings for each node, we learn a function that generates embeddings by sampling and aggregating features from a node's local neighborhood. Our algorithm outperforms strong baselines on three inductive node-classification benchmarks: we classify the category of unseen nodes in evolving information graphs based on citation and Reddit post data, and we show that our algorithm generalizes to completely unseen graphs using a multi-graph dataset of protein-protein interactions.


Domain randomization is a technique used to enhance the variability of synthetic data and has been shown to substantially increase performance in the real world #REFR .

A recent survey categorized hundreds of synthetic datasets and the use cases they have been devised for #OTHEREFR .

Initially used to address low-level computer vision tasks such as optical flow #OTHEREFR , synthetic datasets are increasingly used to generate training datasets for high-level tasks such as, e.g., object recognition and detection #OTHEREFR , pose estimation #OTHEREFR , segmentation #OTHEREFR , human action recognition #OTHEREFR and pedestrian tracking and ReID #OTHEREFR .

Works in this field typically build onto well-known repositories, including millions of virtual models with known categories or properties, which can be programmatically manipulated to automate both data generation and its labelling #OTHEREFR .

Popular approaches for collecting synthetic data also include the use of video games #OTHEREFR , or fusing real and virtual data via compositing techniques and placing, e.g., virtual models onto real background images #OTHEREFR .

One of the main challenges associated with synthetic data is the domain shift between real and synthetic images, which can be tackled through transfer learning or domain adaptation #OTHEREFR .

With ever increasing CGI fidelity, the synthetic-to-real domain gap is progressively reducing.

Recent exciting results showed that training DNNs on very large and diverse synthetic datasets can outperform using public real datasets on tasks such as pedestrian tracking and ReID, even without fine-tuning on real data #OTHEREFR .

Bent & Broken Bicycles: Leveraging Synthetic Data For Damaged Object Re-Identification

Abstract: Instance-level object re-identification is a fundamental computer vision task, with applications from image retrieval to intelligent monitoring and fraud detection. In this work, we propose the novel task of damaged object re-identification, which aims at distinguishing changes in visual appearance due to deformations or missing parts from subtle intra-class variations. To explore this task, we leverage the power of computer-generated imagery to create, in a semi-automatic fashion, high-quality synthetic images of the same bike before and after a damage occurs. The resulting dataset, Bent & Broken Bicycles (BB-Bicycles), contains 39,200 images and 2,800 unique bike instances spanning 20 different bike models. As a baseline for this task, we propose TransReI3D, a multi-task, transformer-based deep network unifying damage detection (framed as a multi-label classification task) with object re-identification. The BBBicycles dataset is available at https://tinyurl.com/37tepf7m keywords instance-level retrieval; re-identification; synthetic data; damage detection; transformers

Domain Randomization For Transferring Deep Neural Networks From Simulation To The Real World

Abstract: Bridging the ‘reality gap’ that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator. With enough variability in the simulator, the real world may appear to the model as just another variation. We focus on the task of object localization, which is a stepping stone to general robotic manipulation skills. We find that it is possible to train a real-world object detector that is accurate to 1.5 cm and robust to distractors and partial occlusions using only data from a simulator with non-realistic random textures. To demonstrate the capabilities of our detectors, we show they can be used to perform grasping in a cluttered environment. To our knowledge, this is the first successful transfer of a deep neural network trained only on simulated RGB images (without pre-training on real images) to the real world for the purpose of robotic control.


To achieve the goal of learning contextual representations of time series with a bidirectional encoding scheme, we decide to construct the corrupt input with masking strategies by following #REFR .

Suppose that S v /S m is the number of masked/visible positions.

In this way, more comprehensive contextual representations of each position, including previous and future contexts, can be encoded, simultaneously.

For clarity, we use Z v to denote the embeddings of visible (unmasked) positions while letting Z m be the embeddings of masked ones.

Although some heuristic masking strategies, e.g., block masking #OTHEREFR , have been developed before, we adopt random masking strategies to form the corrupted inputs.

It means that each sub-series unit has the same probability of being masked in constructing selfsupervised signals.

Timemae: Self-Supervised Representations Of Time Series With Decoupled Masked Autoencoders

Abstract: Enhancing the expressive capacity of deep learning-based time series models with self-supervised pre-training has become ever-increasingly prevalent in time series classification. Even though numerous efforts have been devoted to developing self-supervised models for time series data, we argue that the current methods are not sufficient to learn optimal time series representations due to solely unidirectional encoding over sparse point-wise input units. In this work, we propose TimeMAE, a novel self-supervised paradigm for learning transferrable time series representations based on transformer networks. The distinct characteristics of the TimeMAE lie in processing each time series into a sequence of non-overlapping sub-series via window-slicing partitioning, followed by random masking strategies over the semantic units of localized sub-series. Such a simple yet effective setting can help us achieve the goal of killing three birds with one stone, i.e., (1) learning enriched contextual representations of time series with a bidirectional encoding scheme; (2) increasing the information density of basic semantic units; (3) efficiently encoding representations of time series using transformer networks. Nevertheless, it is a non-trivial to perform reconstructing task over such a novel formulated modeling paradigm. To solve the discrepancy issue incurred by newly injected masked embeddings, we design a decoupled autoencoder architecture, which learns the representations of visible (unmasked) positions and masked ones with two different encoder modules, respectively. Furthermore, we construct two types of informative targets to accomplish the corresponding pretext tasks. One is to create a tokenizer module that assigns a codeword to each masked region, allowing the masked codeword classification (MCC) task to be completed effectively. Another one is to adopt a siamese network structure to generate target representations for each masked input unit, aiming at performing the masked representation regression (MRR) optimization. Comprehensively pre-trained, our model can efficiently learn transferrable time series representations, thus benefiting the classification of time series. We extensively perform experiments on five benchmark datasets to verify the effectiveness of the TimeMAE. Experimental results show that the TimeMAE can significantly surpass previous competitive baselines. Furthermore, we also demonstrate the universality of the learned representations by performing transfer learning experiments. For the reproducibility of our results, we make our experiment codes publicly available to facilitate the self-supervised representations of time series in https://github.com/Mingyue-Cheng/TimeMAE.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


MEGA #REFR takes both temporal information and semantic similarity into account and aggregates local and global features jointly.

For example, Seq-NMS #OTHEREFR associates the bounding boxes from different frames with the IoU threshold; T-CNN #OTHEREFR links the object detection results from each frame according to optical flows.

These methods are not trained end-to-end, and the performances are always sub-optimal.

Feature-aggregation based models enhance the feature representations of the current frame by fusing those from the neighboring frames.

FGFA #OTHEREFR and MANet #OTHEREFR weight average the features from the neighboring frames after being warped based on optical flows.

SELSA #OTHEREFR , and Temporal ROI Align #OTHEREFR fuse the neighboring frames according to their semantic similarity.

Different from the methods mentioned above, which generate weights for aggregation using cosine similarity, TF-Blender #OTHEREFR applies a learnable network to predict the weights.

All the methods mentioned above are designed for the classic object detectors like Faster R-CNN #OTHEREFR , or Center-Net #OTHEREFR .

Recently, with DETR series #OTHEREFR introduced for the image domain tasks, researchers have begun to focus on how to use these Transformer-based models for video object detection.

Among them, TransVOD series #OTHEREFR introduces two key modules to boost the performance: Temporal Query Encoder to fuse object queries and Temporal Deformable Transformer Decoder (TDTD) to obtain current frame detection results.

Similarly, PT-SEFormer #OTHEREFR proposes the Spatial Transition Awareness Model to fuse the temporal and spatial information for a better prediction.

Faq: Feature Aggregated Queries For Transformer-Based Video Object Detectors

Abstract: Video object detection needs to solve feature degradation situations that rarely happen in the image domain. One solution is to use the temporal information and fuse the features from the neighboring frames. With Transformerbased object detectors getting a better performance on the image domain tasks, recent works began to extend those methods to video object detection. However, those existing Transformer-based video object detectors still follow the same pipeline as those used for classical object detectors, like enhancing the object feature representations by aggregation. In this work, we take a different perspective on video object detection. In detail, we improve the qualities of queries for the Transformer-based models by aggregation. To achieve this goal, we first propose a vanilla query aggregation module that weighted averages the queries according to the features of the neighboring frames. Then, we extend the vanilla module to a more practical version, which generates and aggregates queries according to the features of the input frames. Extensive experimental results validate the effectiveness of our proposed methods: On the challenging ImageNet VID benchmark, when integrated with our proposed modules, the current state-of-the-art Transformerbased object detectors can be improved by more than 2.4% on mAP and 4.2% on AP 50 .

Memory Enhanced Global-Local Aggregation For Video Object Detection

Abstract: How do humans recognize an object in a piece of video? Due to the deteriorated quality of single frame, it may be hard for people to identify an occluded object in this frame by just utilizing information within one image. We argue that there are two important cues for humans to recognize objects in videos: the global semantic information and the local localization information. Recently, plenty of methods adopt the self-attention mechanisms to enhance the features in key frame with either global semantic information or local localization information. In this paper we introduce memory enhanced global-local aggregation (MEGA) network, which is among the first trials that takes full consideration of both global and local information. Furthermore, empowered by a novel and carefully-designed Long Range Memory (LRM) module, our proposed MEGA could enable the key frame to get access to much more content than any previous methods. Enhanced by these two sources of information, our method achieves state-of-the-art performance on ImageNet VID dataset. Code is available at \url{https://github.com/Scalsol/mega.pytorch}.


Utility also includes an MLP classifier consisting in a 2-layer network with a hidden layer using Batch-Normalization #REFR and ReLU non-linearity.

We also considered non-linear feature selection methods (e.g., #OTHEREFR ), but settled for this simpler one.

• Nearest Neighbor classification. We also implemented Nearest Neighbor Classification as a classifier.

In practice, a test feature vector is assigned a labeled corresponding to the label of the closest training sample (in L2-norm).

For scalability, we implemented nearest neighbor retrieval using the approximate nearest neighbor library FAISS 97 .

The network is optimized using the Adam optimizer #OTHEREFR with learning rate generally set to 3.0 × 10 −4 unless otherwise stated, and use early-stopping with patience of 3 on F1-score computed on the hold-out set.

• MLP classification / Adversarial debiasing.

Our MLP classifier optionally includes the option of using adversarial debiasing #OTHEREFR as a method to mitigate bias.

This method essentially corresponds to introducing an adversary network that that is trained to infer the protected labels of the value of the protected variables of the input samples.

In our implementation the adversary has the same architecture as the original network, but a hidden size that is 4 times smaller, and is optimized by alternating its update with the update of the original network.

Auditing And Generating Synthetic Data With Controllable Trust Trade-Offs

Abstract: Data collected from the real world tends to be biased, unbalanced, and at risk of exposing sensitive and private information. This reality has given rise to the idea of creating synthetic datasets to alleviate risk, bias, harm, and privacy concerns inherent in the real data. This concept relies on Generative AI models to produce unbiased, privacy-preserving synthetic data while being true to the real data. In this new paradigm, synthetic data creators, users, and auditors must answer critical questions, such as "How can we tell if this approach delivers on its promises?" and "How can we determine whether a synthetic dataset and a model built from it conform to trust requirements, such as fairness, privacy, safety, and fidelity while still being valuable for downstream tasks of interest?". We present an auditing framework that offers a holistic assessment of synthetic datasets and AI models trained on them, centered around bias and discrimination prevention, fidelity to the real data, utility, robustness, and privacy preservation. We showcase our framework by auditing multiple generative models on diverse use cases, including education, healthcare, banking, human resources, and across different modalities, from tabular, to time-series, to natural language. Our use cases demonstrate the importance of a holistic assessment in order to ensure compliance with socio-technical safeguards that regulators and policymakers are increasingly enforcing. For this purpose, we introduce the trust index that ranks multiple synthetic datasets based on their prescribed safeguards and their desired trade-offs. Moreover, we devise a trust-index-driven model selection and cross-validation procedure via auditing in the training loop that we showcase on a class of transformer models that we dub TrustFormers, across different modalities. This trust-driven model selection allows for controllable trust trade-offs in the resulting synthetic data. We instrument our auditing framework with workflows that connect different stakeholders (e.g., data scientists, data governance experts, internal reviewers, external certifiers, and regulators) from model development to audit and certification via a synthetic data auditing report. We believe this mode of transparent reporting should become a standard to prevent bias, discrimination, and privacy breaches and ensure compliance with policy requirements, accountability, safety, and performance guarantees. Main Generative models have demonstrated impressive results in synthesizing high-quality data across multiple modalities from text 1, 2 , to images 3-6 , to chemistry 7 , to tabular and time-series data 8-10. We are entering a new era in training AI models, where synthetic data can be used to augment real data 11 , or as a complete replacement, in the most extreme case 12. One of the main motivations behind controllable synthetic data usage in training AI models is its promise to synthesize privacy-preserving data that enables safe sharing without putting the privacy of real users and individuals at risk, potentially circumventing cumbersome processes that are at the heart of many highly regulated fields such as financial services 13 and healthcare, for example 11, 14-16. Another motivation comes from controlling the generation process in order to balance the training data and reduce biases against protected groups and sensitive communities 17. Finally, synthetic data also offers new opportunities in simulating non-existing scenarios, providing grounding for causal inference via the generation of counterfactuals that would help explain some observations in the absence of real data 18. Synthetic data can be seedless 19, 20 , in the sense that it is not trained on real data but relies on a knowledge base or graph and on a rule-based generation process. While this seems ideal regarding privacy and fairness as this process does not rely on any real data, the underlying knowledge grounding can be biased and may also be at risk of linkage attacks that would leak the identities of individuals present in the real data. On the other side of the spectrum, we have synthetic data coming from inference from generative AI models trained on real data. This approach to synthetic data raises its own set of risks and pitfalls, as it often leads to memorization, privacy breaches, and data leakage 21. Another concern is the potential amplification of bias and disparate

Batch Normalization: Accelerating Deep Network Training By Reducing Internal Covariate Shift

Abstract: Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.