However, we found that Demucs V2 performs better than HuBERT-SEP N , possibly due to the more complex encoder and decoder design of Demucs V2, which was also reported in #REFR .

All experiments on the original Res-U-Net and Demucs V2 were reproduced by us to ensure a fair comparison.

Time-domain models were trained at 16 kHz to utilize the pretrained HuBERT model, so the baseline Demucs V2 results differ from #OTHEREFR .

Additionally, we provide the reported performance of several state-of-the-art models, although some of them (denoted in grey) used additional training data or were developed for the MDX challenge #OTHEREFR , which directly used the MusDB18 test set as a validation set for model selection.

From the comparison between the original Demucs V2, Res-U-Net, HuBERT-SEP N , and Pa -HuBERT N models (i.e., without pretraining), we observe that Pa -HuBERT N generally outperforms Res-U-Net.

The potential reason is that the transformer bottleneck processes the encoder features more efficiently than the CNN bottleneck model in Res-U-Net.

When fine-tuning the pretrained HuBERT and Pa -HuBERT models on the separation task, we observe an improvement in SDR for all four sources compared to models without pretraining.

This demonstrates the effectiveness of HuBERT in contributing to audio separation tasks beyond speech, and also shows that our proposed Pa -HuBERT model, pretrained with primitive auditory features from music data, can improve the performance of TF-domain separation models.

When comparing between Pa -HuBERT-SEP SSL and Pa -HuBERT-SEP 2SSL , we observe that 2SSL yields better performance on vocals and drums separation, but is degraded on bass and other separation.

We believe that the design of iterative pretraining can be further improved, including exploring different layers of latent features for pretraining and training varying numbers of K-Means models for different separation sources. This will be a focus of our future work.

Specifically, HuBERT shows a larger improvement than Pa -HuBERT, which can potentially be attributed to the size of the pretraining dataset (60 000 hours for HuBERT vs. 890 hours for Pa -HuBERT).

Pa -Hubert: Self-Supervised Music Source Separation Via Primitive Auditory Clustering And Hidden-Unit Bert

Abstract: In spite of the progress in music source separation research, the small amount of publicly-available clean source data remains a constant limiting factor for performance. Thus, recent advances in selfsupervised learning present a largely-unexplored opportunity for improving separation models by leveraging unlabelled music data. In this paper, we propose a self-supervised learning framework for music source separation inspired by the HuBERT speech representation model. We first investigate the potential impact of the original HuBERT model by inserting an adapted version of it into the wellknown Demucs V2 time-domain separation model architecture. We then propose a time-frequency-domain self-supervised model, Pa-HuBERT (pronounced Pac-HuBERT, for primitive auditory clustering HuBERT), that we later use in combination with a Res-U-Net decoder for source separation. Pa-HuBERT uses primitive auditory features of music as unsupervised clustering labels to initialize the self-supervised pretraining process using the Free Music Archive (FMA) dataset. The resulting framework achieves better source-todistortion ratio (SDR) performance on the MusDB18 test set than the original Demucs V2 and Res-U-Net models. We further demonstrate that it can boost performance with small amounts of supervised data. Ultimately, our proposed framework is an effective solution to the challenge of limited clean source data for music source separation.

Music Source Separation In The Waveform Domain

Abstract: Source separation for music is the task of isolating contributions, or stems, from different instruments recorded individually and arranged together to form a song. Such components include voice, bass, drums and any other accompaniments. Contrarily to many audio synthesis tasks where the best performances are achieved by models that directly generate the waveform, the state-of-the-art in source separation for music is to compute masks on the magnitude spectrum. In this paper, we first show that an adaptation of Conv-Tasnet (Luo & Mesgarani, 2019), a waveform-to-waveform model for source separation for speech, significantly beats the state-of-the-art on the MusDB dataset, the standard benchmark of multi-instrument source separation. Second, we observe that Conv-Tasnet follows a masking approach on the input signal, which has the potential drawback of removing parts of the relevant source without the capacity to reconstruct it. We propose Demucs, a new waveform-to-waveform model, which has an architecture closer to models for audio generation with more capacity on the decoder. Experiments on the MusDB dataset show that Demucs beats previously reported results in terms of signal to distortion ratio (SDR), but lower than Conv-Tasnet. Human evaluations show that Demucs has significantly higher quality (as assessed by mean opinion score) than Conv-Tasnet, but slightly more contamination from other sources, which explains the difference in SDR. Additional experiments with a larger dataset suggest that the gap in SDR between Demucs and Conv-Tasnet shrinks, showing that our approach is promising.


GroupNorm was created to allow for smaller batch sizes, as compared to standard BatchNorm #REFR .

For high-resolution images, smaller batches of size one or two are preferred, whereas BatchNorm requires larger batch sizes to perform well.

GroupNorm, does not normalize in batches, but instead normalizes along the feature dimension by considering groups of features.

GroupNorm has been shown to work well for batches of size two, and it may enhance object segmentation and detection.

An Empirical Analysis Of The Shift And Scale Parameters In Batchnorm

Abstract: Batch Normalization (BatchNorm) is a technique that improves the training of deep neural networks, especially Convolutional Neural Networks (CNN). It has been empirically demonstrated that BatchNorm increases performance, stability, and accuracy, although the reasons for such improvements are unclear. BatchNorm includes a normalization step as well as trainable shift and scale parameters. In this paper, we empirically examine the relative contribution to the success of BatchNorm of the normalization step, as compared to the re-parameterization via shifting and scaling. To conduct our experiments, we implement two new optimizers in PyTorch, namely, a version of BatchNorm that we refer to as AffineLayer, which includes the re-parameterization step without normalization, and a version with just the normalization step, that we call BatchNorm-minus. We compare the performance of our AffineLayer and BatchNorm-minus implementations to standard BatchNorm, and we also compare these to the case where no batch normalization is used. We experiment with four ResNet architectures (ResNet18, ResNet34, ResNet50, and ResNet101) over a standard image dataset and multiple batch sizes. Among other findings, we provide empirical evidence that the success of BatchNorm may derive primarily from improved weight initialization.

Group Normalization

Abstract: Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems --- BN's error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN's usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN's computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform its BN-based counterparts for object detection and segmentation in COCO, and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.


In addition to 0.00001, 1 and 100 tested in previous work #REFR , we also test λ = 0.0001, 0.001, 0.01, 0.1, 1 and 10 to demonstrate the impact brought by the choice of penalty coefficient λ.

ECRL is compared to several state-of-the-art algorithms including ERL #OTHEREFR , RCPO #OTHEREFR , and IPO #OTHEREFR .

In addition, we implement two variants of ERL with RCPO #OTHEREFR and a reward shape method #OTHEREFR using different coefficient values for constrained optimisation, as compared algorithms.

RCPO implemented with SAC is denoted as "RCPO" and IPO implemented with PPO is denoted as "IPO" in the experiment results, respectively.

Adopting RCPO to ERL is straightforward and not detailed here. The resulted algorithm is referred to as "RCPO-ERL".

When adopting the reward shape method to ERL, it is hard to determine the best value for penalty coefficient λ.

Instances of ERL with reward shape using different values of λ are referred to as "ERL * ", where * indicates the value of penalty coefficient λ.

"ERL" refers to the original ERL without reward shaping or RCPO.

Evolving Constrained Reinforcement Learning Policy

Abstract: Evolutionary algorithms have been used to evolve a population of actors to generate diverse experiences for training reinforcement learning agents, which helps to tackle the temporal credit assignment problem and improves the exploration efficiency. However, when adapting this approach to address constrained problems, balancing the trade-off between the reward and constraint violation is hard. In this paper, we propose a novel evolutionary constrained reinforcement learning (ECRL) algorithm, which adaptively balances the reward and constraint violation with stochastic ranking, and at the same time, restricts the policy's behaviour by maintaining a set of Lagrange relaxation coefficients with a constraint buffer. Extensive experiments on robotic control benchmarks show that our ECRL achieves outstanding performance compared to state-of-the-art algorithms. Ablation analysis shows the benefits of introducing stochastic ranking and constraint buffer. Index Terms-Evolutionary constrained reinforcement learning, evolutionary reinforcement learning, constrained reinforcement learning, stochastic ranking, robotic control This work is accepted by the 2023 International Joint Conference on Neural Networks (IJCNN).

Reward Constrained Policy Optimization

Abstract: Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.


The pioneering method in the family is CAM #REFR , which produces saliency maps by linearly combining the convolutional feature maps Figure 1 . An example of SUNY explanations. SUNY highlights sufficient and necessary input regions w.r.t. the model's prediction towards the target class.

In computer vision, an important research direction is to give Convolutional Neural Network (CNN) more transparency to extend its deployment on a broader basis.

This paper addresses the eXplainable Artificial Intelligence (XAI) #OTHEREFR problem corresponding to CNN for natural image classification, i.e., reasoning why a classifier makes particular decisions.

Specifically, we study a leading family of techniques called Class-Activation-Maps (CAMs), which present heatmaps highlighting image portions associated with a model's class prediction.

The 2D saliency map is the first-of-itskind visual explanation design to the best of our knowledge.

with weights generated by a global average pooling layer.

CAM's structural restriction on the global average pooling layer prohibits its use for general CNNs, and a series of gradient-weighted CAMs #OTHEREFR 31] have been proposed to overcome the restriction.

However, such CAMs suffer from gradients' saturation and vanishing issues, which cause the resulting explanations to be noisy #OTHEREFR .

To bypass the shortcomings of gradients, Score-CAM #OTHEREFR and Group-CAM #OTHEREFR weight the feature maps by contribution scores, referring to the corresponding input features' importance to the model output.

Suny: A Visual Interpretation Framework For Convolutional Neural Networks From A Necessary And Sufficient Perspective

Abstract: Researchers have proposed various methods for visually interpreting the Convolutional Neural Network (CNN) via saliency maps, which include Class-Activation-Map (CAM) based approaches as a leading family. However, in terms of the internal design logic, existing CAM-based approaches often overlook the causal perspective that answers the core "why" question to help humans understand the explanation. Additionally, current CNN explanations lack the consideration of both necessity and sufficiency, two complementary sides of a desirable explanation. This paper presents a causality-driven framework, SUNY, designed to rationalize the explanations toward better human understanding. Using the CNN model's input features or internal filters as hypothetical causes, SUNY generates explanations by bi-directional quantifications on both the necessary and sufficient perspectives. Extensive evaluations justify that SUNY not only produces more informative and convincing explanations from the angles of necessity and sufficiency, but also achieves performances competitive to other approaches across different CNN architectures over largescale datasets, including ILSVRC2012 and CUB-200-2011.

Learning Deep Features For Discriminative Localization

Abstract: In this work, we revisit the global average pooling layer proposed in [13], and shed light on how it explicitly enables the convolutional neural network (CNN) to have remarkable localization ability despite being trained on imagelevel labels. While this technique was previously proposed as a means for regularizing training, we find that it actually builds a generic localizable deep representation that exposes the implicit attention of CNNs on an image. Despite the apparent simplicity of global average pooling, we are able to achieve 37.1% top-5 error for object localization on ILSVRC 2014 without training on any bounding box annotation. We demonstrate in a variety of experiments that our network is able to localize the discriminative image regions despite just being trained for solving classification task1.


Although using inplace distillation #REFR for sub-networks prevents the model from collapsing, there is still a big gap between the results of S-Net+Distill and training each model individually for SimSiam.

To address this problem in the context of supervised learning, the family of slimmable networks (S-Net) and universally slimmable networks (US-Net) #OTHEREFR were proposed, which can switch freely among different widths by training only once.

Driven by the success of slimmable networks, a question arises: Can we train a self-supervised model that can run at arbitrary width? A naïve solution is to replace the supervised loss with self-supervised loss based on the US-Net framework.

However, we find that this solution doesn't work directly after empirical studies.

Table 1 shows that the phenomenon in self-supervised scenarios is very different.

The model directly collapses after applying the popular SSL method SimSiam #OTHEREFR to slimmable networks #OTHEREFR .

So why is the situation so different in SSL and how to further improve the performance (i.e., close the gap)?

In this paper, we present a unified perspective to explain the differences and propose corresponding measures to bridge the gap.

From a unified gradient perspective, we find that the key is that the guidance to sub-networks should be consistent between iterations, and we analyze which components of SSL incur the temporal inconsistency problem and why US-Net works in supervised learning.

Based on these theoretical analyses, we propose three guidelines for the loss design of US-Net training to ensure temporal consistency.

As long as one of them is satisfied, US-Net can work well, no matter in supervised or self-supervised scenarios.

Three Guidelines You Should Know For Universally Slimmable Self-Supervised Learning

Abstract: We propose universally slimmable self-supervised learning (dubbed as US3L) to achieve better accuracy-efficiency trade-offs for deploying self-supervised models across different devices. We observe that direct adaptation of selfsupervised learning (SSL) to universally slimmable networks misbehaves as the training process frequently collapses. We then discover that temporal consistent guidance is the key to the success of SSL for universally slimmable networks, and we propose three guidelines for the loss design to ensure this temporal consistency from a unified gradient perspective. Moreover, we propose dynamic sampling and group regularization strategies to simultaneously improve training efficiency and accuracy. Our US3L method has been empirically validated on both convolutional neural networks and vision transformers. With only once training and one copy of weights, our method outperforms various state-ofthe-art methods (individually trained or not) on benchmarks including recognition, object detection and instance segmentation. Our code is available at https://github.com/ megvii-research/US3L-CVPR2023.

Universally Slimmable Networks And Improved Training Techniques

Abstract: Slimmable networks are a family of neural networks that can instantly adjust the runtime width. The width can be chosen from a predefined widths set to adaptively optimize accuracy-efficiency trade-offs at runtime. In this work, we propose a systematic approach to train universally slimmable networks (US-Nets), extending slimmable networks to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers. We further propose two improved training techniques for US-Nets, named the sandwich rule and inplace distillation, to enhance training process and boost testing accuracy. We show improved performance of universally slimmable MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with individually trained ones and 4-switch slimmable network baselines. We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning. Extensive ablation experiments on these representative tasks demonstrate the effectiveness of our proposed methods. Our discovery opens up the possibility to directly evaluate FLOPs-Accuracy spectrum of network architectures. Code and models are available at: this https URL


The Lorenz system is a well-known 3-dimensional chaotic system with a strange attractor, driven by the equations: #REFR with parameters σ = 10, r = 28, b = 8/3.

We perform experiments on three dynamical systems: the simple pendulum, a 3-dimensional dynamical system modeling the lowdimensional attractor of a nonlinear high-dimensional fluid flow, and the chaotic Lorenz-63 system.

The first two benchmarks are taken from #OTHEREFR and we refer the reader to this paper for further detail.

dx dt = σ(y − x); dy dt = x(r − z) − y; dz dt = xy − bz

1 Should we constrain K to be perfectly orthogonal, a real logarithm would always exist since O(d) is a matrix Lie group, whose Lie algebra is the set of skew-symmetric matrices #OTHEREFR As for the Lorenz-63 system, despite the simplicity of our architecture, it is still able to learn to behave in a chaotic way, or at least to simulate a chaotic behavior, even when the initialization is outside of the attractor.

We obtain Lyapunov exponents 0.61, 0.07, −14.34 for simulated trajectories of our model, which is close to the estimated real exponents 0.9056, 0, −14.5721 of the Lorenz system #OTHEREFR .

We also manage to model the attractor when initializing our predictions outside of it.

Note that, while the method from #OTHEREFR is also able to model the Lorenz-63 dynamics faithfully, classical models like recurrent neural networks are known to fail at this task. We plot our simulated trajectories in figure 1.

Leveraging Neural Koopman Operators To Learn Continuous Representations Of Dynamical Systems From Scarce Data

Abstract: Over the last few years, several works have proposed deep learning architectures to learn dynamical systems from observation data with no or little knowledge of the underlying physics. A line of work relies on learning representations where the dynamics of the underlying phenomenon can be described by a linear operator, based on the Koopman operator theory. However, despite being able to provide reliable long-term predictions for some dynamical systems in ideal situations, the methods proposed so far have limitations, such as requiring to discretize intrinsically continuous dynamical systems, leading to data loss, especially when handling incomplete or sparsely sampled data. Here, we propose a new deep Koopman framework that represents dynamics in an intrinsically continuous way, leading to better performance on limited training data, as exemplified on several datasets arising from dynamical systems.

Learning Koopman Invariant Subspaces For Dynamic Mode Decomposition

Abstract: Spectral decomposition of the Koopman operator is attracting attention as a tool for the analysis of nonlinear dynamical systems. Dynamic mode decomposition is a popular numerical algorithm for Koopman spectral analysis; however, we often need to prepare nonlinear observables manually according to the underlying dynamics, which is not always possible since we may not have any a priori knowledge about them. In this paper, we propose a fully data-driven method for Koopman spectral analysis based on the principle of learning Koopman invariant subspaces from observed data. To this end, we propose minimization of the residual sum of squares of linear least-squares regression to estimate a set of functions that transforms data into a form in which the linear regression fits well. We introduce an implementation with neural networks and evaluate performance empirically using nonlinear dynamical systems and applications.


LRP and its variants, which are based on deep Taylor decomposition #REFR , backpropagate the relevance score from output layer to input layer through specific rules to directly obtain the contribution of each pixel in the input image.

Therefore, as an extension of CAM, Grad-CAM #OTHEREFR , Score-CAM #OTHEREFR and other CAM-based methods have been proposed.

However, before Layer-CAM #OTHEREFR , all CAM-based methods could only be applied to the final convolutional layer of the model, and if the target layer became shallow, the faithfulness (measure how accurately the explantion reveals the model decision) of the explanations would be significantly reduced, thus only coarse-grained visual explanations could be generated, which provides coarse location of the target object, limiting the performance on tasks that need pixel-accurate object locations #OTHEREFR .

Layer-CAM, on the other hand, although outperforming other CAM-based methods on shallow convolutional layers, suffers from shattered gradient problem #OTHEREFR as the network gets deep and the target layer becomes shallow, which can lead to problems such as making visual explanation noisy and reduced faithfulness (not as dramatically as other CAM-based methods, but still not negligible).

As shown in Figure 1 , as the target layer becomes shallower, the accuracy of the explanations generated by Grad-CAM and Score-CAM will decrease significantly and even become unusable (unable to extract contributing pixels or filter out non-contributing pixels).

The visual explanation of Layer-CAM has more and more noise, and the accuracy has decreased, while the FG-Grad-CAM and FG-Score-CAM still show good performance.

They can generate the explanations at the same resolution as the input image. However, the explanations have low faithfulness.

In order to generate fine-grained visual explanation with high faithfulness, this paper proposes a novel method: FG-CAM.

Different from other CAM-based methods which are directly applied to shallow layers to obtain high-resolution explanations, FG-CAM uses the relationship between two adjacent layers of feature maps with resolution difference to gradually increase the resolution of explanations produced on the last convolutional layer, while finding the contributing pixels and filtering out the pixels that do not contribute at each step.

FG-CAM not only solves the problem that CAM-based methods can only provide rough explanations without changing their characteristics, but also generates fine-grained explanations that have higher faithfulness than LRP and its variants.

This paper also presents FG-CAM with denoising, which is able to generate less noisy explanations with almost no change in explanation faithfulness.

Fine-Grained And High-Faithfulness Explanations For Convolutional Neural Networks

Abstract: Recently, explaining CNNs has become a research hotspot. CAM (Class Activation Map)-based methods and LRP (Layer-wise Relevance Propagation) method are two common explanation methods. However, due to the small spatial resolution of the last convolutional layer, the CAMbased methods can often only generate coarse-grained visual explanations that provide a coarse location of the target object. LRP and its variants, on the other hand, can generate fine-grained explanations. But the faithfulness of the explanations is too low. In this paper, we propose FG-CAM (fine-grained CAM), which extends the CAM-based methods to generate fine-grained visual explanations with high faithfulness. FG-CAM uses the relationship between two adjacent layers of feature maps with resolution difference to gradually increase the explanation resolution, while finding the contributing pixels and filtering out the pixels that do not contribute at each step. Our method not only solves the shortcoming of CAM-based methods without changing their characteristics, but also generates fine-grained explanations that have higher faithfulness than LRP and its variants. We also present FG-CAM with denoising, which is a variant of FG-CAM and is able to generate less noisy explanations with almost no change in explanation faithfulness. Experimental results show that the performance of FG-CAM is almost unaffected by the explanation resolution. FG-CAM outperforms existing CAM-based methods significantly in the both shallow and intermediate convolutional layers, and outperforms LRP and its variations significantly in the input layer.

Explaining Nonlinear Classification Decisions With Deep Taylor Decomposition

Abstract: Nonlinear methods such as Deep Neural Networks (DNNs) are the gold standard for various challenging machine learning problems such as image recognition. Although these methods perform impressively well, they have a significant disadvantage, the lack of transparency, limiting the interpretability of the solution and thus the scope of application in practice. Especially DNNs act as black boxes due to their multilayer nonlinear structure. In this paper we introduce a novel methodology for interpreting generic multilayer neural networks by decomposing the network classification decision into contributions of its input elements. Although our focus is on image classification, the method is applicable to a broad set of input data, learning tasks and network architectures. Our method called deep Taylor decomposition efficiently utilizes the structure of the network by backpropagating the explanations from the output to the input layer. We evaluate the proposed method empirically on the MNIST and ILSVRC data sets. HighlightsA novel method to explain nonlinear classification decisions in terms of input variables is introduced.The method is based on Taylor expansions and decomposes the output of a deep neural network in terms of input variables.The resulting deep Taylor decomposition can be applied directly to existing neural networks without retraining.The method is tested on two large-scale neural networks for image classification: BVLC CaffeNet and GoogleNet.


We use a modified version of Xception #REFR as the backbone network, with model parameters initialized by pre-training on ImageNet.

Note that there are three versions of FF++ in terms of compression level, i.e., raw, lightly compressed (HQ), and heavily compressed (LQ).

Since realistic forgeries often have a limited quality, the HQ and LQ versions are used in experiments.

By default, the HQ version is adopted, in line with other works #OTHEREFR .

If there is any deviation from this default, it will be explicitly stated.

Face extraction and alignment are performed using DLIB #OTHEREFR .

Following previous works #OTHEREFR , the aligned faces are resized to 256 × 256 for both the training and testing.

We use the Adam #OTHEREFR for optimization with the learning rate of 0.0002, and the batch size is fixed as 32. In the overall loss function in Eq.

(9), we set λ 1 to λ 3 as 0.1, 0.3, 0.05 empirically. The margin α in Eq. (5) is set to 3.

We also apply some widely used data augmentations, i.e., image compression, horizontal flip, and random brightness contrast.

Ucf: Uncovering Common Features For Generalizable Deepfake Detection

Abstract: Deepfake detection remains a challenging task due to the difficulty of generalizing to new types of forgeries. This problem primarily stems from the overfitting of existing detection methods to forgery-irrelevant features and methodspecific patterns. The latter is often ignored by previous works. This paper presents a novel approach to address the two types of overfitting issues by uncovering common forgery features. Specifically, we first propose a disentanglement framework that decomposes image information into three distinct components: forgery-irrelevant, methodspecific forgery, and common forgery features. To ensure the decoupling of method-specific and common forgery features, a multi-task learning strategy is employed, including a multi-class classification that predicts the category of the forgery method and a binary classification that distinguishes the real from the fake. Additionally, a conditional decoder is designed to utilize forgery features as a condition along with forgery-irrelevant features to generate reconstructed images. Furthermore, a contrastive regularization technique is proposed to encourage the disentanglement of the common and specific forgery features. Ultimately, we only utilize the common forgery features for the purpose of generalizable deepfake detection. Extensive evaluations demonstrate that our framework can perform superior generalization than current state-of-the-art methods.

Faceforensics++: Learning To Detect Manipulated Facial Images

Abstract: The rapid progress in synthetic image generation and manipulation has now come to a point where it raises significant concerns for the implications towards society. At best, this leads to a loss of trust in digital content, but could potentially cause further harm by spreading false information or fake news. This paper examines the realism of state-of-the-art image manipulations, and how difficult it is to detect them, either automatically or by humans. To standardize the evaluation of detection methods, we propose an automated benchmark for facial manipulation detection. In particular, the benchmark is based on DeepFakes, Face2Face, FaceSwap and NeuralTextures as prominent representatives for facial manipulations at random compression level and size. The benchmark is publicly available and contains a hidden test set as well as a database of over 1.8 million manipulated images. This dataset is over an order of magnitude larger than comparable, publicly available, forgery datasets. Based on this data, we performed a thorough analysis of data-driven forgery detectors. We show that the use of additional domainspecific knowledge improves forgery detection to unprecedented accuracy, even in the presence of strong compression, and clearly outperforms human observers.


Problem Statement: Unsupervised Few-Shot Learning

Note that D meta train and D meta test can come from the same domain (i.e., the standard in-domain setting) or different domains (i.e., cross-domain) as suggested by #REFR .

At the meta-test phase, we transfer the model to new few-shot tasks {T i } ∼ D meta test where each task T i aims to classify query samples {x q } among N labels using support (i.e., training)

We here assume the task T i consists of K support samples for each label y ∈ {1, . . .

, N }, which is referred to as N -way K-shot classification.

Unsupervised Meta-Learning Via Few-Shot Pseudo-Supervised Contrastive Learning

Abstract: Unsupervised meta-learning aims to learn generalizable knowledge across a distribution of tasks constructed from unlabeled data. Here, the main challenge is how to construct diverse tasks for meta-learning without label information; recent works have proposed to create, e.g., pseudo-labeling via pretrained representations or creating synthetic samples via generative models. However, such a task construction strategy is fundamentally limited due to heavy reliance on the immutable pseudo-labels during meta-learning and the quality of the representations or the generated samples. To overcome the limitations, we propose a simple yet effective unsupervised meta-learning framework, coined Pseudo-supervised Contrast (PsCo), for few-shot classification. We are inspired by the recent self-supervised learning literature; PsCo utilizes a momentum network and a queue of previous batches to improve pseudo-labeling and construct diverse tasks in a progressive manner. Our extensive experiments demonstrate that PsCo outperforms existing unsupervised meta-learning methods under various in-domain and cross-domain few-shot classification benchmarks. We also validate that PsCo is easily scalable to a large-scale benchmark, while recent prior-art meta-schemes are not.

A Closer Look At Few-Shot Classification

Abstract: Few-shot classification aims to learn a classifier to recognize unseen classes during training with limited labeled examples. While significant progress has been made, the growing complexity of network designs, meta-learning algorithms, and differences in implementation details make a fair comparison difficult. In this paper, we present 1) a consistent comparative analysis of several representative few-shot classification algorithms, with results showing that deeper backbones significantly reduce the performance differences among methods on datasets with limited domain differences, 2) a modified baseline method that surprisingly achieves competitive performance when compared with the state-of-the-art on both the \miniI and the CUB datasets, and 3) a new experimental setting for evaluating the cross-domain generalization ability for few-shot classification algorithms. Our results reveal that reducing intra-class variation is an important factor when the feature backbone is shallow, but not as critical when using deeper backbones. In a realistic cross-domain evaluation setting, we show that a baseline method with a standard fine-tuning practice compares favorably against other state-of-the-art few-shot learning algorithms.


Semi-Supervised Learning Under Extremely Low Data Regimes

As can be seen in Table 5 , the proposed method significantly outperforms MURAUER #REFR under the three out of the four scenarios.

We compare the proposed method with MURAUER #OTHEREFR , which to the best of our knowledge is the only depth-based 3D hand pose estimation method examined under scenarios of severe labeled data scarcity.

The only scenario where the proposed method is inferior is when there are only 10 labeled examples.

This is because such a small amount of labeled training data lacks adequate information about the nature of the task.

MURAUER #OTHEREFR compensates for this lack of information by pre-training on synthetic data. However, the proposed method does not use any pre-training.

More importantly, in contrast to MURAUER #OTHEREFR , the proposed method achieves this performance without the application-limiting requirement for multi-view real-data.

Remarkably, in the case where there are only 100 labeled samples, the proposed method achieves 12.11 mm, which clearly demonstrates the practicality of the proposed method in cases of severely limited access to labeled data.

Pushing The Envelope For Depth-Based Semi-Supervised 3D Hand Pose Estimation With Consistency Training

Abstract: Despite the significant progress that depth-based 3D hand pose estimation methods have made in recent years, they still require a large amount of labeled training data to achieve high accuracy. However, collecting such data is both costly and time-consuming. To tackle this issue, we propose a semi-supervised method to significantly reduce the dependence on labeled training data. The proposed method consists of two identical networks trained jointly: a teacher network and a student network. The teacher network is trained using both the available labeled and unlabeled samples. It leverages the unlabeled samples via a loss formulation that encourages estimation equivariance under a set of affine transformations. The student network is trained using the unlabeled samples with their pseudo-labels provided by the teacher network. For inference at test time, only the student network is used. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art semi-supervised methods by large margins.

Murauer: Mapping Unlabeled Real Data For Label Austerity

Abstract: Data labeling for learning 3D hand pose estimation models is a huge effort. Readily available, accurately labeled synthetic data has the potential to reduce the effort. However, to successfully exploit synthetic data, current state-of-the-art methods still require a large amount of labeled real data. In this work, we remove this requirement by learning to map from the features of real data to the features of synthetic data mainly using a large amount of synthetic and unlabeled real data. We exploit unlabeled data using two auxiliary objectives, which enforce that (i) the mapped representation is pose specific and (ii) at the same time, the distributions of real and synthetic data are aligned. While pose specifity is enforced by a self-supervisory signal requiring that the representation is predictive for the appearance from different views, distributions are aligned by an adversarial term. In this way, we can significantly improve the results of the baseline system, which does not use unlabeled data and outperform many recent approaches already with about 1% of the labeled real data. This presents a step towards faster deployment of learning based hand pose estimation, making it accessible for a larger range of applications.


The current implementation enables two users to engage in a virtual video call while displaying their avatar faces. arXiv:2304.01443v1 [cs.HC] 4 Apr 2023 #REFR .

Furthermore, our solution leads to other applications, such as a virtual Omegle chat and a setup for virtual YouTubers.

The former provides a safe space for one-on-one video chats with random users while displaying facial movements and expressions on avatar mesh.

The latter eliminates the need for costly tracking headsets by using software to track and animate characters with a webcam.

To offer a partially immersive Metaverse experience, we provide a solution to replace the 3D conference call feature from Meta, which requires a costly VR headset #OTHEREFR .

In brief, we use an AI Face Mesh model to track the user's facial expressions, position, and orientation, to render 3D mesh data into a 2D screen through a rendering pipeline done on WebGL, allowing for a simulated 3D conference experience using only a webcam, microphone, and web browser.

In summary, we demonstrate that companies like Meta can use this product to offer an entry-level Metaverse experience through a web portal rather than requiring consumers to invest in expensive hardware.

By providing these accessible applications, our solution is expected to bridge the gap for consumers to experience the immersive Metaverse.

Virtual Avatar Stream: A Cost-Down Approach To The Metaverse Experience

Abstract: The Metaverse through VR headsets is a rapidly growing concept, but the high cost of entry currently limits access for many users. This project aims to provide an accessible entry point to the immersive Metaverse experience by leveraging web technologies. The platform developed allows users to engage with rendered avatars using only a web browser, microphone, and webcam. By employing the WebGL and MediaPipe face tracking AI model from Google, the application generates real-time 3D face meshes for users. It uses a client-to-client streaming cluster to establish a connection, and clients negotiate SRTP protocol through WebRTC for direct data streaming. Additionally, the project addresses backend challenges through an architecture that is serverless, distributive, auto-scaling, highly resilient, and secure. The platform offers a scalable, hardware-free solution for users to experience a near-immersive Metaverse, with the potential for future integration with game server clusters. This project provides an important step toward a more inclusive Metaverse accessible to a wider audience.

Real-Time Facial Surface Geometry From Monocular Video On Mobile Gpus

Abstract: We present an end-to-end neural network-based model for inferring an approximate 3D mesh representation of a human face from single camera input for AR applications. The relatively dense mesh model of 468 vertices is well-suited for face-based AR effects. The proposed model demonstrates super-realtime inference speed on mobile GPUs (100-1000+ FPS, depending on the device and model variant) and a high prediction quality that is comparable to the variance in manual annotations of the same image.


Additionally, it was shown by Gribanov & Veselov #REFR that it is possible to prove a variant of the Flatness Theorem for simplices such that the function ω will depend only on the sub-determinants spectrum of A instead of n.

Due to #OTHEREFR Corollary 9] (see also #OTHEREFR ), the integer linear optimization problem can be solved with O n + ∆ 2 • log(∆) • log(∆ gcd ) operations.

Due to Gribanov & Malyshev #OTHEREFR and a modification from Gribanov, Malyshev & Zolotykh #OTHEREFR , the integer points counting problem can be solved with O(n 4 •∆ 3 ) operations.

Due to Gribanov, Malyshev, Pardalos & Veselov #OTHEREFR (see also #OTHEREFR ), the lattice-width can be computed with poly(n, ∆, ∆ ext )-operations, where ∆ ext = ∆(A b) and (A b) is the system's extended matrix.

Additionally, for empty simplices, the complexity dependence on ∆ ext can be avoided, which gives the bound poly(n, ∆).

The analogous result for lattice-simplices defined by convex hulls of their vertices is presented in Gribanov, Malyshev & Veselov #OTHEREFR .

More precisely, the following statement is true: for any empty simplex S, the inequality width(S) < ∆ min (A) holds, where ∆ min (A) is the minimum between all rankorder nonzero sub-determinants of A taken by an absolute value.

Additionally, if the inequality is not satisfied, then some integer point inside S can be found by a polynomial-time algorithm.

The last inequality for empty simplices was improved to width(S) < ⌊∆ min /2⌋ in the work #OTHEREFR of Henk, Kuhlmann & Weismantel.

For a general ∆-modular empty polytope P, the inequality width

Enumeration And Unimodular Equivalence Of Empty Delta-Modular Simplices ⋆

Abstract: Consider a class of simplices defined by systems Ax ≤ b of linear inequalities with ∆-modular matrices. A matrix is called ∆modular, if all its rank-order sub-determinants are bounded by ∆ in an absolute value. In our work we call a simplex ∆-modular, if it can be defined by a system Ax ≤ b with a ∆-modular matrix A. And we call a simplex empty, if it contains no points with integer coordinates. In literature, a simplex is called lattice-simplex, if all its vertices have integer coordinates. And a lattice-simplex called empty, if it contains no points with integer coordinates excluding its vertices. Recently, assuming that ∆ is fixed, it was shown in [37] that the number of ∆-modular empty simplices modulo the unimodular equivalence relation is bounded by a polynomial on dimension. We show that the analogous fact holds for the class of ∆-modular empty lattice-simplices. As the main result, assuming again that the value of the parameter ∆ is fixed, we show that all unimodular equivalence classes of simplices of the both types can be enumerated by a polynomial-time algorithm. As the secondary result, we show the existence of a polynomial-time algorithm for the problem to check the unimodular equivalence relation for a given pair of ∆-modular, not necessarily empty, simplices.

On Integer Programing With Bounded Determinants

Abstract: Let A be an \((m \times n)\) integral matrix, and let \(P=\{ x :A x \le b\}\) be an n-dimensional polytope. The width of P is defined as \( w(P)=min\{ x\in \mathbb {Z}^n{\setminus }\{0\} :max_{x \in P} x^\top u - min_{x \in P} x^\top v \}\). Let \(\varDelta (A)\) and \(\delta (A)\) denote the greatest and the smallest absolute values of a determinant among all \(r(A) \times r(A)\) sub-matrices of A, where r(A) is the rank of the matrix A. We prove that if every \(r(A) \times r(A)\) sub-matrix of A has a determinant equal to \(\pm \varDelta (A)\) or 0 and \(w(P)\ge (\varDelta (A)-1)(n+1)\), then P contains n affine independent integer points. Additionally, we present similar results for the case of k-modular matrices. The matrix A is called totally k-modular if every square sub-matrix of A has a determinant in the set \(\{0,\, \pm k^r :r \in \mathbb {N} \}\). When P is a simplex and \(w(P)\ge \delta (A)-1\), we describe a polynomial time algorithm for finding an integer point in P.


Sometimes the transform φ θ (•) may not be learnable (e.g., the fixed DFT #REFR ), and in those situations the optimization variable only includes Z.

Suppose that the noisy HSI Y ∈ R h×w×b admits Y = X + N , where X denotes the clean HSI and N denotes noise.

To infer the underlying clean HSI X from the observed Y, t-SVD method generally formulates the following model:

Here, L(Y, X ) denotes the fidelity term and ψ(Z (k) ) represents the low-rank characterization of Z (k) (which denotes the k-th frontal (spatial) slice of Z ∈ R h×w×b #OTHEREFR ).

φ θ (•) : R h×w×b → R h×w×b denotes a transform with learnable parameters θ, which transforms the low-rank representation Z into the original domain.

The philosophy of the t-SVD model (1) is to minimize the rank in the transformed domain, which can model the implicit low-rankness of HSI.

There are naturally two key building blocks for exactly modeling the implicit low-rankness, i.e., the selection of the transform φ θ (•) and the exact low-rank characterization ψ(•) of the transformed frontal slice Z (k) .

Most t-SVD-based methods focus on the design of different transforms φ θ (•) (see examples in #OTHEREFR , #OTHEREFR , #OTHEREFR ), but all of them pay less attention to the exact characterization of the transformed frontal slice.

They mostly adopt shallow representations to characterize Z (k) , e.g., MF #OTHEREFR , #OTHEREFR , QR factorization #OTHEREFR , and nuclear norm #OTHEREFR , #OTHEREFR .

However, these shallow representations may not be expressive enough to capture fine details of the clean HSI.

H2Tf For Hyperspectral Image Denoising: Where Hierarchical Nonlinear Transform Meets Hierarchical Matrix Factorization

Abstract: Recently, tensor singular value decomposition (t-SVD) has emerged as a promising tool for hyperspectral image (HSI) processing. In the t-SVD, there are two key building blocks: (i) the low-rank enhanced transform and (ii) the accompanying low-rank characterization of transformed frontal slices. Previous t-SVD methods mainly focus on the developments of (i), while neglecting the other important aspect, i.e., the exact characterization of transformed frontal slices. In this letter, we exploit the potentiality in both building blocks by leveraging the Hierarchical nonlinear transform and the Hierarchical matrix factorization to establish a new Tensor Factorization (termed as H2TF). Compared to shallow counter partners, e.g., low-rank matrix factorization or its convex surrogates, H2TF can better capture complex structures of transformed frontal slices due to its hierarchical modeling abilities. We then suggest the H2TF-based HSI denoising model and develop an alternating direction method of multipliers-based algorithm to address the resultant model. Extensive experiments validate the superiority of our method over state-of-the-art HSI denoising methods.

Tensor Robust Principal Component Analysis With A New Tensor Nuclear Norm

Abstract: In this paper, we consider the Tensor Robust Principal Component Analysis (TRPCA) problem, which aims to exactly recover the low-rank and sparse components from their sum. Our model is based on the recently proposed tensor-tensor product (or t-product) [15]. Induced by the t-product, we first rigorously deduce the tensor spectral norm, tensor nuclear norm, and tensor average rank, and show that the tensor nuclear norm is the convex envelope of the tensor average rank within the unit ball of the tensor spectral norm. These definitions, their relationships and properties are consistent with matrix cases. Equipped with the new tensor nuclear norm, we then solve the TRPCA problem by solving a convex program and provide the theoretical guarantee for the exact recovery. Our TRPCA model and recovery guarantee include matrix RPCA as a special case. Numerical experiments verify our results, and the applications to image recovery and background modeling problems demonstrate the effectiveness of our method.


It is responsible for adaptively adjusting the phase shift of each reflecting element in terms of the acquired CSI through periodic channel estimation #REFR .

with the Rician factor Γ, the LOS component H LOS , the multipath component H N LOS consisting of independent entries that follow CN (0, 1), and the BS-IRS path loss σ 2 h .

Consider a macro-cell scenario with far-field assumption, where there is no reflector surrounding the BS and IRS, the signal transmission more likes free-space propagation, which can be computed by

where L 0 stands for the path loss at the reference distance of 1 m, and α means the path loss exponent.

A smart controller of the IRS is connected to the BS with a wired or wireless link.

The signal reflection of a typical element n for user k is mathematically modeled by a reflection coefficient kn = a kn e jφ kn , where φ kn ∈ [0, 2π) denotes an induced phase shift, and a kn ∈ [0, 1] stands for amplitude attenuation.

As mentioned by #OTHEREFR , a kn = 1, ∀n, k is the optimal attenuation that maximizes the strength of the received signal and simplifies the implementation complexity.

Hence, the reflection optimization only focuses on the phase shifts φ kn , ∀n, k.

By ignoring hardware impairments such as quantified phase shifts #OTHEREFR and phase noise #OTHEREFR , the k th UE observes the received signal

Orthogonal And Non-Orthogonal Multiple Access For Intelligent Reflection Surface In 6G Systems

Abstract: Intelligent reflecting surface (IRS) is envisioned to become a key technology for the upcoming six-generation (6G) wireless system due to its potential of reaping high performance in a power-efficient and cost-efficient way. With its disruptive capability and hardware constraint, the integration of IRS imposes some fundamental particularities on the coordination of multiuser signal transmission. Consequently, the conventional orthogonal and non-orthogonal multiple-access schemes are hard to directly apply because of the joint optimization of active beamforming at the base station and passive reflection at the IRS. Relying on an alternating optimization method, we develop novel schemes for efficient multiple access in IRS-aided multiuser multi-antenna systems in this paper. Achievable performance in terms of the sum spectral efficiency is theoretically analyzed. A comprehensive comparison of different schemes and configurations is conducted through Monte-Carlo simulations to clarify which scheme is favorable for this emerging 6G paradigm.

Channel Estimation For Intelligent Reflecting Surface Assisted Multiuser Communications

Abstract: In intelligent reflecting surface (IRS) assisted communication systems, the acquisition of channel state information (CSI) is a crucial impediment for achieving the beamforming gain of IRS because of the considerable overhead required for channel estimation. Specifically, under the current beamforming design for IRS-assisted communications, $KMN+KM$ channel coefficients should be estimated, where $K$, $N$ and $M$ denote the numbers of users, IRS reflecting elements, and antennas at the base station (BS), respectively. To accurately estimate such a large number of channel coefficients within a short time interval, we propose a novel three-phase pilot-based channel estimation framework in this paper for IRS-assisted uplink multiuser communications. Under this framework, we analytically prove that a time duration consisting of $K+N+\max(K-1,\lceil (K-1)N/M \rceil)$ pilot symbols is sufficient for the BS to perfectly recover all the $KMN+KM$ channel coefficients for the case without receiver noise at the BS. In contrast to the channel estimation for conventional uplink communications without IRS where the minimum channel estimation time is independent of the number of receive antennas at the BS, our result reveals the crucial role of massive MIMO (multiple-input multiple-output) in reducing the channel estimation time for IRS-assisted communications. Further, for the case with receiver noise, the user pilot sequences, IRS reflecting coefficients, and BS linear minimum mean-squared error (LMMSE) channel estimators are characterized in closed-form, and the corresponding estimation mean-squared error (MSE) is quantified.


Figure 3 illustrates a few of such basis functions for the generative decoder of the IM-Net model #REFR .

In representation learning the aim is to explain observations with a small number of latent variables #OTHEREFR , which we will denote by l.

Generative models f (x; l, Θ), such as (variational) auto-encoders or generative adversarial networks, attempt to map novel latent codes to novel outputs within the same distribution as the observations.

When interpolating between two latent codes, the appearance of the generated output changes continuously #OTHEREFR .

For a generative neural implicit shape model, the basis for the boundary movement is locally described by b = −∇ l f / ∇ x f (Equation 3).

The decoder is trained as part of an auto-encoder reconstructing the entire ShapeNet dataset #OTHEREFR from l ∈ R 256 latent variables.

We use a pretrained model available at https://github.com/czq142857/ IM-NET-pytorch, which is implemented as an MLP with 8 layers of 1024 neurons each and leaky-ReLU activations.

These basis functions can be observed to encode semantics, such as rotation and reflection symmetries, and segment semantically meaningful parts, such as windscreen on the car, rim of the lamp and different surfaces on chairs.

For similar parts, such as the two chairs, the bases show similar behaviour.

However, it is known that not all directions in the latent space are (equally) viable #OTHEREFR , hence, a similar argument can be made for the bases.

Neural Implicit Shape Editing Using Boundary Sensitivity

Abstract: Neural fields are receiving increased attention as a geometric representation due to their ability to compactly store detailed and smooth shapes and easily undergo topological changes. Compared to classic geometry representations, however, neural representations do not allow the user to exert intuitive control over the shape. Motivated by this, we leverage boundary sensitivity to express how perturbations in parameters move the shape boundary. This allows to interpret the effect of each learnable parameter and study achievable deformations. With this, we perform geometric editing: finding a parameter update that best approximates a globally prescribed deformation. Prescribing the deformation only locally allows the rest of the shape to change according to some prior, such as semantics or deformation rigidity. Our method is agnostic to the model its training and updates the NN in-place. Furthermore, we show how boundary sensitivity helps to optimize and constrain objectives (such as surface area and volume), which are difficult to compute without first converting to another representation, such as a mesh.

Learning Implicit Fields For Generative Shape Modeling

Abstract: We advocate the use of implicit fields for learning generative models of shapes and introduce an implicit field decoder, called IM-NET, for shape generation, aimed at improving the visual quality of the generated shapes. An implicit field assigns a value to each point in 3D space, so that a shape can be extracted as an iso-surface. IM-NET is trained to perform this assignment by means of a binary classifier. Specifically, it takes a point coordinate, along with a feature vector encoding a shape, and outputs a value which indicates whether the point is outside the shape or not. By replacing conventional decoders by our implicit decoder for representation learning (via IM-AE) and shape generation (via IM-GAN), we demonstrate superior results for tasks such as generative shape modeling, interpolation, and single-view 3D reconstruction, particularly in terms of visual quality. Code and supplementary material are available at https://github.com/czq142857/implicit-decoder.


Figure 5 gives the contents of the views in the view tree from Figure 2 , where R, S, and T are relations over a ring D with payloads p i ∈ D for i ∈ #REFR .

If X is a free variable, however, then we retain it in the view schema without applying the lifting function to its values.

The schema of the view consists of dep(X) and the free variables in the subtree of ω rooted at X.

τ (variable order ω, free variables F) : view tree switch ω: Figure 3 : Creating a view tree τ (ω, F) for a variable order ω and a set of free variables F. Example 8.

Figure 2 shows the view tree constructed by the function τ from Figure 3 over the variable order ω and the empty set of free variables.

Figure 4 depicts the view tree constructed over the same variable order but for the set F = {A, C} of free variables.

Assume that D is the Z ring, each tuple in these relations is mapped to 1, i.e., p i = 1 for i ∈ #OTHEREFR , and the lifting functions map all values to 1.

Then, the view tree computes the COUNT query from Example 3 and the root view V @A

.ω k let T i = τ (ω i , F), ∀i ∈ [k] let V @ω i rels i [keys i ] = root of T i , ∀i ∈ [k] let keys = dep(X) ∪ (F ∩ vars(ω)) let rels = i∈[k] rels i if X / ∈ F V @X rels [keys] = X i∈[k] V @ω i rels i [keys i ] else V @X rels [keys] = i∈[k] V @ω i rels i [keys i ] return      V @X rels [keys] T 1 . . . T k

V @A RST [A, C ] V @B R [A] V @C ST [A, C ] V @D T [C ] V @E S [A, C ] R[A, B ] S[A, C , E ] T[C , D] V @A RST [A, C ] = V @B R [A] ⊗ V @C ST [A, C ] V @C ST [A, C ] = V @D T [C ] ⊗ V @E S [A, C ] V @E S [A, C ] = E S[A, C , E ] V @B R [A] = B R[A, B ] V @D T [C ] = D T[C , D]

F-Ivm: Analytics Over Relational Databases Under Updates

Abstract: This article describes F-IVM, a unified approach for maintaining analytics over changing relational data. We exemplify its versatility in four disciplines: processing queries with group-by aggregates and joins; learning linear regression models using the covariance matrix of the input features; building Chow-Liu trees using pairwise mutual information of the input features; and matrix chain multiplication. F-IVM has three main ingredients: higher-order incremental view maintenance; factorized computation; and ring abstraction. F-IVM reduces the maintenance of a task to that of a hierarchy of simple views. Such views are functions mapping keys, which are tuples of input values, to payloads, which are elements from a ring. F-IVM also supports efficient factorized computation over keys, payloads, and updates. Finally, F-IVM treats uniformly seemingly disparate tasks. In the key space, all tasks require joins and variable marginalization. In the payload space, tasks differ in the definition of the sum and product ring operations. We implemented F-IVM on top of DBToaster and show that it can outperform classical first-order and fully recursive higher-order incremental view maintenance by orders of magnitude while using less memory.

Answering Conjunctive Queries Under Updates

Abstract: We consider the task of enumerating and counting answers to k-ary conjunctive queries against relational databases that may be updated by inserting or deleting tuples. We exhibit a new notion of q-hierarchical conjunctive queries and show that these can be maintained efficiently in the following sense. During a linear time pre-processing phase, we can build a data structure that enables constant delay enumeration of the query results; and when the database is updated, we can update the data structure and restart the enumeration phase within constant time. For the special case of self-join free conjunctive queries we obtain a dichotomy: if a query is not q-hierarchical, then query enumeration with sublinear *) delay and sublinear update time (and arbitrary preprocessing time) is impossible. For answering Boolean conjunctive queries and for the more general problem of counting the number of solutions of k-ary queries we obtain complete dichotomies: if the query's homomorphic core is q-hierarchical, then size of the the query result can be computed in linear time and maintained with constant update time. Otherwise, the size of the query result cannot be maintained with sublinear update time. All our lower bounds rely on the OMv-conjecture, a conjecture on the hardness of online matrix-vector multiplication that has recently emerged in the field of fine-grained complexity to characterise the hardness of dynamic problems. The lower bound for the counting problem additionally relies on the orthogonal vectors conjecture, which in turn is implied by the strong exponential time hypothesis.*) By sublinear we mean O(n(1-e) for some e > 0, where n is the size of the active domain of the current database.


We conclude that the probabilistic objective (12) substantially reduces bias compared with the non-probabilistic DSLA affordance objective #REFR .

Ablation studies We evaluate the advantage of our proposed predictive world modeling approach [15] for learning navigational patterns from sampled plausible completed worldsx instead of partially observed worlds x.

We conduct an experiment using unaugmented samples to quantify the performance contribution of our geometric data augmentation method #OTHEREFR on real-world data.

We conduct experiments on dataset splits including a different number of regions to estimate how performance increases with additional data or experience.

Table I presents evaluation results for the predicted probability fields.

Our proposed DSLP model optimized with the information balance regularizer α IB (9) predicts the least biased probability field among all models trained and evaluated on accumulated past observation inputs.

Training and evaluating on accumulated past and future observation inputs in an offline map creation manner (i.e.

full obs) reduces bias, demonstrating that more comprehensively observed environments result in better performance.

We performed experiments with different constant α values to #OTHEREFR .

The best constant weight α value 0.1, found over five hyperparameter experiments, result in worse performance than using α IB .

We demonstrate the merit of dynamic, per-sample computed α IB values (9) by running an experiment with the constant mean α IB value 0.122 computed over all training samples, which results in worse performance. See Fig. 7 for probability field visualizations.

Learning To Predict Navigational Patterns From Partial Observations

Abstract: Human beings cooperatively navigate ruleconstrained environments by adhering to mutually known navigational patterns, which may be represented as directional pathways or road lanes. Inferring these navigational patterns from incompletely observed environments is required for intelligent mobile robots operating in unmapped locations. However, algorithmically defining these navigational patterns is nontrivial. This paper presents the first self-supervised learning (SSL) method for learning to infer navigational patterns in realworld environments from partial observations only. We explain how geometric data augmentation, predictive world modeling, and an information-theoretic regularizer enables our model to predict an unbiased local directional soft lane probability (DSLP) field in the limit of infinite data. We demonstrate how to infer global navigational patterns by fitting a maximum likelihood graph to the DSLP field. Experiments show that our SSL model outperforms two SOTA supervised lane graph prediction models on the nuScenes dataset. We propose our SSL method as a scalable and interpretable continual learning paradigm for navigation by perception. Code released upon publication.

Learning A Directional Soft Lane Affordance Model For Road Scenes Using Self-Supervision

Abstract: Humans navigate complex environments in an organized yet flexible manner, adapting to the context and implicit social rules. Understanding these naturally learned patterns of behavior is essential for applications such as autonomous vehicles. However, algorithmically defining these implicit rules of human behavior remains difficult. This work proposes a novel self-supervised method for training a probabilistic network model to estimate the regions humans are most likely to drive in as well as a multimodal representation of the inferred direction of travel at each point. The model is trained on individual human trajectories conditioned on a representation of the driving environment. The model is shown to successfully generalize to new road scenes, demonstrating potential for real-world application as a prior for socially acceptable driving behavior in challenging or ambiguous scenarios which are poorly handled by explicit traffic rules.


In all cases of generator tuning, CSIM scores are higher than the zero-shot case and comparable with each other since we minimize the perceptual distance between the reference and generated frame #REFR .

In contrast to existing lip-sync generative models, we introduce poseaware masking for lip region by utilizing a 3D parametric mesh predictor, which inherits the pose information in the mask itself.

Designing a smooth lip motion by using the moving-average based latent smoothing in the continuous and linear latent space, StyleLipSync can generate temporally consistent lip motion.

Furthermore, we propose a fewshot lip-sync adaptation method for a single person who lies in the out-of-distribution, which uses a few seconds of the target person's video.

Experimental results show that our StyleLipSync can generate realistic lip-sync video from arbitrary audio even with the zero-shot setting and the proposed adaptation method enhance the person-specific information without losing the lip-sync generality. Unseen Face Adaptation.

We plot the CSIM [9] scores in Figure 9 under the same set-up in Figure 7 (b) to further support the necessity of the sync regularizer R sync .

However, the sync regularizer R sync enforces maintaining the lip-sync generality, as illustrated in Figure 7 Figure 9 .

Cosine similarity between face embedding (CSIM) #OTHEREFR tendency of the adaptation method.

Stylelipsync: Style-Based Personalized Lip-Sync Video Generation

Abstract: In this paper, we present StyleLipSync, a style-based personalized lip-sync video generative model that can generate identity-agnostic lip-synchronizing video from arbitrary audio. To generate a video of arbitrary identities, we leverage expressive lip prior from the semantically rich latent space of a pre-trained StyleGAN, where we can also design a video consistency with a linear transformation. In contrast to the previous lip-sync methods, we introduce pose-aware masking that dynamically locates the mask to improve the naturalness over frames by utilizing a 3D parametric mesh predictor frame by frame. Moreover, we propose a few-shot lip-sync adaptation method for an arbitrary person by introducing a sync regularizer that preserves lips-sync generalization while enhancing the person-specific visual information. Extensive experiments demonstrate that our model can generate accurate lip-sync videos even with the zeroshot setting and enhance characteristics of an unseen face using a few seconds of target video through the proposed adaptation method. Please refer to our project page.

The Unreasonable Effectiveness Of Deep Features As A Perceptual Metric

Abstract: While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on the ImageNet classification task has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called"perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new Full Reference Image Quality Assessment (FR-IQA) dataset of perceptual human judgments, orders of magnitude larger than previous datasets. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by huge margins. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.


Note that instance segmentation for the airplane category is not reported because PartNet #REFR does not have ground-truth labels. The results are reported in Tab. 3.

Since our bounding boxes are based on part-level segmentation, we also measure its performance by measuring the mean average precision (mAP) of instance segmentation with the labels obtained from PartNet #OTHEREFR .

In addition to HA [34] and CA [37], we also compare with BSP-Net #OTHEREFR which was used to get our initial segment.

We follow the evaluation setting of PartNet #OTHEREFR to report the instance segmentation results.

The segmentation was obtained by identifying the nearest point for each primitive and assigning them to obtain the segmentation mask.

Subsequently, we computed the IoU between each segmentation mask and the ground-truth mask and considered a segmentation mask as true positive only when IoU was greater than 0.5.

Compared to other baselines we achieve the best result showing that SMART captures the individual parts in the shape better than other works having more alignment with the human perception of the shape decomposition.

Split, Merge, And Refine: Fitting Tight Bounding Boxes Via Learned Over-Segmentation And Iterative Search

Abstract: Figure 1: Our framework dubbed SMART finds a set of tight bounding boxes of a 3D shape. Given a 3D shape represented as a volumetric mesh, it first takes over-segmentation from the output of BSP-Net [8] and post-processing and then performs hierarchical merging with tightness-aware criteria (second row). Then, it refines the bounding box parameters with a sequence of actions in a discrete space (third row). The results are further improved with long-sighted exploration by our MCTS accelerated with our technique (last row).

Partnet: A Large-Scale Benchmark For Fine-Grained And Hierarchical Part-Level 3D Object Understanding

Abstract: We present PartNet: a consistent, large-scale dataset of 3D objects annotated with fine-grained, instance-level, and hierarchical 3D part information. Our dataset consists of 573,585 part instances over 26,671 3D models covering 24 object categories. This dataset enables and serves as a catalyst for many tasks such as shape analysis, dynamic 3D scene modeling and simulation, affordance analysis, and others. Using our dataset, we establish three benchmarking tasks for evaluating 3D part recognition: fine-grained semantic segmentation, hierarchical semantic segmentation, and instance segmentation. We benchmark four state-of-the-art 3D deep learning algorithms for fine-grained semantic segmentation and three baseline methods for hierarchical semantic segmentation. We also propose a baseline method for part instance segmentation and demonstrate its superior performance over existing methods.


To generate a dataset that probes for model understanding on objects and their relationship to contextual cues found in an image's background, we collected samples from MSCOCO #REFR consisting of 80 objects.

These samples are used to probe model reliance on background cues and reliance on co-occurrence between objects.

An overview of Probe-B is shown in Figure 2 .

For each group, there is an unmodified image x 0 , an image with a random patch on the backgroundx 0 , a modified image where the background is removedx 1 and 80 or less prompts.

We have two splits in this data, the first removing the background but keeping all objects Probe-B M R and the other removing both background and all other objects Probe-B R .

Probe-B R aims to probe models on whether they use conceptual maps on object co-occurrence to improve recognition.

Probing Conceptual Understanding Of Large Visual-Language Models

Abstract: We present a novel framework for probing and improving relational, compositional and contextual understanding of large visual-language models (V+L). While large V+L models have achieved success in various downstream tasks, it is not clear if they have a conceptual grasp of the content. We propose a novel benchmarking dataset for probing three aspects of content understanding. Our probes are grounded in cognitive science and help determine if a V+L model can, for example, determine if snow garnished with a man is implausible, or if it can identify beach furniture by knowing it is located on a beach. We have experimented with 5 well known models, such as CLIP and ViLT, and found that they mostly fail to demonstrate a conceptual understanding. That said, we find interesting insights such as cross-attention helps learning conceptual understandingWe use these insights to propose a new finetuning technique that rewards the three conceptual understanding measures we proposed. We hope that the presented benchmarks will help the community assess and improve the conceptual understanding capabilities of large V+L models. Visual-language (V+L) models are recently and greatly successful deep learning models that learn representations of image and text in a shared space. These representations are useful for downstream tasks like image classification, visual-question answering, image retrieval and more [42, 37, 27, 26, 39, 1]. However, for use in real-world applications, it is also vital that models "understand" rather than memorize to perform on more general tasks [24]. While large-language models have been shown to have a moderate amount of "theory of mind", as measured by conceptual consistency [29], V+L models have not

Microsoft Coco: Common Objects In Context

Abstract: We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.


Different from DPLTM #REFR , our method does not have the privacy budget as we do not use the private data while generating the model structure.

Unlike the selection method in DPLTM #OTHEREFR , for the winning ticket selection, there are M alternative tickets, and each ticket is given a score V (w j , D pub ) that equals to the correct number of samples for inference.

To adjust the trade-off between the accuracy and the degree of network pruning, we use the softmax function #OTHEREFR over the preference value V (w j , D pub ) also known as the score to select WT, which ensures that all WTs are explored:

where j denotes one of possibly many winning tickets and P j is its corresponding probability.

It is worth noting that we use LTH to generate WTs on the public dataset, and each client performs the training process with DP on their private data.

Efficient Federated Learning With Enhanced Privacy Via Lottery Ticket Pruning In Edge Computing

Abstract: Federated learning (FL) is a collaborative learning paradigm for decentralized private data from mobile terminals (MTs). However, it suffers from issues in terms of communication, resource of MTs, and privacy. Existing privacy-preserving FL methods usually adopt the instance-level differential privacy (DP), which provides a rigorous privacy guarantee but with several bottlenecks: severe performance degradation, transmission overhead, and resource constraints of edge devices such as MTs. To overcome these drawbacks, we propose Fed-LTP, an efficient and privacy-enhanced FL framework with Lottery Ticket Hypothesis (LTH) and zero-concentrated DP (zCDP). It generates a pruned global model on the server side and conducts sparse-to-sparse training from scratch with zCDP on the client side. On the server side, two pruning schemes are proposed: (i) the weight-based pruning (LTH) determines the pruned global model structure; (ii) the iterative pruning further shrinks the size of the pruned model's parameters. Meanwhile, the performance of Fed-LTP is also boosted via model validation based on the Laplace mechanism. On the client side, we use sparse-to-sparse training to solve the resource-constraints issue and provide tighter privacy analysis to reduce the privacy budget. We evaluate the effectiveness of Fed-LTP on several real-world datasets in both independent and identically distributed (IID) and non-IID settings. The results clearly confirm the superiority of Fed-LTP over state-of-the-art (SOTA) methods in communication, computation, and memory efficiencies while realizing a better utility-privacy trade-off.

The Differentially Private Lottery Ticket Mechanism

Abstract: We propose the differentially private lottery ticket mechanism (DPLTM). An end-to-end differentially private training paradigm based on the lottery ticket hypothesis. Using "high-quality winners", selected via our custom score function, DPLTM significantly improves the privacy-utility trade-off over the state-of-the-art. We show that DPLTM converges faster, allowing for early stopping with reduced privacy budget consumption. We further show that the tickets from DPLTM are transferable across datasets, domains, and architectures. Our extensive evaluation on several public datasets provides evidence to our claims.


It was later extended to the iterative version of FGSM (I-FGSM) #REFR , which generates adversarial examples that fool the model θ with a higher success rate.

Given a pretrained DNN θ, the adversarial attack is implemented by solving the following optimization problem:

where the L(x, y, θ) denotes the loss function that guides the DNN θ to predict the class of image x as ground-truth y, is the maximum allowed magnitude of perturbation, and x * is the generated adversarial example. Some gradient-based optimizers were proposed to solve (1). Szegedy et al.

were the first to use a box-constrained L-BFGS optimizer #OTHEREFR .

Then, a one-step optimizer, i.e., fast gradient sign method (FGSM) was proposed to efficiently generate adversarial examples #OTHEREFR .

The generated adversarial example is shown to have the ability to attack an unknown model, and this ability is called the transferability of the adversarial example.

In this scenario, the model θ used to generate the adversarial example is called the surrogate model, and the unknown model to be attacked is called the victim model.

This type of adversarial attack is called the transfer-based attack. Dong et al.

systematically investigated the transferability of adversarial examples and found an interesting phenomenon that is although the iterative method (I-FGSM) is a stronger optimizer than the one-step method (FGSM), the transferability of the adversarial examples generated by I-FGSM is worse than FGSM #OTHEREFR .

Specifically, their experimental results on the white-box attacks show that an Inception-v3 has only a 72.3% probability of being fooled by FGSM, but a 100% probability of being fooled by I-FGSM.

Generating Adversarial Examples With Better Transferability Via Masking Unimportant Parameters Of Surrogate Model

Abstract: Deep neural networks (DNNs) have been shown to be vulnerable to adversarial examples. Moreover, the transferability of the adversarial examples has received broad attention in recent years, which means that adversarial examples crafted by a surrogate model can also attack unknown models. This phenomenon gave birth to the transfer-based adversarial attacks, which aim to improve the transferability of the generated adversarial examples. In this paper, we propose to improve the transferability of adversarial examples in the transfer-based attack via masking unimportant parameters (MUP). The key idea in MUP is to refine the pretrained surrogate models to boost the transfer-based attack. Based on this idea, a Taylor expansionbased metric is used to evaluate the parameter importance score and the unimportant parameters are masked during the generation of adversarial examples. This process is simple, yet can be naturally combined with various existing gradient-based optimizers for generating adversarial examples, thus further improving the transferability of the generated adversarial examples. Extensive experiments are conducted to validate the effectiveness of the proposed MUP-based methods.

Adversarial Examples In The Physical World

Abstract: Most existing machine learning classifiers are highly vulnerable to adversarial examples. An adversarial example is a sample of input data which has been modified very slightly in a way that is intended to cause a machine learning classifier to misclassify it. In many cases, these modifications can be so subtle that a human observer does not even notice the modification at all, yet the classifier still makes a mistake. Adversarial examples pose security concerns because they could be used to perform an attack on machine learning systems, even if the adversary has no access to the underlying model. Up to now, all previous work have assumed a threat model in which the adversary can feed data directly into the machine learning classifier. This is not always the case for systems operating in the physical world, for example those which are using signals from cameras and other sensors as an input. This paper shows that even in such physical world scenarios, machine learning systems are vulnerable to adversarial examples. We demonstrate this by feeding adversarial images obtained from cell-phone camera to an ImageNet Inception classifier and measuring the classification accuracy of the system. We find that a large fraction of adversarial examples are classified incorrectly even when perceived through the camera.


It is an important step towards improving model interpretability as most of the deep neural networks are typically over-confident in their predictions #REFR .

Confidence calibration is the problem of matching the output confidence level with the actual likelihood of the model.

Of note, A model is said to be perfectly calibrated when the confidence level of a prediction represents the true probability of the prediction being correct #OTHEREFR .

Mathematically speaking, if input X is considered with class labels Y , the predicted class isŶ andP is its associated confidence, then for perfect calibration, the probability Pis:

where the probability is over the joint distribution.

1) Temperature Scaling: Temperature scaling is the simplest extension of Platt scaling #OTHEREFR and uses a single parameter T > 0 for all cases. Guo et al.

Towards Reliable Colorectal Cancer Polyps Classification Via Vision Based Tactile Sensing And Confidence-Calibrated Neural Networks

Abstract: In this study, toward addressing the over-confident outputs of existing artificial intelligence-based colorectal cancer (CRC) polyp classification techniques, we propose a confidencecalibrated residual neural network. Utilizing a novel visionbased tactile sensing (VS-TS) system and unique CRC polyp phantoms, we demonstrate that traditional metrics such as accuracy and precision are not sufficient to encapsulate model performance for handling a sensitive CRC polyp diagnosis. To this end, we develop a residual neural network classifier and address its over-confident outputs for CRC polyps classification via the post-processing method of temperature scaling. To evaluate the proposed method, we introduce noise and blur to the obtained textural images of the VSTS and test the model's reliability for non-ideal inputs through reliability diagrams and other statistical metrics.

On Calibration Of Modern Neural Networks

Abstract: Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.


When the distribution is mostly unimodal, we can obtain a measure of uncertainty as the covariance of P around ξ * #REFR .

where ξ(p) transforms a 2D point p from BEV to map coordinate frame.

The confidence C masks the correlation to ignore some parts of the BEV space, such as occluded areas.

This formulation benefits from an efficient implementation by rotating T K times and performing a single convolution as a batched multiplication in the Fourier domain #OTHEREFR .

Pose inference: We estimate a single pose by maximum likelihood: ξ * = argmax ξ P (ξ|I, map, ξ prior ).

We train a single model that generalizes well across many datasets.

OrienterNet handles different cameras, street-level viewpoints, and unseen cities and is thus suitable for both AR and robotics.

Overlayed on the input maps, the single-image predictions (black arrow ) are close to the ground truth (red arrow) and more accurate than the noisy GPS (blue dot •).

The model effectively leverages building corners and boundaries, crosswalks, sidewalks, road intersections, trees, and other common urban objects.

Likelihood maps can be multi-modal in scenes with repeated elements (last two rows) -we show the predicted orientations at local maxima as small arrows. •GPS •single-frame •sequence •GT Figure 5 . Multi-frame fusion resolves ambiguities.

Orienternet: Visual Localization In 2D Public Maps With Neural Matching

Abstract: Humans can orient themselves in their 3D environments using simple 2D maps. Differently, algorithms for visual localization mostly rely on complex 3D point clouds that are expensive to build, store, and maintain over time. We bridge this gap by introducing OrienterNet, the first deep neural network that can localize an image with sub-meter accuracy using the same 2D semantic maps that humans use. Orienter-Net estimates the location and orientation of a query image by matching a neural Bird's-Eye View with open and globally available maps from OpenStreetMap, enabling anyone to localize anywhere such maps are available. OrienterNet is supervised only by camera poses but learns to perform semantic matching with a wide range of map elements in an end-to-end manner. To enable this, we introduce a large crowd-sourced dataset of images captured across 12 cities from the diverse viewpoints of cars, bikes, and pedestrians. OrienterNet generalizes to new datasets and pushes the state of the art in both robotics and AR scenarios. The code and trained model will be released publicly.

Masking By Moving: Learning Distraction-Free Radar Odometry From Pose Information

Abstract: This paper presents an end-to-end radar odometry system which delivers robust, real-time pose estimates based on a learned embedding space free of sensing artefacts and distractor objects. The system deploys a fully differentiable, correlation-based radar matching approach. This provides the same level of interpretability as established scan-matching methods and allows for a principled derivation of uncertainty estimates. The system is trained in a (self-)supervised way using only previously obtained pose information as a training signal. Using 280km of urban driving data, we demonstrate that our approach outperforms the previous state-of-the-art in radar odometry by reducing errors by up 68% whilst running an order of magnitude faster.


Rq2: Can Our Classifier Learn Vulnerability In A Biased Setting?

Compared to the next best model #REFR , our model with Focal Loss shows an improvement of 22.91% in Precision, 23.53% in Recall, and 18.04% on F1 score.

Observing the last two rows in Table 5 , we find that, initially, we tested our model using Weighted Loss (WL) only, and later we tested with Focal Loss, which is a combination of weighted loss α with the hyperparameter γ.

In both cases, we observe that our model has surpassed previous models in terms of Precision, Recall, and F1 score, indicating lower false positive and false negative rates with WL and with FL.

However, these numbers improved slightly when compared with performance between weighted loss and Focal Loss, with the exception of the precision metric.

From the results, it is observed that with Focal Loss, we achieved an improvement of 11.18% in Precision, 1.06% in Recall, and 0.61% on our F1 score compared to the weighted loss.

Nevertheless, using either weighted loss or Focal Loss, the accuracy of our model drops by almost 1% compared to #OTHEREFR , indicating that these previous models are highly biased with an assumption of non-vulnerability.

Thus, Focal Loss improves precision, recall and F1 on imbalanced data while not causing a negating impact on a balanced dataset only by adjusting the parameters α and γ RQ3: Is our classifier generalized enough to detect vulnerabilities in N-day and zero-day program sam- We evaluated our classifier's performance based on its ability to accurately predict vulnerability with 273 Nday real-world sample programs. These sample programs are never used during training.

We also used 4 zero-day examples in order to evaluate our classifier on predicting zero-day vulnerabilities as well.

The classifier predicts the vulnerability classif the vulnerability exists in the code and predicts non-vulnerable when the vulnerability does not exist.

Out of these 273 N-day and 4, zero-day code samples, some vulnerability classes exist that are not part of our VulF dataset from table 1.

For example, the VulF dataset has no code samples for CWE-787 for training.

An Unbiased Transformer Source Code Learning With Semantic Vulnerability Graph

Abstract: Over the years, open-source software systems have become prey to threat actors. Even highly-adopted software has been crippled by unforeseeable attacks, leaving millions of devices exposed. Even as open-source communities act quickly to patch the breach, code vulnerability screening should be an integral part of agile software development from the beginning. Unfortunately, current vulnerability screening techniques are ineffective at identifying novel vulnerabilities or providing developers with code vulnerability and classification. Furthermore, the datasets used for vulnerability learning often exhibit distribution shifts from the real-world testing distribution due to novel attack strategies deployed by adversaries and as a result, the machine learning model's performance may be hindered or biased. To address these issues, we propose a joint interpolated multitasked unbiased vulnerability classifier comprising a transformer "RoBERTa" and graph convolution neural network (GCN). We present a training process utilizing a semantic vulnerability graph (SVG) representation from source code, created by integrating edges from a sequential flow, control flow, and data flow, as well as a novel flow dubbed Poacher Flow (PF). Poacher flow edges reduce the gap between dynamic and static program analysis and handle complex long-range dependencies. Moreover, our approach reduces biases of classifiers regarding unbalanced datasets by integrating Focal Loss objective function along with SVG. Remarkably, experimental results show that our classifier outperforms state-of-the-art results on vulnerability detection with fewer false negatives and false positives. After testing our model across multiple datasets, it shows an improvement of at least 2.41% and 18.75% in the bestcase scenario. Evaluations using N-day program samples demonstrate that our proposed approach achieves a 93% accuracy and was able to detect 4, zero-day vulnerabilities from popular GitHub repositories. Our code and data are

Devign: Effective Vulnerability Identification By Learning Comprehensive Program Semantics Via Graph Neural Networks

Abstract: Vulnerability identification is crucial to protect the software systems from attacks for cyber security. It is especially important to localize the vulnerable functions among the source code to facilitate the fix. However, it is a challenging and tedious process, and also requires specialized security expertise. Inspired by the work on manually-defined patterns of vulnerabilities from various code representation graphs and the recent advance on graph neural networks, we propose Devign, a general graph neural network based model for graph-level classification through learning on a rich set of code semantic representations. It includes a novel Conv module to efficiently extract useful features in the learned rich node representations for graph-level classification. The model is trained over manually labeled datasets built on 4 diversified large-scale open-source C projects that incorporate high complexity and variety of real source code instead of synthesis code used in previous works. The results of the extensive evaluation on the datasets demonstrate that Devign outperforms the state of the arts significantly with an average of 10.51% higher accuracy and 8.68\% F1 score, increases averagely 4.66% accuracy and 6.37% F1 by the Conv module.