One exception to this is the popularity lift metric in the case of music recommendations; however, this result is in line with our previous work #REFR , in which repeat consumption settings have been studied.

In this paper, we have studied the interconnection between accuracy, miscalibration, and popularity bias for different user groups in three different domains.

Here, we measured popularity bias in terms of popularity lift, a metric that compares the popularity of items in recommendation lists to the popularity of items in user profiles.

Additionally, we investigated miscalibration, a metric that compares the genre spectrums in user profiles with the ones in recommendation lists.

We find that, in general, the inconsistency of recommendations in terms of miscalibration and popularity lift is aligned with lower accuracy performance.

Additionally, we find that different genres contribute differently to miscalibration and popularity lift.

That finding is particularly pronounced in the case of anime recommendations for LowPop users and for genres that are unpopular among other user groups.

Another contribution of our work is that we publicly share our datasets and source code investigated in this study with the research community.

One limitation of our work is that we have focused solely on datasets from the multimedia/entertainment domains, namely music, movies, and animes.

A Study On Accuracy, Miscalibration, And Popularity Bias In Recommendations

Abstract: Recent research has suggested different metrics to measure the inconsistency of recommendation performance, including the accuracy difference between user groups, miscalibration, and popularity lift. However, a study that relates miscalibration and popularity lift to recommendation accuracy across different user groups is still missing. Additionally, it is unclear if particular genres contribute to the emergence of inconsistency in recommendation performance across user groups. In this paper, we present an analysis of these three aspects of five well-known recommendation algorithms for user groups that differ in their preference for popular content. Additionally, we study how different genres affect the inconsistency of recommendation performance, and how this is aligned with the popularity of the genres. Using data from Last.fm, MovieLens, and MyAnimeList, we present two key findings. First, we find that users with little interest in popular content receive the worst recommendation accuracy and that this is aligned with miscalibration and popularity lift. Second, our experiments show that particular genres contribute to a different extent to the inconsistency of recommendation performance, especially in terms of miscalibration in the case of the MyAnimeList dataset.

The Unfairness Of Popularity Bias In Music Recommendation: A Reproducibility Study

Abstract: Research has shown that recommender systems are typically biased towards popular items, which leads to less popular items being underrepresented in recommendations. The recent work of Abdollahpouri et al. in the context of movie recommendations has shown that this popularity bias leads to unfair treatment of both long-tail items as well as users with little interest in popular items. In this paper, we reproduce the analyses of Abdollahpouri et al. in the context of music recommendation. Specifically, we investigate three user groups from the this http URL music platform that are categorized based on how much their listening preferences deviate from the most popular music among all this http URL users in the dataset: (i) low-mainstream users, (ii) medium-mainstream users, and (iii) high-mainstream users. In line with Abdollahpouri et al., we find that state-of-the-art recommendation algorithms favor popular items also in the music domain. However, their proposed Group Average Popularity metric yields different results for this http URL than for the movie domain, presumably due to the larger number of available items (i.e., music artists) in the this http URL dataset we use. Finally, we compare the accuracy results of the recommendation algorithms for the three user groups and find that the low-mainstreaminess group significantly receives the worst recommendations.


B. Integrated Sensing And Communication (Isac)

To prevent RX saturation in mono-static shared-antenna BS deployments and reduce the inherent masking effect of the direct leakage sidelobes, efficient analog and digital SIC solutions, tailored to the orthogonal frequency division multiplexing (OFDM) radar use, were introduced in #REFR .

In-band FD is being considered as a key enabling technology for ISAC applications due to its STAR capability, with promising theoretical results demonstrating advantages of joint design #OTHEREFR .

State-of-the-art: In #OTHEREFR , #OTHEREFR , single-antenna FD systems were considered for ISAC operation, where both communication and radar waveforms were optimized for sensing performance.

Frequency-domain radar processing building on the' LTE or the 5G New Radio (NR) time-frequency resource grid, and complemented with interpolation to account for missing samples due to the null subcarriers within the transmit waveform passband, was presented in #OTHEREFR .

It was shown that the large channel bandwidths supported by 5G NR provide good sensing performance already at below 6 GHz frequencies.

With 100 MHz channel bandwidth, range estimation accuracy in the order of 1 m and target detection probability exceeding 90% were shown to be feasible at signalto-noise ratio (SNR) values lower than −30 dB.

The various provided measurements verified successful sensing of static and moving targets, such as drones and cars, while evidencing measured TX/RX isolation at the FD node of approximately 100 dB.

Apart from the typical optimization of the UL and DL sum rate, FD radios have been considered to continuously update the channel state information at the transmitter (CSIT), which is required to compute the DL precoding matrix in multiuser (massive) MIMO systems, as schematically shown in Fig. 9 .

By design principle, FD operation leverages channel reciprocity for open-loop UL training to estimate the DL channels. However, the UL transmission of training might Fig. 9 .

FD-enabled simultaneous multi-user DL data transmission and channel estimation in the UL, leveraging TDD. create interference at the DL receiving mobile nodes.

Most predominantly, the DL transmission contaminates the reception of the training symbols in the UL.

Full-Duplex Wireless For 6G: Progress Brings New Opportunities And Challenges

Abstract: The use of in-band full-duplex (FD) enables nodes to simultaneously transmit and receive on the same frequency band, which challenges the traditional assumption in wireless network design. The full-duplex capability enhances spectral efficiency and decreases latency, which are two key drivers pushing the performance expectations of next-generation mobile networks. In less than ten years, in-band FD has advanced from being demonstrated in research labs to being implemented in standards, presenting new opportunities to utilize its foundational concepts. Some of the most significant opportunities include using FD to enable wireless networks to sense the physical environment, integrate sensing and communication applications, develop integrated access and backhaul solutions, and work with smart signal propagation environments powered by reconfigurable intelligent surfaces. However, these new opportunities also come with new challenges for large-scale commercial deployment of FD technology, such as managing self-interference, combating cross-link interference in multi-cell networks, and coexistence of dynamic time division duplex, subband FD and FD networks.

Full-Duplex Ofdm Radar With Lte And 5G Nr Waveforms: Challenges, Solutions, And Measurements

Abstract: This article studies the processing principles, implementation challenges, and performance of orthogonal frequency-division multiplexing (OFDM)-based radars, with particular focus on the fourth-generation Long-Term Evolution (LTE) and fifth-generation (5G) New Radio (NR) mobile networks’ base stations and their utilization for radar/sensing purposes. First, we address the problem stemming from the unused subcarriers within the LTE and NR transmit signal passbands and their impact on frequency-domain radar processing. In particular, we formulate and adopt a computationally efficient interpolation approach to mitigate the effects of such empty subcarriers in the radar processing. We evaluate the target detection and the corresponding range and velocity estimation performance through computer simulations and show that high-quality target detection as well as high-precision range and velocity estimation can be achieved. In particular, 5G NR waveforms, through their impressive channel bandwidths and configurable subcarrier spacing, are shown to provide very good radar/sensing performance. Then, a fundamental implementation challenge of transmitter–receiver (TX–RX) isolation in OFDM radars is addressed, with specific emphasis on shared-antenna cases, where the TX–RX isolation challenges are the largest. It is confirmed that from the OFDM radar processing perspective, limited TX–RX isolation is primarily a concern in the detection of static targets, while moving targets are inherently more robust to transmitter self-interference (SI). Properly tailored analog/RF and digital SI cancellation solutions for OFDM radars are also described and implemented and shown through RF measurements to be key technical ingredients for practical deployments, particularly from static and slowly moving targets’ point of view.

mono-static shared-antenna BS
OFDM) radar


In contrast to generative adversarial network (GAN) based methods #REFR , diffusion models offer greater stability during training without relying on the complex adversarial training process, and enable precise control over generation quality and diversity during the diffusion process.

The field of text-to-image synthesis has greatly benefited from recent advancements in diffusion models.

By leveraging the semantics of textual input, large-scale text-guided diffusion models such as DALLE-2 #OTHEREFR , Imagen #OTHEREFR , and Stable Diffusion #OTHEREFR have revolutionized the way we generate high-quality images.

These models are capable of following user instructions, generating customized high-fidelity images with specified content, style, and attributes.

Recently, several studies #OTHEREFR have focused on gaining more fine-grained control over the generation process by leveraging auxiliary information such as edge maps, segmentation maps, color maps, etc.

Users can specify desired attributes to guide and restrict the generation process by including such additional input, resulting in more precise control over the generated images.

Despite some limitations in image diversity due to auxiliary input dependency, these methods have produced impressive results in generating high-quality images that meet specific requirements.

Glyphdraw: Learning To Draw Chinese Characters In Image Synthesis Models Coherently

Abstract: Recent breakthroughs in the field of language-guided image generation have yielded impressive achievements, enabling the creation of high-quality and diverse images based on user instructions. Although the synthesis performance is fascinating, one significant limitation of current image generation models is their insufficient ability to generate coherent text within images, particularly for complex glyph structures like Chinese characters. To address this problem, we introduce GlyphDraw, a general learning framework aiming at endowing image generation models with the capacity to generate images embedded with coherent text. To the best of our knowledge, this is the first work in the field of image synthesis to address the generation of Chinese characters. We first sophisticatedly design the image-text dataset's construction strategy, then build our model specifically on a diffusionbased image generator and carefully modify the network structure to allow the model to learn drawing Chinese characters with the help of glyph and position information. Furthermore, we maintain the model's open-domain image synthesis capability by preventing catastrophic forgetting by using a variety of training techniques. Extensive qualitative and quantitative experiments demonstrate that our method not only produces accurate Chinese characters as in prompts, but also naturally blends the generated text into the background. Please refer to our project page.

Generative Adversarial Networks

Abstract: We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.


NPCR #REFR represents point features by 3D Convolution Neural Network (CNN), and converts the 3D representation to multi-plane images.

Traditional Point Rendering [38, #OTHEREFR is based on computer graphics, which generates images from point clouds by simulating the physical process of imaging, considering geometry #OTHEREFR , material #OTHEREFR , BRDF #OTHEREFR , and lighting #OTHEREFR . The rendering pipeline is general for arbitrary scenes.

But it cannot fill missing points and thus generate vacant pixels.

In the deep learning era, neural-based point renderers #OTHEREFR made great progress in generating images from point clouds.

They first extract multi-scale projected features from point clouds and then use a decoder to generate images.

NPBG #OTHEREFR augments each point by a neural descriptor to encode local geometry and appearance.

Different from these methods, our Point2Pix combines a more discriminative point encoder with the renderer pipeline of NeRF, thus achieving better performance.

Point2Pix: Photo-Realistic Point Cloud Rendering Via Neural Radiance Fields

Abstract: Synthesizing photo-realistic images from a point cloud is challenging because of the sparsity of point cloud representation. Recent Neural Radiance Fields and extensions are proposed to synthesize realistic images from 2D input. In this paper, we present Point2Pix as a novel point renderer to link the 3D sparse point clouds with 2D dense image pixels. Taking advantage of the point cloud 3D prior and NeRF rendering pipeline, our method can synthesize high-quality images from colored point clouds, generally for novel indoor scenes. To improve the efficiency of ray sampling, we propose point-guided sampling, which focuses on valid samples. Also, we present Point Encoding to build Multiscale Radiance Fields that provide discriminative 3D point features. Finally, we propose Fusion Encoding to efficiently synthesize high-quality images. Extensive experiments on the ScanNet and ArkitScenes datasets demonstrate the effectiveness and generalization.

Neural Point Cloud Rendering Via Multi-Plane Projection

Abstract: We present a new deep point cloud rendering pipeline through multi-plane projections. The input to the network is the raw point cloud of a scene and the output are image or image sequences from a novel view or along a novel camera trajectory. Unlike previous approaches that directly project features from 3D points onto 2D image domain, we propose to project these features into a layered volume of camera frustum. In this way, the visibility of 3D points can be automatically learnt by the network, such that ghosting effects due to false visibility check as well as occlusions caused by noise interferences are both avoided successfully. Next, the 3D feature volume is fed into a 3D CNN to produce multiple layers of images w.r.t. the space division in the depth directions. The layered images are then blended based on learned weights to produce the final rendering results. Experiments show that our network produces more stable renderings compared to previous methods, especially near the object boundaries. Moreover, our pipeline is robust to noisy and relatively sparse point cloud for a variety of challenging scenes.


A thorough description of DNNs and their training can be found in #REFR , in the work herein the DNNs utilised only made use of fully connected layers.

where J denotes the dimension of the latent space and

The number of samples to take from the latent space to evaluate the expectation can even be 1 as in the original formulation of Kingma et al. #OTHEREFR .

As mentioned above, in the VAE model the encoder and decoder functions are approximated using deep neural networks (DNNs).

DNNs are a very widely used class of models that exploit multiple neural network layers applied one after another in order to approximate very complex functions more efficiently than shallow networks #OTHEREFR .

In fully connected layers, the transform performed by each layer consists of the matrix multiplication of the input vector with a trainable weights matrix and the addition of a trainable bias vector.

A nonlinear activation function, often a tanh or sigmoid is then applied element-wise to the output of this operation.

This results in a very flexible and powerful model for learning general nonlinear relations (32).

Vprom: A Novel Variational Autoencoder-Boosted Reduced Order Model For The Treatment Of Parametric Dependencies In Nonlinear Systems A Preprint

Abstract: Reduced Order Models (ROMs) are of considerable importance in many areas of engineering in which computational time presents difficulties. Established approaches employ projection-based reduction such as Proper Orthogonal Decomposition, however, such methods can become inefficient or fail in the case of parameteric or strongly nonlinear models. Such limitations are usually tackled via a library of local reduction bases each of which being valid for a given parameter vector. The success of such methods, however, is strongly reliant upon the method used to relate the parameter vectors to the local bases, this is typically achieved using clustering or interpolation methods. We propose the replacement of these methods with a Variational Autoencoder (VAE) to be used as a generative model which can infer the local basis corresponding to a given parameter vector in a probabilistic manner. The resulting VAE-boosted parametric ROM VpROM still retains the physical insights of a projection-based method but also allows for better treatment of problems where model dependencies or excitation traits cause the dynamic behavior to span multiple response regimes. Moreover, the probabilistic treatment of the VAE representation allows for uncertainty quantification on the reduction bases which may then be propagated to the ROM response. The performance of the proposed approach is validated on an open-source simulation benchmark featuring hysteresis and multi-parametric dependencies, and on a large-scale wind turbine tower characterised by nonlinear material behavior and model uncertainty.

Generative Adversarial Nets

Abstract: We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to ½ everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.


Our model is implemented based on Pytorch #REFR and trained on a PC with an RTX 3090 GPU.

The input sources are all resized to 384 × 384 and all the three stages use the mini-batch of size 4 for training.

First, we use some RGB-D saliency datasets #OTHEREFR to train the static predictor network.

Once the first stage of training is finished, we adopt the DAVIS 16 training set to train the multi-source fusion network.

In the process, the parameters of the static predictor are frozen, and we just train the moving object predictor.

We use a total of 8, 398 labeled samples including video data (2, 000 + frames) and RGB-D SOD data (6, 300 + images) in the first and second phases. The depth map is obtained by the depth camera.

Adaptive Multi-Source Predictor For Zero-Shot Video Object Segmentation

Abstract: Both static and moving objects usually exist in real-life videos. Most video object segmentation methods only focus on exacting and exploiting motion cues to perceive moving objects. Once faced with static objects frames, moving object predictors may predict failed results caused by uncertain motion information, such as low-quality optical flow maps. Besides, many sources such as RGB, depth, optical flow and static saliency can provide useful information about the objects. However, existing approaches only utilize the RGB or RGB and optical flow. In this paper, we propose a novel adaptive multi-source predictor for zero-shot video object segmentation. In the static object predictor, the RGB source is converted to depth and static saliency sources, simultaneously. In the moving object predictor, we propose the multi-source fusion structure. First, the spatial importance of each source is highlighted with the help of the interoceptive spatial attention module (ISAM). Second, the motion-enhanced module (MEM) is designed to generate pure foreground motion attention for improving both static and moving features used in the decoder. Furthermore, we design a feature purification module (FPM) to filter the inter-source incompatible features. By the ISAM, MEM and FPM, the multi-source features are effectively fused. In addition, we put forward an adaptive predictor fusion network (APF) to evaluate the quality of optical flow and fuse the predictions from the static object predictor and the moving object predictor in order to prevent over-reliance on the failed results caused by low-quality optical flow maps. Experiments show that the proposed model outperforms the state-of-the-art methods on three challenging ZVOS benchmarks. And, the static object predictor can precisely predicts a high-quality depth map and static saliency map at the same time.

Pytorch: An Imperative Style, High-Performance Deep Learning Library

Abstract: Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. ::: In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. ::: We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.


Extending the results of #REFR to the continuous setting, or proving the uniform convergence in partition size of the infinite-width limit are both promising directions.

We showed that in the infinite-width-then-depth limit, these converge weakly to GPs indexed on path space endowed with neural signature kernels satisfying certain (possibly non-linear) PDEs varying according to the choice of activation function ϕ.

In the special case where ϕ is the identity, we showed that the equation reduces to a linear PDE and the limiting kernel agrees with the signature kernel.

In this setting, we also showed that the width-depth limits commute.

Finally, we showed that in the infinite-depth regime, finite-width controlled ResNets converge in distribution to Neural CDEs with random vector fields which are either time-independent and Gaussian, if the system is homogeneous, or behave like a matrix-valued Brownian motion, if the system is inhomogeneous.

An interesting question that remains to be answered is that of commutativity of the limits in the general setting.

We believe that a rigorous investigation of the functional analytic properties of the reproducing kernel Hilbert spaces (RKHSs) associated to the new family of neural signature kernels is also a compelling future research direction.

In particular, it would allow to build an understanding of the expressivity and generalization properties of these kernels.

Last but not least, establishing expressions and analyzing the associated Neural Tangent Kernels (NTK) #OTHEREFR would provide quantitative insights on the training mechanism of Neural CDEs by gradient descent.

Neural Signature Kernels As Infinite-Width-Depth-Limits Of Controlled Resnets

Abstract: Motivated by the paradigm of reservoir computing, we consider randomly initialized controlled ResNets defined as Euler-discretizations of neural controlled differential equations (Neural CDEs). We show that in the infinite-width-then-depth limit and under proper scaling, these architectures converge weakly to Gaussian processes indexed on some spaces of continuous paths and with kernels satisfying certain partial differential equations (PDEs) varying according to the choice of activation function ϕ. In the special case where ϕ is the identity, we show that the equation reduces to a linear PDE and the limiting kernel agrees with the signature kernel of Salvi et al. (2021a). In this setting, we also show that the width-depth limits commute. We name this new family of limiting kernels neural signature kernels. Finally, we show that in the infinite-depth regime, finite-width controlled ResNets converge in distribution to Neural CDEs with random vector fields which, depending on whether the weights are shared across layers, are either time-independent and Gaussian or behave like a matrix-valued Brownian motion.

Tensor Programs I: Wide Feedforward Or Recurrent Neural Networks Of Any Architecture Are Gaussian Processes

Abstract: Wide neural networks with random weights and biases are Gaussian processes, as observed by Neal (1995) for shallow networks, and more recently by Lee et al.~(2018) and Matthews et al.~(2018) for deep fully-connected networks, as well as by Novak et al.~(2019) and Garriga-Alonso et al.~(2019) for deep convolutional networks. We show that this Neural Network-Gaussian Process correspondence surprisingly extends to all modern feedforward or recurrent neural networks composed of multilayer perceptron, RNNs (e.g. LSTMs, GRUs), (nD or graph) convolution, pooling, skip connection, attention, batch normalization, and/or layer normalization. More generally, we introduce a language for expressing neural network computations, and our result encompasses all such expressible neural networks. This work serves as a tutorial on the \emph{tensor programs} technique formulated in Yang (2019) and elucidates the Gaussian Process results obtained there. We provide open-source implementations of the Gaussian Process kernels of simple RNN, GRU, transformer, and batchnorm+ReLU network at github.com/thegregyang/GP4A. Please see our arxiv version for the complete and up-to-date version of this paper.


We set a weight decay of 10 −4 , and use the cosine learning rate scheduling #REFR from the initial learning rate of 0.1.

Unless otherwise noted, we train each model for 200K updates for CIFAR-10 models, and 1M updates for ImageNet models.

For training AENIB models, we use α = 10.0, β = 0.0001 unless otherwise noted.

We use different training configurations depending on the encoder architecture, i.e., whether is it ResNet or ViT: (a) For ResNet-based models, we train the encoder part (f ) via stochastic gradient descent (SGD) with batch size of 64 using Nesterov momentum of weight 0.9 without dampening.

For the remainder parts of our AENIB architecture, e.g., the decoder g and discriminator MLPs, on the other hand, we follow the training practices of GAN instead: specifically, we use Adam #OTHEREFR with (α, β 1 , β 2 ) = (0.0002, 0.5, 0.999), following the hyperparameter practices explored by #OTHEREFR .

(b) For ViT-based models, on the other hand, we train both (transformer-based) encoder and decoder models via AdamW #OTHEREFR with a weight decay of 10 −4 , using batch size 128 and (α, β 1 , β 2 ) = (0.0002, 0.9, 0.999) with the cosine learning rate scheduling #OTHEREFR .

We use 2K and 100K steps of a linear warm-up phase in learning rate for CIFAR and ImageNet models, respectively.

Overall, we observe that a stable training of ViT (even for CIFAR-10) requires much stronger regularization compared to ResNets, otherwise they often significantly suffer from overfitting.

In this respect, we apply the regularization practices those are now widely used for ViTs on ImageNet, namely mixup #OTHEREFR , CutMix #OTHEREFR , and RandAugment #OTHEREFR , following those established in #OTHEREFR .

Enhancing Multiple Reliability Measures Via Nuisance-Extended Information Bottleneck

Abstract: In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition (i.e., less generalizable), so that one cannot prevent a model from co-adapting on such (socalled) "shortcut" signals: this makes the model fragile in various distribution shifts. To bypass such failure modes, we consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training. This motivates us to extend the standard information bottleneck to additionally model the nuisance information. We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training concerning both convolutional-and Transformerbased architectures. Our experimental results show that the proposed scheme improves robustness of learned representations (remarkably without using any domain-specific knowledge), with respect to multiple challenging reliability measures. For example, our model could advance the stateof-the-art on a recent challenging OBJECTS benchmark in novelty detection by 78.4% → 87.2% in AUROC, while simultaneously enjoying improved corruption, background and (certified) adversarial robustness. Code is available at https://github.com/jh-jeong/nuisance_ib.

Sgdr: Stochastic Gradient Descent With Warm Restarts

Abstract: Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR

cosine learning rate
initial learning rate


Traditional data poisoning attacks corrupt training samples to downgrade the overall performance of ML models #REFR .

Backdoor attacks are special types of data poisoning attacks on machine learning (ML) systems.

However, backdoor attacks maintain original performance until the model is provided with an input sample containing a "backdoor trigger." Such a backdoor trigger causes a pre-determined output, y t , beneficial to an adversary #OTHEREFR .

An adversary can deploy backdoor attacks by manipulating the training data and the corresponding labels.

For the case of GNNs, where the input samples are graphs, existing backdoor attacks inject triggers in the form of subgraphs g t . Fig.

3 illustrates the flow of a subgraphbased backdoor attack against GNNs #OTHEREFR .

First, a backdoor trigger and a target label y t (e.g., class 1, i.e., y t = 1) are determined ( Fig. 3 1 ) .

Poisonedgnn: Backdoor Attack On Graph Neural Networks-Based Hardware Security Systems

Abstract: Graph neural networks (GNNs) have shown great success in detecting intellectual property (IP) piracy and hardware Trojans (HTs). However, the machine learning community has demonstrated that GNNs are susceptible to data poisoning attacks, which result in GNNs performing abnormally on graphs with pre-defined backdoor triggers (realized using crafted subgraphs). Thus, it is imperative to ensure that the adoption of GNNs should not introduce security vulnerabilities in critical security frameworks. Existing backdoor attacks on GNNs generate random subgraphs with specific sizes/densities to act as backdoor triggers. However, for Boolean circuits, backdoor triggers cannot be randomized since the added structures should not affect the functionality of a design. We explore this threat and develop PoisonedGNN as the first backdoor attack on GNNs in the context of hardware design. We design and inject backdoor triggers into the register-transfer-or the gate-level representation of a given design without affecting the functionality to evade some GNN-based detection procedures. To demonstrate the effectiveness of PoisonedGNN, we consider two case studies: (i) Hiding HTs and (ii) IP piracy. Our experiments on TrustHub datasets demonstrate that PoisonedGNN can hide HTs and IP piracy from advanced GNN-based detection platforms with an attack success rate of up to 100%.

Poisoning Attacks Against Support Vector Machines

Abstract: We investigate a family of poisoning attacks against Support Vector Machines (SVM). Such attacks inject specially crafted training data that increases the SVM's test error. Central to the motivation for these attacks is the fact that most learning algorithms assume that their training data comes from a natural or well-behaved distribution. However, this assumption does not generally hold in security-sensitive settings. As we demonstrate, an intelligent adversary can, to some extent, predict the change of the SVM's decision function due to malicious input and use this ability to construct malicious data. ::: ::: The proposed attack uses a gradient ascent strategy in which the gradient is computed based on properties of the SVM's optimal solution. This method can be kernelized and enables the attack to be constructed in the input space even for non-linear kernels. We experimentally demonstrate that our gradient ascent procedure reliably identifies good local maxima of the non-convex validation error surface, which significantly increases the classifier's test error.


We evaluate the 3D detection performance on the widely used benchmark, nuScenes #REFR , a public dataset for autonomous driving.

It is collected from challenging urban scenes, consisting of 1000 annotated sequences.

Among them, 700 sequences are used as the training phase, 150 as validation, and the remaining 150 as the test phase for online evaluation.

The bounding boxes are labeled with not only categories but also some attributes like the velocity, scale, orientation and translation.

Besides the point cloud data from LiDAR, it also provides the 360 • image modality from cameras and signal from the Radar sensor.

In this paper, we only utilize the point cloud from LiDAR.

Link: Linear Kernel For Lidar-Based 3D Perception

Abstract: Extending the success of 2D Large Kernel to 3D perception is challenging due to: 1. the cubically-increasing overhead in processing 3D data; 2. the optimization difficulties from data scarcity and sparsity. Previous work has taken the first step to scale up the kernel size from 3 × 3 × 3 to 7 × 7 × 7 by introducing block-shared weights. However, to reduce the feature variations within a block, it only employs modest block size and fails to achieve larger kernels like the 21 × 21 × 21. To address this issue, we propose a new method, called LinK, to achieve a wider-range perception receptive field in a convolution-like manner with two core designs. The first is to replace the static kernel matrix with a linear kernel generator, which adaptively provides weights only for non-empty voxels. The second is to reuse the precomputed aggregation results in the overlapped blocks to reduce computation complexity. The proposed method successfully enables each voxel to perceive context within a range of 21 × 21 × 21. Extensive experiments on two basic perception tasks, 3D object detection and 3D semantic segmentation, demonstrate the effectiveness of our method. Notably, we rank 1st on the public leaderboard of the 3D detection benchmark of nuScenes (LiDAR track), by simply incorporating a LinK-based backbone into the basic detector, CenterPoint. We also boost the strong segmentation baseline's mIoU with 2.7% in the SemanticKITTI test set.

Nuscenes: A Multimodal Dataset For Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image-based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first published dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online at this http URL.

3D detection performance
autonomous driving


Our work was also compared with the results from µVulDeePecker #REFR , which evaluated 40 classes.

We prevented training a biased model by using the Focal Loss function during the model's training.

We tested VulF dataset with our proposed RoBERTa-PFGCN and two other publicly available models, Devign and VELVET.

Moreover, we also implemented µVulDeePecker #OTHEREFR ourselves based on their proposed architecture since their model is not publicly available.

Our experimental results show that our model achieves the highest accuracy compared to Devign, VELVET, and µVulDeePecker.

Table 4 shows that our model improved accuracy, precision, recall, and F1 score by 11.79%, 11.06%, 9.98%, and 11.39%, respectively, which is at least 11% higher than other models.

Table  4 shows that our model outperforms µVulDeePecker on their MVD dataset by almost 3.80% on F1 score.

We provide an F1 score comparison for all 40 classes by our model against µVulDeePecker with the MVD dataset in Figure 4 .

This bar chart shows a comparative analysis of the F1 score with our model vs.

Table 4 demonstrates our model's performance with two other datasets.

An Unbiased Transformer Source Code Learning With Semantic Vulnerability Graph

Abstract: Over the years, open-source software systems have become prey to threat actors. Even highly-adopted software has been crippled by unforeseeable attacks, leaving millions of devices exposed. Even as open-source communities act quickly to patch the breach, code vulnerability screening should be an integral part of agile software development from the beginning. Unfortunately, current vulnerability screening techniques are ineffective at identifying novel vulnerabilities or providing developers with code vulnerability and classification. Furthermore, the datasets used for vulnerability learning often exhibit distribution shifts from the real-world testing distribution due to novel attack strategies deployed by adversaries and as a result, the machine learning model's performance may be hindered or biased. To address these issues, we propose a joint interpolated multitasked unbiased vulnerability classifier comprising a transformer "RoBERTa" and graph convolution neural network (GCN). We present a training process utilizing a semantic vulnerability graph (SVG) representation from source code, created by integrating edges from a sequential flow, control flow, and data flow, as well as a novel flow dubbed Poacher Flow (PF). Poacher flow edges reduce the gap between dynamic and static program analysis and handle complex long-range dependencies. Moreover, our approach reduces biases of classifiers regarding unbalanced datasets by integrating Focal Loss objective function along with SVG. Remarkably, experimental results show that our classifier outperforms state-of-the-art results on vulnerability detection with fewer false negatives and false positives. After testing our model across multiple datasets, it shows an improvement of at least 2.41% and 18.75% in the bestcase scenario. Evaluations using N-day program samples demonstrate that our proposed approach achieves a 93% accuracy and was able to detect 4, zero-day vulnerabilities from popular GitHub repositories. Our code and data are

$\Mu$Vuldeepecker: A Deep Learning-Based System For Multiclass Vulnerability Detection

Abstract: Fine-grained software vulnerability detection is an important and challenging problem. Ideally, a detection system (or detector) not only should be able to detect whether or not a program contains vulnerabilities, but also should be able to pinpoint the type of a vulnerability in question. Existing vulnerability detection methods based on deep learning can detect the presence of vulnerabilities (i.e., addressing the binary classification or detection problem), but cannot pinpoint types of vulnerabilities (i.e., incapable of addressing multiclass classification). In this paper, we propose the first deep learning-based system for multiclass vulnerability detection, dubbed $\mu$VulDeePecker. The key insight underlying $\mu$VulDeePecker is the concept of code attention, which can capture information that can help pinpoint types of vulnerabilities, even when the samples are small. For this purpose, we create a dataset from scratch and use it to evaluate the effectiveness of $\mu$VulDeePecker. Experimental results show that $\mu$VulDeePecker is effective for multiclass vulnerability detection and that accommodating control-dependence (other than data-dependence) can lead to higher detection capabilities.


We use the CodeSearchNet #REFR dataset for our experiments: it is a carefully de-duplicated, multi-project dataset, which allows cross-project testing.

De-duplication is key: Code duplication in machine learning models can deceptively inflate performance metrics by as much as 100% when measured on duplicated code datasets, compared to de-duplicated datasets #OTHEREFR .

It is part of the CodeXGLUE #OTHEREFR benchmark, which comprises 14 datasets for 10 software engineering tasks. Many models have been evaluated on this dataset.

The CodeXGLUE dataset contains thousands of samples from six different programming languages (i.e., Java, Python, JavaScript, Ruby, Go, PHP).

However, we did not use the entire test dataset; instead, we selected 250 samples uniformly at random from each language.

Since the original dataset is cross-project and we sampled it uniformly, our subsample remains cross-project.

Improving Few-Shot Prompts With Relevant Static Analysis Products

Abstract: Large Language Models (LLM) are a new class of computation engines, "programmed" via prompt engineering. Researchers are still learning how to best "program" these LLMs to help developers. We start with the intuition that developers tend to consciously and unconsciously have a collection of semantics facts in mind when working on coding tasks. Mostly these are shallow, simple facts arising from a quick read. For a function, examples of facts might include parameter and local variable names, return expressions, simple pre-and post-conditions, and basic control and data flow, etc. One might assume that the powerful multi-layer architecture of transformer-style LLMs makes them inherently capable of doing this simple level of "code analysis" and extracting such information, implicitly, while processing code: but are they, really? If they aren't, could explicitly adding this information help? Our goal here is to investigate this question, using the code summarization task and evaluate whether automatically augmenting an LLM's prompt with semantic facts explicitly, actually helps. Prior work shows that LLM performance on code summarization benefits from few-shot samples drawn either from the same-project or from examples found via information retrieval methods (such as BM25). While summarization performance has steadily increased since the early days, there is still room for improvement: LLM performance on code summarization still lags its performance on natural-language tasks like translation and text summarization. We find that adding semantic facts actually does help! This approach improves performance in several different settings suggested by prior work, including for two different Large Language Models.

Codesearchnet Challenge: Evaluating The State Of Semantic Code Search

Abstract: Semantic code search is the task of retrieving relevant code given a natural language query. While related to other information retrieval tasks, it requires bridging the gap between the language used in code (often abbreviated and highly technical) and natural language more suitable to describe vague concepts and ideas. ::: To enable evaluation of progress on code search, we are releasing the CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which consists of 99 natural language queries with about 4k expert relevance annotations of likely results from CodeSearchNet Corpus. The corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation. In this article, we describe the methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task. ::: We hope that CodeSearchNet Challenge encourages researchers and practitioners to study this interesting task further and will host a competition and leaderboard to track the progress on the challenge. We are also keen on extending CodeSearchNet Challenge to more queries and programming languages in the future.


BERT #REFR can be seen as a development of DAE, which masks input tokens and trains the model to predict the missing content.

mmTransformer #OTHEREFR is designed to hierarchically aggregate the past trajectories, the road information, and the social interaction.

For predicting multiagents future trajectories, AgentFormer #OTHEREFR and AutoBots #OTHEREFR have given solutions to model the time dimension and social dimension simultaneously.

The enhancement of the encoder's ability to model information in both dimensions is an interesting and central focus of this work.

Self-supervised Learning has shown significant success in natural language processing and computer vision fields recently, especially the autoencoding method.

Denoising autoencoders (DAE) #OTHEREFR is a learning representation method that reconstructs original signals from corrupted inputs.

With the Masked Language Modeling (MLM) task proposed in BERT, many MLM variants #OTHEREFR are proposed to improve the performance of transformer pretraining.

Similarly, in computer vision, the autoencoding method often focuses on different pretext tasks for pre- training #OTHEREFR .

One of the most popular methods is MAE #OTHEREFR , which randomly masks input patches and trains the model to recover masked patches in pixel space.

Continuous progress #OTHEREFR based on MAE has verified its effectiveness.

Following the concept of MAE, our approach concentrate on utilizing MAE as a tool to pre-train model encoders with powerful feature extraction capability.

Traj-Mae: Masked Autoencoders For Trajectory Prediction

Abstract: Trajectory prediction has been a crucial task in building a reliable autonomous driving system by anticipating possible dangers. One key issue is to generate consistent trajectory predictions without colliding. To overcome the challenge, we propose an efficient masked autoencoder for trajectory prediction (Traj-MAE) that better represents the complicated behaviors of agents in the driving environment. Specifically, our Traj-MAE employs diverse masking strategies to pre-train the trajectory encoder and map encoder, allowing for the capture of social and temporal information among agents while leveraging the effect of environment from multiple granularities. To address the catastrophic forgetting problem that arises when pre-training the network with multiple masking strategies, we introduce a continual pre-training framework, which can help Traj-MAE learn valuable and diverse information from various strategies efficiently. Our experimental results in both multi-agent and single-agent settings demonstrate that Traj-MAE achieves competitive results with state-of-the-art methods and significantly outperforms our baseline model. The code will be made publicly available upon publication.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


To verify the robustness of our model to occlusion, we evaluated the models on 132 images with occlusion from the test split of the Matteport3D dataset #REFR .

In this experiment, occlusion was defined as a situation where one or more corners of the ground truth are not visible, as illustrated in Fig. 6 .

The Pano S2D3D dataset #OTHEREFR was not included in this experiment because of its cuboid nature, which does not cause occlusion.

Quantitative results on the occlusion dataset are presented in Tab. 3.

We reported the best performance of our model, a 2.37% improvement in both 2D and 3D IoU compared with LGT-Net #OTHEREFR . In comparison to Tab.

2, the difference in 2D and 3D IoU between our model and other models was greater on the occlusion dataset.

Shape-Net: Room Layout Estimation From Panoramic Images Robust To Occlusion Using Knowledge Distillation With 3D Shapes As Additional Inputs

Abstract: Estimating the layout of a room from a single-shot panoramic image is important in virtual/augmented reality and furniture layout simulation. This involves identifying three-dimensional (3D) geometry, such as the location of corners and boundaries, and performing 3D reconstruction. However, occlusion is a common issue that can negatively impact room layout estimation, and this has not been thoroughly studied to date. It is possible to obtain 3D shape information of rooms as drawings of buildings and coordinates of corners from image datasets, thus we propose providing both 2D panoramic and 3D information to a model to effectively deal with occlusion. However, simply feeding 3D information to a model is not sufficient to utilize the shape information for an occluded area. Therefore, we improve the model by introducing 3D Intersection over Union (IoU) loss to effectively use 3D information. In some cases, drawings are not available or the construction deviates from a drawing. Considering such practical cases, we propose a method for distilling knowledge from a model trained with both images and 3D information to a model that takes only images as input. The proposed model, which is called Shape-Net, achieves state-of-the-art (SOTA) performance on benchmark datasets. We also confirmed its effectiveness in dealing with occlusion through significantly improved accuracy on images with occlusion compared with existing models.

Matterport3D: Learning From Rgb-D Data In Indoor Environments

Abstract: Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms. However, existing datasets still cover only a limited number of views or a restricted scale of spaces. In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. Annotations are provided with surface reconstructions, camera poses, and 2D and 3D semantic segmentations. The precise global alignment and comprehensive, diverse panoramic set of views over entire buildings enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification.


We compare our results to that of a Dense, Spatial AE and Dense VAE model (Baur et al., 2020) and an SSIM AE #REFR .

After obtaining the optimal models from each hyperparameter study, we test the proposed models on the hold-out testing data as a final ablation study to showcase the added contribution of each proposed method.

These models are trained on PET only, however we also implement them with 2 channel inputs with PET and CT to showcase the effect of CT conditioning on the baseline models in comparison to the effects of conditioning for our proposed method.

We additionally implement F-AnoGan for a further baseline comparison #OTHEREFR .

Testing is carried out on 36 samples of which the best achievable DICE score is calculated in addition to the area under the precision-recall curve (AUPRC) as a suitable measure for segmentation performance under class imbalance.

These results can be seen tabulated in Table  2 and visualised in Fig.

7 qualitatively showcasing the best two state-of-the-art models against our added contributions.

Cross Attention Transformers For Multi-Modal Unsupervised Whole-Body Pet Anomaly Detection

Abstract: Cancer is a highly heterogeneous condition that can occur almost anywhere in the human body. 18 F-fluorodeoxyglucose (18 F-FDG PET) is an imaging modality commonly used to detect cancer due to its high sensitivity and clear visualisation of the pattern of metabolic activity. Nonetheless, as cancer is highly heterogeneous, it is challenging to train general-purpose discriminative cancer detection models, with data availability and disease complexity often cited as a limiting factor. Unsupervised learning methods, more specifically anomaly detection models, have been suggested as a putative solution. These models learn a healthy representation of tissue and detect cancer by predicting deviations from the healthy norm, which requires models capable of accurately learning long-range interactions between organs, their imaging patterns, and other abstract features with high levels of expressivity. Such characteristics are suitably satisfied by transformers, which have been shown to generate state-of-the-art results in unsupervised anomaly detection by training on normal data. This work expands upon such approaches by introducing multi-modal conditioning of the transformer via cross-attention i.e. supplying anatomical reference information from paired CT images to aid the PET anomaly detection task. Furthermore, we show the importance and impact of codebook sizing within a Vector Quantized Variational Autoencoder, on the ability of the transformer network to fulfill the task of anomaly detection. Using 294 whole-body PET/CT samples containing various cancer types, we show that our anomaly detection method is robust and capable of achieving accurate cancer localization results even in cases where normal training data is unavailable. In addition, we show the efficacy of this approach on out-of-sample data showcasing the generalizability of this approach even with limited training data. Lastly, we propose to combine model uncertainty with a new kernel density estimation approach, and show that it provides clinically and statistically significant improvements in accuracy and robustness, when compared to the classic residual-based anomaly maps. Overall, a superior performance is demonstrated against leading state-of-the-art alternatives, drawing attention to the potential of these approaches.

Improving Unsupervised Defect Segmentation By Applying Structural Similarity To Autoencoders

Abstract: Convolutional autoencoders have emerged as popular methods for unsupervised defect segmentation on image data. Most commonly, this task is performed by thresholding a per-pixel reconstruction error based on an p-distance. This procedure, however, leads to large residuals whenever the reconstruction includes slight localization inaccuracies around edges. It also fails to reveal defective regions that have been visually altered when intensity values stay roughly consistent. We show that these problems prevent these approaches from being applied to complex real-world scenarios and that they cannot be easily avoided by employing more elaborate architectures such as variational or feature matching autoencoders. We propose to use a perceptual loss function based on structural similarity that examines inter-dependencies between local image regions, taking into account luminance, contrast, and structural information, instead of simply comparing single pixel values. It achieves significant performance gains on a challenging real-world dataset of nanofibrous materials and a novel dataset of two woven fabrics over state-of-the-art approaches for unsupervised defect segmentation that use per-pixel reconstruction error metrics.


B. Meta Learning-Based Few-Shot Kg Reasoning

FSRL #REFR proposed a novel method that captures both the different relation types and the different impacts of local neighbors.

Meta-learning is an effective learning paradigm across curating meta-tasks in order to transfer them to a new task #OTHEREFR .

A large portion of FS-KGR methods employ the meta-learning paradigm to effectively tackle the issue of data scarcity in the target few-shot task #OTHEREFR [16] #OTHEREFR .

These models can be broadly categorized into two types, i.e., metric-based models, and optimization-based models.

The former attempts to acquire a universal measure and the corresponding matching functions from a group of training examples and GMatching #OTHEREFR focus on learning a matching metric that relies on entity embedding and subgraph structure.

However, such approaches typically rely on many manual-created meta-training tasks towards specific relations during the training process, leading to poor robustness if the meta-training set is not welldesigned.

Moreover, creating an excessive number of metatraining sets from the background knowledge graph would eliminate a large proportion of edges, result in sparsity in the graph that hard to learn since the number of relations is limited in real KG, which limits the expressive ability of the model.

Sarf: Aliasing Relation Assisted Self-Supervised Learning For Few-Shot Relation Reasoning

Abstract: Few-shot relation reasoning on knowledge graphs (FS-KGR) aims to infer long-tail data-poor relations, which has drawn increasing attention these years due to its practicalities. The pre-training of previous methods needs to manually construct the meta-relation set, leading to numerous labor costs. Self-supervised learning (SSL) is treated as a solution to tackle the issue, but still at an early stage for FS-KGR task. Moreover, most of the existing methods ignore leveraging the beneficial information from aliasing relations (AR), i.e., data-rich relations with similar contextual semantics to the target data-poor relation. Therefore, we proposed a novel Self-Supervised Learning model by leveraging Aliasing Relations to assist FS-KGR, termed SARF. Concretely, four main components are designed in our model, i.e., SSL reasoning module, AR-assisted mechanism, fusion module, and scoring function. We first generate the representation of the co-occurrence patterns in a generative manner. Meanwhile, the representations of aliasing relations are learned to enhance reasoning in the AR-assist mechanism. Besides, multiple strategies i.e., simple summation and learnable fusion, are offered for representation fusion. Finally, the generated representation is used for scoring. Extensive experiments on three few-shot benchmarks demonstrate that SARF achieves state-of-the-art performance compared with other methods in most cases.

Few-Shot Knowledge Graph Completion

Abstract: Knowledge graphs (KGs) serve as useful resources for various natural language processing applications. Previous KG completion approaches require a large number of training instances (i.e., head-tail entity pairs) for every relation. The real case is that for most of the relations, very few entity pairs are available. Existing work of one-shot learning limits method generalizability for few-shot scenarios and does not fully use the supervisory information; however, few-shot KG completion has not been well studied yet. In this work, we propose a novel few-shot relation learning model (FSRL) that aims at discovering facts of new relations with few-shot references. FSRL can effectively capture knowledge from heterogeneous graph structure, aggregate representations of few-shot references, and match similar entity pairs of reference set for every relation. Extensive experiments on two public datasets demonstrate that FSRL outperforms the state-of-the-art.


A deep learning-based COVID-Net model designed by Wang and Wong #REFR acquired an accuracy of 92.4% with 53 COVID-19 samples out of the 16,756 total images. Ozturk et al.

As no benchmark dataset is yet available to analyze COVID-19 cases in chest x-ray images, authors collected images from different sources to build the datasets and conduct experiments. Table 5 . presents comparison of the proposed framework with state-of-the-arts.

#OTHEREFR tested the proposed DarkCovidNet by using 125 COVID-19, 500 pneumonia, and 500 normal images and achieved 98.08% and 87.02% for binary and ternary class scenario. Ioannis et al. #OTHEREFR detection of COVID-19 in two datasets.

The first dataset comprises 224 COVID-19, 504 normal, and 700 bacterial pneumonia images and achieved an overall accuracy of 98.75% and 93.48% for 2-class and 3-class respectively with the VGG-19 model.

The second dataset contains 224 COVID-19, 504 normal and, 714 pneumonia includes both viral and bacterial images.

And for 2-class and 3-class scenarios achieved an accuracy of 96.78% and 94.72% using pretrained MobileNet-V2.

In the proposed work, however, we used different datasets for 2-class and 3-class classification problem and achieved an accuracy of 94.8% and 92.8% by fine-tuning the pre-trained VGG-16 model.

Detection Of Covid-19 In Chest X-Ray Images Using Transfer Learning

Abstract: COVID-19a highly contagious disease infected millions of people worldwide. With limited testing components, screening tools such as chest radiography can assist the clinicians in the diagnosis and assessing the progress of disease. The performance of deep learning-based systems for diagnosis of COVID-19 disease in radiograph images has been encouraging. This paper investigates the concept of transfer learning using two of the most well-known VGGNet architectures, namely VGG-16 and VGG-19. The classifier block and hyperparameters are fine-tuned to adopt the models for automatic detection of Covid-19 in chest x-ray images. We generated two different datasets to evaluate the performance of the proposed system for the identification of positive Covid-19 instances in a multiclass and binary classification problems. The experimental outcome demonstrates the usefulness of transfer learning for small-sized datasets particularly in the field of medical imaging, not only to prevent over-fitting and convergence problems but also to attain optimal classification performance as well.

Covid-Net: A Tailored Deep Convolutional Neural Network Design For Detection Of Covid-19 Cases From Chest Radiography Images

Abstract: The COVID-19 pandemic continues to have a devastating effect on the health and well-being of the global population. A critical step in the fight against COVID-19 is effective screening of infected patients, with one of the key screening approaches being radiological imaging using chest radiography. Motivated by this, a number of artificial intelligence (AI) systems based on deep learning have been proposed and results have been shown to be quite promising in terms of accuracy in detecting patients infected with COVID-19 using chest radiography images. However, to the best of the authors' knowledge, these developed AI systems have been closed source and unavailable to the research community for deeper understanding and extension, and unavailable for public access and use. Therefore, in this study we introduce COVID-Net, a deep convolutional neural network design tailored for the detection of COVID-19 cases from chest radiography images that is open source and available to the general public. We also describe the chest radiography dataset leveraged to train COVID-Net, which we will refer to as COVIDx and is comprised of 16,756 chest radiography images across 13,645 patient cases from two open access data repositories. Furthermore, we investigate how COVID-Net makes predictions using an explainability method in an attempt to gain deeper insights into critical factors associated with COVID cases, which can aid clinicians in improved screening. By no means a production-ready solution, the hope is that the open access COVID-Net, along with the description on constructing the open source COVIDx dataset, will be leveraged and build upon by both researchers and citizen data scientists alike to accelerate the development of highly accurate yet practical deep learning solutions for detecting COVID-19 cases and accelerate treatment of those who need it the most.


This differs from the objective studied in previous work, which is the "standard" Binary Search objective of minimizing the worst-case total number of queries, though #REFR also established hardness results for the case when nodes have known query costs.

Through the user's interactions with q, the system learns about the user's preferences, which can be modeled as the direction to t from q.

However, serving the user q may impose a cost which increases in the distance between q and t.

For example, repeatedly serving content that the user dislikes may cause them to stop using the service altogether with some probability.

Hence, such a scenario fits well into the generalization of binary search that we consider.

With these potential motivations in mind, in combination with the applications given in #OTHEREFR , it is natural that the cost of a proposal should increase with the distance in G from the proposal to the correct node.

This naturally motivates extending a study of our non-uniform cost model to the case of arbitrary undirected graphs.

We ask: can any non-trivial approximation guarantees (or hardness) be achieved for interesting classes of cost functions? Even for the cost function that is identically 1 (i.e., the goal is to minimize the number of queries), minimizing the number of queries is not possible in polynomial time under the ETH.

However, no approximation hardness or nontrivial positive approximation results (beyond the trivial O(log n)) are known; obtaining such results for more general cost functions would be of great interest.

Another generalization of standard binary search that has been extensively studied is the case when the answers received by the algorithm could be erroneous.

Indeed, optimal algorithms for this problem have been studied under different error models, on the line, on trees, and on general graphs. #OTHEREFR .

Binary Search With Distance-Dependent Costs

Abstract: We introduce a search problem generalizing the typical setting of Binary Search on the line. Similar to the setting for Binary Search, a target is chosen adversarially on the line, and in response to a query, the algorithm learns whether the query was correct, too high, or too low. Different from the Binary Search setting (which is the special case of a constant function), the cost of a query is a monotone non-decreasing function of the distance between the query and the correct answer; different functions can be used for queries that are too high vs. those that are too low. The algorithm's goal is to identify an adversarially chosen target with minimum total cost. Note that the algorithm does not even know the cost it incurred until the end, when the target is revealed. This abstraction captures many natural settings in which a principal experiments by setting a quantity (such as an item price, bandwidth, tax rate, medicine dosage, etc.) where the cost or regret increases the further the chosen setting is from the optimal one. First, we show that for arbitrary symmetric cost functions (i.e., the case when overshooting vs. undershooting by the same amount leads to the same cost), the standard Binary Search algorithm is a 4-approximation. We then show that when the cost functions are bounded-degree polynomials of the distance (including the asymmetric case), the problem can be solved optimally using Dynamic Programming; this relies on a careful encoding of the combined cost of past queries (which, recall, will only be revealed in the future). We then generalize the setting to finding a node on a tree; here, the response to a query is the direction on the tree in which the target is located, and the cost is increasing in the distance on the tree from the query to the target. Using the k-cut search tree framework of Berendsohn and Kozma and the ideas we developed for the case of the line, we give a PTAS when the cost function is a bounded-degree polynomial.

Deterministic And Probabilistic Binary Search In Graphs

Abstract: We consider the following natural generalization of Binary Search: in a given undirected, positively weighted graph, one vertex is a target. The algorithm's task is to identify the target by adaptively querying vertices. In response to querying a node $q$, the algorithm learns either that $q$ is the target, or is given an edge out of $q$ that lies on a shortest path from $q$ to the target. We study this problem in a general noisy model in which each query independently receives a correct answer with probability $p>\frac{1}{2}$ (a known constant), and an (adversarial) incorrect one with probability $1-p$. Our main positive result is that when $p = 1$ (i.e., all answers are correct), $\log_2 n$ queries are always sufficient. For general $p$, we give an (almost information-theoretically optimal) algorithm that uses, in expectation, no more than $(1 - \delta) \frac{\log n}{1 - H(p)} + o(\log n) + O(\log^2 (1/\delta))$ queries, and identifies the target correctly with probability at leas $1-\delta$. Here, $H(p) = -(p \log p + (1-p) \log(1-p))$ denotes the entropy. The first bound is achieved by the algorithm that iteratively queries a 1-median of the nodes not ruled out yet; the second bound by careful repeated invocations of a multiplicative weights algorithm. Even for $p = 1$, we show several hardness results for the problem of determining whether a target can be found using $K$ queries. Our upper bound of $\log_2 n$ implies a quasipolynomial-time algorithm for undirected connected graphs; we show that this is best-possible under the Strong Exponential Time Hypothesis (SETH). Furthermore, for directed graphs, or for undirected graphs with non-uniform node querying costs, the problem is PSPACE-complete. For a semi-adaptive version, in which one may query $r$ nodes each in $k$ rounds, we show membership in $\Sigma_{2k-1}$ in the polynomial hierarchy, and hardness for $\Sigma_{2k-5}$.


This is important when the normalization layer is used in conjunction with Exponential Moving Average #REFR (commonly used in DDPMs).

The visual encoder maps the raw image sequence into a latent embedding O t and is trained end-to-end with the diffusion policy.

Different camera views use separate encoders, and images in each timestep are encoded independently and then concatenated to form O t .

We used a standard ResNet-18 (without pretraining) as the encoder with the following modifications: 1) Replace the global average pooling with a spatial softmax pooling to maintain spatial information #OTHEREFR . 2) Replace BatchNorm with GroupNorm #OTHEREFR for stable training.

Diffusion Policy: Visuomotor Policy Learning Via Action Diffusion

Abstract: Fig. 1: Policy Representations. a) Explicit policy with different types of action representations. b) Implicit policy learns an energy function conditioned on both action and observation and optimizes for actions that minimize the energy landscape c) Diffusion policy refines noise into actions via a learned gradient field. This formulation provides stable training, allows the learned policy to accurately model multimodal action distributions, and accommodates high-dimensional action sequences.

Momentum Contrast For Unsupervised Visual Representation Learning

Abstract: We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.


In particular, Aspect-MF #REFR analyses the dynamicity of temporal preference aspects using a component-based approach, and identifies the aspects that are easy to drift.

User preference may shift over time for many reasons, including the changes of user features (e.g., income and pregnancy) and environment factors (e.g., seasonal variation).

For example, in the scenario of food recommendation, a user might become liking expensive but healthy food if the user's income increases.

Since such preference shifts are frequent in the real-world scenarios, the recommender models should update the user representations adaptively over time.

Ignoring the shifting nature of preference to learn user representation will lead to inappropriate recommendations.

In addition to disentangled recommendation and sequential recommendation, there exists some work with the potential of addressing this issue.

ST-LDA #OTHEREFR learns region-dependent personal interests and crowd preference to adapt to preference shifts.

Lastly, MTUPD #OTHEREFR utilizes a forgetting curve function to calculate the correlations of user preference in different time periods.

• Long-and short-term interest modeling.

Another possibly related direction considers the modeling of both long-and short-term interests, where short-term interests are inferred from recent interactions to capture the preference shifts, and long-term interests represent the stable preference over time.

Many studies have shed light on the modeling of long-and short-term user interests, for instance, PLSPL #OTHEREFR utilizes attention mechanism to characterize the long-term preference while integrating the location and category information to capture the short-term preference.

Causal Disentangled Recommendation Against User Preference Shifts

Abstract: Recommender systems easily face the issue of user preference shifts. User representations will become outof-date and lead to inappropriate recommendations if user preference has shifted over time. To solve the issue, existing work focuses on learning robust representations or predicting the shifting pattern. There lacks a comprehensive view to discover the underlying reasons for user preference shifts. To understand the preference shift, we abstract a causal graph to describe the generation procedure of user interaction sequences. Assuming user preference is stable within a short period, we abstract the interaction sequence as a set of chronological environments. From the causal graph, we find that the changes of some unobserved factors (e.g., becoming pregnant) cause preference shifts between environments. Besides, the fine-grained user preference over categories sparsely affects the interactions with different items. Inspired by the causal graph, our key considerations to handle preference shifts lie in modeling the interaction generation procedure by: 1) capturing the preference shifts across environments for accurate preference prediction, and 2) disentangling the sparse influence from user preference to interactions for accurate effect estimation of preference. To this end, we propose a Causal Disentangled Recommendation (CDR) framework, which captures preference shifts via a temporal variational autoencoder and learns the sparse influence from multiple environments. Specifically, an encoder is adopted to infer the unobserved factors from user interactions while a decoder is to model the interaction generation process. Besides, we introduce two learnable matrices to disentangle the sparse influence from user preference to interactions. Lastly, we devise a multi-objective loss to optimize CDR. Extensive experiments on three datasets show the superiority of CDR in enhancing the generalization ability under user preference shifts.

Modelling And Analysis Of Temporal Preference Drifts Using A Component-Based Factorised Latent Approach

Abstract: textabstractIn recommender systems, human preferences are identified by a number of individual components with complicated interactions and properties. Recently, the dynamicity of preferences has been the focus of several studies. The changes in user preferences can originate from substantial reasons, like personality shift, or transient and circumstantial ones, like seasonal changes in item popularities. Disregarding these temporal drifts in modelling user preferences can result in unhelpful recommendations. Moreover, different temporal patterns can be associated with various preference domains, and preference components and their combinations. These components comprise preferences over features, preferences over feature values, conditional dependencies between features, socially-influenced preferences, and bias. For example, in the movies domain, the user can change his rating behaviour (bias shift), her preference for genre over language (feature preference shift), or start favouring drama over comedy (feature value preference shift). In this paper, we first propose a novel latent factor model to capture the domain-dependent component-specific temporal patterns in preferences. The component-based approach followed in modelling the aspects of preferences and their temporal effects enables us to arbitrarily switch components on and off. We evaluate the proposed method on three popular recommendation datasets and show that it significantly outperforms the most accurate state-of-the-art static models. The experiments also demonstrate the greater robustness and stability of the proposed dynamic model in comparison with the most successful models to date. We also analyse the temporal behaviour of different preference components and their combinations and show that the dynamic behaviour of preference components is highly dependent on the preference dataset and domain. Therefore, the results also highlight the importance of modelling temporal effects but also underline the advantages of a component-based architecture that is better suited to capture domain-specific balances in the contributions of the aspects.


In addition, we train and also report the results of CenterNet #REFR for comparing the detection performance and execution time, since the code is available.

We compare the results of EvCenterNet with four baseline methods.

We include two sampling-based baselines, which utilize MC dropout and an Ensemble of 5 networks for uncertainty estimation.

Note that we use #OTHEREFR for both the sampling-based baselines. For MC dropout, we perform five forward passes.

For a sampling-free baseline, we implemented the CertainNet * baseline ourselves as no code was available.

When available, we also report the results presented by the authors of CertainNet #OTHEREFR in their paper.

I compares the detection performance of the models on the validation set of the KITTI dataset.

EvCenterNet performs slightly better than the sampling-free #OTHEREFR baseline on almost all the categories.

The largest difference can be seen in the cyclist class, where EvCenterNet obtains 4.48pp (easy), 2.62pp (medium), and 2.96pp (hard) in comparison to CertainNet.

We attribute the improvement in the small class to the proposed uncertainty-aware loss with weighting approach.

Evcenternet: Uncertainty Estimation For Object Detection Using Evidential Learning

Abstract: Uncertainty estimation is crucial in safety-critical settings such as automated driving as it provides valuable information for several downstream tasks including high-level decision-making and path planning. In this work, we propose EvCenterNet, a novel uncertainty-aware 2D object detection framework utilizing evidential learning to directly estimate both classification and regression uncertainties. To employ evidential learning for object detection, we devise a combination of evidential and focal loss functions for the sparse heatmap inputs. We introduce class-balanced weighting for regression and heatmap prediction to tackle the class imbalance encountered by evidential learning. Moreover, we propose a learning scheme to actively utilize the predicted heatmap uncertainties to improve the detection performance by focusing on the most uncertain points. We train our model on the KITTI dataset and evaluate it on challenging out-of-distribution datasets including BDD100K and nuImages. Our experiments demonstrate that our approach improves the precision and minimizes the execution time loss in relation to the base model.

Objects As Points

Abstract: Detection identifies objects as axis-aligned boxes in an image. Most successful object detectors enumerate a nearly exhaustive list of potential object locations and classify each. This is wasteful, inefficient, and requires additional post-processing. In this paper, we take a different approach. We model an object as a single point --- the center point of its bounding box. Our detector uses keypoint estimation to find center points and regresses to all other object properties, such as size, 3D location, orientation, and even pose. Our center point based approach, CenterNet, is end-to-end differentiable, simpler, faster, and more accurate than corresponding bounding box based detectors. CenterNet achieves the best speed-accuracy trade-off on the MS COCO dataset, with 28.1% AP at 142 FPS, 37.4% AP at 52 FPS, and 45.1% AP with multi-scale testing at 1.4 FPS. We use the same approach to estimate 3D bounding box in the KITTI benchmark and human pose on the COCO keypoint dataset. Our method performs competitively with sophisticated multi-stage methods and runs in real-time.


Compared with A2J #REFR , certain performance improvement can be achieved on two datasets, which proves that A2J-Transformer has a strong generalization ability.

NYU Table 5 . Performance comparison on HANDS 2017 dataset #OTHEREFR . Our method can get state-of-the-art performance on this dataset.

state-of-the-art depth based single hand estimation methods on NYU and HANDS 2017 dataset are given in Table 4 and Table 5 .

Since A2J-Transformer is not specifically designed for single hand estimation on depth image, we just changed the input channel to verify the generalization ability of our model through this experiment. We can summarize that:

• Although A2J-Transformer is based on the RGB image of interacting hands, it still achieves state-of-the-art performance on HANDS 2017 dataset and gets comparable result on NYU dataset.

This relys on the strong ability of A2J-Transformer to grasp the articulated hand information and the fitting ability of 3D anchor points.

A2J-Transformer: Anchor-To-Joint Transformer Network For 3D Interacting Hand Pose Estimation From A Single Rgb Image

Abstract: 3D interacting hand pose estimation from a single RGB image is a challenging task, due to serious self-occlusion and inter-occlusion towards hands, confusing similar appearance patterns between 2 hands, ill-posed joint position mapping from 2D to 3D, etc.. To address these, we propose to extend A2J-the state-of-the-art depth-based 3D single hand pose estimation method-to RGB domain under interacting hand condition. Our key idea is to equip A2J with strong local-global aware ability to well capture interacting hands' local fine details and global articulated clues among joints jointly. To this end, A2J is evolved under Transformer's non-local encoding-decoding framework to build A2J-Transformer. It holds 3 main advantages over A2J. First, self-attention across local anchor points is built to make them global spatial context aware to better capture joints' articulation clues for resisting occlusion. Secondly, each anchor point is regarded as learnable query with adaptive feature learning for facilitating pattern fitting capacity, instead of having the same local representation with the others. Last but not least, anchor point locates in 3D space instead of 2D as in A2J, to leverage 3D pose prediction. Experiments on challenging InterHand 2.6M demonstrate that, A2J-Transformer can achieve state-ofthe-art model-free performance (3.38mm MPJPE advancement in 2-hand case) and can also be applied to depth domain with strong generalization. The code is avaliable at https://github.com/ChanglongJiangGit/ A2J-Transformer.

A2J: Anchor-To-Joint Regression Network For 3D Articulated Pose Estimation From A Single Depth Image

Abstract: For 3D hand and body pose estimation task in depth image, a novel anchor-based approach termed Anchor-to-Joint regression network (A2J) with the end-to-end learning ability is proposed. Within A2J, anchor points able to capture global-local spatial context information are densely set on depth image as local regressors for the joints. They contribute to predict the positions of the joints in ensemble way to enhance generalization ability. The proposed 3D articulated pose estimation paradigm is different from the state-of-the-art encoder-decoder based FCN, 3D CNN and point-set based manners. To discover informative anchor points towards certain joint, anchor proposal procedure is also proposed for A2J. Meanwhile 2D CNN (i.e., ResNet-50) is used as backbone network to drive A2J, without using time-consuming 3D convolutional or deconvolutional layers. The experiments on 3 hand datasets and 2 body datasets verify A2J's superiority. Meanwhile, A2J is of high running speed around 100 FPS on single NVIDIA 1080Ti GPU.


While each approach is based on PoseNet #REFR , they can be applied to other pose estimation models.

This section presents a chronological review of the results from the four frameworks proposed for conformalized visual odometry (CVO) and compared in Table I .

The aim of these methods is to develop reliable prediction intervals that can capture true uncertainty in data-driven pose estimation with guaranteed confidence.

In highlighting the differences between the methods, we use the data from scene 2 of RGB-D Scenes Dataset v.2 #OTHEREFR as visuals of its pose trajectory are easy to comprehend.

All results illustrate uncertainty bands that exhibit at least 90% guaranteed marginal coverage through CI.

III-A, we introduced univariate conformalized quantile regression (CQR) to generate uncertainty bands. However, as shown in Fig.

5 , the resulting prediction interval is overly cautious and box-shaped despite its ability to adjust to the data's heteroscedasticity.

Lightweight, Uncertainty-Aware Conformalized Visual Odometry

Abstract: Data-driven visual odometry (VO) is a critical subroutine for autonomous edge robotics, and recent progress in the field has produced highly accurate point predictions in complex environments. However, emerging autonomous edge robotics devices like insect-scale drones and surgical robots lack a computationally efficient framework to estimate VO's predictive uncertainties. Meanwhile, as edge robotics continue to proliferate into mission-critical application spaces, awareness of model's the predictive uncertainties has become crucial for risk-aware decision-making. This paper addresses this challenge by presenting a novel, lightweight, and statistically robust framework that leverages conformal inference (CI) to extract VO's uncertainty bands. Our approach represents the uncertainties using flexible, adaptable, and adjustable prediction intervals that, on average, guarantee the inclusion of the ground truth across all degrees of freedom (DOF) of pose estimation. We discuss the architectures of generative deep neural networks for estimating multivariate uncertainty bands along with point (mean) prediction. We also present techniques to improve the uncertainty estimation accuracy, such as leveraging Monte Carlo dropout (MC-dropout) for data augmentation. Finally, we propose a novel training loss function that combines interval scoring and calibration loss with traditional training metrics-mean-squared error and KLdivergence-to improve uncertainty-aware learning. Our simulation results demonstrate that the presented framework consistently captures true uncertainty in pose estimations across different datasets, estimation models, and applied noise types, indicating its wide applicability.

Posenet: A Convolutional Network For Real-Time 6-Dof Camera Relocalization

Abstract: We present a robust and real-time monocular six degree of freedom relocalization system. Our system trains a convolutional neural network to regress the 6-DOF camera pose from a single RGB image in an end-to-end manner with no need of additional engineering or graph optimisation. The algorithm can operate indoors and outdoors in real time, taking 5ms per frame to compute. It obtains approximately 2m and 3 degrees accuracy for large scale outdoor scenes and 0.5m and 5 degrees accuracy indoors. This is achieved using an efficient 23 layer deep convnet, demonstrating that convnets can be used to solve complicated out of image plane regression problems. This was made possible by leveraging transfer learning from large scale classification data. We show that the PoseNet localizes from high level features and is robust to difficult lighting, motion blur and different camera intrinsics where point based SIFT registration fails. Furthermore we show how the pose feature that is produced generalizes to other scenes allowing us to regress pose with only a few dozen training examples.


As Table 7 shows, B-Seg performs favorably over other methods for most sub-categories, except that it performs slightly worse than PointGroup #REFR on three categories in the Campus.

2022] , though it is clusteringfree, it needs to encode every instance proposal into a kernel feature for the dynamic convolution. While DyCo3D #OTHEREFR

2021 ] also adopts a dynamic-convolution-based model, it still requires a pointwise grouping first.

On the contrary, B-Seg adopts a simple but effective strategy by aggregating points with a relation matrix and merging points only through a small set of candidates. The visualization result can be seen in Fig. 6 .

Fine-grained Building Instance Segmentation Performance.

We further evaluate the segmentation performance of all methods on the building sub-categories.

These categories are overall harder to distinguish and their features are more similar in Campus, so requiring a stronger feature extractor.

Note also that we do not have results for some categories in Wuhu and Longhua (see "-" in the table), as the number of associated buildings in these cases is either too small or not existent.

From the experiments, we can see the strength of B-Seg in analyzing buildings in 3D urban scenes and its potential for supporting building-aware 3D machine learning tasks.

We train the methods collectively on samples of all scenes.

Urbanbis: A Large-Scale Benchmark For Fine-Grained Urban Building Instance Segmentation

Abstract: Figure 1: UrbanBIS provides 2.5 billion 3D point samples over six scenes, covering a vast area of 10.78 km 2. Particularly, this large 3D dataset is annotated not only in the urban semantic level (top right) but also in the building instance level (bottom right) with fine-grained building categories (bottom left).

Pointgroup: Dual-Set Point Grouping For 3D Instance Segmentation

Abstract: Instance segmentation is an important task for scene understanding. Compared to the fully-developed 2D, 3D instance segmentation for point clouds have much room to improve. In this paper, we present PointGroup, a new end-to-end bottom-up architecture, specifically focused on better grouping the points by exploring the void space between objects. We design a two-branch network to extract point features and predict semantic labels and offsets, for shifting each point towards its respective instance centroid. A clustering component is followed to utilize both the original and offset-shifted point coordinate sets, taking advantage of their complementary strength. Further, we formulate the ScoreNet to evaluate the candidate instances, followed by the Non-Maximum Suppression (NMS) to remove duplicates. We conduct extensive experiments on two challenging datasets, ScanNet v2 and S3DIS, on which our method achieves the highest performance, 63.6% and 64.0%, compared to 54.9% and 54.4% achieved by former best solutions in terms of mAP with IoU threshold 0.5.