In the white box attack, the adversary has complete access to the target model, including the model architecture and parameters, allowing the attacker to use the model's gradient to produce the most effective adversarial examples #REFR .

The adversarial example is an input to machine learning models that an attacker has deliberately designed to cause the model to make a mistake.

In particular, it is a corrupted version of a valid input, where the corruption is done by adding a perturbation of a small magnitude to it.

The adversarial example aims to appear "normal" to humans but causes misclassification by the targeted machine learning model.

An adversarial attack is then a method to generate adversarial examples.

We distinguish two kinds of adversarial attacks: a white box attack and a black box attack.

On the contrary, in the black box attack scenario, the attackers do not have full access to the model and can only observe the output of its prediction #OTHEREFR . Fig. 1 . The proposed scheme.

We consider three neural networks (NN): classification, simulation, and adversarial.

The classification NN is a black box trained on an unknown 'target' dataset and presented by an unknown architecture.

The simulation NN is not trainable and has one layer with 'simulation' kernels and serves for computing loss difference between the original and the adversarial output.

The adversarial NN is a simple architecture, highly distinct from the black box one and trained on a different 'adversarial' dataset.

Effective Black Box Adversarial Attack With Handcrafted Kernels

Abstract: We propose a new, simple framework for crafting adversarial examples for black box attacks. The idea is to simulate the substitution model with a non-trainable model compounded of just one layer of handcrafted convolutional kernels and then train the generator neural network to maximize the distance of the outputs for the original and generated adversarial image. We show that fooling the prediction of the first layer causes the whole network to be fooled and decreases its accuracy on adversarial inputs. Moreover, we do not train the neural network to obtain the first convolutional layer kernels, but we create them using the technique of F-transform. Therefore, our method is very time and resource effective.

Intriguing Properties Of Neural Networks

Abstract: Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. ::: First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. ::: Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. We can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.


We train our model on Voxceleb2 #REFR , which consists of in-the-wild talking face videos collected from YouTube.

It contains more than 145,000 videos of about 6,100 identities in the train set and more than 4,900 videos of more than 110 identities in the test set.

We convert all videos into frames with 25 fps and then crop and resize those frames into 256 × 256 resolution, following the method of #OTHEREFR .

For more semantically rich face priors, we only use videos where the detected face bounding boxes' height (and width) is longer than 256.

After pre-processing, the remaining 11051 videos and 340 videos are used as the training set and the test set, respectively.

All audios are re-sampled to 16kHz and converted into a mel-spectrogram to be used as our audio representation similar to the method in #OTHEREFR .

Stylelipsync: Style-Based Personalized Lip-Sync Video Generation

Abstract: In this paper, we present StyleLipSync, a style-based personalized lip-sync video generative model that can generate identity-agnostic lip-synchronizing video from arbitrary audio. To generate a video of arbitrary identities, we leverage expressive lip prior from the semantically rich latent space of a pre-trained StyleGAN, where we can also design a video consistency with a linear transformation. In contrast to the previous lip-sync methods, we introduce pose-aware masking that dynamically locates the mask to improve the naturalness over frames by utilizing a 3D parametric mesh predictor frame by frame. Moreover, we propose a few-shot lip-sync adaptation method for an arbitrary person by introducing a sync regularizer that preserves lips-sync generalization while enhancing the person-specific visual information. Extensive experiments demonstrate that our model can generate accurate lip-sync videos even with the zeroshot setting and enhance characteristics of an unseen face using a few seconds of target video through the proposed adaptation method. Please refer to our project page.

Voxceleb2: Deep Speaker Recognition

Abstract: The objective of this paper is speaker recognition under noisy and unconstrained conditions. ::: We make two key contributions. First, we introduce a very large-scale audio-visual speaker recognition dataset collected from open-source media. Using a fully automated pipeline, we curate VoxCeleb2 which contains over a million utterances from over 6,000 speakers. This is several times larger than any publicly available speaker recognition dataset. ::: Second, we develop and compare Convolutional Neural Network (CNN) models and training strategies that can effectively recognise identities from voice under various conditions. The models trained on the VoxCeleb2 dataset surpass the performance of previous works on a benchmark dataset by a significant margin.


In contrast to the previous studies in #REFR , our noise model as given in (5) incorporating channel fading effects and non-coherent detection at the BS side is more intricate.

In #OTHEREFR , the authors considered BP based GT decoding under a noise model that combines the addition and dilution models.

In the context of mMTC, BP has been previously studied as a means for unsourced random access the users share the same codebook and the DRAFT March 14, 2023 decoder is only required to provide an unordered list of user messages #OTHEREFR .

Recognizing the suitability of BP in sparse recovery problems including GT, we aim to employ a BP based strategy for the problem of active device identification in the non-coherent

In BP based user identification, we estimate the set of active devices by approximately performing bitwise-MAP detection through message passing on a bipartite graph with devices on one side and the binary energy measurements on the other side #OTHEREFR .

Note that in #OTHEREFR , Polyanskiy proposed an alternating BP scheme as a candidate practical solution for random access in a quasi-static fading multiple access channel where each user's transmissions are attenuated by random fading gains, unknown to the receiver.

However, #OTHEREFR considers iterative coding scheme based on LDPC codes and slow fading scenarios whereas our work focus on energy-efficient OOK signaling in a fast fading non-coherent ( , k)−MnAC.

To identify the activity status vector β using the binary vector of energy detector outputs Z = (Z 1 , . . .

, Z n ), BP relies on an approximate Maximum Aposteriori Probability (MAP)

based decoding where instead of maximizing the posterior as a function of all the β i 's, we use the marginals of the posterior distribution. i.e., β i = arg max

Non-Coherent Active Device Identification For Massive Random Access

Abstract: Massive Machine-Type Communications (mMTC) is a key service category in the current generation of wireless networks featuring an extremely high density of energy and resource-limited devices with sparse and sporadic activity patterns. In order to enable random access in such mMTC networks, base station needs to identify the active devices while operating within stringent access delay constraints. In this paper, an energy efficient active device identification protocol is proposed in which active devices transmit On-Off Keying (OOK) modulated preambles jointly and base station employs non-coherent energy detection avoiding channel estimation overheads. The minimum number of channel-uses required by the active user identification protocol is characterized in the asymptotic regime of total number of devices when the number of active devices k scales as k = Θ(1) along with an achievability scheme relying on the equivalence of activity detection to a group testing problem. Several practical schemes based on Belief Propagation (BP) and Combinatorial Orthogonal Matching Pursuit (COMP) are also proposed. Simulation results show that BP strategies outperform COMP significantly and can operate close to the theoretical achievability bounds. In a partial-recovery setting where few misdetections are allowed, BP continues to perform well.

Note On Noisy Group Testing: Asymptotic Bounds And Belief Propagation Reconstruction

Abstract: An information theoretic perspective on group testing problems has recently been proposed by Atia and Saligrama, in order to characterise the optimal number of tests. Their results hold in the noiseless case, where only false positives occur, and where only false negatives occur. We extend their results to a model containing both false positives and false negatives, developing simple information theoretic bounds on the number of tests required. Based on these bounds, we obtain an improved order of convergence in the case of false negatives only. Since these results are based on (computationally infeasible) joint typicality decoding, we propose a belief propagation algorithm for the detection of defective items and compare its actual performance to the theoretical bounds.


Most of these models have been benchmarked on large datasets such as COCO, Cityscapes (Lin et al., 2014; #REFR .

Among them, Semantic Segmentation is an important task where the model has to learn the classification of each pixel to a semantic class. In 2014, Jonathan Long et al.

proposed a Fully Convolutional Network (FCN), which was the first CNN with only convolutional and transposed convolutional layers and without any fully connected (or dense) layers #OTHEREFR .

With the advent of FCNs, researchers have proposed and developed several other states of the art models for the semantic segmentation task.

Among them, the popular ones include -UNet, LinkNet, PSPNet, DeepLabV3+ #OTHEREFR .

These models have one thing in common -i.e., they are all variations of encoder-decoder architectures.

Cityscapes, a labeled dataset used as a benchmark for semantic segmentation tasks, contains 5000 high quality labeled images and 20000 weakly labeled images. Marios Krestenitis et al.

benchmarked some of these models on the Oil Spill Detection Dataset, a dataset developed by compiling the images extracted from the satellite Synthetic Aperture Radar (SAR) data #OTHEREFR .

This dataset contains a meager 1002 training image, which is relatively much lower than the Cityscapes dataset. In their research, Marios Krestenitis et al.

trained the models by dividing the original images of size 1250 × 650 into multiple patches.

They used the image patches as input for the models. They used different image patch sizes for different models.

Oil Spill Segmentation Using Deep Encoder-Decoder Models

Abstract: Crude oil is an integral component of the modern world economy. With the growing demand for crude oil due to its widespread applications, accidental oil spills are unavoidable. Even though oil spills are in and themselves difficult to clean up, the first and foremost challenge is to detect spills. In this research, the authors test the feasibility of deep encoder-decoder models that can be trained effectively to detect oil spills. The work compares the results from several segmentation models on high dimensional satellite Synthetic Aperture Radar (SAR) image data. Multiple combinations of models are used in running the experiments. The best-performing model is the one with the ResNet-50 encoder and DeepLabV3+ decoder 1. It achieves a mean Intersection over Union (IoU) of 64.868% and a class IoU of 61.549% for the "oil spill" class when compared with the current benchmark model, which achieved a mean IoU of 65.05% and a class IoU of 53.38% for the "oil spill" class.

The Cityscapes Dataset For Semantic Urban Scene Understanding

Abstract: Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotations, 20 000 additional images have coarse annotations to enable methods that leverage large volumes of weakly-labeled data. Crucially, our effort exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity. Our accompanying empirical study provides an in-depth analysis of the dataset characteristics, as well as a performance evaluation of several state-of-the-art approaches based on our benchmark.


This method is limited by the often inaccurate (uncalibrated) predictive confidence learned by the neural network #REFR .

Boundary Calibration The classical local outlier factor (LOF) #OTHEREFR method is a custom metric that calculates the local density deviation of a data point from its neighbors.

However, there is not a principled rule on how to choose the outlier detection threshold when using LOF. #OTHEREFR ; #OTHEREFR

(2022) added open category data into the validation set to estimate or grid-search the proper threshold.

So motivated, #OTHEREFR fit the output logits of the classifier to a Weibull distribution, but still use a validation set that contains the open category to select the confidence threshold. Further, #OTHEREFR

(2017) employed one-versus-rest binary classifiers and then calculates the threshold over the confidence score space by fitting it to a Gaussian distribution.

Adaptive decision boundary #OTHEREFR , illustrated in Figure 1 (c), was recently proposed to learn bounded spherical regions for known categories to contain known class samples.

Though this post-processing approach achieves state-of-theart performance, it still suffers from the issue that the tight-bounded spheres may not exist or cannot be well-defined in representation space.

Due to the fact that high-dimensional data representations usually lie on a low-dimensional manifold #OTHEREFR , a sphere defined in a Euclidean space can be restrictive as a decision boundary.

Moreover, the issue can be more severe if certain categories follow multimodal or skewed distributions.

Representation Learning DeepUnk (Lin and Xu, 2019a) trains the feature extractor with Large Margin Cosine Loss #OTHEREFR .

Open World Classification With Adaptive Negative Samples

Abstract: Open world classification is a task in natural language processing with key practical relevance and impact. Since the open or unknown category data only manifests in the inference phase, finding a model with a suitable decision boundary accommodating for the identification of known classes and discrimination of the open category is challenging. The performance of existing models is limited by the lack of effective open category data during the training stage or the lack of a good mechanism to learn appropriate decision boundaries. We propose an approach based on adaptive negative samples (ANS) designed to generate effective synthetic open category samples in the training stage and without requiring any prior knowledge or external datasets. Empirically, we find a significant advantage in using auxiliary one-versus-rest binary classifiers, which effectively utilize the generated negative samples and avoid the complex thresholdseeking stage in previous works. Extensive experiments on three benchmark datasets show that ANS achieves significant improvements over state-of-the-art methods.

On Calibration Of Modern Neural Networks

Abstract: Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.


By contrast, JT-VAE #REFR , a motif-sequence-based generator, generates molecular graphs by a procedure that generates a treestructured scaffold over chemical substructures first and then combines them into a molecule.

When generating a graph, there is often a constraint demand that only specific combinations of the nodes and edges are valid in context #OTHEREFR , #OTHEREFR , #OTHEREFR .

RGVAE #OTHEREFR generates a semantically valid graph by adding a penalty term that regularizes the output distribution of the decoder to the objective function of VAEs.

Graphite #OTHEREFR repeats the process that constructs an intermediate weighted graph and iteratively refines the reconstructed feature matrix via message passing.

The final distribution of graph is obtained via the inner product of feature matrix. The other category is sequential generation method.

CGVAE #OTHEREFR , node-by-node generator, repeats the process that chooses which node is connected to the current node and selects and labels edges between nodes to generate a graph.

But Autoencoderbassed graph generation methods have limitations in that they are not scalable to be applied on large graphs.

A Survey On Graph Diffusion Models: Generative Ai In Science For Molecule, Protein And Material

Abstract: Diffusion models have become a new SOTA generative modeling method in various fields, for which there are multiple survey works that provide an overall survey. With the number of articles on diffusion models increasing exponentially in the past few years, there is an increasing need for survey of diffusion models on specific fields. In this work, we are committed to conducting a survey on the graph diffusion models. Even though our focus is to cover the progress of diffusion models in graphs, we first briefly summarize how other generative modeling methods are used for graphs. After that, we introduce the mechanism of diffusion models in various forms, which facilitates the discussion on the graph diffusion models. The applications of graph diffusion models mainly fall into the catogory of AI-generated content (AIGC) in science, for which we mainly focus on how graph diffusion models are utilized for generating molecules and proteins but also cover other cases, including materials design. Moreover, we discuss the issue of evaluating diffusion models in the graph domain and the existing challenges.

Junction Tree Variational Autoencoder For Molecular Graph Generation

Abstract: We seek to automate the design of molecules based on specific chemical properties. In computational terms, this task involves continuous embedding and generation of molecular graphs. Our primary contribution is the direct realization of molecular graphs, a task previously approached by generating linear SMILES strings instead of graphs. Our junction tree variational autoencoder generates molecular graphs in two phases, by first generating a tree-structured scaffold over chemical substructures, and then combining them into a molecule with a graph message passing network. This approach allows us to incrementally expand molecules while maintaining chemical validity at every step. We evaluate our model on multiple tasks ranging from molecular generation to optimization. Across these tasks, our model outperforms previous state-of-the-art baselines by a significant margin.


Since SuperQuadrics #REFR uses a restricted family of 3D shapes to reconstruct an input scene, it has worst qualitative performances for instance segmentation when compared to our model, which learns scene-specific prototypes and can provide semantic information. Table 7 : Results on the Earth Parser Dataset.

We show in Figure 9 the ground truth semantic segmentation, our predicted semantic segmentation, our reconstruction and our learned prototypes.

They showcase the quality, interpretability, and diversity of use cases of our model on this dataset of aerial LiDAR scans.

We show in Figure 10 a comparison of the instance segmentation produced by Su-perQuadrics #OTHEREFR and our Learnable Earth Parser.

We report the quality of the reconstruction (Cham.) and semantic segmentation (mIoU) for the models presented in the main paper and other variations.

Bold numbers indicate the best results of the models shown in the main paper, while green bold numbers indicate the best results across all variations.

↓ indicates that the variation results in a significant drop in performance, while ↑ indicates a performance boost.

We also show the number of prototypes selected by our post-processing selection algorithm. Ours Figure 10 : Instance Segmentation.

We represent the instances predicted with our algorithm and by SuperQuadric #OTHEREFR .

Learnable Earth Parser: Discovering 3D Prototypes In Aerial Scans

Abstract: We propose an unsupervised method for parsing large 3D scans of real-world scenes into interpretable parts. Our goal is to provide a practical tool for analyzing 3D scenes with unique characteristics in the context of aerial surveying and mapping, without relying on applicationspecific user annotations. Our approach is based on a probabilistic reconstruction model that decomposes an input 3D point cloud into a small set of learned prototypical shapes. Our model provides an interpretable reconstruction of complex scenes and leads to relevant instance and semantic segmentations. To demonstrate the usefulness of our results, we introduce a novel dataset of seven diverse aerial LiDAR scans. We show that our method outperforms state-of-the-art unsupervised methods in terms of decomposition accuracy while remaining visually interpretable. Our method offers significant advantage over existing approaches, as it does not require any manual annotations, making it a practical and efficient tool for 3D scene analysis. Our code and dataset are available at https://imagine.enpc.fr/˜loiseaur/ learnable-earth-parser. Windturbines Forest Crop fields Urban Power plant Greenhouses Marina

Superquadrics Revisited: Learning 3D Shape Parsing Beyond Cuboids

Abstract: ing complex 3D shapes with parsimonious part-based representations has been a long standing goal in computer vision. This paper presents a learning-based solution to this problem which goes beyond the traditional 3D cuboid representation by exploiting superquadrics as atomic elements. We demonstrate that superquadrics lead to more expressive 3D scene parses while being easier to learn than 3D cuboid representations. Moreover, we provide an analytical solution to the Chamfer loss which avoids the need for computational expensive reinforcement learning or iterative prediction. Our model learns to parse 3D objects into consistent superquadric representations without supervision. Results on various ShapeNet categories as well as the SURREAL human body dataset demonstrate the flexibility of our model in capturing fine details and complex poses that could not have been modelled using cuboids.


For each task, we calculate the Exact Match and F1 score as defined in #REFR .

First, for the few-shot examples we only provide the line within the paragraph containing the answer as the context.

Second, for the test example, we index the chunks of the context using the embeddings from the text-embedding-ada-002 model.

Given the question, the closest chunk in the full context is retrieved and used in the prompt for the test example.

We use a maximum chunk size of 100 in our experiments and use the implementation for retrieval provided in the LangChain 3 library.

By doing this,we minimize the space taken by the context tokens in our prompt.

For our experiments we consider the following three tasks: 1.

TyDiQA (Clark et al., 2020) is a QA dataset covering 11 typologically diverse languages.

The task consists of two sub-tasks -passage selection and minimum answer span (Gold-P).

For our experiments, we consider the Gold-P task and evaluate Monolingual and Zero-Shot Cross-Lingual prompting strategies.

Since the labels do not directly transfer one-to-one across translation for QA tasks as they do for classification and require the use of alignment algorithms, we skip translate-test prompting for this task. 2.

Mega: Multilingual Evaluation Of Generative Ai

Abstract: Generative AI models have impressive performance on many Natural Language Processing tasks such as language understanding, reasoning and language generation. One of the most important questions that is being asked by the AI community today is about the capabilities and limits of these models, and it is clear that evaluating generative AI is very challenging. Most studies on generative LLMs are restricted to English and it is unclear how capable these models are at understanding and generating other languages. We present the first comprehensive benchmarking of generative LLMs-MEGA, which evaluates models on standard NLP benchmarks, covering 8 diverse tasks and 33 typologically diverse languages. We also compare the performance of generative LLMs to State of the Art (SOTA) non-autoregressive models on these tasks to determine how well generative models perform compared to the previous generation of LLMs. We present a thorough analysis of the performance of models across languages and discuss some of the reasons why generative LLMs are currently not optimal for all languages. We create a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.

Squad: 100,000+ Questions For Machine Comprehension Of Text

Abstract: We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. ::: The dataset is freely available at this https URL


This result is not surprising, as RMSProp and ADAM are known for yielding higher convergence speed than SGD since they use more information: the moving average of gradients helps RMSProp and ADAM driving the optimization towards the optimum #REFR .

This probability distribution verifies the hypothesis of our theorem, hence it brings theoretically an improvement on the convergence speed compared to the uniform sampling scheme. The experiments show this improvement is not only theoretical.

For SGD and SGD with momentum performed with p 1/2 , the loss decrease is faster than the loss decrease obtained with the uniform sampling, and slower than the one obtained with the optimal sampling scheme (in terms of iterations). Fig.

2 lower right compares SGD, SGD with momentum, RMSProp and ADAM with the uniform sampling for completeness.

Even though this comparison is not the core of our study, the following can be noted.

First, RMSProp and ADAM provide a faster loss decrease in the early stages of the optimization process than SGD (with or without momentum).

However, more surprising is the final result of the optimization: the final loss is lower for SGD than for RMSProp and ADAM.

This can be justified by the bias induced by the update equations of RMSProp and ADAM.

Whereas SGD minimizes the empirical loss, as proven by #OTHEREFR , it is not the case for RMSProp and ADAM. Fig.

2 cannot be fully appreciated without Table 1 , where the computational time of each sampling scheme used in this work is reported.

Whatever the optimizer, the computational time is approximately the same for a given sampling scheme.

Importance Sampling For Stochastic Gradient Descent In Deep Neural Networks

Abstract: Stochastic gradient descent samples uniformly the training set to build an unbiased gradient estimate with a limited number of samples. However, at a given step of the training process, some data are more helpful than others to continue learning. Importance sampling for training deep neural networks has been widely studied to propose sampling schemes yielding better performance than the uniform sampling scheme. After recalling the theory of importance sampling for deep learning, this paper reviews the challenges inherent to this research area. In particular, we propose a metric allowing the assessment of the quality of a given sampling scheme; and we study the interplay between the sampling scheme and the optimizer used. Preprint. Under review.

Adam: A Method For Stochastic Optimization

Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.


Internal Lr Feature Guided Multiscale Attention

In this study, scaled dot-product attention #REFR is chosen to implement multiscale attention. In Fig.

The details of multiscale attention for internal LR and HR feature fusion are shown in Fig. 7 .

Unlike the original attention model in Transformer #OTHEREFR , three types of inputs, key (K), query (Q), and value (V), are visual feature maps, and K has a different scale than Q and V.

In other words, K is small-scale and takes linearly transformed internal LR features ℋ(ℓ) as the input. In addition, K represents deformed LR features. In contrast, Q and V are large-scale.

The internal HR feature maps are assigned to Q and V after applying linear transformation.

6 , the HR hourglass follows the encoder-decoder framework, and thus, multiscale attention is implemented slightly differently depending on the encoder and decoder.

For the encoder of the HR hourglass, multiscale attention is designed as follows: Here, represents the internal HR features transferred by the encoder via within-hourglass skip connection, as shown in Fig.

6 , and then added to the scaled dot-product attention result.

In the proposed multiscale attention, internal LR features are jointly learned with internal HR features via between-hourglass skip connection.

Trap-Based Pest Counting: Multiscale And Deformable Attention Centernet Integrating Internal Lr And Hr Joint Feature Learning

Abstract: Pest counting, which predicts the number of pests in the early stage, is very important because it enables rapid pest control, reduces damage to crops, and improves productivity. In recent years, light traps have been increasingly used to lure and photograph pests for pest counting. However, pest images have a wide range of variability in pest appearance owing to severe occlusion, wide pose variation, and even scale variation. This makes pest counting more challenging. To address these issues, this study proposes a new pest counting model referred to as multiscale and deformable attention CenterNet (Mada-CenterNet) for internal low-resolution (LR) and high-resolution (HR) joint feature learning. Compared with the conventional CenterNet, the proposed Mada-CenterNet adopts a multiscale heatmap generation approach in a two-step fashion to predict LR and HR heatmaps adaptively learned to scale variations, that is, changes in the number of pests. In addition, to overcome the pose and occlusion problems, a new between-hourglass skip connection based on deformable and multiscale attention is designed to ensure internal LR and HR joint feature learning and incorporate geometric deformation, thereby resulting in an improved pest counting accuracy. Through experiments, the proposed Mada-CenterNet is verified to generate the HR heatmap more accurately and improve pest counting accuracy owing to multiscale heatmap generation, joint internal feature learning, and deformable and multiscale attention. In addition, the proposed model is confirmed to be effective in overcoming severe occlusions and variations in pose and scale. The experimental results show that the proposed model outperforms state-of-the-art crowd counting and object detection models.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


As the projection only checks against intersection in the endconfiguration of the validation trajectory, it is not guaranteed that the resulting projected trajectory is intersection-free in #REFR .

G ([0, α opt ]) O (o) exp = ∅, α opt < 1 − α opt < α opt, min , 2 A ⊕ B = {a + v | b ∈ A, v ∈ B}

where we adapt α opt, min to the robot dynamics.

Finally, we use a trajectory planner to plan a trajectory χ P from p (r) 0 to an ISS in an -bound around c (r) V , which ensures that the first L steps of the trajectory have a constant control input.

Therefore, if χ P would result in an intersection, we can repetitively reduce α for M times to find a more conservative projection point.

If no temporarily safe trajectory χ P can be found, we execute the zero action and continue to ensure safety with our safety shield.

Reducing Safety Interventions In Provably Safe Reinforcement Learning

Abstract: Deep Reinforcement Learning (RL) has shown promise in addressing complex robotic challenges. In real-world applications, RL is often accompanied by failsafe controllers as a last resort to avoid catastrophic events. While necessary for safety, these interventions can result in undesirable behaviors, such as abrupt braking or aggressive steering. This paper proposes two safety intervention reduction methods: action replacement and projection, which change the agent's action if it leads to an unsafe state. These approaches are compared to the state-of-the-art constrained RL on the OpenAI safety gym benchmark and a human-robot collaboration task. Our study demonstrates that the combination of our method with provably safe RL leads to high-performing policies with zero safety violations and a low number of failsafe interventions. Our versatile method can be applied to a wide range of realworld robotics tasks, while effectively improving safety without sacrificing task performance.

Virtual-To-Real Deep Reinforcement Learning: Continuous Control Of Mobile Robots For Mapless Navigation

Abstract: We present a learning-based mapless motion planner by taking the sparse 10-dimensional range findings and the target position with respect to the mobile robot coordinate frame as input and the continuous steering commands as output. Traditional motion planners for mobile ground robots with a laser range sensor mostly depend on the obstacle map of the navigation environment where both the highly precise laser sensor and the obstacle map building work of the environment are indispensable. We show that, through an asynchronous deep reinforcement learning method, a mapless motion planner can be trained end-to-end without any manually designed features and prior demonstrations. The trained planner can be directly applied in unseen virtual and real environments. The experiments show that the proposed mapless motion planner can navigate the nonholonomic mobile robot to the desired targets without colliding with any obstacles.


This type of error guarantees were previously provided by a body of work on coresets for logistic regression; see the works of Munteanu et al. #REFR , Mai et al.

In this section, we consider approximations to the logistic loss function.

#OTHEREFR for state-of-the-art results and bounds and references therein for motivation and prior work on this topic.

We begin by proving a general lower bound on the space complexity of any data structure which approximates logistic loss to ǫ-relative error for every parameter vector β ∈ R d .

Our lower bound shows that the coreset construction of Mai et al.

#OTHEREFR has optimal space complexity up to anÕ(d • µ y (X) 2 ) factor, where µ y (X) 2 is a data complexity measure introduced by Munteanu et al. #OTHEREFR (see also Definition 1 below).

Furthermore, we provide an efficient linear programming formulation to compute this data complexity measure on real data sets, refuting an earlier conjecture that the factor was hard to compute #OTHEREFR .

Feature Space Sketching For Logistic Regression

Abstract: We present novel bounds for coreset construction, feature selection, and dimensionality reduction for logistic regression. All three approaches can be thought of as sketching the logistic regression inputs. On the coreset construction front, we resolve open problems from prior work and present novel bounds for the complexity of coreset construction methods. On the feature selection and dimensionality reduction front, we initiate the study of forward error bounds for logistic regression. Our bounds are tight up to constant factors and our forward error bounds can be extended to Generalized Linear Models.

On Coresets For Logistic Regression

Abstract: Coresets are one of the central methods to facilitate the analysis of large data sets. We continue a recent line of research applying the theory of coresets to logistic regression. First, we show a negative result, namely, that no strongly sublinear sized coresets exist for logistic regression. To deal with intractable worst-case instances we introduce a complexity measure $\mu(X)$, which quantifies the hardness of compressing a data set for logistic regression. $\mu(X)$ has an intuitive statistical interpretation that may be of independent interest. For data sets with bounded $\mu(X)$-complexity, we show that a novel sensitivity sampling scheme produces the first provably sublinear $(1\pm\varepsilon)$-coreset. We illustrate the performance of our method by comparing to uniform sampling as well as to state of the art methods in the area. The experiments are conducted on real world benchmark data for logistic regression.


Visualization Of The Consistency Of Orientation Estimation

To visualize Figure 9 , we create a sequence of 480 × 640 images augmented by random in-plane rotation with Gaussian noise sourced by ILSVRC2012 #REFR .

We provide more examples for Figure 5 of the main paper, which visualize the consistency of orientation estimation. Additionally, we show the similarity map w.r.t. a keypoint under varying rotations.

Figure 9 shows the qualitative comparison of the estimated orientation consistency.

Given the dominant orientations estimated from the image pair, we calculate the relative angle between the corresponding keypoint orientations and measure the difference between the relative angle and the ground-truth rotation.

We evaluate the relative angle to be correct i.e., the dominant orientation estimation is consistent if the difference with the ground-truth rotation is within a 30 • threshold.

Our rotation-equivariant model trained with the orientation alignment loss inspired by #OTHEREFR consistently estimates more correct keypoint orientations than LF-Net #OTHEREFR and RF-Net #OTHEREFR .

Figure 10 shows the similarity maps with respect to a keypoint under varying rotations of images with a resolution of 180 × 180, with uniform rotation intervals of 45 • .

Learning Rotation-Equivariant Features For Visual Correspondence

Abstract: Extracting discriminative local features that are invariant to imaging variations is an integral part of establishing correspondences between images. In this work, we introduce a self-supervised learning framework to extract discriminative rotation-invariant descriptors using groupequivariant CNNs. Thanks to employing group-equivariant CNNs, our method effectively learns to obtain rotationequivariant features and their orientations explicitly, without having to perform sophisticated data augmentations. The resultant features and their orientations are further processed by group aligning, a novel invariant mapping technique that shifts the group-equivariant features by their orientations along the group dimension. Our group aligning technique achieves rotation-invariance without any collapse of the group dimension and thus eschews loss of discriminability. The proposed method is trained end-to-end in a self-supervised manner, where we use an orientation alignment loss for the orientation estimation and a contrastive descriptor loss for robust local descriptors to geometric/photometric variations. Our method demonstrates state-of-the-art matching accuracy among existing rotationinvariant descriptors under varying rotation and also shows competitive results when transferred to the task of keypoint matching and camera pose estimation.

Imagenet Large Scale Visual Recognition Challenge

Abstract: The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements.


Unit disk graph: We give an algorithm to compute a maximum clique in a unit disk graph in O(n 2.5 log n)-time, which improves the previously best known running time of O(n 3 log n) #REFR .

In this paper we make significant progress on the maximum clique problem for unit disk graphs, disk graphs with disk radii lying in the interval [1, 1 + ε] , and intersection graphs of unit disks and axis-parallel rectangles.

Our algorithm is based on a divide-and-conquer approach that, unlike the previous algorithms that search a clique over all the lenses, shows how to efficiently merge solutions to the subproblems to achieve a faster time complexity.

Such techniques have previously been used to accelerate computation for other computational geometry problems, e.g., when finding a closest pair in a point set #OTHEREFR , but appeared to be highly non-trivial while adapting it for the unit disk graph setting.

Intersection graph of unit disks and axis-parallel rectangles: We extend the co-2-subdivision approach to prove a (4448/4449 ≈ 0.9997)-inapproximability result for computing a maximum clique in an intersection graph that contains both unit disks and axis-parallel rectangles, and thus improve the previously known inapproximability factor of 7633010347/7633010348 ≈ 0.9999 #OTHEREFR .

Note that the use of a co-2-subdivision approach was previously thought to be unlikely in this setting by Bonnet et al. #OTHEREFR .

The key idea behind our NP-hardness reduction is to show that every Hamiltonian cubic graph admits a well-behaved edge orientation and vertex labeling, i.e., its vertices can be labeled and the edges can be oriented such that every vertex has two outgoing or two incoming edges where the labels of these corresponding neighbors are consecutive.

Finding A Maximum Clique In A Disk Graph

Abstract: A disk graph is an intersection graph of disks in the Euclidean plane, where the disks correspond to the vertices of the graph and a pair of vertices are adjacent if and only if their corresponding disks intersect. The problem of determining the time complexity of computing a maximum clique in a disk graph is a long-standing open question that has been very well studied in the literature. The problem is known to be open even when the radii of all the disks are in the interval [1, (1 + ε)], where ε > 0. If all the disks are unit disks then there exists an O(n 3 log n)-time algorithm to compute a maximum clique, which is the best-known running time for over a decade. Although the problem of computing a maximum clique in a disk graph remains open, it is known to be APX-hard for the intersection graphs of many other convex objects such as intersection graphs of ellipses, triangles, and a combination of unit disks and axis-parallel rectangles. Here we obtain the following results.-We give an algorithm to compute a maximum clique in a unit disk graph in O(n 2.5 log n)-time, which improves the previously best known running time of O(n 3 log n) [Eppstein '09].-We extend a widely used 'co-2-subdivision approach' to prove that computing a maximum clique in a combination of unit disks and axis-parallel rectangles is NP-hard to approximate within 4448/4449 ≈ 0.9997. The use of a 'co-2-subdivision approach' was previously thought to be unlikely in this setting [Bonnet et al. '20]. Our result improves the previously known inapproximability factor of 7633010347/7633010348 ≈ 0.9999.-We show that the parameter minimum lens width of the disk arrangement may be used to make progress in the case when disk radii are in [1, (1 + ε)]. For example, if the minimum lens width is at least 0.265 and ε ≤ 0.0001, which still allows for non-Helly triples in the arrangement, then one can find a maximum clique in polynomial time.

Graph-Theoretic Solutions To Computational Geometry Problems

Abstract: Many problems in computational geometry are not stated in graph-theoretic terms, but can be solved efficiently by constructing an auxiliary graph and performing a graph-theoretic algorithm on it. Often, the efficiency of the algorithm depends on the special properties of the graph constructed in this way. We survey the art gallery problem, partition into rectangles, minimum-diameter clustering, rectilinear cartogram construction, mesh stripification, angle optimization in tilings, and metric embedding from this perspective.


In addition to name similarity, the ranking of candidates utilizes the cosine similarity between the sentence embeddings #REFR of the context and the descriptions of the candidate entity from Wikipedia and Wikidata.

Existing tools and models in NLP Workbench include: Named Entity Recognition The task, known as NER, is to identify mentions to entities such as people, organizations, and locations.

We incorporated the NER model from PURE #OTHEREFR , which achieved good performance by simply fine-tuning BERT #OTHEREFR .

Coreference Resolution To determine which entity a pronoun refers to, we adopted the heuristic algorithm by #OTHEREFR that is based on recency and type agreement.

Entity Linking Mentions to entities in the text are disambiguated and linked to Wikidata (Vrandečić and Krötzsch, 2014) entities.

Candidate entities are generated by a fuzzy match on name.

Nlp Workbench: Efficient And Extensible Integration Of State-Of-The-Art Text Mining Tools

Abstract: NLP Workbench is a web-based platform for text mining that allows non-expert users to obtain semantic understanding of large-scale corpora using state-of-the-art text mining models. The platform is built upon latest pretrained models and open source systems from academia that provide semantic analysis functionalities, including but not limited to entity linking, sentiment analysis, semantic parsing, and relation extraction. Its extensible design enables researchers and developers to smoothly replace an existing model or integrate a new one. To improve efficiency, we employ a microservice architecture that facilitates allocation of acceleration hardware and parallelization of computation. This paper presents the architecture of NLP Workbench and discusses the challenges we faced in designing it. We also discuss diverse use cases of NLP Workbench and the benefits of using it over other approaches. The platform is under active development, with its source code released under the MIT license 1. A website 2 and a short video 3 demonstrating our platform are also available.

Sentence-Bert: Sentence Embeddings Using Siamese Bert-Networks

Abstract: BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. ::: In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. ::: We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.


To address this limitation, we propose to divide the city into irregular areas based on its road network #REFR , and create two traffic flow prediction datasets incorporating POI data.

The POI-MetaBlock adopts the self-attention structure #OTHEREFR and integrates time information to achieve dynamic attention, enabling our module to capture traffic patterns dynamically.

We treat the POI distribution as metadata for parameter generation #OTHEREFR , which enables the model to accommodate different traffic patterns across various regions.

To model functionality correlations that are overlooked by the original geographic topology, we establish a second topological structure (represented by the red line in Figure 1 ) using cosine similarity of POI distribution and apply a graph convolution network #OTHEREFR to refine attention results in each region based on the functionality topological structure.

It is noteworthy that the proposed module can be easily integrated into conventional models through a residual connection, allowing existing models to additionally account for the impact of functional attributes on the original basis.

In commonly used traffic prediction datasets, the urban area is often obtained through the grid method, which can lead to a loss of the original functional areas of the city.

In summary, our contributions are mainly summarized into the following aspects:

• We propose a novel module named POI-MetaBlock, which leverages POI&time data to dynamically generate prediction parameters and fit traffic patterns of different areas at different times.

• The proposed POI-MetaBlock is a lightweight module that can be easily integrated into other traffic flow prediction models.

It allows the original model to take advantage of functional correlations during traffic flow prediction.

• We constructed two new datasets by partitioning the city into irregular areas using POI information.

Urban Regional Function Guided Traffic Flow Prediction

Abstract: The prediction of traffic flow is a challenging yet crucial problem in spatial-temporal analysis, which has recently gained increasing interest. In addition to spatial-temporal correlations, the functionality of urban areas also plays a crucial role in traffic flow prediction. However, the exploration of regional functional attributes mainly focuses on adding additional topological structures, ignoring the influence of functional attributes on regional traffic patterns. Different from the existing works, we propose a novel module named POI-MetaBlock, which utilizes the functionality of each region (represented by Point of Interest distribution) as metadata to further mine different traffic characteristics in areas with different functions. Specifically, the proposed POI-MetaBlock employs a self-attention architecture and incorporates POI and time information to generate dynamic attention parameters for each region, which enables the model to fit different traffic patterns of various areas at different times. Furthermore, our lightweight POI-MetaBlock can be easily integrated into conventional traffic flow prediction models. Extensive experiments demonstrate that our module significantly improves the performance of traffic flow prediction and outperforms state-of-the-art methods that use metadata.

Predicting Citywide Crowd Flows In Irregular Regions Using Multi-View Graph Convolutional Networks

Abstract: Being able to predict the crowd flows in each and every part of a city, especially in irregular regions, is strategically important for traffic control, risk assessment, and public safety. However, it is very challenging because of interactions and spatial correlations between different regions. In addition, it is affected by many factors: i) multiple temporal correlations among different time intervals: closeness, period, trend; ii) complex external influential factors: weather, events; iii) meta features: time of the day, day of the week, and so on. In this paper, we formulate crowd flow forecasting in irregular regions as a spatio-temporal graph (STG) prediction problem in which each node represents a region with time-varying flows. By extending graph convolution to handle the spatial information, we propose using spatial graph convolution to build a multi-view graph convolutional network (MVGCN) for the crowd flow forecasting problem, where different views can capture different factors as mentioned above. We evaluate MVGCN using four real-world datasets (taxicabs and bikes) and extensive experimental results show that our approach outperforms the adaptations of state-of-the-art methods. And we have developed a crowd flow forecasting system for irregular regions that can now be used internally.


Despite being similar to previous works in structure #REFR and utilizing little in terms of novel improvements, our system was able to achieve SOTA performance on FEVER and the highest label accuracy on SciFact.

We presented BEVERS, a strong baseline approach for the FEVER and SciFact datasets.

We primarily attribute these improvements to diligent hyperparameter tuning and error analysis.

While several previous works have shown novel contributions to portions of the pipeline can yield improvements, in this work we show a well-tuned baseline is very strong.

Bevers: A General, Simple, And Performant Framework For Automatic Fact Verification

Abstract: Automatic fact verification has become an increasingly popular topic in recent years and among datasets the Fact Extraction and VERification (FEVER) dataset is one of the most popular. In this work we present BEVERS, a tuned baseline system for the FEVER dataset. Our pipeline uses standard approaches for document retrieval, sentence selection, and final claim classification, however, we spend considerable effort ensuring optimal performance for each component. The results are that BEV-ERS achieves the highest FEVER score and label accuracy among all systems, published or unpublished. We also apply this pipeline to another fact verification dataset, Scifact, and achieve the highest label accuracy among all systems on that dataset as well. We also make our full code available 1 .

Bert For Evidence Retrieval And Claim Verification

Abstract: Motivated by the promising performance of pre-trained language models, we investigate BERT in an evidence retrieval and claim verification pipeline for the FEVER fact extraction and verification challenge. To this end, we propose to use two BERT models, one for retrieving potential evidence sentences supporting or rejecting claims, and another for verifying claims based on the predicted evidence sets. To train the BERT retrieval system, we use pointwise and pairwise loss functions, and examine the effect of hard negative mining. A second BERT model is trained to classify the samples as supported, refuted, and not enough information. Our system achieves a new state of the art recall of 87.1 for retrieving top five sentences out of the FEVER documents consisting of 50K Wikipedia pages, and scores second in the official leaderboard with the FEVER score of 69.7.


Following the origginal Transformer #REFR , each Transformer layer contains the multi-head attention with c = 512 dimensional states and 8 attention heads.

We use Dynamic Time Warping (DTW) #OTHEREFR , an elastic matching technique for aligning the given two sequences, to calculate the distance between the generated and real characters.

Moreover, we use Content Score #OTHEREFR to measure the structure correctness of generated characters, and adopt Style Score #OTHEREFR to quantify the style similarity between the generated and real handwritings.

We also conduct user preference studies to quantify the subjective quality of the output characters. More details are provided in Appendix A.1.1. Implementation details.

We set the number of the style reference to K = 15, and resize the reference style and content images to 64×64.

Moreover, each Transformer encoder contains 2 self-attention layers, while the Transformer decoder has 4 layers for obtaining style representations (2 for writer-wise and 2 for character-wise).

Contrary to HWT #OTHEREFR , where F is concatenated before the Transformer encoder, we process each feature sequence f ∈ F individually.

Moreover, we apply sinusoidal positional encoding #OTHEREFR to input tokens before feeding them to the Transformer encoder and decoder.

For training, we first pre-train the content encoder with 138k iterations (batch size of 256) for character classification on training samples and then train the whole model with 148k iterations (batch size of 128), on a single RTX3090 GPU.

The optimizer is Adam #OTHEREFR , with a learning rate of 0.0002 and gradient clipping of 5.0.

The sampling ratio α is determined through a search over 0.25, 0.5, 0.75, 1.00, and 0.25 is chosen. Following #OTHEREFR , we set λ = 2. Further implementation details are provided in Appendix A.1.2. Compared methods.

Disentangling Writer And Character Styles For Handwriting Generation

Abstract: Training machines to synthesize diverse handwritings is an intriguing task. Recently, RNN-based methods have been proposed to generate stylized online Chinese characters. However, these methods mainly focus on capturing a person's overall writing style, neglecting subtle style inconsistencies between characters written by the same person. For example, while a person's handwriting typically exhibits general uniformity (e.g., glyph slant and aspect ratios), there are still small style variations in finer details (e.g., stroke length and curvature) of characters. In light of this, we propose to disentangle the style representations at both writer and character levels from individual handwritings to synthesize realistic stylized online handwritten characters. Specifically, we present the styledisentangled Transformer (SDT), which employs two complementary contrastive objectives to extract the style commonalities of reference samples and capture the detailed style patterns of each sample, respectively. Extensive experiments on various language scripts demonstrate the effectiveness of SDT. Notably, our empirical findings reveal that the two learned style representations provide information at different frequency magnitudes, underscoring the importance of separate style extraction. Our source code is public at: https://github.com/dailenson/SDT.

Attention Is All You Need


We also evaluated the binary classification of KL-0 vs KL-2 on #REFR modifying the output units of their Siamese-based model from five to two.

As presented in Section 2.1, instead of using max pooling layers for down-sampling, we used stride and more convolutional layers to preserve more spatial information, which increased the nunber of parameters to a certain extant compared to the work of Tiulpin et al. #OTHEREFR .

Using the same dataset, the accuracy reached 86.2% with 0.15M parameters.

As shown in Table 4 , our proposed Siamese-GAP model achieved an accuracy of 88.38% with 2.71M parameters.

It is noteworthy that our model only needs a very small number of parameters compared to common evaluated CNN models (i.e., DenseNet, ResNet, and VGG).

Hence, compared to #OTHEREFR , the greater number of parameters in our proposed model appears to be acceptable, given the significant improvement in performance.

A Confident Labelling Strategy Based On Deep Learning For Improving Early Detection Of Knee Osteoarthritis

Abstract: Knee OsteoArthritis (KOA) is a prevalent musculoskeletal disorder that causes decreased mobility in seniors. The diagnosis provided by physicians is subjective, however, as it relies on personal experience and the semi-quantitative Kellgren-Lawrence (KL) scoring system. KOA has been successfully diagnosed by Computer-Aided Diagnostic (CAD) systems that use deep learning techniques like Convolutional Neural Networks (CNN). In this paper, we propose a novel Siamese-based network, and we introduce a new hybrid loss strategy for the early detection of KOA. The model extends the classical Siamese network by integrating a collection of Global Average Pooling (GAP) layers for feature extraction at each level. Then, to improve the classification performance, a novel training strategy that partitions each training batch into low-, medium-and high-confidence subsets, and a specific hybrid loss function are used for each new label attributed to each sample. The final loss function is then derived by combining the latter loss functions with optimized weights. Our test results demonstrate that our proposed approach significantly improves the detection performance.

Automatic Knee Osteoarthritis Diagnosis From Plain Radiographs: A Deep Learning-Based Approach

Abstract: Knee osteoarthritis (OA) is the most common musculoskeletal disorder. OA diagnosis is currently conducted by assessing symptoms and evaluating plain radiographs, but this process suffers from subjectivity. In this study, we present a new transparent computer-aided diagnosis method based on the Deep Siamese Convolutional Neural Network to automatically score knee OA severity according to the Kellgren-Lawrence grading scale. We trained our method using the data solely from the Multicenter Osteoarthritis Study and validated it on randomly selected 3,000 subjects (5,960 knees) from Osteoarthritis Initiative dataset. Our method yielded a quadratic Kappa coefficient of 0.83 and average multiclass accuracy of 66.71% compared to the annotations given by a committee of clinical experts. Here, we also report a radiological OA diagnosis area under the ROC curve of 0.93. Besides this, we present attention maps highlighting the radiological features affecting the network decision. Such information makes the decision process transparent for the practitioner, which builds better trust toward automatic methods. We believe that our model is useful for clinical decision making and for OA research; therefore, we openly release our training codes and the data set created in this study.


The first scene is from the Waymo Open Dataset (WOD) #REFR , and has a 20-second clip acquired by five pinhole cameras and one 64-beam LiDAR sensor at 10 Hz. We use all five camera views for our experiments.

In our evaluation, we follow the setting proposed in #OTHEREFR .

Specifically, we use three scenes for quantitative evaluation and use 13/12/11 sessions respectively to optimize the parameters of our neural scene representation.

We then use environment maps from five other recording sessions to relight each scene and measure average PSNR and MSE between the rendered and ground-truth images.

To remove dynamic objects, sky and vegetation pixels we again follow #OTHEREFR and use the segmentation masks predicted by an off-the-shelf semantic segmentation network #OTHEREFR .

Driving dataset includes two scenes captured by autonomous vehicles (AV) in an urban environment.

The second set of scenes is also from a high-quality AV dataset (dubbed RoadData) acquired in-house.

It is captured using eight high resolution (3848x2168 pixel) cameras with calibrated distortion and one 128-beam LiDAR.

We only use images from the front-facing 120 FoV camera.

For both scenes, we additionally rely on LiDAR point clouds for depth supervision.

The Driving dataset is challenging as it records large street environments with complex geometry, lighting, and occlusion, and typically with a fast camera motion.

Neural Fields Meet Explicit Geometric Representations For Inverse Rendering Of Urban Scenes

Abstract: Figure 1. We present FEGR, an approach for reconstructing scene geometry and recovering intrinsic properties of the scene from posed camera images. Our approach works both for single and multi-illumination captured data. FEGR enables various downstream applications such as VR and AR where users may want to control the lighting of the environment and insert desired 3D objects into the scene.

Scalability In Perception For Autonomous Driving: Waymo Open Dataset

Abstract: The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the overall viability of the technology. In an effort to help align the research community's contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data captured across a range of urban and suburban geographies. It is 15x more diverse than the largest camera+LiDAR dataset available based on our proposed diversity metric. We exhaustively annotated this data with 2D (camera image) and 3D (LiDAR) bounding boxes, with consistent identifiers across frames. Finally, we provide strong baselines for 2D as well as 3D detection and tracking tasks. We further study the effects of dataset size and generalization across geographies on 3D detection methods. Find data, code and more up-to-date information at http://www.waymo.com/open.


We set the perturbation bound in #REFR to = 4 for MNIST and = 1.5 for CIFAR10.

which is consistent with the perturbation computed from the fast gradient method (FGM) #OTHEREFR .

In our experiments, we use both the MNIST #OTHEREFR and CIFAR10 #OTHEREFR datasets, and randomly generate a graph with 20 nodes, shown in Fig. 1 .

We limit our simulations to binary classification in this example.

For this reason, we consider samples with digits 0 and 1 from MNIST, and images for airplanes and automobiles from CI-FAR10.

In the test phase, we compute the average classification error across the network to measure the performance of the multi-agent system against perturbations of different strengths.

We first illustrate the convergence of our algorithm, as anticipated by Theorem 1. From Fig.

2 , we observe a steady decrease in the classification error towards a limiting value.

The robust behavior of the proposed algorithm is illustrated in Fig. 3 for both MNIST and CIFAR10.

We explain the curves for MNIST and similar remarks hold for CIFAR10.

Multi-Agent Adversarial Training Using Diffusion Learning

Abstract: This work focuses on adversarial learning over graphs. We propose a general adversarial training framework for multi-agent systems using diffusion learning. We analyze the convergence properties of the proposed scheme for convex optimization problems, and illustrate its enhanced robustness to adversarial attacks.

Graphdefense: Towards Robust Graph Convolutional Networks

Abstract: In this paper, we study the robustness of graph convolutional networks (GCNs). Despite the good performance of GCNs on graph semi-supervised learning tasks, previous works have shown that the original GCNs are very unstable to adversarial perturbations. In particular, we can observe a severe performance degradation by slightly changing the graph adjacency matrix or the features of a few nodes, making it unsuitable for security-critical applications. Inspired by the previous works on adversarial defense for deep neural networks, and especially adversarial training algorithm, we propose a method called GraphDefense to defend against the adversarial perturbations. In addition, for our defense method, we could still maintain semi-supervised learning settings, without a large label rate. We also show that adversarial training in features is equivalent to adversarial training for edges with a small perturbation. Our experiments show that the proposed defense methods successfully increase the robustness of Graph Convolutional Networks. Furthermore, we show that with careful design, our proposed algorithm can scale to large graphs, such as Reddit dataset.


Similarly to #REFR , we find that training with verb phrase supervision helps for zero-shot performance on tasks with shorter sentences. Temporal understanding in videos.

#OTHEREFR learn a separate multi-modal embedding space for each PoS tag and then combine these embeddings for fine-grained action retrieval.

#OTHEREFR construct a hierarchical semantic graph and use graph reasoning for local-global alignments.

Most closely related to our work, #OTHEREFR use a PoS based token contrastive loss.

Our work differs in that: (i) we apply a verb phrase contrastive loss, as opposed to separate verb and noun losses; (ii) we extract verb phrases using a LLM and show this performs better than PoS tagging with NLTK #OTHEREFR (Tab.

5); (iii) we evaluate our methods on verb-focused downstream tasks.

A long term goal in computer vision is temporal understanding in videos #OTHEREFR .

However, current training and test datasets have a strong visual bias towards objects and backgrounds as well as static actions #OTHEREFR , with some works #OTHEREFR demonstrating strong results with a single frame.

Despite these challenges, many recent works in video-only self-supervised learning propose pretext tasks for improving temporal modelling #OTHEREFR .

Unlike these works that use only video, #OTHEREFR focus on fine-grained temporal videotext alignment via localization of text sub-tokens.

#OTHEREFR also leverage before/after relations in captions to create artifical training samples for video-text.

Verbs In Action: Improving Verb Understanding In Video-Language Models

Abstract: two brown horses eating grass LLM Original caption Video Add prompt two brown horses running on the grass two brown horses fighting on the grass two brown horses lying on the grass eating grass Verb phrase Hard verb negative captions two brown horses sleeping on the grass two brown horses playing on the grass eating grass cleaning camera standing squatting Verb phrase loss two brown horses eating grass two brown horses lying on the grass a woman standing in a post office person squatting at the gym a man cleaning his camera Hard negative loss Generated hard negative caption Negative captions in the batch Negative verb phrases in the batch Batch Figure 1. Verb-Focused Contrastive (VFC) learning: (Left): Given a video and its corresponding caption, we leverage a Large Language Model (LLM) to output (1) hard negative captions, where only the verb has been changed while keeping the remaining context, and (2) verb phrases which succinctly describe the action in the video. (Right): To encourage better verb reasoning, we subsequently enforce (1) a calibrated hard negative loss, using our generated hard negative captions and other captions in the batch, and (2) a fine-grained, verb phrase loss. We show that VFC improves verb understanding of video-language models compared to the standard contrastive loss.

Large-Scale Weakly-Supervised Pre-Training For Video Action Recognition

Abstract: Current fully-supervised video datasets consist of only a few hundred thousand videos and fewer than a thousand domain-specific labels. This hinders the progress towards advanced video architectures. This paper presents an in-depth study of using large volumes of web videos for pre-training video models for the task of action recognition. Our primary empirical finding is that pre-training at a very large scale (over 65 million videos), despite on noisy social-media videos and hashtags, substantially improves the state-of-the-art on three challenging public action recognition datasets. Further, we examine three questions in the construction of weakly-supervised video action datasets. First, given that actions involve interactions with objects, how should one construct a verb-object pre-training label space to benefit transfer learning the most? Second, frame-based models perform quite well on action recognition; is pre-training for good image features sufficient or is pre-training for spatio-temporal features valuable for optimal transfer learning? Finally, actions are generally less well-localized in long videos vs. short videos; since action labels are provided at a video level, how should one choose video clips for best performance, given some fixed budget of number or minutes of videos?


Federated learning #REFR takes the distributed data and trains the model locally for data security.

This type of federated learning is suitable for scenarios where the samples are different, but the features between the samples are similar.

• The applicable scenario of vertical federated learning is the opposite of horizontal federated learning, i.e., the union of features of cross-users under different businesses.

During the learning process, each participant is unaware of the existence of other parties and only uploads their model parameters, but the prediction requires joint modeling and collaboration to complete the task.

• Federated transfer learning can be considered useful when there is no significant overlap between either the samples or their features.

The key problem is to identify similarities between the source and target domains.

With a trusted channel established by secure algorithms such as homomorphic encryption, the results of the model can be aggregated to a central node for training to obtain the final training model.

Federated learning ensures both parity and independence among the participants.

Moreover, the entire process of learning and training achieves data isolation to prevent data leakage, which well protects the privacy of users #OTHEREFR .

In such a distributed machine learning framework, the various participants can be coordinated to build more accurate global models. Until now, there are many studies about federated learning.

Federated Learning For Metaverse: A Survey

Abstract: The metaverse, which is at the stage of innovation and exploration, faces the dilemma of data collection and the problem of private data leakage in the process of development. This can seriously hinder the widespread deployment of the metaverse. Fortunately, federated learning (FL) is a solution to the above problems. FL is a distributed machine learning paradigm with privacy-preserving features designed for a large number of edge devices. Federated learning for metaverse (FL4M) will be a powerful tool. Because FL allows edge devices to participate in training tasks locally using their own data, computational power, and model-building capabilities. Applying FL to the metaverse not only protects the data privacy of participants but also reduces the need for high computing power and high memory on servers. Until now, there have been many studies about FL and the metaverse, respectively. In this paper, we review some of the early advances of FL4M, which will be a research direction with unlimited development potential. We first introduce the concepts of metaverse and FL, respectively. Besides, we discuss the convergence of key metaverse technologies and FL in detail, such as big data, communication technology, the Internet of Things, edge computing, blockchain, and extended reality. Finally, we discuss some key challenges and promising directions of FL4M in detail. In summary, we hope that our up-to-date brief survey can help people better understand FL4M and build a fair, open, and secure metaverse. CCS CONCEPTS • Computing methodologies → Federated learning.

Federated Learning: Challenges, Methods, And Future Directions

Abstract: Federated learning involves training statistical models over remote devices or siloed data centers, such as mobile phones or hospitals, while keeping data localized. Training in heterogeneous and potentially massive networks introduces novel challenges that require a fundamental departure from standard approaches for large-scale machine learning, distributed optimization, and privacy-preserving data analysis. In this article, we discuss the unique characteristics and challenges of federated learning, provide a broad overview of current approaches, and outline several directions of future work that are relevant to a wide range of research communities.


It is folklore that the peeling exponential mechanism is equivalent to taking the top-after adding Gumbel noise, as formally shown in Lemma 4.2 of #REFR .

We show a similar property here for AboveThreshold with Gumbel noise that it is equivalent to iteratively running the exponential mechanism.

Unbounded Di Erentially Private Quantile And Maximum Estimation

Abstract: In this work we consider the problem of di erentially private computation of quantiles for the data, especially the highest quantiles such as maximum, but with an unbounded range for the dataset. We show that this can be done e ciently through a simple invocation of AboveThreshold, a subroutine that is iteratively called in the fundamental Sparse Vector Technique, even when there is no upper bound on the data. In particular, we show that this procedure can give more accurate and robust estimates on the highest quantiles with applications towards clipping that is essential for di erentially private sum and mean estimation. In addition, we show how two invocations can handle the fully unbounded data setting. Within our study, we show that an improved analysis of AboveThreshold can improve the privacy guarantees for the widely used Sparse Vector Technique that is of independent interest. We give a more general characterization of privacy loss for AboveThreshold which we immediately apply to our method for improved privacy guarantees. Our algorithm only requires one () pass through the data, which can be unsorted, and each subsequent query takes (1) time. We empirically compare our unbounded algorithm with the state-of-the-art algorithms in the bounded setting. For inner quantiles, we nd that our method often performs better on non-synthetic datasets. For the maximal quantiles, which we apply to di erentially private sum computation, we nd that our method performs signi cantly better.

Practical Differentially Private Top-$K$ Selection With Pay-What-You-Get Composition

Abstract: We study the problem of top-k selection over a large domain universe subject to user-level differential privacy. Typically, the exponential mechanism or report noisy max are the algorithms used to solve this problem. However, these algorithms require querying the database for the count of each domain element. We focus on the setting where the data domain is unknown, which is different than the setting of frequent itemsets where an apriori type algorithm can help prune the space of domain elements to query. We design algorithms that ensures (approximate) differential privacy and only needs access to the true top-k' elements from the data for any chosen k' ≥ k. This is a highly desirable feature for making differential privacy practical, since the algorithms require no knowledge of the domain. We consider both the setting where a user's data can modify an arbitrary number of counts by at most 1, i.e. unrestricted sensitivity, and the setting where a user's data can modify at most some small, fixed number of counts by at most 1, i.e. restricted sensitivity. Additionally, we provide a pay-what-you-get privacy composition bound for our algorithms. That is, our algorithms might return fewer than k elements when the top-k elements are queried, but the overall privacy budget only decreases by the size of the outcome set.


We began by constructing a BPE tokenizer #REFR ) with a vocabulary size of 50,265 tokens using the HeDC4 dataset.

The model was trained in two stages: in the first stage the model was trained on four NVIDIA GeForce GTX 1080 Ti 12GB GPUs for 35 days. The checkpoint was then saved for LongHeRo initialization.

The second stage of training was identical to the first.

Each stage started with the learning rate of 1e-4 with linear decay and the AdamW optimizer (Loshchilov and Hutter, 2017) .

Hero: Roberta And Longformer Hebrew Language Models

Abstract: In this paper, we fill in an existing gap in resources available to the Hebrew NLP community by providing it with the largest so far pre-train dataset HeDC4, a state-ofthe-art pre-trained language model HeRo for standard length inputs and an efficient transformer LongHeRo for long input sequences. The HeRo model was evaluated on the sentiment analysis, the named entity recognition, and the question answering tasks while the LongHeRo model was evaluated on the document classification task with a dataset composed of long documents. Both HeRo and LongHeRo presented state-of-the-art performance. The dataset and model checkpoints used in this work are publicly available 1 .

Neural Machine Translation Of Rare Words With Subword Units

Abstract: Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character ngram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English!German and English!Russian by up to 1.1 and 1.3 BLEU, respectively.


Inspired by the mixup #REFR approach for 2D data, we propose to mix the feature embeddings obtained by F , but from multiple domains in the latent space.

Unlike the methods for 2D data where the input images are blended by an alpha factor #OTHEREFR , we propose mixing the feature embeddings, since the feature embeddings from the deeper lay-ers of the network contains information about the global shape of the point cloud and local point-point interaction, as demonstrated in #OTHEREFR applied to STDA set-up.

Specifically, we linearly interpolate the source (F s ) and target feature (F Ti ) embeddings to obtain F m i and the corresponding mixed soft domain labels L m i as:

where L s and L Ti denote the domain labels of source and target domain which are set to 1 and 0, respectively.

Mensa: Mix-Up Ensemble Average For Unsupervised Multi Target Domain Adaptation On 3D Point Clouds

Abstract: Unsupervised domain adaptation (UDA) addresses the problem of distribution shift between the unlabeled target domain and labelled source domain. While the single target domain adaptation (STDA) is well studied in both 2D and 3D vision literature, multi-target domain adaptation (MTDA) is barely explored for 3D data despite its wide realworld applications such as autonomous driving systems for various geographical and climatic conditions. We establish an MTDA baseline for 3D point cloud data by proposing to mix the feature representations from all domains together to achieve better domain adaptation performance by an ensemble average, which we call Mixup Ensemble Average or MEnsA. With the mixed representation, we use a domain classifier to improve at distinguishing the feature representations of source domain from those of target domains in a shared latent space. In extensive empirical validations on the challenging PointDA-10 dataset, we showcase a clear benefit of our simple method over previous unsupervised STDA and MTDA methods by large margins (up to 17.10% and 4.76% on averaged over all domain shifts). We make the code publicly available here.

Mixup: Beyond Empirical Risk Minimization

Abstract: Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.


The AutoML system TPOT uses multi-objective optimization to balance pipeline size and performance #REFR .

In this section we discuss multiobjective and constrained optimization, then describe how it is used to handle fairness as an additional objective, and lastly, highlight other findings that relate to fairness and AutoML.

Multi-objective and constrained optimization HPO methods, AutoML systems, and NAS methods have been extended to the multi-objective optimization case to take into account additional objectives.

A related solution is given by constrained optimization, in which one or more objectives are optimized to meet a certain constraint value #OTHEREFR .

HPO has been used to optimize ML models to take multiple cost functions #OTHEREFR or resource consumption #OTHEREFR into account.

The goal of multi-objective optimization is to approximate the Pareto set representing the best possible combination of outcomes -solutions that can not be improved w.r.t. one objective without sacrificing performance w.r.t. another.

AutoXGBoostMC proposes tuning several additional metrics, such as fairness, robustness, or interpretability metrics #OTHEREFR .

The NAS technique LEMONADE #OTHEREFR searches for neural architectures that balance performance, latency and size. We refer to Morales-Hernández et al. 2022and #OTHEREFR

(2022) for reviews of multi-objective HPO and to Benmeziane et al.

(2021) for a review on hardware-aware NAS, a subfield of multi-objective NAS.

Fairness as a constraint or additional objective Many fairness-aware ML algorithms consider a constrained optimization approach.

Can Fairness Be Automated? Guidelines And Opportunities For Fairness-Aware Automl

Abstract: The problem is scoped and translated into an ML task including real-world success criteria, requirements, and constraints. 2. Data understanding and preparation. 3 It is determined which data is required, followed by an iterative sequence of data collection, exploration, cleaning, and aggregation.

Evaluation Of A Tree-Based Pipeline Optimization Tool For Automating Data Science

Abstract: As the field of data science continues to grow, there will be an ever-increasing demand for tools that make machine learning accessible to non-experts. In this paper, we introduce the concept of tree-based pipeline optimization for automating one of the most tedious parts of machine learning--pipeline design. We implement an open source Tree-based Pipeline Optimization Tool (TPOT) in Python and demonstrate its effectiveness on a series of simulated and real-world benchmark data sets. In particular, we show that TPOT can design machine learning pipelines that provide a significant improvement over a basic machine learning analysis while requiring little to no input nor prior knowledge from the user. We also address the tendency for TPOT to design overly complex pipelines by integrating Pareto optimization, which produces compact pipelines without sacrificing classification accuracy. As such, this work represents an important step toward fully automating machine learning pipeline design.


Inspired by the success of current defensive approaches #REFR , we conclude the properties of operations on graph structure for robustness and design representative defensive operations in our search space accordingly.

M (l) = A (l−1) LRA A (l−1) = U (l−1) S (l−1) (V (l−1) ) T , M (l) = U (l−1) r S (l−1) r (V (l−1) r ) T NFS m (l) ij = 0, if a (l−1) ij > 0 and J ij < τ a (l−1) ij , otherwise NIE m (l) ij = βm (l−1) ij + (1 − β)α (l) ij VPO M (l) = V v=1 θ v (A (l−1) ) v

j where m ij = 0 indicates a complete edge pruning and m ij = 1 means no modification to the original edge.

The defensive operation aims to assign fewer weights to potential perturbed edges.

In this way, we can choose the most appropriate defensive strategies when confronting perturbed graphs.

To our best knowledge, this is the first time the search space to be designed with a specific purpose to enhance the robustness of GNNs.

Specifically, we include five graph structure mask operations in the search space.

Identity keeps the same graph structure as the previous layer.

Low Rank Approximation (LRA) reconstructs the adjacency matrix in the l-th layer from the top-r components of singular value decomposition from adjacency matrix in the previous layer.

Adversarially Robust Neural Architecture Search For Graph Neural Networks

Abstract: Graph Neural Networks (GNNs) obtain tremendous success in modeling relational data. Still, they are prone to adversarial attacks, which are massive threats to applying GNNs to risk-sensitive domains. Existing defensive methods neither guarantee performance facing new data/tasks or adversarial attacks nor provide insights to understand GNN robustness from an architectural perspective. Neural Architecture Search (NAS) has the potential to solve this problem by automating GNN architecture designs. Nevertheless, current graph NAS approaches lack robust design and are vulnerable to adversarial attacks. To tackle these challenges, we propose a novel Robust Neural Architecture search framework for GNNs (G-RNA). Specifically, we design a robust search space for the message-passing mechanism by adding graph structure mask operations into the search space, which comprises various defensive operation candidates and allows us to search for defensive GNNs. Furthermore, we define a robustness metric to guide the search procedure, which helps to filter robust architectures. In this way, G-RNA helps understand GNN robustness from an architectural perspective and effectively searches for optimal adversarial robust GNNs. Extensive experimental results on benchmark datasets show that G-RNA significantly outperforms manually designed robust GNNs and vanilla graph NAS baselines by 12.1% to 23.4% under adversarial attacks.

Adversarial Attack And Defense On Graph Data: A Survey

Abstract: Deep neural networks (DNNs) have been widely applied to various applications including image classification, text generation, audio recognition, and graph data analysis. However, recent studies have shown that DNNs are vulnerable to adversarial attacks. Though there are several works studying adversarial attack and defense strategies on domains such as images and natural language processing, it is still difficult to directly transfer the learned knowledge to graph structure data due to its representation challenges. Given the importance of graph analysis, an increasing number of works start to analyze the robustness of machine learning models on graph data. Nevertheless, current studies considering adversarial behaviors on graph data usually focus on specific types of attacks with certain assumptions. In addition, each work proposes its own mathematical formulation which makes the comparison among different methods difficult. Therefore, in this paper, we aim to survey existing adversarial learning strategies on graph data and first provide a unified formulation for adversarial learning on graph data which covers most adversarial learning studies on graph. Moreover, we also compare different attacks and defenses on graph data and discuss their corresponding contributions and limitations. In this work, we systemically organize the considered works based on the features of each topic. This survey not only serves as a reference for the research community, but also brings a clear image researchers outside this research domain. Besides, we also create an online resource and keep updating the relevant papers during the last two years. More details of the comparisons of various studies based on this survey are open-sourced at https://github.com/YingtongDou/graph-adversarial-learning-literature.