One aspect in which they can prove useful is in the training of algorithm selection models #REFR , as they can significantly increase the size and variety of training data, which is an important consideration towards testing generalizability.

This is achieved through several transformations, the most common of which is moving the optimum to a different location in the domain.

The results we present show that the way in which these optimal locations are chosen can have a large impact on the performance of optimization algorithms.

Since the optima are not distributed uniformly in the domain, some functions have different kinds of bias, which can be exploited by an algorithm.

The question on how to fairly consider different instance generation mechanisms when making use of function combination is thus highly interlinked with questions about how well performance observed on a set of BBOB instances generalizes.

Even with these challenges in mind, there are many potential use cases for these affine function combinations.

One final aspect in which the benchmark data on these function combinations can be further utilized is by linking it back to the exploratory landscape analysis which inspired their creation.

Since the combinations can smoothly fill the landscape feature space, this can be combined with algorithm performance to get a more fine-grained view of the way in which the landscape interacts with different algorithms #OTHEREFR .

Using Affine Combinations Of Bbob Problems For Performance Assessment

Abstract: Benchmarking plays a major role in the development and analysis of optimization algorithms. As such, the way in which the used benchmark problems are defined significantly affects the insights that can be gained from any given benchmark study. One way to easily extend the range of available benchmark functions is through affine combinations between pairs of functions. From the perspective of landscape analysis, these function combinations smoothly transition between the two base functions. In this work, we show how these affine function combinations can be used to analyze the behavior of optimization algorithms. In particular, we highlight that by varying the weighting between the combined problems, we can gain insights into the effects of added global structure on the performance of optimization algorithms. By analyzing performance trajectories on more function combinations, we also show that aspects such as the scaling of objective functions and placement of the optimum can greatly impact how these results are interpreted.

Automated Algorithm Selection: Survey And Perspectives

Abstract: It has long been observed that for practically any computational problem that has been intensely studied, different instances are best solved using different algorithms. This is particularly pronounced for computationally hard problems, where in most cases, no single algorithm defines the state of the art; instead, there is a set of algorithms with complementary strengths. This performance complementarity can be exploited in various ways, one of which is based on the idea of selecting, from a set of given algorithms, for each problem instance to be solved the one expected to perform best. The task of automatically selecting an algorithm from a given set is known as the per-instance algorithm selection problem and has been intensely studied over the past 15 years, leading to major improvements in the state of the art in solving a growing number of discrete combinatorial problems, including propositional satisfiability and AI planning. Per-instance algorithm selection also shows much promise for boosting performance in solving continuous and mixed discrete/continuous optimisation problems. This survey provides an overview of research in automated algorithm selection, ranging from early and seminal works to recent and promising application areas. Different from earlier work, it covers applications to discrete and continuous problems, and discusses algorithm selection in context with conceptually related approaches, such as algorithm configuration, scheduling or portfolio selection. Since informative and cheaply computable problem instance features provide the basis for effective per-instance algorithm selection systems, we also provide an overview of such features for discrete and continuous problems. Finally, we provide perspectives on future work in the area and discuss a number of open research challenges.


And we employ focal loss #REFR to increase weighting of detections that are especially hard to classify correctly and to better address the severe class imbalance.

We choose to use the position in the Cartesian vehicle coordinate system (CS) at the time of recording the most recent scan in the point cloud. The timestamp is relative to that moment.

Moreover, we add the detection's position in the polar sensor CS, which is more useful in some cases, and the measuring sensor's ID.

When upsampling of the input point cloud is required, that is done by duplicating randomly selected detections. Replicas Finally, we also adjust the training setup. Input data is normalized via standardization.

To prevent a distortion of following distance calculations, the variances of the Cartesian coordinates are averaged before scaling.

The learning rate is varied between 10 −9 and 10 −3 according to a cyclical learning rate policy.

The remaining training parameters are as described in #OTHEREFR .

Tackling Clutter In Radar Data -Label Generation And Detection Using Pointnet++

Abstract: Radar sensors employed for environment perception, e.g. in autonomous vehicles, output a lot of unwanted clutter. These points, for which no corresponding real objects exist, are a major source of errors in following processing steps like object detection or tracking. We therefore present two novel neural network setups for identifying clutter. The input data, network architectures and training configuration are adjusted specifically for this task. Special attention is paid to the downsampling of point clouds composed of multiple sensor scans. In an extensive evaluation, the new setups display substantially better performance than existing approaches. Because there is no suitable public data set in which clutter is annotated, we design a method to automatically generate the respective labels. By applying it to existing data with object annotations and releasing its code, we effectively create the first freely available radar clutter data set representing realworld driving scenarios. Code and instructions are accessible at www.github.com/kopp-j/clutter-ds.

Focal Loss For Dense Object Detection

Abstract: The highest accuracy object detectors to date are based on a two-stage approach popularized by R-CNN, where a classifier is applied to a sparse set of candidate object locations. In contrast, one-stage detectors that are applied over a regular, dense sampling of possible object locations have the potential to be faster and simpler, but have trailed the accuracy of two-stage detectors thus far. In this paper, we investigate why this is the case. We discover that the extreme foreground-background class imbalance encountered during training of dense detectors is the central cause. We propose to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples. Our novel Focal Loss focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training. To evaluate the effectiveness of our loss, we design and train a simple dense detector we call RetinaNet. Our results show that when trained with the focal loss, RetinaNet is able to match the speed of previous one-stage detectors while surpassing the accuracy of all existing state-of-the-art two-stage detectors. Code is at: https://github.com/facebookresearch/Detectron .


For example, SMAC #REFR is based on the popular real-time strategy game StarCraft II, focuses on micromanagement challenges, and is applicable to studying cooperative MARL.

Multifarious emerging benchmarks have accelerated MARL research in recent years and provide various evaluation criteria for different application scenarios and research domains.

Some game platforms have been the most popular benchmarks for evaluating MARL algorithms.

GRF #OTHEREFR , an environment for playing football tasks of varying difficulty in a physics-based 3D simulation, focuses on multi-level, multi-agent learning.

Wimblepong #OTHEREFR is a 2-player version of the Atari game Pong, where each player controls a paddle to play a ball with the other, and it is a purely competitive scenario.

Some environments evolve from single-agent tasks, which decompose single-agent control tasks into multi-agent tasks.

For instance, MaMujoco #OTHEREFR , based on a single-agent robotic MuJoCo control suite, provides a wide variety of continuous multi-agent robotic control scenarios in which multiple agents within a single robot try to complete a task cooperatively.

DM Control #OTHEREFR is a set of Python RL environments powered by the MuJoCo physics engine and includes multi-agent soccer simulation environments.

Neuronsmae: A Novel Multi-Agent Reinforcement Learning Environment For Cooperative And Competitive Multi-Robot Tasks

Abstract: Multi-agent reinforcement learning (MARL) has achieved remarkable success in various challenging problems. Meanwhile, more and more benchmarks have emerged and provided some standards to evaluate the algorithms in different fields. On the one hand, the virtual MARL environments lack knowledge of real-world tasks and actuator abilities, and on the other hand, the current task-specified multi-robot platform has poor support for the generality of multi-agent reinforcement learning algorithms and lacks support for transferring from simulation to the real environment. Bridging the gap between the virtual MARL environments and the real multi-robot platform becomes the key to promoting the practicability of MARL algorithms. This paper proposes a novel MARL environment for real multi-robot tasks named NeuronsMAE (Neurons Multi-Agent Environment). This environment supports cooperative and competitive multi-robot tasks and is configured with rich parameter interfaces to study the multi-agent policy transfer from simulation to reality. With this platform, we evaluate various popular MARL algorithms and build a new MARL benchmark for multi-robot tasks. We hope that this platform will facilitate the research and application of MARL algorithms for real robot tasks. Information about the benchmark and the open-source code will be released. Index Terms-multi-agent reinforcement learning, benchamark, multi-robot.

The Starcraft Multi-Agent Challenge

Abstract: In the last few years, deep multi-agent reinforcement learning (RL) has become a highly active area of research. A particularly challenging class of problems in this area is partially observable, cooperative, multi-agent learning, in which teams of agents must learn to coordinate their behaviour while conditioning only on their private observations. This is an attractive research area since such problems are relevant to a large number of real-world systems and are also more amenable to evaluation than general-sum problems. Standardised environments such as the ALE and MuJoCo have allowed single-agent RL to move beyond toy domains, such as grid worlds. However, there is no comparable benchmark for cooperative multi-agent RL. As a result, most papers in this field use one-off toy problems, making it difficult to measure real progress. In this paper, we propose the StarCraft Multi-Agent Challenge (SMAC) as a benchmark problem to fill this gap. SMAC is based on the popular real-time strategy game StarCraft II and focuses on micromanagement challenges where each unit is controlled by an independent agent that must act based on local observations. We offer a diverse set of challenge maps and recommendations for best practices in benchmarking and evaluations. We also open-source a deep multi-agent RL learning framework including state-of-the-art algorithms. We believe that SMAC can provide a standard benchmark environment for years to come. Videos of our best agents for several SMAC scenarios are available at: https://youtu.be/VZ7zmQ_obZ0.


Following #REFR , we rasterize neural points V to multi-scale feature maps S(i, V ), i ∈ [0, K), K = 3.

We use a differentiable point rasterizer implemented by PyTorch3D #OTHEREFR to assign every pixel a neural descriptor and a binary scalar that indicates a non-empty pixel.

We consider the binary mask as a point-based occupancy mask. Background regularization.

To handle pixels without corresponding point cloud projection, we assign a learnable descriptor as the background.

During texture enhancement optimization, we additionally add a regularization to encourage the scene to be rendered with a white background according to the binary occupancy mask mentioned above. Deferred neural rendering.

For deferred rendering of the point clouds, we use a 2D U-Net architecture with gated convolutions #OTHEREFR .

Make-It-3D: High-Fidelity 3D Creation From A Single Image With Diffusion Prior

Abstract: Reference Normal Novel Views Reference Normal Novel Views Figure 1: Make-It-3D can create high-fidelity 3D content from only a single image. We show the normal map and novel-view renderings of created 3D content, showcasing fine geometry and faithful textures with stunning quality at novel views.

Neural Point-Based Graphics

Abstract: We present a new point-based approach for modeling the appearance of real scenes. The approach uses a raw point cloud as the geometric representation of a scene, and augments each point with a learnable neural descriptor that encodes local geometry and appearance. A deep rendering network is learned in parallel with the descriptors, so that new views of the scene can be obtained by passing the rasterizations of a point cloud from new viewpoints through this network. The input rasterizations use the learned descriptors as point pseudo-colors. We show that the proposed approach can be used for modeling complex scenes and obtaining their photorealistic views, while avoiding explicit surface estimation and meshing. In particular, compelling results are obtained for scene scanned using hand-held commodity RGB-D sensors as well as standard RGB cameras even in the presence of objects that are challenging for standard mesh-based modeling.


Then, we adopt a GNN model, namely graph isomorphism network (GIN) #REFR , to extract the structural information of cell graphs.

After constructing the cell graphs, we generate the embeddings of cell nodes and input images for the report generation task.

For embedding generation of cell nodes, we first use a trainable CNN to process the extracted cell nodes in Section 3.1 and obtain the output

|V| ] , which captures morphology features of cell nodes.

In this way, the network can learn to extract essential morphology features for report generation during training.

Specifically, one layer of the GNN model is defined as follows:

where h ( ) i denotes the hidden embedding of cell node i in a cell graph at the th layer.

( ) is a learnable parameter to distinguish central nodes from neighbors.

N i denotes the set of neighbors of cell node i. MLP denotes a multi-layer perceptron. H (0) is the input of the GNN model.

Gnnformer: A Graph-Based Framework For Cytopathology Report Generation Technical Report

Abstract: Cytopathology report generation is a necessary step for the standardized examination of pathology images. However, manually writing detailed reports brings heavy workloads for pathologists. To improve efficiency, some existing works have studied automatic generation of cytopathology reports, mainly by applying image caption generation frameworks with visual encoders originally proposed for natural images. A common weakness of these works is that they do not explicitly model the structural information among cells, which is a key feature of pathology images and provides significant information for making diagnoses. In this paper, we propose a novel graph-based framework called GNNFormer, which seamlessly integrates graph neural network (GNN) and Transformer into the same framework, for cytopathology report generation. To the best of our knowledge, GNNFormer is the first report generation method that explicitly models the structural information among cells in pathology images. It also effectively fuses structural information among cells, fine-grained morphology features of cells and background features to generate high-quality reports. Experimental results on the NMI-WSI dataset show that GNNFormer can outperform other state-of-the-art baselines.

How Powerful Are Graph Neural Networks?

Abstract: Graph Neural Networks (GNNs) are an effective framework for representation learning of graphs. GNNs follow a neighborhood aggregation scheme, where the representation vector of a node is computed by recursively aggregating and transforming representation vectors of its neighboring nodes. Many GNN variants have been proposed and have achieved state-of-the-art results on both node and graph classification tasks. However, despite GNNs revolutionizing graph representation learning, there is limited understanding of their representational properties and limitations. Here, we present a theoretical framework for analyzing the expressive power of GNNs to capture different graph structures. Our results characterize the discriminative power of popular GNN variants, such as Graph Convolutional Networks and GraphSAGE, and show that they cannot learn to distinguish certain simple graph structures. We then develop a simple architecture that is provably the most expressive among the class of GNNs and is as powerful as the Weisfeiler-Lehman graph isomorphism test. We empirically validate our theoretical findings on a number of graph classification benchmarks, and demonstrate that our model achieves state-of-the-art performance.


Additionally, we incorporate the entropy loss term H(p k θ ), following the SAC's settings, to encourage exploration in the learning process #REFR .

The loss function is computed over a horizon of K unrolled steps.

The reward loss l r measures the difference between the estimated reward r k and target reward u k .

Similarly, The value loss l v indicates the difference between the estimated value v k and bootstrapped target value z k .

The policy loss l p represents the distance between the output policy p k and the root visit distribution of MCTS π k .

In order to address the challenge of insufficient supervisory signal in dynamic networks in MuZero, we introduce consistency loss l c which maximizes the similarity between the predicted next-stateŝ k+1 and the ground-truth next-state s k+1 .

Real-Time Scheduling Of Renewable Power Systems Through Planning-Based Reinforcement Learning

Abstract: The growing renewable energy sources have posed significant challenges to traditional power scheduling. It is difficult for operators to obtain accurate day-ahead forecasts of renewable generation, thereby requiring the future scheduling system to make real-time scheduling decisions aligning with ultra-short-term forecasts. Restricted by the computation speed, traditional optimization-based methods can not solve this problem. Recent developments in reinforcement learning (RL) have demonstrated the potential to solve this challenge. However, the existing RL methods are inadequate in terms of constraint complexity, algorithm performance, and environment fidelity. We are the first to propose a systematic solution 1

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning With A Stochastic Actor

Abstract: Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy - that is, succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.


Specifically, the authors in #REFR consider a two-cell network and propose a centralized DRL-based approach, where only a single agent is employed to control the beamformers and transmit power of both the two cells.

In #OTHEREFR - #OTHEREFR , the DL-based approaches exploiting expert knowledge, i.e., the known structure of optimal solutions, are studied for the beamforming optimization in a single-cell MU-MISO downlink system.

Particularly, the exploitation of expert knowledge can improve the performance of the DL-based approaches #OTHEREFR .

In #OTHEREFR , a bipartite graph neural network based approach is further developed to realize a scalable DL-based solution.

In #OTHEREFR , a deep unrolling approach is proposed to reduce the number of required iterations by unfolding the iterative optimization procedures as graph neural networks, however, it still requires the real-time global CSI and some iterations to obtain near-optimal beamformers.

In #OTHEREFR , #OTHEREFR , the joint beamforming, power control and interference coordination problem in cellular networks is investigated.

For the more general multi-cell networks, a distributed beamforming coordination approach based on multi-agent DRL is proposed in #OTHEREFR .

It is worth noting that the DRL-based schemes in #OTHEREFR , #OTHEREFR are designed for cellular networks with only a single user per cell, and thus can not be directly extended to more general multi-cell multi-user cellular networks.

Moreover, the codebook-based method is adopted in #OTHEREFR , #OTHEREFR , where the optimal beamformers can only be selected from a predefined set of available beamformers.

As such, when the channel characteristics are more complex, e.g., the channels in a lower frequency band instead of the millimeter wave band considered in #OTHEREFR , the optimal beamformers may not fall into the predefined set, and it is almost impossible to obtain the optimal beamformers with the codebook-based method due to the mismatch between the channel characteristics and the codebook.

Inspired by the huge potential of ML-based beamforming optimization approaches, we propose a DRL-based distributed dynamic coordinated beamforming (DDCBF) framework for a massive MIMO mobile cellular network.

Deep Reinforcement Learning For Distributed Dynamic Coordinated Beamforming In Massive Mimo Cellular Networks

Abstract: To accommodate the explosive wireless traffics, massive multiple-input multiple-output (MIMO) is regarded as one of the key enabling technologies for next-generation communication systems. In massive MIMO cellular networks, coordinated beamforming (CBF), which jointly designs the beamformers of multiple base stations (BSs), is an efficient method to enhance the network performance. In this paper, we investigate the sum rate maximization problem in a massive MIMO mobile cellular network, where in each cell a multi-antenna BS serves multiple mobile users simultaneously via downlink beamforming. Although existing optimization-based CBF algorithms can provide near-optimal solutions, they require realtime and global channel state information (CSI), in addition to their high computation complexity. It is almost impossible to apply them in practical wireless networks, especially highly dynamic mobile cellular networks. Motivated by this, we propose a deep reinforcement learning based distributed dynamic coordinated beamforming (DDCBF) framework, which enables each BS to determine the beamformers with only local CSI and some historical information from other BSs.Besides, the beamformers can be calculated with a considerably lower computational complexity by exploiting neural networks and expert knowledge, i.e., a solution structure observed from the iterative procedure of the weighted minimum mean square error (WMMSE) algorithm. Moreover, we provide extensive numerical simulations to validate the effectiveness of the proposed DRL-based approach. With lower computational complexity

Deep Reinforcement Learning For 5G Networks: Joint Beamforming, Power Control, And Interference Coordination

Abstract: The fifth generation of wireless communications (5G) promises massive increases in traffic volume and data rates, as well as improved reliability in voice calls. Jointly optimizing beamforming, power control, and interference coordination in a 5G wireless network to enhance the communication performance to end users poses a significant challenge. In this paper, we formulate the joint design of beamforming, power control, and interference coordination as a non-convex optimization problem to maximize the signal to interference plus noise ratio (SINR) and solve this problem using deep reinforcement learning. By using the greedy nature of deep Q-learning to estimate future rewards of actions and using the reported coordinates of the users served by the network, we propose an algorithm for voice bearers and data bearers in sub-6 GHz and millimeter wave (mmWave) frequency bands, respectively. The algorithm improves the performance measured by SINR and sum-rate capacity. In realistic cellular environments, the simulation results show that our algorithm outperforms the link adaptation industry standards for sub-6 GHz voice bearers. For data bearers in the mmWave frequency band, our algorithm approaches the maximum sum rate capacity, but with less than 4% of the required run time.


In addition, annotating the image for each condition is very laborious when the individual images contain many cells. 2015; #REFR .

proposed Mask R-CNN, which enables instance segmentation by adding mask branches to the head of the Faster R-CNN architecture #OTHEREFR

(2015) and thereby allowing segmentation of each detected object #OTHEREFR . Fujita et al.

utilized Mask R-CNN for cell detection and segmentation #OTHEREFR . To reduce annotation costs, Nishimura et al. proposed U-Net Ronneberger et al.

(2015) , which uses a heatmap as training data #OTHEREFR .

However, these methods would not work well at cell detection if the domains of the training data (source domain) and test data (target domain) are different (domain shift).

This method uses a domain discriminator that distinguishes the source and target domain and, for adaptation, it tries to fool the discriminator into not distinguishing between domains. For the cell segmentation task, Haq et al.

utilized adversarial learning for domain adaption and introduced an autoencoder to extract invariant features between the source and target domains #OTHEREFR .

However, this approach does not consider essential features e.g.,(class features in classification task) and only tries to match features between domains. To solve this problem, Saito et al.

proposed to use the maximum classifier discrepancy that matches features among classes between domains #OTHEREFR .

The design of an adversarial domain adaptation is more complex for the heatmap prediction task since the network has both an encoder and decoder.

Medical Image Analysis

Abstract: Cell detection is an important task in biomedical research. Recently, deep learning methods have made it possible to improve the performance of cell detection. However, a detection network trained with training data under a specific condition (source domain) may not work well on data under other conditions (target domains), which is called the domain shift problem. In particular, cells are cultured under different conditions depending on the purpose of the research. Characteristics, e.g., the shapes and density of the cells, change depending on the conditions, and such changes may cause domain shift problems. Here, we propose an unsupervised domain adaptation method for cell detection using a pseudo-cell-position heatmap, where the cell centroid is at the peak of a Gaussian distribution in the map and selective pseudo-labeling. In the prediction result for the target domain, even if the peak location is correct, the signal distribution around the peak often has a non-Gaussian shape. The pseudo-cell-position heatmap is thus regenerated using the peak positions in the predicted heatmap to have a clear Gaussian shape. Our method selects confident pseudo-cell-position heatmaps based on uncertainty and curriculum learning. We conducted numerous experiments showing that, compared with the existing methods, our method improved detection performance under different conditions.

Abstract: Adversarial learning methods are a promising approach to training robust deep networks, and can generate complex samples across diverse domains. They can also improve recognition despite the presence of domain shift or dataset bias: recent adversarial approaches to unsupervised domain adaptation reduce the difference between the training and test domain distributions and thus improve generalization performance. However, while generative adversarial networks (GANs) show compelling visualizations, they are not optimal on discriminative tasks and can be limited to smaller shifts. On the other hand, discriminative approaches can handle larger domain shifts, but impose tied weights on the model and do not exploit a GAN-based loss. In this work, we first outline a novel generalized framework for adversarial adaptation, which subsumes recent state-of-the-art approaches as special cases, and use this generalized view to better relate prior approaches. We then propose a previously unexplored instance of our general framework which combines discriminative modeling, untied weight sharing, and a GAN loss, which we call Adversarial Discriminative Domain Adaptation (ADDA). We show that ADDA is more effective yet considerably simpler than competing domain-adversarial methods, and demonstrate the promise of our approach by exceeding state-of-the-art unsupervised adaptation results on standard domain adaptation tasks as well as a difficult cross-modality object classification task.


Remarkably, I3D #REFR reveals that successful 2D networks could be seamlessly inflated to corresponding 3D networks and even their parameter.

The advancement of visual backbone networks prospers various downstream tasks tremendously #OTHEREFR .

Based on the modality of input data, visual networks can be classified into two types, image networks (i.e. 2D networks) and video networks (i.e. 3D networks).

C3D #OTHEREFR pioneeringly devises an 11-layer CNN with 3D-CNN to adapt to video inputs.

Subsequent P3D #OTHEREFR , S3D #OTHEREFR and R(2+1)D #OTHEREFR observe that disentangled spatial and temporal convolutions results in a more favorable speed-accuracy trade-off than the pure 3D convolution.

In recent days, a shift in backbone architecture, from CNNs to Transformers (ViT) #OTHEREFR , has begun.

Especially, Swin Transformer #OTHEREFR reintroduces the inductive bias of convolutions (i.e., locality, translation invariance and hierarchy), which enables it to serve as a general-purpose backbone.

The success of image Transformer leads to further investigation of Transformer-based video networks (e.g., ViViT #OTHEREFR , MViT #OTHEREFR , Video Swin Transformer #OTHEREFR ).

Among all characteristics of Transformers, the patch-wise operations inherently differentiate the edges of patches, thus making them ideal for handling input sampled by GMS. Figure 2 . Illustration of the framework of Zoom-VQA. The overall architecture consists of two parts.

One part is a perception network based on image input that obtains global information.

Zoom-Vqa: Patches, Frames And Clips Integration For Video Quality Assessment

Abstract: Video quality assessment (VQA) aims to simulate the human perception of video quality, which is influenced by factors ranging from low-level color and texture details to high-level semantic content. To effectively model these complicated quality-related factors, in this paper, we decompose video into three levels (i.e., patch level, frame level, and clip level), and propose a novel Zoom-VQA architecture to perceive spatio-temporal features at different levels. It integrates three components: patch attention module, frame pyramid alignment, and clip ensemble strategy, respectively for capturing region-of-interest in the spatial dimension, multi-level information at different feature levels, and distortions distributed over the temporal dimension. Owing to the comprehensive design, Zoom-VQA obtains state-of-the-art results on four VQA benchmarks and achieves 2nd place in the NTIRE 2023 VQA challenge. Notably, Zoom-VQA has outperformed the previous best results on two subsets of LSVQ, achieving 0.8860 (+1.0%) and 0.7985 (+1.9%) of SRCC on the respective subsets. Adequate ablation studies further verify the effectiveness of each component. Codes and models are released in https://github.com/k-zha14/Zoom-VQA.

Quo Vadis, Action Recognition? A New Model And The Kinetics Dataset

Abstract: The paucity of videos in current action classification datasets (UCF-101 and HMDB-51) has made it difficult to identify good video architectures, as most methods obtain similar performance on existing small-scale benchmarks. This paper re-evaluates state-of-the-art architectures in light of the new Kinetics Human Action Video dataset. Kinetics has two orders of magnitude more data, with 400 human action classes and over 400 clips per class, and is collected from realistic, challenging YouTube videos. We provide an analysis on how current architectures fare on the task of action classification on this dataset and how much performance improves on the smaller benchmark datasets after pre-training on Kinetics. ::: We also introduce a new Two-Stream Inflated 3D ConvNet (I3D) that is based on 2D ConvNet inflation: filters and pooling kernels of very deep image classification ConvNets are expanded into 3D, making it possible to learn seamless spatio-temporal feature extractors from video while leveraging successful ImageNet architecture designs and even their parameters. We show that, after pre-training on Kinetics, I3D models considerably improve upon the state-of-the-art in action classification, reaching 80.9% on HMDB-51 and 98.0% on UCF-101.


Overfitting is a general issue in the domain of supervised machine learning and cannot be avoided #REFR .

The top performing model achieved 49.36% accuracy at the validation level.

Given three balanced classes of the input data, the result outperforms random guessing by 48% at the aggregate level.

The shape of both curves indicate that the training was a subject of overfitting.

An overfitted model may fail to properly generalize features that it is supposed to learn and instead fits the idiosyncrasies of the training sample itself.

Such model would perform well during the training but unsatisfactorily to any other data but the one on which it was trained.

To contain the effect of overfitting, we added a dropout layer that discarded 20% of random units to the end of the network just prior to the classification.

Dropout is generally considered very effective technique against overfitting #OTHEREFR , #OTHEREFR .

Predicting Stock Price Movement As An Image Classification Problem

Abstract: The paper studies intraday price movement of stocks that is considered as an image classification problem. Using a CNN-based model we make a compelling case for the highlevel relationship between the first hour of trading and the close. The algorithm managed to adequately separate between the two opposing classes and investing according to the algorithm's predictions outperformed all alternative constructs but the theoretical maximum. To support the thesis, we ran several additional tests. The findings in the paper highlight the suitability of computer vision techniques for studying financial markets and in particular prediction of stock price movements.

Overfitting In Adversarially Robust Deep Learning

Abstract: It is common practice in deep learning to use overparameterized networks and train for as long as possible; there are numerous studies that show, both theoretically and empirically, that such practices surprisingly do not unduly harm the generalization performance of the classifier. In this paper, we empirically study this phenomenon in the setting of adversarially trained deep networks, which are trained to minimize the loss under worst-case adversarial perturbations. We find that overfitting to the training set does in fact harm robust performance to a very large degree in adversarially robust training across multiple datasets (SVHN, CIFAR-10, CIFAR-100, and ImageNet) and perturbation models ($\ell_\infty$ and $\ell_2$). Based upon this observed effect, we show that the performance gains of virtually all recent algorithmic improvements upon adversarial training can be matched by simply using early stopping. We also show that effects such as the double descent curve do still occur in adversarially trained models, yet fail to explain the observed overfitting. Finally, we study several classical and modern deep learning remedies for overfitting, including regularization and data augmentation, and find that no approach in isolation improves significantly upon the gains achieved by early stopping. All code for reproducing the experiments as well as pretrained model weights and training logs can be found at this https URL.


Meanwhile, the augmented kNN-graph has approximately L∼n vi •M edges, where n vi =nn+rn > 2 #REFR .

4.5b) , and the ordering of NN can result from measurement errors.

We assume that the number of neighbors nn must meet two conditions.

First, the kNN-graph should be fully connected (or approximately, that is, the size of the largest component should be comparable to the size of the entire graph).

Second, the kNN-graph augmented with approximately rn edges should be at least a minimal n-rigid graph (in 2D: 2-rigid).

The lower band of the number of connections L, required to make the 2-rigid augmented k NN-graph, is L∼2•M.

As our experience shows, the probability that the largest connected component is rigid (or approximately rigid) is very high.

In summary, to obtain the largest connected component approximately equal to the full kNN-graph, the number of nearest neighbors nn can be very low (mostly nn = 2, but for some specific datasets with very similar samples, it can be a bit larger).

Assuming additionally that rn = 1, we can obtain a stable and rigid 2-D embedding of the kNN-graph.

This way, instead of the O(M 2 ) floating point D matrix, we have as input data O(nn • M) integers -the list of edges of kNN graph.

The indices of rn random neighbors can be generated ad hoc during the embedding process.

Abstract: Interactive visual exploration of large, high-dimensional datasets plays a very important role in various fields of science, which requires aggregated information about mutual relationships between numerous objects. It enables not only to recognize their important structural features and forms, such as clusters of vertices and their connectivity patterns, but also to assess their mutual relationships in terms of position, distance, shape, and connection density. The structural properties of these large datasets can be scrutinized throughout their interactive visualization. We argue that the visualization of very high-dimensional data is well approximated by the two-dimensional (2D) problem of embedding undirected kNN-graphs. In the advent of the big data era, the size of complex networks (datasets) G(V, E) (|V|=M∼10 6+) represents a great challenge for today's computer systems and still requires more efficient ND→2D dimensionality reduction (DR) algorithms. The existing DR methods, which involve more computational and memory complexities than O(M), are too slow for interactive manipulation on large networks that involve millions of vertices. We show that high-quality embeddings can be produced with minimal time&memory complexity. Very efficient IVHD (interactive visualization of high-dimensional data) and IVHD-CUDA algorithms are presented and then compared to the state-of-the-art DR methods (both CPU and GPU versions): t-SNE, UMAP, TriMAP, PaCMAP, BH-SNE-CUDA, and AtSNE-CUDA. We show that the memory and time requirements for IVHD are radically lower than those for the baseline codes. For example, IVHD-CUDA is almost 30 times faster in embedding (without the kNN graph generation procedure, which is the same for all methods) of one of the largest datasets used, YAHOO (M=1.4 • 10 6), than AtSNE-CUDA. We conclude that at the expense of a minor deterioration of embedding quality, compared to baseline algorithms, IVHD well preserves the main structural properties of ND data in 2D for a significantly lower computational budget. We also present a meta-algorithm that enables using any unsupervised DR method in supervised fashion and as a result allows for flexible control of global and local properties of the embedding. Thus, our methods can be a good candidate for an interactive visualization of truly big data (M=10 8+) and can be further used to inspect and interpret relationships between alternative representations of observations learned by artificial neural networks (ANN). Additionally, we have provided a framework for testing the trade-off between preservation of global structure and local structure of DR.

2-D Embedding Of Large And High-Dimensional Data With Minimal Memory And Computational Time Requirements

Abstract: In the advent of big data era, interactive visualization of large data sets consisting of M*10^5+ high-dimensional feature vectors of length N (N ~ 10^3+), is an indispensable tool for data exploratory analysis. The state-of-the-art data embedding (DE) methods of N-D data into 2-D (3-D) visually perceptible space (e.g., based on t-SNE concept) are too demanding computationally to be efficiently employed for interactive data analytics of large and high-dimensional datasets. Herein we present a simple method, ivhd (interactive visualization of high-dimensional data tool), which radically outperforms the modern data-embedding algorithms in both computational and memory loads, while retaining high quality of N-D data embedding in 2-D (3-D). We show that DE problem is equivalent to the nearest neighbor nn-graph visualization, where only indices of a few nearest neighbors of each data sample has to be known, and binary distance between data samples -- 0 to the nearest and 1 to the other samples -- is defined. These improvements reduce the time-complexity and memory load from O(M log M) to O(M), and ensure minimal O(M) proportionality coefficient as well. We demonstrate high efficiency, quality and robustness of ivhd on popular benchmark datasets such as MNIST, 20NG, NORB and RCV1.


After conducting linear probing, the vanilla knowledge distillation #REFR method is used to train students with hard and soft labels.

Our model reprogramming in TDMR is to design a proper reprogramming space for KD, that is, a projector with a classifier is simple yet well-suited for the KD on downstream task.

(ii) In distillation phase, the KD in MRKD usually leverages one-stage distillation, a brute-forced transfer, while our KD in TDMR is a twostage progressive distillation, a mild and more efficient way, which will be demonstrated in the following.

(iii) More importantly, we reprogram students according to the proxy space, i.e., we consider the knowledge of the reprogrammed teacher classifier, which is missing in MRKD. Comparison with Linear Probing.

For the foundation model application to downstream tasks, a common transfer method is the linear probing #OTHEREFR , #OTHEREFR which just modifies the output dimension of the teacher classifier to the total number of categories of the target data.

Then, the classifier is trained from scratch with the feature extractor frozen instead of fully fine-tuning the source model with high computation cost.

The whole process above is denoted by "Lin." We conduct experiments with various combinations of teachers and students on Caltech-256-60 and Oxford-102 to compare the common method with ours.

From the results in Table 6 , we can see that our method outperforms the common method "Lin.".

Moreover, our method exhibits a better improvement on the Oxford-102 dataset than the Caltech-256 dataset.

It can be attributed to the larger gap between the Oxford dataset and the pretraining dataset.

These results provide further evidence of the superior performance of our proposed method, especially in scenarios with larger domain gaps.

Towards Efficient Task-Driven Model Reprogramming With Foundation Models

Abstract: Vision foundation models exhibit impressive power, benefiting from the extremely large model capacity and broad training data. However, in practice, downstream scenarios may only support a small model due to the limited computational resources or efficiency considerations. Moreover, the data used for pretraining foundation models are usually invisible and very different from the target data of downstream tasks. This brings a critical challenge for the real-world application of foundation models: one has to transfer the knowledge of a foundation model to the downstream task that has a quite different architecture with only downstream target data. Existing transfer learning or knowledge distillation methods depend on either the same model structure or finetuning of the foundation model. Thus, naively introducing these methods can be either infeasible or very inefficient. How to leverage the knowledge from the foundation model to boost the small model has not been well studied. To address this, we propose a Task-Driven Model Reprogramming (TDMR) framework. Specifically, we reprogram the foundation model to project the knowledge into a proxy space, which alleviates the adverse effect of task mismatch and domain inconsistency. In this stage, we maintain the foundation model as a powerful feature extractor frozen. Then, we reprogram the target model via progressive distillation from the proxy space to efficiently learn the knowledge from the reprogrammed foundation model. TDMR is compatible with different pre-trained model types (CNN, transformer or their mix) and limited target data, and promotes the wide applications of vision foundation models to downstream tasks in a cost-effective manner. Extensive experiments on different downstream classification tasks and target model structures demonstrate the effectiveness of our methods with both CNNs and transformer foundation models. For example, on CUB-200, TDMR improves the accuracy of MobileNetV2 from 62.90% to 72.60% using the ResNet-50 as a teacher and to 76.04% using the Swin transformer as a teacher.

Distilling The Knowledge In A Neural Network

Abstract: A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.


Another method proposes the use of multiple monocular 2D keypoint estimates from spread out viewpoints to estimate the poses of vehicles #REFR .

A recent work combines Stereo 2D keypoint detections within a trained network to estimate the 3D positions of each keypoint for transparent and translucent objects #OTHEREFR .

The results show an improvement over the state of the art by a factor of 1.5-3.5, using only two known views from a stereo camera.

The objects are unoccluded as opposed to placed in bins, however, and only one instance of the object can be present in the scene.

While this approach is faster, the network architecture is fixed to take in exactly two stereo images as input, and to output exactly one keypoint prediction.

Multi-object detection will likely prove difficult as repeated features and occlusions from different instances of the same object may confuse the depth estimation within the network.

The transformations between these viewpoints are unknown; instead the approach adds constraints based on object rigidity and the relative positions of different keypoints in the image to solve for the vehicle poses.

We borrow the object rigidity contraint from this paper, but also make use of known transformations between viewpoints from a robot arm, to allow more precise fusion of estimates between views.

CosyPose is able to achieve higher performance in more cluttered scenes by utilizing a larger number of views #OTHEREFR .

The scene is assumed to be static across the different viewpoints, and the poses of all objects within the scene are estimated.

This is then used to estimate the camera pose across different viewpoints, followed by bundle adjustment to refine the estimates and generate a globally consistent scene across all views.

Multi-View Keypoints For Reliable 6D Object Pose Estimation

Abstract: 6D Object pose estimation is a fundamental component in robotics enabling efficient interaction with the environment. It is particularly challenging in bin-picking applications, where many objects are low-feature and reflective, and self-occlusion between objects of the same type is common. We propose a novel multi-view approach leveraging known camera transformations from an eye-in-hand setup to combine heatmap and keypoint estimates into a probability density map over 3D space. The result is a robust approach that is scalable in the number of views. It relies on a confidence score composed of keypoint probabilities and point-cloud alignment error, which allows reliable rejection of false positives. We demonstrate an average pose estimation error of approximately 0.5 mm and 2 degrees across a variety of difficult low-feature and reflective objects in the ROBI dataset, while also surpassing the stateof-art correct detection rate, measured using the 10% object diameter threshold on ADD error.

Vehicle Pose And Shape Estimation Through Multiple Monocular Vision

Abstract: In this paper, we present a method to estimate a vehicle's pose and shape from off-board multi-view images. These images are taken from monocular cameras with small overlaps. We utilize state-of-the-art Convolutional Neural Networks (CNNs) to extract vehicles' semantic keypoints and introduce a Cross Projection Optimization (CPO) method to estimate the 3D pose. During the iterative CPO process, an adaptive shape adjustment method named Hierarchical Wireframe Constraint (HWC) is implemented to estimate the shape. Our approach is evaluated under both simulated and real-world scenes for performance verification. It's shown that our algorithm outperforms other existing monocular and stereo methods for vehicles' pose and shape estimation. This approach provides a new and robust solution for off-board visual vehicle localization and tracking, which can be applied to massive surveillance camera networks for intelligent transportation.


A. Current Adversarial Techniques 1) Gradient-Based Evasion Attack:

Adversarial sampling based on addition of perturbation #REFR We can carry out this type of attack by trial and error method as we don't know in advance, the exact data manipulation that will break the model and make it to classify.

In gradient base evasion attack, a perturbed image which seems like untampered to human eyes is made to be misclassified by neural network model (Fig. 1 ) #OTHEREFR . Fig. 1 .

Let say we want to probe the boundaries of a machine learning model designed to filter out spam emails, it is possible for us to experiment by sending different emails to see what gets through.

And so, a model has been trained for certain words like "momentum", and now we want to make an exceptions for emails that contains other words, if we want to attack, we can craft email with enough extraneous words which will eventually make the model to misclassify it.

Adversarial Sampling For Fairness Testing In Deep Neural Network

Abstract: In this research, we focus on the usage of adversarial sampling to test for the fairness in the prediction of deep neural network model across different classes of image in a given dataset. While several framework had been proposed to ensure robustness of machine learning model against adversarial attack, some of which includes adversarial training algorithm. There is still the pitfall that adversarial training algorithm tends to cause disparity in accuracy and robustness among different group. Our research is aimed at using adversarial sampling to test for fairness in the prediction of deep neural network model across different classes or categories of image in a given dataset. We successfully demonstrated a new method of ensuring fairness across various group of input in deep neural network classifier. We trained our neural network model on the original image, and without training our model on the perturbed or attacked image. When we feed the adversarial samplings to our model, it was able to predict the original category/ class of the image the adversarial sample belongs to. We also introduced and used the separation of concern concept from software engineering whereby there is an additional standalone filter layer that filters perturbed image by heavily removing the noise or attack before automatically passing it to the network for classification, we were able to have accuracy of 93.3%. Cifar-10 dataset have ten categories of dataset, and so, in order to account for fairness, we applied our hypothesis across each categories of dataset and were able to get a consistent result and accuracy.

Explaining And Harnessing Adversarial Examples

Abstract: Several machine learning models, including neural networks, consistently misclassify adversarial examples---inputs formed by applying small but intentionally worst-case perturbations to examples from the dataset, such that the perturbed input results in the model outputting an incorrect answer with high confidence. Early attempts at explaining this phenomenon focused on nonlinearity and overfitting. We argue instead that the primary cause of neural networks' vulnerability to adversarial perturbation is their linear nature. This explanation is supported by new quantitative results while giving the first explanation of the most intriguing fact about them: their generalization across architectures and training sets. Moreover, this view yields a simple and fast method of generating adversarial examples. Using this approach to provide examples for adversarial training, we reduce the test set error of a maxout network on the MNIST dataset.


Future research will focus on implementing an SNN version of CMLs based on resonate-and-fire (RF) neurons #REFR .

b) The EXP model for CML A can be trained with task-agnostic proxy symbols, which are then mapped to application specific inputs x and s D .

A major challenge to the greater adoption of HDC is the need for algorithms to map real-valued sensor data to hypervector symbols #OTHEREFR .

Yet since ANNs have a rich history in classification: mapping raw sensor data to arbitrary class labels, there is recent work training ANNs as task-agnostic feature extractors #OTHEREFR then mapping these sparse feature vectors to arbitrary hypervector symbols for subsequent HDC computation #OTHEREFR .

This approach, in effect, turns ANNs themselves into modular ML components, functioning as the ML equivalent of analog to digital (A2D) converters.

Lastly, the CML algorithm operates over real-valued neural networks; but the illustrative biology examples described are based on spiking neural networks (SNN).

These types of SNNs encode information based on the time a neuron spikes within period τ as opposed to rate encoding, measuring the number of spikes within a time window.

Importantly, a spike at time t with respect to a local oscillator of period τ may be expressed as a complex valued phasor (or phase vector).

RF neurons therefore also facilitate HDC interfacing via Holographic Reduced Representations (HRR), based on complex phasors #OTHEREFR .

Modularizing And Assembling Cognitive Map Learners Via Hyperdimensional Computing

Abstract: Biological organisms must learn how to control their own bodies to achieve deliberate locomotion, that is, predict their next body position based on their current position and selected action. Such learning is goal-agnostic with respect to maximizing (minimizing) an environmental reward (penalty) signal. A cognitive map learner (CML) is a collection of three separate yet collaboratively trained artificial neural networks which learn to construct representations for the node states and edge actions of an arbitrary bidirectional graph. In so doing, a CML learns how to traverse the graph nodes; however, the CML does not learn when and why to move from one node state to another. This work created CMLs with node states expressed as high dimensional vectors suitable for hyperdimensional computing (HDC), a form of symbolic machine learning (ML). In so doing, graph knowledge (CML) was segregated from target node selection (HDC), allowing each ML approach to be trained independently. The first approach used HDC to engineer an arbitrary number of hierarchical CMLs, where each graph node state specified target node states for the next lower level CMLs to traverse to. Second, an HDC-based stimulus-response experience model was demonstrated per CML. Because hypervectors may be in superposition with each other, multiple experience models were added together and run in parallel without any retraining. Lastly, a CML-HDC ML unit was modularized: trained with proxy symbols such that arbitrary, application-specific stimulus symbols could be operated upon without retraining either CML or HDC model. These methods provide a template for engineering heterogenous ML systems.

Robust Computation With Rhythmic Spike Patterns

Abstract: Information coding by precise timing of spikes can be faster and more energy efficient than traditional rate coding. However, spike-timing codes are often brittle, which has limited their use in theoretical neuroscience and computing applications. Here, we propose a type of attractor neural network in complex state space and show how it can be leveraged to construct spiking neural networks with robust computational properties through a phase-to-timing mapping. Building on Hebbian neural associative memories, like Hopfield networks, we first propose threshold phasor associative memory (TPAM) networks. Complex phasor patterns whose components can assume continuous-valued phase angles and binary magnitudes can be stored and retrieved as stable fixed points in the network dynamics. TPAM achieves high memory capacity when storing sparse phasor patterns, and we derive the energy function that governs its fixed-point attractor dynamics. Second, we construct 2 spiking neural networks to approximate the complex algebraic computations in TPAM, a reductionist model with resonate-and-fire neurons and a biologically plausible network of integrate-and-fire neurons with synaptic delays and recurrently connected inhibitory interneurons. The fixed points of TPAM correspond to stable periodic states of precisely timed spiking activity that are robust to perturbation. The link established between rhythmic firing patterns and complex attractor dynamics has implications for the interpretation of spike patterns seen in neuroscience and can serve as a framework for computation in emerging neuromorphic devices.


BAE-Net trains an implicit shape representation that uses multiple decoder "heads," where each head tends to represent the same localized part across many shape instances #REFR .

Several methods focus on convex polyhedra, either decomposing individual shapes in isolation #OTHEREFR or training a neural network to produce similar convex decompositions for similar shapes from a category #OTHEREFR .

Another option is to decompose the input shape into pieces which can be represented as generalized cylinders #OTHEREFR .

These approaches produce clean geometry with a better fit to the input shape than paramatric primitives allow, but they offer no control over the type of decomposition produced.

They also typically need many primitives to fit the input shape well, making them non-compact and not well-suited for shape editing.

Recent research in this space has focused on decomposing shapes using neural primitives.

Other approaches represent neural parts as star domains #OTHEREFR or deformed sphere meshes #OTHEREFR .

These approaches produce decompositions that fit the input shape well using a small number of primitives.

However, their output geometry can exhibit undesirable artifacts, and they provide no control over the type of decomposition produced.

We compare our algorithm to one of these approaches later in the paper and show that ours achieves even better reconstruction accuracy while also producing qualitatively better decompositions.

Modeling by retrieval and assembly: A large body of work in computer graphics has considered computerassisted or fully-automated 3D modeling via retrieving and assembling pre-existing 3D shapes.

Unsupervised 3D Shape Reconstruction By Part Retrieval And Assembly

Abstract: Figure 1. Our system takes target 3D shapes together with a 3D part library as input and outputs a set of retrieved and transformed parts from the part library that recreates the input target shapes.

Bae-Net: Branched Autoencoder For Shape Co-Segmentation

Abstract: We treat shape co-segmentation as a representation learning problem and introduce BAE-NET, a branched autoencoder network, for the task. The unsupervised BAE-NET is trained with all shapes in an input collection using a shape reconstruction loss, without ground-truth segmentations. Specifically, the network takes an input shape and encodes it using a convolutional neural network, whereas the decoder concatenates the resulting feature code with a point coordinate and outputs a value indicating whether the point is inside/outside the shape. Importantly, the decoder is branched: each branch learns a compact representation for one commonly recurring part of the shape collection, e.g., airplane wings. By complementing the shape reconstruction loss with a label loss, BAE-NET is easily tuned for one-shot learning. We show unsupervised, weakly supervised, and one-shot learning results by BAE-NET, demonstrating that using only a couple of exemplars, our network can generally outperform state-of-the-art supervised methods trained on hundreds of segmented shapes.


Comparison Of Models Used To Determine Node Importance

Although GNNExplainer #REFR was also not designed for unsupervised node embeddings, we could still use the same objective function to quantify the importance of each node by leveraging the available labels.

However, LIME is not designed for generating global node-level explanations for unsupervised node embeddings.

For a fair comparison, similar to our GRAPH-wGD, we computed MEAN{LIME

(v c |v i ), v c ∈ N(v i )} by regarding v c as local features for v i . N G (v i |ψ)

also returns the neighbors with the same size of neighbor set, ψ.

Thus, the mutual information between predictions from an input graph and a perturbed graph by v i was leveraged for the importance score of v i .

In this case, we assumed that nodes that return low mutual information values were more important.

Generating Post-Hoc Explanations For Skip-Gram-Based Node Embeddings By Identifying Important Nodes With Bridgeness

Abstract: Node representation learning in a network is an important machine learning technique for encoding relational information in a continuous vector space while preserving the inherent properties and structures of the network. Recently, unsupervised node embedding methods such as DeepWalk [1], LINE [2], struc2vec [3], PTE [4], UserItem2vec [5], and RWJBG [6] have emerged from the Skip-gram model [7] and perform better performance in several downstream tasks such as node classification and link prediction than the existing relational models. However, providing posthoc explanations of Skip-gram-based embeddings remains a challenging problem because of the lack of explanation methods and theoretical studies applicable for embeddings. In this paper, we first show that global explanations to the Skip-gram-based embeddings can be found by computing bridgeness under a spectral cluster-aware local perturbation. Moreover, a novel gradient-based explanation method, which we call GRAPH-wGD, is proposed that allows the top-q global explanations about learned graph embedding vectors more efficiently. Experiments show that the ranking of nodes by scores using GRAPH-wGD is highly correlated with true bridgeness scores. We also observe that the top-q node-level explanations selected by GRAPH-wGD have higher importance scores and produce more changes in class label prediction when perturbed, compared with the nodes selected by recent alternatives, using five real-world graphs.

Gnnexplainer: Generating Explanations For Graph Neural Networks

Abstract: Graph Neural Networks (GNNs) are a powerful tool for machine learning on graphs. GNNs combine node feature information with the graph structure by recursively passing neural messages along edges of the input graph. However, incorporating both graph structure and feature information leads to complex models and explaining predictions made by GNNs remains unsolved. Here we propose GnnExplainer, the first general, model-agnostic approach for providing interpretable explanations for predictions of any GNN-based model on any graph-based machine learning task. Given an instance, GnnExplainer identifies a compact subgraph structure and a small subset of node features that have a crucial role in GNN's prediction. Further, GnnExplainer can generate consistent and concise explanations for an entire class of instances. We formulate GnnExplainer as an optimization task that maximizes the mutual information between a GNN's prediction and distribution of possible subgraph structures. Experiments on synthetic and real-world graphs show that our approach can identify important graph structures as well as node features, and outperforms alternative baseline approaches by up to 43.0% in explanation accuracy. GnnExplainer provides a variety of benefits, from the ability to visualize semantically relevant structures to interpretability, to giving insights into errors of faulty GNNs.


Related Works In Counterfactual Explanations

Many works have addressed the issue of generating natural-sounding counterfactual explanations in text using language representation models such as BERT #REFR .

Simply removing words from the text to generate counterfactual explanations is not effective. #OTHEREFR

(2020) addressed this issue by ensuring that replaced words are grammatically correct.

They demonstrate their approach on a sentiment analysis task, introducing two lists of words: one containing words that are suitable for replacement based on grammar, and another containing words with opposite senses to those in the sentiment dictionary.

They then identified the intersection of these two lists and replaced words in the main text with words from this intersection until the predicted class was changed.

This approach helps to generate counterfactual explanations that are more understandable to humans.

#OTHEREFR proposed one example of such an approach.

They first generated a candidate set of words to replace each word in the text.

They then used BERT as a language model to determine the probability of each candidate token for a given position.

In the second step, they found the best combination of changes using shapley values #OTHEREFR and generated the explanations using beam search.

This approach allows the generation of more coherent and understandable counterfactual explanations in text.

Explaining Recommendation System Using Counterfactual Textual Explanations

Abstract: Currently, there is a significant amount of research being conducted in the field of artificial intelligence to improve the explainability and interpretability of deep learning models. It is found that if end-users understand the reason for the production of some output, it is easier to trust the system. Recommender systems are one example of systems that great efforts have been conducted to make their output more explainable. One method for producing a more explainable output is using counterfactual reasoning, which involves altering minimal features to generate a counterfactual item that results in changing the output of the system. This process allows the identification of input features that have a significant impact on the desired output, leading to effective explanations. In this paper, we present a method for generating counterfactual explanations for both tabular and textual features. We evaluated the performance of our proposed method on three real-world datasets and demonstrated a +5% improvement on finding effective features (based on model-based measures) compared to the baseline method.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


The cost of the walk S in G is at most OPT G + 2 max k S k (see Lemma 5.1 in #REFR ).

Therefore, an optimal solution to RMCCP on V i will have a total length at most 2 i OPT G (otherwise, the optimal solution to min-max weighted latency problem has cost more than OPT G ).

As the solution to RMCCP is partitioned into 2 i walks in Line 6, length of the walk W i,j is at most αOPT G where α is the approximation ratio of RMCCP.

The walk S k is given by [W 0,j0 , ν, W 1,j1 , ν, . . .

, ν, W log t,j log t ] where k = j i ( mod 2 i ) for 0 ≤ i < log t.

Since, there are log ρ G walks in S k , and those walks are connected using the recharging vertex, and (ν, v) < OPT G /2 for all v ∈ V , the length of the walk S k is at most O(α log ρ G )OPT G .

Also OPT G ≤ OPT G ≤ 2OPT G (see Lemma 3.2 in #OTHEREFR ).

Using the approximation ratio of RMCCP, the maximum weighted latency of the walk returned by algorithm 3 is O(min{log n, log D log log D } log ρ)OPT G .

Algorithm 3 is used as a subroutine in Algorithm 4 to find walks for R robots.

The following result characterizes the cost of the solution returned by Algorithm 4.

Multi-Robot Persistent Monitoring: Minimizing Latency And Number Of Robots With Recharging Constraints

Abstract: In this paper we study multi-robot path planning for persistent monitoring tasks. We consider the case where robots have a limited battery capacity with a discharge time D. We represent the areas to be monitored as the vertices of a weighted graph. For each vertex, there is a constraint on the maximum allowable time between robot visits, called the latency. The objective is to find the minimum number of robots that can satisfy these latency constraints while also ensuring that the robots periodically charge at a recharging depot. The decision version of this problem is known to be PSPACE-complete. We present a O(log D log log D log ρ) approximation algorithm for the problem where ρ is the ratio of the maximum and the minimum latency constraints. We also present an orienteering based heuristic to solve the problem and show empirically that it typically provides higher quality solutions than the approximation algorithm. We extend our results to provide an algorithm for the problem of minimizing the maximum weighted latency given a fixed number of robots. We evaluate our algorithms on large problem instances in a patrolling scenario and in a wildfire monitoring application. We also compare the algorithms with an existing solver on benchmark instances.

Persistent Monitoring In Discrete Environments: Minimizing The Maximum Weighted Latency Between Observations

Abstract: In this paper, we consider the problem of planning a path for a robot to monitor a known set of features of interest in an environment. We represent the environment as a graph with vertex weights and edge lengths. The vertices represent regions of interest, edge lengths give travel times between regions, and the vertex weights give the importance of each region. As the robot repeatedly performs a closed walk on the graph, we define the weighted latency of a vertex to be the maximum time between visits to that vertex, weighted by the importance (vertex weight) of that vertex. Our goal is to find a closed walk that minimizes the maximum weighted latency of any vertex. We show that there does not exist a polynomial time algorithm for the problem. We then provide two approximation algorithms; an $O(\log n)$-approximation algorithm and an $O(\log \rho_G)$-approximation algorithm, where $\rho_G$ is the ratio between the maximum and minimum vertex weights. We provide simulation results which demonstrate that our algorithms can be applied to problems consisting of thousands of vertices, and a case study for patrolling a city for crime.


Although this is the same as the complexity of a tSVD algorithm based on the rank-revealing QR decomposition, the Randomized SVD has been shown to be experimentally faster than that algorithm #REFR .

where E X is the expectation with respect to X, and || • || F is the Frobenius norm.

We show the algorithm of the Randomized SVD in Algorithm 1.

In the algorithm, the asymptotic computational complexity of the QR factorization (Line 2) and SVD (Line 4) are O(mp 2 ) and O(np 2 ), respectively, and the random projection (Line 1) is O(mnp).

Therefore, since p m, n typically, the random projection is the most expensive computation in the algorithm, and the total asymptotic complexity of the Randomized SVD is O(mnp).

Furthermore, to reduce the complexity of the random projection, there are some studies on using random matrices instead of the Gaussian random matrix.

For instance, when using subsampled random Fourier transform, the computational complexity is O(mn • log(p)) #OTHEREFR .

We can also use sparse random matrices for the random projection, although they were not originally proposed in the context of RandNLA.

These matrices are sparse from the perspective of non-zero elements and the choice of element values.

For instance, Achlioptas proposes a random matrix Ω ∈ R n×q where the (i, j) element is decided as follows #OTHEREFR :

Mixed-Precision Random Projection For Randnla On Tensor Cores

Abstract: Random projection can reduce the dimension of data while capturing its structure and is a fundamental tool for machine learning, signal processing, and information retrieval, which deal with a large amount of data today. RandNLA (Randomized Numerical Linear Algebra) leverages random projection to reduce the computational complexity of low-rank decomposition of tensors and solve least-square problems. While the computation of the random projection is a simple matrix multiplication, its asymptotic computational complexity is typically larger than other operations in a RandNLA algorithm. Therefore, various studies propose methods for reducing its computational complexity. We propose a fast mixed-precision random projection method on NVIDIA GPUs using Tensor Cores for single-precision tensors. We exploit the fact that the random matrix requires less precision, and develop a highly optimized matrix multiplication between FP32 and FP16 matrices-SHGEMM (Single and Half GEMM)-on Tensor Cores, where the random matrix is stored in FP16. Our method can compute Randomized SVD 1.28 times faster and Random projection high order SVD 1.75 times faster than baseline single-precision implementations while maintaining accuracy.

Finding Structure With Randomness: Probabilistic Algorithms For Constructing Approximate Matrix Decompositions

Abstract: Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets. This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed---either explicitly or implicitly---to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, speed, and robustness. These claims are supported by extensive numerical experiments and a detailed error analysis.

Randomized SVD
rank-revealing QR decomposition


C. Performance Evaluation Of The Proposed Resource Management Strategy

Specifically, we compare the random realization of channel model #REFR in each training round and the ideal static channel channel model (i.e.

Furthermore, with a more powerful server, the performance improvements brought by power control and subchannel allocation grow.

This is because, in this case, the training latency is primarily limited by the computing latency of the client devices and data exchange latency.

Again, we observe that optimizing the cut layer selection brings better performance improvement than power control and subchannel allocation when the server's computing capability varies. Fig.

13 shows the effect of channel variation on the performance of the proposed solution.

Our layer split decision remains unchanged for a period (in the simulations, it remains the same until the model converges), and therefore it is important to evaluate how channel variation would impact its performance.

channel gain remain unchanged, which is unrealistic but used as the benchmark).

We observe that channel variation has little impact on the performance of the proposed solution, which demonstrates its robustness in dynamic wireless channel conditions.

Efficient Parallel Split Learning Over Resource-Constrained Wireless Edge Networks

Abstract: The increasingly deeper neural networks hinder the democratization of privacy-enhancing distributed learning, such as federated learning (FL), to resource-constrained devices. To overcome this challenge, in this paper, we advocate the integration of edge computing paradigm and parallel split learning (PSL), allowing multiple client devices to offload substantial training workloads to an edge server via layer-wise model split. By observing that existing PSL schemes incur excessive training latency and large volume of data transmissions, we propose an innovative PSL framework, namely, efficient parallel split learning (EPSL), to accelerate model training. To be specific, EPSL parallelizes client-side model training and reduces the dimension of activations' gradients for back propagation (BP) via last-layer gradient aggregation, leading to a significant reduction in server-side training and communication latency. Moreover, by considering the heterogeneous channel conditions and computing capabilities at client devices, we jointly optimize subchannel allocation, power control, and cut layer selection to minimize the per-round latency. Simulation results show that the proposed EPSL framework significantly decreases the training latency needed to achieve a target accuracy compared with the stateof-the-art benchmarks, and the tailored resource management and layer split strategy can considerably reduce latency than the counterpart without optimization.

Probabilistic Omnidirectional Path Loss Models For Millimeter-Wave Outdoor Communications

Abstract: This paper presents a probabilistic omnidirectional millimeter-wave path loss model based on real-world 28 GHz and 73 GHz measurements collected in New York City. The probabilistic path loss approach uses a free space line-of-sight propagation model, and for non-line-of-sight conditions uses either a close-in free space reference distance path loss model or a floating-intercept path loss model. The probabilistic model employs a weighting function that specifies the line-of-sight probability for a given transmitter-receiver separation distance. Results show that the probabilistic path loss model offers virtually identical results whether one uses a non-line-of-sight close-in free space reference distance path loss model, with a reference distance of 1 meter, or a floating-intercept path loss model. This letter also shows that site-specific environmental information may be used to yield the probabilistic weighting function for choosing between line-of-sight and non-line-of-sight conditions.


In a systematic review of user studies of multi-factor authentication, Das et al. consistently find low adoption #REFR .

Researchers have also found that in practice, users conflate encryption and authentication in their mental models of security indicators in the website security context #OTHEREFR .

Our work parallels extensive research into these passive authentication indicators in browsers, and brings a user-and data-driven analysis of passive verification indicators on social media.

User perception of security has been studied in a wide variety of online contexts. For example, Dechand et al.

study user perception of endto-end encryption on WhatsApp and find that users largely do not trust it #OTHEREFR .

Alshamsi and Andras study perception of Bitcoin usability among novice users and find that they find credit or debit cards to be more usable #OTHEREFR . Ur et al.

investigate whether user perception of password security match reality and find significant differences across users' understanding of possible attacks #OTHEREFR . Perceptions of social media users have also been studied.

However, work in this area has focused on perceptions of social media website quality #OTHEREFR , level of control over information shared #OTHEREFR , and protection from abuse and harrassment #OTHEREFR .

Usability of security features on social media platforms has only been analyzed in the context of security notices. Benson et al.

find that users disclose more information in their presence #OTHEREFR . : News consumption and social media usage.

Account Verification On Social Media: User Perceptions And Paid Enrollment

Abstract: We study the gap between user perceptions of social media verification indicators and their actual meaning, and the type of behavior that emerges when such a gap is present. We use recent changes to Twitter's verification process as a unique case study wherein the meaning of a verification indicator has rapidly shifted. The study consists of a U.S. demographicallyrepresentative survey of 300 respondents and quantitative and qualitative analyses of results, and an analysis of verified Twitter accounts sampled from a large-scale dataset of 15 million Tweets collected in October 2022. The survey addresses differences in user-perceived and actual requirements for verification marks on popular social media platforms, with a focus on evolving perceptions of verification marks on Twitter. We find that more than half of survey respondents misunderstood Twitter's criteria for assigning blue verification check marks to user accounts; more than 80% of survey respondents did not understand what differentiated blue check marks from gold and grey check marks. We also note interesting correlations between respondent age and perception of verification marks. From our qualitative analysis of verified user accounts, we find that cryptocurrency promotion accounts constitute significantly more Blue subscribers than our randomly sampled control dataset, indicating that a significant number of Blue users may be leveraging the confusion between legacy and Blue verification to promote their own commodities. Finally, we provide recommendations for improving verification indicators and processes on social media.

Evaluating User Perception Of Multi-Factor Authentication: A Systematic Review

Abstract: Security vulnerabilities of traditional single factor authentication has become a major concern for security practitioners and researchers. To mitigate single point failures, new and technologically advanced Multi-Factor Authentication (MFA) tools have been developed as security solutions. However, the usability and adoption of such tools have raised concerns. An obvious solution can be viewed as conducting user studies to create more user-friendly MFA tools. To learn more, we performed a systematic literature review of recently published academic papers (N = 623) that primarily focused on MFA technologies. While majority of these papers (m = 300) proposed new MFA tools, only 9.1% of papers performed any user evaluation research. Our meta-analysis of user focused studies (n = 57) showed that researchers found lower adoption rate to be inevitable for MFAs, while avoidance was pervasive among mandatory use. Furthermore, we noted several reporting and methodological discrepancies in the user focused studies. We identified trends in participant recruitment that is indicative of demographic biases.


The second one uses an existing NLP model (T5 #REFR ) to produce sentences from structural data.

Overall, MEDIMP with all medical prompts results in the best predictions at 2 and 4 years post-transplantation.

To demonstrate the relevance of the proposed approach for medical prompt generation, we compare our main model with two other approaches that produce text information.

The first one is noted as "Manual" and comprises all the templates indicated by the medical experts, corresponding to only one sentence per variable of interest.

Note that this is the base of our proposed medical prompting without using the prompt expansion method described in Section 2.2.

For a fair comparison, we train the T5 model on the WebNLG 2020 data #OTHEREFR and infer it on our data to generate text, denoted as "T5 WebNLG".

The results are summarised in Table 2 , highlighting the superiority of our method.

The "T5 WebNLG" approach offers a competitive F1 for all the predictions, although the AUC is the lowest except for the 2 years prediction.

We show in Appendix B examples of generated texts from these three approaches.

"Manual" approach lacks diversity in the text data, and therefore the training process of our proposed approach without text augmentations is more challenging.

Medimp: Medical Images And Prompts For Renal Transplant Representation Learning

Abstract: Renal transplantation emerges as the most effective solution for end-stage renal disease. Occurring from complex causes, a substantial risk of transplant chronic dysfunction persists and may lead to graft loss. Medical imaging plays a substantial role in renal transplant monitoring in clinical practice. However, graft supervision is multidisciplinary , notably joining nephrology, urology, and radiology, while identifying robust biomarkers from such high-dimensional and complex data for prognosis is challenging. In this work, taking inspiration from the recent success of Large Language Models (LLMs), we propose MEDIMP-Medical Images and Prompts-a model to learn meaningful multi-modal representations of renal transplant Dynamic Contrast-Enhanced Magnetic Resonance Imaging (DCE MRI) by incorporating structural clinicobiological data after translating them into text prompts. MEDIMP is based on contrastive learning from joint text-image paired embeddings to perform this challenging task. Moreover, we propose a framework that generates medical prompts using automatic textual data augmentations from LLMs. Our goal is to learn meaningful manifolds of renal transplant DCE MRI, interesting for the prognosis of the transplant or patient status (2, 3, and 4 years after the transplant), fully exploiting the available multi-modal data in the most efficient way. Extensive experiments and comparisons with other renal transplant representation learning methods with limited data prove the effectiveness of MEDIMP in a relevant clinical setting, giving new directions toward medical prompts. Our code is available at https://github.com/leomlck/MEDIMP.

Exploring The Limits Of Transfer Learning With A Unified Text-To-Text Transformer

Abstract: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.


The simplicity of our framework makes it also easier to study than negative sampling with the techniques described in #REFR , for instance.

In the earlier case all the information related to the corpus (the graph) is represented in the form of a sparse matrix, hence the offline implementation of EDRep works optimally and significantly outperformed Node2Vec both in terms of speed and of accuracy.

In the latter case, when encoding the text into a matrix, all the complex relations between the words should be captured by the matrix structure.

This is a non-trivial task that might penalize infrequent co-occurrences between pairs of words.

We believe, however, that a crucial advantage of our approach lies in its high interpretability.

For a given practical deployment of our algorithm, a practitioner only needs to define a sampling strategy that meaningfully encodes proximity for the problem at hand.

Moreover, our main result holds in a wider range of settings with respect to the one explored and it can be easily generalized to non symmetric, or non normalized P matrices.

Efficient Distributed Representations Beyond Negative Sampling

Abstract: This article describes an efficient method to learn distributed representations, also known as embeddings. This is accomplished minimizing an objective function similar to the one introduced in the Word2Vec algorithm and later adopted in several works. The optimization computational bottleneck is the calculation of the softmax normalization constants for which a number of operations scaling quadratically with the sample size is required. This complexity is unsuited for large datasets and negative sampling is a popular workaround, allowing one to obtain distributed representations in linear time with respect to the sample size. Negative sampling consists, however, in a change of the loss function and hence solves a different optimization problem from the one originally proposed. Our contribution is to show that the sotfmax normalization constants can be estimated in linear time, allowing us to design an efficient optimization strategy to learn distributed representations. We test our approximation on two popular applications related to word and node embeddings. The results evidence competing performance in terms of accuracy with respect to negative sampling with a remarkably lower computational time.

The Spectral Underpinning Of Word2Vec

Abstract: word2vec due to Mikolov \textit{et al.} (2013) is a word embedding method that is widely used in natural language processing. Despite its great success and frequent use, theoretical justification is still lacking. The main contribution of our paper is to propose a rigorous analysis of the highly nonlinear functional of word2vec. Our results suggest that word2vec may be primarily driven by an underlying spectral method. This insight may open the door to obtaining provable guarantees for word2vec. We support these findings by numerical simulations. One fascinating open question is whether the nonlinear properties of word2vec that are not captured by the spectral method are beneficial and, if so, by what mechanism.