MPNN's expressive power in terms of distinguishing non-isomorphic graphs is upper bounded by 2-WL #REFR .

MPNNs are a class of GNNs that learns node representations by iteratively encoding and aggregating messages from neighboring nodes.

Let h t v be the node representation for v ∈ V in the t-th iteration. It is usually initialized with the node's intrinsic attributes.

In the (t + 1)-th iteration, it is updated by:

where W t 1 and W t 2 are two learnable functions.

Subgraph GNNs first represent the input graph by a collection of subgraphs based on certain subgraph selection policies.

They then encode the subgraphs using backbone GNNs and aggregate subgraph representations into the graph representation.

We note that there exist some other variants of subgraph GNNs #OTHEREFR , but in this paper, we focus on a specific type of subgraph GNNs without information exchange between subgraphs, which covers reconstruction GNNs #OTHEREFR , ID-GNNs #OTHEREFR , and nested GNNs (Zhang & Li, 2021; #OTHEREFR .

We will show that this type of subgraph GNNs is powerful enough in terms of counting connected substructures.

Efficiently Counting Substructures By Subgraph Gnns Without Running Gnn On Subgraphs

Abstract: Using graph neural networks (GNNs) to approximate specific functions such as counting graph substructures is a recent trend in graph learning. Among these works, a popular way is to use subgraph GNNs, which decompose the input graph into a collection of subgraphs and enhance the representation of the graph by applying GNN to individual subgraphs. Although subgraph GNNs are able to count complicated substructures, they suffer from high computational and memory costs. In this paper, we address a non-trivial question: can we count substructures efficiently with GNNs? To answer the question, we first theoretically show that the distance to the rooted nodes within subgraphs is key to boosting the counting power of subgraph GNNs. We then encode such information into structural embeddings, and precompute the embeddings to avoid extracting information over all subgraphs via GNNs repeatedly. Experiments on various benchmarks show that the proposed model can preserve the counting power of subgraph GNNs while running orders of magnitude faster.

How Powerful Are Graph Neural Networks?

Abstract: Graph Neural Networks (GNNs) are an effective framework for representation learning of graphs. GNNs follow a neighborhood aggregation scheme, where the representation vector of a node is computed by recursively aggregating and transforming representation vectors of its neighboring nodes. Many GNN variants have been proposed and have achieved state-of-the-art results on both node and graph classification tasks. However, despite GNNs revolutionizing graph representation learning, there is limited understanding of their representational properties and limitations. Here, we present a theoretical framework for analyzing the expressive power of GNNs to capture different graph structures. Our results characterize the discriminative power of popular GNN variants, such as Graph Convolutional Networks and GraphSAGE, and show that they cannot learn to distinguish certain simple graph structures. We then develop a simple architecture that is provably the most expressive among the class of GNNs and is as powerful as the Weisfeiler-Lehman graph isomorphism test. We empirically validate our theoretical findings on a number of graph classification benchmarks, and demonstrate that our model achieves state-of-the-art performance.

MPNN's expressive power
non-isomorphic graphs


This type of domain adaptation (DA) is heterogeneous and unlike homogeneous DA approaches, there is not much work focused on heterogeneous DA as far as depth approaches are concerned, and the solution for heterogeneous deep DA is still similar to some homogeneous DA approaches #REFR .

In this paper we utilize the final classification head of the original model as h t .

This function is learned solely through supervised signal update gradients.

There are a large number of labeled image that can be studied directly for static image so h s are not the object of this study.

Critically, we want to provide a generalization of g t which can pave the way for learning of h t to improve the generalizability of SNN.

As mentioned before, due to the different sensory devices, the two data domains have different feature spaces and they have different dimensions.

In this paper we use a discrepancy-based approach.

The embedding function g would be modeled by network sharing between the source and target domains using all layers before the last classification layer.

At this point the shared g t = g s = g, the objective of optimization is finding the satisfied g in its hypothetical space G:

where X a s and X a t refer to the same classes of data in the source and target domains while X a s and X c t mean the data from different classes.

Improving The Performance Of Spiking Neural Networks On Event-Based Datasets With Knowledge Transfer

Abstract: Spiking neural networks (SNNs) have rich spatialtemporal dynamics, which are suitable for processing neuromorphic, event-based data. However, event-based datasets are usually less annotated than static datasets used in traditional deep learning. Small data scale makes SNNs prone to overfitting and limits the performance of the SNN. To enhance the generalizability of SNNs on eventbased datasets, we propose a knowledge-transfer framework that leverages static images to assist in the training on neuromorphic datasets. Our method proposes domain loss and semantic loss to exploit both domain-invariant and unique features of these two domains, providing SNNs with more generalized knowledge for subsequent targeted training on neuromorphic data. Specifically, domain loss aligns the feature space and aims to capture common features between static and event-based images, while semantic loss emphasizes that the differences between samples from different categories should be as large as possible. Experimental results demonstrate that our method outperforms existing methods on all mainstream neuromorphic vision datasets. In particular, we achieve significant performance improvement of 2.7% and 9.8% when using only 10% training data of CIFAR10-DVS and N-Caltech 101 datasets, respectively.

Deep Visual Domain Adaptation: A Survey

Abstract: Deep domain adaptation has emerged as a new learning technique to address the lack of massive amounts of labeled data. Compared to conventional methods, which learn shared feature subspaces or reuse important source instances with shallow representations, deep domain adaptation methods leverage deep networks to learn more transferable representations by embedding domain adaptation in the pipeline of deep learning. There have been comprehensive surveys for shallow domain adaptation, but few timely reviews the emerging deep learning based methods. In this paper, we provide a comprehensive survey of deep domain adaptation methods for computer vision applications with four major contributions. First, we present a taxonomy of different deep domain adaptation scenarios according to the properties of data that define how two domains are diverged. Second, we summarize deep domain adaptation approaches into several categories based on training loss, and analyze and compare briefly the state-of-the-art methods under these categories. Third, we overview the computer vision applications that go beyond image classification, such as face recognition, semantic segmentation and object detection. Fourth, some potential deficiencies of current methods and several future directions are highlighted.


This model is based on the pre-trained model BERT proposed by #REFR with a stack of 2 Transformer blocks on top, finalized with a conditional random field (CRF) prediction layer.

Table 2 displays the information regarding the number and type of entities found in the specified datasets. this study, we only consider the coarse-grained entities.

We defined the three prompts presented in Table 1 depending on the entity type differences between datasets and to respect their corresponding labels casing.

Even if the IOB/BIO 9 format is explicitly demanded for each word, tokenization by ChatGPT was inconsistent with IOB tokenized dataset files, thus, for evaluation purposes, an alignment and verification process was required to ensure evaluation consistency.

Table 3 presents the performance of ChatGPT over coarse-grained (high-level entity types) NERC in terms of strict and fuzzy microlevel precision (P), recall (R) and F-measure (F1) evaluated with CLEF-HIPE-2020-scorer #OTHEREFR .

For positioning and comparison, we also present the performance of two LM-based state-of-the-art NERC systems. Stacked NERC.

Stacked NERC proved increased performance in historical documents, while did not degrade the performance over modern data #OTHEREFR .

The same architecture was utilized as a baseline in the description of the NewsEye dataset #OTHEREFR .

Yes But.. Can Chatgpt Identify Entities In Historical Documents?

Abstract: Large language models (LLMs) have been leveraged for several years now, obtaining state-of-the-art performance in recognizing entities from modern documents. For the last few months, the conversational agent ChatGPT has "prompted" a lot of interest in the scientific community and public due to its capacity of generating plausible-sounding answers. In this paper, we explore this ability by probing it in the named entity recognition and classification (NERC) task in primary sources (e.g., historical newspapers and classical commentaries) in a zero-shot manner and by comparing it with state-of-the-art LM-based systems. Our findings indicate several shortcomings in identifying entities in historical text that range from the consistency of entity annotation guidelines, entity complexity, and code-switching, to the specificity of prompting. Moreover, as expected, the inaccessibility of historical archives to the public (and thus on the Internet) also impacts its performance. CCS CONCEPTS • Information systems → Language models; Information extraction; • Computing methodologies → Natural language processing; • Applied computing → Arts and humanities.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


Results show that our proposed model can achieve better performance compared to the well-known MobileNetV3-small model #REFR and provide better computational efficiency.

In this paper, we propose a framework for IoTbased edge computing-enabled non-speech emotion recognition systems.

We have made the following contributions: 1) We propose to leverage edge computing to design a low-latency non-speech-based emotion recognition system for resource-constrained devices.

2) We develop a computationally efficient nonspeech emotion detection system by utilising knowledge distillation.

3) We provide a detailed discussion on the potential of using non-speech emotions for various applications such as healthcare, rescue services, etc.

4) We show the effectiveness of the proposed framework by evaluating the system using two publicly available datasets.

Emotions Beyond Words: Non-Speech Audio Emotion Recognition With Edge Computing

Abstract: Non-speech emotion recognition has a wide range of applications including healthcare, crime control and rescue, and entertainment, to name a few. Providing these applications using edge computing has great potential, however, recent studies are focused on speech-emotion recognition using complex architectures. In this paper, a non-speech-based emotion recognition system is proposed, which can rely on edge computing to analyse emotions conveyed through non-speech expressions like screaming and crying. In particular, we explore knowledge distillation to design a computationally efficient system that can be deployed on edge devices with limited resources without degrading the performance significantly. We comprehensively evaluate our proposed framework using two publicly available datasets and highlight its effectiveness by comparing the results with the well-known MobileNet model. Our results demonstrate the feasibility and effectiveness of using edge computing for non-speech emotion detection, which can potentially improve applications that rely on emotion detection in communication networks. To the best of our knowledge, this is the first work on an edgecomputing-based framework for detecting emotions in non-speech audio, offering promising directions for future research.

Searching For Mobilenetv3

Abstract: We present the next generation of MobileNets based on a combination of complementary search techniques as well as a novel architecture design. MobileNetV3 is tuned to mobile phone CPUs through a combination of hardware-aware network architecture search (NAS) complemented by the NetAdapt algorithm and then subsequently improved through novel architecture advances. This paper starts the exploration of how automated search algorithms and network design can work together to harness complementary approaches improving the overall state of the art. Through this process we create two new MobileNet models for release: MobileNetV3-Large and MobileNetV3-Small which are targeted for high and low resource use cases. These models are then adapted and applied to the tasks of object detection and semantic segmentation. For the task of semantic segmentation (or any dense pixel prediction), we propose a new efficient segmentation decoder Lite Reduced Atrous Spatial Pyramid Pooling (LR-ASPP). We achieve new state of the art results for mobile classification, detection and segmentation. MobileNetV3-Large is 3.2\% more accurate on ImageNet classification while reducing latency by 15\% compared to MobileNetV2. MobileNetV3-Small is 4.6\% more accurate while reducing latency by 5\% compared to MobileNetV2. MobileNetV3-Large detection is 25\% faster at roughly the same accuracy as MobileNetV2 on COCO detection. MobileNetV3-Large LR-ASPP is 30\% faster than MobileNetV2 R-ASPP at similar accuracy for Cityscapes segmentation.


The time branch is based on the SincNet #REFR due to its promising results on speaker recognition.

2 , our TFN contains two branches (a time branch and a frequency branch) and a fusion module.

The frequency branch is a multi-layer CNN with a frequency spectrum representation (such as MFCC) as input and the fusion module is designed to fuse the time-domain feature and the frequencydomain feature.

It is worth noting that although the frequency spectrum also contains the time information, its time resolution is much lower than the raw signal due to the window-based transformation (such as STFT).

Time-Frequency Network For Robust Speaker Recognition

Abstract: The wide deployment of speech-based biometric systems usually demands high-performance speaker recognition algorithms. However, most of the prior works for speaker recognition either process the speech in the frequency domain or time domain, which may produce suboptimal results because both time and frequency domains are important for speaker recognition. In this paper, we attempt to analyze the speech signal in both time and frequency domains and propose the timefrequency network (TFN) for speaker recognition by extracting and fusing the features in the two domains. Based on the recent advance of deep neural networks, we propose a convolution neural network to encode the raw speech waveform and the frequency spectrum into domain-specific features, which are then fused and transformed into a classification feature space for speaker recognition. Experimental results on the publicly available datasets TIMIT and LibriSpeech show that our framework is effective to combine the information in the two domains and performs better than the state-of-the-art methods for speaker recognition.

Speech And Speaker Recognition From Raw Waveform With Sincnet

Abstract: Deep neural networks can learn complex and abstract representations, that are progressively obtained by combining simpler ones. A recent trend in speech and speaker recognition consists in discovering these representations starting from raw audio samples directly. Differently from standard hand-crafted features such as MFCCs or FBANK, the raw waveform can potentially help neural networks discover better and more customized representations. The high-dimensional raw inputs, however, can make training significantly more challenging. This paper summarizes our recent efforts to develop a neural architecture that efficiently processes speech from audio waveforms. In particular, we propose SincNet, a novel Convolutional Neural Network (CNN) that encourages the first layer to discover meaningful filters by exploiting parametrized sinc functions. In contrast to standard CNNs, which learn all the elements of each filter, only low and high cutoff frequencies of band-pass filters are directly learned from data. This inductive bias offers a very compact way to derive a customized front-end, that only depends on some parameters with a clear physical meaning. Our experiments, conducted on both speaker and speech recognition, show that the proposed architecture converges faster, performs better, and is more computationally efficient than standard CNNs.


Existing methods maximize the mutual information between the augmented data pairs 28, #REFR and adopt InfoNCE as the contrastive loss.

Contrastive learning is one of the most effective unsupervised learning methods in the field of representation learning.

Contrastive learning aims at seeking a latent space that separates the pairs from different classes and maximizes the similarity for samples with the same classes 28 .

However, our method uses the multi-view omics data as the compared pairs to maximize the consistency across different omics data.

Clclsa: Cross-Omics Linked Embedding With Contrastive Learning And Self Attention For Multi-Omics Integration With Incomplete Multi-Omics Data

Abstract: Integration of heterogeneous and high-dimensional multi-omics data is becoming increasingly important in understanding genetic data. Each omics technique only provides a limited view of the underlying biological process and integrating heterogeneous omics layers simultaneously would lead to a more comprehensive and detailed understanding of diseases and phenotypes. However, one obstacle faced when performing multi-omics data integration is the existence of unpaired multi-omics data due to instrument sensitivity and cost. Studies may fail if certain aspects of the subjects are missing or incomplete. In this paper, we propose a deep learning method for multi-omics integration with incomplete data by Cross-omics Linked unified embedding with Contrastive Learning and Self Attention (CLCLSA). Utilizing complete multi-omics data as supervision, the model employs cross-omics autoencoders to learn the feature representation across different types of biological data. The multi-omics contrastive learning, which is used to maximize the mutual information between different types of omics, is employed before latent feature concatenation. In addition, the feature-level self-attention and omics-level self-attention are employed to dynamically identify the most informative features for multi-omics data integration. Extensive experiments were conducted on four public multi-omics datasets. The experimental results indicated that the proposed CLCLSA outperformed the state-of-the-art approaches for multi-omics data classification using incomplete multiomics data.

A Simple Framework For Contrastive Learning Of Visual Representations

Abstract: This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.


The stochastic nature of the set function f (S) requires the use the Stochastic Continuous Greedy (SCG) algorithm #REFR .

This is a stochastic variant of the continuous greedy algorithm (method) #OTHEREFR , to solve (1).

The SCG algorithm uses a common averaging technique in stochastic optimization and computes the estimated gradient d t by the recursion

where ρ t is a positive step size and the algorithm initially starts with d 0 = y 0 = 0.

Then, it proceeds in iterations, where in the t-th iteration it finds a feasible solution as follows

Stochastic Submodular Maximization Via Polynomial Estimators

Abstract: In this paper, we study stochastic submodular maximization problems with general matroid constraints, that naturally arise in online learning, team formation, facility location, influence maximization, active learning and sensing objective functions. In other words, we focus on maximizing submodular functions that are defined as expectations over a class of submodular functions with an unknown distribution. We show that for monotone functions of this form, the stochastic continuous greedy algorithm [20] attains an approximation ratio (in expectation) arbitrarily close to (1 − 1/e) ≈ 63% using a polynomial estimation of the gradient. We argue that using this polynomial estimator instead of the prior art that uses sampling eliminates a source of randomness and experimentally reduces execution time.

Stochastic Conditional Gradient Methods: From Convex Minimization To Submodular Maximization

Abstract: This paper considers stochastic optimization problems for a large class of objective functions, including convex and continuous submodular. Stochastic proximal gradient methods have been widely used to solve such problems; however, their applicability remains limited when the problem dimension is large and the projection onto a convex set is costly. Instead, stochastic conditional gradient methods are proposed as an alternative solution relying on (i) Approximating gradients via a simple averaging technique requiring a single stochastic gradient evaluation per iteration; (ii) Solving a linear program to compute the descent/ascent direction. The averaging technique reduces the noise of gradient approximations as time progresses, and replacing projection step in proximal methods by a linear program lowers the computational complexity of each iteration. We show that under convexity and smoothness assumptions, our proposed method converges to the optimal objective function value at a sublinear rate of $O(1/t^{1/3})$. Further, for a monotone and continuous DR-submodular function and subject to a general convex body constraint, we prove that our proposed method achieves a $((1-1/e)OPT-\eps)$ guarantee with $O(1/\eps^3)$ stochastic gradient computations. This guarantee matches the known hardness results and closes the gap between deterministic and stochastic continuous submodular maximization. Additionally, we obtain $((1/e)OPT -\eps)$ guarantee after using $O(1/\eps^3)$ stochastic gradients for the case that the objective function is continuous DR-submodular but non-monotone and the constraint set is down-closed. By using stochastic continuous optimization as an interface, we provide the first $(1-1/e)$ tight approximation guarantee for maximizing a monotone but stochastic submodular set function subject to a matroid constraint and $(1/e)$ approximation guarantee for the non-monotone case.


In addition, we unlearn the 'Bang' feature from ProgGAN trained with CelebA-HQ dataset #REFR , whose resolution is higher than the other two datasets.

Note that FID is calculated using the entire dataset, which yields a slightly higher FID value for the unlearned and baseline models, but there is no significant difference between the two models.

Figure 5 presents the qualitative visualization result of our unlearning framework.

The top row shows the images generated from the original generator, and the bottom row shows those generated from the unlearned generator.

By visualizing generated images using the same latent vector, we observe that the target feature has been effectively erased in each case.

The qualitative results in Figure 1 show the approach also works well with high-resolution images. Table 3 : ROC-AUC score of feature identification method.

Feature Unlearning For Generative Models Via Implicit Feedback

Abstract: We tackle the problem of feature unlearning from a pretrained image generative model. Unlike a common unlearning task where an unlearning target is a subset of the training set, we aim to unlearn a specific feature, such as hairstyle from facial images, from the pretrained generative models. As the target feature is only presented in a local region of an image, unlearning the entire image from the pretrained model may result in losing other details in the remaining region of the image. To specify which features to unlearn, we develop an implicit feedback mechanism where a user can select images containing the target feature. From the implicit feedback, we identify a latent representation corresponding to the target feature and then use the representation to unlearn the generative model. Our framework is generalizable for the two well-known families of generative models: GANs and VAEs. Through experiments on MNIST and CelebA datasets, we show that target features are successfully removed while keeping the fidelity of the original models.

Progressive Growing Of Gans For Improved Quality, Stability, And Variation

Abstract: We describe a new training methodology for generative adversarial networks. The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses. This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2. We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10. Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator. Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation. As an additional contribution, we construct a higher-quality version of the CelebA dataset.


The authors of #REFR discuss three types of priority: per-service, per-VNF, and per-flow, where service prioritization is based on the delay attribute and per VNF priority is assigned randomly.

Authors in #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR proposed reinforcement learningbased approaches.

In a realistic scenario, each service has its own level of importance based on its type and requirements.

Following the FIFO technique can cause failure in deploying emergency services over best-effort ones.

Methods that classify SFCs as either Premium (Pr) or Best-Effort (BE), constantly prioritizes Pr services over the BE ones, resulting in severe service deprivation for low-priority services #OTHEREFR .

#OTHEREFR and #OTHEREFR demonstrated the effectiveness of mapping the services depending on the priority of VNFs.

This is an iterative process that adjusts the solution after each iteration, which is not scalable considering future requirements.

Also the predefined number of flows that a VNF can handle, ignores the operational dynamics.

Priority-based scheduling and deployment is still an issue requiring an 'intelligent' system to dynamically establish the online service priority and re-schedule the services according to the urgency and benefit.

Researchers have proposed Integer Linear Program (ILP) or heuristic models in the literature, which are either computationally costly or produce sub-optimal solutions.

To overcome this drawback, we propose ML-based algorithms to dynamically assign priority to online services and train the model to re-arrange the scheduling queue according to the needs.

Dynamic Prioritization And Adaptive Scheduling Using Deep Deterministic Policy Gradient For Deploying Microservice-Based Vnfs

Abstract: The Network Function Virtualization (NFV)-Resource Allocation (RA) problem is NP-Hard. Traditional deployment methods revealed the existence of a starvation problem, which the researchers failed to recognize. Basically, starvation here, means the longer waiting times and eventual rejection of low-priority services due to a 'time out'. The contribution of this work is threefold: a) explain the existence of the starvation problem in the existing methods and their drawbacks, b) introduce 'Adaptive Scheduling' (AdSch) which is an 'intelligent scheduling' scheme using a three-factor approach (priority, threshold waiting time, and reliability), which proves to be more reasonable than traditional methods solely based on priority, and c) a 'Dynamic Prioritization' (DyPr), allocation method is also proposed for unseen services and the importance of macro-and microlevel priority. We presented a zero-touch solution using Deep Deterministic Policy Gradient (DDPG) for adaptive scheduling and an online-Ridge Regression (RR) model for dynamic prioritization. The DDPG successfully identified the 'Beneficial and Starving' services, efficiently deploying twice as many low-priority services as others, reducing the starvation problem. Our online-RR model learns the pattern in less than 100 transitions, and the prediction model has an accuracy rate of more than 80%.

Reducing Service Deployment Cost Through Vnf Sharing

Abstract: Thanks to its computational and forwarding capabilities, the mobile network infrastructure can support several third-party ("vertical") services, each composed of a graph of virtual (network) functions (VNFs). Importantly, one or more VNFs are often common to multiple services, thus the services deployment cost could be reduced by letting the services share the same VNF instance instead of devoting a separate instance to each service. By doing that, however, it is critical that the target KPI (key performance indicators) of all services are met. To this end, we study the VNF sharing problem and make decisions on (i) when sharing VNFs among multiple services is possible, (ii) how to adapt the virtual machines running the shared VNFs to the combined load of the assigned services, and (iii) how to prioritize the services traffic within shared VNFs. All decisions aim to minimize the cost for the mobile operator, subject to requirements on end-to-end service performance, e.g., total delay. Notably, we show that the aforementioned priorities should be managed dynamically and vary across VNFs. We then propose the FlexShare algorithm to provide near-optimal VNF-sharing and priority assignment decisions in polynomial time. We prove that FlexShare is within a constant factor from the optimum and, using real-world VNF graphs, we show that it consistently outperforms baseline solutions.


Specifically, our Burstormer lead to 0.79 dB boost when compared with existing multi-level PCD alignment #REFR .

In the baseline model, we use Resblocks #OTHEREFR for feature extraction, simple concatenation-based fusion, and the pixel-shuffle operation for upsampling.

It can be seen that inclusion of the proposed RBFE in feature alignment stage leads to substantial PSNR boost of 1.02 dB.

This performance gain is further increased by 1.49 dB when we add the proposed burst fusion (NRFE) and upsampling modules.

Overall, when deployed all our modules, we achieve 5.67 dB increment over the baseline.

Further, Table 2 shows that replacing the proposed alignment and fusion methods with other existing techniques causes significant performance drop, i.e., 0.43 dB and 0.34 dB, respectively.

The proposed RBFE module with local-non-local feature extraction ability is a key difference between the existing PCD and our enhanced deformable alignment.

Further, we observe 0.34 dB drop in PSNR when we replace the proposed NRFE (fusion module) with existing compute extensive PBFF #OTHEREFR .

Ablation experiments show that with compute efficient in nature Figure 5 . Ablation experiments for Burstormer contributions.

PSNR is reported on SyntheticBurst dataset #OTHEREFR for 4× SR. All our major components contribute significantly.

As given in Table, our Burstormer achieves 5.67 dB gain over the baseline approach.

Burstormer: Burst Image Restoration And Enhancement Transformer

Abstract: On a shutter press, modern handheld cameras capture multiple images in rapid succession and merge them to generate a single image. However, individual frames in a burst are misaligned due to inevitable motions and contain multiple degradations. The challenge is to properly align the successive image shots and merge their complimentary information to achieve high-quality outputs. Towards this direction, we propose Burstormer: a novel transformer-based architecture for burst image restoration and enhancement. In comparison to existing works, our approach exploits multi-scale local and non-local features to achieve improved alignment and feature fusion. Our key idea is to enable inter-frame communication in the burst neighborhoods for information aggregation and progressive fusion while modeling the burst-wide context. However, the input burst frames need to be properly aligned before fusing their information. Therefore, we propose an enhanced deformable alignment module for aligning burst features with regards to the reference frame. Unlike existing methods, the proposed alignment module not only aligns burst features but also exchanges feature information and maintains focused communication with the reference frame through the proposed referencebased feature enrichment mechanism, which facilitates handling complex motions. After multi-level alignment and enrichment, we re-emphasize on inter-frame communication within burst using a cyclic burst sampling module. Finally, the inter-frame information is aggregated using the proposed burst feature fusion module followed by progressive upsampling. Our Burstormer outperforms state-ofthe-art methods on burst super-resolution, burst denoising and burst low-light enhancement. Our codes and pretrained models are available at https:// github.com/ akshaydudhane16/ Burstormer .

Edvr: Video Restoration With Enhanced Deformable Convolutional Networks

Abstract: Video restoration tasks, including super-resolution, deblurring, etc, are drawing increasing attention in the computer vision community. A challenging benchmark named REDS is released in the NTIRE19 Challenge. This new benchmark challenges existing methods from two aspects: (1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. In this work, we propose a novel Video Restoration framework with Enhanced Deformable networks, termed EDVR, to address these challenges. First, to handle large motions, we devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, we propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration. Thanks to these modules, our EDVR wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. EDVR also demonstrates superior performance to state-of-the-art published methods on video super-resolution and deblurring. The code is available at this https URL.


In a realistic public health scenario, these techniques cannot be used to model and optimize continuous factors such as the number of vaccinations needed and the duration of lockdowns #REFR .

The recent work on the automatic control of epidemics and optimization has focused on the use of either greedy algorithms #OTHEREFR

2020) or reinforcement learning frameworks such as Deep Q-learning variants #OTHEREFR .

Techniques such as Q-learning and greedy optimization can only choose from a predetermined set of intervention strategies (states and actions) e.g.

high vs low vaccination; choose one among n types of masks.

Studies using continuous reinforcement learning frameworks such as Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG, and Proximal Policy Optimization (PPO) #OTHEREFR have been proposed to handle such continuous actions.

However, these studies are limited in the simulation objectives (only one or two of vaccination, lockdown, economy), scale (few thousand individuals), or use equation-based models which are not suited for intervention studies.

This is very important since the use of inappropriate models or optimization algorithms can have unintended and often negative consequences #OTHEREFR .

We overcome these challenges and demonstrate a Deep Deterministic Policy Gradient (DDPG) #OTHEREFR

2017) based policy optimization framework on a large-scale epidemiological simulation where we optimize for multiple objectives (health and economy).

Epidemic Control On A Large-Scale-Agent-Based Epidemiology Model Using Deep Deterministic Policy Gradient

Abstract: To mitigate the impact of the pandemic, several measures include lockdowns, rapid vaccination programs, school closures, and economic stimulus. These interventions can have positive or unintended negative consequences. Current research to model and determine an optimal intervention automatically through roundtripping is limited by the simulation objectives, scale (a few thousand individuals), model types that are not suited for intervention studies, and the number of intervention strategies they can explore (discrete vs continuous). We address these challenges using a Deep Deterministic Policy Gradient (DDPG) based policy optimization framework on a large-scale (100,000 individual) epidemiological agent-based simulation where we perform multi-objective optimization. We determine the optimal policy for lockdown and vaccination in a minimalist age-stratified multi-vaccine scenario with a basic simulation for economic activity. With no lockdown and vaccination (mid-age and elderly), results show optimal economy (individuals below the poverty line) with balanced health objectives (infection, and hospitalization). An in-depth simulation is needed to further validate our results and open-source our framework.

Continuous Control With Deep Reinforcement Learning

Abstract: We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.


To aid with our re-defined tasks we introduce the Semantic Anchors, a combination of the general Word2Vec #REFR semantic space and class centers based on the image representations output by the pre-trained model.

1 (b,c) we graph a preliminary evaluation showing that Sketch-an-Anchor indeed succeeds at this goal.

We've evaluated the DINO S/8 pre-trained model and our own finetuned one at within-domain and cross-domain retrieval, showing the former is already great at ZSIBIR (image domain) while ours maintains that performance at the same time as it significantly moves the needle on withindomain Sketch-based Sketch Retrieval and cross-domain ZSSBIR. The t-SNE projection in Fig.

1 (a) corroborates our finding, showing how the DINO space is discriminative for images but messy for sketches, and then how Sketch-an-Anchor keeps the image relationships while learning better representation for the sketches.

Rethinking the sub-tasks has another remarkable benefit: since we do not need to learn the ZSIBIR sub-task, our model requires only a few iterations, 99.1% less iterations than the average ZSSBIR model to be exact, making it the fastest training ZSSBIR model in the literature.

We then use the Semantic Anchors and their pairwise similarities to design our novel Anchored Contrastive Loss, elevating a strong baseline model to SoTA performance.

To summarize, our contributions are the following:

• Re-think the ZSSBIR solution on matching sketch representations to an existing pre-trained space, since those are already good image retrieval models;

• The ability to train a model for ZSSBIR in under an epoch and achieve performance competitive with SoTA;

• Introduce Semantic Anchors, which learns from visual and word-based semantic spaces with a Graph Convolutional Net and guide the fast model adaptation;

Sketch-An-Anchor: Sub-Epoch Fast Model Adaptation For Zero-Shot Sketch-Based Image Retrieval

Abstract: Sketch-an-Anchor is a novel method to train state-of-theart Zero-shot Sketch-based Image Retrieval (ZSSBIR) models in under an epoch. Most studies break down the problem of ZSSBIR into two parts: domain alignment between images and sketches, inherited from SBIR, and generalization to unseen data, inherent to the zero-shot protocol. We argue one of these problems can be considerably simplified and re-frame the ZSSBIR problem around the alreadystellar yet underexplored Zero-shot Image-based Retrieval performance of off-the-shelf models. Our fast-converging model keeps the single-domain performance while learning to extract similar representations from sketches. To this end we introduce our Semantic Anchors-guiding embeddings learned from word-based semantic spaces and features from off-the-shelf models-and combine them with our novel Anchored Contrastive Loss. Empirical evidence shows we can achieve state-of-the-art performance on all benchmark datasets while training for 100x less iterations than other methods.

Distributed Representations Of Words And Phrases And Their Compositionality

Abstract: The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. ::: ::: An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.

Semantic Anchors
general Word2Vec semantic


Stain normalisation is employed as a preprocessing step in computational pathology algorithms to regulate variances and has a significant impact on the outcome of automated image analysis methods #REFR .

Auxiliary predictions Histology samples stained with H&E often vary in colour between and within laboratories from one batch to the next.

In our work, we employed StainTools #OTHEREFR , using the Vahadane technique to normalize the stain variance in the target image and the Luminosity Standardizer function to standardize the brightness, both of which facilitate consistent interpretation of the images #OTHEREFR .

Evaluation (a) (b) (c) (d) (f) (g) ST Spots Main network predictions (iv) (iii) (ii) (i) (e)

Breast Cancer Histopathology Image-Based Gene Expression Prediction Using Spatial Transcriptomics Data And Deep Learning

Abstract: Tumour heterogeneity in breast cancer poses challenges in predicting outcome and response to therapy. Spatial transcriptomics technologies may address these challenges, as they provide a wealth of information about gene expression at the cell level, but they are expensive, hindering their use in large-scale clinical oncology studies. Predicting gene expression from hematoxylin and eosin stained histology images provides a more affordable alternative for such studies. Here we present BrST-Net, a deep learning framework for predicting gene expression from histopathology images using spatial transcriptomics data. Using this framework, we trained and evaluated 10 state-of-the-art deep learning models without utilizing pretrained weights for the prediction of 250 genes. To enhance the generalisation performance of the main network, we introduce an auxiliary network into the framework. Our methodology outperforms previous studies, with 237 genes identified with positive correlation, including 24 genes with a median correlation coefficient greater than 0.50. This is a notable improvement over previous studies, which could predict only 102 genes with positive correlation, with the highest correlation values ranging from 0.29 to 0.34.

The Importance Of Stain Normalization In Colorectal Tissue Classification With Convolutional Networks

Abstract: The development of reliable imaging biomarkers for the analysis of colorectal cancer (CRC) in hematoxylin and eosin (H&E) stained histopathology images requires an accurate and reproducible classification of the main tissue components in the image. In this paper, we propose a system for CRC tissue classification based on convolutional networks (ConvNets). We investigate the importance of stain normalization in tissue classification of CRC tissue samples in H&E-stained images. Furthermore, we report the performance of ConvNets on a cohort of rectal cancer samples and on an independent publicly available dataset of colorectal H&E images.

computational pathology
Stain normalisation


More specially, for AVs, we will focus on vehicle trajectory tracking control at the microscopic level, while more efforts on collaborative control will be put for CAVs at the macroscopic level #REFR .

AVs and CAVs as the key instantiation of AD and CDA become more and more widely deployed and they benefit both individuals and society #OTHEREFR .

In the past few decades, manufacturers and scholars have been thriving in automotive industrialization to maximize benefits from AD and CDA #OTHEREFR .

The hierarchical framework, including perception, planning, and control modules, is the popular pipeline for both AV and CDA techniques.

One of the most challenging tasks for AD and CDA is to develop safe and efficient vehicle control modules for both AV and CAV #OTHEREFR .

To clarify, vehicle control is to make an AV or CAV follow a desired route or trajectory #OTHEREFR .

Before diving into vehicle control, it is worth mentioning that accurate and reliable vehicle state information is the prerequisite and essential for vehicle control in AVs and CAVs #OTHEREFR .

However, key states such as longitudinal and lateral velocity, sideslip angle, orientation, and tire forces cannot be directly measured for commercial vehicles.

In other words, these vital states can only be estimated indirectly.

Among these states, sideslip angle comprising longitudinal and lateral velocity is the most important state for vehicle stabilization, motion planning, road condition estimation, handover modules, and vehicle navigation #OTHEREFR .

In other words, without an accurate sideslip angle, the performance of the applications mentioned above will be compromised and even sustain failure inevitably.

A Systematic Survey Of Control Techniques And Applications: From Autonomous Vehicles To Connected And Automated Vehicles

Abstract: Vehicle control is one of the most critical challenges in autonomous vehicles (AVs) and connected and automated vehicles (CAVs), and it is paramount in vehicle safety, passenger comfort, transportation efficiency, and energy saving. This survey attempts to provide a comprehensive and thorough overview of the current state of vehicle control technology, focusing on the evolution from vehicle state estimation and trajectory tracking control in AVs at the microscopic level to collaborative control in CAVs at the macroscopic level. First, this review starts with vehicle key state estimation, specifically vehicle sideslip angle, which is the most pivotal state for vehicle trajectory control, to discuss representative approaches. Then, we present symbolic vehicle trajectory tracking control approaches for AVs. On top of that, we further review the collaborative control frameworks for CAVs and corresponding applications. Finally, this survey concludes with a discussion of future research directions and the challenges. This survey aims to provide a contextualized and indepth look at state of the art in vehicle control for AVs and CAVs, identifying critical areas of focus and pointing out the potential areas for further exploration.

Deep Reinforcement Learning For Autonomous Driving: A Survey

Abstract: With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms, provides a taxonomy of automated driving tasks where (D)RL methods have been employed, highlights the key challenges algorithmically as well as in terms of deployment of real world autonomous driving agents, the role of simulators in training agents, and finally methods to evaluate, test and robustifying existing solutions in RL and imitation learning.


To evaluate the efficacy of our method, we performed experiments with ResNet18 #REFR and wide residual network [23] .

In particular, we evaluated ResNet18 on two popular datasets, CIFAR-10 and CIFAR-100 #OTHEREFR and WRN22-8 on CIFAR-100.

We used PyTorch API to define and train our models on an Nvidia RTX 2080 Ti GPU.

To perform stage 1 and 2, we followed the same hyperparameters as #OTHEREFR .

To perform joint ReLU and depth reduction during fine-tuning stage, we initialized a PR model with the weights that provided the best accuracy during the mask evaluation stage (stage 2).

We then trained the model for 180 epochs with starting LR of 0.01 that decayed by a factor of 10 at epochs 90, 140, and 160.

Making Models Shallow Again: Jointly Learning To Reduce Non-Linearity And Depth For Latency-Efficient Private Inference

Abstract: Large number of ReLU and MAC operations of Deep neural networks make them ill-suited for latency and compute-efficient private inference. In this paper, we present a model optimization method that allows a model to learn to be shallow. In particular, we leverage the ReLU sensitivity of a convolutional block to remove a ReLU layer and merge its succeeding and preceding convolution layers to a shallow block. Unlike existing ReLU reduction methods, our joint reduction method can yield models with improved reduction of both ReLUs and linear operations by up to 1.73× and 1.47×, respectively, evaluated with ResNet18 on CIFAR-100 without any significant accuracy-drop.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


For downstream tasks, we evaluate SEMAP on four datasets: CI-FAR10 [1], CIFAR100 [1], F-MNIST #REFR , and STL10 [2].

Models and Datasets. Our target model is pre-trained on ImageNet-1k #OTHEREFR .

We use four different kinds of model structures: ResNet18 (RN18) #OTHEREFR , ResNet50 (RN50) #OTHEREFR , Instagram ResNeXt (Ins) #OTHEREFR , and BiT-M ResNet50 (BiT) #OTHEREFR . ImageNet-21k #OTHEREFR and then fine-tuned on ImageNet-1k.

Our implementation is based on a visual prompt learning framework #OTHEREFR . Baseline Methods.

We compare our methods with the most recent visual prompt learning methods #OTHEREFR , where random mapping and frequency-based mapping are adopted. Therefore, we name the methods as RM-VP and FM-VP.

Further, we compare our methods with traditional fine-tuning, which is denoted as FT.

Note that visual prompt learning and finetuning are two different ways to adapt pre-trained models to downstream tasks.

From Visual Prompt Learning To Zero-Shot Transfer: Mapping Is All You Need

Abstract: Visual prompt learning, as a newly emerged technique, leverages the knowledge learned by a large-scale pre-trained model and adapts it to downstream tasks through the usage of prompts. While previous research has focused on designing effective prompts, in this work, we argue that compared to prompt design, a good mapping strategy matters more. In this sense, we propose SEMAP, a more effective mapping using the semantic alignment between the pre-trained model's knowledge and the downstream task. Our experimental results show that SEMAP can largely boost the performance of visual prompt learning. Moreover, our experiments show that SEMAP is capable of achieving competitive zero-shot transfer, indicating that it can perform the downstream task without any fine-tuning on the corresponding dataset. This demonstrates the potential of our proposed method to be used in a broader range of applications where the zero-shot transfer is desired. Results suggest that our proposed SEMAP could lead to significant advancements in both visual prompt learning and zero-shot transfer. We hope with SEMAP, we can help the community move forward to more efficient and lightweight utilization of large vision models.

Fashion-Mnist: A Novel Image Dataset For Benchmarking Machine Learning Algorithms

Abstract: We present Fashion-MNIST, a new dataset comprising of 28x28 grayscale images of 70,000 fashion products from 10 categories, with 7,000 images per category. The training set has 60,000 images and the test set has 10,000 images. Fashion-MNIST is intended to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms, as it shares the same image size, data format and the structure of training and testing splits. The dataset is freely available at this https URL


Following #REFR we assume that Problem 1 is robustly feasible with minimum clearance ε > 0.

In this section, we provide the probabilistic proof for our algorithm.

Hence, ∃u ∈ U such that it produces a system (1) trajectory φ, where φ : [0, T ] → X safe , φ(0) = x init , φ(T ) = x goal , and

∀t ∈ [0, T ]; Ψ r b (φ(t)) ≥ 0, ∀φ ∈ C 0 ∩ • • • ∩ C r b . Remark 4.

The assumption on the sets of initial state X init and goal state X goal are required because the CBF constraints #OTHEREFR are checked at every time steps during edge extension.

If the assumption does not hold, there's a probability such that certain states x init , x goal will be rejected, and the state trajectory will never reach the goal.

We can ensure the assumption holds by choosing appropriate HOCBF hyperparameters. Lemma 1.

Efficient Lqr-Cbf-Rrt*: Safe And Optimal Motion Planning

Abstract: Control Barrier Functions (CBF) are a powerful tool for designing safety-critical controllers and motion planners. The safety requirements are encoded as a continuously differentiable function that maps from state variables to a real value, in which the sign of its output determines whether safety is violated. In practice, the CBFs can be used to enforce safety by imposing itself as a constraint in a Quadratic Program (QP) solved point-wise in time. However, this approach costs computational resources and could lead to infeasibility in solving the QP. In this paper, we propose a novel motion planning framework that combines samplingbased methods with Linear Quadratic Regulator (LQR) and CBFs. Our approach does not require solving the QPs for control synthesis and avoids explicit collision checking during samplings. Instead, it uses LQR to generate optimal controls and CBF to reject unsafe trajectories. To improve sampling efficiency, we employ the Cross-Entropy Method (CEM) for importance sampling (IS) to sample configurations that will enhance the path with higher probability and store computed optimal gain matrices in a hash table to avoid re-computation during rewiring procedure. We demonstrate the effectiveness of our method on nonlinear control affine systems in simulation.

Sampling-Based Algorithms For Optimal Motion Planning

Abstract: United States. Air Force Office of Scientific Research (Michigan/AFRL Collaborative Center in Control Science Grant FA 8650-07-2-3744)


In frameworks like Pytorch #REFR , the precision of h l , g t are automatically cast into 16-bit precision, and recast to 32-bit when necessary.

In this work, we investigate a combination of the following three quantisation strategies:

1. Gradients and intermediate activations:

have been studied extensively in literature #OTHEREFR .

For instance, in #OTHEREFR the gradients and activations were quantised to 1-bit and still yielded reasonable performance compared to the 32-bit models. However, not all operations are well suited for quantisation.

This technique known as automatic mixed precision (AMP) can yield considerable reduction of memory costs #OTHEREFR . 2.

Optimiser states: Until recently, the main focus of quantisation in deep learning has been on compressing g t , h l , W.

In a recent work in #OTHEREFR , it was shown that the optimiser states, m t , v t can use up to 75% of memory.

By quantising optimiser states to 8-bits using a dynamic quantisation scheme, this work reduces the memory consumption during model training. 3.

Model weights: The resolution of the hypothesis space is controlled by the precision of the weights, W.

Operating Critical Machine Learning Models In Resource Constrained Regimes

Abstract: The accelerated development of machine learning methods, primarily deep learning, are causal to the recent breakthroughs in medical image analysis and computer aided intervention. The resource consumption of deep learning models in terms of amount of training data, compute and energy costs are known to be massive. These large resource costs can be barriers in deploying these models in clinics, globally. To address this, there are cogent efforts within the machine learning community to introduce notions of resource efficiency. For instance, using quantisation to alleviate memory consumption. While most of these methods are shown to reduce the resource utilisation, they could come at a cost in performance. In this work, we probe into the trade-off between resource consumption and performance, specifically, when dealing with models that are used in critical settings such as in clinics.

Pytorch: An Imperative Style, High-Performance Deep Learning Library

Abstract: Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. ::: In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. ::: We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.


We have also tried using positional encoding with ReLU, like what is used in NeRF #REFR . The result is shown in table 11.

It can be seen that joint training achieves significantly better performance than training on a single corruption across all three tasks, suggesting the effectiveness of the LINR.

For images containing noise in all joint-training experiments, the training weight is set to 0.1.

This aims to reduce the misleading effect of this type of corruption. For all other images, the weight is 1.

Our work is based on SIREN #OTHEREFR , which uses sine as the activation function.

We also replace it with other activation functions for super-resolution tasks as an ablation study here. We have tried Tanh, Sigmoid, ReLU, etc.

With other activation functions, INR performed poorly on SR with the same number of training iterations(500).

SIREN can represent high-quality images in a very short time, whereas the other activation functions can not.

If given more training iterations, the performance of ReLU-PE can increase significantly (with an average PSNR of 26.23 on Set5 after 2000 iterations), but is still much worse than the result of SIREN (because ReLU cannot model higher-order derivatives and also lack the ability to extract enough global information #OTHEREFR .) However, it is significantly better than the result of Nearest (24.50) , which shows that with other activation functions, INR also has the capacity to restore high-quality images.

Revisiting Implicit Neural Representations In Low-Level Vision

Abstract: Implicit Neural Representation (INR) has been emerging in computer vision in recent years. It has been shown to be effective in parameterising continuous signals such as dense 3D models from discrete image data, e.g. the neural radius field (NeRF). However, INR is under-explored in 2D image processing tasks. Considering the basic definition and the structure of INR, we are interested in its effectiveness in low-level vision problems such as image restoration. In this work, we revisit INR and investigate its application in low-level image restoration tasks including image denoising, super-resolution, inpainting, and deblurring. Extensive experimental evaluations suggest the superior performance of INR in several low-level vision tasks with limited resources, outperforming its counterparts by over 2dB. Code and models are available at: https://github.com/WenTXuL/LINR.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


Characteristics Of In-The-Wild Har Datasets

To characterize the distribution of noisy labels, a noise transition matrix is often used, where element corresponds to the probability of mislabeling a data sample with ground truth label to label #REFR .

1 , we conclude ExtraSensory is much noisier than RealWorld since the former is crowdsourced data.

What also distinguishes the two datasets is the distribution of label noises.

Specifically, for RealWorld, most mislabeling happens in the climbing up/down trials when the ground labels are "walk on a flat ground" or standing.

In contrast, in ExtraSensory, mislabeling exists almost between any two activities.

When mislabels occur equally likely for all classes other than the true class, the associate noise pattern is called symmetric noise.

Otherwise, if there is a dominant off-diagonal element in each row in , the associate noise pattern is called asymmetric noise.

Table 1 shows the noise transition matrix of data in three locomotion classes and one location class in ExtraSensory by comparing their curated labels (row headings) and the original ones (column headings).

As ExtraSensory is a multi-label dataset with many classes, only top-5 mutually exclusive labels are included in the table.

We observe that with the exception of "running", noise transition probabilities of all classes are best modeled as symmetric noise.

Valerian: Invariant Feature Learning For Imu Sensor-Based Human Activity Recognition In The Wild

Abstract: Deep neural network models for IMU sensor-based human activity recognition (HAR) that are trained from controlled, well-curated datasets suffer from poor generalizability in practical deployments. However, data collected from naturalistic settings often contains significant label noise. In this work, we examine two in-the-wild HAR datasets and DivideMix, a state-of-the-art learning with noise labels (LNL) method to understand the extent and impacts of noisy labels in training data. Our empirical analysis reveals that the substantial domain gaps among diverse subjects cause LNL methods to violate a key underlying assumption, namely, neural networks tend to fit simpler (and thus clean) data in early training epochs. Motivated by the insights, we design VALERIAN, an invariant feature learning method for in-the-wild wearable sensor-based HAR. By training a multi-task model with separate task-specific layers for each subject, VALERIAN allows noisy labels to be dealt with individually while benefiting from shared feature representation across subjects. We evaluated VALERIAN on four datasets, two collected in a controlled environment and two in the wild. Experimental results show that VALERIAN significantly outperforms baseline approaches. VALERIAN can correct 75%-93% of label errors in the source domains. When only 10-second clean labeled data per class is available from a new target subject, even with 40% label noise in training data, it achieves ∼ 83% test accuracy.

Co-Teaching: Robust Training Of Deep Neural Networks With Extremely Noisy Labels

Abstract: Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called Co-teaching for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.


Yolo is faster than Faster RCNN, but its improved modules introduced in #REFR bring more parameters.

This suggests that the two detectors can identify the streaks properly but the detected bounding boxes may not be always accurate.

Instead, the Faster RCNN performs well for both AP@0.3 and AP@0.5, showing that this detector is more consistent than the other two detectors and is less likely to produce inaccurate detection boxes.

This is easy to understand because the Faster RCNN has a specific classification head that can clearly identify partially aligned less accurate bounding boxes.

Regarding the overall performance for AP@[0.3:0.95], the Faster RCNN shows much better consistency than the other two detectors, which further validates its superior robustness.

In addition to the accuracy, if considering complexity like GFLOPs, parameters, and inference time, it shows that the Faster RCNN is the heaviest method with the highest GFLOPs, the highest number of parameters, and the longest processing time, while the UNet-SD is the most lightweighted.

To sum up, for the RSO streak detection task, our benchmarking results illustrate that the Faster RCNN is the most robust and accurate method but it has a higher cost.

Our developed UNet-SD method is very efficient and also produces promising performance if not pursuing the highest accuracy.

Yolo is roughly in the middle of the two methods, but its accuracy in the standard of AP@0.5 is greatly affected by the low-resolution of input images and small sizes of streaks.

A Simulation-Augmented Benchmarking Framework For Automatic Rso Streak Detection In Single-Frame Space Images

Abstract: Effective space-based Resident Space Objects (RSOs) detection provides a solution to avoiding potential impact or collision of space debris with other satellites. This is achieved by providing timely observations that help track and infer the behaviours of targets of interest. Recently, deep convolutional neural networks (DCNNs) have shown superior performance in object detection compared to traditional methods when large-scale datasets are available to train their models. However, collecting rich, diversified, and meaningful data of RSOs is difficult due to very few occurrences in the space images. Without sufficient data, it is challenging to comprehensively train DCNN detectors and make them effective for detecting RSOs in space images, let alone to evaluate and estimate whether a detector has sufficient capacity to detect RSOs robustly. The lack of meaningful evaluation of different detectors could further affect the design and application of detection methods. To tackle this issue, we propose that the space images containing RSOs can be simulated to complement the shortage of raw data for better benchmarking. Accordingly, we introduce a novel simulation-augmented benchmarking framework for RSO detection (SAB-RSOD). In our framework, by making the best use of the hardware parameters of the sensor that captures real-world space images, we first develop a high-fidelity RSO simulator that can generate various realistic space images to enrich the dataset for training detectors. Then, we use this simulator to generate images that contain diversified RSOs in space and annotate them automatically. Later, we mix the synthetic images with the available real-world images, obtaining around 500 images for training detectors with only the real-world images for evaluation. Under SAB-RSOD, we can train different popular object detectors like Yolo[1] and Faster RCNN[2] effectively, enabling us to evaluate their performance thoroughly. The evaluation results have shown that the amount of available data and image resolution are two important factors for robust RSO detection. Moreover, if using a lower resolution for higher efficiency, we demonstrated that a simple UNet-based[3] detection method can already access high-accuracy RSO detection, providing new insights to the community. The dataset and evaluation code will be released.

Yolov3: An Incremental Improvement

Abstract: We present some updates to YOLO! We made a bunch of little design changes to make it better. We also trained this new network that's pretty swell. It's a little bigger than last time but more accurate. It's still fast though, don't worry. At 320x320 YOLOv3 runs in 22 ms at 28.2 mAP, as accurate as SSD but three times faster. When we look at the old .5 IOU mAP detection metric YOLOv3 is quite good. It achieves 57.9 mAP@50 in 51 ms on a Titan X, compared to 57.5 mAP@50 in 198 ms by RetinaNet, similar performance but 3.8x faster. As always, all the code is online at this https URL


Recent Advances Based On Neural Networks

For extracting nonlinear dynamics, a method with Minimum Predictive Information Regularization (MPIR) #REFR is introduced.

A feature selection procedure termed as a non-uniform embedding (NUE) is proposed in NN-GC #OTHEREFR to identify the significant Granger causes in the MLP model.

By greedily adding lagged components of predictor time series as input, an MLP is updated iteratively.

A predictor time series is claimed a significant Granger cause of the target time series if at least one of its lagged components is added when the procedure is terminated.

In RNN-GC #OTHEREFR , the NUE is extended by replacing MLPs with gated RNN models, However, as this technique requires training and comparing many candidate models, it's costly in high-dimensional settings.

DL-extensions with Information Regularization.

It leverages learnable corruption for predictor variables and minimizes a mutual information-regularized risk, which combines the benefits of the Granger causality paradigm with deep learning models.

In MPIR, the author states that the naive way to combine neural nets with Granger causality suffers from two major drawbacks: instability and inefficiency.

The solution is to encourage each x t−K:t−1 i to provide as little information to x t j as possible while maintaining good prediction via learned corruption, replacing the naive way which predicts x t j with one x t−K:t−1 i missing at a time. The risk is defined as follows:

R X,xj [f θ , n] = E X t−1 ,x t j ,u [(x t j − f θ (X t−K:t−1 (n) )) 2 ] + λ • d p=1 I(X t−K:t−1 i(n) ; X t−K:t−1 i ), whereX t−K:t−1 (n)

:= X t−K:t−1 + n e (or its element-wise representation,X t−K:t−1

Causal Discovery From Temporal Data: An Overview And New Perspectives

Abstract: Temporal data, representing chronological observations of complex systems, has always been a typical data structure that can be widely generated by many domains, such as industry, medicine and finance. Analyzing this type of data is extremely valuable for various applications. Thus, different temporal data analysis tasks, e.g., classification, clustering and prediction, have been proposed in the past decades. Among them, causal discovery, learning the causal relations from temporal data, is considered an interesting yet critical task and has attracted much research attention. Existing casual discovery works can be divided into two highly correlated categories according to whether the temporal data is calibrated, i.e., multivariate time series casual discovery, and event sequence casual discovery. However, most previous surveys are only focused on the time series casual discovery and ignore the second category. In this paper, we specify the correlation between the two categories and provide a systematical overview of existing solutions. Furthermore, we provide public datasets, evaluation metrics and new perspectives for temporal data casual discovery. Keywords Causal discovery • Temporal data analysis • Relational learning * Entries correspond to methods reviewed which are mainly for non-temporal settings. † Mainly about causalities related to the Hawkes process.

Discovering Nonlinear Relations With Minimum Predictive Information Regularization

Abstract: Identifying the underlying directional relations from observational time series with nonlinear interactions and complex relational structures is key to a wide range of applications, yet remains a hard problem. In this work, we introduce a novel minimum predictive information regularization method to infer directional relations from time series, allowing deep learning models to discover nonlinear relations. Our method substantially outperforms other methods for learning nonlinear relations in synthetic datasets, and discovers the directional relations in a video game environment and a heart-rate vs. breath-rate dataset.


Besides, all the experiments are conducted on evolutionary multi-objective optimization platform PlatEMO 2 #REFR .

In this section, the performance of the proposed CLMEA is first investigated regarding the efficacy of classifier-assisted rank-based learning pre-screening, HV-based non-dominated search, and local search in sparse object space.

CLMEA is also compared with five state-of-the-art algorithms (CPS-MOEA #OTHEREFR , K-RVEA #OTHEREFR , CSEA #OTHEREFR , END-ARMOEA #OTHEREFR , and MCEA/D #OTHEREFR ) on DTLZ and ZDT benchmark suites (detailed characteristics of the benchmark problems in Table S-I in the supplementary material).

The source code of CLMEA is publicly available in MATLAB to help readers reproduce the results #OTHEREFR .

In addition, the computational complexity of the algorithms is analyzed and compared.

Finally, a real-world heat extraction optimization of geothermal reservoir is also employed to further test the performance of the proposed CLMEA.

Rank-Based Learning And Local Model Based Evolutionary Algorithm For High-Dimensional Expensive Multi-Objective Problems

Abstract: Surrogate-assisted evolutionary algorithms have been widely developed to solve complex and computationally expensive multi-objective optimization problems in recent years. However, when dealing with high-dimensional optimization problems, the performance of these surrogate-assisted multiobjective evolutionary algorithms deteriorate drastically. In this work, a novel Classifier-assisted rank-based learning and Local Model based multi-objective Evolutionary Algorithm (CLMEA) is proposed for high-dimensional expensive multi-objective optimization problems. The proposed algorithm consists of three parts: classifier-assisted rank-based learning, hypervolume-based non-dominated search, and local search in the relatively sparse objective space. Specifically, a probabilistic neural network is built as classifier to divide the offspring into a number of ranks. The offspring in different ranks uses rank-based learning strategy to generate more promising and informative candidates for real function evaluations. Then, radial basis function networks are built as surrogates to approximate the objective functions. After searching non-dominated solutions assisted by the surrogate model, the candidates with higher hypervolume improvement are selected for real evaluations. Subsequently, in order to maintain the diversity of solutions, the most uncertain sample point from the non-dominated solutions measured by the crowding distance is selected as the guided parent to further infill in the uncertain region of the front. The experimental results of benchmark problems and a real-world application on geothermal reservoir heat extraction optimization demonstrate that the proposed algorithm shows superior performance compared with the stateof-the-art surrogate-assisted multi-objective evolutionary algorithms. The source code for this work is available at https://github.com/JellyChen7/CLMEA.

Platemo: A Matlab Platform For Evolutionary Multi-Objective Optimization

Abstract: Over the last three decades, a large number of evolutionary algorithms have been developed for solving multi-objective optimization problems. However, there lacks an upto-date and comprehensive software platform for researchers to properly benchmark existing algorithms and for practitioners to apply selected algorithms to solve their real-world problems. The demand of such a common tool becomes even more urgent, when the source code of many proposed algorithms has not been made publicly available. To address these issues, we have developed a MATLAB platform for evolutionary multi-objective optimization in this paper, called PlatEMO, which includes more than 50 multiobjective evolutionary algorithms and more than 100 multi-objective test problems, along with several widely used performance indicators. With a user-friendly graphical user interface, PlatEMO enables users to easily compare several evolutionary algorithms at one time and collect statistical results in Excel or LaTeX files. More importantly, PlatEMO is completely open source, such that users are able to develop new algorithms on the basis of it. This paper introduces the main features of PlatEMO and illustrates how to use it for performing comparative experiments, embedding new algorithms, creating new test problems, and developing performance indicators. Source code of PlatEMO is now available at: http://bimk.ahu.edu.cn/index.php?s=/Index/Software/index.html.


Most of the common NN architectures employ Batch Normalization (BN) #REFR , a technique for improving the training of NNs to make them converge faster and to a more stable solution.

Unfortunately, real-world data is often non-IID, and it is well known that this scenario poses critical issues to FL #OTHEREFR .

In a non-IID setting, the data statistics of a single client may be unrepresentative of the global statistics and make the model diverge from the intended solution. Interestingly, Huang at al.

show that if the loss surface of the optimization problem is both smooth and convex (which is hardly true in a real scenario), then FedAvg will also converge when the data is non-IID #OTHEREFR .

Recent works have proposed several FL algorithms to cope with non-IIDness problems, such as FedProx #OTHEREFR , FedNova #OTHEREFR , SCAFFOLD #OTHEREFR , and FedCurv #OTHEREFR , which has been tested in #OTHEREFR , #OTHEREFR .

Notice that all these algorithms are modified versions of FedAvg and they preserve the principle underneat FedAvg: to average the weights in all the layers of the NN.

BN works by standardizing the layers' input for each mini-batch.

In this work, we investigate two aspects of the training FL models which, differently from cenytralized case, they happen to be hyper-parameters that can be optimized: the normalization layers and the frequency of models aggregation (epochs per round).

We show that the most popular normalization layer (BN) does not couple well with FL for non-IID data and that substituting BN with alternative normalization FL a better model can be produced for both the non-IID and IID case.

We also show that building a global model aggregating local models at each epoch is not a good strategy, neither for quality of the model nor for the execution time.

We experiment with two network architectures and five different normalization layers on two public image datasets: MNIST #OTHEREFR and CIFAR-10 #OTHEREFR .

Experimenting With Normalization Layers In Federated Learning On Non-Iid Scenarios

Abstract: Training Deep Learning (DL) models require large, high-quality datasets, often assembled with data from different institutions. Federated Learning (FL) has been emerging as a method for privacy-preserving pooling of datasets employing collaborative training from different institutions by iteratively globally aggregating locally trained models. One critical performance challenge of FL is operating on datasets not independently and identically distributed (non-IID) among the federation participants. Even though this fragility cannot be eliminated, it can be debunked by a suitable optimization of two hyperparameters: layer normalization methods and collaboration frequency selection. In this work, we benchmark five different normalization layers for training Neural Networks (NNs), two families of non-IID data skew, and two datasets. Results show that Batch Normalization, widely employed for centralized DL, is not the best choice for FL, whereas Group and Layer Normalization consistently outperform Batch Normalization. Similarly, frequent model aggregation decreases convergence speed and mode quality.

Batch Normalization: Accelerating Deep Network Training By Reducing Internal Covariate Shift

Abstract: Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.


With the same number of trainable parameters, EfficientNet neural networks show the highest accuracy compared to other neural network architectures in the ImageNet #REFR classification problem.

The problem of extracting the watermark w from the image I w can be interpreted as a multilabel image classification problem.

Thus, if the bit of the watermark w i is equal to 1, it means that image belongs to the class with label i.

Neural network architectures specialized on classification problems can be used as extraction neural network D w .

In the implementation of the proposed method, the EfficientNet-B2 #OTHEREFR architecture has been chosen.

The EfficientNet architecture class was obtained using the NAS (Neural Architecture Search) method.

Docmarking: Real-Time Screen-Cam Robust Document Image Watermarking

Abstract: This paper focuses on investigation of confidential documents leaks in the form of screen photographs. Proposed approach does not try to prevent leak in the first place but rather aims to determine source of the leak. Method works by applying on the screen a unique identifying watermark as semi-transparent image that is almost imperceptible for human eyes. Watermark image is static and stays on the screen all the time thus watermark present on every captured photograph of the screen. The key components of the approach are three neural networks. The first network generates an image with embedded message in a way that this image is almost invisible when displayed on the screen. The other two neural networks are used to retrieve embedded message with high accuracy. Developed method was comprehensively tested on different screen and cameras. Test results showed high efficiency of the proposed approach. Index Terms-document leakage investigation, screen-cam robust watermarking, blind watermarking I. INTRODUCTION

Imagenet Large Scale Visual Recognition Challenge

Abstract: The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements.

ImageNet classification problem
EfficientNet neural networks


In #REFR , after candidate constraint sets are constructed, the Lagrange multipliers of the IO problem are used to infer constraints.

Constraints can also be modeled in the form of control barrier functions.

The authors in #OTHEREFR , use expert demonstrations to learn control barrier functions that enjoy provable safety guarantees.

Moreover, the authors in #OTHEREFR infer parameters in the objective function utilized to obtain control policies in constraint environments.

However, it should be noted that unknown parameters exist only in the objective function and not in the constraints.

The KKT optimality conditions of an optimization problem can also be used to infer constraint sets.

The closest work to our framework is #OTHEREFR , in which the authors use the KKT conditions to construct safe and unsafe areas of the state-action space, and hence implicitly the constraints themselves.

Although this approach can also infer uncertainty in the objective function, we choose to focus solely on the constraint recovery task.

Inverse Optimization: Inverse optimization, broadly refers to the problem of inferring parameters of an optimization problem by observing the optimal solution #OTHEREFR .

The majority of the work revolves around estimating parameters in the objective function.

For linear objective functions and constraints, #OTHEREFR formulate the inverse optimization problem also as a linear problem.

Constraint Inference In Control Tasks From Expert Demonstrations Via Inverse Optimization

Abstract: Inferring unknown constraints is a challenging and crucial problem in many robotics applications. When only expert demonstrations are available, it becomes essential to infer the unknown domain constraints to deploy additional agents effectively. In this work, we propose an approach to infer affine constraints in control tasks after observing expert demonstrations. We formulate the constraint inference problem as an inverse optimization problem, and we propose an alternating optimization scheme that infers the unknown constraints by minimizing a KKT residual objective. We demonstrate the effectiveness of our method in a number of simulations, and show that our method can infer less conservative constraints than a recent baseline method while maintaining comparable safety guarantees.

Constrained Inverse Optimal Control With Application To A Human Manipulation Task

Abstract: This brief presents an inverse optimal control methodology and its application to training a predictive model of human motor control from a manipulation task. It introduces a convex formulation for learning both objective function and constraints of an infinite-horizon constrained optimal control problem with nonlinear system dynamics. The inverse approach utilizes Bellman's principle of optimality to formulate the infinite-horizon optimal control problem as a shortest path problem and the Lagrange multipliers to identify constraints. We highlight the key benefit of using the shortest path formulation, i.e., the possibility of training the predictive model with short and selected trajectory segments. The method is applied to training a predictive model of movements of a human subject from a manipulation task. The study indicates that individual human movements can be predicted with low error using an infinite-horizon optimal control problem with constraints on the shoulder movement.


Similarly, CoverNet #REFR uses CNN to encode the scene context while it proposes dozens of trajectory anchors based on the bicycle model and converts prediction from position regression to anchor classification.

In this section, we first briefly review the general frameworks of learning-based trajectory prediction and then introduce the literature closely related to our approach. General trajectory prediction architecture.

Trajectory prediction for moving agents is important for the safe planning of autonomous driving #OTHEREFR .

One branch of trajectory prediction consists of the approaches #OTHEREFR based on typical neural networks (CNN, GNN #OTHEREFR , RNN #OTHEREFR ). Cui et al.

#OTHEREFR propose to use agents' states and features extracted by CNN from Bird-Eye-View images to generate a multimodal prediction.

A hierarchical GNN named VectorNet #OTHEREFR encapsulates the sequential features of map elements and past trajectories with instancewise subgraphs and models interactions with a global graph.

TNT #OTHEREFR and DenseTNT #OTHEREFR are extensions of #OTHEREFR that focus on predicting reasonable destinations.

Social LSTM #OTHEREFR models the trajectories of individual agents from separate LSTM networks and aggregates the LSTM hidden cues to model their interactions.

CL-SGR #OTHEREFR considers the sample replay model in a continuous trajectory prediction scenario setting to avoid catastrophic forgetting.

The other branch #OTHEREFR models the interaction among the agents based on the attention mechanism.

Ipcc-Tp: Utilizing Incremental Pearson Correlation Coefficient For Joint Multi-Agent Trajectory Prediction

Abstract: Reliable multi-agent trajectory prediction is crucial for the safe planning and control of autonomous systems. Compared with single-agent cases, the major challenge in simultaneously processing multiple agents lies in modeling complex social interactions caused by various driving intentions and road conditions. Previous methods typically leverage graph-based message propagation or attention mechanism to encapsulate such interactions in the format of marginal probabilistic distributions. However, it is inherently sub-optimal. In this paper, we propose IPCC-TP, a novel relevance-aware module based on Incremental Pearson Correlation Coefficient to improve multi-agent interaction modeling. IPCC-TP learns pairwise joint Gaussian Distributions through the tightly-coupled estimation of the means and covariances according to interactive incremental movements. Our module can be conveniently embedded into existing multi-agent prediction methods to extend original motion distribution decoders. Extensive experiments on nuScenes and Argoverse 2 datasets demonstrate that IPCC-TP improves the performance of baselines by a large margin.

Covernet: Multimodal Behavior Prediction Using Trajectory Sets

Abstract: We present CoverNet, a new method for multimodal, probabilistic trajectory prediction for urban driving. Previous work has employed a variety of methods, including multimodal regression, occupancy maps, and 1-step stochastic policies. We instead frame the trajectory prediction problem as classification over a diverse set of trajectories. The size of this set remains manageable due to the limited number of distinct actions that can be taken over a reasonable prediction horizon. We structure the trajectory set to a) ensure a desired level of coverage of the state space, and b) eliminate physically impossible trajectories. By dynamically generating trajectory sets based on the agent's current state, we can further improve our method's efficiency. We demonstrate our approach on public, real-world self-driving datasets, and show that it outperforms state-of-the-art methods.