The architectures are inspired from the work of #REFR and are presented in a generic fashion in Table 2 .

In this work, we use again the implementation provided by the authors of E(2)-CNNs, who re-implement HNets in their own framework, for convenience.

• Regarding our B-CNNs, four setups are considered to achieve SO(2) or O(2), with or without scale invariance (denoted by the presence or not of "+" in our tables and figures), with the computation of k max as described by Equation (4.7).

Another setup for SO(2) invariance with a stronger cutoff frequency, that corresponds to half the initial k max , is also considered.

This last setup is motivated by the empirical observation that it often leads to better performances.

• Finally, a vanilla CNN with the same architecture than for the other methods, as well as a ResNet-18 #OTHEREFR are also trained for reference.

Note that the size of the filters is larger than conventional sizes in CNNs.

The reason why it is preferable to increase the size of the filters in those cases is explained in Section 6.3.

The same template architecture is used for all the methods (except for the ResNet-18 architecture that is kept unmodified) and data sets. Nonetheless, minor modifications are sometimes performed.

Firstly, the number of filters in each convolutional layer should be adapted from one method to another, in order to keep the same number of trainable parameters.

To do so, a parameter λ is introduced to manually scale the number of filters and guarantee the same number of trainable parameters for all the methods.

So(2) And O(2) Equivariance In Image Recognition With Bessel-Convolutional Neural Networks

Abstract: For many years, it has been shown how much exploiting equivariances can be beneficial when solving image analysis tasks. For example, the superiority of convolutional neural networks (CNNs) compared to dense networks mainly comes from an elegant exploitation of the translation equivariance. Patterns can appear at arbitrary positions and convolutions take this into account to achieve translation invariant operations through weight sharing. Nevertheless, images often involve other symmetries that can also be exploited. It is the case of rotations and reflections that have drawn particular attention and led to the development of multiple equivariant CNN architectures. Among all these methods, Bessel-convolutional neural networks (B-CNNs) exploit a particular decomposition based on Bessel functions to modify the key operation between images and filters and make it by design equivariant to all the continuous set of planar rotations. In this work, the mathematical developments of B-CNNs are presented along with several improvements, including the incorporation of reflection and multi-scale equivariances. Extensive study is carried out to assess the performances of B-CNNs compared to other methods. Finally, we emphasize the theoretical advantages of B-CNNs by giving more insights and in-depth mathematical details.

General $E(2)$-Equivariant Steerable Cnns

Abstract: The big empirical success of group equivariant networks has led in recent years to the sprouting of a great variety of equivariant network architectures. A particular focus has thereby been on rotation and reflection equivariant CNNs for planar images. Here we give a general description of $E(2)$-equivariant convolutions in the framework of Steerable CNNs. The theory of Steerable CNNs thereby yields constraints on the convolution kernels which depend on group representations describing the transformation laws of feature spaces. We show that these constraints for arbitrary group representations can be reduced to constraints under irreducible representations. A general solution of the kernel space constraint is given for arbitrary representations of the Euclidean group $E(2)$ and its subgroups. We implement a wide range of previously proposed and entirely new equivariant network architectures and extensively compare their performances. $E(2)$-steerable convolutions are further shown to yield remarkable gains on CIFAR-10, CIFAR-100 and STL-10 when used as a drop-in replacement for non-equivariant convolutions.


We compare our method with the existing dynamic pose estimation networks, i.e., DKD #REFR and Skip-Convolution [12] .

A trace of models' accuracy and delay for online videobased action recognition (left) and pose estimation (right).

In the areas masked by the gray color, we can observe a rapid increase of the baseline model's delay (see the second row), which corresponds to the high-load status of the system.

Here, we consider three (i.e., m = 3) resolution candidates: [128, 192, 256] and follow the work of #OTHEREFR to use 3 deconvolutional layers to build the task head for pose estimation.

Moreover, we replace LSTM with a ConvL-STM #OTHEREFR to accommodate for 2D feature maps.

Similarly, we re-implement DKD and Skip-Convolutions as there is no publicly available code. Experiment settings.

To build dynamic systems with varying system status, we design multiple background processes (i.e., matrix calculators, video compressors, and large deep learning models) to occupy various amount of computational resources.

We can then generate dynamic system load trajectories using these processes to simulate dynamic system load environments.

During model training, we randomly generate dynamic system trajectories in each iteration to simulate various dynamic environments to train our SAN.

During testing, for fair comparisons, we first generate 3 dynamic trajectories unseen in training, and then run all the evaluation experiments on these 3 fixed trajectories and report the average result.

System-Status-Aware Adaptive Network For Online Streaming Video Understanding

Abstract: Recent years have witnessed great progress in deep neural networks for real-time applications. However, most existing works do not explicitly consider the general case where the device's state and the available resources fluctuate over time, and none of them investigate or address the impact of varying computational resources for online video understanding tasks. This paper proposes a Systemstatus-aware Adaptive Network (SAN) that considers the device's real-time state to provide high-quality predictions with low delay. Usage of our agent's policy improves efficiency and robustness to fluctuations of the system status. On two widely used video understanding tasks, SAN obtains state-of-the-art performance while constantly keeping processing delays low. Moreover, training such an agent on various types of hardware configurations is not easy as the labeled training data might not be available, or can be computationally prohibitive. To address this challenging problem, we propose a Meta Self-supervised Adaptation (MSA) method that adapts the agent's policy to new hardware configurations at test-time, allowing for easy deployment of the model onto other unseen hardware platforms.

Dynamic Kernel Distillation For Efficient Pose Estimation In Videos

Abstract: Existing video-based human pose estimation methods extensively apply large networks onto every frame in the video to localize body joints, which suffer high computational cost and hardly meet the low-latency requirement in realistic applications. To address this issue, we propose a novel Dynamic Kernel Distillation (DKD) model to facilitate small networks for estimating human poses in videos, thus significantly lifting the efficiency. In particular, DKD introduces a light-weight distillator to online distill pose kernels via leveraging temporal cues from the previous frame in a one-shot feed-forward manner. Then, DKD simplifies body joint localization into a matching procedure between the pose kernels and the current frame, which can be efficiently computed via simple convolution. In this way, DKD fast transfers pose knowledge from one frame to provide compact guidance for body joint localization in the following frame, which enables utilization of small networks in video-based pose estimation. To facilitate the training process, DKD exploits a temporally adversarial training strategy that introduces a temporal discriminator to help generate temporally coherent pose kernels and pose estimation results within a long range. Experiments on Penn Action and Sub-JHMDB benchmarks demonstrate outperforming efficiency of DKD, specifically, 10x flops reduction and 2x speedup over previous best model, and its state-of-the-art accuracy.


In this project, we have successfully replicated the results from the original paper #REFR in terms of extracting the 3D bounding boxes from a single view for three different dataset categories.

We then extended the 3D bounding box accuracies, using the light-weight MobileNet-v2 and EfficientNet-v2 feature extractors, and later a similar performance boost is also observed with our modified multibin architecture.

As a result, we conclude that MobileNet-v2 and EfficientNet-v2 outperform VGG-19 architecture in 3D bounding box estimation, from a single view of an RGB image.

Oricon3D: Effective 3D Object Detection Using Orientation And Confidence

Abstract: We introduce a technique for detecting 3D objects and estimating their position from a single image. Our method is built on top of a similar state-of-the-art technique [1], but with improved accuracy. The approach followed in this research first estimates common 3D properties of an object using a Deep Convolutional Neural Network (DCNN), contrary to other frameworks that only leverage centre-point predictions. We then combine these estimates with geometric constraints provided by a 2D bounding box to produce a complete 3D bounding box. The first output of our network estimates the 3D object orientation using a discrete-continuous loss [1]. The second output predicts the 3D object dimensions with minimal variance. Here we also present our extensions by augmenting lightweight feature extractors and a customized multibin architecture. By combining these estimates with the geometric constraints of the 2D bounding box, we can accurately (or comparatively) determine the 3D object pose better than our baseline [1] on the KITTI 3D detection benchmark [2].

3D Bounding Box Estimation Using Deep Learning And Geometry

Abstract: We present a method for 3D object detection and pose estimation from a single image. In contrast to current techniques that only regress the 3D orientation of an object, our method first regresses relatively stable 3D object properties using a deep convolutional neural network and then combines these estimates with geometric constraints provided by a 2D object bounding box to produce a complete 3D bounding box. The first network output estimates the 3D object orientation using a novel hybrid discrete-continuous loss, which significantly outperforms the L2 loss. The second output regresses the 3D object dimensions, which have relatively little variance compared to alternatives and can often be predicted for many object types. These estimates, combined with the geometric constraints on translation imposed by the 2D bounding box, enable us to recover a stable and accurate 3D object pose. We evaluate our method on the challenging KITTI object detection benchmark [2] both on the official metric of 3D orientation estimation and also on the accuracy of the obtained 3D bounding boxes. Although conceptually simple, our method outperforms more complex and computationally expensive approaches that leverage semantic segmentation, instance level segmentation and flat ground priors [4] and sub-category detection [23][24]. Our discrete-continuous loss also produces state of the art results for 3D viewpoint estimation on the Pascal 3D+ dataset[26].


When the model is updated to learn the current task, we observe catastrophic forgetting #REFR in the model due to deviation from the optimal parameters for the past learned tasks.

The reconstruction prior used in VAEs is local low-variance Bournourli distribution.

This requirement can be relaxed to more global distributions seen in GAN [7] , or distributed throughout the full model as in diffusion models #OTHEREFR .

Our method employs ideas of (i) and (iii).

When used on a unimodal Gaussian VAE, it is a GMVAE with 2 d components, with deterministic assignment conditioned on the approximate posterior, and unimodal Gaussian reparameterization to distributed the mass in the latent embedding.

Continual Learning aims to learn a set of sequentially arriving tasks using a shared model.

CL algorithms employ three primary strategies to mitigate catastrophic forgetting.

One approach involves regularizing a fixed shared model, such that different information pathways are used to learn each task's weights #OTHEREFR .

The aim is to identify critical model parameters that encode the learned knowledge of each task and consolidate these parameters while updating the model to learn new tasks.

The downside is that the learning capacity of the model is compromised as more weights are consolidated.

Another approach relies on model expansion #OTHEREFR , which involves adding new weights to a base model and customizing the network to learn new tasks via these additional weights.

Encoding Binary Concepts In The Latent Space Of Generative Models For Enhancing Data Representation

Abstract: Binary concepts 1 are empirically used by humans to generalize efficiently. And they are based on Bernoulli distribution which is the building block of information. These concepts span both low-level and high-level features such as "large vs small" and "a neuron is active or inactive". Binary concepts are ubiquitous features and can be used to transfer knowledge to improve model generalization. We propose a novel binarized regularization to facilitate learning of binary concepts to improve the quality of data generation in autoencoders. We introduce a binarizing hyperparameter r in data generation process to disentangle the latent space symmetrically. We demonstrate that this method can be applied easily to existing variational autoencoder (VAE) variants to encourage symmetric disentanglement, improve reconstruction quality, and prevent posterior collapse without computation overhead. We also demonstrate that this method can boost existing models to learn more transferable representations and generate more representative samples for the input distribution which can alleviate catastrophic forgetting using generative replay under continual learning settings.

Overcoming Catastrophic Forgetting In Neural Networks

Abstract: The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks that they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on a hand-written digit dataset and by learning several Atari 2600 games sequentially.

past learned tasks
catastrophic forgetting


Consequently, albeit being defined on the full input space and over a fairness similarity metric, because of its worst-case nature, IF has been linked to adversarial robustness #REFR .

Several calls for caution have, however, recently been raised about their deployment in tasks where fairness is of concern #OTHEREFR .

In fact, Neural Networks (NNs) have been found to reinforce negative biases from sensitive datasets #OTHEREFR , discriminating against individuals on the basis of attributes such as gender or race.

To address this, research efforts have been directed at both measuring the fairness of NNs, their de-biased training, as well as defining precise notions of fairness.

Given a BNN f and a similarity metric between individuals d f air , which encodes a task-dependent notion of similarity #OTHEREFR , Individual Fairness (IF) enforces that all pairs of similar individuals in the input space get treated similarly by f #OTHEREFR .

As opposed to the statistical nature of group fairness #OTHEREFR , IF aims at computing worst-case bias measures over a model input space.

As it has recently been show that Bayesian Neural Networks (BNNs) have a tendency to be less fragile to adversarial attacks than their frequentist counter-parts #OTHEREFR , it is natural to wonder whether approximate Bayesian inference may also have a positive impact over the IF of a neural network.

However, to the best of our knowledge, no work has been conducted along these lines of inquire.

In this paper, we investigate the IF of BNNs and empirically evaluate it on various benchmarks.

While exact computations of IF in BNNs is infeasible due to their non-convexity, we exploit the relationship between IF and adversarial robustness #OTHEREFR to develop a framework for the adaptation of adversarial attack methods for IF.

In particular, we explicitly instantiate Fair-FGSM and Fair-PGD as extensions of their corresponding adversarial attacks #OTHEREFR by employing gradient steps modifications and projections specific to d f air metrics commonly used in the fairness literature.

Individual Fairness In Bayesian Neural Networks

Abstract: We study Individual Fairness (IF) for Bayesian neural networks (BNNs). Specifically, we consider the-δ-individual fairness notion, which requires that, for any pair of input points that are-similar according to a given similarity metrics, the output of the BNN is within a given tolerance δ > 0. We leverage bounds on statistical sampling over the input space and the relationship between adversarial robustness and individual fairness to derive a framework for the systematic estimation of-δ-IF, designing Fair-FGSM and Fair-PGD as global, fairness-aware extensions to gradient-based attacks for BNNs. We empirically study IF of a variety of approximately inferred BNNs with different architectures on fairness benchmarks, and compare against deterministic models learnt using frequentist techniques. Interestingly, we find that BNNs trained by means of approximate Bayesian inference consistently tend to be markedly more individually fair than their deterministic counterparts.

Individual Fairness Revisited: Transferring Techniques From Adversarial Robustness

Abstract: We turn the definition of individual fairness on its head---rather than ascertaining the fairness of a model given a predetermined metric, we find a metric for a given model that satisfies individual fairness. This can facilitate the discussion on the fairness of a model, addressing the issue that it may be difficult to specify a priori a suitable metric. Our contributions are twofold: First, we introduce the definition of a minimal metric and characterize the behavior of models in terms of minimal metrics. Second, for more complicated models, we apply the mechanism of randomized smoothing from adversarial robustness to make them individually fair under a given weighted $L^p$ metric. Our experiments show that adapting the minimal metrics of linear models to more complicated neural networks can lead to meaningful and interpretable fairness guarantees at little cost to utility.

fairness similarity metric
adversarial robustness


Overall, HRNet is the most robust against the PGD and CR-PGD attacks, in that it has the smallest PixAcc drop when increases. On the other hand, PSPNet is the most vulnerable. We note that #REFR has similar observations. Running time comparison.

This is because neighboring pixels with small certified radii can easily affect each other, which naturally forms the groups in the certified radius map (also see Figure 1 (i)).

Figure 5 verifies this intuition, where we test 10 random testing images in Pascal VOC.

We can see that a majority of the perturbations in CR-PGD are assigned to the pixels with relatively smaller certified radii, in order to wrongly predict more pixels.

In contrast, most of the perturbations in PGD are assigned to the pixels with relatively larger certified radii.

As wrongly predicting these pixels requires a larger perturbation, PGD misclassifies much fewer pixels than our CR-PGD. Different models have different robustness.

Over all testing images in the three datasets, the average time of CR-PGD is 4.0 seconds, while that of PGD is 3.6 seconds. The overhead of CR-PGD over PGD is 11%.

A Certified Radius-Guided Attack Framework To Image Segmentation Models

Abstract: Image segmentation is an important problem in many safety-critical applications such as medical imaging and autonomous driving. Recent studies show that modern image segmentation models are vulnerable to adversarial perturbations, while existing attack methods mainly follow the idea of attacking image classification models. We argue that image segmentation and classification have inherent differences, and design an attack framework specially for image segmentation models. Our goal is to thoroughly explore the vulnerabilities of modern segmentation models, i.e., aiming to misclassify as many pixels as possible under a perturbation budget in both white-box and black-box settings. Our attack framework is inspired by certified radius, which was originally used by defenders to defend against adversarial perturbations to classification models. We are the first, from the attacker perspective, to leverage the properties of certified radius and propose a certified radius guided attack framework against image segmentation models. Specifically, we first adapt randomized smoothing, the state-of-theart certification method for classification models, to derive the pixel's certified radius. A larger certified radius of a pixel means the pixel is theoretically more robust to adversarial perturbations. This observation inspires us to focus more on disrupting pixels with relatively smaller certified radii. Accordingly, we design a pixel-wise certified radius guided loss, when plugged into any existing white-box attack, yields our certified radius-guided white-box attack. Next, we propose the first black-box attack to image segmentation models via bandit. A key challenge is no gradient information is available. To address it, we design a novel gradient estimator, based on bandit feedback, which is queryefficient and provably unbiased and stable. We use this gradient estimator to design a projected bandit gradient descent (PBGD) attack. We further use pixels' certified radii and design a certified radius-guided PBGD (CR-PBGD) attack. We prove our PBGD and CR-PBGD attacks can achieve asymptotically optimal attack performance with an optimal rate. We evaluate our certified-radius guided white-box and black-box attacks on multiple modern image segmentation models and datasets. Our results validate the effectiveness of our certified radius-guided attack framework.

On The Robustness Of Semantic Segmentation Models To Adversarial Attacks

Abstract: Deep Neural Networks (DNNs) have been demonstrated to perform exceptionally well on most recognition tasks such as image classification and segmentation. However, they have also been shown to be vulnerable to adversarial examples. This phenomenon has recently attracted a lot of attention but it has not been extensively studied on multiple, large-scale datasets and complex tasks such as semantic segmentation which often require more specialised networks with additional components such as CRFs, dilated convolutions, skip-connections and multiscale processing. In this paper, we present what to our knowledge is the first rigorous evaluation of adversarial attacks on modern semantic segmentation models, using two large-scale datasets. We analyse the effect of different network architectures, model capacity and multiscale processing, and show that many observations made on the task of classification do not always transfer to this more complex task. Furthermore, we show how mean-field inference in deep structured models and multiscale processing naturally implement recently proposed adversarial defenses. Our observations will aid future efforts in understanding and defending against adversarial examples. Moreover, in the shorter term, we show which segmentation models should currently be preferred in safety-critical applications due to their inherent robustness.


We evaluate Split-CIFAR100, Split-MiniImagenet, and Ima-geNet32 using the protocol from #REFR with 100 training epochs training per task.

We report the mean and standard error over 3 runs.

Split-CIFAR100 and Split-MiniImageNet We consider Split-CIFAR100 and Split-MiniImageNet with 20 tasks of 5 classes each.

The results can be found in Figure 2 , for Split-CIFAR100 and Split-MiniImageNet, using different buffer sizes for ER.

In this setting, we can observe that our Figure 4 . Class-incremental accuracy on 20-Task Split-CIFAR100(left) and Split-MiniImageNet(right).

We observe that PRD outperforms not only other replay-free baselines but also ER, M=5, and is on par with ER, M=20, without storing any data.

Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning

Abstract: In Continual learning (CL) balancing effective adaptation while combating catastrophic forgetting is a central challenge. Many of the recent best-performing methods utilize various forms of prior task data, e.g. a replay buffer, to tackle the catastrophic forgetting problem. Having access to previous task data can be restrictive in many real-world scenarios, for example when task data is sensitive or proprietary. To overcome the necessity of using previous tasks data, in this work, we start with strong representation learning methods that have been shown to be less prone to forgetting. We propose a holistic approach to jointly learn the representation and class prototypes while maintaining the relevance of old class prototypes and their embedded similarities. Specifically, samples are mapped to an embedding space where the representations are learned using a supervised contrastive loss. Class prototypes are evolved continually in the same latent space, enabling learning and prediction at any point. To continually adapt the prototypes without keeping any prior task data, we propose a novel distillation loss that constrains class prototypes to maintain relative similarities as compared to new task data. This method yields state-of-the-art performance in the task-incremental setting where we are able to outperform other methods that both use no data as well as approaches relying on large amounts of data. Our method is also shown to provide strong performance in the class-incremental setting without using any stored data points.

Online Continual Learning With Maximally Interfered Retrieval

Abstract: Continual learning, the setting where a learning agent is faced with a never ending stream of data, continues to be a great challenge for modern machine learning systems. In particular the online or "single-pass through the data" setting has gained attention recently as a natural setting that is difficult to tackle. Methods based on replay, either generative or from a stored memory, have been shown to be effective approaches for continual learning, matching or exceeding the state of the art in a number of standard benchmarks. These approaches typically rely on randomly selecting samples from the replay memory or from a generative model, which is suboptimal. In this work, we consider a controlled sampling of memories for replay. We retrieve the samples which are most interfered, i.e. whose prediction will be most negatively impacted by the foreseen parameters update. We show a formulation for this sampling criterion in both the generative replay and the experience replay setting, producing consistent gains in performance and greatly reduced forgetting. We release an implementation of our method at https://github.com/optimass/Maximally_Interfered_Retrieval.


We do this by explicity analyzing the Chebyshev polynomial (as opposed to a polynomial approximation to a threshold function in #REFR ) and demonstrate that the output of Algorithm 2.2 is at least as good as outputting the aformentioned vector v (see Lemma 7.7 for details).

A (1 + ε) relative-error solution to the above cost corresponds to an additive ε 2/3 error.

The standard analysis of Krylov iteration #OTHEREFR , states that after q = log(n)/ √ ζ iterations, for any 0 < ζ < 1, the algorithm outputs a vector v such that v A Av A 2 op − ζσ 2 2 . By Pythagorean theorem,

A I − vv 2 F = A 2 op − Av 2 2 min u 2 =1 A I − uu + σ 2 2 ζ.

Since σ 2 2 1, it suffices to set ζ = ε 2/3 and thus q = O(log(n)/ε 1/3 ) iterations suffice.

We strengthen this analysis by showing that a significantly lower dimensional Krylov subspace (corresponding to O log(1/ε)/ε 1/3 iterations) spans a vector v such that A I − vv 2 F min u 2 =1 A I − uu + ε 2/3 .

In the complementary case, we deviate significantly from any prior analysis of Krylov iteration, including #OTHEREFR .

Here, we know that the number of singular values in the range [1/2, 1 − ε] is at most O(ε −1/3 ).

We therefore construct an entirely different polynomial, which is no longer based on Chebyshev polynomials.

This polynomial is designed to explicitly zero out all singular values in the range [1/2, 1 − ε].

We note that the degree of this polynomial, p 0 , is only O(ε −1/3 ), and it allows us to remove the contribution of all medium sized singular values, similar to starting with a larger block size.

Krylov Methods Are (Nearly) Optimal For Low-Rank Approximation

Abstract: We consider the problem of rank-1 low-rank approximation (LRA) in the matrix-vector product model under various Schatten norms: min u 2 =1 A(I − uu) S p , where M S p denotes the p norm of the singular values of M. Given ε > 0, our goal is to output a unit vector v such that A I − vv S p (1 + ε) min u 2 =1 A I − uu S p. Our main result shows that Krylov methods (nearly) achieve the information-theoretically optimal 1 number of matrix-vector products for Spectral (p = ∞), Frobenius (p = 2) and Nuclear (p = 1) LRA. In particular, for Spectral LRA, we show that any algorithm requires Ω log(n)/ε 1/2 matrix-vector products, exactly matching the upper bound obtained by Krylov methods [MM15]. Our lower bound addresses Open Question 1 in [Woo14], providing evidence for the lack of progress on algorithms for Spectral LRA and resolves Open Question 1.2 in [BCW22]. Next, we show that for any fixed constant p, i.e. 1 p = O(1), there is an upper bound of O log(1/ε)/ε 1/3 matrix-vector products, implying that the complexity does not grow as a function of input size. This improves the O log(n/ε)/ε 1/3 bound recently obtained in [BCW22], and matches their Ω 1/ε 1/3 lower bound, to a log(1/ε) factor.

Randomized Block Krylov Methods For Stronger And Faster Approximate Singular Value Decomposition

Abstract: Since being analyzed by Rokhlin, Szlam, and Tygert and popularized by Halko, Martinsson, and Tropp, randomized Simultaneous Power Iteration has become the method of choice for approximate singular value decomposition. It is more accurate than simpler sketching algorithms, yet still converges quickly for any matrix, independently of singular value gaps. After $\tilde{O}(1/\epsilon)$ iterations, it gives a low-rank approximation within $(1+\epsilon)$ of optimal for spectral norm error. ::: We give the first provable runtime improvement on Simultaneous Iteration: a simple randomized block Krylov method, closely related to the classic Block Lanczos algorithm, gives the same guarantees in just $\tilde{O}(1/\sqrt{\epsilon})$ iterations and performs substantially better experimentally. Despite their long history, our analysis is the first of a Krylov subspace method that does not depend on singular value gaps, which are unreliable in practice. ::: Furthermore, while it is a simple accuracy benchmark, even $(1+\epsilon)$ error for spectral norm low-rank approximation does not imply that an algorithm returns high quality principal components, a major issue for data applications. We address this problem for the first time by showing that both Block Krylov Iteration and a minor modification of Simultaneous Iteration give nearly optimal PCA for any matrix. This result further justifies their strength over non-iterative sketching methods. ::: Finally, we give insight beyond the worst case, justifying why both algorithms can run much faster in practice than predicted. We clarify how simple techniques can take advantage of common matrix properties to significantly improve runtime.


To relax this heavy reliance on the parameter space, some approaches directly regress 3D positions of mesh vertices instead of predicting the model's parameters. Among these approaches, Kolotouros et al. #REFR and Hongsuk et al.

The fitted mesh is then used as a supervisory signal to train a feed-forward network with a mesh convolutional decoder. Spurr et al.

#OTHEREFR introduce biomechanical constraints to guide the network to predict feasible hand poses with weakly-annotated real-world data. Chen et al.

#OTHEREFR use 2D joints extracted from an off-the-shelf 2D pose estimator as a supervisory signal to train a modelbased autoencoder to estimate 3D hand pose and shape.

However, similar to the model based approaches, they do not exploit correlation between joints and mesh vertices, yet our proposed AMVUR model addresses this issue and improves the feature representation of joints and vertices.

Model-free Methods: Although hand parametric models such as MANO serve as a strong structural prior to support 3D hand reconstruction, help to handle severe occlusions and help to accommodate weakly-annotated data, approaches that rely on this can easily get stuck in the model's parameter space, resulting in a non-minimal representation problem #OTHEREFR .

#OTHEREFR combine an image-based CNN and a GraphCNN to estimate human mesh coordinates directly. Lin et al.

#OTHEREFR argue that GraphCNN can only capture the local interactions between neighboring vertices of the triangle mesh, so they use a self-attention mechanism to capture global interactions between the vertices and joints. Most recently, Hampali et al.

#OTHEREFR first extract joint features by localizing them on CNN feature maps, then take these features and their spatial encodings as the input to a transformer model for 3D hand pose estimation.

However, spatial encoding is ambiguous to describe joints' 3D locations, especially for overlapping 3D joints in 2D images.

Different from the above approaches, in AMVUR, a cross-attention module is proposed to learn the correlation between joints and mesh vertices, followed by a self-attention module to learn the correlation between different vertices.

A Probabilistic Attention Model With Occlusion-Aware Texture Regression For 3D Hand Reconstruction From A Single Rgb Image

Abstract: Recently, deep learning based approaches have shown promising results in 3D hand reconstruction from a single RGB image. These approaches can be roughly divided into model-based approaches, which are heavily dependent on the model's parameter space, and model-free approaches, which require large numbers of 3D ground truths to reduce depth ambiguity and struggle in weakly-supervised scenarios. To overcome these issues, we propose a novel probabilistic model to achieve the robustness of model-based approaches and reduced dependence on the model's parameter space of model-free approaches. The proposed probabilistic model incorporates a model-based network as a prior-net to estimate the prior probability distribution of joints and vertices. An Attention-based Mesh Vertices Uncertainty Regression (AMVUR) model is proposed to capture dependencies among vertices and the correlation between joints and mesh vertices to improve their feature representation. We further propose a learning based occlusionaware Hand Texture Regression model to achieve highfidelity texture reconstruction. We demonstrate the flexibility of the proposed probabilistic model to be trained in both supervised and weakly-supervised scenarios. The experimental results demonstrate our probabilistic model's stateof-the-art accuracy in 3D hand and texture reconstruction from a single image in both training schemes, including in the presence of severe occlusions.

Convolutional Mesh Regression For Single-Image Human Shape Reconstruction

Abstract: This paper addresses the problem of 3D human pose and shape estimation from a single image. Previous approaches consider a parametric model of the human body, SMPL, and attempt to regress the model parameters that give rise to a mesh consistent with image evidence. This parameter regression has been a very challenging task, with model-based approaches underperforming compared to nonparametric solutions in terms of pose estimation. In our work, we propose to relax this heavy reliance on the model's parameter space. We still retain the topology of the SMPL template mesh, but instead of predicting model parameters, we directly regress the 3D location of the mesh vertices. This is a heavy task for a typical network, but our key insight is that the regression becomes significantly easier using a Graph-CNN. This architecture allows us to explicitly encode the template mesh structure within the network and leverage the spatial locality the mesh has to offer. Image-based features are attached to the mesh vertices and the Graph-CNN is responsible to process them on the mesh structure, while the regression target for each vertex is its 3D location. Having recovered the complete 3D geometry of the mesh, if we still require a specific model parametrization, this can be reliably regressed from the vertices locations. We demonstrate the flexibility and the effectiveness of our proposed graph-based mesh regression by attaching different types of features on the mesh vertices. In all cases, we outperform the comparable baselines relying on model parameter regression, while we also achieve state-of-the-art results among model-based pose estimation approaches.


With perfect execution, success can generally be achieved within 50 steps, thisis similar to other short-horizon, continuousspace, manipulation tasks #REFR . See Appendix. B.1 and Table.

We also include proprioceptive sensors corresponding to the agent's arm position.

The arm of a Stretch RE1 agent uses a telescoping mechanism to move forward and back, may move up and down, and the gripper has one rotational degree of freedom allowing for changes in yaw, see Figures 2 and 3 for 3rd person views of the stretch robot.

The robotic arm of the Stretch RE1 robot is orthogonal to the agent's forward and backward movement and so, to move the arm laterally, the agent must move its body in the forward and backward direction.

To highlight the study of irreversible transitions in our benchmark, and as to not add additional complexity to STRETCH-P&P, we restrict the robot body to not rotate in training, although the wrist of the agent may do so.

The maximum rotation for the wrist is 2 • per step, and horizontal/vertical arm movement is limited to 5cm per step.

B.1 for further details regarding the observation and action spaces.

When Learning Is Out Of Reach, Reset: Generalization In Autonomous Visuomotor Reinforcement Learning

Abstract: Figure 1: Episodic, Reset-Free, and Reset-Minimizing RL. In standard (i.e. episodic) reinforcement learning (RL) agents have their environments reset after every success or failure, an expensive operation in the real world. In Reset-Free RL (RF-RL), researchers have designed "reset games" which allow for learning so long as special care is taken to avoid irreversible transitions (e.g. an apple falling out of reach). We consider Reset-Minimizing RL (RM-RL) where in realistic and dynamic environments agents may request human interventions but should minimize these requests.

Meta-World: A Benchmark And Evaluation For Multi-Task And Meta Reinforcement Learning

Abstract: Meta-reinforcement learning algorithms can enable robots to acquire new skills much more quickly, by leveraging prior experience to learn how to learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used meta-reinforcement learning benchmark uses different running velocities for a simulated robot as different tasks. When policies are meta-trained on such narrow task distributions, they cannot possibly generalize to more quickly acquire entirely new tasks. Therefore, if the aim of these methods is to enable faster acquisition of entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors. In this paper, we propose an open-source simulated benchmark for meta-reinforcement learning and multi-task learning consisting of 50 distinct robotic manipulation tasks. Our aim is to make it possible to develop algorithms that generalize to accelerate the acquisition of entirely new, held-out tasks. We evaluate 6 state-of-the-art meta-reinforcement learning and multi-task learning algorithms on these tasks. Surprisingly, while each task and its variations (e.g., with different object positions) can be learned with reasonable success, these algorithms struggle to learn with multiple tasks at the same time, even with as few as ten distinct training tasks. Our analysis and open-source environments pave the way for future research in multi-task learning and meta-learning that can enable meaningful generalization, thereby unlocking the full potential of these methods.


Artistic style transfer, as an efficient way to create a new painting by combining the content of a natural images and the style of an existing painting image, is a major research topic in computer graphics and computer vision #REFR .

If a picture is worth a thousand words, then an artwork may tell the whole story.

The art style depicts the visual appearance of an artwork and characterizes how the artist expresses a theme and shows his/her creativity.

The features that identify an artwork, such as the artist's use of stroke, color and composition, determine the style.

The main challenges of arbitrary style transfer are extracting styles from artistic images and mapping a specific realistic image into an artistic one in a controllable way.

The core problem for style extraction is to find an effective representation of styles since it is in general hard to give explicit definitions across different styles.

To build a reasonable style feature space, it is necessary to explore the relationship and distribution of styles in order to capture both individual and holistic characteristics.

For the mapping process, several generative mechanisms are adopted to address different issues, such as auto-encoder #OTHEREFR , neural flow model #OTHEREFR and visual transformer #OTHEREFR .

In contrast to the goal of those methods, we propose to improve arbitrary style transfer via a unified framework that offers the guidance of proper artistic style representation and works for various generative backbones.

A Unified Arbitrary Style Transfer Framework Via Adaptive Contrastive Learning

Abstract: Flow-based backbone [An et al. 2021] in UCAST ViT-based backbone [Deng et al. 2022] in UCAST CNN-based backbone [Huang and Belongie 2017] in UCAST (b) Aquarelle (d) Aquarelle (k) Impressionism (g) Line Art (h) Ink and Wash (i) Impressionism (j) Ink and Wash

Neural Style Transfer: A Review

Abstract: The seminal work of Gatys et al. demonstrated the power of Convolutional Neural Networks (CNNs) in creating artistic imagery by separating and recombining image content and style. This process of using CNNs to render a content image in different styles is referred to as Neural Style Transfer (NST). Since then, NST has become a trending topic both in academic literature and industrial applications. It is receiving increasing attention and a variety of approaches are proposed to either improve or extend the original NST algorithm. In this paper, we aim to provide a comprehensive overview of the current progress towards NST. We first propose a taxonomy of current algorithms in the field of NST. Then, we present several evaluation methods and compare different NST algorithms both qualitatively and quantitatively. The review concludes with a discussion of various applications of NST and open problems for future research. A list of papers discussed in this review, corresponding codes, pre-trained models and more comparison results are publicly available at this https URL.


The proposed framework addresses this issue by simultaneously training and optimizing OOD detection, and a MixMatch #REFR based semi-SL in a multitask learning framework.

3) ReMixMatch: ReMixMatch #OTHEREFR enhances the Mix-Match #OTHEREFR framework by introducing the principles of distribution alignment and augmentation anchoring.

The notion of distribution alignment replaces the sharpening step of the MixMatch #OTHEREFR framework in an attempt to match model aggregated class prediction to that of the marginal distribution of the given ground truth.

Further, ReMixMatch #OTHEREFR introduces another principle of augmentation alignment in place of the consistency regularization step of MixMatch #OTHEREFR to encourage each output to be close to the prediction for a weakly-augmented version of the same input.

By incorporating these changes, ReMixMatch #OTHEREFR has been reported to be data efficient compared to MixMatch #OTHEREFR frameworks.

4) Multi-task Curriculum learning (MTL) Framework Guided Semi-SL: Unlike previous methods that focused on developing strong semi-SL models, the goal of MTL #OTHEREFR guided semi-SL is to explicitly address the issues associated with the presence of open-set samples in semi-SL.

The paper proposes a novel OOD detector based on the model's capacity to identify noisy labeled training data.

The OOD detector ensures that the subsequent semi-SL framework, based on MixMatch #OTHEREFR , is only trained with the inliers samples by selecting an appropriate threshold on the OOD score and filtering out the outlier samples from the unlabelled data.

The method significantly produced superior results in settings where samples from a different distribution contaminates the unlabelled data.

Along with these frameworks, a few studies have also reexamined the original MixMatch #OTHEREFR algorithm itself.

One such study found that the performance degradation caused by open-set samples in unlabelled data is primarily due to the Pseudo-Labelling (PL) task of MixMatch.

Robust Semi-Supervised Learning For Histopathology Images Through Self-Supervision Guided Out-Of-Distribution Scoring

Abstract: Semi-supervised learning (semi-SL) is a promising alternative to supervised learning for medical image analysis when obtaining good quality supervision for medical imaging is difficult. However, semi-SL assumes that the underlying distribution of unaudited data matches that of the few labeled samples, which is often violated in practical settings, particularly in medical images. The presence of out-of-distribution (OOD) samples in the unlabeled training pool of semi-SL is inevitable and can reduce the efficiency of the algorithm. Common preprocessing methods to filter out outlier samples may not be suitable for medical images that involve a wide range of anatomical structures and rare morphologies. In this paper, we propose a novel pipeline for addressing open-set supervised learning challenges in digital histology images. Our pipeline efficiently estimates an OOD score for each unlabelled data point based on self-supervised learning to calibrate the knowledge needed for a subsequent semi-SL framework. The outlier score derived from the OOD detector is used to modulate sample selection for the subsequent semi-SL stage, ensuring that samples conforming to the distribution of the few labeled samples are more frequently exposed to the subsequent semi-SL framework. Our framework is compatible with any semi-SL framework, and we base our experiments on the popular Mixmatch semi-SL framework. We conduct extensive studies on two digital pathology datasets, Kather colorectal histology dataset and a dataset derived from TCGA-BRCA whole slide images, and establish the effectiveness of our method by comparing with popular methods and frameworks in semi-SL algorithms through various experiments. Index Terms-Semi Supervised learning, open-set, label-noise, mixmatch I. INTRODUCTION Medical image analysis requires large volumes of supervised data to train deep learning models effectively, but obtaining good quality supervision for medical imaging is inherently difficult due to the associated labor, expertise, and time required [1]-[3]. In such scenarios, semi-supervised learning (semi-SL) offers an efficient alternative, especially when there are only a few labeled samples but plenty of unlabeled or unaudited data. Semi-SL algorithms can leverage the vast pool of unaudited training data by extracting discriminative information from the structure of unlabeled data that complements the knowledge gained from a small number of supervisory data samples. However, semi-SL assumes that the underlying distribution of the unaudited data matches that of the few labeled samples [4], [5].

Mixmatch: A Holistic Approach To Semi-Supervised Learning

Abstract: Semi-supervised learning has proven to be a powerful paradigm for leveraging unlabeled data to mitigate the reliance on large labeled datasets. In this work, we unify the current dominant approaches for semi-supervised learning to produce a new algorithm, MixMatch, that works by guessing low-entropy labels for data-augmented unlabeled examples and mixing labeled and unlabeled data using MixUp. We show that MixMatch obtains state-of-the-art results by a large margin across many datasets and labeled data amounts. For example, on CIFAR-10 with 250 labels, we reduce error rate by a factor of 4 (from 38% to 11%) and by a factor of 2 on STL-10. We also demonstrate how MixMatch can help achieve a dramatically better accuracy-privacy trade-off for differential privacy. Finally, we perform an ablation study to tease apart which components of MixMatch are most important for its success.


Attention scores of inner product used in the original transformer #REFR is not well-defined in our spiking transformer.

(b) Architecture of Spiking Spatiotemporal Attention that aims to address the one-way flow of information over time in SNNs, thus compensating for missing pose information, particularly for the early time steps.

Normalized Hamming similarity is proposed for the attention score, which has been shown to be equivalent to inner product similarity between real valued vectors used in original transformer.

where pos represents the position in the sequence, while 2i or 2i + 1 denotes the position of C k channel.

If the spike key is a zero vector, s k j = 0, then the attention score will always be zero for any spike query s q i , that is 0 • s q j = 0.

This means the inner product used in #OTHEREFR is not able to accurately determine the similarity between two spike vectors.

Event-Based Human Pose Tracking By Spiking Spatiotemporal Transformer

Abstract: Event camera, as an emerging biologically-inspired vision sensor for capturing motion dynamics, presents new potential for 3D human pose tracking, or video-based 3D human pose estimation. However, existing works in pose tracking either require the presence of additional gray-scale images to establish a solid starting pose, or ignore the temporal dependencies all together by collapsing segments of event streams to form static image frames. Meanwhile, although the effectiveness of Artificial Neural Networks (ANNs, a.k.a. dense deep learning) has been showcased in many event-based tasks, the use of ANNs tends to neglect the fact that compared to the dense frame-based image sequences, the occurrence of events from an event camera is spatiotemporally much sparser. Motivated by the above mentioned issues, we present in this paper a dedicated end-to-end sparse deep learning approach for event-based pose tracking: 1) to our knowledge this is the first time that 3D human pose tracking is obtained from events only, thus eliminating the need of accessing to any frame-based images as part of input; 2) our approach is based entirely upon the framework of Spiking Neural Networks (SNNs), which consists of Spike-Element-Wise (SEW) ResNet and our proposed spiking spatiotemporal transformer; 3) a large-scale synthetic dataset is constructed that features a broad and diverse set of annotated 3D human motions, as well as longer hours of event stream data, named SynEventHPD. Empirical experiments demonstrate the superiority of our approach in both performance and efficiency measures. For example, with comparable performance to the state-of-the-art ANNs counterparts, our approach achieves a computation reduction of 20% in FLOPS. Our implementation is made available at https://github.com/JimmyZou/HumanPoseTracking SNN and dataset will be released upon paper acceptance.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


For the rectangular elements, the approximation of the singular function follows closely the proof in #REFR , and we present it below for the convenience of the reader.

Note that the approximation error for the smooth term is of higher order.

By summing over all rectangles Γ j of the mesh of the screen and all components, we conclude that

p(u)−Π x Π t p(u) r,− 1 2 ,Γ, * hβ Re ν * −ε if ∆t ≤ min{h 1 , h 2 }.

(a), cone singularity: To discuss the approximation of p(u) in the cone geometry, for simplicity, we let Γ be the squareR = [0, 1] 2 .

Figure 4 shows how to reduce the mesh on the cone to this case by an affine map, with the exception of a small number of triangular elements.

For the additional triangular elements in Figure 4 (b) with linear basis functions, the crucial observation is that their angles are independent of h, leading to a shape-regular mesh.

In particular, the quotient ρ of the radii of the smallest circumscribed to the largest inscribed circle remains bounded and the expected interpolation inequalities hold: For the linear interpolant p of a function f determined by the vertices of a triangle T of circumscribed radius ≤ h, one has

Here, s ∈ [0, 1] and the constant C 0 only depends on ρ and s.

The respective proofs for the regular part φ φ φ 0 and the singular function r λ−1 b i in this way directly apply to the arising triangles.

Higher-Order Time Domain Boundary Elements For Elastodynamicsgraded Meshes And Hp Versions

Abstract: The solution to the elastodynamic equation in the exterior of a polyhedral domain or a screen exhibits singular behavior from the corners and edges. The detailed expansion of the singularities implies quasi-optimal estimates for piecewise polynomial approximations of the Dirichlet trace of the solution and the traction. The results are applied to hp and graded versions of the time domain boundary element method for the weakly singular and the hypersingular integral equations. Numerical examples confirm the theoretical results for the Dirichlet and Neumann problems for screens and for polygonal domains in 2d. They exhibit the expected quasi-optimal convergence rates and the singular behavior of the solutions.

Boundary Elements With Mesh Refinements For The Wave Equation

Abstract: The solution of the wave equation in a polyhedral domain in $\mathbb{R}^3$ admits an asymptotic singular expansion in a neighborhood of the corners and edges. In this article we formulate boundary and screen problems for the wave equation as equivalent boundary integral equations in time domain, study the regularity properties of their solutions and the numerical approximation. Guided by the theory for elliptic equations, graded meshes are shown to recover the optimal approximation rates known for smooth solutions. Numerical experiments illustrate the theory for screen problems. In particular, we discuss the Dirichlet and Neumann problems, as well as the Dirichlet-to-Neumann operator and applications to the sound emission of tires.


The achieved performance is comparable to the traditionally designed TVM compiler #REFR for deep learning.

To control the application of rewrite rules, strategy languages, such as Stratego #OTHEREFR ] have been proposed. #OTHEREFR provides a recent overview of the field.

These strategy languages enable strategic rewriting by providing combinators to compose rewrite rules into larger strategies.

Stratego is an integral part of the Spoofax language workbench by Kats and #OTHEREFR designed to declaratively specify languages and tailored IDEs to work with them.

The Stratego strategy language is used here to write interpreters and compilers purely using compositions of rewrites. #OTHEREFR

[ , 2020 describes the ELEVATE strategy language and how it is used to encode and control the application of traditional compiler optimizations, such as loop-tiling, compositionally.

In fact, ELEVATE shows how to rethink the design of "user-schedulable languages" as strategy languages, as highlighted by #OTHEREFR .

Schedules allow experts precise control over what compiler optimizations to apply, an idea popularized by the domain-specific compiler Halide #OTHEREFR

2013] and now widely adopted in other optimizing compilers, such as TVM.

Closely related to strategy languages are tactic languages in automatic theorem proving that allow to control the arrangement of individual proof steps.

These ideas go all the way back to #OTHEREFR

Traced Types For Safe Strategic Rewriting

Abstract: Strategy languages enable programmers to compose rewrite rules into strategies and control their application. This is useful in programming languages, e.g., for describing program transformations compositionally, but also in automated theorem proving, where related ideas have been studies with tactics languages. Clearly, not all compositions of rewrites are correct, but how can we assist programmers in writing correct strategies? In this paper, we present a static type system for strategy languages. We combine a structural type system capturing how rewrite strategies transform the shape of the rewritten syntax with a novel tracing system that keeps track of all possible legal strategy execution paths. Our type system raises warnings when parts of a composition are guaranteed to fail at runtime, and errors when no legal execution for a strategy is possible. We present a formalization of our strategy language and novel tracing type system, and formally prove its type soundness. We present formal results, showing that ill-traced strategies are guaranteed to fail at runtime and that well-traced strategy executions "can't go wrong", meaning that they are guaranteed to have a possible successful execution path.

Tvm: An Automated End-To-End Optimizing Compiler For Deep Learning

Abstract: There is an increasing need to bring machine learning to a wide diversity of hardware devices. Current frameworks rely on vendor-specific operator libraries and optimize for a narrow range of server-class GPUs. Deploying workloads to new platforms -- such as mobile phones, embedded devices, and accelerators (e.g., FPGAs, ASICs) -- requires significant manual effort. We propose TVM, a compiler that exposes graph-level and operator-level optimizations to provide performance portability to deep learning workloads across diverse hardware back-ends. TVM solves optimization challenges specific to deep learning, such as high-level operator fusion, mapping to arbitrary hardware primitives, and memory latency hiding. It also automates optimization of low-level programs to hardware characteristics by employing a novel, learning-based cost modeling method for rapid exploration of code optimizations. Experimental results show that TVM delivers performance across hardware back-ends that are competitive with state-of-the-art, hand-tuned libraries for low-power CPU, mobile GPU, and server-class GPUs. We also demonstrate TVM's ability to target new accelerator back-ends, such as the FPGA-based generic deep learning accelerator. The system is open sourced and in production use inside several major companies.


Adding noise to the gradients of the Wasserstein distance is more efficient than adding noise to the final parameters directly with respect to preserving privacy #REFR .

Although AFL has distinct privacy advantages like FL, current research shows that sensitive information can still be inferred by using shared parameters during the learning process #OTHEREFR .

To address this issue, we propose a differentially private WGAN-GP model, where differential privacy is achieved in WGAN-GP by adding carefully designed noise to gradients during the learning process. Definition 1.

A randomized function F is considered as ( , δ)-differentially private if the following inequality is satisfied for any two databases X and X differing in a single point and for any output subset S #OTHEREFR :

where F (X) and F (X ) are the outputs of the function F for inputs X and X , respectively.

The gradients ∆w n of discriminator parameter w n after adding noise can be expressed as follows:

where σ n represents the noise scale, and c g is the bound on the gradients of the Wasserstein distance.

According to Lemma 1 in #OTHEREFR , given the sampling rate p = m M (where m is the batch size and M is the total number of training data used in each discriminator iteration), the number of discriminator iterations N d between two generator iterations, and privacy violation δ, then for any positive , the discriminator parameter guarantees ( , δ)-differential privacy with respect to all data used in the generator iteration if we choose

A Vhetnet-Enabled Asynchronous Federated Learning-Based Anomaly Detection Framework For Ubiquitous Iot

Abstract: Anomaly detection for the Internet of Things (IoT) is a major intelligent service required by many fields, including intrusion detection, state monitoring, device-activity analysis, and security supervision. However, the heterogeneous distribution of data and resource-constrained end nodes in ubiquitous IoT systems present challenges for existing anomaly detection models. Due to the advantages of flexible deployment and multidimensional resources, high altitude platform stations (HAPSs) and unmanned aerial vehicles (UAVs), which are important components of vertical heterogeneous networks (VHetNets), have significant potential for sensing, computing, storage, and communication applications in ubiquitous IoT systems. In this paper, we propose a novel VHetNet-enabled asynchronous federated learning (AFL) framework to enable decentralized UAVs to collaboratively train a global anomaly detection model based on their local sensory data from ubiquitous IoT devices. In the VHetNet-enabled AFL framework, a HAPS operates as a central aerial server, and the local models trained in UAVs are uploaded to the HAPS for global aggregation due to its wide coverage and strong storage and computation capabilities. We also introduce a UAV selection strategy into the AFL framework to prevent UAVs with low local model quality and large energy consumption from affecting the learning efficiency and detection accuracy of the global model. To ensure the security of transmissions between UAVs and the HAPS via wireless links, we add designed noise to local model parameters in UAVs to achieve differential privacy during the information exchange process. Moreover, we propose a compound-action actor-critic (CA2C)-based joint device association, UAV selection, and UAV trajectory planning algorithm to further enhance the overall federated execution efficiency and detection model accuracy under the UAV energy constraints. Extensive experimental evaluation on a real-world dataset demonstrates that the proposed algorithm can achieve high detection accuracy with short federated execution time and low energy consumption. Index Terms-Anomaly detection, ubiquitous Internet of Things (IoT), vertical heterogeneous network (VHetNet), asynchronous federated learning (AFL), differential privacy.

Differentially Private Generative Adversarial Network

Abstract: Generative Adversarial Network (GAN) and its variants have recently attracted intensive research interests due to their elegant theoretical foundation and excellent empirical performance as generative models. These tools provide a promising direction in the studies where data availability is limited. One common issue in GANs is that the density of the learned generative distribution could concentrate on the training data points, meaning that they can easily remember training samples due to the high model complexity of deep networks. This becomes a major concern when GANs are applied to private or sensitive data such as patient medical records, and the concentration of distribution may divulge critical patient information. To address this issue, in this paper we propose a differentially private GAN (DPGAN) model, in which we achieve differential privacy in GANs by adding carefully designed noise to gradients during the learning procedure. We provide rigorous proof for the privacy guarantee, as well as comprehensive empirical evidence to support our analysis, where we demonstrate that our method can generate high quality data points at a reasonable privacy level.


The key idea is to let them use separate batch normalization (BN) layers while sharing all the other parameters of the network; following the domain-specific batch normalization #REFR we call such a set of separate BNs lighting-condition specific BN (LSBN).

As our target model deals with extremely low-light images, it suffers from the significantly low quality of inputs.

To alleviate this, we propose a new method for learning the target model using the paired well-lit images as privileged information #OTHEREFR , additional high-quality input data accessible only in training.

Our method introduces another model called teacher that takes the privileged information as input and provides rich supervision to the target model called student.

This method for learning using privileged information (LUPI) allows the student to simulate the internal behavior of the teacher as well as learn to predict human poses.

To further exploit the privileged information, we design a single concise architecture that integrates the teacher and the student.

This design choice allows the student to enjoy the strong representation learned using the well-lit images (i.e., the privileged information) while capturing specific characteristics of low-light images through the separate BN parameters.

Our model architecture and LUPI strategy are depicted in Fig. 5 .

Note that, before being fed to the student, low-light images are scaled automatically by adjusting their average pixel intensity value to a predefined constant.

On the other hand, the teacher takes as input well-lit images as-is.

Both of the teacher and the student are trained by a common pose estimation loss:

Human Pose Estimation In Extremely Low-Light Conditions

Abstract: We study human pose estimation in extremely low-light images. This task is challenging due to the difficulty of collecting real low-light images with accurate labels, and severely corrupted inputs that degrade prediction quality significantly. To address the first issue, we develop a dedicated camera system and build a new dataset of real lowlight images with accurate pose labels. Thanks to our camera system, each low-light image in our dataset is coupled with an aligned well-lit image, which enables accurate pose labeling and is used as privileged information during training. We also propose a new model and a new training strategy that fully exploit the privileged information to learn representation insensitive to lighting conditions. Our method demonstrates outstanding performance on real extremely low-light images, and extensive analyses validate that both of our model and dataset contribute to the success.

Domain-Specific Batch Normalization For Unsupervised Domain Adaptation

Abstract: We propose a novel unsupervised domain adaptation framework based on domain-specific batch normalization in deep neural networks. We aim to adapt to both domains by specializing batch normalization layers in convolutional neural networks while allowing them to share all other model parameters, which is realized by a two-stage algorithm. In the first stage, we estimate pseudo-labels for the examples in the target domain using an external unsupervised domain adaptation algorithm---for example, MSTN or CPUA---integrating the proposed domain-specific batch normalization. The second stage learns the final models using a multi-task classification loss for the source and target domains. Note that the two domains have separate batch normalization layers in both stages. Our framework can be easily incorporated into the domain adaptation techniques based on deep neural networks with batch normalization layers. We also present that our approach can be extended to the problem with multiple source domains. The proposed algorithm is evaluated on multiple benchmark datasets and achieves the state-of-the-art accuracy in the standard setting and the multi-source domain adaption scenario.

lighting-condition specific BN
domain-specific batch normalization


In addition, besides AdaptiveMix loss, we can use mixingbased cross-entropy loss in the learning objective of image classification following augmentation #REFR , since we use Mixup to generate hard samples (See supplementary material for more details on classification).

Benchmark #OTHEREFR Table 12 +3.5% F1 ↑ class-aware separation, we then introduce the orthogonal constraint to initialize W, which is defined as:

Improving Gan Training Via Feature Space Shrinkage

Abstract: Due to the outstanding capability for data generation, Generative Adversarial Networks (GANs) have attracted considerable attention in unsupervised learning. However, training GANs is difficult, since the training distribution is dynamic for the discriminator, leading to unstable image representation. In this paper, we address the problem of training GANs from a novel perspective, i.e., robust image classification. Motivated by studies on robust image representation, we propose a simple yet effective module, namely AdaptiveMix, for GANs, which shrinks the regions of training data in the image representation space of the discriminator. Considering it is intractable to directly bound feature space, we propose to construct hard samples and narrow down the feature distance between hard and easy samples. The hard samples are constructed by mixing a pair of training images. We evaluate the effectiveness of our AdaptiveMix with widely-used and state-of-the-art GAN architectures. The evaluation results demonstrate that our AdaptiveMix can facilitate the training of GANs and effectively improve the image quality of generated samples. We also show that our AdaptiveMix can be further applied to image classification and Out-Of-Distribution (OOD) detection tasks, by equipping it with state-of-theart methods. Extensive experiments on seven publicly available datasets show that our method effectively boosts the performance of baselines. The code is publicly available at https : / / github. com / WentianZhang-ML/AdaptiveMix.

Mixup: Beyond Empirical Risk Minimization

Abstract: Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.

image classification
mixingbased cross-entropy loss


Relevant experimental work on noisy segmentation include #REFR where the authors considered three different label noise models and observed that increased noise level caused worse performance.

This is especially important for this work because it closely relates to the studied problems.

In binary classification, optimal threshold based classifiers for the F 1 metric was discussed in #OTHEREFR and further elaborated by others.

Because of the close relationship between the Dice metric in segmentation and the F 1 metric in binary classification, this idea was later taken to the segmentation context and used to give a characterization of the optimal segmentations with respect to Dice #OTHEREFR .

Furthermore, sharp bounds of the volume of these solutions were provided.

Similar inspiration from work in binary classification [2, 3] made authors propose that calibration can be a property of importance for explaining the good performance obtained by soft-Dice when evaluation is done by Dice #OTHEREFR .

A similar experimental study also including experiments on biased noise was later described in #OTHEREFR .

Finally, experimental work on soft labels with other loss functions than soft-Dice include #OTHEREFR .

For recent general reviews on work on noisy segmentation and imperfect data, see #OTHEREFR and #OTHEREFR .

Noisy Image Segmentation With Soft-Dice

Abstract: This paper presents a study on the soft-Dice loss, one of the most popular loss functions in medical image segmentation, for situations where noise is present in target labels. In particular, the set of optimal solutions are characterized and sharp bounds on the volume bias of these solutions are provided. It is further shown that a sequence of soft segmentations converging to optimal soft-Dice also converges to optimal Dice when converted to hard segmentations using thresholding. This is an important result because soft-Dice is often used as a proxy for maximizing the Dice metric. Finally, experiments confirming the theoretical results are provided.

Imperfect Segmentation Labels: How Much Do They Matter?

Abstract: Labeled datasets for semantic segmentation are imperfect, especially in medical imaging where borders are often subtle or ill-defined. Little work has been done to analyze the effect that label errors have on the performance of segmentation methodologies. Here we present a large-scale study of model performance in the presence of varying types and degrees of error in training data. We trained U-Net, SegNet, and FCN32 several times for liver segmentation with 10 different modes of ground-truth perturbation. Our results show that for each architecture, performance steadily declines with boundary-localized errors, however, U-Net was significantly more robust to jagged boundary errors than the other architectures. We also found that each architecture was very robust to non-boundary-localized errors, suggesting that boundary-localized errors are fundamentally different and more challenging problem than random label errors in a classification setting.


This robustness verification method (Cohen et al., 2019; #REFR computes the 2 -norm certificates around an input sample x by counting which class is most likely to be returned when x is perturbed by isotropic Gaussian noise.

Even though an adversariallytrained network is resilient to attacks created during training, it can still be susceptible to unseen new attacks.

To overcome this problem, certified defenses formally guarantee the stability of the prediction in a neighbourhood of the input.

In other words, a neural network f is certifiably robust for the input x ∈ R d , if the prediction for all perturbed versions

x remains unchanged such that x − x p ≤ , where • p is the p -norm around x of size > 0.

Formally, given a soft classifier F , randomized smoothing considers a smooth version of F defined as:

where σ > 0 represents the standard deviation.

As previously, we define the hard version of G(x) as g(x) = arg max y∈Y G(x) y .

Contrary to other formal verification methods, randomized smoothing does not make any assumptions regarding the model's properties, allowing certification to be scaled to larger and more complex networks. Cohen et al.

Diffusion Denoised Smoothing For Certified And Adversarial Robust Out-Of-Distribution Detection

Abstract: As the use of machine learning continues to expand, the importance of ensuring its safety cannot be overstated. A key concern in this regard is the ability to identify whether a given sample is from the training distribution, or is an "Out-Of-Distribution" (OOD) sample. In addition, adversaries can manipulate OOD samples in ways that lead a classifier to make a confident prediction. In this study, we present a novel approach for certifying the robustness of OOD detection within a 2-norm around the input, regardless of network architecture and without the need for specific components or additional training. Further, we improve current techniques for detecting adversarial attacks on OOD samples, while providing high levels of certified and adversarial robustness on in-distribution samples. The average of all OOD detection metrics on CIFAR10/100 shows an increase of ∼ 13%/5% relative to previous approaches.

Certified Robustness To Adversarial Examples With Differential Privacy

Abstract: Adversarial examples that fool machine learning models, particularly deep neural networks, have been a topic of intense research interest, with attacks and defenses being developed in a tight back-and-forth. Most past defenses are best effort and have been shown to be vulnerable to sophisticated attacks. Recently a set of certified defenses have been introduced, which provide guarantees of robustness to norm-bounded attacks. However these defenses either do not scale to large datasets or are limited in the types of models they can support. This paper presents the first certified defense that both scales to large networks and datasets (such as Google’s Inception network for ImageNet) and applies broadly to arbitrary model types. Our defense, called PixelDP, is based on a novel connection between robustness against adversarial examples and differential privacy, a cryptographically-inspired privacy formalism, that provides a rigorous, generic, and flexible foundation for defense.


As an early prototype system of global connectivity, narrowband IoT #REFR has enjoyed significant success in the long-term evolution of terrestrial networks (TNs), which has motivated considerable interest in moving from human-type communication to machine-type communication (MTC) .

As an indispensable component of the space-air-ground integrated network, low earth orbit (LEO) satellites have received extensive attention in the research of beyond fifth generation (B5G) and sixth generation (6G) mobile communication systems #OTHEREFR - #OTHEREFR .

Extensive efforts have been devoted to the construction of satellite constellations over the past few decades, for example, the Iridium system in the 1990s and Starlink LEO constellation projects more recently #OTHEREFR .

With the evolution of space and communication technologies, satellite communication (SatCom) has extended from its original narrowband voice service to broadband multimedia service, which also brings more opportunities to the ubiquitous space-air-ground integrated connectivity.

Given the explosive growth of data traffic, advanced MTC applications will be more data-intensive, and the demands of advanced IoT-enabled applications will shift from low-rate short packet transmission to more rigorous low-latency, broadband, and reliable information interaction such as industrial Internet, smart cities, intelligent transportation, industrial metaverse, holographic communications, and so on #OTHEREFR .

On the other hand, remote and disaster areas also still face challenges in accessing the network due to the high cost of terrestrial infrastructure.

Therefore, it is also of interest to promote broadband LEO satellite-enabled non-terrestrial networks (NTNs) #OTHEREFR as a part of the communication infrastructure.

Due to the massive number of potential MTC user terminals (UTs) and the long propagation delay between ground and LEO satellites, it is inefficient to coordinate the channel resources for uplink access through traditional handshaking protocols.

Grant free-random access (GF-RA) is a compelling paradigm for massive access in MTC since it allows UTs to directly transmit their respective preamble and payload data to base stations (BSs) without the aforehand handshaking process.

Quasi-Synchronous Random Access For Massive Mimo-Based Leo Satellite Constellations

Abstract: Low earth orbit (LEO) satellite constellation-enabled communication networks are expected to be an important part of many Internet of Things (IoT) deployments due to their unique advantage of providing seamless global coverage. In this paper, we investigate the random access problem in massive multiple-input multiple-output-based LEO satellite systems, where the multi-satellite cooperative processing mechanism is considered. Specifically, at edge satellite nodes, we conceive a training sequence padded multi-carrier system to overcome the issue of imperfect synchronization, where the training sequence is utilized to detect the devices' activity and estimate their channels. Considering the inherent sparsity of terrestrial-satellite links and the sporadic traffic feature of IoT terminals, we utilize the orthogonal approximate message passing-multiple measurement vector algorithm to estimate the delay coefficients and user terminal activity. To further utilize the structure of the receive array, a two-dimensional estimation of signal parameters via rotational invariance technique is performed for enhancing channel estimation. Finally, at the central server node, we propose a majority voting scheme to enhance activity detection by aggregating backhaul information from multiple satellites. Moreover, multi-satellite cooperative linear data detection and multi-satellite cooperative Bayesian dequantization data detection are proposed to cope with perfect and quantized backhaul, respectively. Simulation results verify the effectiveness of our proposed schemes in terms of channel estimation, activity detection, and data detection for quasi-synchronous random access in satellite systems. Index Terms Internet of Things (IoT), low earth orbit (LEO) satellite, massive multiple-input multiple-output (mMIMO), random access

A Primer On 3Gpp Narrowband Internet Of Things

Abstract: Narrowband Internet of Things (NB-IoT) is a new cellular technology introduced in 3GPP Release 13 for providing wide-area coverage for IoT. This article provides an overview of the air interface of NB-IoT. We describe how NB-IoT addresses key IoT requirements such as deployment flexibility, low device complexity, long battery lifetime, support of massive numbers of devices in a cell, and significant coverage extension beyond existing cellular technologies. We also share the various design rationales during the standardization of NB-IoT in Release 13 and point out several open areas for future evolution of NB-IoT.


As a representative work, DeepCluster #REFR ) adopts a standard k-means for clustering, but it contains degenerate solutions.

Instance discrimination #OTHEREFR Radford et al., 2021) is realized with a contrastive loss which targets at pulling closer samples from the same instance while pushing away samples from different instances.

Despite the impressive performance, instance-wise contrastive learning can not capture the semantic information from the training data because it is trained to ignore the similarity between different instances.

Cluster discrimination #OTHEREFR is processed with iterative steps: the clustering step to assign pseudo class labels for each sample, and then the classification step to map each sample to its assigned label.

Since one cluster has more than one instance, learning representations with clusters will gather similar instances together, which can explore potential semantic structures in data.

To this end, recent research work #OTHEREFR focuses on improving the label assignment during clustering but employs a standard cross-entropy loss during discrimination.

In this paper, we only employ one step of off-line clustering but design a robust classifier to achieve good feature representation when training on the automatically clustered large-scale data.

Image retrieval task typically relies on fine-tuning pre-trained visual models #OTHEREFR and can be divided into two learning categories: supervised and unsupervised metric learning.

For supervised metric learning, pair-wise loss #OTHEREFR and cross-entropy loss #OTHEREFR are extensively studied and recent bench-marking results #OTHEREFR indicate that the margin-based softmax loss (e.g., ArcFace #OTHEREFR ) can achieve state-of-the-art performance.

Unicom: Universal And Compact Representation Learning For Image Retrieval

Abstract: Modern image retrieval methods typically rely on fine-tuning pre-trained encoders to extract image-level descriptors. However, the most widely used models are pre-trained on ImageNet-1K with limited classes. The pre-trained feature representation is therefore not universal enough to generalize well to the diverse open-world classes. In this paper, we first cluster the large-scale LAION 400M dataset into one million pseudo classes based on the joint textual and visual features extracted by the CLIP model. Due to the confusion of label granularity, the automatically clustered dataset inevitably contains heavy inter-class conflict. To alleviate such conflict, we randomly select partial inter-class prototypes to construct the margin-based softmax loss. To further enhance the low-dimensional feature representation, we randomly select partial feature dimensions when calculating the similarities between embeddings and class-wise prototypes. The dual random partial selections are with respect to the class dimension and the feature dimension of the prototype matrix, making the classification conflict-robust and the feature embedding compact. Our method significantly outperforms state-of-the-art unsupervised and supervised image retrieval approaches on multiple benchmarks. The code and pre-trained models are released to facilitate future research https://github.com/deepglint/unicom.

Deep Clustering For Unsupervised Learning Of Visual Features

Abstract: Clustering is a class of unsupervised learning methods that has been extensively applied and studied in computer vision. Little work has been done to adapt it to the end-to-end training of visual features on large scale datasets. In this work, we present DeepCluster, a clustering method that jointly learns the parameters of a neural network and the cluster assignments of the resulting features. DeepCluster iteratively groups the features with a standard clustering algorithm, k-means, and uses the subsequent assignments as supervision to update the weights of the network. We apply DeepCluster to the unsupervised training of convolutional neural networks on large datasets like ImageNet and YFCC100M. The resulting model outperforms the current state of the art by a significant margin on all the standard benchmarks.


In fact, for the effectively acyclic case, #REFR only provides a sufficient condition that a given footprint yields a framepreserving update but it gives no algorithm for computing such a footprint.

This way, one can focus on the least flow, which is guaranteed to exist if one applies standard fixed point theorems, imposing only mild assumptions on the edge functions. However, cancellativity is inherently incompatible with standard domain-theoretic prerequisites.

For instance, the only ordered cancellative commutative monoid that is a directed cpo is the trivial one: M 0 = {0}.

Similarly, M 0 is the only such monoid that has a greatest element.

For cases where unique flows are desired, #OTHEREFR imposes additional requirements on the edge functions (nil-potent) or the graph structure (effectively acyclic). The former is quite restrictive in terms of expressivity.

The latter again complicates the computation of frame-preserving updates: one now has to ensure that no cycles are introduced when the updated graph h 2 is composed with its frame h 1 .

In this paper, we propose a new meta theory of flows based on flow monoids that form ω-cpos (but need not be cancellative).

The cpo requirement yields the desired least fixed point semantics.

The differences in the requirements on the flow monoid necessitate a new notion of flow graph composition.

In particular, for a least fixed point semantics of flows, h = h 1 * h 2 is only defined if the flows of h 1 and h 2 do not vanish.

Make Flows Small Again: Revisiting The Flow Framework

Abstract: We present a new flow framework for separation logic reasoning about programs that manipulate general graphs. The framework overcomes problems in earlier developments: it is based on standard fixed point theory, guarantees least flows, rules out vanishing flows, and has an easy to understand notion of footprint as needed for soundness of the frame rule. In addition, we present algorithms for automating the frame rule, which we evaluate on graph updates extracted from linearizability proofs for concurrent data structures. The evaluation demonstrates that our algorithms help to automate key aspects of these proofs that have previously relied on user guidance or heuristics.

Local Reasoning For Global Graph Properties

Abstract: Separation logics are widely used for verifying programs that manipulate complex heap-based data structures. These logics build on so-called separation algebras, which allow expressing properties of heap regions such that modifications to a region do not invalidate properties stated about the remainder of the heap. This concept is key to enabling modular reasoning and also extends to concurrency. While heaps are naturally related to mathematical graphs, many ubiquitous graph properties are non-local in character, such as reachability between nodes, path lengths, acyclicity and other structural invariants, as well as data invariants which combine with these notions. Reasoning modularly about such graph properties remains notoriously difficult, since a local modification can have side-effects on a global property that cannot be easily confined to a small region. ::: In this paper, we address the question: What separation algebra can be used to avoid proof arguments reverting back to tedious global reasoning in such cases? To this end, we consider a general class of global graph properties expressed as fixpoints of algebraic equations over graphs. We present mathematical foundations for reasoning about this class of properties, imposing minimal requirements on the underlying theory that allow us to define a suitable separation algebra. Building on this theory we develop a general proof technique for modular reasoning about global graph properties over program heaps, in a way which can be integrated with existing separation logics. To demonstrate our approach, we present local proofs for two challenging examples: a priority inheritance protocol and the non-blocking concurrent Harris list.


Thereby, the performance on the previously-known classes is similar to the baseline even without including a distillation loss #REFR .

The quantitative results of our semantic segmentation method, reported in Tab.

2, demonstrate, that the empty classes are "filled" with the novel concepts human and car.

For the car class, our method outperforms the baseline with respect to IoU (+2.87 pp), precision (+0.55 pp) and recall (+3.06 pp).

We lose performance in terms of IoU for the human class due to a higher tendency for false positives.

However, the false negative rate is significantly reduced, which is indicated by an increase in the recall value of 26.89 pp. The improved recall score is also visible in Fig.

9 , showing two examples from the Cityscapes validation dataset.

In the top row, several pedestrians are crossing the street, which are mostly segmented by our DNN, whereas the baseline DNN mostly misses the persons in the center as well as all heads.

Detecting Novelties With Empty Classes

Abstract: For open world applications, deep neural networks (DNNs) need to be aware of previously unseen data and adaptable to evolving environments. Furthermore, it is desirable to detect and learn novel classes which are not included in the DNNs underlying set of semantic classes in an unsupervised fashion. The method proposed in this article builds upon anomaly detection to retrieve out-of-distribution (OoD) data as candidates for new classes. We thereafter extend the DNN by k empty classes and fine-tune it on the OoD data samples. To this end, we introduce two loss functions, which 1) entice the DNN to assign OoD samples to the empty classes and 2) to minimize the inner-class feature distances between them. Thus, instead of ground truth which contains labels for the different novel classes, the DNN obtains a single OoD label together with a distance matrix, which is computed in advance. We perform several experiments for image classification and semantic segmentation, which demonstrate that a DNN can extend its own semantic space by multiple classes without having access to ground truth.

Knowledge Distillation For Incremental Learning In Semantic Segmentation

Abstract: Although deep learning architectures have shown remarkable results in scene understanding problems, they exhibit a critical drop of overall performance due to catastrophic forgetting when they are required to incrementally learn to recognize new classes without forgetting the old ones. This phenomenon impacts on the deployment of artificial intelligence in real world scenarios where systems need to learn new and different representations over time. Current approaches for incremental learning deal only with the image classification and object detection tasks. In this work we formally introduce the incremental learning problem for semantic segmentation. To avoid catastrophic forgetting we propose to distill the knowledge of the previous model to retain the information about previously learned classes, whilst updating the current model to learn the new ones. We developed three main methodologies of knowledge distillation working on both the output layers and the internal feature representations. Furthermore, differently from other recent frameworks, we do not store any image belonging to the previous training stages while only the last model is used to preserve high accuracy on previously learned classes. Extensive results were conducted on the Pascal VOC2012 dataset and show the effectiveness of the proposed approaches in different incremental learning scenarios.