In addition, the 4DRT-based RTNH achieves similar 3D object detection results to LiDAR point cloud-based (LPC-based) neural network, PointPillars #REFR , in road environments under clear weather conditions, and significantly outperforms the LPC-based network in adverse weather conditions such as sleet and heavy snow.

Due to the density, an RT provides rich information regarding the environment, but at a cost of a large amount of memory for storage and computations.

With the availability of dense RTs, many studies #OTHEREFR , #OTHEREFR - #OTHEREFR have proposed RT-based object detection networks that achieve similar detection performance to camera and Lidarbased object detection networks.

In particular, the K-Radar dataset #OTHEREFR provides a collection of 4D Radar Tensor (4DRT) that consists of power measurements along the Doppler, range, azimuth, and elevation dimensions.

This is in contrast to the conventional 3D Radar tensor (3DRT) #OTHEREFR - #OTHEREFR that do not provide elevation information.

The importance of the additional elevation information has been shown in #OTHEREFR , where the 4DRT-based Radar Tensor Network with Height (RTNH) significantly outperforms the Radar Tensor Network without height (RTN) in the 3D object detection task.

These results indicate the importance of 4D Radar sensors for a robust perception in adverse weather conditions.

While the advantages of 4DRT-based networks are clear, it remains challenging to conduct experiments on the 4DRT data.

This is mainly because of the size of the 4DRT data is prohibitively large (i.e., ∼12TB).

In the prior work #OTHEREFR , the size of the 4DRT data is reduced by performing a density reduction online during training, where the ouput density level is chosen arbitrarily.

The effects of density reduction on the detection performance and memory consumption, however, remains largely unknown.

Enhanced K-Radar: Optimal Density Reduction To Improve Detection Performance And Accessibility Of 4D Radar Tensor-Based Object Detection

Abstract: Recent works have shown the superior robustness of four-dimensional (4D) Radar-based three-dimensional (3D) object detection in adverse weather conditions. However, processing 4D Radar data remains a challenge due to the large data size, which require substantial amount of memory for computing and storage. In previous work, an online density reduction is performed on the 4D Radar Tensor (4DRT) to reduce the data size, in which the density reduction level is chosen arbitrarily. However, the impact of density reduction on the detection performance and memory consumption remains largely unknown. In this paper, we aim to address this issue by conducting extensive hyperparamter tuning on the density reduction level. Experimental results show that increasing the density level from 0.01% to 50% of the original 4DRT density level proportionally improves the detection performance, at a cost of memory consumption. However, when the density level is increased beyond 5%, only the memory consumption increases, while the detection performance oscillates below the peak point. In addition to the optimized density hyperparameter, we also introduce 4D Sparse Radar Tensor (4DSRT), a new representation for 4D Radar data with offline density reduction, leading to a significantly reduced raw data size. An optimized development kit for training the neural networks is also provided, which along with the utilization of 4DSRT, improves training speed by a factor of 17.1 compared to the stateof-the-art 4DRT-based neural networks.

Pointpillars: Fast Encoders For Object Detection From Point Clouds

Abstract: Object detection in point clouds is an important aspect of many robotics applications such as autonomous driving. In this paper, we consider the problem of encoding a point cloud into a format appropriate for a downstream detection pipeline. Recent literature suggests two types of encoders; fixed encoders tend to be fast but sacrifice accuracy, while encoders that are learned from data are more accurate, but slower. In this work, we propose PointPillars, a novel encoder which utilizes PointNets to learn a representation of point clouds organized in vertical columns (pillars). While the encoded features can be used with any standard 2D convolutional detection architecture, we further propose a lean downstream network. Extensive experimentation shows that PointPillars outperforms previous encoders with respect to both speed and accuracy by a large margin. Despite only using lidar, our full detection pipeline significantly outperforms the state of the art, even among fusion methods, with respect to both the 3D and bird’s eye view KITTI benchmarks. This detection performance is achieved while running at 62 Hz: a 2 - 4 fold runtime improvement. A faster version of our method matches the state of the art at 105 Hz. These benchmarks suggest that PointPillars is an appropriate encoding for object detection in point clouds.


Unlike some previous methods, we do not use any post-processing techniques like k-reciprocal re-ranking #REFR to refine our results.

On VeRi-776 and CityFlow-ReID, the initial learning rate is 1e-4 and the margin of triplet loss is set as 0.5 empirically.

The number of training epochs is 80 and the learning rate is decreased by a factor of 0.1 at 20th, 40th, and 60th epoch.

On VehicleID, the margin is 0.7 and the number of learning epochs is 120.

The learning rate is increased linearly from 0 to 1e-4 during the first 10 epochs, decreased with cosine scheduler to 1e-7 at 100th epoch, and to 0 at the last epoch.

We evaluate our approach by four widely used metrics in ReID literature, i.e., image-to-track retrieval mean Average Precision (tmAP) (if tracks are available in one dataset), image-to-image retrieval mAP (imAP), Top-1, and Top-5 accuracy.

Particularly, we report both tmAP and imAP on VeRi-776 for comprehensive evaluation.

These scores are shown as percentages and the best are marked in bold.

In Table 1 , 2, and 3, ES (Y/N) indicates whether Extra Supervision besides ID labels is employed to train a corresponding method.

Self-Supervised Geometric Features Discovery Via Interpretable Attention For Vehicle Re-Identification And Beyond (Complete Version)

Abstract: To learn distinguishable patterns, most of recent works in vehicle re-identification (ReID) struggled to redevelop official benchmarks to provide various supervisions, which requires prohibitive human labors. In this paper, we seek to achieve the similar goal but do not involve more human efforts. To this end, we introduce a novel framework, which successfully encodes both geometric local features and global representations to distinguish vehicle instances, optimized only by the supervision from official ID labels. Specifically, given our insight that objects in ReID share similar geometric characteristics, we propose to borrow self-supervised representation learning to facilitate geometric features discovery. To condense these features, we introduce an interpretable attention module, with the core of local maxima aggregation instead of fully automatic learning, whose mechanism is completely understandable and whose response map is physically reasonable. To the best of our knowledge, we are the first that perform self-supervised learning to discover geometric features. We conduct comprehensive experiments on three most popular datasets for vehicle ReID, i.e., VeRi-776, CityFlow-ReID, and VehicleID. We report our stateof-the-art (SOTA) performances and promising visualization results. We also show the excellent scalability of our approach on other ReID related tasks, i.e., person ReID and multi-target multi-camera (MTMC) vehicle tracking. The code is available at https://github.com/ ming1993li/Self-supervised-Geometric.

Re-Ranking Person Re-Identification With K-Reciprocal Encoding

Abstract: When considering person re-identification (re-ID) as a retrieval process, re-ranking is a critical step to improve its accuracy. Yet in the re-ID community, limited effort has been devoted to re-ranking, especially those fully automatic, unsupervised solutions. In this paper, we propose a k-reciprocal encoding method to re-rank the re-ID results. Our hypothesis is that if a gallery image is similar to the probe in the k-reciprocal nearest neighbors, it is more likely to be a true match. Specifically, given an image, a k-reciprocal feature is calculated by encoding its k-reciprocal nearest neighbors into a single vector, which is used for re-ranking under the Jaccard distance. The final distance is computed as the combination of the original distance and the Jaccard distance. Our re-ranking method does not require any human interaction or any labeled data, so it is applicable to large-scale datasets. Experiments on the large-scale Market-1501, CUHK03, MARS, and PRW datasets confirm the effectiveness of our method.


Random forests were grown with the Scikit-learn Python package #REFR and are composed of 500 trees each. We used the default parameters.

The CAE has been trained twice, once for selecting p and once for selecting 2p features. The model has been trained for 10 epochs.

Each hidden layer is composed of 32 neurons, and followed by a 20% dropout.

DeepPINK requires knockoff features, that we generated in two different ways depending on the nature of the dataset.

For the DAG dataset, we relied on the framework of Model-X Knockoff features designed by Candès et al. #OTHEREFR . Details of Model-X Knockoff features are explained in Suppl. Mat. 1.2.

For the remaining 4 datasets, knockoff features were generated by simply sampling a uniform distribution (the marginal distribution of the original data matrix X is multivariate but uniform).

We selected features from trained random forests using the widely-used impurity-based feature importance scores, as implemented in Scikit-Learn.

More specifically, the importance of a feature is defined as the total reduction in Gini impurity caused by all node splits involving that feature, averaged across all trees in the forest.

How Good Neural Networks Interpretation Methods Really Are? A Quantitative Benchmark

Abstract: Saliency Maps (SMs) have been extensively used to interpret deep learning models decision by highlighting the features deemed relevant by the model. They are used on highly nonlinear problems, where linear feature selection (FS) methods fail at highlighting relevant explanatory variables. However, the reliability of gradient-based feature attribution methods such as SM has mostly been only qualitatively (visually) assessed, and quantitative benchmarks are currently missing, partially due to the lack of a definite ground truth on image data. Concerned about the apophenic biases introduced by visual assessment of these methods, in this paper we propose a synthetic quantitative benchmark for Neural Networks (NNs) interpretation methods. For this purpose, we built synthetic datasets with nonlinearly separable classes and increasing number of decoy (random) features, illustrating the challenge of FS in high-dimensional settings. We also compare these methods to conventional approaches such as mRMR or Random Forests. Our results show that our simple synthetic datasets are sufficient to challenge most of the benchmarked methods. TreeShap, mRMR and LassoNet are the best performing FS methods. We also show that, when quantifying the relevance of a few non linearly-entangled predictive features diluted in a large number of irrelevant noisy variables, neural network-based FS and interpretation methods are still far from being reliable.

Scikit-Learn: Machine Learning In Python

Abstract: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.sourceforge.net.


Another line of works perform point convolution #REFR [20] on radius neighborhood points to alleviate the impact of density.

These methods lack robustness to varying density point clouds, since density directly influences the k-nearest neighbor search.

PointConv #OTHEREFR takes point density into account and uses the point position as input of the weight function, and employs another density function to re-weight the weighting function.

However, this re-weighting neglects the internal correlation between point density and position.

This deteriorates the network robustness and increases the computational complexity of convolution.

These methods directly predict kernel weights to map input features to output features, which comes at a high computational cost and also increases the memory traffic during the learning process.

For example, MCC #OTHEREFR computes the point density by Kernel Density Estimation (KDE) to directly adjust convolution kernel weights, where the weight function is approximated by MLPs.

This also has the same limitation as PointConv #OTHEREFR .

KPConv #OTHEREFR associates convolutional weights with a series of explicit kernel points in a local region, and predicts the kernel points weight through a linear correlation function of relative position between kernel points and neighbor points.

The linear coefficient is chosen according to the input density to characterize the influence of density to kernel weights.

However, the linear coefficient must be set offline i.e., the convolution needs to be specifically optimized for different datasets according to their point density.

Danet: Density Adaptive Convolutional Network With Interactive Attention For 3D Point Clouds

Abstract: Local features and contextual dependencies are crucial for 3D point cloud analysis. Many works have been devoted to designing better local convolutional kernels that exploit the contextual dependencies. However, current point convolutions lack robustness to varying point cloud density. Moreover, contextual modeling is dominated by non-local or self-attention models which are computationally expensive. To solve these problems, we propose density adaptive convolution, coined DAConv. The key idea is to adaptively learn the convolutional weights from geometric connections obtained from the point density and position. To extract precise context dependencies with fewer computations, we propose an interactive attention module (IAM) that embeds spatial information into channel attention along different spatial directions. DAConv and IAM are integrated in a hierarchical network architecture to achieve local density and contextual direction-aware learning for point cloud analysis. Experiments show that DAConv is significantly more robust to point density compared to existing methods and extensive comparisons on challenging 3D point cloud datasets show that our network achieves state-of-theart classification results of 93.6% on ModelNet40, competitive semantic segmentation results of 68.71% mIoU on S3DIS and part segmentation results of 86.7% mIoU on ShapeNet.

Monte Carlo Convolution For Learning On Non-Uniformly Sampled Point Clouds

Abstract: Deep learning systems extensively use convolution operations to process input data. Though convolution is clearly defined for structured data such as 2D images or 3D volumes, this is not true for other data types such as sparse point clouds. Previous techniques have developed approximations to convolutions for restricted conditions. Unfortunately, their applicability is limited and cannot be used for general point clouds. We propose an efficient and effective method to learn convolutions for non-uniformly sampled point clouds, as they are obtained with modern acquisition techniques. Learning is enabled by four key novelties: first, representing the convolution kernel itself as a multilayer perceptron; second, phrasing convolution as a Monte Carlo integration problem, third, using this notion to combine information from multiple samplings at different levels; and fourth using Poisson disk sampling as a scalable means of hierarchical point cloud learning. The key idea across all these contributions is to guarantee adequate consideration of the underlying non-uniform sample distribution function from a Monte Carlo perspective. To make the proposed concepts applicable to real-world tasks, we furthermore propose an efficient implementation which significantly reduces the GPU memory required during the training process. By employing our method in hierarchical network architectures we can outperform most of the state-of-the-art networks on established point cloud segmentation, classification and normal estimation benchmarks. Furthermore, in contrast to most existing approaches, we also demonstrate the robustness of our method with respect to sampling variations, even when training with uniformly sampled data only. To support the direct application of these concepts, we provide a ready-to-use TensorFlow implementation of these layers at https://github.com/viscom-ulm/MCCNN.


Normalizing Batch Normalization #REFR Yes Either add noise to Batch norm mean and mean of squares calculation; or use public data to calculate these statistics.

Layer Normalization #OTHEREFR No GroupNormalziation #OTHEREFR No Weight Normalization #OTHEREFR No Specialized Layers GNN #OTHEREFR Yes Node or edge level GNN layers augment the features of the instance (node or edge) with features and labels of their direct neighbours, making this process not private.

Further, graph structure is also leaked through such aggregation #OTHEREFR .

An additional complication of GNN networks is that during inference time, the same (training) graph structure is reused for predictions, and it needs to be DP-protected as well (on top of DP-Training of GNN models) #OTHEREFR .

There are attempts at adding noise to the aggregation function, but the authors also had to change the structure of the network #OTHEREFR , while Sajadmanesh & Gatica-Perez (2021) considered nodes and labels private but treated edges as public data.The area of DP with GNN models is very much nascent.

Optimizers SGD #OTHEREFR No As long as DP-SGD version is used #OTHEREFR , which dictates per example clipping and adding noise to the aggregated batch gradients Adaptive and Accelerated Firstorder Optimizers (Adam #OTHEREFR , Adagrad #OTHEREFR

How To Dp-Fy Ml: A Practical Guide To Machine Learning With Differential Privacy

Abstract: Machine Learning (ML) models are ubiquitous in real world applications and are a constant focus of research. Modern ML models have become more complex, deeper, and harder to reason about. At the same time, the community has started to realize the importance of protecting the privacy of the training data that goes into these models. Differential Privacy (DP) has become a gold standard for making formal statements about data anonymization. However, while some adoption of DP has happened in industry, attempts to apply DP to real world complex ML models are still few and far between. The adoption of DP is hindered by limited practical guidance of what DP protection entails, what privacy guarantees to aim for, and the difficulty of achieving good privacy-utility-computation trade-offs for ML models. Tricks for tuning and maximizing performance are scattered among papers or stored in the heads of practitioners, particularly with respect to the challenging task of hyperparameter tuning. Furthermore, the literature seems to present conflicting evidence on how and whether to apply architectural adjustments and which components are "safe" to use with DP. In this survey paper, we attempt to create a self-contained guide that gives a in-depth overview of the field of DP ML. We aim to assemble information about achieving the best possible DP ML model with rigorous privacy guarantees. Our target audience is both researchers and practitioners. Researchers interested in DP for ML will benefit from a clear overview of current advances and areas for improvement. We also include theory-focused sections that highlight important topics such as privacy accounting and its assumptions, and convergence. For a practitioner, this survey provides a background in DP theory and a clear step-by-step guide for choosing an appropriate privacy definition and approach, implementing DP training, potentially updating the model architecture, and tuning hyperparameters. For both researchers and practitioners, consistently and fully reporting privacy guarantees is critical, and so we propose a set of specific best practices for stating guarantees. With sufficient computation and a sufficiently large training set or supplemental non-private data, both good accuracy (that is, almost as good as a non-private model) and good privacy can often be achievable. And even when computation and dataset size are limited, there are advantages to training with even a weak (but still finite) formal DP guarantee. Hence, we hope this work will facilitate more widespread deployments of DP ML models.

Batch Normalization: Accelerating Deep Network Training By Reducing Internal Covariate Shift

Abstract: Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.

public data
Normalizing Batch Normalization


Another notable aggregation technique is the Generalized Mean (GeM) #REFR which is a learnable generalized form of global pooling. Building on GeM, Berton et al.

#OTHEREFR proposed Gated NetVLAD, which uses a gating mechanism that incorporates attention in the computation of NetVLAD residuals.

Other techniques focus on regions of interest in the feature maps.

Among the first techniques is MAC #OTHEREFR , a simple aggregation method that applies max-pooling on each individual feature map, selecting only the most activated neurons. Building on that, Tolias et al.

#OTHEREFR introduced R-MAC (Regional Maximum Activations of Convolutions) that consists of extracting multiple Region of Interest (RoI) directly from the CNN feature maps to form representations.

These techniques showed impressive performance on the task of image retrieval and have since been used in VPR.

#OTHEREFR recently proposed CosPlace, a lightweight aggregation technique that combines GeM with a linear projection layer.

Their method showed impressive performance on the task of VPR, outperforming GeM and NetVLAD and achieving state-of-the-art results on multiple benchmarks.

Another trend in recent VPR works #OTHEREFR is to consider using a two-stage retrieval strategy, which consists of running a first global retrieval step to retrieve, for each query, the top k candidates from the reference database.

This step is generally more efficient because it uses k-NN on the global descriptors.

Then, a second computationally heavy step is performed where the candidates are re-ranked according to their local features #OTHEREFR .

Mixvpr: Feature Mixing For Visual Place Recognition

Abstract: Visual Place Recognition (VPR) is a crucial part of mobile robotics and autonomous driving as well as other computer vision tasks. It refers to the process of identifying a place depicted in a query image using only computer vision. At large scale, repetitive structures, weather and illumination changes pose a real challenge, as appearances can drastically change over time. Along with tackling these challenges, an efficient VPR technique must also be practical in real-world scenarios where latency matters. To address this, we introduce MixVPR, a new holistic feature aggregation technique that takes feature maps from pre-trained backbones as a set of global features. Then, it incorporates a global relationship between elements in each feature map in a cascade of feature mixing, eliminating the need for local or pyramidal aggregation as done in NetVLAD or TransVPR. We demonstrate the effectiveness of our technique through extensive experiments on multiple large-scale benchmarks. Our method outperforms all existing techniques by a large margin while having less than half the number of parameters compared to CosPlace and NetVLAD. We achieve a new all-time high recall@1 score of 94.6% on Pitts250k-test, 88.0% on MapillarySLS, and more importantly, 58.4% on Nordland. Finally, our method outperforms two-stage retrieval techniques such as Patch-NetVLAD, TransVPR and SuperGLUE all while being orders of magnitude faster. Our code and trained models are available at https://github.com/amaralibey/MixVPR.

Fine-Tuning Cnn Image Retrieval With No Human Annotation

Abstract: Image descriptors based on activations of Convolutional Neural Networks (CNNs) have become dominant in image retrieval due to their discriminative power, compactness of the representation, and the efficiency of search. Training of CNNs, either from scratch or fine-tuning, requires a large amount of annotated data, where high quality of the annotation is often crucial. In this work, we propose to fine-tune CNNs for image retrieval on a large collection of unordered images in a fully automatic manner. Reconstructed 3D models, obtained by the state-of-the-art retrieval and structure-from-motion methods, guide the selection of the training data. We show that both hard positive and hard negative examples, selected by exploiting the geometry and the camera positions available from the 3D models, enhance the performance in particular object retrieval. CNN descriptor whitening discriminatively learned from the same training data outperforms the commonly used PCA whitening. We propose a novel trainable Generalized-Mean (GeM) pooling layer that generalizes max and average pooling and show that it boosts retrieval performance. Applying the proposed method on VGG network achieves state-of-the-art performance on standard benchmarks: Oxford Buildings, Paris, and Holidays datasets.


We compare our panoptic one-click segmentation approach to a traditional click segmentation method #REFR adapted to this setting.

This creates large datasets with minimal expense associated with the labelling (i.e.

single click per object compared to pixel-wise segmentation) for training DL algorithms.

In this paper we present an efficient and accurate method for weak labelling enabled by panoptic segmentation.

We introduce panoptic one-click segmentation, a novel offline tool to produce pseudo-labels from a single click (annotation) per object in the image.

This approach greatly simplifies the creation of novel datasets (e.g. for semi-supervised learning).

This traditional approach learns to produce a segmentation mask for an object based on the following input: an RGB image, one click for the object of interest and potentially one click for all of the other objects in the image.

This is a computationally inefficient technique as it processes each of the N objects individually, this leads to long training times.

We overcome this inefficiency by employing panoptic segmentation #OTHEREFR to jointly resolve the segmentation mask for all N objects simultaneously, as depicted in Figure 1 .

We evaluate our panoptic-based approach on two challenging arable farming (crop/weed) datasets from sugar beet and corn fields (see Section IV-A).

Furthermore, we demonstrate the practical applicability of this one-click segmentation approach to train Mask R-CNN using just 10% of the manually annotated ground truth; the remaining 90% of the data is generated as pseudo-labels using our approach. In doing so, we make the following contributions:

Panoptic One-Click Segmentation: Applied To Agricultural Data

Abstract: In weed control, precision agriculture can help to greatly reduce the use of herbicides, resulting in both economical and ecological benefits. A key element here is the ability to locate and segment all the plants (crop & weed) from image data. Modern instance segmentation techniques can achieve this, however, training such systems requires large amounts of handlabelled data which is expensive and laborious to obtain. Weakly supervised training can help to greatly reduce labelling efforts and costs. In this paper we propose panoptic one-click segmentation, an efficient and accurate offline tool to produce pseudo-labels from click inputs and thereby reduce labelling effort when creating novel datasets. Our approach jointly estimates the pixel-wise location of all N objects in the scene, compared to traditional approaches which iterate independently through all N objects. This results in a highly efficient technique with greatly reduced training times. Using just 10% of the data to train our panoptic one-click segmentation approach yields 68.1% and 68.8% mean object intersection over union (IoU) on challenging sugar beet and corn image data respectively, providing comparable performance to traditional one-click approaches while being approximately 12 times (an order of magnitude) faster to train. We demonstrate the practical applicability of our system by generating pseudo-labels from click annotations for the remaining 90% of the data. These pseudo-labels are then used to train Mask R-CNN, in a semisupervised manner, improving the absolute performance (of mean foreground IoU) by 9.4 and 7.9 points for sugar beet and corn data respectively, demonstrating the potential of our approach to rapidly annotate challenging data. Finally, we show that our panoptic one-click segmentation technique is able to recover missed clicks during annotation outlining a further benefit over traditional approaches.

Deep Interactive Object Selection

Abstract: Interactive object selection is a very important research problem and has many applications. Previous algorithms require substantial user interactions to estimate the foreground and background distributions. In this paper, we present a novel deep-learning-based algorithm which has much better understanding of objectness and can reduce user interactions to just a few clicks. Our algorithm transforms user-provided positive and negative clicks into two Euclidean distance maps which are then concatenated with the RGB channels of images to compose (image, user interactions) pairs. We generate many of such pairs by combining several random sampling strategies to model users' click patterns and use them to finetune deep Fully Convolutional Networks (FCNs). Finally the output probability maps of our FCN-8s model is integrated with graph cut optimization to refine the boundary segments. Our model is trained on the PASCAL segmentation dataset and evaluated on other datasets with different object classes. Experimental results on both seen and unseen objects demonstrate that our algorithm has a good generalization ability and is superior to all existing interactive object selection approaches.


According to #REFR , when the batch-size becomes N times the original size, the learning rate should be increased to √ N times the original size, so the learning rate on VisDA-C (RS-UT) is 0.02.

We implement our framework in PyTorch #OTHEREFR .

We use ResNet-50 [ DomainNet, the batch-size is set to be 32 and the learning rate to be 0.01.

For VisDA-C (RS-UT), the batch-size is set to be 128 for acceleration.

We use stochastic gradient descent (SGD) optimizer with momentum 0.9 to optimize the models.

The learning rate is decayed with the factor of (1 + γ iter 10000 ) p with γ = 10, p = −0.75. Please refer to our code for more details.

We adopt the HOS score [35] as evaluation metric.

As there exists class imbalance problem, we set the per-class average accuracy over the known classes as OS * and the accuracy of the unknown samples as UNK.

HOS is the harmonic mean of OS * and UNK.

Imbalanced Open Set Domain Adaptation Via Moving-Threshold Estimation And Gradual Alignment

Abstract: Multimedia applications are often associated with cross-domain knowledge transfer, where Unsupervised Domain Adaptation (UDA) can be used to reduce the domain shifts. Open Set Domain Adaptation (OSDA) aims to transfer knowledge from a well-labeled source domain to an unlabeled target domain under the assumption that the target domain contains unknown classes. Existing OSDA methods consistently lay stress on the covariate shift, ignoring the potential label shift problem. The performance of OSDA methods degrades drastically under intradomain class imbalance and inter-domain label shift. However, little attention has been paid to this issue in the community. In this paper, the Imbalanced Open Set Domain Adaptation (IOSDA) is explored where the covariate shift, label shift and category mismatch exist simultaneously. To alleviate the negative effects raised by label shift in OSDA, we propose Open-set Movingthreshold Estimation and Gradual Alignment (OMEGA)-a novel architecture that improves existing OSDA methods on classimbalanced data. Specifically, a novel unknown-aware target clustering scheme is proposed to form tight clusters in the target domain to reduce the negative effects of label shift and intra-domain class imbalance. Furthermore, moving-threshold estimation is designed to generate specific thresholds for each target sample rather than using one for all. Extensive experiments on IOSDA, OSDA and OPDA benchmarks demonstrate that our method could significantly outperform existing state-of-the-arts. Code and data are available at https://github.com/mendicant04/OMEGA.

One Weird Trick For Parallelizing Convolutional Neural Networks

Abstract: I present a new way to parallelize the training of convolutional neural networks across multiple GPUs. The method scales significantly better than all alternatives when applied to modern convolutional neural networks.


In contrast to our work, they also suffer from the lossy nature of grid-wise occupancy discretization #REFR .

In line with our approach, #OTHEREFR uses a GNN-based encoder to learn structurally-aware state representations, but in the context of RL-based robotic manipulators.

Road occupancy as a standalone prediction task is widely covered in existing works #OTHEREFR - #OTHEREFR .

With the objective of encoding traffic scenes similar to ours (albeit not in the context of motion planning), encoder architectures for learning representations of occupancy maps have been proposed #OTHEREFR - #OTHEREFR .

Using graphical or otherwise spatially-aware encoders similar to ours, recent works such as #OTHEREFR - #OTHEREFR predict occupancy grids #OTHEREFR as an intermediate learning target for guiding the training of neural motion planners.

However, these approaches do not provide global, low-dimensional representations appropriate for decoupled RL agents.

Deep Occupancy-Predictive Representations For Autonomous Driving

Abstract: Manually specifying features that capture the diversity in traffic environments is impractical. Consequently, learning-based agents cannot realize their full potential as neural motion planners for autonomous vehicles. Instead, this work proposes to learn which features are task-relevant. Given its immediate relevance to motion planning, our proposed architecture encodes the probabilistic occupancy map as a proxy for obtaining pre-trained state representations of the environment. By leveraging a map-aware traffic graph formulation, our agent-centric encoder generalizes to arbitrary road networks and traffic situations. We show that our approach significantly improves the downstream performance of a reinforcement learning agent operating in urban traffic environments.

Design Space Of Behaviour Planning For Autonomous Driving

Abstract: We explore the complex design space of behaviour planning for autonomous driving. Design choices that successfully address one aspect of behaviour planning can critically constrain others. To aid the design process, in this work we decompose the design space with respect to important choices arising from the current state of the art approaches, and describe the resulting trade-offs. In doing this, we also identify interesting directions of future work.


For the training of the student model, similar to the setting in PATE #REFR , in the case of CIFAR-10, the student has access to 9,000 samples that are partially labeled through the noisy aggregation mechanism (step (b) in Prom-PATE) in Section 3.2.

Regarding the training of each re-teacher model, since the source model is pretrained on ImageNet, the visual prompt has a dimension of 224 × 224.

When training the re-teacher model, we optimize the model with Adam whilst using a learning rate of 0.05 with a decay rate of 70%, batch size of 16, and training epoch of 10.

In Section 4.7, we also investigate the effect of the binary mask M on visual prompt performance.

For label mapping, we randomly select ten classes from the 1,000 source classes as a one-to-one mapping.

We also investigate using FC layers as the label mapping function in Section 4.8.

The performance is evaluated on the remaining 1,000 samples in the testing set.

Meanwhile, in the case of Blood-MNIST #OTHEREFR , the student has access to 2,421 samples that are as well partially labeled with privacy.

We use Rényi DP (RDP, see the definition in the Supplementary Materials) privacy accountant 2 to calculate the privacy budget .

Exploring The Benefits Of Visual Prompting In Differential Privacy

Abstract: Visual Prompting (VP) is an emerging and powerful technique that allows sample-efficient adaptation to downstream tasks by engineering a well-trained frozen source model. In this work, we explore the benefits of VP in constructing compelling neural network classifiers with differential privacy (DP). We explore and integrate VP into canonical DP training methods and demonstrate its simplicity and efficiency. In particular, we discover that VP in tandem with PATE, a state-of-the-art DP training method that leverages the knowledge transfer from an ensemble of teachers, achieves the state-of-the-art privacy-utility tradeoff with minimum expenditure of privacy budget. Moreover, we conduct additional experiments on cross-domain image classification with a sufficient domain gap to further unveil the advantage of VP in DP. Lastly, we also conduct extensive ablation studies to validate the effectiveness and contribution of VP under DP consideration.

Scalable Private Learning With Pate

Abstract: The rapid adoption of machine learning has increased concerns about the privacy implications of machine learning models trained on sensitive data, such as medical records or other personal information. To address those concerns, one promising approach is Private Aggregation of Teacher Ensembles, or PATE, which transfers to a "student" model the knowledge of an ensemble of "teacher" models, with intuitive privacy provided by training teachers on disjoint data and strong privacy guaranteed by noisy aggregation of teachers' answers. However, PATE has so far been evaluated only on simple classification tasks like MNIST, leaving unclear its utility when applied to larger-scale learning tasks and real-world datasets. ::: In this work, we show how PATE can scale to learning tasks with large numbers of output classes and uncurated, imbalanced training data with errors. For this, we introduce new noisy aggregation mechanisms for teacher ensembles that are more selective and add less noise, and prove their tighter differential-privacy guarantees. Our new mechanisms build on two insights: the chance of teacher consensus is increased by using more concentrated noise and, lacking consensus, no answer need be given to a student. The consensus answers used are more likely to be correct, offer better intuitive privacy, and incur lower-differential privacy cost. Our evaluation shows our mechanisms improve on the original PATE on all measures, and scale to larger tasks with both high utility and very strong privacy ($\varepsilon$ < 1.0).

student model
noisy aggregation mechanism


FL has proven to be very efficient in training models with strong performance, while avoiding the need for data sharing #REFR .

Federated learning (FL) #OTHEREFR aims to overcome the challenges of data sharing by enabling collaborative training, which does not require that the involved parties share their training data.

Therefore, the data remains private to each local node within the FL network, such that only the model updates are shared and integrated in a centralized model.

Federated averaging #OTHEREFR is the most common form of FL.

In this setting, a centralized server is connected to N entities, which have their own training data.

The central server orchestrates the collaborative training process as follows: (1) the initial model is distributed amongst all entities, (2) each entity performs a training iteration on their local model using their own training data, typically one epoch, and shares its resulting model parameters with the central server, (3) the server averages the model parameters shared by all entities and distributes the resulting (averaged) model amongst all entities, and (4) steps (2) and (3) are repeated sequentially until a performance threshold or a specific number of training iterations is achieved.

However, FL might be vulnerable to privacy issues such as reconstruction attacks #OTHEREFR , thus requiring that it is combined with other privacy-preserving methods to ensure robust privacy guarantees (Nguyen et al., 2022).

Iclr 2023 Workshop On Trustworthy Machine Learning For Healthcare Privacy-Preserving Machine Learning For Healthcare: Open Challenges And Future Per-Spectives

Abstract: Machine Learning (ML) has recently shown tremendous success in modeling various healthcare prediction tasks, ranging from disease diagnosis and prognosis to patient treatment. Due to the sensitive nature of medical data, privacy must be considered along the entire ML pipeline, from model training to inference. In this paper, we conduct a review of recent literature concerning Privacy-Preserving Machine Learning (PPML) for healthcare. We primarily focus on privacy-preserving training and inference-as-a-service, and perform a comprehensive review of existing trends, identify challenges, and discuss opportunities for future research directions. The aim of this review is to guide the development of private and efficient ML models in healthcare, with the prospects of translating research efforts into real-world settings.

Communication-Efficient Learning Of Deep Networks From Decentralized Data

Abstract: Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.


However, an inappropriate size for z can still lead to mode collapse, with the generator mapping multiple z vectors to the same output #REFR .

Unlike other techniques, such as principal component analysis (PCA) and others, the determination of the latent vector's (z) size can be challenging in GANs.

The deep convolutional neural networks utilised for both generator and discriminator allow our model to generate samples of higher quality, as they capture the hierarchical features of the target data distribution.

This not only stabilises the training process but also helps to avoid mode collapse.

Especially when z is small, the possibility of the generator's failure to cover the entire training dataset distribution increases, and it may produce many invalid designs and/or designs with minimal diversity.

Obviously, a larger z may resolve this, but not without cost since large vectors correspond to high-dimensional design spaces when performing shape optimisation, which increases the computational complexity of the entire simulation-driven design pipeline #OTHEREFR .

Therefore, for estimating a sufficient but not redundant size of z, we perform PCA and use the number of eigenvalues required for achieving a target variance as a reasonable estimation of the initial size of z. As it can be easily seen from Fig.

15 , 30 latent features in z can capture 99% of geometric variance.

We, therefore, set the initial size of z to 30 and then reduce it iteratively while measuring the diversity, novelty, and maximum mean discrepancy (MMD) #OTHEREFR of generated designs.

Shiphullgan: A Generic Parametric Modeller For Ship Hull Design Using Deep Convolutional Generative Model

Abstract: Figure 1: The generic capability of the ShipHullGAN model enables the creation of parametric design variations for a wide range of ship hulls, including both traditional and unconventional forms. In this work, we introduce ShipHullGAN, a generic parametric modeller built using deep convolutional generative adversarial networks (GANs) for the versatile representation and generation of ship hulls. At a high level, the new model intends to address the current conservatism in the parametric ship design paradigm, where parametric modellers can only handle a particular ship type. We trained ShipHullGAN on a large dataset of 52,591 physically validated designs from a wide range of existing ship types, including container ships, tankers, bulk carriers, tugboats, and crew supply vessels. We developed a new shape extraction and representation strategy to convert all training designs into a common geometric representation of the same resolution, as typically GANs can only accept vectors of fixed dimension as input. A space-filling layer is placed right after the generator component to ensure that the trained generator can cover all design classes. During training, designs are provided in the form of a shape-signature tensor (SST) which harnesses the compact geometric representation using geometric moments that further enable the inexpensive incorporation of physics-informed elements in ship design. We have shown through extensive comparative studies and optimisation cases that ShipHullGAN can generate designs with augmented features resulting in versatile design spaces that produce traditional and novel designs with geometrically valid and practically feasible shapes.

Nips 2016 Tutorial: Generative Adversarial Networks

Abstract: This report summarizes the tutorial presented by the author at NIPS 2016 on generative adversarial networks (GANs). The tutorial describes: (1) Why generative modeling is a topic worth studying, (2) how generative models work, and how GANs compare to other generative models, (3) the details of how GANs work, (4) research frontiers in GANs, and (5) state-of-the-art image models that combine GANs with other methods. Finally, the tutorial contains three exercises for readers to complete, and the solutions to these exercises.


Global shape latent codes cannot represent detailed shape geometries, and they often lead to learning shape recognition rather than shape reconstruction, as pointed out by "What do single-view 3d reconstruction networks learn?" #REFR in 2019.

Similar to Section 4, a significant amount of works in 3D deep learning use a global shape latent code to encode the shape, such as SurfNet #OTHEREFR , 3D-R2N2 #OTHEREFR , OGN #OTHEREFR , HSP #OTHEREFR , ShapeHD #OTHEREFR , AtlasNet #OTHEREFR , Im2Struct #OTHEREFR , Matryoshka Networks #OTHEREFR , SkeletonNet #OTHEREFR , IM-Net #OTHEREFR , OccNet #OTHEREFR , Deep Level Sets #OTHEREFR , Deep Meta Functionals #OTHEREFR , topology-modifying AtlasNet #OTHEREFR , Pix2Vox #OTHEREFR , PQ-NET #OTHEREFR , BSP-Net #OTHEREFR , Cvxnet #OTHEREFR , LDIF #OTHEREFR , Neural Template #OTHEREFR , AutoSDF #OTHEREFR .

That is, an encoder-decoder structured neural network with a global shape latent code is likely to simply memorize the shapes in the training set during training, and "retrieve" a shape from the memory bank as output during testing.

Therefore, in the following, we will only discuss works that take local features into account.

Deforming a sphere mesh with local image features.

Pixel2Mesh #OTHEREFR progressively deforms and subdivides a sphere mesh via graph convolutional networks.

It extracts image features with a VGG-16 CNN, and then pools image features into the vertices of the mesh to enrich the vertex features, so that the graph convolutional networks can learn local-feature-aware deformations.

A Review Of Deep Learning-Powered Mesh Reconstruction Methods

Abstract: With the recent advances in hardware and rendering techniques, 3D models have emerged everywhere in our life. Yet creating 3D shapes is arduous and requires significant professional knowledge. Meanwhile, Deep learning has enabled high-quality 3D shape reconstruction from various sources, making it a viable approach to acquiring 3D shapes with minimal effort. Importantly, to be used in common 3D applications, the reconstructed shapes need to be represented as polygonal meshes, which is a challenge for neural networks due to the irregularity of mesh tessellations. In this survey, we provide a comprehensive review of mesh reconstruction methods that are powered by machine learning. We first describe various representations for 3D shapes in the deep learning context. Then we review the development of 3D mesh reconstruction methods from voxels, point clouds, single images, and multi-view images. Finally, we identify several challenges in this field and propose potential future directions.

What Do Single-View 3D Reconstruction Networks Learn?

Abstract: Convolutional networks for single-view object reconstruction have shown impressive performance and have become a popular subject of research. All existing techniques are united by the idea of having an encoder-decoder network that performs non-trivial reasoning about the 3D structure of the output space. In this work, we set up two alternative approaches that perform image classification and retrieval respectively. These simple baselines yield better results than state-of-the-art methods, both qualitatively and quantitatively. We show that encoder-decoder methods are statistically indistinguishable from these baselines, thus indicating that the current state of the art in single-view object reconstruction does not actually perform reconstruction but image classification. We identify aspects of popular experimental procedures that elicit this behavior and discuss ways to improve the current state of research.

Global shape latent
single-view 3d reconstruction


This vulnerability is also known in the literature as "external-function" #REFR or "SWC-100: Function Default Visibility" (SmartContractSecurity, 2020).

This defect refers to the case of functions that are written solely to be used by other contracts (i.e., not within the contract).

Such functions should be marked with the external modifier instead of public. The public modifier allows both external and internal calls.

Marking a function with external results in gas savings, as every invocation will be using calldata (a special memory region to store arguments, which cannot be later modified by the function) and can avoid unnecessary read and write operations to memory, which occur with internal calls (i.e., that do not use calldata).

Openscv: An Open Hierarchical Taxonomy For Smart Contract Vulnerabilities

Abstract: Smart contracts are nowadays at the core of most blockchain systems, as they specify and allow an agreement between entities that wish to perform a transaction. As any computer program, smart contracts are subject to the presence of residual faults, including severe security vulnerabilities, which require that the vulnerable contract is terminated in the blockchain. In this context, research began to be developed to prevent the deployment of smart contract holding vulnerabilities, mostly in the form of vulnerability detection tools. Along with these efforts, several and heterogeneous vulnerability classification schemes arised (e.g., most notably DASP and SWC). At the time of writing, these are mostly outdated initiatives, despite the fact that smart contract vulnerabilities are continuously being discovered and the associated rich information being mostly disregarded. In this paper, we propose OpenSCV, a new and Open hierarchical taxonomy for Smart Contract Vulnerabilities, which is open to community contributions and matches the current state of the practice, while being prepared to handle future modifications and evolution. The taxonomy was built based on the analysis of research on vulnerability classification, community-maintained classification schemes, and research on smart contract vulnerability detection. We show how Open-SCV covers the announced detection ability of current vulnerability detection tools, and highlight its usefulness as a resource in smart contract vulnerability research.

Securify: Practical Security Analysis Of Smart Contracts

Abstract: Permissionless blockchains allow the execution of arbitrary programs (called smart contracts), enabling mutually untrusted entities to interact without relying on trusted third parties. Despite their potential, repeated security concerns have shaken the trust in handling billions of USD by smart contracts. ::: To address this problem, we present Securify, a security analyzer for Ethereum smart contracts that is scalable, fully automated, and able to prove contract behaviors as safe/unsafe with respect to a given property. Securify's analysis consists of two steps. First, it symbolically analyzes the contract's dependency graph to extract precise semantic information from the code. Then, it checks compliance and violation patterns that capture sufficient conditions for proving if a property holds or not. To enable extensibility, all patterns are specified in a designated domain-specific language. ::: Securify is publicly released, it has analyzed >18K contracts submitted by its users, and is regularly used to conduct security audits by experts. We present an extensive evaluation of Securify over real-world Ethereum smart contracts and demonstrate that it can effectively prove the correctness of smart contracts and discover critical violations.


In #REFR , Aaronson investigated the concept on quantum state complexity, leading to the state synthesis problem.

Classical and quantum complexity theory typically concentrates on the computational difficulty of solving problems with classical inputs and outputs.

However, quantum computers have the ability to handle not only classical problems, but also quantum tasks, such as synthesizing quantum states.

The most famous example is preparing ground states of a physical system #OTHEREFR , which even dates back to Feynman's original ideas #OTHEREFR .

Analogous tasks are also commonplace in quantum cryptography and generalized notions of the pseudorandomness, such as quantum money #OTHEREFR and pseudorandom quantum states #OTHEREFR .

This motivates the study of complexity of synthesizing quantum states.

This problem involves generating a quantum state ρ from the all-zero state based on a quantum circuit with a succinct description acting on n qubits with the depth up to exponential.

The resulting state ρ is supposed to be close to the designated target state |ψ 1 .

This problem is solvable in (quantum) polynomial space (PSPACE), scilicet a quantum computer running in exponential time but using polynomially many gates can generate a state that well approximates the target state.

Quantum computers are seemingly not capable of solving any PSPACE problem in polynomial time, while polynomially many messages interactive protocols with the help of an all-powerful and untrusted prover (known as interactive proofs, IP) captures the full computational power of polynomial-space computation, referred to as the celebrated IP = PSPACE theorem #OTHEREFR Sha92] .

A recent line of works #OTHEREFR INN + 22, MY23] initializes the study on the interactive state synthesis problem.

Quantum Merlin-Arthur Proof Systems For Synthesizing Quantum States

Abstract: Complexity theory typically focuses on the difficulty of solving computational problems using classical inputs and outputs, even with a quantum computer. In the quantum world, it is natural to apply a different notion of complexity, namely the complexity of synthesizing quantum states. We investigate a state-synthesizing counterpart of the class NP, referred to as stateQMA, which is concerned with preparing certain quantum states through a polynomial-time quantum verifier with the aid of a single quantum message from an allpowerful but untrusted prover. This is a subclass of the class stateQIP recently introduced by Rosenthal and Yuen (ITCS 2022), which permits polynomially many interactions between the prover and the verifier. Our main result consists of the basic properties of this class (as well as a variant with an exponentially small gap), such as error reduction, and its relationship to other fundamental state synthesizing classes, viz., states generated by uniform polynomial-time quantum circuits (stateBQP) and space-uniform polynomial-space quantum circuits (statePSPACE). Additionally, we demonstrate that stateQCMA is closed under perfect completeness. Our proof techniques are based on the quantum singular value transformation introduced by Gilyén, Su, Low, and Wiebe (STOC 2019), and its adaption to achieve exponential precision with a bounded space.

The Complexity Of Quantum States And Transformations: From Quantum Money To Black Holes

Abstract: These are lecture notes from a weeklong course in quantum complexity theory taught at the Bellairs Research Institute in Barbados, February 21-25, 2016. The focus is quantum circuit complexity---i.e., the minimum number of gates needed to prepare a given quantum state or apply a given unitary transformation---as a unifying theme tying together several topics of recent interest in the field. Those topics include the power of quantum proofs and advice states; how to construct quantum money schemes secure against counterfeiting; and the role of complexity in the black-hole information paradox and the AdS/CFT correspondence (through connections made by Harlow-Hayden, Susskind, and others). The course was taught to a mixed audience of theoretical computer scientists and quantum gravity / string theorists, and starts out with a crash course on quantum information and computation in general.


We found that the classification accuracy (ACC) is only 46% which is much lower than the ACC of the classifier in Faster R-CNN #REFR (about 90%).

Different solutions are proposed to leverage these models to enable a detector that can detect novel objects.

Some researchers #OTHEREFR use these models to generate pseudo-label and train their own model with pseudo-label and the image-caption pair, while other trains a prompt #OTHEREFR to turn the CLIP into a detector.

However, all these methods need additional caption and image data, which may not available in the real-life zero-shot detection setting.

In contrast, ViLD #OTHEREFR learns a mapping from image regions to CLIP feature space by distillation for detecting the novel object, which is applicable when no additional data is provided.

To know how good the features are for distillation, we first apply CLIP to classify the instances in the COCO #OTHEREFR dataset.

This indicates the domain gap between the training data of CLIP and the detection dataset, making the mapping from the image region to the vision-language feature space harder to learn.

In addition, since the distillation is conducted on some specific image regions, how to select such a region is an important question.

ViLD uses the proposals from an RPN trained with base category annotations as distillation regions.

These proposals bias toward the region with base categories instances, which limits the novel information obtained by the detector and harms the distillation efficiency. Therefore, ViLD suffers from an extremely long training schedule.

To address these problems, we propose Efficient Feature Distillation for Zero-Shot Detection (EZSD).

Efficient Feature Distillation For Zero-Shot Detection

Abstract: The large-scale vision-language models (e.g., CLIP) are leveraged by different methods to detect unseen objects. However, most of these works require additional captions or images for training, which is not feasible in the context of zero-shot detection. In contrast, the distillation-based method is an extra-data-free method, but it has its limitations. Specifically, existing work creates distillation regions that are biased to the base categories, which limits the distillation of novel category information and harms the distillation efficiency. Furthermore, directly using the raw feature from CLIP for distillation neglects the domain gap between the training data of CLIP and the detection datasets, which makes it difficult to learn the mapping from the image region to the vision-language feature space-an essential component for detecting unseen objects. As a result, existing distillation-based methods require an excessively long training schedule. To solve these problems, we propose Efficient feature distillation for Zero-Shot Detection (EZSD). Firstly, EZSD adapts the CLIP's feature space to the target detection domain by re-normalizing CLIP to bridge the domain gap; Secondly, EZSD uses CLIP to generate distillation proposals with potential novel instances, to avoid the distillation being overly biased to the base categories. Finally, EZSD takes advantage of semantic meaning for regression to further improve the model performance. As a result, EZSD achieves state-of-the-art performance in the COCO zero-shot benchmark with a much shorter training schedule and outperforms previous work by 4% in LVIS overall setting with 1/10 training time.

Faster R-Cnn: Towards Real-Time Object Detection With Region Proposal Networks

Abstract: State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with ’attention’ mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps ( including all steps ) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available.


Lemma 2 (Lemma 4.1 in #REFR ): For any δ ∈ (0, 1), with probability at least 1 − δ, P ∈ P i for all i.

This allows us to update the occupancy measure efficiently by simply solving a linear program over the polytope Q Pi .

Lemma 1 (Theorem 4.2 in #OTHEREFR ): If P i is defined by Eq.

(12), Q Pi is a non-empty polytope and can be described by

Q Pi = Q ∩ q ∈ R XAX + , ǫ ∈ R XAX + q(x, a, x ′ ) −P i (x ′ |x, a) y∈X h+1 q(x, a, y) ≤ ǫ(x, a, x ′ ), ∀x ∈ X h , a ∈ A, x ′ ∈ X h+1 , h ∈ [L − 1], P i (x ′ |x, a) y∈X h+1 q(x, a, y) − q(x, a, x ′ ) ≤ ǫ(x, a, x ′ ), ∀x ∈ X h , a ∈ A, x ′ ∈ X h+1 , h ∈ [L − 1], x ′ ∈X h+1 ǫ(x, a, x ′ ) ≤ ǫ i (x, a) x ′ ∈X h+1 q(x, a, x ′ ), ∀x ∈ X h , a ∈ A, h ∈ [L − 1] . (13)

Next, we consider the following lemma, which states that P i contains P with high probability.

Similarly, the empirical estimation of r in epoch i is

where t(i) denotes the starting time of epoch i, I h denotes the realized reward at step h in round t.

The confidence radius, also referred to as "bonus term," which comes from the incentive for exploration, is defined by

Online Reinforcement Learning In Markov Decision Process Using Linear Programming

Abstract: We consider online reinforcement learning in episodic Markov decision process (MDP) with an unknown transition matrix and stochastic rewards drawn from a fixed but unknown distribution. The learner aims to learn the optimal policy and minimize their regret over a finite time horizon through interacting with the environment. We devise a simple and efficient model-based algorithm that achieves O(LX √ T A) regret with high probability, where L is the episode length, T is the number of episodes, and X and A are the cardinalities of the state space and the action space, respectively. The proposed algorithm, which is based on the concept of "optimism in the face of uncertainty", maintains confidence sets of transition and reward functions and uses occupancy measures to connect the online MDP with linear programming. It achieves a tighter regret bound compared to the existing works that use a similar confidence sets framework and improves the computational effort compared to those that use a different framework but with a slightly tighter regret bound.

Online Convex Optimization In Adversarial Markov Decision Processes

Abstract: We consider online learning in episodic loop-free Markov decision processes (MDPs), where the loss function can change arbitrarily between episodes, and the transition function is not known to the learner. We show $\tilde{O}(L|X|\sqrt{|A|T})$ regret bound, where $T$ is the number of episodes, $X$ is the state space, $A$ is the action space, and $L$ is the length of each episode. Our online algorithm is implemented using entropic regularization methodology, which allows to extend the original adversarial MDP model to handle convex performance criteria (different ways to aggregate the losses of a single episode) , as well as improve previous regret bounds.


We base our architecture off the conditional version of StyleGAN2 #REFR , comprised of two networks G = G • M .

The mapping network w = M (z, c) maps the inputs into a "style" vector w, which modulates a series of upsampling convolutional layers in the synthesis network G(w) to map a learned constant tensor to an output image x.

Convolution is the main engine to generate all output pixels, with the w vector as the only source of information to model conditioning.

To handle the highly diverse distribution of internet images, we aim to increase the capacity of convolution kernels.

However, increasing the width of the convolution layers becomes too demanding, as the same operation is repeated across all locations.

Scaling Up Gans For Text-To-Image Synthesis

Abstract: The recent success of text-to-image synthesis has taken the world by storm and captured the general public's imagination. From a technical standpoint, it also marked a drastic change in the favored architecture to design generative image models. GANs used to be the de facto choice, with techniques like StyleGAN. With DALL•E 2, autoregressive and diffusion models became the new standard for largescale generative models overnight. This rapid shift raises a fundamental question: can we scale up GANs to benefit from large datasets like LAION? We find that naïvely increasing the capacity of the StyleGAN architecture quickly becomes unstable. We introduce GigaGAN, a new GAN architecture that far exceeds this limit, demonstrating GANs as a viable option for text-to-image synthesis. GigaGAN offers three major advantages. First, it is orders of magnitude faster at inference time, taking only 0.13 seconds to synthesize a 512px image. Second, it can synthesize high-resolution images, for example, 16-megapixel images in 3.66 seconds. Finally, GigaGAN supports various latent space editing applications such as latent interpolation, style mixing, and vector arithmetic operations.

Analyzing And Improving The Image Quality Of Stylegan

Abstract: The style-based GAN architecture (StyleGAN) yields state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of its characteristic artifacts, and propose changes in both model architecture and training methods to address them. In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images. In addition to improving image quality, this path length regularizer yields the additional benefit that the generator becomes significantly easier to invert. This makes it possible to reliably attribute a generated image to a particular network. We furthermore visualize how well the generator utilizes its output resolution, and identify a capacity problem, motivating us to train larger models for additional quality improvements. Overall, our improved model redefines the state of the art in unconditional image modeling, both in terms of existing distribution quality metrics as well as perceived image quality.


In general, our CNDE-based methods are more stable over a long prediction period and show much better performance than other baselines. #REFR We also observe that the temporal model (e.g.

According to Table 1 and Table 2 , the CNDE-based methods in general outperform other baselines for velocity components { , , }.

We also have several observations from these tables: (1) When comparing the CNDE-based methods with SR baselines and DCS/MS model, we observe that these baseline methods cannot recover turbulent flow well and get the worse performance in terms of SSIM and Laplacian difference. Temporal Analysis.

In the temporal analysis from the FIT dataset, we show the change in performance as we reconstruct DNS data over 20 time steps after the training data.

The performance change using SSIM and Laplacian difference is shown in Fig. 4 and Fig. 5 , respectively.

Several observations are highlighted from these results: (1) With larger time intervals between training data and prediction data, the performance becomes worse.

LSTM) can bring significant improvement in longterm prediction by comparing DCS/MS with CTN.

(3) By comparing CNDEp-based methods and CNDE-based methods, the CNDE-based methods using refinement outperform the CNDEp-based methods without using this adjustment over time.

It demonstrates that the refinement method can adjust the state bias in the long-term prediction of flow data.

#OTHEREFR We also find that CNDEp-based methods achieve better performance after the 5th time steps compared with the temporal baseline CTN model.

It proves the advantage of the proposed RKTU structure in the long-term prediction.

Reconstructing Turbulent Flows Using Physics-Aware Spatio-Temporal Dynamics And Test-Time Refinement

Abstract: Simulating turbulence is critical for many societally important applications in aerospace engineering, environmental science, the energy industry, and biomedicine. Large eddy simulation (LES) has been widely used as an alternative to direct numerical simulation (DNS) for simulating turbulent flows due to its reduced computational cost. However, LES is unable to capture all of the scales of turbulent transport accurately. Reconstructing DNS from low-resolution LES is critical for many scientific and engineering disciplines, but it poses many challenges to existing super-resolution methods due to the spatio-temporal complexity of turbulent flows. In this work, we propose a new physics-guided neural network for reconstructing the sequential DNS from low-resolution LES data. The proposed method leverages the partial differential equation that underlies the flow dynamics in the design of spatio-temporal model architecture. A degradation-based refinement method is also developed to enforce physical constraints and further reduce the accumulated reconstruction errors over long periods. The results on two different types of turbulent flow data confirm the superiority of the proposed method in reconstructing the high-resolution DNS data and preserving the physical characteristics of flow transport.

Fast, Accurate, And Lightweight Super-Resolution With Cascading Residual Network

Abstract: In recent years, deep learning methods have been successfully applied to single-image super-resolution tasks. Despite their great performances, deep learning methods cannot be easily applied to real-world applications due to the requirement of heavy computation. In this paper, we address this issue by proposing an accurate and lightweight deep network for image super-resolution. In detail, we design an architecture that implements a cascading mechanism upon a residual network. We also present variant models of the proposed cascading residual network to further improve efficiency. Our extensive experiments show that even with much fewer parameters and operations, our models achieve performance comparable to that of state-of-the-art methods.


More details of the ProGAN architecture used in this study can be found in #REFR .

The ProGANs were trained on 4 NVIDIA Quadro RTX 8000 GPUs by use of Tensorflow #OTHEREFR .

A stochastic gradient method that employs the Adam algorithm #OTHEREFR was employed as the optimizer to train the ProGANs.

A ProGAN architecture with the initial image resolution of 4 × 4 and a 64dimensional latent space were employed.

The ProGANs were trained by use of the publicly available ProGAN code (https://github.com/tkarras/ progressive_growing_of_gans).

After the ProGAN was trained, the MCMC technique with pCN proposal was employed to sample the GAN's latent variable for use in computing the IO test statistic.

The corresponding proposal density function and the acceptance probability were defined in Eq. 15 and Eq. 16, respectively.

Each Markov chain was run for 200,000 iterations on a single NVIDIA Quadro RTX 8000 GPU.

A burn-in period corresponding to the first 10,000 iterations was discarded from each Markov chain.

Ideal Observer Computation By Use Of Markov-Chain Monte Carlo With Generative Adversarial Networks

Abstract: Medical imaging systems are often evaluated and optimized via objective, or task-specific, measures of image quality (IQ) that quantify the performance of an observer on a specific clinically-relevant task. The performance of the Bayesian Ideal Observer (IO) sets an upper limit among all observers, numerical or human, and has been advocated for use as a figure-of-merit (FOM) for evaluating and optimizing medical imaging systems. However, the IO test statistic corresponds to the likelihood ratio that is intractable to compute in the majority of cases. A sampling-based method that employs Markov-Chain Monte Carlo (MCMC) techniques was previously proposed to estimate the IO performance. However, current applications of MCMC methods for IO approximation have been limited to a small number of situations where the considered distribution of to-be-imaged objects can be described by a relatively simple stochastic object model (SOM). As such, there remains an important need to extend the domain of applicability of MCMC methods to address a large variety of scenarios where IO-based assessments are needed but the associated SOMs have not been available. In this study, a novel MCMC method that employs a generative adversarial network (GAN)-based SOM, referred to as MCMC-GAN, is described and evaluated. The MCMC-GAN method was quantitatively validated by use of test-cases for which reference solutions were available. The results demonstrate that the MCMC-GAN method can extend the domain of applicability of MCMC methods for conducting IO analyses of medical imaging systems.

Progressive Growing Of Gans For Improved Quality, Stability, And Variation

Abstract: We describe a new training methodology for generative adversarial networks. The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses. This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2. We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10. Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator. Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation. As an additional contribution, we construct a higher-quality version of the CelebA dataset.


The candidate images for the original dataset are crawled from Google, MSN, Yahoo, and Flickr #REFR .

The original annotation procedure consists of the following four stages [77, #OTHEREFR .

We replicate only the crowdsourcing stages (2) and (3) that are directly related to the generation of annotation byproducts.

Our replication is based on the description in the original ImageNet [77] and ImageNetV2 #OTHEREFR papers.

For stage (1), we use the 1,000-class subset of the original 21,841 WordNet concepts #OTHEREFR , corresponding to the ILSVRC2012 subset, also known as the ImageNet1K [77] .

Preparing candidate images I cand c for each class c ∈ C.

The search keywords are formulated by combining the class names and their "synsets" in WordNet #OTHEREFR Figure 3 : ImageNet annotation interface. We replicate the interface in #OTHEREFR .

Annotators read the category description in the instruction panel, select all the images corresponding to "French bulldog", and click on the submit button.

the front-end and back-end to record and save the annotation byproducts in the database.

The annotation interface and crowdsourcing details are explained in Appendix C.1. Number of annotators per image.

The original ImageNet annotation procedure presents each image to 10 annotators for more precise annotations. This would require 240k USD for the annotation.

Neglected Free Lunch -Learning Image Classifiers Using Annotation Byproducts

Abstract: Supervised learning of image classifiers distills human knowledge into a parametric model f θ through pairs of images and corresponding labels {(X i , Y i)} N i=1. We argue that this simple and widely used representation of human knowledge neglects rich auxiliary information from the annotation procedure, such as the time-series of mouse traces and clicks left after image selection. Our insight is that such annotation byproducts Z provide approximate human attention that weakly guides the model to focus on the foreground cues, reducing spurious correlations and discouraging shortcut learning. To verify this, we create ImageNet-AB and COCO-AB. They are ImageNet and COCO training sets enriched with sample-wise annotation byproducts, collected by replicating the respective original annotation tasks. We refer to the new paradigm of training models with annotation byproducts as learning using annotation byproducts (LUAB). We show that a simple multitask loss for regressing Z together with Y already improves the generalisability and robustness of the learned models. Compared to the original supervised learning, LUAB does not require extra annotation costs. ImageNet-AB and COCO-AB are at github.com/naverai/NeglectedFreeLunch.

Do Imagenet Classifiers Generalize To Imagenet?

Abstract: We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.


The drawback of #REFR is the ignorance of correlated patterns in historical samples of the load and renewable generation.

The iterative methods in #OTHEREFR - #OTHEREFR focus on solving a particular instance of the UC problem.

Therefore, they can be practical in daily electricity market operations, but they encounter significant performance challenges in the context of short-term operations.

Alternatively, the notion of umbrella constraint was introduced when identifying redundant constraints which do not alter the feasibility region of the original UC problem when they are removed from the original problem #OTHEREFR .

Similarly, references #OTHEREFR - #OTHEREFR use a bound tightening technique #OTHEREFR and solve two optimization problems for each transmission line in the power system over linear programming (LP) relaxations of the feasible region to remove as many redundant constraints as possible from the full UC formulation.

Also, the authors of #OTHEREFR used Clarkson's redundancy removal algorithm that is based on LP and requires solving multiple maximization problems #OTHEREFR for each possible line flow contingency constraint in a SCOPF.

Recently, an analytical approach was proposed by #OTHEREFR based on a constraint generation algorithm and heuristics to eliminate nonbinding constraints from UC problems on top of redundant constraints.

In addition, the variability and uncertainty of renewable power generation have introduced new challenges to the UC problem #OTHEREFR .

The representation of uncertainty can take the form of scenarios or uncertainty sets for stochastic and robust optimization approaches, respectively.

The computational burden of stochastic programming prompts to migrate towards tractable approaches for optimization under uncertainty, namely robust and chance-constrained optimization #OTHEREFR , #OTHEREFR .

Moreover, renewable energy sources (RES) manifest crosscorrelation over space and time.

Tight Data-Driven Linear Relaxations For Constraint Screening In Robust Unit Commitment

Abstract: and with the Groupe d'études et de recherche en analyse des décisions (GERAD),

Fast Security-Constrained Optimal Power Flow Through Low-Impact And Redundancy Screening

Abstract: Network-constrained dispatch decisions are not only restricted by the thermal limits of the power lines, but also by complex security requirements related to contingency scenarios, e.g. the N-1-criterion. For real-world networks, the security-constrained optimal power flow (SCOPF) leads to a prohibitive increase in complexity even for linearized (DC) power flow equations. This paper presents a set of methods for fast, yet exact, SCOPF computation by reducing the power flow and contingency constraints to a minimal subset. The constraint reduction is performed by an iterative algorithm with performance guarantees based on both general algebraic theory and exploiting peculiarities of the optimal dispatch problem. The method is showcased on the classic IEEE 118-bus test system and on a full-scale real-world system. The numerical experiments rely on open-source software and data and our implementation is published supplementary to the paper.


Our model can handle multi-label inpainting: we extend a previous method #REFR to include tissue labels within dropout regions rather than a binary mask.

Additionally, since the dropout masks contain not just CSF but all tissue types, the model learns to synthesize tissue boundaries realistically.

Since our downstream goal is cortical surface reconstruction, a plausible GM/CSF boundary is critical.

Once trained, we apply our model to synthetically atrophied images obtained using a registration-based method. Our main contributions are:

The use of CSF inpainting for synthetic cortical atrophy induction is the first of its kind, and provides an interesting new challenge for the MIDL community. 2.

We combine several existing 2D methods and extend them to 3D, including a selfsupervised dropout scheme #OTHEREFR , edge priors #OTHEREFR , and sinusoidal positional encoding (SPE) #OTHEREFR . 4.

Our model synthesizes better images than 2D methods and can adapt to unseen data.

Figure 2: Self-supervised training pipeline for a T1w.

We pre-processed (green) T1w and FLAIR images (gray) to obtain 5 input channels.

Self-Supervised Csf Inpainting With Synthetic Atrophy For Improved Accuracy Validation Of Cortical Surface Analyses

Abstract: Accuracy validation of cortical thickness measurement is a difficult problem due to the lack of ground truth data. To address this need, many methods have been developed to synthetically induce gray matter (GM) atrophy in an MRI via deformable registration, creating a set of images with known changes in cortical thickness. However, these methods often cause blurring in atrophied regions, and cannot simulate realistic atrophy within deep sulci where cerebrospinal fluid (CSF) is obscured or absent. In this paper, we present a solution using a self-supervised inpainting model to generate CSF in these regions and create images with more plausible GM/CSF boundaries. Specifically, we introduce a novel, 3D GAN model that incorporates patch-based dropout training, edge map priors, and sinusoidal positional encoding, all of which are established methods previously limited to 2D domains. We show that our framework significantly improves the quality of the resulting synthetic images and is adaptable to unseen data with fine-tuning. We also demonstrate that our resulting dataset can be employed for accuracy validation of cortical segmentation and thickness measurement.

Context Encoders: Feature Learning By Inpainting

Abstract: We present an unsupervised visual feature learning algorithm driven by context-based pixel prediction. By analogy with auto-encoders, we propose Context Encoders -- a convolutional neural network trained to generate the contents of an arbitrary image region conditioned on its surroundings. In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s). When training context encoders, we have experimented with both a standard pixel-wise reconstruction loss, as well as a reconstruction plus an adversarial loss. The latter produces much sharper results because it can better handle multiple modes in the output. We found that a context encoder learns a representation that captures not just appearance but also the semantics of visual structures. We quantitatively demonstrate the effectiveness of our learned features for CNN pre-training on classification, detection, and segmentation tasks. Furthermore, context encoders can be used for semantic inpainting tasks, either stand-alone or as initialization for non-parametric methods.


However, after the substi-tute model is well trained with sufficient data #REFR , BagAmmo performs more efficiently and exhibits its advantage. . We can draw two conclusions from this figure. First, the package-level classifier is more difficult to attack.

The vertical axis reflects perturbation ratio, and the horizontal axis indicates the number of queries.

It can be seen that in all cases, BagAmmo always has a higher convergence speed.

Moreover, BagAmmo always requires fewer queries before the perturbation ratio is kept below a certain threshold (e.g., 0.1).

Note that the difference between two methods in the initial phase is relatively small.

It is because the substitute model has not been well trained in this phase.

This is because package-level FCGs contain much more nodes than family-level FCGs, resulting in a larger search space for BagAmmo.

Second, using the substitute model reduces the number of queries in almost all cases and helps enhance the attack efficiency.

Black-Box Adversarial Example Attack Towards Fcg Based Android Malware Detection Under Incomplete Feature Information

Abstract: The function call graph (FCG) based Android malware detection methods have recently attracted increasing attention due to their promising performance. However, these methods are susceptible to adversarial examples (AEs). In this paper, we design a novel black-box AE attack towards the FCG based malware detection system, called BagAmmo. To mislead its target system, BagAmmo purposefully perturbs the FCG feature of malware through inserting "never-executed" function calls into malware code. The main challenges are twofold. First, the malware functionality should not be changed by adversarial perturbation. Second, the information of the target system (e.g., the graph feature granularity and the output probabilities) is absent. To preserve malware functionality, BagAmmo employs the try-catch trap to insert function calls to perturb the FCG of malware. Without the knowledge about feature granularity and output probabilities, BagAmmo adopts the architecture of generative adversarial network (GAN), and leverages a multi-population co-evolution algorithm (i.e., Apoem) to generate the desired perturbation. Every population in Apoem represents a possible feature granularity, and the real feature granularity can be achieved when Apoem converges. Through extensive experiments on over 44k Android apps and 32 target models, we evaluate the effectiveness, efficiency and resilience of BagAmmo. BagAmmo achieves an average attack success rate of over 99.9% on MaMaDroid, APIGraph and GCN, and still performs well in the scenario of concept drift and data imbalance. Moreover, BagAmmo outperforms the state-of-the-art attack SRL in attack success rate.

Android Hiv: A Study Of Repackaging Malware For Evading Machine-Learning Detection

Abstract: Machine learning-based solutions have been successfully employed for the automatic detection of malware on Android. However, machine learning models lack robustness to adversarial examples, which are crafted by adding carefully chosen perturbations to the normal inputs. So far, the adversarial examples can only deceive detectors that rely on syntactic features ( e.g. , requested permissions, API calls, etc. ), and the perturbations can only be implemented by simply modifying application’s manifest. While recent Android malware detectors rely more on semantic features from Dalvik bytecode rather than manifest, existing attacking/defending methods are no longer effective. In this paper, we introduce a new attacking method that generates adversarial examples of Android malware and evades being detected by the current models. To this end, we propose a method of applying optimal perturbations onto Android APK that can successfully deceive the machine learning detectors. We develop an automated tool to generate the adversarial examples without human intervention. In contrast to existing works, the adversarial examples crafted by our method can also deceive recent machine learning-based detectors that rely on semantic features such as control-flow-graph. The perturbations can also be implemented directly onto APK’s Dalvik bytecode rather than Android manifest to evade from recent detectors. We demonstrate our attack on two state-of-the-art Android malware detection schemes, MaMaDroid and Drebin. Our results show that the malware detection rates decreased from 96% to 0% in MaMaDroid, and from 97% to 0% in Drebin, with just a small number of codes to be inserted into the APK.


B. Self-Supervised Learning In Remote Sensing

Generative adversarial networks (GANs) #REFR and reconstruction with auto-encoding are primarily used for representation learning.

However, further attempts are being made to learn multi-sensor and temporal information #OTHEREFR , #OTHEREFR , #OTHEREFR .

4) Generative learning: The primary goal of generative learning is to train a model to reconstruct an original input even when it is partially corrupted.

In general computer vision, removing spatial and temporal information from the original input is employed.

Similarly, a comparable method is used in the field of remote sensing.

Generation from missing spatial information involves learning features by reconstructing the corrupted area of the original image using the characteristics of the surrounding space.

In MARTA GANs #OTHEREFR , which is related to generative adversarial networks, the discriminator serves as the encoder for the scene classification model.

On the other hand, numerous studies focus on reconstruction with auto-encoding #OTHEREFR , #OTHEREFR , #OTHEREFR .

The auto-encoding method primarily employs a corrupted image as input with an encoder-decoder structure model using L1 or L2 distance metrics.

Additionally, a study on reconstruction by reflecting the characteristics of ground sample distance (GSD), an essential feature in remote sensing images, has been conducted #OTHEREFR .

Generation from missing temporal information is similar to video frame prediction in the computer vision field.

A Billion-Scale Foundation Model For Remote Sensing Images

Abstract: As the potential of foundation models in visual tasks has garnered significant attention, pretraining these models before downstream tasks has become a crucial step. The three key factors in pretraining foundation models are the pretraining method, the size of the pretraining dataset, and the number of model parameters. Recently, research in the remote sensing field has focused primarily on the pretraining method and the size of the dataset, with limited emphasis on the number of model parameters. This paper addresses this gap by examining the effect of increasing the number of model parameters on the performance of foundation models in downstream tasks such as rotated object detection and semantic segmentation. We pretrained foundation models with varying numbers of parameters, including 86M, 605.26M, 1.3B, and 2.4B, to determine whether performance in downstream tasks improved with an increase in parameters. To the best of our knowledge, this is the first billionscale foundation model in the remote sensing field. Furthermore, we propose an effective method for scaling up and fine-tuning a vision transformer in the remote sensing field. To evaluate general performance in downstream tasks, we employed the DOTA v2.0 and DIOR-R benchmark datasets for rotated object detection, and the Potsdam and LoveDA datasets for semantic segmentation. Experimental results demonstrated that, across all benchmark datasets and downstream tasks, the performance of the foundation models and data efficiency improved as the number of parameters increased. Moreover, our models achieve the state-of-the-art performance on several datasets including DIOR-R, Postdam, and LoveDA.

Generative Adversarial Networks

Abstract: We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

representation learning
Generative adversarial networks