Data augmentation benefits generalizable, transferable, and robust representation learning by correctly extrapolating the input training space to a larger region #REFR .

Akin to hand-crafting Figure 1 : InfoTS is composed of three parts: (1) candidate transformation that generates different augmentations of the original inputs, (2) a meta-learner network that selects the optimal augmentations, (3) an encoder that learns representations of time series instances.

The meta-learner is learned in tandem with contrastive encoder learning.

features, hand-picking choices of data augmentations are undesirable from the learning perspective.

The diversity and heterogeneity of real-life time series data further hinder these methods away from wide applicability.

To address the challenges, we first introduce the criteria for selecting good data augmentations in contrastive learning.

The positive instances enclose a discriminative zone in which all the data points should be similar to the original instance.

The desired data augmentations for contrastive representation learning should have both high fidelity and high variety.

High fidelity encourages the augmented data to maintain the semantic identity that is invariant to transformations #OTHEREFR .

For example, if the downstream task is classification, then the generated augmentations of inputs should be classpreserving.

Meanwhile, generating augmented samples with high variety benefits representation learning by increasing the generalization capacity #OTHEREFR .

Time Series Contrastive Learning With Information-Aware Augmentations

Abstract: Various contrastive learning approaches have been proposed in recent years and achieve significant empirical success. While effective and prevalent, contrastive learning has been less explored for time series data. A key component of contrastive learning is to select appropriate augmentations imposing some priors to construct feasible positive samples, such that an encoder can be trained to learn robust and discriminative representations. Unlike image and language domains where "desired" augmented samples can be generated with the rule of thumb guided by prefabricated human priors, the ad-hoc manual selection of time series augmentations is hindered by their diverse and human-unrecognizable temporal structures. How to find the desired augmentations of time series data that are meaningful for given contrastive learning tasks and datasets remains an open question. In this work, we address the problem by encouraging both high fidelity and variety based upon information theory. A theoretical analysis leads to the criteria for selecting feasible data augmentations. On top of that, we propose a new contrastive learning approach with informationaware augmentations, InfoTS, that adaptively selects optimal augmentations for time series representation learning. Experiments on various datasets show highly competitive performance with up to 12.0% reduction in MSE on forecasting tasks and up to 3.7% relative improvement in accuracy on classification tasks over the leading baselines.

Learning Invariances Using The Marginal Likelihood

Abstract: Generalising well in supervised learning tasks relies on correctly extrapolating the training data to a large region of the input space. One way to achieve this is to constrain the predictions to be invariant to transformations on the input that are known to be irrelevant (e.g. translation). Commonly, this is done through data augmentation, where the training set is enlarged by applying hand-crafted transformations to the inputs. We argue that invariances should instead be incorporated in the model structure, and learned using the marginal likelihood, which correctly rewards the reduced complexity of invariant models. We demonstrate this for Gaussian process models, due to the ease with which their marginal likelihood can be estimated. Our main contribution is a variational inference scheme for Gaussian processes containing invariances described by a sampling procedure. We learn the sampling procedure by back-propagating through it to maximise the marginal likelihood.

Data augmentation
representation learning


In addition to its adaptability and high precision, it can compute the camera trajectory and a sparse 3D reconstruction of the world in real-time for a variety of situations, from brief desk-related hand-held sequences to car driving through many city blocks #REFR .

ORB-SLAM3 is a visual sensor-based SLAM solution designed for Monocular, Binocular, Stereo, and RGB-D cameras #OTHEREFR .

The ORB-SLAM3 architecture's foundation is built using ORB-SLAM2 and ORBSLAM-VI #OTHEREFR .

It is a complete multimap and multi-session system that may operate in either visual-inertial or pure visual modes, that supports both pinhole and fisheye lens models.

The ORB-SLAM3 pipeline, with its primary system components, is depicted in Figure 2 .

Over the years, ORB-SLAM3 has proven to be more useful in low-light environments compared to ORB-SLAM2 because it incorporates improvements in several aspects that enhance its ability to operate in challenging lighting conditions.

Firstly, ORB-SLAM3 includes a new feature descriptor called COdes of Low Light Features (COLLF), which is designed specifi-cally to capture and match features in low-light environments.

Twilight Slam: A Comparative Study Of Low-Light Visual Slam Pipelines

Abstract: This paper presents a comparative study of lowlight visual SLAM pipelines, specifically focusing on determining an efficient combination of the state-of-the-art low-light image enhancement algorithms with standard and contemporary Simultaneous Localization and Mapping (SLAM) frameworks by evaluating their performance in challenging low-light conditions. In this study, we investigate the performance of several different low-light SLAM pipelines for dark and/or poorly-lit datasets as opposed to just partially dim-lit datasets like other works in the literature. Our study takes an experimental approach to qualitatively and quantitatively compare the chosen combinations of modules to enhance the feature-based visual SLAM.

Orb-Slam2: An Open-Source Slam System For Monocular, Stereo And Rgb-D Cameras

Abstract: We present ORB-SLAM2 a complete SLAM system for monocular, stereo and RGB-D cameras, including map reuse, loop closing and relocalization capabilities. The system works in real-time on standard CPUs in a wide variety of environments from small hand-held indoors sequences, to drones flying in industrial environments and cars driving around a city. Our back-end based on bundle adjustment with monocular and stereo observations allows for accurate trajectory estimation with metric scale. Our system includes a lightweight localization mode that leverages visual odometry tracks for unmapped regions and matches to map points that allow for zero-drift localization. The evaluation on 29 popular public sequences shows that our method achieves state-of-the-art accuracy, being in most cases the most accurate SLAM solution. We publish the source code, not only for the benefit of the SLAM community, but with the aim of being an out-of-the-box SLAM solution for researchers in other fields.

sparse 3D reconstruction
camera trajectory


Specifically, we compare SAM to a U-Net model #REFR that has been highly optimized for the benchmark in a prior study.

For a given benchmark problem we also report the segmentation accuracy of two "task specific models"; these models are been trained with full supervision on that benchmark.

We refer to this as "UNet-Opt" and it approximates the state-ofthe-art performance that can be achieved on that benchmark, given substantial investment.

For the Unet-Opt model we report the results reported in a prior study involving the particular benchmark under consideration.

We also compare SAM with a conventional Unet model that we have trained here, reflecting the accuracy achievable without substantial investment, but assuming the availability of (often substantial) labeled training data. We refer this model as "UNet".

Segment Anything, From Space?

Abstract: Recently, the first foundation model developed specifically for vision tasks was developed, termed the "Segment Anything Model" (SAM). SAM can segment objects in input imagery based upon cheap input prompts, such as one (or more) points, a bounding box, or a mask. The authors examined the zero-shot image segmentation accuracy of SAM on a large number of vision benchmark tasks and found that SAM usually achieved recognition accuracy similar to, or sometimes exceeding, vision models that had been trained on the target tasks. The impressive generalization of SAM for segmentation has major implications for vision researchers working on natural imagery. In this work, we examine whether SAM's impressive performance extends to overhead imagery problems, and help guide the community's response to its development. We examine SAM's performance on a set of diverse and widely-studied benchmark tasks. We find that SAM does often generalize well to overhead imagery, although it fails in some cases due to the unique characteristics of overhead imagery and the target objects. We report on these unique systematic failure cases for remote sensing imagery that may comprise useful future research for the community. Note that this is a working paper, and it will be updated as additional analysis and results are completed.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .


The pix2pix structure belongs to the family of conditional generative adversarial networks (cGANs), comprising a discriminator network D and a generator network G, which cooperate in order to generate realistic reconstructions #REFR .

As an alternative to the U-Net model, we also evaluated the pix2pix neural network architecture.

In our experiments, the generator network maps an input image, X i ∈ R n×n , derived from the first stage, to an associated reconstruction, G(X i ) ∈ R n×n .

On the other hand, the discriminator network takes in a pair of images, (X, Y ) ∈ R n×n × R n×n , as input and produces a scalar-valued output ranging between 0 and 1.

The two networks are jointly trained in a competitive manner, wherein the generator network endeavors to generate images G(X i ) that resemble the target images Y i , while the discriminator network attempts to differentiate between real image pairs (X i , Y i ) and generated image pairs (X i , G(X i )), with the output indicating the probability that an input image pair is real (i.e., it originates from the training dataset) or generated (i.e., it was produced by the generator network). The training process is illustrated in Figure 3 .

The pix2pix algorithm utilizes a loss function consisting of two distinct components.

The first component, denoted as the adversarial loss, is expressed in terms of a binary cross-entropy loss, and is represented as follows:

A Data-Assisted Two-Stage Method For The Inverse Random Source Problem

Abstract: We propose a data-assisted two-stage method for solving an inverse random source problem of the Helmholtz equation. In the first stage, the regularized Kaczmarz method is employed to generate initial approximations of the mean and variance based on the mild solution of the stochastic Helmholtz equation. A dataset is then obtained by sampling the approximate and corresponding true profiles from a certain a-priori criterion. The second stage is formulated as an image-to-image translation problem, and several data-assisted approaches are utilized to handle the dataset and obtain enhanced reconstructions. Numerical experiments demonstrate that the data-assisted two-stage method provides satisfactory reconstruction for both homogeneous and inhomogeneous media with fewer realizations.

Image-To-Image Translation With Conditional Adversarial Networks

Abstract: We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of the pix2pix software associated with this paper, a large number of internet users (many of them artists) have posted their own experiments with our system, further demonstrating its wide applicability and ease of adoption without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.

realistic reconstructions
conditional generative adversarial


Existing query embedding methods (Zhang et al., 2021; Ren et al., 2020; #REFR ; Ren and Leskovec, 2020) apply multi-layer perceptron networks to accomplish this task.

As illustrated in Figure 1 , each logical query can be represented as a directed acyclic graph (DAG) tree, where the tree nodes correspond to constants/anchor node entities or variables, and the edges correspond to atom relations or logical operations in a query.

Logical operations are performed along the DAG tree from constants to the target answer variable. Figure 3 visualizes these operations on 2D complex space. The logical operators can be defined as follows

Relational rotating projection Given a set of entities S ⊂ E and a relation r ∈ R, the projection operator selects the neighbouring entities S ⊂ E by relation such that S = {e ∈ S, e ∈ S : r(e, e ) = T rue}.

They do not accommodate the learning of potential KG relational patterns which might help in reasoning logical queries.

Motivated by RotatE (Sun et al., 2019) , we represent each relation r as a counterclockwise relational rotation on query embeddings about the origin of the complex plane such that r = (r U , r L ), where |r {U,L} | = 1, and r U , r L ∈ C d .

Given the query embedding q = (h U , h L ) and a relation r, the selected query

where • is the Hadmard (element-wise) product, and θ θ θ ax,r , θ θ θ ap,r correspond to the equivalent relational rotation on θ θ θ ax , θ θ θ ap .

Modeling Relational Patterns For Logical Query Answering Over Knowledge Graphs

Abstract: Answering first-order logical (FOL) queries over knowledge graphs (KG) remains a challenging task mainly due to KG incompleteness. Query embedding approaches this problem by computing the low-dimensional vector representations of entities, relations, and logical queries. KGs exhibit relational patterns such as symmetry and composition and modeling the patterns can further enhance the performance of query embedding models. However, the role of such patterns in answering FOL queries by query embedding models has not been yet studied in the literature. In this paper, we fill in this research gap and empower FOL queries reasoning with pattern inference by introducing an inductive bias that allows for learning relation patterns. To this end, we develop a novel query embedding method, RoConE, that defines query regions as geometric cones and algebraic query operators by rotations in complex space. RoConE combines the advantages of Cone as a wellspecified geometric representation for query embedding, and also the rotation operator as a powerful algebraic operation for pattern inference. Our experimental results on several benchmark datasets confirm the advantage of relational patterns for enhancing logical query answering task.

Embedding Logical Queries On Knowledge Graphs

Abstract: Learning low-dimensional embeddings of knowledge graphs is a powerful approach used to predict unobserved or missing edges between entities. However, an open challenge in this area is developing techniques that can go beyond simple edge prediction and handle more complex logical queries, which might involve multiple unobserved edges, entities, and variables. For instance, given an incomplete biological knowledge graph, we might want to predict"em what drugs are likely to target proteins involved with both diseases X and Y?"-- a query that requires reasoning about all possible proteins that {\em might} interact with diseases X and Y. Here we introduce a framework to efficiently make predictions about conjunctive logical queries -- a flexible but tractable subset of first-order logic -- on incomplete knowledge graphs. In our approach, we embed graph nodes in a low-dimensional space and represent logical operators as learned geometric operations (e.g., translation, rotation) in this embedding space. By performing logical operations within a low-dimensional embedding space, our approach achieves a time complexity that is linear in the number of query variables, compared to the exponential complexity required by a naive enumeration-based approach. We demonstrate the utility of this framework in two application studies on real-world datasets with millions of relations: predicting logical relationships in a network of drug-gene-disease interactions and in a graph-based representation of social interactions derived from a popular web forum.


It should be noted that adversarial training #REFR is different from our method because the perturbation for the worst case is conducted on the input space, not on the parameter space.

Sharpness-Aware Minimization (SAM) (Foret et al., 2020) is an optimizer for the model parameter, which regularizes the locality region of θ to be the flat minima on the loss curvature as follows:

Here, is the perturbation vector to the parameter; and ρ denotes the maximum size of the perturbation vector.

As the objective is a function defined by both input and model parameter, it is possible to solve the generalization of a model parameter through the optimization of input data.

However, there is no such study, which improves the generalization of the perturbed parameter space via optimizing the input data variable, to the best of our knowledge.

Loss-Curvature Matching For Dataset Selection And Condensation

Abstract: Training neural networks on a large dataset requires substantial computational costs. Dataset reduction selects or synthesizes data instances based on the large dataset, while minimizing the degradation in generalization performance from the full dataset. Existing methods utilize the neural network during the dataset reduction procedure, so the model parameter becomes important factor in preserving the performance after reduction. By depending upon the importance of parameters, this paper introduces a new reduction objective, coined LCMat, which Matches the Loss Curvatures of the original dataset and reduced dataset over the model parameter space, more than the parameter point. This new objective induces a better adaptation of the reduced dataset on the perturbed parameter region than the exact point matching. Particularly, we identify the worst case of the loss curvature gap from the local parameter region, and we derive the implementable upper bound of such worst-case with theoretical analyses. Our experiments on both coreset selection and condensation benchmarks illustrate that LCMat shows better generalization performances than existing baselines.

Theoretically Principled Trade-Off Between Robustness And Accuracy

Abstract: We identify a trade-off between robustness and accuracy that serves as a guiding principle in the design of defenses against adversarial examples. Although this problem has been widely studied empirically, much remains unknown concerning the theory underlying this trade-off. In this work, we decompose the prediction error for adversarial examples (robust error) as the sum of the natural (classification) error and boundary error, and provide a differentiable upper bound using the theory of classification-calibrated loss, which is shown to be the tightest possible upper bound uniform over all probability distributions and measurable predictors. Inspired by our theoretical analysis, we also design a new defense method, TRADES, to trade adversarial robustness off against accuracy. Our proposed algorithm performs well experimentally in real-world datasets. The methodology is the foundation of our entry to the NeurIPS 2018 Adversarial Vision Challenge in which we won the 1st place out of ~2,000 submissions, surpassing the runner-up approach by $11.41\%$ in terms of mean $\ell_2$ perturbation distance.


It changes the global average pooling layer into an attention pooling mechanism, where a "QKV" attention is implemented #REFR .

For our reported results, we take accuracy as the metric and count results on the 30-second segment level.

Besides, we take ResNet #OTHEREFR with multi-head attention as the backbone according to our preliminary experiment.

Table 1 presents the results of our experiments on data pruning and regularization strategies based on three acoustic features on the Shipsear and Deepship datasets.

First, we observe that adaptive data pruning can consistently improve performance for various acoustic features and datasets.

These results further demonstrate the importance of addressing the double-descent issue.

We also find that data pruning works particularly well on low-resource Shipsear, which can bring up to a 6.03% performance improvement.

This is because the scarcity of data causes the model to be more likely to memorize repeated data and fall into local bias.

Advancing Underwater Acoustic Target Recognition Via Adaptive Data Pruning And Smoothness-Inducing Regularization

Abstract: Underwater acoustic recognition for ship-radiated signals has high practical application value due to the ability to recognize non-line-of-sight targets. However, due to the difficulty of data acquisition, the collected signals are scarce in quantity and mainly composed of mechanical periodic noise. According to the experiments, we observe that the repeatability of periodic signals leads to a double-descent phenomenon, which indicates a significant local bias toward repeated samples. To address this issue, we propose a strategy based on cross-entropy to prune excessively similar segments in training data. Furthermore, to compensate for the reduction of training data, we generate noisy samples and apply smoothness-inducing regularization based on KL divergence to mitigate overfitting. Experiments show that our proposed data pruning and regularization strategy can bring stable benefits and our framework significantly outperforms the state-of-the-art in low-resource scenarios.

Non-Local Neural Networks

Abstract: Both convolutional and recurrent operations are building blocks that process one local neighborhood at a time. In this paper, we present non-local operations as a generic family of building blocks for capturing long-range dependencies. Inspired by the classical non-local means method in computer vision, our non-local operation computes the response at a position as a weighted sum of the features at all positions. This building block can be plugged into many computer vision architectures. On the task of video classification, even without any bells and whistles, our non-local models can compete or outperform current competition winners on both Kinetics and Charades datasets. In static image recognition, our non-local models improve object detection/segmentation and pose estimation on the COCO suite of tasks. Code is available at this https URL .


After that, an automatic background cropping (based on monocular depth estimation #REFR ) is employed to remove the background objects.

Our automatic calibration of infrastructure LiDARs and cameras, which outputs the precise pose of these sensors, is the most fundamental part of the framework.

In order to calibrate the sensors in the real world, we propose an automatic target-less LiDAR-camera calibration model.

We use the calibration method proposed in #OTHEREFR as a baseline and extend it to outdoor scenes captured by infrastructure roadside sensors of a different manufacturer.

To improve the robustness of the model under different external conditions, such as different scene complexities, lighting conditions, or sensor conditions, we introduce various automatic preprocessing submodules (see Figure 3 ).

If there is shadow on the ground, the automatic shadow filtering module will be activated to filter the shadow.

After the preprocessing, the Canny edge detector #OTHEREFR is adopted to extract 2D edges Fig. 3 : Automatic calibration pipeline.

We integrate four camera image and seven LiDAR point cloud preprocessing modules into our pipeline in order to increase the robustness of real-world outdoor calibration of roadside sensors.

The algorithm takes the image and point cloud that is published continuously on the live system as input and outputs both, the calibration results and qualitative projections of point clouds into camera images.

Infradet3D: Multi-Modal 3D Object Detection Based On Roadside Infrastructure Camera And Lidar Sensors

Abstract: Current multi-modal object detection approaches focus on the vehicle domain and are limited in the perception range and the processing capabilities. Roadside sensor units (RSUs) introduce a new domain for perception systems and leverage altitude to observe traffic. Cameras and LiDARs mounted on gantry bridges increase the perception range and produce a full digital twin of the traffic. In this work, we introduce InfraDet3D, a multi-modal 3D object detector for roadside infrastructure sensors. We fuse two LiDARs using early fusion and further incorporate detections from monocular cameras to increase the robustness and to detect small objects. Our monocular 3D detection module uses HD maps to ground object yaw hypotheses, improving the final perception results. The perception framework is deployed on a real-world intersection that is part of the A9 Test Stretch in Munich, Germany. We perform several ablation studies and experiments and show that fusing two LiDARs with two cameras leads to an improvement of +1.90 mAP compared to a camera-only solution. We evaluate our results on the A9 infrastructure dataset and achieve 68.48 mAP on the test set. The dataset and code will be available at https://a9-dataset.com to allow the research community to further improve the perception results and make autonomous driving safer.

Towards Robust Monocular Depth Estimation: Mixing Datasets For Zero-Shot Cross-Dataset Transfer

Abstract: The success of monocular depth estimation relies on large and diverse training sets. Due to the challenges associated with acquiring dense ground-truth depth across different environments at scale, a number of datasets with distinct characteristics and biases have emerged. We develop tools that enable mixing multiple datasets during training, even if their annotations are incompatible. In particular, we propose a training objective that is invariant to changes in depth range and scale. Armed with this objective, we explore an abundant source of training data: 3D films. We demonstrate that despite pervasive inaccuracies, 3D films constitute a useful source of data that is complementary to existing training sets. We evaluate the presented approach on diverse datasets, focusing on zero-shot cross-dataset transfer: testing the generality of the learned model by evaluating it on datasets that were not seen during training. The experiments confirm that mixing data from complementary sources yields improved depth estimates, particularly on previously unseen datasets. Some results are shown in the supplementary video: this https URL


Following the prior work #REFR , the IOR (index of refraction) of air is set to 1.0003 and the IOR of transparent material (glass) is set to 1.4723.

A hierarchical sampling strategy is used to sample points along a ray in a coarse-to-fine manner for volume render-Li et al. #OTHEREFR DRT #OTHEREFR Ours ing.

We first uniformly sample 64 points along the ray, and then iteratively conduct importance sampling #OTHEREFR to sample more points on top of coarse probability estimation for 4 times.

The positional encoding is applied to the spatial location with 5 frequencies.

The hyper-parameters used in the experiments are set as ω 1 = 0.0001, ω 2 = 0.1, ω 3 = 0.1.

Neto:Neural Reconstruction Of Transparent Objects With Self-Occlusion Aware Refraction-Tracing

Abstract: We present a novel method, called NeTO, for capturing 3D geometry of solid transparent objects from 2D images via volume rendering. Reconstructing transparent objects is a very challenging task, which is ill-suited for generalpurpose reconstruction techniques due to the specular light transport phenomena. Although existing refraction-tracing based methods, designed specially for this task, achieve impressive results, they still suffer from unstable optimization and loss of fine details, since the explicit surface representation they adopted is difficult to be optimized, and the self-occlusion problem is ignored for refraction-tracing. In this paper, we propose to leverage implicit Signed Distance Function (SDF) as surface representation, and optimize the SDF field via volume rendering with a self-occlusion aware refractive ray tracing. The implicit representation enables our method to be capable of reconstructing high-quality reconstruction even with a limited set of images, and the selfocclusion aware strategy makes it possible for our method to accurately reconstruct the self-occluded regions. Experiments show that our method achieves faithful reconstruction results and outperforms prior works by a large margin. Visit our project page at https://www.xxlong. site/NeTO/.

Full 3D Reconstruction Of Transparent Objects

Abstract: This paper presents a fully automatic approach for reconstructing complete 3D shapes of transparent objects.


Compared to the NNQS results reported in #REFR , we see that the IS method with the symmetric function ansatz is able to better predict the overlap of the above state with the |GHZ n + .

In an experiment involving symmetric states, first studied in #OTHEREFR , the fidelity of |GHZ n + is estimated with respect to another mixed state of the form, ρ n,p = (1 − p) |GHZ n + GHZ n

+ | + p |GHZ n − GHZ n − | for p ∈ [0, 1]

It was been reported that for this mixed state example, generative models learned using the Neural Net Quantum States (NNQS) ansatz [7, #OTHEREFR struggled to find the correct fidelity as p got closer to 1.

The results of applying our method on this problem with a symmetric function ansatz are given in Fig. 4d .

Learning Energy Based Representations Of Quantum Many-Body States

Abstract: Efficient representation of quantum many-body states on classical computers is a problem of enormous practical interest. An ideal representation of a quantum state combines a succinct characterization informed by the system's structure and symmetries, along with the ability to predict the physical observables of interest. A number of machine learning approaches have been recently used to construct such classical representations [1-6] which enable predictions of observables [7] and accounts for physical symmetries [8]. However, the structure of a quantum state gets typically lost unless a specialized ansatz is employed based on prior knowledge of the system [9-12]. Moreover, most such approaches give no information about what states are easier to learn in comparison to others. Here, we propose a new generative energy-based representation of quantum many-body states derived from Gibbs distributions used for modeling the thermal states of classical spin systems. Based on the prior information on a family of quantum states, the energy function can be specified by a small number of parameters using an explicit low-degree polynomial or a generic parametric family such as neural nets, and can naturally include the known symmetries of the system. Our results show that such a representation can be efficiently learned from data using exact algorithms in a form that enables the prediction of expectation values of physical observables. Importantly, the structure of the learned energy function provides a natural explanation for the hardness of learning for a given class of quantum states.

Predicting Many Properties Of A Quantum System From Very Few Measurements

Abstract: Predicting properties of complex, large-scale quantum systems is essential for developing quantum technologies. We present an efficient method for constructing an approximate classical description of a quantum state using very few measurements of the state. This description, called a classical shadow, can be used to predict many different properties: order $\log M$ measurements suffice to accurately predict $M$ different functions of the state with high success probability. The number of measurements is independent of the system size, and saturates information-theoretic lower bounds. Moreover, target properties to predict can be selected after the measurements are completed. We support our theoretical findings with extensive numerical experiments. We apply classical shadows to predict quantum fidelities, entanglement entropies, two-point correlation functions, expectation values of local observables, and the energy variance of many-body local Hamiltonians, which allows applications to speedup variational quantum algorithms. The numerical results highlight the advantages of classical shadows relative to previously known methods.


However, a more complicated head will not further improve the performance, which is consistent with the observation in #REFR .

Now we will demonstrate the effect of different components in TeS by ablation study. Experiments are conducted with ResNet-50 on CIFAR-100.

#Layer in Projection h h contains multi-layer MLP to align vision representations with text classifier.

We vary the number of layers and summarize the results in Table 7 .

When h has 1-layer MLP, it degenerates to a linear projection.

If increasing the number of layers to 2, the nonlinear mapping helps improve the performance by 0.36%.

Compared to ResNet-50, the computational overhead introduced by the 2-layer MLP is negligible, which keeps the efficiency of the proposed method.

Weight of Text Regularization λ T λ T in Eqn.

7 weights the loss for projecting the vision representation to the text space spanned by the text classifier.

We vary it in {0.1, 0.3, 0.5, 0.7, 0.9, 1.1, 1.3, 1.5} and Table 8 When λ T is small, the projection head cannot be learned sufficiently and results in an inaccurate estimation for the reference distribution.

By increasing λ T , the projection head can approximate the text space effectively and a satisfied performance is obtained.

Improved Visual Fine-Tuning With Natural Language Supervision

Abstract: Fine-tuning a pre-trained model can leverage the semantic information from large-scale pre-training data and mitigate the over-fitting problem on downstream tasks with limited training examples. While the problem of catastrophic forgetting in backbone has been extensively studied, the potential bias existing in a pre-trained model due to the corresponding pre-training task and data, attracts less attention. In this work, we investigate this problem by demonstrating that the obtained classifier after fine-tuning will be close to that induced by the pre-trained model. To reduce the bias in the classifier effectively, we introduce a reference distribution obtained from a fixed text classifier, which can help regularize the learned vision classifier. The proposed method, Text Supervised fine-tuning (TeS), is evaluated with diverse pre-trained vision models including ResNet and ViT, and text encoders including BERT and CLIP, on 11 downstream tasks. The consistent improvement with a clear margin over distinct scenarios confirms the effectiveness of our proposal.

A Simple Framework For Contrastive Learning Of Visual Representations

Abstract: This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.


The edge map is concatenated with the masked image and fed into a UNet #REFR to reconstruct the original image.

We assign a learnable weight α to edges, which is enforced to be positive by SoftPlus #OTHEREFR .

This weight is learned during training and shared across all edges and all object instances in a dataset.

Finally, we take the maximum at each pixel of the heatmaps to obtain the edge map S ∈ R H×W ,

Taking the maximum at each pixel avoids the entanglement of the uncertainty and the convolution kernel weights #OTHEREFR .

We minimize the L 1 loss and ViT perceptual loss #OTHEREFR between the reconstructed images I ′ and the original images I,

Few-Shot Geometry-Aware Keypoint Localization

Abstract: Supervised keypoint localization methods rely on large manually labeled image datasets, where objects can deform, articulate, or occlude. However, creating such large keypoint labels is time-consuming and costly, and is often error-prone due to inconsistent labeling. Thus, we desire an approach that can learn keypoint localization with fewer yet consistently annotated images. To this end, we present a novel formulation that learns to localize semantically consistent keypoint definitions, even for occluded regions, for varying object categories. We use a few user-labeled 2D images as input examples, which are extended via self-supervision using a larger unlabeled dataset. Unlike unsupervised methods, the few-shot images act as semantic shape constraints for object localization. Furthermore, we introduce 3D geometryaware constraints to uplift keypoints, achieving more accurate 2D localization. Our general-purpose formulation paves the way for semantically conditioned generative modeling and attains competitive or state-of-the-art accuracy on several datasets, including human faces, eyes, animals, cars, and never-before-seen mouth interior (teeth) localization tasks, not attempted by the previous few-shot methods.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .


However, they only studied LP relaxation methods and were unable to include the SDP certification method #REFR .

Nevertheless, the SDPs are general enough, and this is straightforward from our framework.

Similarly, we implemented a tool for ℓ 2 -robustness certification, which is natural from our framework.

However, even though it has been a while since the SDP for ℓ ∞ -robustness certification was proposed #OTHEREFR , we are unaware of any existing SDP tools to certify ℓ 2 -robustness.

The ℓ 2 -robustness certification is an active topic #OTHEREFR .

[55] tried to unify efficient robustness certification works, which the authors claimed is a convex relaxation barrier.

As we have discussed, SDP is strictly more powerful than LP.

Our framework can be viewed as complementing #OTHEREFR towards a tighter convex relaxation barrier.

Efficient Symbolic Reasoning For Neural-Network Verification

Abstract: The neural network has become an integral part of modern software systems. However, they still suffer from various problems, in particular, vulnerability to adversarial attacks. In this work, we present a novel program reasoning framework for neural-network verification, which we refer to as symbolic reasoning. The key components of our framework are the use of the symbolic domain and the quadratic relation. The symbolic domain has very flexible semantics, and the quadratic relation is quite expressive. They allow us to encode many verification problems for neural networks as quadratic programs. Our scheme then relaxes the quadratic programs to semidefinite programs, which can be efficiently solved. This framework allows us to verify various neural-network properties under different scenarios, especially those that appear challenging for non-symbolic domains. Moreover, it introduces new representations and perspectives for the verification tasks. We believe that our framework can bring new theoretical insights and practical tools to verification problems for neural networks.

Semidefinite Relaxations For Certifying Robustness To Adversarial Examples

Abstract: Despite their impressive performance on diverse tasks, neural networks fail catastrophically in the presence of adversarial inputs---imperceptibly but adversarially perturbed versions of natural inputs. We have witnessed an arms race between defenders who attempt to train robust networks and attackers who try to construct adversarial examples. One promise of ending the arms race is developing certified defenses, ones which are provably robust against all attackers in some family. These certified defenses are based on convex relaxations which construct an upper bound on the worst case loss over all attackers in the family. Previous relaxations are loose on networks that are not trained against the respective relaxation. In this paper, we propose a new semidefinite relaxation for certifying robustness that applies to arbitrary ReLU networks. We show that our proposed relaxation is tighter than previous relaxations and produces meaningful robustness guarantees on three different "foreign networks" whose training objectives are agnostic to our proposed relaxation.

SDP certification method
LP relaxation methods


To emulate a variety of scenes, we randomly select video clips from YouTube UGC video dataset #REFR , including gaming, television, animation, etc., to form a 1-minute long video, and then repetitively render it during a live RTVC session spanning about 10 minutes.

For real-world tests in the wild, we rent a commercial cloud server from a leading Internet service vendor, and set the server-to-receiver down-link with 4G, 5G, or WiFi accesses in various environment (walking, bus/subway riding, café, etc.), for simulating a variety of real-world scenarios.

Note that sender and receiver clients are deployed in different cities worldwide.

As discussed in the measurement study in Sec.

III-A, we play pre-cached video files to simplify the simulations.

The sender encodes live scenes in real-time, and then transmits the video stream to the receiver client.

To best mimic the real RTVC applications in the large-scale Internet, we configure to encode this long video at a relative low resolution (i.e., 720P) to guarantee both the video quality and the playback fluency.

At the receiver client, we collect the RTT, packet loss rate measured by the WebRTC internals #OTHEREFR and proactively measure the stalling rate along with the video consumption. Fig. 5 .

Echo is a real-time video conferencing testbed that complies with the WebRTC standard.

Placing the server node differently allows us to perform the trace-driven simulations in a managed lab or directly run real-world field trials in the wild. Four ABR solutions are deployed for study.

We will later make this testbed open to public for reproducible research.

Improving Adaptive Real-Time Video Communication Via Cross-Layer Optimization

Abstract: Effective Adaptive BitRate (ABR) algorithm or policy is of paramount importance for Real-Time Video Communication (RTVC) amid this pandemic to pursue uncompromised quality of experience (QoE). Existing ABR methods mainly separate the network bandwidth estimation and video encoder control, and fine-tune video bitrate towards estimated bandwidth, assuming the maximization of bandwidth utilization yields the optimal QoE. However, the QoE of a RTVC system is jointly determined by the quality of compressed video, fluency of video playback, and interaction delay. Solely maximizing the bandwidth utilization without comprehensively considering compound impacts incurred by both network and video application layers, does not assure the satisfactory QoE. And the decoupling of network and video layer further exacerbates the user experience due to network-codec incoordination. This work therefore proposes the Palette, a reinforcement learning based ABR scheme that unifies the processing of network and video application layers to directly maximize the QoE formulated as the weighted function of video quality, stalling rate and delay. To this aim, a crosslayer optimization is proposed to derive fine-grained compression factor of upcoming frame(s) using cross-layer observations like network conditions, video encoding parameters, and video content complexity. As a result, Palette manages to resolve the network-codec incoordination and to best catch up with the network fluctuation. Compared with state-of-the-art schemes in real-world tests, Palette not only reduces 3.1%-46.3% of the stalling rate, 20.2%-50.8% of the delay, but also improves 0.2%-7.2% of the video quality with comparable bandwidth consumption, under a variety of application scenarios.

Youtube Ugc Dataset For Video Compression Research

Abstract: Non-professional video, commonly known as User Generated Content (UGC) has become very popular in today's video sharing applications. However, traditional metrics used in compression and quality assessment, like BD-Rate and PSNR, are designed for pristine originals. Thus, their accuracy drops significantly when being applied on non-pristine originals (the majority of UGC). Understanding difficulties for compression and quality assessment in the scenario of UGC is important, but there are few public UGC datasets available for research. This paper introduces a large scale UGC dataset (1500 20 sec video clips) sampled from millions of YouTube videos. The dataset covers popular categories like Gaming, Sports, and new features like High Dynamic Range (HDR). Besides a novel sampling method based on features extracted from encoding, challenges for UGC compression and quality evaluation are also discussed. Shortcomings of traditional reference-based metrics on UGC are addressed. We demonstrate a promising way to evaluate UGC quality by no-reference objective quality metrics, and evaluate the current dataset with three no-reference metrics (Noise, Banding, and SLEEQ).


Approximation Algorithms For The Robf Problem

In #REFR it has been shown that solving the nominal problem min k∈F c c c k , for linear cost functions f , gives a z(F)-approximation to (13).

F ∈F min k∈F f (x x x,c c c k ) F ∈F max k∈F f (x x x,c c c k ) ∈ [0, 1]

Observe that (13) is a special case of the mixed-uncertainty robust optimization problem discussed in #OTHEREFR , where uncertainty sets are given and we would like to solve

min x x x∈X F ∈F m(F ) max k∈F f (x x x, c c c k ),

In combination with Theorem 4, this allows us to derive approximation algorithms for a wide range of the ROBF problems if the underling deterministic problem (1) is polynomially solvable, in particular for a wide class of network problems (see #OTHEREFR for a comprehensive review).

Note also that in most cases γ = 1 if (13) is a convex problem (in particular linear programing one).

Robust Optimization With Belief Functions

Abstract: In this paper, an optimization problem with uncertain objective function coefficients is considered. The uncertainty is specified by providing a discrete scenario set, containing possible realizations of the objective function coefficients. The concept of belief function in the traditional and possibilistic setting is applied to define a set of admissible probability distributions over the scenario set. The generalized Hurwicz criterion is then used to compute a solution. In this paper, the complexity of the resulting problem is explored. Some exact and approximation methods of solving it are proposed.

Mixed Uncertainty Sets For Robust Combinatorial Optimization

Abstract: In robust optimization, the uncertainty set is used to model all possible outcomes of uncertain parameters. In the classic setting, one assumes that this set is provided by the decision maker based on the data available to her. Only recently it has been recognized that the process of building useful uncertainty sets is in itself a challenging task that requires mathematical support. ::: In this paper, we propose an approach to go beyond the classic setting, by assuming multiple uncertainty sets to be prepared, each with a weight showing the degree of belief that the set is a "true" model of uncertainty. We consider theoretical aspects of this approach and show that it is as easy to model as the classic setting. In an extensive computational study using a shortest path problem based on real-world data, we auto-tune uncertainty sets to the available data, and show that with regard to out-sample performance, the combination of multiple sets can give better results than each set on its own.


CDAN #REFR injects class-specific information into the discriminator to facilitate the alignment of multi-modal distributions.

For example, #OTHEREFR try to minimize the maximum mean discrepancy (MMD) #OTHEREFR across domains, while MCC #OTHEREFR minimizes the class confusion loss without explicit domain alignment.

Margin disparity discrepancy (MDD) #OTHEREFR is to perform domain alignment with analyses in terms of generalization bounds.

#OTHEREFR uses the Wasserstein distance for distributional alignment to achieve better transfer effects.

On the other hand, adversarial-based methods align distributions across domains through adversarial training #OTHEREFR .

For example, DANN #OTHEREFR adversarially learns a domain discriminator to distinguish samples in two domains and thus enables the feature extractor to confuse the domain discriminator.

SDAT #OTHEREFR uses sharpness-aware minimization #OTHEREFR to seek a flat minimize for better generalization, while ELS #OTHEREFR improves over SDAT by introducing label smoothing to domain labels and achieves better transfer performance.

Adversarial training can also be used to generate samples to bridge the source and target domains.

CyCADA #OTHEREFR adopts GAN for image-to-image translation via the cycle consistency loss #OTHEREFR and semantic consistency loss.

Specifically, it directly changes the style of source images to that of the target domain to obtain a better classifier for the target domain.

#OTHEREFR uses two cross-domain generators to synthesize data of each domain conditioned on the other, and learns two task-specific classifiers.

Diffusion-Based Target Sampler For Unsupervised Domain Adaptation

Abstract: Limited transferability hinders the performance of deep learning models when applied to new application scenarios. Recently, unsupervised domain adaptation (UDA) has achieved significant progress in addressing this issue via learning domain-invariant features. However, large domain shifts and the sample scarcity in the target domain make existing UDA methods achieve suboptimal performance. To alleviate these issues, we propose a plug-andplay Diffusion-based Target Sampler (DTS) to generate high fidelity and diversity pseudo target samples. By introducing class-conditional information, the labels of the generated target samples can be controlled. The generated samples can well simulate the data distribution of the target domain and help existing UDA methods transfer from the source domain to the target domain more easily, thus improving the transfer performance. Extensive experiments on various benchmarks demonstrate that the performance of existing UDA methods can be greatly improved through the proposed DTS method. The code will be released soon.

Conditional Adversarial Domain Adaptation

Abstract: Adversarial learning has been embedded into deep networks to learn disentangled and transferable representations for domain adaptation. Existing adversarial domain adaptation methods may not effectively align different domains of multimodal distributions native in classification problems. In this paper, we present conditional adversarial domain adaptation, a principled framework that conditions the adversarial adaptation models on discriminative information conveyed in the classifier predictions. Conditional domain adversarial networks (CDANs) are designed with two novel conditioning strategies: multilinear conditioning that captures the cross-covariance between feature representations and classifier predictions to improve the discriminability, and entropy conditioning that controls the uncertainty of classifier predictions to guarantee the transferability. With theoretical guarantees and a few lines of codes, the approach has exceeded state-of-the-art results on five datasets.


This was observed before by #REFR and we observe the same trend in our experiments (Section 5).

V is O i:µ i <µ a * σ 2 i ∆ i + b log n , where b

is an upper bound on the absolute value of the rewards.

In summary, variance adaptation in UCB-V incurs only a small penalty of O(bK log n). #OTHEREFR

(2018) proposed an elimination-based variant of UCB-V that attains the optimal gap-free regret of O( √ Kn), as opposing to the original O( √ Kn log n).

While empirical Bernstein bounds are general, they tend to be conservative in practice.

Our work can be viewed as a similar development with Thompson sampling.

We show that Thompson sampling with unknown reward variances (Section 4) incurs only a slightly higher regret than the one with the known variances (Section 3), by a multiplicative factor. Comparing to UCB-V, the algorithm is highly practical. A recent work by Zhang et al.

(2021) extended the above results to d-dimensional linear bandits Abbasi-Yadkori et al. 2011; Dani et al. (2008b,a) , where the arm set can be infinite.

Their regret bound isÕ(poly(d) 1 + n s=1 σ 2 s ), where σ 2 s is the unknown reward variance in round s. They also extended their result to linear mixture MDPs.

Another related work on variance-sensitive multi-armed bandits is Zhu and Tan (2020).

Only Pay For What Is Uncertain: Variance-Adaptive Thompson Sampling

Abstract: Most bandit algorithms assume that the reward variance or its upper bound is known. While variance overestimation is usually safe and sound, it increases regret. On the other hand, an underestimated variance may lead to linear regret due to committing early to a suboptimal arm. This motivated prior works on variance-aware frequentist algorithms. We lay foundations for the Bayesian setting. In particular, we study multi-armed bandits with known and unknown heterogeneous reward variances, and develop Thompson sampling algorithms for both and bound their Bayes regret. Our regret bounds decrease with lower reward variances, which make learning easier. The bound for unknown reward variances captures the effect of the prior on learning reward variances and is the first of its kind. Our experiments show the superiority of varianceaware Bayesian algorithms and also highlight their robustness.

The Kl-Ucb Algorithm For Bounded Stochastic Bandits And Beyond

Abstract: This paper presents a finite-time analysis of the KL-UCB algorithm, an online, horizon-free index policy for stochastic bandit problems. We prove two distinct results: first, for arbitrary bounded rewards, the KL-UCB algorithm satisfies a uniformly better regret bound than UCB or UCB2; second, in the special case of Bernoulli rewards, it reaches the lower bound of Lai and Robbins. Furthermore, we show that simple adaptations of the KL-UCB algorithm are also optimal for specific classes of (possibly unbounded) rewards, including those generated from exponential families of distributions. A large-scale numerical study comparing KL-UCB with its main competitors (UCB, UCB2, UCB-Tuned, UCB-V, DMED) shows that KL-UCB is remarkably efficient and stable, including for short time horizons. KL-UCB is also the only method that always performs better than the basic UCB policy. Our regret bounds rely on deviations results of independent interest which are stated and proved in the Appendix. As a by-product, we also obtain an improved regret bound for the standard UCB algorithm.


For a comprehensive and fair comparison of the CIFAR10 experiment, we followed the same experiment setting of #REFR .

All the compared methods (except UASD #OTHEREFR ) were built upon the open-source Pytorch implementation 1 by #OTHEREFR .

As UASD has not released its implementation, we implemented UASD by ourselves.

For DS3L, we implemented it based on the released code 2 .

For L2RW #OTHEREFR , we used the open-source Pytorch implementation 3 and adapted it to the SSL settings.

For MWN #OTHEREFR , we used the authors' implementation 4 and adapt to SSL. experiments.

All the compared methods were built upon the open-source Pytorch implementation by #OTHEREFR .

The code and datasets are temporarily available for reviewing purposes at here 5 .

Our code and datasets are also submitted as supplementary material.

1 https://github.com/perrying/realistic-ssl-evaluation-pytorch 2 https://github.com/guolz-ml/DS3L

3 https://github.com/danieltan07/learning-to-reweight-examples #OTHEREFR https://github.com/xjtushujun/meta-weight-net #OTHEREFR https://anonymous.4open.science/r/WR-SSL-406F/README.md Hyperparameter setting.

Table Of Contents

Abstract: 2 Uncertainty Framework Overview. Subjective Bayesian GNN (a) is designed for estimating the different types of uncertainties. The loss function includes a square error (d) to reduce bias, GKDE (b) to reduce errors in uncertainty estimation, and teacher network (c) to refine class probability.

Realistic Evaluation Of Deep Semi-Supervised Learning Algorithms

Abstract: Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. SSL algorithms based on deep neural networks have recently proven successful on standard benchmark tasks. However, we argue that these benchmarks fail to address many issues that these algorithms would face in real-world applications. After creating a unified reimplementation of various widely-used SSL techniques, we test them in a suite of experiments designed to address these issues. We find that the performance of simple baselines which do not use unlabeled data is often underreported, that SSL methods differ in sensitivity to the amount of labeled and unlabeled data, and that performance can degrade substantially when the unlabeled dataset contains out-of-class examples. To help guide SSL research towards real-world applicability, we make our unified reimplemention and evaluation platform publicly available.


Relatedly, DeepEnsembles #REFR seeks to approximately marginalise over the parameters of a given NN architecture.

Neural Ensemble Search via Bayesian Sampling #OTHEREFR approximates the posterior distribution over architectures p(α | D) with a variational distribution of the form

, where i iterates over the connections within a cell, o is the operation for connection i, and θ i are the variational parameters for q i .

The form of q i is chosen to be a softmax over θ i .

The ensemble is then selected by using Stein Variational Gradient Descent with Regularised Diversity to select a diverse set of M samples from (a continuous relaxation of) the variational distribution.

The architecture is trained from several random initialisations, and the ensemble makes a prediction as an equally weighed sum of these.

This is orthogonal to the work above (and, indeed, is orthogonal our work), which seeks to construct ensembles of different architectures, rather than ensembles of different parameter settings of the same architecture.

Bayesian Quadrature For Neural Ensemble Search

Abstract: Ensembling can improve the performance of Neural Networks, but existing approaches struggle when the architecture likelihood surface has dispersed, narrow peaks. Furthermore, existing methods construct equally weighted ensembles, and this is likely to be vulnerable to the failure modes of the weaker architectures. By viewing ensembling as approximately marginalising over architectures we construct ensembles using the tools of Bayesian Quadrature-tools which are well suited to the exploration of likelihood surfaces with dispersed, narrow peaks. Additionally, the resulting ensembles consist of architectures weighted commensurate with their performance. We show empirically-in terms of test likelihood, accuracy, and expected calibration error-that our method outperforms state-of-the-art baselines, and verify via ablation studies that its components do so independently.

Simple And Scalable Predictive Uncertainty Estimation Using Deep Ensembles

Abstract: Deep neural networks (NNs) are powerful black box predictors that have recently achieved impressive performance on a wide spectrum of tasks. Quantifying predictive uncertainty in NNs is a challenging and yet unsolved problem. Bayesian NNs, which learn a distribution over weights, are currently the state-of-the-art for estimating predictive uncertainty; however these require significant modifications to the training procedure and are computationally expensive compared to standard (non-Bayesian) NNs. We propose an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates. Through a series of experiments on classification and regression benchmarks, we demonstrate that our method produces well-calibrated uncertainty estimates which are as good or better than approximate Bayesian NNs. To assess robustness to dataset shift, we evaluate the predictive uncertainty on test examples from known and unknown distributions, and show that our method is able to express higher uncertainty on out-of-distribution examples. We demonstrate the scalability of our method by evaluating predictive uncertainty estimates on ImageNet.


For this purpose, we used two towns from the CARLA simulator by Dosovitskiy et al. #REFR namely, Town01 and Town07.

In both the INTERFACES (VR and BASELINE), participants performed a driving task in the TELEOPERATION scenario, where they had to drive a virtually simulated vehicle.

The participants drove in Town01 in one INTERFACE and on Town07 in the other.

The combination of the INTERFACE and the town to drive in was balanced among the participants.

The pre-defined path for driving in the town was visualized using spheres, here shown with a yellow dashed curves in Figure 4 .

We asked the participants to follow this path for driving.

We also asked them to drive as fast as possible and align the position of the vehicle below the spheres as accurately as possible to reduce their deviation from the route.

Remote Monitoring And Teleoperation Of Autonomous Vehicles -Is Virtual Reality An Option?

Abstract: While the promise of autonomous vehicles has led to significant scientific and industrial progress, fully automated, SAE level 5 conform cars will likely not see mass adoption anytime soon. Instead, in many applications, human supervision, such as remote monitoring and teleoperation, will be required for the foreseeable future. While Virtual Reality (VR) has been proposed as one potential interface for teleoperation, its benefits and drawbacks over physical monitoring and teleoperation solutions have not been thoroughly investigated. To this end, we contribute two user studies, comparing the performance of and subjective feedback for a VR-based system with an existing monitoring and teleoperation system, which is in industrial use today. The results of our first user study (n=16), indicate that a VR interface replicating the physical interface does not outperform the physical interface. It also quantifies the negative effects that combined monitoring and teleoperating tasks have on users irrespectively of the interface being used. The results of the second user study (n=24), indicate that the perceptual and ergonomic issues caused by VR outweigh its benefits like better concentration through isolation, which is critical for monitoring tasks. Through these two user studies, we contribute to a better understanding of future virtual monitoring and teleoperation solutions for autonomous vehicles.

Carla: An Open Urban Driving Simulator

Abstract: We introduce CARLA, an open-source simulator for autonomous driving research. CARLA has been developed from the ground up to support development, training, and validation of autonomous urban driving systems. In addition to open-source code and protocols, CARLA provides open digital assets (urban layouts, buildings, vehicles) that were created for this purpose and can be used freely. The simulation platform supports flexible specification of sensor suites and environmental conditions. We use CARLA to study the performance of three approaches to autonomous driving: a classic modular pipeline, an end-to-end model trained via imitation learning, and an end-to-end model trained via reinforcement learning. The approaches are evaluated in controlled scenarios of increasing difficulty, and their performance is examined via metrics provided by CARLA, illustrating the platform's utility for autonomous driving research. The supplementary video can be viewed at this https URL


In the L N -iMAC, the backbone network Φ is ResNet50 #REFR , the number of layers K is 4, the types of region kernel N are 4.

Each video is sampled at one frame per second.

The number of frames per video is set during training with T = 64 but is varied during inference based on the duration of the video.

The set of easy distractors used in the training phase is taken from nearly 100,000 metadatafree videos (with the exception of 528 videos in the core datasets) within the VCDB #OTHEREFR , where each frame-level feature is selected as an easy distractor based on the magnitude threshold λ mag = 40.

The distractor threshold λ di for the thresholding operation on the confidence of DDM during the inference phase is set to 0.5.

The temperature of the tempered sigmoid #OTHEREFR is set to 512 in TGM.

The parameter α for balancing the discrimination loss is set to 0.5, and margins in the frame loss and the video loss are also set to 0.5.

Additionally, PCA is incrementally trained from all metadata-free videos within the VCDB.

Vvs: Video-To-Video Retrieval With Irrelevant Frame Suppression

Abstract: In content-based video retrieval (CBVR), dealing with large-scale collections, efficiency is as important as accuracy. For this reason, several video-level feature-based studies have actively been conducted; nevertheless, owing to the severe difficulty of embedding a lengthy and untrimmed video into a single feature, these studies have shown insufficient for accurate retrieval compared to framelevel feature-based studies. In this paper, we show an insight that appropriate suppression of irrelevant frames can be a clue to overcome the current obstacles of the videolevel feature-based approaches. Furthermore, we propose a Video-to-Video Suppression network (VVS) as a solution. The VVS is an end-to-end framework that consists of an easy distractor elimination stage for identifying which frames to remove and a suppression weight generation stage for determining how much to suppress the remaining frames. This structure is intended to effectively describe an untrimmed video with varying content and meaningless information. Its efficacy is proved via extensive experiments, and we show that our approach is not only state-of-the-art in videolevel feature-based approaches but also has a fast inference time despite possessing retrieval capabilities close to those of frame-level feature-based approaches.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


For example, when we use ImageNet data #REFR for self-supervised pre-training, views from the current image instance x 0 belonging to the 'bloodhound' category is labeled as positive, while views from other image instances belonging to the same category will be labeled as negative.

Different from these methods, our SNCLR shown in (d) measures correlations between the current instance and other instances identified from the candidate neighbor set.

We define the instances that are highly-correlated to the current instance as soft neighbors (e.g., different pandas to the input image belonging to the 'panda' category in Fig. 3 ).

The similarity/correlation, which is based on the cross-attention computation from a shallow attention module, denotes the instance positiveness to compute the contrastive loss.

(Best viewed in colors) and negative pairs constitute the contrastive loss computation process.

In practice, we observe that this rudimentary labeling is not sufficient to represent instance correlations (i.e., similar to the semantic relations in supervised learning) between different images and may hurt the performance of learned representations.

Since we do not know the original training labels during SSL, images belonging to the same semantic category will be labeled differently to limit feature representations.

On the other hand, for the current image instance x 0 , although views from one image instance x 1 belonging to the 'walker hound' category and views from another image instance x 2 belonging to the 'peacock' category are both labeled as negative, the views from x 1 are more correlated to the views from x 0 than those views from x 2 .

This correlation is not properly encoded during contrastive learning as views from both x 1 and x 2 are labeled the same.

Without original training labels, contrastive learning methods are not effective in capturing the correlations between image instances.

The learned feature representations are limited to describing correlated visual contents across different images.

Soft Neighbors Are Positive Supporters In Contrastive Visual Representation Learning

Abstract: Contrastive learning methods train visual encoders by comparing views (e.g., often created via a group of data augmentations on the same instance) from one instance to others. Typically, the views created from one instance are set as positive, while views from other instances are negative. This binary instance discrimination is studied extensively to improve feature representations in self-supervised learning. In this paper, we rethink the instance discrimination framework and find the binary instance labeling insufficient to measure correlations between different samples. For an intuitive example, given a random image instance, there may exist other images in a mini-batch whose content meanings are the same (i.e., belonging to the same category) or partially related (i.e., belonging to a similar category). How to treat the images that correlate similarly to the current image instance leaves an unexplored problem. We thus propose to support the current image by exploring other correlated instances (i.e., soft neighbors). We first carefully cultivate a candidate neighbor set, which will be further utilized to explore the highly-correlated instances. A cross-attention module is then introduced to predict the correlation score (denoted as positiveness) of other correlated instances with respect to the current one. The positiveness score quantitatively measures the positive support from each correlated instance, and is encoded into the objective for pretext training. To this end, our proposed method benefits in discriminating uncorrelated instances while absorbing correlated instances for SSL. We evaluate our soft neighbor contrastive learning method (SNCLR) on standard visual recognition benchmarks, including image classification, object detection, and instance segmentation. The state-of-theart recognition performance shows that SNCLR is effective in improving feature representations from both ViT and CNN encoders.

Imagenet Large Scale Visual Recognition Challenge

Abstract: The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements.


The lottery ticket hypothesis #REFR is the first to discover that there exist trainable subnetworks which are hidden in an over-parameterized dense network but can match the accuracy of their dense network counterparts after being trained in isolation.

This intriguing finding has motivated various followup works that explore the lottery ticket hypothesis under different scenarios.

In particular, #OTHEREFR , #OTHEREFR extend the lottery ticket hypothesis to transfer learning and find that subnetworks inheriting the pretrained model weights as initialization can match the task accuracy of their dense network counterparts after finetuning.

However, their adopted metrics were originally designed for maintaining the accuracy of the identified tickets on the same task and thus do not necessarily help preserve the transferability to downstream tasks.

Motivated by the success of DNNs as a general feature extractor, transfer learning #OTHEREFR has been widely adopted across different domains to benefit downstream tasks from DNNs pretrained on a big data regime.

Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks In Transfer Learning

Abstract: Transfer learning leverages feature representations of deep neural networks (DNNs) pretrained on source tasks with rich data to empower effective finetuning on downstream tasks. However, the pretrained models are often prohibitively large for delivering generalizable representations, which limits their deployment on edge devices with constrained resources. To close this gap, we propose a new transfer learning pipeline, which leverages our finding that robust tickets can transfer better, i.e., subnetworks drawn with properly induced adversarial robustness can win better transferability over vanilla lottery ticket subnetworks. Extensive experiments and ablation studies validate that our proposed transfer learning pipeline can achieve enhanced accuracysparsity trade-offs across both diverse downstream tasks and sparsity patterns, further enriching the lottery ticket hypothesis.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Abstract: Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance. We find that a standard technique for pruning weights naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the"lottery ticket hypothesis": dense, randomly-initialized feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - arrive at comparable test accuracy in a comparable number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective. We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Furthermore, the winning tickets we find above that size learn faster than the original network and exhibit higher test accuracy.


This challenge of circular dependence does not appear in #REFR : their goal was to estimate the unknown mean of an identity-covariance Gaussian, and thus it sufficed to use a linear polynomial f (instead of a quadratic polynomial).

In either case, we will have cleaned up our set of samples and can recurse on each of the returned subsets of S.

Iterating this technique recursively on all of the smaller subsets returned ensures that there is always at least one subset containing the majority of the inliers, and that eventually once it stops having too large of a covariance, we will return an appropriate approximation to Σ.

We want to highlight the main point of difference where our techniques differ notably from #OTHEREFR .

In order to implement the algorithm outlined above, one needs to have good a priori bounds for what the variance of f (x) over the inliers ought to be.

Since f (•) is a quadratic polynomial, the variance of f over the inliers, itself depends on the covariance Σ, which is exactly what we are trying to estimate.

Importantly, the covariance of a linear polynomial does not depend on the (unknown) mean (it depends only on the covariance, which was known in their setting).

In order to overcome this challenge, we observe that if S contains most of the inliers, then the covariance of S cannot be too much smaller than the true covariance Σ.

This allows us to find an upper bound on Σ, which in turn lets us upper bound the variance of f (x) over the good samples (Lemma 3.3).

A Spectral Algorithm For List-Decodable Covariance Estimation In Relative Frobenius Norm

Abstract: We study the problem of list-decodable Gaussian covariance estimation. Given a multiset T of n points in R d such that an unknown α < 1/2 fraction of points in T are i.i.d. samples from an unknown Gaussian N (µ, Σ), the goal is to output a list of O(1/α) hypotheses at least one of which is close to Σ in relative Frobenius norm. Our main result is a poly(d, 1/α) sample and time algorithm for this task that guarantees relative Frobenius norm error of poly(1/α). Importantly, our algorithm relies purely on spectral techniques. As a corollary, we obtain an efficient spectral algorithm for robust partial clustering of Gaussian mixture models (GMMs)-a key ingredient in the recent work of [BDJ + 22] on robustly learning arbitrary GMMs. Combined with the other components of [BDJ + 22], our new method yields the first Sum-of-Squares-free algorithm for robustly learning GMMs. At the technical level, we develop a novel multi-filtering method for list-decodable covariance estimation that may be useful in other settings.

List-Decodable Robust Mean Estimation And Learning Mixtures Of Spherical Gaussians

Abstract: We study the problem of list-decodable Gaussian mean estimation and the related problem of learning mixtures of separated spherical Gaussians. We develop a set of techniques that yield new efficient algorithms with significantly improved guarantees for these problems. {\bf List-Decodable Mean Estimation.} Fix any $d \in \mathbb{Z}_+$ and $0<\alpha<1/2$. We design an algorithm with runtime $O (\mathrm{poly}(n/\alpha)^{d})$ that outputs a list of $O(1/\alpha)$ many candidate vectors such that with high probability one of the candidates is within $\ell_2$-distance $O(\alpha^{-1/(2d)})$ from the true mean. The only previous algorithm for this problem achieved error $\tilde O(\alpha^{-1/2})$ under second moment conditions. For $d = O(1/\epsilon)$, our algorithm runs in polynomial time and achieves error $O(\alpha^{\epsilon})$. We also give a Statistical Query lower bound suggesting that the complexity of our algorithm is qualitatively close to best possible. {\bf Learning Mixtures of Spherical Gaussians.} We give a learning algorithm for mixtures of spherical Gaussians that succeeds under significantly weaker separation assumptions compared to prior work. For the prototypical case of a uniform mixture of $k$ identity covariance Gaussians we obtain: For any $\epsilon>0$, if the pairwise separation between the means is at least $\Omega(k^{\epsilon}+\sqrt{\log(1/\delta)})$, our algorithm learns the unknown parameters within accuracy $\delta$ with sample complexity and running time $\mathrm{poly} (n, 1/\delta, (k/\epsilon)^{1/\epsilon})$. The previously best known polynomial time algorithm required separation at least $k^{1/4} \mathrm{polylog}(k/\delta)$. Our main technical contribution is a new technique, using degree-$d$ multivariate polynomials, to remove outliers from high-dimensional datasets where the majority of the points are corrupted.


We have used the apple leaf disease subset of the PlantVillage dataset, one of the largest open-access sources of leaf images #REFR .

There are 54,309 images of healthy and diseased leaves of 14 crops in the plantVillage dataset, all of which have been annotated by experts.

There are 3,171 pictures of apple leaves, with 3 groups representing diseases and 1 representing healthy leaves. Figure 1 shows examples of images in each category.

The images were captured in a laboratory setting and depict varying degrees of disease spot distribution on the leaf.

Class imbalance is present in the dataset, as the greatest number of 'healthy' samples is 1645, while the number of samples corresponding to 'Cedar apple rust' is only 275.

To combat this imbalance issue, we have included the concept of runtime augmentation.

An Efficient Transfer Learning-Based Approach For Apple Leaf Disease Classification

Abstract: Correct identification and categorization of plant diseases are crucial for ensuring the safety of the global food supply and the overall financial success of stakeholders. In this regard, a wide range of solutions has been made available by introducing deep learning-based classification systems for different staple crops. Despite being one of the most important commercial crops in many parts of the globe, research proposing a smart solution for automatically classifying apple leaf diseases remains relatively unexplored. This study presents a technique for identifying apple leaf diseases based on transfer learning. The system extracts features using a pretrained EfficientNetV2S architecture and passes to a classifier block for effective prediction. The class imbalance issues are tackled by utilizing runtime data augmentation. The effect of various hyperparameters, such as input resolution, learning rate, number of epochs, etc., has been investigated carefully. The competence of the proposed pipeline has been evaluated on the apple leaf disease subset from the publicly available 'PlantVillage' dataset, where it achieved an accuracy of 99.21%, outperforming the existing works.

An Open Access Repository Of Images On Plant Health To Enable The Development Of Mobile Disease Diagnostics

Abstract: Human society needs to increase food production by an estimated 70% by 2050 to feed an expected population size that is predicted to be over 9 billion people. Currently infectious diseases reduce the potential yield by an average of 40% with many farmers in the developing world experiencing yield losses as high as 100%. Infectious diseases of crops are not new and historic examples such as the Irish Potato Famine of 1845-49 demonstrate this. But what is new is the widespread distribution of smartphones among crop growers around the world with an expected 5 billion smartphones by 2020. This offers the potential of turning the smartphone into a valuable tool for diverse communities growing food. One potential application is the development of mobile disease diagnostics through machine learning and crowdsourcing. Computer vision and machine learning have shown their potential to automatically classify images. To do this for plant diseases requires a training set that facilitates the development of the algorithms. Here we announce the release of >50,000 expertly curated images on healthy and infected leaves of crops plants through the existing platform www.PlantVillage.org. We describe both the data and the platform. These data are the beginning of an on-going, crowdsourcing effort to enable computer vision approaches to help solve the problem of yield losses in crop plants due to infectious diseases.


C. Feature Residual Extraction And Compression

The VRN #REFR unit, located between the two convolution layers, utilizes skip connections to mitigate information loss during training and employs parallel convolutional layers with varying kernel sizes, namely 1 × 1 × 1 or 3 × 3 × 3, to capture features at different ranges.

In our implementation, the kernel is defined as a hypercube with a size of 3, namely [−1, 0, 1] 3 , and the Minkowski #OTHEREFR , #OTHEREFR is adopted as the sparse inference engine.

1) Feature Extraction: Using sparse convolution as the essential operation, the feature extraction module is carefully designed to enable high-level embedding prediction.

Following learning-based compression methods #OTHEREFR , #OTHEREFR , the proposed approach works in a bottom-up manner by progressively reducing the spatial resolution and exponentially increasing the receptive field. As shown in Fig.

3(a) , each downsampling block in the feature extractor comprises a strided convolution, a VRN unit #OTHEREFR , #OTHEREFR , and another convolution layer in a cascading manner.

Specifically, the strided convolution reduces the number of points, while the subsequent convolution layer refines extracted features for optimal performance.

By stacking these downsampling blocks, the feature extraction module outputs multiscale sparse tensors, including coordinates and features, that form the latent representation of an entire point cloud.

As such, the extracted features can reconstruct the original geometry information in a hierarchical manner with only a small number of output coordinates as indispensable information to decrease required bits.

We losslessly encode these coordinates using a coordinate encoder #OTHEREFR , and further process multiscale features in the following modules.

2) Feature Warping: The feature warping module is designed to transform the extracted features from the aligned point cloud to the coordinates of the source point cloud.

This process is completed in a coarse-to-fine manner, with each scale entailing a hierarchical concatenation of primary sparse tensors X from the feature extractor and auxiliary sparse tensors X from the previous block, as illustrated in Fig. 3(b) .

Geometric Prior Based Deep Human Point Cloud Geometry Compression

Abstract: The emergence of digital avatars has raised an exponential increase in the demand for human point clouds with realistic and intricate details. The compression of such data becomes challenging with overwhelming data amounts comprising millions of points. Herein, we leverage the human geometric prior in geometry redundancy removal of point clouds, greatly promoting the compression performance. More specifically, the prior provides topological constraints as geometry initialization, allowing adaptive adjustments with a compact parameter set that could be represented with only a few bits. Therefore, we can envisage high-resolution human point clouds as a combination of geometric priors and structural deviations. The priors could first be derived with an aligned point cloud, and subsequently the difference of features is compressed into a compact latent code. The proposed framework can operate in a play-and-plug fashion with existing learning based point cloud compression methods. Extensive experimental results show that our approach significantly improves the compression performance without deteriorating the quality, demonstrating its promise in a variety of applications.

Generative And Discriminative Voxel Modeling With Convolutional Neural Networks

Abstract: When working with three-dimensional data, choice of representation is key. We explore voxel-based models, and present evidence for the viability of voxellated representations in applications including shape modeling and object classification. Our key contributions are methods for training voxel-based variational autoencoders, a user interface for exploring the latent space learned by the autoencoder, and a deep convolutional neural network architecture for object classification. We address challenges unique to voxel-based representations, and empirically evaluate our models on the ModelNet benchmark, where we demonstrate a 51.5% relative improvement in the state of the art for object classification.


At inference time, we use the VGG network for calculating LPIPS #REFR after normalizing pixel values to [-1,1].

We firstly provide experimental details general to all the settings, then present contents specific to different settings.

Common configurations As for training, we use a batch size of 16 object instances, with the number of rays sampled for each object set to 128 and 256 for pretraining and fine tuning, respectively.

The coefficient λ for loss term L mrvm is set to 0.1 at pretraining stage.

We perform ray casting, sampling and volume rendering all in the world coordinate. All the models are implemented using Pytorch #OTHEREFR framework.

For all the settings performed on synthetic datasets except for DTU, note that the images in synthetic dataset all have a blank background, we adopt two technics for better performance: i) We use bounding box sampling strategy at pretraining stage.

The training rays are only sampled within the bounding box of the foreground object.

In this way, it avoids the model to be too empty at initial stage.

ii) We assign a white background color for those pixels sampled from the background to match the renderings in ShapeNet dataset.

Mrvm-Nerf: Mask-Based Pretraining For Neural Radiance Fields

Abstract: Most Neural Radiance Fields (NeRFs) have poor generalization ability, limiting their application when representing multiple scenes by a single model. To ameliorate this problem, existing methods simply condition NeRF models on image features, lacking the global understanding and modeling of the entire 3D scene. Inspired by the significant success of mask-based modeling in other research fields, we propose a masked ray and view modeling method for generalizable NeRF (MRVM-NeRF), the first attempt to incorporate mask-based pretraining into 3D implicit representations. Specifically, considering that the core of NeRFs lies in modeling 3D representations along the rays and across the views, we randomly mask a proportion of sampled points along the ray at fine stage by discarding partial information obtained from multi-viewpoints, targeting at predicting the corresponding features produced in the coarse branch. In this way, the learned prior knowledge of 3D scenes during pretraining helps the model generalize better to novel scenarios after finetuning. Extensive experiments demonstrate the superiority of our proposed MRVM-NeRF under various synthetic and real-world settings, both qualitatively and quantitatively. Our empirical studies reveal the effectiveness of our proposed innovative MRVM which is specifically designed for NeRF models.

The Unreasonable Effectiveness Of Deep Features As A Perceptual Metric

Abstract: While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on the ImageNet classification task has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called"perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new Full Reference Image Quality Assessment (FR-IQA) dataset of perceptual human judgments, orders of magnitude larger than previous datasets. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by huge margins. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.