The nuScenes dataset #REFR contains 1000 sequences of approximately 20s duration with a LiDAR frequency of 20 FPS. The box annotations are provided for every 0.5s.

Each frame has one point cloud and six images covering 360 • horizontal FOV.

In total, there are 40k frames which are split into 28k, 6k, 6k for training, validation, and testing.

As the dataset provides full annotations and information of vehicle pose and timestamp, we can simulate all corruptions.

Thus, we apply all 27 corruptions to the nuScenes validation set with 5 severities to obtain nuScenes-C.

For 3D object detection, the main evaluation metrics are mean Average Precision (mAP) and nuScenes detection score (NDS) computed on 10 object categories.

Benchmarking Robustness Of 3D Object Detection To Common Corruptions In Autonomous Driving

Abstract: 3D object detection is an important task in autonomous driving to perceive the surroundings. Despite the excellent performance, the existing 3D detectors lack the robustness to real-world corruptions caused by adverse weathers, sensor noises, etc., provoking concerns about the safety and reliability of autonomous driving systems. To comprehensively and rigorously benchmark the corruption robustness of 3D detectors, in this paper we design 27 types of common corruptions for both LiDAR and camera inputs considering real-world driving scenarios. By synthesizing these corruptions on public datasets, we establish three corruption robustness benchmarks-KITTI-C, nuScenes-C, and Waymo-C. Then, we conduct large-scale experiments on 24 diverse 3D object detection models to evaluate their corruption robustness. Based on the evaluation results, we draw several important findings, including: 1) motion-level corruptions are the most threatening ones that lead to significant performance drop of all models; 2) LiDAR-camera fusion models demonstrate better robustness; 3) camera-only models are extremely vulnerable to image corruptions, showing the indispensability of LiDAR point clouds. We release the benchmarks and codes at https://github.com/kkkcx/ 3D_Corruptions_AD. We hope that our benchmarks and findings can provide insights for future research on developing robust 3D object detection models.


Namely, the slope of the lines of equal accuracy is determined by the base rates of the outcome within the group #REFR .

The circle at the intersection of the two curves represents the single operating point for which the false positive and false negative rates of the model are equal for both groups (equalized odds), while still being at an "optimal" operating point for both groups.

In other words, all fair operating points besides the circle require a strict trade-off with the model's FP and/or FN rates.

Because the base rates differ between groups, an additional problem arises: operating at the circle means that the model is miscalibrated for one or both of the groups.

Consider the dotted lines, which represent levels of accuracy for the model on each group.

Accuracy is the number of correct predictions divided by the total number of predictions, and is tightly linked to calibration.

Therefore, the model is both calibrated and equally accurate for both groups wherever the lines of equal accuracy intersect within the feasible region.

We denote such an intersection by the square, which, notably, is not on either group's ROC curve. The square and circle will only intersect, i.e.

equalized odds and fair calibration be simultaneously achieved, if the two two groups have identical base rates or the classifier is perfect on both groups.

Thus we have not only an error-fairness, but also a fairness-fairness tradeoff among operating points in the model.

We can relax equalized odds by dividing it into simpler constraints that only ask for equivalence among the positive samples negative ones.

Optimizing Fairness Tradeoffs In Machine Learning With Multiobjective Meta-Models

Abstract: Improving the fairness of machine learning models is a nuanced task that requires decision makers to reason about multiple, conflicting criteria. The majority of fair machine learning methods transform the error-fairness trade-off into a single objective problem with a parameter controlling the relative importance of error versus fairness. We propose instead to directly optimize the errorfairness tradeoff by using multi-objective optimization. We present a flexible framework for defining the fair machine learning task as a weighted classification problem with multiple cost functions. This framework is agnostic to the underlying prediction model as well as the metrics. We use multiobjective optimization to define the sample weights used in model training for a given machine learner, and adapt the weights to optimize multiple metrics of fairness and accuracy across a set of tasks. To reduce the number of optimized parameters, and to constrain their complexity with respect to population subgroups, we propose a novel meta-model approach that learns to map protected attributes to sample weights, rather than optimizing those weights directly. On a set of real-world problems, this approach outperforms current state-of-the-art methods by finding solution sets with preferable error/fairness trade-offs. CCS CONCEPTS • Computing methodologies → Cost-sensitive learning; Supervised learning by classification; Multi-task learning; Randomized search.

On Fairness And Calibration

Abstract: The machine learning community has become increasingly concerned with the potential for bias and discrimination in predictive models, and this has motivated a growing line of work on what it means for a classification procedure to be"fair."In particular, we investigate the tension between minimizing error disparity across different population groups while maintaining calibrated probability estimates. We show that calibration is compatible only with a single error constraint (i.e. equal false-negatives rates across groups), and show that any algorithm that satisfies this relaxation is no better than randomizing a percentage of predictions for an existing classifier. These unsettling findings, which extend and generalize existing results, are empirically confirmed on several datasets.


We employ the proximal policy optimization (PPO) #REFR algorithm to update the model weights during training.

They also suffer from the problems of mode collapse #OTHEREFR , class imbalance #OTHEREFR , and computational expense #OTHEREFR when generating high-dimensional data.

But what we care about most is that a lot of synthetic samples generated are used directly for training a classifier without studying if these samples actually help the classifier learn. Instead, these samples are chosen based on "realness".

Figure 1 shows a comparison of the standard pipeline and our proposed pipeline for feature-generating approaches.

To address the limitations of GANs in synthetic feature selection, we propose a novel reinforcement learning-based approach that automatically selects generated features that improve model performance.

Specifically, we use a transformer model #OTHEREFR for synthetic sample selection and use validation classification accuracy as the reward for RL training.

Our proposed approach aims to pick samples that help classification and not just generate real-looking samples.

We dub our synthetic sample selection method as "SPOT" for Selection using Proximal policy OpTimization.

Furthermore, our proposed approach is model-agnostic and data-agnostic, as we evaluate our method on multiple benchmark datasets in images and videos and various feature-generating models.

Our comprehensive experiments demonstrate that our approach consistently improves model performance across different datasets and models, highlighting the effectiveness and versatility of our proposed method.

By leveraging RL-based synthetic feature selection, we can more effectively generate synthetic data that captures the underlying structure of the data, improving the generalization performance of downstream models.

Synthetic Sample Selection For Generalized Zero-Shot Learning

Abstract: Generalized Zero-Shot Learning (GZSL) has emerged as a pivotal research domain in computer vision, owing to its capability to recognize objects that have not been seen during training. Despite the significant progress achieved by generative techniques in converting traditional GZSL to fully supervised learning, they tend to generate a large number of synthetic features that are often redundant, thereby increasing training time and decreasing accuracy. To address this issue, this paper proposes a novel approach for synthetic feature selection using reinforcement learning. In particular, we propose a transformer-based selector that is trained through proximal policy optimization (PPO) to select synthetic features based on the validation classification accuracy of the seen classes, which serves as a reward. The proposed method is model-agnostic and data-agnostic, making it applicable to both images and videos and versatile for diverse applications. Our experimental results demonstrate the superiority of our approach over existing feature-generating methods, yielding improved overall performance on multiple benchmarks.

Proximal Policy Optimization Algorithms

Abstract: We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.


When extracting images from the original the ImageNet-1K dataset to create the ImageNet-100 dataset, we select the 100 classes used in #REFR .

In the experiments, we use the following datasets: CIFAR-10 ( #OTHEREFR , STL-10 (Coates et al., 2011) , and ImageNet-100 #OTHEREFR .

Note that ImageNet-100 is a subset of the ImageNet-1K dataset #OTHEREFR , where the ImageNet-100 dataset contains images categorized in 100 classes #OTHEREFR .

We also remark that for the experiments with the STL-10 dataset, we use the mixed dataset that consists of the unlabeled images and the labeled training images for pretraining, the labeled training images for the training of the linear head in the stage of linear evaluation, and the labeled test images for computing the accuracy in linear evaluation.

Throughout the experiments, we use the following image size for each dataset: 32 × 32 pixels for CIFAR-10, 96 × 96 pixels for STL-10, and 224 × 224 pixels for ImageNet-100, where the image sizes of CIFAR-10 and STL-10 are the same as the sizes of the original images, respectively, and the image sizes for ImageNet-100 are inspired by those for the ImageNet-1K dataset used in #OTHEREFR ; #OTHEREFR .

Towards Understanding The Mechanism Of Contrastive Learning Via Similarity Structure: A Theoretical Analysis

Abstract: Contrastive learning is an efficient approach to self-supervised representation learning. Although recent studies have made progress in the theoretical understanding of contrastive learning, the investigation of how to characterize the clusters of the learned representations is still limited. In this paper, we aim to elucidate the characterization from theoretical perspectives. To this end, we consider a kernel-based contrastive learning framework termed Kernel Contrastive Learning (KCL), where kernel functions play an important role when applying our theoretical results to other frameworks. We introduce a formulation of the similarity structure of learned representations by utilizing a statistical dependency viewpoint. We investigate the theoretical properties of the kernel-based contrastive loss via this formulation. We first prove that the formulation characterizes the structure of representations learned with the kernel-based contrastive learning framework. We show a new upper bound of the classification error of a downstream task, which explains that our theory is consistent with the empirical success of contrastive learning. We also establish a generalization error bound of KCL. Finally, we show a guarantee for the generalization ability of KCL to the downstream classification task via a surrogate bound.

Contrastive Multiview Coding

Abstract: Humans view the world through many sensory channels, e.g., the long-wavelength light channel, viewed by the left eye, or the high-frequency vibrations channel, heard by the right ear. Each view is noisy and incomplete, but important factors, such as physics, geometry, and semantics, tend to be shared between all views (e.g., a "dog" can be seen, heard, and felt). We hypothesize that a powerful representation is one that models view-invariant factors. Based on this hypothesis, we investigate a contrastive coding scheme, in which a self-supervsied representation is learned that aims to maximize mutual information between different views but is otherwise compact. Our approach scales to any number of views, and is view-agnostic. The resulting learned representations perform above the state of the art for downstream tasks such as object classification, compared to formulations based on predictive learning or single view reconstruction, and improve as more views are added. On the Imagenet linear readoff benchmark, we achieve 68.4% top-1 and 88.2% top-5 accuracies. Code and reference implementations are released on our project page: this http URL.


A (higher indicates better performance) and F (lower indicates less forgetting) are calculated averagely over all learned tasks, while the detailed ACC is calculated on per learned task #REFR . Compared Methods.

Each class has more than 1, 000 images for training and 500 for testing.

Each image is resized to 256×256 before randomly sampling 224 × 224 crops for training.

For testing, the original center crop with 224 × 224 shapes is used.

We employ average incremental accuracy (A), average forgetting measure (F ), and classification accuracies (ACC) for evaluation #OTHEREFR .

We compare two reference methods, fine-tuning (Fine) and joint training (Joint) #OTHEREFR , and several SOTA methods on CIFAR-100 and ImageNet-sub.

They include two parameter isolation methods, LwF #OTHEREFR and LwM #OTHEREFR , two regularization-based methods, EWC #OTHEREFR and MAS #OTHEREFR , two extractoraimed methods, IL2M #OTHEREFR and Lucir #OTHEREFR , and the prototype-aimed method, GFR .

Besides, we directly introduce the embedding distillation into each SOTA method to obtain a modified eXX method, where XX is LwF, LwM, MAS, IL2M, Lucir, or GFR.

Fine updates the network on the newly arrived tasks, while Joint assumes data of all previous tasks are available in each training task.

EWC, MAS, LwF, LwM, and GFR are trained without exemplars, whereas IL2M and Lucir store 20 exemplars for each learned class.

Etag: Class-Incremental Learning With Embedding Distillation And Task-Oriented Generation

Abstract: Class-Incremental Learning (CIL) aims to solve the neural networks' catastrophic forgetting problem, which refers to the fact that once the network updates on a new task, its performance on previouslylearned tasks drops dramatically. Most successful CIL methods incrementally train a feature extractor with the aid of stored exemplars, or estimate the feature distribution with the stored prototypes. However, the stored exemplars would violate the data privacy concerns, while the stored prototypes might not reasonably be consistent with a proper feature distribution, hindering the exploration of real-world CIL applications. In this paper, we propose a method of embedding distillation and Taskoriented generation (eTag) for CIL, which requires neither the exemplar nor the prototype. Instead, eTag achieves a data-free manner to train the neural networks incrementally. To prevent the feature extractor from forgetting, eTag distills the embeddings of the network's intermediate blocks. Additionally, eTag enables a generative network to produce suitable features, fitting the needs of the top incremental classifier. Experimental results confirmed that our proposed eTag considerably outperforms the state-of-the-art methods on CIFAR-100 and ImageNet-sub 1 .

Semantic Drift Compensation For Class-Incremental Learning

Abstract: Class-incremental learning of deep networks sequentially increases the number of classes to be classified. During training, the network has only access to data of one task at a time, where each task contains several classes. In this setting, networks suffer from catastrophic forgetting which refers to the drastic drop in performance on previous tasks. The vast majority of methods have studied this scenario for classification networks, where for each new task the classification layer of the network must be augmented with additional weights to make room for the newly added classes. Embedding networks have the advantage that new classes can be naturally included into the network without adding new weights. Therefore, we study incremental learning for embedding networks. In addition, we propose a new method to estimate the drift, called semantic drift, of features and compensate for it without the need of any exemplars. We approximate the drift of previous tasks based on the drift that is experienced by current task data. We perform experiments on fine-grained datasets, CIFAR100 and ImageNet-Subset. We demonstrate that embedding networks suffer significantly less from catastrophic forgetting. We outperform existing methods which do not require exemplars and obtain competitive results compared to methods which store exemplars. Furthermore, we show that our proposed SDC when combined with existing methods to prevent forgetting consistently improves results.


I Comparison With Augmentation-Based Unsupervised

However, unlike the image domain, they performed worse than CACTUs #REFR , as shown in Table 11 .

META-LEARNING SCHEMES Table 11 : Few-shot test accuracy (%) on 8 datasets from the OpenML-CC18 benchmark #OTHEREFR .

We report the mean test accuracy over 100 different seeds. The bold denotes the highest mean score.

We evaluate UMTRA #OTHEREFR and SES #OTHEREFR ) (also utilizing SNS proposed by Ye et al.

(2022)) on few-shot tabular learning tasks, where we use augmentation strategies used in SubTab #OTHEREFR ) (i.e., Gaussian noise and marginal distribution masking).

Here, we tried our best to improve the performance of SES and UMTRA (e.g., tune variance of Gaussian noise).

We believe that the failures of SES and UMTRA are mainly due to the absence of effective augmentation strategies for tabular data, and developing them will be an interesting future direction.

Stunt: Few-Shot Tabular Learning With Self-Generated Tasks From Unlabeled Tables

Abstract: Learning with few labeled tabular samples is often an essential requirement for industrial machine learning applications as varieties of tabular data suffer from high annotation costs or have difficulties in collecting new samples for novel tasks. Despite the utter importance, such a problem is quite under-explored in the field of tabular learning, and existing few-shot learning schemes from other domains are not straightforward to apply, mainly due to the heterogeneous characteristics of tabular data. In this paper, we propose a simple yet effective framework for few-shot semi-supervised tabular learning, coined Self-generated Tasks from UNlabeled Tables (STUNT). Our key idea is to self-generate diverse few-shot tasks by treating randomly chosen columns as a target label. We then employ a meta-learning scheme to learn generalizable knowledge with the constructed tasks. Moreover, we introduce an unsupervised validation scheme for hyperparameter search (and early stopping) by generating a pseudo-validation set using STUNT from unlabeled data. Our experimental results demonstrate that our simple framework brings significant performance gain under various tabular few-shot learning benchmarks, compared to prior semi-and self-supervised baselines. Code is available at https://github.com/jaehyun513/STUNT.

Unsupervised Learning Via Meta-Learning

Abstract: A central goal of unsupervised learning is to acquire representations from unlabeled data or experience that can be used for more effective learning of downstream tasks from modest amounts of labeled data. Many prior unsupervised learning works aim to do so by developing proxy objectives based on reconstruction, disentanglement, prediction, and other metrics. Instead, we develop an unsupervised meta-learning method that explicitly optimizes for the ability to learn a variety of tasks from small amounts of data. To do so, we construct tasks from unlabeled data in an automatic way and run meta-learning over the constructed tasks. Surprisingly, we find that, when integrated with meta-learning, relatively simple task construction mechanisms, such as clustering embeddings, lead to good performance on a variety of downstream, human-specified tasks. Our experiments across four image datasets indicate that our unsupervised meta-learning approach acquires a learning algorithm without any labeled data that is applicable to a wide range of downstream classification tasks, improving upon the embedding learned by four prior unsupervised learning methods.


Furthermore, similar to #REFR , we find the optimal number of total communication rounds that achieves the best model performance at a fixed privacy budget.

Nevertheless, their algorithm requires performing the exact arg min on local clients, and their theory works only under very restrictive assumptions that implicitly require no collaboration, as each client is assumed to have the same unique minimizer.

In this work, we bridge this gap and provide, to our knowledge, the first analysis of the standard FL algorithms with DP under realistic assumptions with the application to medical image analysis. Our contributions can be summarized as follows:

We analyze one of the most popular algorithms for FL with DP -DP-FedAvg #OTHEREFR under the standard non-restrictive assumptions.

Our analysis reveals that clients benefit from running local steps, and there is an optimal number of local steps each client should take.

We provide an extensive empirical evaluation of the analyzed method.

We show optimal values for the number of local steps and communication rounds that maximize the practical performance of DP-FedAvg as predicted by our theory.

On top of that, the tuned model achieves similar performance as the centralized model, i.e., the model that has access to the full dataset in a single location while providing rigorous privacy guarantees, which are essential for sensitive medical imaging applications.

Improving Performance Of Private Federated Models In Medical Image Analysis

Abstract: Federated learning (FL) is a distributed machine learning (ML) approach that allows data to be trained without being centralized. This approach is particularly beneficial for medical applications because it addresses some key challenges associated with medical data, such as privacy, security, and data ownership. On top of that, FL can improve the quality of ML models used in medical applications. Medical data is often diverse and can vary significantly depending on the patient population, making it challenging to develop ML models that are accurate and generalizable. FL allows medical data to be used from multiple sources, which can help to improve the quality and generalizability of ML models. Differential privacy (DP) is a go-to algorithmic tool to make this process secure and private. In this work, we show that the model performance can be further improved by employing local steps, a popular approach to improving the communication efficiency of FL, and tuning the number of communication rounds. Concretely, given the privacy budget, we show an optimal number of local steps and communications rounds. We provide theoretical motivations further corroborated with experimental evaluations on real-world medical imaging tasks.

Federated Learning With Differential Privacy: Algorithms And Performance Analysis

Abstract: In this paper, to effectively prevent information leakage, we propose a novel framework based on the concept of differential privacy (DP), in which artificial noises are added to the parameters at the clients side before aggregating, namely, noising before model aggregation FL (NbAFL). First, we prove that the NbAFL can satisfy DP under distinct protection levels by properly adapting different variances of artificial noises. Then we develop a theoretical convergence bound of the loss function of the trained FL model in the NbAFL. Specifically, the theoretical bound reveals the following three key properties: 1) There is a tradeoff between the convergence performance and privacy protection levels, i.e., a better convergence performance leads to a lower protection level; 2) Given a fixed privacy protection level, increasing the number $N$ of overall clients participating in FL can improve the convergence performance; 3) There is an optimal number of maximum aggregation times (communication rounds) in terms of convergence performance for a given protection level. Furthermore, we propose a $K$-random scheduling strategy, where $K$ ($1<K<N$) clients are randomly selected from the $N$ overall clients to participate in each aggregation. We also develop the corresponding convergence bound of the loss function in this case and the $K$-random scheduling strategy can also retain the above three properties. Moreover, we find that there is an optimal $K$ that achieves the best convergence performance at a fixed privacy level. Evaluations demonstrate that our theoretical results are consistent with simulations, thereby facilitating the designs on various privacy-preserving FL algorithms with different tradeoff requirements on convergence performance and privacy levels.


This is in contrast to pure 2D keypoint-based matching approaches #REFR which rely on local texture similarities and do not exploit object semantics.

On the object-level, the matching features are determined by an associative graph neural network that reasons *This work has been supported by Cyber Valley and the Max Planck Society.

We are grateful to the Max Planck ETH Center for Learning Systems for supporting Cathrin Elich.

1 Cathrin Elich and Joerg Stueckler are with Embodied Vision Group, Max Planck Institute for Intelligent Systems, Tuebingen, Germany firstname.lastname@tue.mpg.de #OTHEREFR Cathrin Elich is with the Max Planck ETH Center for Learning Systems 3 Iro Armeni and Marc Pollefeys are with the Computer Vision and Geometry Lab, ETH Zurich, Switzerland armeni@ibi.baug.ethz.ch, marc.pollefeys@inf.ethz.ch on spatial and appearance features of object detections in both input images.

To incorporate spatial knowledge into our matching pipeline, we additionally train our features on semantic and spatial auxiliary tasks such as inferring object class, 3D position, and relative distance between objects.

By this, objects can be matched based on semantic and spatial information across large view point changes.

We evaluate our approach on a realistically rendered indoor dataset #OTHEREFR and demonstrate improved accuracy over state-of-the-art object matching approaches.

For large view-point changes, our approach also compares favorably in object matching to a baseline that uses the stateof-the-art keypoint matching method in accuracy and recall.

We provide an ablation study to analyze the contributions of the individual design choices in our method.

In summary, we make the following contributions: (1) We present a novel learning approach for learning relational object features suited for matching detected objects between image pairs.

Our approach uses an attentional graph neural network based on appearance and spatial features extracted from the object bounding boxes.

Learning-Based Relational Object Matching Across Views

Abstract: Intelligent robots require object-level scene understanding to reason about possible tasks and interactions with the environment. Moreover, many perception tasks such as scene reconstruction, image retrieval, or place recognition can benefit from reasoning on the level of objects. While keypoint-based matching can yield strong results for finding correspondences for images with small to medium view point changes, for large view point changes, matching semantically on the object-level becomes advantageous. In this paper, we propose a learningbased approach which combines local keypoints with novel object-level features for matching object detections between RGB images. We train our object-level matching features based on appearance and inter-frame and cross-frame spatial relations between objects in an associative graph neural network. We demonstrate our approach in a large variety of views on realistically rendered synthetic images. Our approach compares favorably to previous state-of-the-art object-level matching approaches and achieves improved performance over a pure keypoint-based approach for large viewpoint changes.

Superglue: Learning Feature Matching With Graph Neural Networks

Abstract: This paper introduces SuperGlue, a neural network that matches two sets of local features by jointly finding correspondences and rejecting non-matchable points. Assignments are estimated by solving a differentiable optimal transport problem, whose costs are predicted by a graph neural network. We introduce a flexible context aggregation mechanism based on attention, enabling SuperGlue to reason about the underlying 3D scene and feature assignments jointly. Compared to traditional, hand-designed heuristics, our technique learns priors over geometric transformations and regularities of the 3D world through end-to-end training from image pairs. SuperGlue outperforms other learned approaches and achieves state-of-the-art results on the task of pose estimation in challenging real-world indoor and outdoor environments. The proposed method performs matching in real-time on a modern GPU and can be readily integrated into modern SfM or SLAM systems. The code and trained weights are publicly available at https://github.com/magicleap/SuperGluePretrainedNetwork.


Metrics We numerically evaluate the results using two complementary metrics: LPIPS #REFR to quantify the extent of structure preservation (lower is better) and a CLIP-based score to quantify how well the generated images comply with the text prompt (higher is better).

Datasets We use two datasets of real images: (i) "modified ImageNet-R-TI2I" from #OTHEREFR with additional examples collected from the internet and from other datasets, (ii) "modified Zero-Shot I2IT", which contains images of 4 classes (Cat, Dog, Horse, Zebra) from #OTHEREFR and from the internet. Figure 9 : Comparisons.

We show results for editing of real images using our inversion by itself and in combination with other methods.

Our approach maintains high fidelity to the input while conforming to the text prompt.

The first dataset comprises 48 images, with 3-5 different target prompts for each image. This results in a total of 212 image-text pairs.

The second dataset has 15 images in each category with one target prompt for each, making 60 image-text pairs in total. Please see App. D for full details.

We also measure running time in seconds, and diversity among generated outputs (higher is better).

Specifically, for each image and source text p src , we generate 8 outputs with target text p tar and calculate the average LPIPS distance over all ( 8 2 ) pairs. Figure 10 : Improving Zero-Shot I2I Translation.

Images generated by Zero Shot I2I suffer from loss of detail.

With our inversion, fine textures, like fur and flowers, are retained. All results use the default cross-attention guidance weight (0.1).

An Edit Friendly Ddpm Noise Space: Inversion And Manipulations

Abstract: Prompt-to-prompt + Our DDPM inversion Zero Shot + Our DDPM inversion cat dog → Our DDPM inversion A sketch of a cat a → A sculpture of a cat A sketch of a cat → A smiling cat A painting of a cat with white flowers→ A painting of a dog with white flowers Real image Our DDPM inversion Zero Shot Our DDPM inversion Figure 1: Edit friendly DDPM inversion. We present a method for extracting a sequence of DDPM noise maps that perfectly reconstruct a given image. These noise maps are distributed differently from those used in regular sampling, and are more edit-friendly. Our method allows diverse editing of real images without fine-tuning the model or modifying its attention maps, and it can also be easily integrated into other algorithms (illustrated here with Prompt-to-Prompt [7] and Zero-Shot I2I [19]).

The Unreasonable Effectiveness Of Deep Features As A Perceptual Metric

Abstract: While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on the ImageNet classification task has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called"perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new Full Reference Image Quality Assessment (FR-IQA) dataset of perceptual human judgments, orders of magnitude larger than previous datasets. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by huge margins. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.


The nuScenes dataset #REFR is a largescale autonomous-driving dataset with 3D object annotations that contains 1000 driving sequences, with 700, 150, and 150 sequences for training, validation, and testing, respectively.

In this section, we first compare state-of-the-art methods on nuScenes.

Then we designed three experiments with LiDAR corruption to show the modal complementarity of CrossFusion at low-quality LiDAR features.

Afterward, we conduct extensive ablation experiments to demonstrate the rationality and importance of each key component of our CrossFusion. nuScenes Dataset.

Each sequence is approximately 20-second long but only provides box annotations every ten frames (0.5s).

Each frame contains a wealth of sensor information, including point clouds and carefully calibrated six 360-degree horizontal FOV images at the same timestamp.

For 3D detection task, the main metrics are mean Average Precision (mAP) and nuScenes detection score (NDS mAP defines a match by considering the 2D center distance thresholds of 0.5m, 1m, 2m, 4m across ten classes on the ground plane.

NDS is a weighted average of mAP and other attribute metrics, including translation(mATE), scale(mASE), orientation(mAOE), velocity(mAVE), and attribute(mAAE).

Crossfusion: Interleaving Cross-Modal Complementation For Noise-Resistant 3D Object Detection

Abstract: The combination of LiDAR and camera modalities is proven to be necessary and typical for 3D object detection according to recent studies. Existing fusion strategies tend to overly rely on the LiDAR modal in essence, which exploits the abundant semantics from the camera sensor insufficiently. However, existing methods cannot rely on information from other modalities because the corruption of Li-DAR features results in a large domain gap. Following this, we propose CrossFusion, a more robust and noise-resistant scheme that makes full use of the camera and LiDAR features with the designed cross-modal complementation strategy. Extensive experiments we conducted show that our method not only outperforms the state-of-the-art methods under the setting without introducing an extra depth estimation network but also demonstrates our model's noise resistance without retraining for the specific malfunction scenarios by increasing 5.2% mAP and 2.4% NDS.

Nuscenes: A Multimodal Dataset For Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image-based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first published dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online at this http URL.

3D object annotations
largescale autonomous-driving dataset


Convergence Rates For Abstract Wave Equations

In conclusion, we obtain the following generalisation of #REFR Cor. 4.3] to contractive time discretisation schemes.

Hence, G satisfies the linear growth condition of Assumption 7.1 with δ = 1−ε 2 .

Repeating the arguments for Λ −1/2 [G(u 1 ) − G(u 2 )] and using c 1 = π 2 /6 results in

The nonlinearity F was already considered in Subsection 7.4.

Theorem 7.11 (Wave equation with white noise).

Suppose that (u 0 , v 0 ) ∈ L p F0 (Ω; X δ ).

Let F and G be Nemytskij operators as above with φ and ψ satisfying (7.14) .

Suppose the covariance operator Q = I on L 2 (O). Let Y = X δ .

Pathwise Uniform Convergence Of Time Discretisation Schemes For Spdes

Abstract: In this paper we prove convergence rates for time discretisation schemes for semilinear stochastic evolution equations with additive or multiplicative Gaussian noise, where the leading operator A is the generator of a strongly continuous semigroup S on a Hilbert space X, and the focus is on non-parabolic problems. The main results are optimal bounds for the uniform strong error E ∞ k := E sup j∈{0,...,N k } U (t j) − U j p 1/p , where p ∈ [2, ∞), U is the mild solution, U j is obtained from a time discretisation scheme, k is the step size, and N k = T /k. The usual schemes such as splitting/exponential Euler, implicit Euler, and Crank-Nicolson, etc. are included as special cases. Under conditions on the nonlinearity and the noise we show • E ∞ k k log(T /k) (linear equation, additive noise, general S); • E ∞ k √ k log(T /k) (nonlinear equation, multiplicative noise, contractive S); • E ∞ k k log(T /k) (nonlinear wave equation, multiplicative noise). The logarithmic factor can be removed if the splitting scheme is used with a (quasi)-contractive S. The obtained bounds coincide with the optimal bounds for SDEs. Most of the existing literature is concerned with bounds for the simpler pointwise strong error E k := sup j∈{0,...,N k } E U (t j) − U j p 1/p. Applications to Maxwell equations, Schrödinger equations, and wave equations are included. For these equations our results improve and reprove several existing results with a unified method.

An Exponential Integrator Scheme For Time Discretization Of Nonlinear Stochastic Wave Equation

Abstract: This work is devoted to convergence analysis of an exponential integrator scheme for semi-discretization in time of nonlinear stochastic wave equation. A unified framework is first set forth, which covers important cases of additive and multiplicative noises. Within this framework, the proposed scheme is shown to converge uniformly in the strong $L^p$-sense with precise convergence rates given. The abstract results are then applied to several concrete examples. Further, weak convergence rates of the scheme are examined for the case of additive noise. To analyze the weak error for the nonlinear case, techniques based on the Malliavin calculus were usually exploited in the literature. Under certain appropriate assumptions on the nonlinearity, this paper provides a weak error analysis, which does not rely on the Malliavin calculus. The rates of weak convergence can, as expected, be improved in comparison with the strong rates. Both strong and weak convergence results obtained here show that the proposed method achieves higher convergence rates than the implicit Euler and Crank-Nicolson time discretizations. Numerical results are finally reported to confirm our theoretical findings.


The recent movement is to combine incremental Delta-Bar-Delta method and Temporal-difference learning #REFR .

HPO in the RL traditionally used a Delta-Bar-Delta method as incremental algorithms to tune parameters #OTHEREFR .

However, this method and its variations were limited to linear supervised learning.

Those methods can not make tuning hyperparameters online and allow the algorithm to more robustly adjust to non-stationarity in a problem at the same time.

A variety of techniques exist to combat this recentlymost notably use of a large experience replay buffers or the use of multiple parallel actors.

These techniques come at the cost of moving away from the online RL problem as it is traditionally formulated.

More sophisticated approaches include Self-Tuning Actor Critic (STAC) #OTHEREFR and Sequential Model-based Bayesian optimisation (SMBO) #OTHEREFR .

However both methods ignored a crucial issue for RL: the possible non-stationarity of the RL problem induces a possible non-stationarity of the hyperparameters.

A Framework For History-Aware Hyperparameter Optimisation In Reinforcement Learning

Abstract: A Reinforcement Learning (RL) system depends on a set of initial conditions (hyperparameters) that affect the system's performance. However, defining a good choice of hyperparameters is a challenging problem. Hyperparameter tuning often requires manual or automated searches to find optimal values. Nonetheless, a noticeable limitation is the high cost of algorithm evaluation for complex models, making the tuning process computationally expensive and time-consuming. In this paper, we propose a framework based on integrating complex event processing and temporal models, to alleviate these trade-offs. Through this combination, it is possible to gain insights about a running RL system efficiently and unobtrusively based on data stream monitoring and to create abstract representations that allow reasoning about the historical behaviour of the RL system. The obtained knowledge is exploited to provide feedback to the RL system for optimising its hyperparameters while making effective use of parallel resources. We introduce a novel history-aware epsilon-greedy logic for hyperparameter optimisation that instead of using static hyperparameters that are kept fixed for the whole training, adjusts the hyperparameters at runtime based on the analysis of the agent's performance over time windows in a single agent's lifetime. We tested the proposed approach in a 5G mobile communications case study that uses DQN, a variant of RL, for its decision-making. Our experiments demonstrated the effects of hyperparameter tuning using history on training stability and reward values. The encouraging results show that the proposed history-aware framework significantly improved performance compared to traditional hyperparameter tuning approaches.

Metatrace: Online Step-Size Tuning By Meta-Gradient Descent For Reinforcement Learning Control

Abstract: Reinforcement learning (RL) has had many successes in both "deep" and "shallow" settings. In both cases, significant hyperparameter tuning is often required to achieve good performance. Furthermore, when nonlinear function approximation is used, non-stationarity in the state representation can lead to learning instability. A variety of techniques exist to combat this --- most notably large experience replay buffers or the use of multiple parallel actors. These techniques come at the cost of moving away from the online RL problem as it is traditionally formulated (i.e., a single agent learning online without maintaining a large database of training examples). Meta-learning can potentially help with both these issues by tuning hyperparameters online and allowing the algorithm to more robustly adjust to non-stationarity in a problem. This paper applies meta-gradient descent to derive a set of step-size tuning algorithms specifically for online RL control with eligibility traces. Our novel technique, Metatrace, makes use of an eligibility trace analogous to methods like $TD(\lambda)$. We explore tuning both a single scalar step-size and a separate step-size for each learned parameter. We evaluate Metatrace first for control with linear function approximation in the classic mountain car problem and then in a noisy, non-stationary version. Finally, we apply Metatrace for control with nonlinear function approximation in 5 games in the Arcade Learning Environment where we explore how it impacts learning speed and robustness to initial step-size choice. Results show that the meta-step-size parameter of Metatrace is easy to set, Metatrace can speed learning, and Metatrace can allow an RL algorithm to deal with non-stationarity in the learning task.


In the spiking transformer, the attention score is normalized Hamming similarity between binary spike vectors, which has been proven in Proposition 1 to be equivalent to inner product similarity between real valued vectors used in the original transformer #REFR .

In this work, we tackle a new problem of tracking 3D human poses from only events using SNNs. Specifically, our approach, shown in Fig.

2 , is an end-to-end model that estimates parametric human poses over time solely from an event stream, without any requirement of dense images.

This model is entirely built upon SNNs, providing significantly greater efficiency than ANNs.

After preprocessing the input event stream to be a sequence of event voxel grids with the same time interval, Spike-Element-Wise (SEW) ResNet #OTHEREFR is employed as the SNNs backbone to extract pose spike features.

This is followed by our proposed spiking spatiotemporal transformer for the bi-directional fusion of pose spike features, allowing for the compensation of missing pose information, especially for early time steps.

The final step is to apply 2D average pooling to the spatiotemporally aggregated spike features and then directly regress the parametric human poses over time.

The experiments show that our proposed SNNs achieve competitive or even better performance of pose tracking than several ANNs (i.e.

ResNet-GRU #OTHEREFR , ResNet-TF #OTHEREFR and Video-Swin #OTHEREFR ), but requires only about 20% of FLOPs, which shows promising prospects of efficient SNNs on various event-based vision tasks.

Additionally, we provide a large-scale synthetic dataset for event-based 3D human pose tracking.

Although an inhouse event-based motion capture dataset, MMHPSD, is introduced in #OTHEREFR , its limited variety of motions restricts the generalization ability of trained models.

Event-Based Human Pose Tracking By Spiking Spatiotemporal Transformer

Abstract: Event camera, as an emerging biologically-inspired vision sensor for capturing motion dynamics, presents new potential for 3D human pose tracking, or video-based 3D human pose estimation. However, existing works in pose tracking either require the presence of additional gray-scale images to establish a solid starting pose, or ignore the temporal dependencies all together by collapsing segments of event streams to form static image frames. Meanwhile, although the effectiveness of Artificial Neural Networks (ANNs, a.k.a. dense deep learning) has been showcased in many event-based tasks, the use of ANNs tends to neglect the fact that compared to the dense frame-based image sequences, the occurrence of events from an event camera is spatiotemporally much sparser. Motivated by the above mentioned issues, we present in this paper a dedicated end-to-end sparse deep learning approach for event-based pose tracking: 1) to our knowledge this is the first time that 3D human pose tracking is obtained from events only, thus eliminating the need of accessing to any frame-based images as part of input; 2) our approach is based entirely upon the framework of Spiking Neural Networks (SNNs), which consists of Spike-Element-Wise (SEW) ResNet and our proposed spiking spatiotemporal transformer; 3) a large-scale synthetic dataset is constructed that features a broad and diverse set of annotated 3D human motions, as well as longer hours of event stream data, named SynEventHPD. Empirical experiments demonstrate the superiority of our approach in both performance and efficiency measures. For example, with comparable performance to the state-of-the-art ANNs counterparts, our approach achieves a computation reduction of 20% in FLOPS. Our implementation is made available at https://github.com/JimmyZou/HumanPoseTracking SNN and dataset will be released upon paper acceptance.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


The vast majority of AI studies consider accuracy as the pertinent performance measure, ignoring the impact ML has on deployed devices and the environment #REFR .

This is ideal in the cases model training simply cannot wait.

Based on these estimates, the same CNN model will be trained in Sweden with a carbon footprint of just 0.21-0.24kg, no matter the time throughout the day.

This footprint is equivalent to a 95% reduction to ML training, even, midday in Cyprus.

Observation: The ML model training process can have a significantly different environmental footprint over different locations.

Therefore, organisations wanting to reduce the environmental impact of their model training should explore potential advantages of moving their workloads to low-carbon grid energy.

Towards this, let us consider three state-of-the-art model architectures for ML classification that can be used for object detection.

Specifically, we will consider ResNet50 (used in Sections III-B and III-C), SqueezeNet and MobileNetV2.

We now focus on ML inference and its impact to the reference edge micro-DC.

For this, we use binary builds from the TensorFlow model zoo for the adopted architectures trained using the reference dataset #OTHEREFR . Table I depicts the results.

We immediately observe the high accuracy of ResNet50, but through the additional metrics we can also label it as both energy-hungry and latent in terms of inference.

Towards Energy Consumption And Carbon Footprint Testing For Ai-Driven Iot Services

Abstract: Energy consumption and carbon emissions are expected to be crucial factors for Internet of Things (IoT) applications. Both the scale and the geo-distribution keep increasing, while Artificial Intelligence (AI) further penetrates the "edge" in order to satisfy the need for highly-responsive and intelligent services. To date, several edge/fog emulators are catering for IoT testing by supporting the deployment and execution of AI-driven IoT services in consolidated test environments. These tools enable the configuration of infrastructures so that they closely resemble edge devices and IoT networks. However, energy consumption and carbon emissions estimations during the testing of AI services are still missing from the current state of IoT testing suites. This study highlights important questions that developers of AIdriven IoT services are in need of answers, along with a set of observations and challenges, aiming to help researchers designing IoT testing and benchmarking suites to cater to user needs.

Green Ai

Abstract: The computations required for deep learning research have been doubling every few months, resulting in an estimated 300,000x increase from 2012 to 2018 [2]. These computations have a surprisingly large carbon footprint [38]. Ironically, deep learning was inspired by the human brain, which is remarkably energy efficient. Moreover, the financial cost of the computations can make it difficult for academics, students, and researchers, in particular those from emerging economies, to engage in deep learning research. This position paper advocates a practical solution by making efficiency an evaluation criterion for research alongside accuracy and related measures. In addition, we propose reporting the financial cost or "price tag" of developing, training, and running models to provide baselines for the investigation of increasingly efficient methods. Our goal is to make AI both greener and more inclusive---enabling any inspired undergraduate with a laptop to write high-quality research papers. Green AI is an emerging focus at the Allen Institute for AI.


A function's curvature is often expressed as the norm of the Hessian at a particular point #REFR .

Analysis of max-pooling from the perspective of curvature and Hessian norms.

Here, we take a step further and investigate why maxpooling impacts the robustness of models.

In order to perform this analysis, we analyze gradient norms, Hessian norms, and the model's curvature.

The curvature of a point is a mathematical quantity that indicates the degree of nonlinearity.

Mainly, robust models are characterized by low gradient norms #OTHEREFR , implying smaller Hessian norms.

In order to investigate robustness independent of non-linearity, (Srinivas et al.) propose normalized curvature, which normalizes the Hessian norm by its corresponding gradient norm.

They defined normalized curvature for a neural network classifier f as C f (x) = ∇ 2 f (x) 2 /( ∇f (x) 2 + ε).

Where ∇f (x) 2 and ∇ 2 f (x) 2 are the 2 -norm of the gradient and the spectral norm of the Hessian, respectively, where ∇f (x) ∈ R d , ∇ 2 f (x) ∈ R d×d , and ε > 0 is a small constant to ensure the proper behavior of the measure. We use this metric to conduct our experiments.

Revisiting Deepfool: Generalization And Improvement

Abstract: Deep neural networks have been known to be vulnerable to adversarial examples, which are inputs that are modified slightly to fool the network into making incorrect predictions. This has led to a significant amount of research on evaluating the robustness of these networks against such perturbations. One particularly important robustness metric is the robustness to minimal 2 adversarial perturbations. However, existing methods for evaluating this robustness metric are either computationally expensive or not very accurate. In this paper, we introduce a new family of adversarial attacks that strike a balance between effectiveness and computational efficiency. Our proposed attacks are generalizations of the well-known Deep-Fool (DF) attack, while they remain simple to understand and implement. We demonstrate that our attacks outperform existing methods in terms of both effectiveness and computational efficiency. Our proposed attacks are also suitable for evaluating the robustness of large models and can be used to perform adversarial training (AT) to achieve state-of-the-art robustness to minimal 2 adversarial perturbations 1 .

Robustness Via Curvature Regularization, And Vice Versa

Abstract: State-of-the-art classifiers have been shown to be largely vulnerable to adversarial perturbations. One of the most effective strategies to improve robustness is adversarial training. In this paper, we investigate the effect of adversarial training on the geometry of the classification landscape and decision boundaries. We show in particular that adversarial training leads to a significant decrease in the curvature of the loss surface with respect to inputs, leading to a drastically more "linear" behaviour of the network. Using a locally quadratic approximation, we provide theoretical evidence on the existence of a strong relation between large robustness and small curvature. To further show the importance of reduced curvature for improving the robustness, we propose a new regularizer that directly minimizes curvature of the loss surface, and leads to adversarial robustness that is on par with adversarial training. Besides being a more efficient and principled alternative to adversarial training, the proposed regularizer confirms our claims on the importance of exhibiting quasi-linear behavior in the vicinity of data points in order to achieve robustness.


To address it, IoU-Net #REFR proposes an individual IoU prediction branch and an IoU-guided NMS to align the classification confidence and regression precision.

1a shows an empirical study on a strong baseline DINO #OTHEREFR , and the recall of BR samples by the well-trained model on the top-k confident outputs in an image is calculated, where a higher recall indicates that more BR samples are selected in final prediction.

As we can see, only 45% and 48% BR samples are covered by HC samples with top-N and top-2N scores, respectively, suggesting that more than half of the well-localized predictions have low confidence scores. In Fig.

1b , the frequency histogram of the HC and BR samples is plotted from 5,000 samples on COCO val.

and a clear discrepancy between the two distributions is observed, revealing that even the highly-optimized model DINO #OTHEREFR suffers from the misalignment problem.

In fact, this problem also appears in CNN-based detectors #OTHEREFR .

A number of alternatives #OTHEREFR introduce an IoU-aware loss or weight to integrate the IoU branch into the original classification branch and adopt a joint training strategy.

These methods are specially designed for NMS-based detectors, but DETR implicitly selects samples by modeling query relations under the guidance of one-to-one matching without an explicit query selection step like NMS, making them less applicable to DETR #OTHEREFR .

To the best of our knowledge, the misalignment problem still remains unexplored in the DETR series.

This paper investigates the solution to the misalignment problem in DETR.

To this end, we propose a novel method, namely Align-DETR.

Align-Detr: Improving Detr With Simple Iou-Aware Bce Loss

Abstract: DETR has set up a simple end-to-end pipeline for object detection by formulating this task as a set prediction problem, showing promising potential. However, despite the significant progress in improving DETR, this paper identifies a problem of misalignment in the output distribution, which prevents the best-regressed samples from being assigned with high confidence, hindering the model's accuracy. We propose a metric, recall of best-regressed samples, to quantitively evaluate the misalignment problem. Observing its importance, we propose a novel Align-DETR that incorporates a localization precision aware classification loss in optimization. The proposed loss, IA-BCE, guides the training of DETR to build a strong correlation between classification score and localization precision. We also adopt the mixed-matching strategy, to facilitate DETR-based detectors with faster training convergence while keeping an end-to-end scheme. Moreover, to overcome the dramatic decrease in sample quality induced by the sparsity of queries, we introduce a prime sample weighting mechanism to suppress the interference of unimportant samples. Extensive experiments are conducted with very competitive results reported. In particular, it delivers a 46 (+3.8)% AP on the DAB-DETR baseline with the ResNet-50 backbone and reaches a new SOTA performance of 50.2% AP in the 1x setting on the COCO validation set when employing the strong baseline DINO. Our code is available at https://github.com/FelixCaae/AlignDETR.

Acquisition Of Localization Confidence For Accurate Object Detection

Abstract: Modern CNN-based object detectors rely on bounding box regression and non-maximum suppression to localize objects. While the probabilities for class labels naturally reflect classification confidence, localization confidence is absent. This makes properly localized bounding boxes degenerate during iterative regression or even suppressed during NMS. In the paper we propose IoU-Net learning to predict the IoU between each detected bounding box and the matched ground-truth. The network acquires this confidence of localization, which improves the NMS procedure by preserving accurately localized bounding boxes. Furthermore, an optimization-based bounding box refinement method is proposed, where the predicted IoU is formulated as the objective. Extensive experiments on the MS-COCO dataset show the effectiveness of IoU-Net, as well as its compatibility with and adaptivity to several state-of-the-art object detectors.


From the perspective of machine learning, early representations through the lower layer play an important role to capture the pos-terior distribution of the underlying explanatory factors for the observed input #REFR .

We recognize that most of the domain variations, such as changes in the visual scenes and image transformations or corruptions, are early layers of features in the semantic hierarchy #OTHEREFR .

They can be effectively captured and modeled by lower layers of the network model.

For instance, in deep neural network models, the early layers of the network tend to respond to corners, edges, or colors.

In contrast, deeper layers respond to more class-specific features #OTHEREFR .

In the corruption testtime adaptation scenario, the class-specific features are always the same because the testing datasets are the corruption of the training domain.

However, the early layers of models can be failed due to corruption.

Therefore, the central challenge in fully test-time UDA lies in how to learn useful early layer representations of the test samples without supervision.

Neuro-Modulated Hebbian Learning For Fully Test-Time Adaptation

Abstract: Fully test-time adaptation aims to adapt the network model based on sequential analysis of input samples during the inference stage to address the cross-domain performance degradation problem of deep neural networks. We take inspiration from the biological plausibility learning where the neuron responses are tuned based on a local synapse-change procedure and activated by competitive lateral inhibition rules. Based on these feed-forward learning rules, we design a soft Hebbian learning process which provides an unsupervised and effective mechanism for online adaptation. We observe that the performance of this feed-forward Hebbian learning for fully test-time adaptation can be significantly improved by incorporating a feedback neuro-modulation layer. It is able to fine-tune the neuron responses based on the external feedback generated by the error back-propagation from the top inference layers. This leads to our proposed neuro-modulated Hebbian learning (NHL) method for fully test-time adaptation. With the unsupervised feed-forward soft Hebbian learning being combined with a learned neuro-modulator to capture feedback from external responses, the source model can be effectively adapted during the testing process. Experimental results on benchmark datasets demonstrate that our proposed method can significantly improve the adaptation performance of network models and outperforms existing stateof-the-art methods.

Representation Learning: A Review And New Perspectives

Abstract: The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks. This motivates longer term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation, and manifold learning.


Even though it is proved in #REFR that motion features are useless for audio-visual event localization, we argue that our experiment clearly demonstrates their significance for dense-localizing audio-visual events.

In addition, we found that the appropriate number of uni-modal blocks is also impor- tant, which reveals that applying self-attention before crossmodal interaction can help the model to focus on informative signals and eliminate noise from each modality. Dependency Modeling and Class-Aware Regression. As shown in Tab.

4, applying temporal dependency modeling and class-aware regression separately can both achieve higher performances than the base model that just contains our transformer encoder with a class-agnostic regression head in the decoder.

Besides, we found that when using both of them, they can promote each other and achieve a further significant performance boost, which clearly demonstrates their effectiveness. The Impact of Motion Features. In Tab.

5, we observe that utilizing both RGB and optical flow features extracted by I3D #OTHEREFR achieves the best performance.

It outperforms the model that uses visual features extracted by ResNet50 #OTHEREFR pre-trained on ImageNet by a large margin (+3.5% at the average mAP).

The Capability of Localizing Concurrent Events.

We further evaluate our models and the state-of-the-art TAL method #OTHEREFR on the videos that contain concurrent events with different overlap rates in Fig. 6 .

We observe that our model equipped with dependency modeling and class-aware regression obviously gains more performance improvement on the videos with higher overlap rates, compared with our baseline and ActionFormer #OTHEREFR .

It suggests that our model has a better ability to localize overlapping audiovisual events in untrimmed videos. Qualitative Results. In Fig.

5 , we present the qualitative results of our model variants that utilize different modalities as input.

Dense-Localizing Audio-Visual Events In Untrimmed Videos: A Large-Scale Benchmark And Baseline

Abstract: Existing audiovisual event localization (AVE) handles manually trimmed videos with only a single instance in each of them. However, this setting is unrealistic as natural videos often contain numerous audiovisual events with different categories. To better adapt to real-life applications, in this paper we focus on the task of denselocalizing audiovisual events, which aims to jointly localize and recognize all audiovisual events occurring in an untrimmed video. The problem is challenging as it requires fine-grained audiovisual scene and context understanding. To tackle this problem, we introduce the first Untrimmed AudioVisual (UnAV-100) dataset, which contains 10K untrimmed videos with over 30K audiovisual events. Each video has 2.8 audiovisual events on average, and the events are usually related to each other and might co-occur as in real-life scenes. Next, we formulate the task using a new learning-based framework, which is capable of fully integrating audio and visual modalities to localize audiovisual events with various lengths and capture dependencies between them in a single pass. Extensive experiments demonstrate the effectiveness of our method as well as the significance of multi-scale cross-modal perception and dependency modeling for this task.

Audio-Visual Event Localization In Unconstrained Videos

Abstract: In this paper, we introduce a novel problem of audio-visual event localization in unconstrained videos. We define an audio-visual event as an event that is both visible and audible in a video segment. We collect an Audio-Visual Event (AVE) dataset to systemically investigate three temporal localization tasks: supervised and weakly-supervised audio-visual event localization, and cross-modality localization. We develop an audio-guided visual attention mechanism to explore audio-visual correlations, propose a dual multimodal residual network (DMRN) to fuse information over the two modalities, and introduce an audio-visual distance learning network to handle the cross-modality localization. Our experiments support the following findings: joint modeling of auditory and visual modalities outperforms independent modeling, the learned attention can capture semantics of sounding objects, temporal alignment is important for audio-visual fusion, the proposed DMRN is effective in fusing audio-visual features, and strong correlations between the two modalities enable cross-modality localization.

dense-localizing audio-visual events
audio-visual event localization


In our ablation studies, we observe that CHATTY, when combined with MCC #REFR gives the best result on most of the domain adaptation tasks.

The sensitivity is reduced by replacing L T L by L T L Cos which normalizes every entry in the matrix Y by the product of the norms.

Empirically, the best adaptation algorithm was seen to be the loss given by Equation 18.

When the loss was modified to Equation 20 it was observed that the convergence was slower than when the class confusion loss was included separately in the total loss.

This could be possibly attributed to the back-propagation updates of M being very slow, owing to the camouflaged nature of M in T 1 MT 2 T .

Nevertheless, it is interesting to study an optimal choice of M.

The evolution of the target classifier accuracy on the domain adaptation from Ar to Cl is plotted in Figure 4 .

We observe that by shifting the samples in the classification (logit) space itself produces a significantly improved accuracy compared to SOTA methods, as shown in Table 1 , and tuning the transport vectors increases the accuracy.

Furthermore, using the MCC loss in synergy with the transport loss favoured accuracy even more.

When using just the transport loss as a means of reducing class confusion, we observe that it does a better job than the MCC loss.

Chatty: Coupled Holistic Adversarial Transport Terms With Yield For Unsupervised Domain Adaptation

Abstract: We propose a new technique called CHATTY: Coupled Holistic Adversarial Transport Terms with Yield for Unsupervised Domain Adaptation. Adversarial training is commonly used for learning domain-invariant representations by reversing the gradients from a domain discriminator head to train the feature extractor layers of a neural network. We propose significant modifications to the adversarial head, its training objective, and the classifier head. With the aim of reducing class confusion, we introduce a sub-network which displaces the classifier outputs of the source and target domain samples in a learnable manner. We control this movement using a novel transport loss that spreads class clusters away from each other and makes it easier for the classifier to find the decision boundaries for both the source and target domains. The results of adding this new loss to a careful selection of previously proposed losses leads to improvement in UDA results compared to the previous state-of-the-art methods on benchmark datasets. We show the importance of the proposed loss term using ablation studies and visualization of the movement of target domain sample in representation space.

Minimum Class Confusion For Versatile Domain Adaptation

Abstract: There are a variety of DA scenarios subject to label sets and domain configurations, including closed-set and partial-set DA, as well as multi-source and multi-target DA. It is notable that existing DA methods are generally designed only for a specific scenario, and may underperform for scenarios they are not tailored to. A versatile method, which can handle several different scenarios without any extra modifications, is still remained to be explored. Towards such purpose, a more general inductive bias other than the domain alignment should be explored. In this paper, we delve into a missing piece of existing methods: class confusion, the tendency that a classifier confuses the predictions between the correct and ambiguous classes for target examples, which exists in all of the scenarios above. We unveil that reducing such pair-wise class confusion brings about significant transfer gains. Based on this, we propose a general loss function: Minimum Class Confusion (MCC). It can be characterized by (1) a non-adversarial DA method without explicitly deploying domain alignment, enjoying fast convergence speed (about 3 times faster than mainstream adversarial methods); (2) a versatile approach that can handle the four existing scenarios: Closed-Set, Partial-Set, Multi-Source, and Multi-Target DA, outperforming the state-of-the-art methods in these scenarios, especially on the largest and hardest dataset to date (7.25% on DomainNet). Strong performance in the two scenarios proposed in this paper: Multi-Source Partial and Multi-Target Partial DA, further proves its versatility. In addition, it can also be used as a general regularizer that is orthogonal and complementary to a variety of existing DA methods, accelerating convergence and pushing those readily competitive methods to a stronger level.


MixMatch #REFR combines the consistency regularization with the entropy minimization to encourage the network to output confident predictions for unlabeled data.

In view of this, we focus on learning from scarce annotations to minimize human labor and achieve accurate animal pose estimation at the same time.

Semi-supervised learning is powerful in leveraging unlabeled data to improve a model's performance when labeled data is limited.

One of the most widely used SSL techniques is pseudo labeling #OTHEREFR , which generates artificial labels for unlabeled images from model predictions.

Another technique is consistency regularization #OTHEREFR , which enforces that the model output should be consistent when the input is randomly perturbed.

More recent Unsupervised Data Augmentation (UDA) #OTHEREFR and FixMatch #OTHEREFR combine pseudo labeling, consistency regularization with strong augmentations #OTHEREFR and achieve superior performance.

FlexMatch #OTHEREFR further improves over UDA and Fixmatch by introducing a curriculum learning scheme.

We also adopt the PL-based techniques in this work given its effectiveness.

To mitigate the negative effect of noisy pseudo labels, we combine PL with reliable pseudo label selection and reusable sample re-labeling.

Scarcenet: Animal Pose Estimation With Scarce Annotations

Abstract: Animal pose estimation is an important but underexplored task due to the lack of labeled data. In this paper, we tackle the task of animal pose estimation with scarce annotations, where only a small set of labeled data and unlabeled images are available. At the core of the solution to this problem setting is the use of the unlabeled data to compensate for the lack of well-labeled animal pose data. To this end, we propose the ScarceNet, a pseudo label-based approach to generate artificial labels for the unlabeled images. The pseudo labels, which are generated with a model trained with the small set of labeled images, are generally noisy and can hurt the performance when directly used for training. To solve this problem, we first use a small-loss trick to select reliable pseudo labels. Although effective, the selection process is improvident since numerous highloss samples are left unused. We further propose to identify reusable samples from the high-loss samples based on an agreement check. Pseudo labels are regenerated to provide supervision for those reusable samples. Lastly, we introduce a student-teacher framework to enforce a consistency constraint since there are still samples that are neither reliable nor reusable. By combining the reliable pseudo label selection with the reusable sample re-labeling and the consistency constraint, we can make full use of the unlabeled data. We evaluate our approach on the challenging AP-10K dataset, where our approach outperforms existing semi-supervised approaches by a large margin. We also test on the TigDog dataset, where our approach can achieve better performance than domain adaptation based approaches when only very few annotations are available. Our code is available at the project website 1 .

Mixmatch: A Holistic Approach To Semi-Supervised Learning

Abstract: Semi-supervised learning has proven to be a powerful paradigm for leveraging unlabeled data to mitigate the reliance on large labeled datasets. In this work, we unify the current dominant approaches for semi-supervised learning to produce a new algorithm, MixMatch, that works by guessing low-entropy labels for data-augmented unlabeled examples and mixing labeled and unlabeled data using MixUp. We show that MixMatch obtains state-of-the-art results by a large margin across many datasets and labeled data amounts. For example, on CIFAR-10 with 250 labels, we reduce error rate by a factor of 4 (from 38% to 11%) and by a factor of 2 on STL-10. We also demonstrate how MixMatch can help achieve a dramatically better accuracy-privacy trade-off for differential privacy. Finally, we perform an ablation study to tease apart which components of MixMatch are most important for its success.


Since our method can adapt faster to the downstream tasks, we train our model for 80K iterations using a Semantic FPN #REFR by default.

To fully preserve the pre-trained knowledge of the θ , we always set the learning rate of θ as 1/10 of the base learning rate. We use γ=1e-4 for the text adapter.

The task-specific settings and training details are elaborated as follows.

The goal of semantic segmentation is to assign pixel-level labels to a given image, which requires a fine-grained high-level understanding of visual content.

We evaluate our method on ADE20K #OTHEREFR , which consists of 20K images for training and 2K images for validation.

We use a global batch size of 16 and set the learning rate as 1e-4.

We use the AdamW optimizer with a weight decay of 1e-4 and warming-up iterations of 1500.

We adopt the polynomial learning rate scheduler with a power of 0.9 and a minimum learning rate of 1e-6.

For the fast schedule (8K iterations), we linear scale the learning rate schedule and set the warming-up iterations to 150.

During inference, we use the slide inference with a crop size 512 × 512 and a stride of 341 × 341.

Unleashing Text-To-Image Diffusion Models For Visual Perception

Abstract: Diffusion models (DMs) have become the new trend of generative models and have demonstrated a powerful ability of conditional synthesis. Among those, text-to-image diffusion models pre-trained on large-scale image-text pairs are highly controllable by customizable prompts. Unlike the unconditional generative models that focus on low-level attributes and details, text-to-image diffusion models contain more high-level knowledge thanks to the vision-language pre-training. In this paper, we propose VPD (Visual Perception with a pre-trained Diffusion model), a new framework that exploits the semantic information of a pre-trained text-to-image diffusion model in visual perception tasks. Instead of using the pre-trained denoising autoencoder in a diffusion-based pipeline, we simply use it as a backbone and aim to study how to take full advantage of the learned knowledge. Specifically, we prompt the denoising decoder with proper textual inputs and refine the text features with an adapter, leading to a better alignment to the pre-trained stage and making the visual contents interact with the text prompts. We also propose to utilize the cross-attention maps between the visual features and the text features to provide explicit guidance. Compared with other pre-training methods, we show that vision-language pre-trained diffusion models can be faster adapted to downstream visual perception tasks using the proposed VPD. Extensive experiments on semantic segmentation, referring image segmentation and depth estimation demonstrates the effectiveness of our method. Notably, VPD attains 0.254 RMSE on NYUv2 depth estimation and 73.3% oIoU on RefCOCOval referring image segmentation, establishing new records on these two benchmarks. Code is available at https: //github.com/wl-zhao/VPD.

Panoptic Feature Pyramid Networks

Abstract: The recently introduced panoptic segmentation task has renewed our community's interest in unifying the tasks of instance segmentation (for thing classes) and semantic segmentation (for stuff classes). However, current state-of-the-art methods for this joint task use separate and dissimilar networks for instance and semantic segmentation, without performing any shared computation. In this work, we aim to unify these methods at the architectural level, designing a single network for both tasks. Our approach is to endow Mask R-CNN, a popular instance segmentation method, with a semantic segmentation branch using a shared Feature Pyramid Network (FPN) backbone. Surprisingly, this simple baseline not only remains effective for instance segmentation, but also yields a lightweight, top-performing method for semantic segmentation. In this work, we perform a detailed study of this minimally extended version of Mask R-CNN with FPN, which we refer to as Panoptic FPN, and show it is a robust and accurate baseline for both tasks. Given its effectiveness and conceptual simplicity, we hope our method can serve as a strong baseline and aid future research in panoptic segmentation.


However, the nonlinearity of the fully connected layers may cause gradient saturation or undesirable artifacts during backpropagation #REFR .

In this way, Grad-CAM can derive the saliency map without needing to modify and retrain the CNN models.

Methods that have followed up Grad-CAM include Grad-CAM++ #OTHEREFR

2018) for the localization of multi-objects belonging to the same class and Score-CAM #OTHEREFR

2020) , which obtains weights for corresponding feature maps without gradient calculation.

These gradient-based methods require only one forward-backward pass and are computationally efficient compared to the perturbation-based methods.

Recent works have developed new strategies to address this issue. For example, Sundararajan et al.

2017developed an integrated gradient method to aggregate the gradients over an entire image as it goes through continuous modifications, to avoid gradient saturation and unexpected artifacts.

The entire image is gradually altered from the original image to an all-black image (i.e., baseline).

The result is an aggregation of all the intermediate results after each alternation.

This method can be considered as a combination of perturbationbased and gradient-based methods and has shown promising results in many tasks #OTHEREFR .

Explainable Geoai: Can Saliency Maps Help Interpret Artificial Intelligence'S Learning Process? An Empirical Study On Natural Feature Detection

Abstract: Improving the interpretability of geospatial artificial intelligence (GeoAI) models has become critically important to open the "black box" of complex AI models, such as deep learning. This paper compares popular saliency map generation techniques and their strengths and weaknesses in interpreting GeoAI and deep learning models' reasoning behaviors, particularly when applied to geospatial analysis and image processing tasks. We surveyed two broad classes of model explanation methods: perturbation-based and gradient-based methods. The former identifies important image areas, which help machines make predictions by modifying a localized area of the input image. The latter evaluates the contribution of every single pixel of the input image to the model's prediction results through gradient backpropagation. In this study, three algorithms-the occlusion method, the integrated gradients method, and the class activation map method-are examined for a natural feature detection task using deep learning. The algorithms' strengths and weaknesses are discussed, and the consistency between model-learned and human-understandable concepts for object recognition is also compared. The experiments used two GeoAIready datasets to demonstrate the generalizability of the research findings.

Learning Important Features Through Propagating Activation Differences

Abstract: The purported "black box" nature of neural networks is a barrier to adoption in applications where interpretability is essential. Here we present DeepLIFT (Deep Learning Important FeaTures), a method for decomposing the output prediction of a neural network on a specific input by backpropagating the contributions of all neurons in the network to every feature of the input. DeepLIFT compares the activation of each neuron to its 'reference activation' and assigns contribution scores according to the difference. By optionally giving separate consideration to positive and negative contributions, DeepLIFT can also reveal dependencies which are missed by other approaches. Scores can be computed efficiently in a single backward pass. We apply DeepLIFT to models trained on MNIST and simulated genomic data, and show significant advantages over gradient-based methods. Video tutorial: http://goo.gl/qKb7pL, code: http://goo.gl/RM8jvH.


We follow the approach proposed in #REFR , which concatenates all the descriptions of a video to a paragraph.

The second one is from #OTHEREFR with 9000 clips for training and 1000 clips for testing.

The last one is the partitions from #OTHEREFR , which uses 9000 clips for training and 1000 clips for testing.

To increase the persuasiveness of the experiment, we conducted experiments on all data partitions.

ActivityNet #OTHEREFR : It is an increasingly popular dataset consisting of densely annotated temporal segments of 20000 YouTube videos.

Each video is an average of 2 minutes long, and there are 72000 video-text pairs in this dataset.

We trained the model with 10009 videos and evaluated our model on the "val1" split, which contains 4917 videos.

LSMDC #OTHEREFR : It contains 118,081 short video clips( 45 s) extracted from 202 movies.

Unlike other datasets, each video in LSMDC only has one description, either extracted from the movie script or the transcribed audio description.

The test set consisted of 1000 videos that were not present in the training set.

Improving Video Retrieval By Adaptive Margin

Abstract: Video retrieval is becoming increasingly important owing to the rapid emergence of videos on the Internet. The dominant paradigm for video retrieval learns video-text representations by pushing the distance between the similarity of positive pairs and that of negative pairs apart from a fixed margin. However, negative pairs used for training are sampled randomly, which indicates that the semantics between negative pairs may be related or even equivalent, while most methods still enforce dissimilar representations to decrease their similarity. This phenomenon leads to inaccurate supervision and poor performance in learning video-text representations. While most video retrieval methods overlook that phenomenon, we propose an adaptive margin changed with the distance between positive and negative pairs to solve the aforementioned issue. First, we design the calculation framework of the adaptive margin, including the method of distance measurement and the function between the distance and the margin. Then, we explore a novel implementation called "Cross-Modal Generalized Self-Distillation" (CMGSD), which can be built on the top of most video retrieval models with few modifications. Notably, CMGSD adds few computational overheads at train time and adds no computational overhead at test time. Experimental results on three widely used datasets demonstrate that the proposed method can yield significantly better performance than the corresponding backbone model, and it outperforms state-of-the-art methods by a large margin. CCS CONCEPTS • Information systems → Video search.

Cross-Modal And Hierarchical Modeling Of Video And Text

Abstract: Visual data and text data are composed of information at multiple granularities. A video can describe a complex scene that is composed of multiple clips or shots, where each depicts a semantically coherent event or action. Similarly, a paragraph may contain sentences with different topics, which collectively conveys a coherent message or story. In this paper, we investigate the modeling techniques for such hierarchical sequential data where there are correspondences across multiple modalities. Specifically, we introduce hierarchical sequence embedding (HSE), a generic model for embedding sequential data of different modalities into hierarchically semantic spaces, with either explicit or implicit correspondence information. We perform empirical studies on large-scale video and paragraph retrieval datasets and demonstrated superior performance by the proposed methods. Furthermore, we examine the effectiveness of our learned embeddings when applied to downstream tasks. We show its utility in zero-shot action recognition and video captioning.


In contrast, with 512 or more basis vectors the relative error in the Hamiltonian is about 10 −6 for the standard cSVD which is in agreement with the experiments from #REFR for the linear wave equation.

The temporary increase in the error curves for m s ≥ 120 and an average basis size of 70 is due to the fact that the basis is too small for the window size m s ≥ 120. Larger window sizes require larger basis sizes.

In Figures 3 and 4 we present the relative error in the Hamiltonian

averaged over the three training parameters in dependence on the time-step.

In Figure 3 we see that using a non-symplectic, standard POD-basis leads to unstable models in the sense that the energy drastically increases if more than 256 basis vectors are used.

As expected, the error in the Hamiltonian is constant for the standard cSVD up to numerical inaccuracies.

These inaccuracies can only be seen in the bottom two curves due to the logarithmic axis.

In Figure 4 , we plot the same errors for the dictionary-based methods.

With n s = 100 selected snapshots per basis update, we observe high error jumps across the basis changes for DB-cSVD for window sizes m s ≥ 120.

However, with n s = 250 snapshots selected, this behavior is no longer observed and a relative error of 10 −6 is achieved as with the standard cSVD.

Dictionary-Based Online-Adaptive Structure-Preserving Model Order Reduction For Parametric Hamiltonian Systems

Abstract: Classical model order reduction (MOR) for parametric problems may become computationally inefficient due to large sizes of the required projection bases, especially for problems with slowly decaying Kolmogorov n-widths. Additionally, Hamiltonian structure of dynamical systems may be available and should be preserved during the reduction. In the current presentation, we address these two aspects by proposing a corresponding dictionary-based, online-adaptive MOR approach. The method requires dictionaries for the state-variable, non-linearities and discrete empirical interpolation (DEIM) points. During the online simulation, local basis extensions/simplifications are performed in an online-efficient way, i.e. the runtime complexity of basis modifications and online simulation of the reduced models do not depend on the full state dimension. Experiments on a linear wave equation and a non-linear Sine-Gordon example demonstrate the efficiency of the approach.

Symplectic Model Reduction Of Hamiltonian Systems

Abstract: In this paper, a symplectic model reduction technique, proper symplectic decomposition (PSD) with symplectic Galerkin projection, is proposed to save the computational cost for the simplification of large-scale Hamiltonian systems while preserving the symplectic structure. As an analogy to the classical proper orthogonal decomposition (POD)-Galerkin approach, PSD is designed to build a symplectic subspace to fit empirical data, while the symplectic Galerkin projection constructs a reduced Hamiltonian system on the symplectic subspace. For practical use, we introduce three algorithms for PSD, which are based upon: the cotangent lift, complex singular value decomposition, and nonlinear programming. The proposed technique has been proven to preserve system energy and stability. Moreover, PSD can be combined with the discrete empirical interpolation method to reduce the computational cost for nonlinear Hamiltonian systems. Owing to these properties, the proposed technique is better suited than the classical POD-Galerkin approach for model reduction of Hamiltonian systems, especially when long-time integration is required. The stability, accuracy, and efficiency of the proposed technique are illustrated through numerical simulations of linear and nonlinear wave equations.


Notably, the utility achieved on these tasks is comparable to the values reported in prior work that use these datasets #REFR , demonstrating effectiveness of METAM to achieve competitive quality while generalizing to a wide variety of tasks.

The goal is to predict number of collisions in NYC using data such as number of daily taxi trips #OTHEREFR over a dataset containing 350 records..

The task uses a random forest regressor and computes the mean absolute error (MAE), returning 1 − M AE as utility. METAM outperforms all baselines.

With only 300 queries, METAM reduces MAE from 0.66 to 0.55.

Other techniques require three times more queries to achieve similar MAE.

We further evaluate predictive analytics (classification and regression) tasks with informative domainspecific data profiles such as feature importance #OTHEREFR and uninformative generic data profiles to understand METAM's flexibility in Section VI-C.

The task takes an input dataset along with a hypothetical update, and outputs the causal impact of the update query on other attributes.

We consider an initial table containing SAT scores of 450 students [43] and ask what attributes would be causally affected if "critical reading score" of students is updated.

Understanding the attributes sheds light on what affects students' reading score, paving the way for the implementation of interventions.

Metam: Goal-Oriented Data Discovery

Abstract: Data is a central component of machine learning and causal inference tasks. The availability of large amounts of data from sources such as open data repositories, data lakes and data marketplaces creates an opportunity to augment data and boost those tasks' performance. However, augmentation techniques rely on a user manually discovering and shortlisting useful candidate augmentations. Existing solutions do not leverage the synergy between discovery and augmentation, thus underexploiting data. In this paper, we introduce METAM, a novel goal-oriented framework that queries the downstream task with a candidate dataset, forming a feedback loop that automatically steers the discovery and augmentation process. To select candidates efficiently, METAM leverages properties of the: i) data, ii) utility function, and iii) solution set size. We show METAM's theoretical guarantees and demonstrate those empirically on a broad set of tasks. All in all, we demonstrate the promise of goal-oriented data discovery to modern data science applications.

Arda: Automatic Relational Data Augmentation For Machine Learning

Abstract: Automatic machine learning (\AML) is a family of techniques to automate the process of training predictive models, aiming to both improve performance and make machine learning more accessible. While many recent works have focused on aspects of the machine learning pipeline like model selection, hyperparameter tuning, and feature selection, relatively few works have focused on automatic data augmentation. Automatic data augmentation involves finding new features relevant to the user's predictive task with minimal ``human-in-the-loop'' involvement. We present \system, an end-to-end system that takes as input a dataset and a data repository, and outputs an augmented data set such that training a predictive model on this augmented dataset results in improved performance. Our system has two distinct components: (1) a framework to search and join data with the input data, based on various attributes of the input, and (2) an efficient feature selection algorithm that prunes out noisy or irrelevant features from the resulting join. We perform an extensive empirical evaluation of different system components and benchmark our feature selection algorithm on real-world datasets.