Among the neural baselines, the transformer-based cross-encoder outperforms the bi-encoder, which is consistent with previous findings #REFR .

Our main results are summarized in Table 2 .

All neural re-ranking models outperform the baseline Expert System in both metrics, suggesting that re-ranking does up-rank the more relevant historytaking questions.

Surprisingly, the LongT5 autoregressive re-ranker, despite having more parameters (220M parameters), also performs worse than the crossencoder (∼110M parameters).

The best performance is achieved by the global re-ranker for both transformer and S4 architectures, regardless of the loss functions chosen.

Among the various loss functions, the pointwise binary cross-entropy (BCE) performs the best.

Our hypothesis is that since our ground truth relevance scores are binary rather than continuous, the current task does not make full use of the listwise loss functions.

The effectiveness of the global re-ranker lies in the fact that it attends to the semantic interactions not only between the dialogue and the candidate questions but also between the candidate questions themselves.

Dialogue-Contextualized Re-Ranking For Medical History-Taking

Abstract: AI-driven medical history-taking is an important component in symptom checking, automated patient intake, triage, and other AI virtual care applications. As historytaking is extremely varied, machine learning models require a significant amount of data to train. To overcome this challenge, existing systems are developed using indirect data or expert knowledge. This leads to a training-inference gap as models are trained on different kinds of data than what they observe at inference time. In this work, we present a two-stage re-ranking approach that helps close the training-inference gap by re-ranking the first-stage question candidates using a dialogue-contextualized model. For this, we propose a new model, global re-ranker, which cross-encodes the dialogue with all questions simultaneously, and compare it with several existing neural baselines. We test both transformer and S4-based language model backbones. We find that relative to the expert system, the best performance is achieved by our proposed global re-ranker with a transformer backbone, resulting in a 30% higher normalized discount cumulative gain (nDCG) and a 77% higher mean average precision (mAP). As part of this work, we also release pre-trained checkpoints for bi-directional and autoregressive S4 models trained on Wikipedia and PubMed data.

Multi-Stage Document Ranking With Bert

Abstract: The advent of deep neural networks pre-trained via language modeling tasks has spurred a number of successful applications in natural language processing. This work explores one such popular model, BERT, in the context of document ranking. We propose two variants, called monoBERT and duoBERT, that formulate the ranking problem as pointwise and pairwise classification, respectively. These two models are arranged in a multi-stage ranking architecture to form an end-to-end search system. One major advantage of this design is the ability to trade off quality against latency by controlling the admission of candidates into each pipeline stage, and by doing so, we are able to find operating points that offer a good balance between these two competing metrics. On two large-scale datasets, MS MARCO and TREC CAR, experiments show that our model produces results that are either at or comparable to the state of the art. Ablation studies show the contributions of each component and characterize the latency/quality tradeoff space.


Please visit the publication #REFR or (Wu et al., 2019) for more details on other parameters.

Our MTFA #OTHEREFR baseline is implemented using Detectron2 framework (Wu et al., 2019) . Our backbone is

ResNet-101 #OTHEREFR with Feature Pyramid Network #OTHEREFR .

Each experiment is set up with a single GPU GeForce RTX 2080Ti with a batch size of 2 images.

The novel phase has a learning rate of 0.00125 inferred from the MTFA configuration.

We set the balance parameters α = 1e −1 and β = 1e −2 when we train the model with instance triplet and instance memory loss function, respectively.

Computer Vision And Image Understanding

Abstract: Camouflaged object detection and segmentation is a new and challenging research topic in computer vision. There is a serious issue of lacking data of camouflaged objects such as camouflaged animals in natural scenes. In this paper, we address the problem of few-shot learning for camouflaged object detection and segmentation. To this end, we first collect a new dataset, CAMO-FS, for the benchmark. We then propose a novel method to efficiently detect and segment the camouflaged objects in the images. In particular, we introduce the instance triplet loss and the instance memory storage. The extensive experiments demonstrated that our proposed method achieves state-of-the-art performance on the newly collected dataset.

Frustratingly Simple Few-Shot Object Detection

Abstract: Detecting rare objects from a few examples is an emerging problem. Prior works show meta-learning is a promising approach. But, fine-tuning techniques have drawn scant attention. We find that fine-tuning only the last layer of existing detectors on rare classes is crucial to the few-shot object detection task. Such a simple approach outperforms the meta-learning methods by roughly 2~20 points on current benchmarks and sometimes even doubles the accuracy of the prior methods. However, the high variance in the few samples often leads to the unreliability of existing benchmarks. We revise the evaluation protocols by sampling multiple groups of training examples to obtain stable comparisons and build new benchmarks based on three datasets: PASCAL VOC, COCO and LVIS. Again, our fine-tuning approach establishes a new state of the art on the revised benchmarks. The code as well as the pretrained models are available at https://github.com/ucbdrive/few-shot-object-detection.


The results in the figure are clean accuracy in DFKD and robust accuracy under AA attack #REFR in DFARD.

According to #OTHEREFR , proper smoothing factor for KD helps to better results.

Different degrees of softness indicates the difficulty of different learning objectives.

In this part, we use different temperatures #OTHEREFR to construct various soft labels as shown in Figure 2 .

At the same time, we carry out experiments in both DFKD and DFARD tasks to obtain comprehensive results. The experimental results are shown in Figure 3 .

(a) and (b) show the student's learning effectiveness of learning objectives with different difficulties, which are adjusted by different distillation temperatures. We distill all student models for 50 epochs.

An obvious conclusion is that, in both DFKD and DFARD tasks, a too-difficult learning target (low distillation temperature) or a too-simple target (high distillation temperature) leads to a decline in the student's performance.

Besides, we also show that maintaining a fixed temperature throughout the training process may affect the student's performance. The results can be found in Table 3 .

Model Robustness Meets Data Privacy: Adversarial Robustness Distillation Without Original Data

Abstract: Large-scale deep learning models have achieved great performance based on large-scale datasets. Moreover, the existing Adversarial Training (AT) can further improve the robustness of these large models. However, these large models are difficult to deploy to mobile devices, and the effect of AT on small models is very limited. In addition, the data privacy issue (e.g., face data and diagnosis report) may lead to the original data being unavailable, which relies on data-free knowledge distillation technology for training. To tackle these issues, we propose a challenging novel task called Data-Free Adversarial Robustness Distillation (DFARD), which tries to train small, easily deployable, robust models without relying on the original data. We find the combination of existing techniques resulted in degraded model performance due to fixed training objectives and scarce information content. First, an interactive strategy is designed for more efficient knowledge transfer to find more suitable training objectives at each epoch. Then, we explore an adaptive balance method to suppress information loss and obtain more data information than previous methods. Experiments show that our method improves baseline performance on the novel task.

Reliable Evaluation Of Adversarial Robustness With An Ensemble Of Diverse Parameter-Free Attacks

Abstract: The field of defense strategies against adversarial attacks has significantly grown over the last years, but progress is hampered as the evaluation of adversarial defenses is often insufficient and thus gives a wrong impression of robustness. Many promising defenses could be broken later on, making it difficult to identify the state-of-the-art. Frequent pitfalls in the evaluation are improper tuning of hyperparameters of the attacks, gradient obfuscation or masking. In this paper we first propose two extensions of the PGD-attack overcoming failures due to suboptimal step size and problems of the objective function. We then combine our novel attacks with two complementary existing ones to form a parameter-free, computationally affordable and user-independent ensemble of attacks to test adversarial robustness. We apply our ensemble to over 40 models from papers published at recent top machine learning and computer vision venues. In all except one of the cases we achieve lower robust test accuracy than reported in these papers, often by more than $10\%$, identifying several broken defenses.


The following lemma characterizes the non-asymptotic behavior of the MLE estimator. It is extracted from #REFR .

Define V s = s t=1 φ it,jt φ it,jt , and θ s as the MLE estimator (3) at round s.

x ∈ R d , with probability at least 1 − δ, | θ s − θ * , x | ≤ 3 κ x 2 V −1 s log(3/δ).

Borda Regret Minimization For Generalized Linear Dueling Bandits

Abstract: Dueling bandits are widely used to model preferential feedback that is prevalent in machine learning applications such as recommendation systems and ranking. In this paper, we study the Borda regret minimization problem for dueling bandits, which aims to identify the item with the highest Borda score while minimizing the cumulative regret. We propose a new and highly expressive generalized linear dueling bandits model, which covers many existing models. Surprisingly, the Borda regret minimization problem turns out to be difficult, as we prove a regret lower bound of order Ω(d 2/3 T 2/3), where d is the dimension of contextual vectors and T is the time horizon. To attain the lower bound, we propose an explore-then-commit type algorithm, which has a nearly matching regret upper bound O(d 2/3 T 2/3). When the number of items/arms K is small, our algorithm can achieve a smaller regret O((d log K) 1/3 T 2/3) with proper choices of hyperparameters. We also conduct empirical experiments on both synthetic data and a simulated real-world environment, which corroborate our theoretical analysis.

Provable Optimal Algorithms For Generalized Linear Contextual Bandits

Abstract: Contextual bandits are widely used in Internet services from news recommendation to advertising, and to Web search. Generalized linear models (logistical regression in particular) have demonstrated stronger performance than linear models in many applications where rewards are binary. However, most theoretical analyses on contextual bandits so far are on linear bandits. In this work, we propose an upper confidence bound based algorithm for generalized linear contextual bandits, which achieves an $\tilde{O}(\sqrt{dT})$ regret over $T$ rounds with $d$ dimensional feature vectors. This regret matches the minimax lower bound, up to logarithmic terms, and improves on the best previous result by a $\sqrt{d}$ factor, assuming the number of arms is fixed. A key component in our analysis is to establish a new, sharp finite-sample confidence bound for maximum-likelihood estimates in generalized linear models, which may be of independent interest. We also analyze a simpler upper confidence bound algorithm, which is useful in practice, and prove it to have optimal regret for certain cases.


Unlike existing multimodal grasping mechanisms (31, #REFR 37) , the same spiral robot with the same strategy can successfully grasp and manipulate a wide variety of objects in size and weight ( Fig. 6A-D, movie S5 ).

To demonstrate the ability to grasp and manipulate a variety of objects, we first designed and fabricated a 2-cable SpiRob (45 cm in length) by 3D printing using soft filaments ( Fig.

S2A ; another origami-based fabrication method is described in detail with Text S1).

This robot was used to demonstrate the potential of the proposed spiral-shaped design principle and operating strategy.

We tested for numerous objects varying in size, weight, and material.

Even in a confined space, the packed body can navigate through obstacles without relative sliding ( Fig. 6 E,  F, S6 , and movie S6).

Interestingly, we observed that the robot takes advantage of its interaction with the obstacles to reach the target instead of trying to avoid it.

Since visual perception and control were not the focus in these experiments, we manually controlled the two cables like a puppet (implementation of a remote-control scheme with a joystick is detailed in Fig. S2, movie S7 , and text S3).

Evidently, given that the base of the robot is fixed in our experiments, objects need to be positioned in a suitable range.

Abstract: Across various species and different scales, certain organisms use their appendages to grasp objects not through clamping but through wrapping. This pattern of movement is found in octopus tentacles, elephant trunks, and chameleon prehensile tails, demonstrating a great versatility to grasp a wide range of objects of various sizes and weights as well as dynamically manipulate them in the 3D space. We observed that the structures of these appendages follow a common pattern-a logarithmic spiral-which is especially challenging for existing robot designs to reproduce. This paper reports the design, fabrication, and operation of a class of cable-driven soft robots that morphologically replicate spiral-shaped wrapping. This amounts to substantially curling in length while actively controlling the curling direction as enabled by two principles: a) the parametric design based on the logarithmic spiral makes it possible to tightly pack to grasp objects that vary in size by more than two orders of magnitude and up to 260 times self-weight and b) asymmetric cable forces allow the swift control of the curling direction for conducting object manipulation. We demonstrate the ability to dynamically operate objects at a sub-second level by exploiting passive compliance. We believe that our study constitutes a step towards engineered systems that wrap to grasp and manipulate, and further sheds some insights into understanding the efficacy of biological spiral-shaped appendages. One-Sentence Summary: Design, fabrication, and operation of spiral soft robots at variable scales that can manipulate objects through wrapping. Main Text: INTRODUCTION Wrapping as a paradigm for grasping and manipulation (1), which are two key objectives in robotics (2, 3), is found in the prehensile tail of chameleons and seahorses with length scales as small as a few millimeters (4), as well as in the tentacles of octopuses and the trunks of elephants as large as a meter (Fig. 1A) (5, 6). These structures curl from their tips, tightly wrap around objects of different sizes and shapes, and continuously control the curling direction to manipulate them with unmatched efficiency (Fig. 1B). Roboticists have successfully achieved effective control of an object's motion using grasping-defined as the force-closure of an object (7). For example, clamping (8), digging (9), hooking (10), vacuum suction (11), magnetic attraction (12), and so on. Nevertheless, grasping and manipulating objects through wrapping remains by-and-large unaddressed, noting the succinct difference between the continuous deformation/compliant interaction (13) exhibited by biological systems and the discrete joints/rigid materials employed in robotic systems (7) that scientists and engineers have long worked on. Soft robots (13-17) made of flexible materials are well tailored to produce continuous deformations and interact with the environment (18-20). In particular, soft manipulators have shown high compliance and adaptability, enabling the development of grippers capable of grasping in an

Cartman: The Low-Cost Cartesian Manipulator That Won The Amazon Robotics Challenge

Abstract: The Amazon Robotics Challenge enlisted sixteen teams to each design a pick-and-place robot for autonomous warehousing, addressing development in robotic vision and manipulation. This paper presents the design of our custom-built. cost-effective robot system Cartman, which won first place in the competition finals by stowing 14 (out of 16) and picking all 9 items in 27 minutes, scoring a total of 272 points. We highlight our experience-centred design methodology and key aspects of our system that contributed to our competitiveness. We believe these aspects are crucial to building robust and effective robotic systems.


Feature Denoising (FD) #REFR proposes a block to purify the adversarial feature map and obtain state-of-theart on ImageNet.

A body of work uses new objective functions and regularizations to further improve the effectiveness of standard adversarial training.

TRADES #OTHEREFR analysis the trade-off between clean accuracy and adversarial accuracy, and uses Kullback-Leibler divergence #OTHEREFR to balance them.

Misclassification Aware adveRsarial Training (MART) #OTHEREFR emphasizes the misclassified examples.

Adversarial logit pairing (ALP) #OTHEREFR defines a pairing loss that pulls adversarial logit and natural logit together.

Adversarial Neural Pruning (ANP) #OTHEREFR uses Bayesian methods to prune vulnerable features.

#OTHEREFR uses sample reweighting techniques to improve the adversarial robustness against PGD attack, but not against AutoAttack.

Latent Feature Relation Consistency For Adversarial Robustness

Abstract: Deep neural networks have been applied in many computer vision tasks and achieved state-of-the-art performance. However, misclassification will occur when DNN predicts adversarial examples which add humanimperceptible adversarial noise to natural examples. This limits the application of DNN in security-critical fields. To alleviate this problem, we first conducted an empirical analysis of the latent features of both adversarial and natural examples and found the similarity matrix of natural examples is more compact than those of adversarial examples. Motivated by this observation, we propose Latent Feature Relation Consistency (LFRC), which constrains the relation of adversarial examples in latent space to be consistent with the natural examples. Importantly, our LFRC is orthogonal to the previous method and can be easily combined with them to achieve further improvement. To demonstrate the effectiveness of LFRC, we conduct extensive experiments using different neural networks on benchmark datasets. For instance, LFRC can bring 0.78% further improvement compared to AT, and 1.09% improvement compared to TRADES, against AutoAttack on CIFAR10. Code is available at https://github.com/ liuxingbin/LFRC.

Feature Denoising For Improving Adversarial Robustness

Abstract: Adversarial attacks to image classification systems present challenges to convolutional networks and opportunities for understanding them. This study suggests that adversarial perturbations on images lead to noise in the features constructed by these networks. Motivated by this observation, we develop new network architectures that increase adversarial robustness by performing feature denoising. Specifically, our networks contain blocks that denoise the features using non-local means or other filters; the entire networks are trained end-to-end. When combined with adversarial training, our feature denoising networks substantially improve the state-of-the-art in adversarial robustness in both white-box and black-box attack settings. On ImageNet, under 10-iteration PGD white-box attacks where prior art has 27.9% accuracy, our method achieves 55.7%; even under extreme 2000-iteration PGD white-box attacks, our method secures 42.6% accuracy. Our method was ranked first in Competition on Adversarial Attacks and Defenses (CAAD) 2018 --- it achieved 50.6% classification accuracy on a secret, ImageNet-like test dataset against 48 unknown attackers, surpassing the runner-up approach by ~10%. Code is available at https://github.com/facebookresearch/ImageNet-Adversarial-Training.


Judging by the comparably high predictive performance of local approximations such as LA and DE #REFR 29] , we conclude that a small amount of functional modes is reasonable to assume in practice. Our qualitative experiments in Section 5 support this supposition.

, π ν be the respective probabilities of the ν functionally diverse modes to be visited by an independently started Markov chain and Π J := j∈J π j . Then, given ρ chains,

The proof can be found in Supplementary Material A.6.

Note that this bound is independent of the NN architecture and only depends on the assumptions about the number and probabilities of functionally diverse modes ν, disregarding symmetric copies.

Proposition 2 can be used to calculate the number of MCMC chains given certain assumptions -for example, from domain knowledge, or in a worst-case scenario calculation -and thus provides practical guidance for MCMC sampling of MLPs.

As an example of applying Proposition 2, assume ν = 3 functionally diverse modes in a reference set with π 1 = 0.57, π 2 = 0.35, π 3 = 0.08 (chosen to represent a rather diverse functional mode set).

An upper bound of ρ = 1274 chains ensures that we observe all functionally diverse modes with probability P(G < ρ) ≥ 0.99.

Towards Efficient Mcmc Sampling In Bayesian Neural Networks By Exploiting Symmetry

Abstract: Bayesian inference in deep neural networks is challenging due to the high-dimensional, strongly multi-modal parameter posterior density landscape. Markov chain Monte Carlo approaches asymptotically recover the true posterior but are considered prohibitively expensive for large modern architectures. Local methods, which have emerged as a popular alternative, focus on specific parameter regions that can be approximated by functions with tractable integrals. While these often yield satisfactory empirical results, they fail, by definition, to account for the multi-modality of the parameter posterior. In this work, we argue that the dilemma between exact-but-unaffordable and cheap-but-inexact approaches can be mitigated by exploiting symmetries in the posterior landscape. Such symmetries, induced by neuron interchangeability and certain activation functions, manifest in different parameter values leading to the same functional output value. We show theoretically that the posterior predictive density in Bayesian neural networks can be restricted to a symmetry-free parameter reference set. By further deriving an upper bound on the number of Monte Carlo chains required to capture the functional diversity, we propose a straightforward approach for feasible Bayesian inference. Our experiments suggest that efficient sampling is indeed possible, opening up a promising path to accurate uncertainty quantification in deep learning.

Simple And Scalable Predictive Uncertainty Estimation Using Deep Ensembles

Abstract: Deep neural networks (NNs) are powerful black box predictors that have recently achieved impressive performance on a wide spectrum of tasks. Quantifying predictive uncertainty in NNs is a challenging and yet unsolved problem. Bayesian NNs, which learn a distribution over weights, are currently the state-of-the-art for estimating predictive uncertainty; however these require significant modifications to the training procedure and are computationally expensive compared to standard (non-Bayesian) NNs. We propose an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates. Through a series of experiments on classification and regression benchmarks, we demonstrate that our method produces well-calibrated uncertainty estimates which are as good or better than approximate Bayesian NNs. To assess robustness to dataset shift, we evaluate the predictive uncertainty on test examples from known and unknown distributions, and show that our method is able to express higher uncertainty on out-of-distribution examples. We demonstrate the scalability of our method by evaluating predictive uncertainty estimates on ImageNet.


SciCite #REFR is a much larger dataset built from 6,627 papers and has 11,020 instances tagged with 3 categories of coarse-grained citation intents.

We use two standard citation intent classification datasets, ACL-ARC and SciCite.

ACL-ARC #OTHEREFR is built from 186 papers from the ACL Anthology Reference Corpus #OTHEREFR and consists of 1,941 instances labeled with 6 citation intent labels.

Citeprompt: Using Prompts To Identify Citation Intent In Scientific Papers

Abstract: Citations in scientific papers not only help us trace the intellectual lineage but also are a useful indicator of the scientific significance of the work. Citation intents prove beneficial as they specify the role of the citation in a given context. We present a tool CitePrompt which uses the hitherto unexplored approach of prompt learning for citation intent classification. We argue that with the proper choice of the pretrained language model, the prompt template, and the prompt verbalizer, we can not only get results that are better than or comparable to those obtained with the state-of-the-art methods but also do it with much less exterior information about the scientific document. We report state-of-the-art results on the ACL-ARC dataset, and also show significant improvement on the SciCite dataset over all baseline models except one. As suitably large labelled datasets for citation intent classification can be quite hard to find, in a first, we propose the conversion of this task to the few-shot and zero-shot settings. For the ACL-ARC dataset, we report a 53.86% F1 score for the zero-shot setting, which improves to 63.61% and 66.99% for the 5-shot and 10-shot settings respectively. CCS CONCEPTS • Computing methodologies → Natural language processing; • Information systems → Digital libraries and archives.

Structural Scaffolds For Citation Intent Classification In Scientific Publications

Abstract: Identifying the intent of a citation in scientific papers (e.g., background information, use of methods, comparing results) is critical for machine reading of individual publications and automated analysis of the scientific literature. We propose structural scaffolds, a multitask model to incorporate structural information of scientific papers into citations for effective classification of citation intents. Our model achieves a new state-of-the-art on an existing ACL anthology dataset (ACL-ARC) with a 13.3% absolute increase in F1 score, without relying on external linguistic resources or hand-engineered features as done in existing methods. In addition, we introduce a new dataset of citation intents (SciCite) which is more than five times larger and covers multiple scientific domains compared with existing datasets. Our code and data are available at: this https URL.


In addition, skip connections between the encoder and the decoder facilitate training, especially by reducing the vanishing gradient issue #REFR ).

The architecture used in this chapter is the UNet architecture #OTHEREFR depicted in Fig. 1 .

UNet is an autoencoder architecture with the advantage of being fully convolutional, meaning it has translation equivariance properties (translations of the input result in translations of the output).

Introduced in 2015, UNet has been used in various domains and has demonstrated its importance for segmentation of SAR observations #OTHEREFR , #OTHEREFR , #OTHEREFR .

The output of the model always contains a single convolution kernel, activated by the ReLU function to ensure that the prediction is in the interval [0, +∞[.

All convolution kernels in the hidden layers are also activated by ReLU functions.

The model is set to take input of 256x256 pixels during training, but since the weights only describe convolution kernels, it is possible to use the model for inference on images of any shape as long as the input resolution remains at 100 m/px.

Variants of the model are trained with different numbers of input channels.

Reduction Of Rain-Induced Errors For Wind Speed Estimation On Sar Observations Using Convolutional Neural Networks

Abstract: Synthetic Aperture Radar is known to be able to provide high-resolution estimates of surface wind speed. These estimates usually rely on a Geophysical Model Function (GMF) that has difficulties accounting for non-wind processes such as rain events. Convolutional neural network, on the other hand, have the capacity to use contextual information and have demonstrated their ability to delimit rainfall areas. By carefully building a large dataset of SAR observations from the Copernicus Sentinel-1 mission, collocated with both GMF and atmospheric model wind speeds as well as rainfall estimates, we were able to train a wind speed estimator with reduced errors under rain. Collocations with in-situ wind speed measurements from buoys show a root mean square error that is reduced by 27% (resp. 45%) under rainfall estimated at more than 1 mm/h (resp. 3 mm/h). These results demonstrate the capacity of deep learning models to correct rain-related errors in SAR products.

On The Difficulty Of Training Recurrent Neural Networks

Abstract: There are two widely known issues with properly training recurrent neural networks, the vanishing and the exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to improve the understanding of the underlying issues by exploring these problems from an analytical, a geometric and a dynamical systems perspective. Our analysis is used to justify a simple yet effective solution. We propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. We validate empirically our hypothesis and proposed solutions in the experimental section.


Given the lack of generative methods that can scale up to this data, we compare with a monocular depth method from #REFR , also trained on Taskonomy.

We also wish to showcase how our method scales to real data, and past the object-centric nature of ShapeNet that Table 4 : Evaluation on Taskonomy. Please refer to Fig. 4 for qualitative examples.

For this purpose we turn to Taskonomy #OTHEREFR , which contains a large dataset of scanned indoor scenes with high-quality depth maps, and convert them into point clouds by sampling and unprojecting 8192 points per image.

We sample points inversely proportionally to pixel depth, to emulate per-surface-area densities. This yields a rich image-point cloud dataset.

We first adjust the absolute scale and shift of its output by comparing with ground truth depth (as in the loss function of MiDaS #OTHEREFR ) and proceed by unprojecting with the same procedure as when creating the dataset.

For GECCO, we directly predict the point clouds in absolute units, using the (u, v, l) reparameterization introduced in Sec. 3.2. We use 2048 points for both training and evaluation. We compare the two approaches qualitatively in Fig.

4 , and quantitatively in Table 4 using the same metric as in Sec. 4.2 and Table 3 .

This experiment confirms our method extends beyond object-centric views to real scenes, and is competitive with similarly-sized baselines benefitting from years of research on monocular depth.

As in Sec 4.2, we also report results with ICP.

Gecco : Geometrically-Conditioned Point Diffusion Models

Abstract: Diffusion models generating images conditionally on text, such as Dall-E 2 [51] and Stable Diffusion[53], have recently made a splash far beyond the computer vision community. Here, we tackle the related problem of generating point clouds, both unconditionally, and conditionally with images. For the latter, we introduce a novel geometricallymotivated conditioning scheme based on projecting sparse image features into the point cloud and attaching them to each individual point, at every step in the denoising process. This approach improves geometric consistency and yields greater fidelity than current methods relying on unstructured, global latent codes. Additionally, we show how to apply recent continuous-time diffusion schemes [59, 21]. Our method performs on par or above the state of art on conditional and unconditional experiments on synthetic data, while being faster, lighter, and delivering tractable likelihoods. We show it can also scale to diverse indoors scenes.

Learning To Navigate Using Mid-Level Visual Priors

Abstract: How much does having visual priors about the world (e.g. the fact that the world is 3D) assist in learning to perform downstream motor tasks (e.g. navigating a complex environment)? What are the consequences of not utilizing such visual priors in learning? We study these questions by integrating a generic perceptual skill set (a distance estimator, an edge detector, etc.) within a reinforcement learning framework (see Fig. 1). This skill set ("mid-level vision") provides the policy with a more processed state of the world compared to raw images. ::: Our large-scale study demonstrates that using mid-level vision results in policies that learn faster, generalize better, and achieve higher final performance, when compared to learning from scratch and/or using state-of-the-art visual and non-visual representation learning methods. We show that conventional computer vision objectives are particularly effective in this regard and can be conveniently integrated into reinforcement learning frameworks. Finally, we found that no single visual representation was universally useful for all downstream tasks, hence we computationally derive a task-agnostic set of representations optimized to support arbitrary downstream tasks.


Given the sequence embedding e , a deep neural network model (e.g., SASRec #REFR ) represented as (•) is utilized to learn the representation of the sequence. Where represents the parameters of the sequential model.

The output representation H ∈ R × is calculated as:

The last vector h ∈ R in H = [h 0 , h 1 , ..., h ] is chosen as the representation of the sequence #OTHEREFR .

Meta-Optimized Contrastive Learning For Sequential Recommendation

Abstract: Contrastive Learning (CL) performances as a rising approach to address the challenge of sparse and noisy recommendation data. Although having achieved promising results, most existing CL methods only perform either hand-crafted data or model augmentation for generating contrastive pairs to find a proper augmentation operation for different datasets, which makes the model hard to generalize. Additionally, since insufficient input data may lead the encoder to learn collapsed embeddings, these CL methods expect a relatively large number of training data (e.g., large batch size or memory bank) to contrast. However, not all contrastive pairs are always informative and discriminative enough for the training processing. Therefore, a more general CL-based recommendation model called Meta-optimized Contrastive Learning for sequential Recommendation (MCLRec) is proposed in this work. By applying both data augmentation and learnable model augmentation operations, this work innovates the standard CL framework by contrasting data and model augmented views for adaptively capturing the informative features hidden in stochastic data augmentation. Moreover, MCLRec utilizes a meta-learning manner to guide the updating of the model augmenters, which helps to improve the quality of contrastive pairs without enlarging the amount of input data. Finally, a contrastive regularization term is considered to encourage the augmentation model to generate more informative augmented views and avoid too similar contrastive pairs within

Self-Attentive Sequential Recommendation

Abstract: Sequential dynamics are a key feature of many modern recommender systems, which seek to capture the 'context' of users' activities on the basis of actions they have performed recently. To capture such patterns, two approaches have proliferated: Markov Chains (MCs) and Recurrent Neural Networks (RNNs). Markov Chains assume that a user's next action can be predicted on the basis of just their last (or last few) actions, while RNNs in principle allow for longer-term semantics to be uncovered. Generally speaking, MC-based methods perform best in extremely sparse datasets, where model parsimony is critical, while RNNs perform better in denser datasets where higher model complexity is affordable. The goal of our work is to balance these two goals, by proposing a self-attention based sequential model (SASRec) that allows us to capture long-term semantics (like an RNN), but, using an attention mechanism, makes its predictions based on relatively few actions (like an MC). At each time step, SASRec seeks to identify which items are 'relevant' from a user's action history, and use them to predict the next item. Extensive empirical studies show that our method outperforms various state-of-the-art sequential models (including MC/CNN/RNN-based approaches) on both sparse and dense datasets. Moreover, the model is an order of magnitude more efficient than comparable CNN/RNN-based models. Visualizations on attention weights also show how our model adaptively handles datasets with various density, and uncovers meaningful patterns in activity sequences.


A coarse-to-fine spatial pyramid network is employed in SPynet #REFR to estimate optical flow at multiple scales.

Optical flow estimation aims to estimate the per-pixel motion between adjacent frames.

Deep learning-based optical flow models have been considerably advanced since Flownet #OTHEREFR , a pioneering work to learn convolutional neural models from synthetic data.

Flownet2.0 #OTHEREFR improves the accuracy of optical flow estimation by stacking subnetworks for iterative refinement.

PWC-Net #OTHEREFR employs feature warping operation at different resolution and uses a cost volume layer to refine the estimated flow at each resolution.

RAFT #OTHEREFR proposes a lightweight recurrent network that is weight-sharing during the iterative process.

Flow-Former #OTHEREFR designs an encoder to output latent tokens and a recurrent decoder to decode features and refine the estimated flow iteratively.

In video synthesis, optical flow for downstream tasks #OTHEREFR is also a hot research topic.

Based on these approaches, for the video prediction task, we aim to design a flow estimation network that can specifically transform based on each sample.

A Dynamic Multi-Scale Voxel Flow Network For Video Prediction

Abstract: The performance of video prediction has been greatly boosted by advanced deep neural networks. However, most of the current methods suffer from large model sizes and require extra inputs, e.g., semantic/depth maps, for promising performance. For efficiency consideration, in this paper, we propose a Dynamic Multi-scale Voxel Flow Network (DMVFN) to achieve better video prediction performance at lower computational costs with only RGB images, than previous methods. The core of our DMVFN is a differentiable routing module that can effectively perceive the motion scales of video frames. Once trained, our DMVFN selects adaptive sub-networks for different inputs at the inference stage. Experiments on several benchmarks demonstrate that our DMVFN is an order of magnitude faster than Deep Voxel Flow [32] and surpasses the state-of-the-art iterative-based OPT [59] on generated image quality.

Optical Flow Estimation Using A Spatial Pyramid Network

Abstract: We learn to compute optical flow by combining a classical spatial-pyramid formulation with deep learning. This estimates large motions in a coarse-to-fine approach by warping one image of a pair at each pyramid level by the current flow estimate and computing an update to the flow. Instead of the standard minimization of an objective function at each pyramid level, we train one deep network per level to compute the flow update. Unlike the recent FlowNet approach, the networks do not need to deal with large motions, these are dealt with by the pyramid. This has several advantages. First, our Spatial Pyramid Network (SPyNet) is much simpler and 96% smaller than FlowNet in terms of model parameters. This makes it more efficient and appropriate for embedded applications. Second, since the flow at each pyramid level is small (


Finally, AR applications allow users to interact with their interfaces using more natural approaches (natural interactions) #REFR .

How can we tell users that a hyper-realistic virtual object is not genuine without hindering the user experience and immersion?

Besides the realism of virtual objects, the immersiveness and interactivity of these environments can also impact the effectiveness of dark patterns.

Similar to the effects of audio on the shoppers' buying intentions in retail stores #OTHEREFR , AR applications can create immersive dark patterns using visual, audio, and haptic feedback.

The AR ecosystem can augment the effect of dark patterns into influencing users' behaviours.

It is necessary to analyze the main factors influencing users when dark patterns (using visual, audio, and haptic) are applied in AR environments, and how can we protect users in these immersive experiences.

These natural interactions can also be the subject of dark pattern techniques.

For example, when a user clicks a UI button via gaze or using the index fingertip as a controller.

In this case, a dark pattern can hide (hijack) the authentic cursor behind a virtual object and use a fake cursor to click another UI interface (e.g., an ad banner) (H. #OTHEREFR .

These virtual elements, being inserted during the input interactions of users, require framework-driven regulations within the AR ecosystem. Following the work in (H.

#OTHEREFR , future AR platforms should consider the defense against dark pattern attacks, whose techniques are implemented during the interaction process (e.g., cursor movement).

The Dark Side Of Augmented Reality: Exploring Manipulative Designs In Ar

Abstract: Augmented Reality (AR) applications are becoming more mainstream, with successful examples in the mobile environment like Pokemon GO. Current malicious techniques can exploit these environments' immersive and mixed nature (physicalvirtual) to trick users into providing more personal information, i.e., dark patterns. Dark patterns are deceiving techniques (e.g., interface tricks) designed to influence individuals' behavioural decisions. However, there are few studies regarding dark patterns' potential issues in AR environments. In this work, using scenario construction to build our prototypes, we investigate the potential future approaches that dark patterns can have. We use VR mockups in our user study to analyze the effects of dark patterns in AR. Our study indicates that dark patterns are effective in immersive scenarios, and the use of novel techniques such as 'haptic grabbing' to drag participants' attention can influence their movements. Finally, we discuss the impact of such malicious techniques and what techniques can mitigate them.

Emerging Natural User Interfaces In Mobile Computing: A Bottoms-Up Survey

Abstract: Mobile and wearable interfaces and interaction paradigms are highly constrained by the available screen real estate, and the computational and power resources. Although there exist many ways of displaying information to mobile users, inputting data to a mobile device is, usually, limited to a conventional touch based interaction, that distracts users from their ongoing activities. Furthermore, emerging applications, like augmented, mixed and virtual reality (AR/MR/VR), require new types of input methods in order to interact with complex virtual worlds, challenging the traditional techniques of Human-Computer Interaction (HCI). Leveraging of Natural User Interfaces (NUIs), as a paradigm of using natural intuitive actions to interact with computing systems, is one of many ways to meet these challenges in mobile computing and its modern applications. Brain-Machine Interfaces that enable thought-only hands-free interaction, Myoelectric input methods that track body gestures and gaze-tracking input interfaces - are the examples of NUIs applicable to mobile and wearable interactions. The wide adoption of wearable devices and the penetration of mobile technologies, alongside with the growing market of AR/MR/VR, motivates the exploration and implementation of new interaction paradigms. The concurrent development of bio-signal acquisition techniques and accompanying ecosystems offers a useful toolbox to address open challenges. In this survey, we present state-of-the-art bio-signal acquisition methods, summarize and evaluate recent developments in the area of NUIs and outline potential application in mobile scenarios. The survey will provide a bottoms-up overview starting from (i) underlying biological aspects and signal acquisition techniques, (ii) portable NUI hardware solutions, (iii) NUI-enabled applications, as well as (iv) research challenges and open problems.


As reported in #REFR , the solid mechanics and fluid mechanics can be studied in a unified framework.

The contact phenomena between different phases of materials is perfectly satisfied automatically. The distorted mesh problem is also not existent. There is no need to perform the self-adaptive mesh.

Thus, the advantages of the Eulerian and Lagrangian frameworks are both included in this method.

The N-S equation is the governing equation for the fluid mechanics.

But actually, it is also able to study the solid mechanics.

There is no need to distinguish the solid mechanics and the fluid mechanics in numerical methods.

The solid mechanics is different from the fluid mechanics because of its stress-history-dependency, meaning that the current stress increment is not only dependent on its current deformation status, but also dependent on the history of the accumulated plastic strain, the previous stress history and the potential internal variable history.

To consider solid mechanics using N-S equation, one extra stress term is added to represent the history-dependent stress. The corresponding VMS-FEM formulation is derived in this paper.

The non-local stress theory is a relative-distance-weighted averaging algorithm that can clean the noise of the original stress field.

Typically, the source point's stress is calculated as a relative-distance-weighted summation of the stress values of all the nearby field points.

The relative distance between the source point and the field point determines the weight function uniquely.

A Vms-Fem For The Stress-History-Dependent Materials (Fluid Or Solid) Interacting With The Rigid Body Structure Part A: Formulation, Numerical Verification And Application In The Pipe-Soil-Water Interaction Analysis First Author

Abstract: The proposed VMS-FEM shows very good accuracy in the benchmark problems. The pipe-soil-water interaction analysis is presented.

A Unified Continuum And Variational Multiscale Formulation For Fluids, Solids, And Fluid-Structure Interaction

Abstract: We develop a unified continuum modeling framework for viscous fluids and hyperelastic solids using the Gibbs free energy as the thermodynamic potential. This framework naturally leads to a pressure primitive variable formulation for the continuum body, which is well-behaved in both compressible and incompressible regimes. Our derivation also provides a rational justification of the isochoric-volumetric additive split of free energies in nonlinear continuum mechanics. The variational multiscale analysis is performed for the continuum model to construct a foundation for numerical discretization. We first consider the continuum body instantiated as a hyperelastic material and develop a variational multiscale formulation for the hyper-elastodynamic problem. The generalized-alpha method is applied for temporal discretization. A segregated algorithm for the nonlinear solver is designed and carefully analyzed. Second, we apply the new formulation to construct a novel unified formulation for fluid-solid coupled problems. The variational multiscale formulation is utilized for spatial discretization in both fluid and solid subdomains. The generalized-alpha method is applied for the whole continuum body, and optimal high-frequency dissipation is achieved in both fluid and solid subproblems. A new predictor multi-corrector algorithm is developed based on the segregated algorithm to attain a good balance between robustness and efficiency. The efficacy of the new formulations is examined in several benchmark problems. The results indicate that the proposed modeling and numerical methodologies constitute a promising technology for biomedical and engineering applications, particularly those necessitating incompressible models.


They also train on multiple classes, a strategy that #REFR showed will slow down the overfitting.

Quite the contrary; the research in Section 2 supports their strategies.

Their AUC decreased from the public to the private leaderboard, but it was still within the confidence interval, suggesting that they have avoided overfitting.

They had a low rank on the public leaderboard (880th out of 3300), and they have avoided the general overfitting that can be seen in Fig. 8 .

To avoid overfitting, they use additional data from previous years' competitions, and they also do data augmentation.

They chose ensemble learning as a strategy to avoid overfitting, a strategy implicitly supported by #OTHEREFR .

To estimate multiplicity effect on the AUC scores is not as straightforward as with accuracy.

The uncertainty of AUC can be calculated through bootstrapping, but without access to the predictions for each data point in the test set, the analyst must make some assumptions about the shape of the ROC curve.

However, with high AUC, the shape has less impact, and the analyst's choices are not suggestive of the overall conclusion. If we assume i.d.d.

classifiers and AUC = 0.9352, the the upper bound of the 95% confidence interval is 0.9490.

And since there are 427 teams with AUC > 0.9352, we can expect observations outside the confidence interval.

What Is The State Of The Art? Accounting For Multiplicity In Machine Learning Benchmark Performance

Abstract: Machine learning methods are commonly evaluated and compared by their performance on data sets from public repositories, where methods can be evaluated under identical conditions and across time. Theoretical results play an important role in machine learning research, but for the last couple of decades the proven success of increasingly complex methods has pushed the proof towards the pudding; performance on benchmark data sets. The numbers don't lie: if method A performs better than method B on a specific data set, that is what it did. We think of a data set, for example a collection of images, as a random sample from an underlying population. In science we are interested in the performance on the population, and the observed performance acts as an estimate. A good estimator gives a reliable result, but overfitting to the data set, originating from wrong use of cross-validation, data leakage and data reuse, can give overly optimistic observations. An independent test set, often referred to as a hold-out set, is a sample that is not used for development of the method, and gives an unbiased estimator. Competitions and challenges with prize money typically withhold the test set so that it becomes impossible for the participants to use it for development. Although we know these things we have a tendency to extrapolate the performance metric and interpret it as something about the world rather than something about a data set. If we always appended "on MNIST" when we said so-and-so method had accuracy so-and-so nobody would care. Who cares about MNIST? We hold out data precisely because we want to estimate some sort of long-term behavior on similar data.

The Advantages Of Multiple Classes For Reducing Overfitting From Test Set Reuse

Abstract: Excessive reuse of holdout data can lead to overfitting. However, there is little concrete evidence of significant overfitting due to holdout reuse in popular multiclass benchmarks today. Known results show that, in the worst-case, revealing the accuracy of $k$ adaptively chosen classifiers on a data set of size $n$ allows to create a classifier with bias of $\Theta(\sqrt{k/n})$ for any binary prediction problem. We show a new upper bound of $\tilde O(\max\{\sqrt{k\log(n)/(mn)},k/n\})$ on the worst-case bias that any attack can achieve in a prediction problem with $m$ classes. Moreover, we present an efficient attack that achieve a bias of $\Omega(\sqrt{k/(m^2 n)})$ and improves on previous work for the binary setting ($m=2$). We also present an inefficient attack that achieves a bias of $\tilde\Omega(k/n)$. Complementing our theoretical work, we give new practical attacks to stress-test multiclass benchmarks by aiming to create as large a bias as possible with a given number of queries. Our experiments show that the additional uncertainty of prediction with a large number of classes indeed mitigates the effect of our best attacks. ::: Our work extends developments in understanding overfitting due to adaptive data analysis to multiclass prediction problems. It also bears out the surprising fact that multiclass prediction problems are significantly more robust to overfitting when reusing a test (or holdout) dataset. This offers an explanation as to why popular multiclass prediction benchmarks, such as ImageNet, may enjoy a longer lifespan than what intuition from literature on binary classification suggests.


The results are shown in Table 5 : although adversarial training with PGD could effectively defend the PGD attack with a precision 75.6 (IoU=0.5), using a different attacker, Carlini & Wagner Attack #REFR , will largely degrade the precision to 43.2 (IoU=0.5).

Specifically, we compare the current output with the previous output for consensus verification.

This can help to improve efficiency because we can reduce the times of model forward caused by individual perception. The results are shown in Table 4 .

We see that FPS is improved from 14.2 to 19.2, yet the performance is still comparable to using individual perception results as a reference.

Generalizability. We compare the generalizability of ROBOSAC to adversarial training.

We use PGD #OTHEREFR which is the strongest one-stage gradient-based adversarial attack during adversarial training.

In contrast, our method can achieve comparable precision under both two attackers (77.9@IoU0.5 under PGD attack, and 74.5@IoU=0.5 under C&W attack).

Our better generalizability stems from being attacker-agnostic: we do not rely on the knowledge of attackers while adversarial training does. Attacker ratio estimation.

In practice, using a budget of five samplings within a single frame yields acceptable results.

As shown in Table 7 , within the initial few frames, the actual attacker ratio can be efficiently probed.

The estimated ratio can then be used to carry out ROBOSAC steps.

Among Us: Adversarially Robust Collaborative Perception By Consensus

Abstract: Multiple robots could perceive a scene (e.g., detect objects) collaboratively better than individuals, although easily suffer from adversarial attacks when using deep learning. This could be addressed by the adversarial defense, but its training requires the often-unknown attacking mechanism. Differently, we propose ROBOSAC, a novel samplingbased defense strategy generalizable to unseen attackers. Our key idea is that collaborative perception should lead to consensus rather than dissensus in results compared to individual perception. This leads to our hypothesize-and-verify framework: perception results with and without collaboration from a random subset of teammates are compared until reaching a consensus. In such a framework, more teammates in the sampled subset often entail better perception performance but require longer sampling time to reject potential attackers. Thus, we derive how many sampling trials are needed to ensure the desired size of an attacker-free subset, or equivalently, the maximum size of such a subset that we can successfully sample within a given number of trials. We validate our method on the task of collaborative 3D object detection in autonomous driving scenarios.

Towards Evaluating The Robustness Of Neural Networks

Abstract: Neural networks provide state-of-the-art results for most machine learning tasks. Unfortunately, neural networks are vulnerable to adversarial examples: given an input $x$ and any target classification $t$, it is possible to find a new input $x'$ that is similar to $x$ but classified as $t$. This makes it difficult to apply neural networks in security-critical areas. Defensive distillation is a recently proposed approach that can take an arbitrary neural network, and increase its robustness, reducing the success rate of current attacks' ability to find adversarial examples from $95\%$ to $0.5\%$. ::: In this paper, we demonstrate that defensive distillation does not significantly increase the robustness of neural networks by introducing three new attack algorithms that are successful on both distilled and undistilled neural networks with $100\%$ probability. Our attacks are tailored to three distance metrics used previously in the literature, and when compared to previous adversarial example generation algorithms, our attacks are often much more effective (and never worse). Furthermore, we propose using high-confidence adversarial examples in a simple transferability test we show can also be used to break defensive distillation. We hope our attacks will be used as a benchmark in future defense attempts to create neural networks that resist adversarial examples.


There is similar corroboration in other scenarios, such as the positive effect of the sparsity regularity on generalization ability #REFR .

Theorem 1 indicates the generalization depends on the size of the global model (i.e., d) reduced by the number of clients |C| (the first term), as the global model is shared by all clients.

It also depends on the size of the gating layers (the second term), and it is not reduced by |C| since each client has its own personalized parameters.

Besides, the generalization is also affected by the Lipschitz constants of the global model, sparse model, and the gating layer, as they together constrain the parameter space that our method can search.

Specifically, as the sparsity s i decreases, the smaller theÑ , and the better the generalization.

This is also to some extent a reflection of Ockham's Razor in our approach.

Efficient Personalized Federated Learning Via Sparse Model-Adaptation

Abstract: Federated Learning (FL) aims to train machine learning models for multiple clients without sharing their own private data. Due to the heterogeneity of clients' local data distribution, recent studies explore the personalized FL that learns and deploys distinct local models with the help of auxiliary global models. However, the clients can be heterogeneous in terms of not only local data distribution, but also their computation and communication resources. The capacity and efficiency of personalized models are restricted by the lowest-resource clients, leading to suboptimal performance and limited practicality of personalized FL. To overcome these challenges, we propose a novel approach named pFedGate for efficient personalized FL by adaptively and efficiently learning sparse local models. With a lightweight trainable gating layer, pFedGate enables clients to reach their full potential in model capacity by generating different sparse models accounting for both the heterogeneous data distributions and resource constraints. Meanwhile, the computation and communication efficiency are both improved thanks to the adaptability between the model sparsity and clients' resources. Further, we theoretically show that the proposed pFedGate has superior complexity with guaranteed convergence and generalization error. Extensive experiments show that pFedGate achieves superior global accuracy, individual accuracy and efficiency simultaneously over state-of-the-art methods. We also demonstrate that pFedGate performs better than competitors in the novel clients participation and partial clients participation scenarios, and can learn meaningful sparse local models adapted to different data distributions.

Structured Sparsity And Generalization

Abstract: We present a data dependent generalization bound for a large class of regularized algorithms which implement structured sparsity constraints. The bound can be applied to standard squared-norm regularization, the Lasso, the group Lasso, some versions of the group Lasso with overlapping groups, multiple kernel learning and other regularization schemes. In all these cases competitive results are obtained. A novel feature of our bound is that it can be applied in an infinite dimensional setting such as the Lasso in a separable Hilbert space or multiple kernel learning with a countable number of kernels.

generalization ability
sparsity regularity


B.2. More Analyses Of Human-Centric Tasks

Another interesting finding is that, although the mean AP score on MSCOCO #REFR is reduced after joint training, the percentage of the low scores is reduced as well, as shown in Fig. 13(b) and Fig. 13(d) .

This issue has become more serious in artificial scenes of Human-Art. For instance, in Fig. 13 (e) and Fig.

13 (f) , although the pose estimation model shows a relatively high confidence score in most images, a large proportion of the estimation outputs' Figure 12 . Annotated examples in Human-Art.

We randomly select 3 images from each category to show the annotation quality and image diversity of Human-Art.

Images in Human-Art are varied in terms of human shape, pose, texture and size. AP scores range from 0 to 0.25. (2) The contrast between before training and after training.

The recurring finding is that training can reduce the percentage of both low confidence scores and low AP scores, which is consistent with common sense.

This may be because the more evenly distributed human size and the richer depictions in Human-Art help the model to obtain better adaptability on hard poses in real-world scenarios.

(3) The contrast between human detection and pose estimation tasks.

After training, human detection shows a more uniform AP distribution along the horizontal axis.

However, the pose estimation model shows concentrated distributions in low and high AP scores.

This may be due to the differences in the two methods' targets.

Human-Art: A Versatile Human-Centric Dataset Bridging Natural And Artificial Scenes

Abstract: Humans have long been recorded in a variety of forms since antiquity. For example, sculptures and paintings were the primary media for depicting human beings before the invention of cameras. However, most current human-centric computer vision tasks like human pose estimation and human image generation focus exclusively on natural images in the real world. Artificial humans, such as those in sculptures, paintings, and cartoons, are commonly neglected, making existing models fail in these scenarios. As an abstraction of life, art incorporates humans in both natural and artificial scenes. We take advantage of it and introduce the Human-Art dataset to bridge related tasks in natural and artificial scenarios. Specifically, Human-Art contains 50k high-quality images with over 123k person instances from 5 natural and 15 artificial scenarios, which are annotated with bounding boxes, keypoints, self-contact points, and text information for humans represented in both 2D and 3D. It is, therefore, comprehensive and versatile for various downstream tasks. We also provide a rich set of baseline results and detailed analyses for related tasks, including human detection, 2D and 3D human pose estimation, image generation, and motion transfer. As a challenging dataset, we hope Human-Art can provide insights for relevant research and open up new research questions. * Equal contribution. † Work done during an internship at IDEA.

Microsoft Coco: Common Objects In Context

Abstract: We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.


Empirical Investigation 5.1 Finding 1: Neural Models Excel On Datasets With Larger Sizes In Memorization

Nevertheless, it is also worth noting that several non-neural models also perform well, e.g., linear models, which is consistent with the findings from existing work (e.g., #REFR ) in weak generalization settings.

For example, Ease, as a linear recommendation model, outperforms neural models on both utility metrics.

We examine the differences between neural and non-neural models in the so-called reranking memorization task.

We observe that neural models are among the ones with the best reranking performance.

A potential reason could be that the high model capacity enables neural models to perform well in memorizing all implicit user feedback that has been seen during training.

Memorization under Different Dataset Sizes.

Based on the previous results, we further zoom in on the differences regarding the fitting capabilities of neural models and non-neural models.

Specifically, we compute the average performance ranking in the reranking memorization task for each baseline model over datasets with the top-smallest and largest sizes (in terms of the total number of implicit user feedback).

We present the results for = 2 , we present the results of the two different memorization tasks, respectively.

In (c), we show the generalization results in terms of recommendation utility.

When Newer Is Not Better: Does Deep Learning Really Benefit Recommendation From Implicit Feedback?

Abstract: In recent years, neural models have been repeatedly touted to exhibit state-of-the-art performance in recommendation. Nevertheless, multiple recent studies have revealed that the reported stateof-the-art results of many neural recommendation models cannot be reliably replicated. A primary reason is that existing evaluations are performed under various inconsistent protocols. Correspondingly, these replicability issues make it difficult to understand how much benefit we can actually gain from these neural models. It then becomes clear that a fair and comprehensive performance comparison between traditional and neural models is needed. Motivated by these issues, we perform a large-scale, systematic study to compare recent neural recommendation models against traditional ones in top-recommendation from implicit data. We propose a set of evaluation strategies for measuring memorization performance, generalization performance, and subgroup-specific performance of recommendation models. We conduct extensive experiments with 13 popular recommendation models (including two neural models and 11 traditional ones as baselines) on nine commonly used datasets. Our experiments demonstrate that even with extensive hyper-parameter searches, neural models do not dominate traditional models in all aspects, e.g., they fare worse in terms of average HitRate. We further find that there are areas where neural models seem to outperform non-neural models, for example, in recommendation diversity and robustness between different subgroups of users and items. Our work illuminates the relative advantages and disadvantages of neural models in recommendation and is therefore an important step towards building better recommender systems.

Are We Really Making Much Progress? A Worrying Analysis Of Recent Neural Recommendation Approaches

Abstract: Deep learning techniques have become the method of choice for researchers working on algorithmic aspects of recommender systems. With the strongly increased interest in machine learning in general, it has, as a result, become difficult to keep track of what represents the state-of-the-art at the moment, e.g., for top-n recommendation tasks. At the same time, several recent publications point out problems in today's research practice in applied machine learning, e.g., in terms of the reproducibility of the results or the choice of the baselines when proposing new models. In this work, we report the results of a systematic analysis of algorithmic proposals for top-n recommendation tasks. Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques. The remaining one clearly outperformed the baselines but did not consistently outperform a well-tuned non-neural linear ranking method. Overall, our work sheds light on a number of potential problems in today's machine learning scholarship and calls for improved scientific practices in this area.


This is in line with the theoretical findings from #REFR , which states that GINECONV is strictly more expressive than all other Message Passing Layers.

Possible explanations for this are that 1.) the query graphs are rather small so 2 steps of message-passing are sufficient, and 2.) fewer layers avoid the oversmoothing problem #OTHEREFR , where node representations become increasingly similar, which deteriorates the model performance.

! , | ! | # , | # | … $ , | $ | min − log ( ) Training !"# $ $%! Prediction New

Our custom Message Passing function, which we term TPN Message Passing, is a modulation of the GINECONV #OTHEREFR message passing function defined as

We initially evaluated several different Message Passing Layers but found GINECONV to be superior to the others.

It also aligns with results from LSS [Zhao et al., 2021] , which also uses GINECONV as the Message Passing Function.

θ is a multilayer perception with a ReLU Activation function.

N (i) are all neighboring nodes of node i with corresponding edges e j,i .

Cardinality Estimation Over Knowledge Graphs With Embeddings And Graph Neural Networks

Abstract: Cardinality Estimation over Knowledge Graphs (KG) is crucial for query optimization, yet remains a challenging task due to the semi-structured nature and complex correlations of typical Knowledge Graphs. In this work, we propose GNCE, a novel approach that leverages knowledge graph embeddings and Graph Neural Networks (GNN) to accurately predict the cardinality of conjunctive queries. GNCE first creates semantically meaningful embeddings for all entities in the KG, which are then integrated into the given query, which is processed by a GNN to estimate the cardinality of the query. We evaluate GNCE on several KGs in terms of q-Error and demonstrate that it outperforms state-of-the-art approaches based on sampling, summaries, and (machine) learning in terms of estimation accuracy while also having lower execution time and less parameters. Additionally, we show that GNCE can inductively generalize to unseen entities, making it suitable for use in dynamic query processing scenarios. Our proposed approach has the potential to significantly improve query optimization and related applications that rely on accurate cardinality estimates of conjunctive queries.

How Powerful Are Graph Neural Networks?

Abstract: Graph Neural Networks (GNNs) are an effective framework for representation learning of graphs. GNNs follow a neighborhood aggregation scheme, where the representation vector of a node is computed by recursively aggregating and transforming representation vectors of its neighboring nodes. Many GNN variants have been proposed and have achieved state-of-the-art results on both node and graph classification tasks. However, despite GNNs revolutionizing graph representation learning, there is limited understanding of their representational properties and limitations. Here, we present a theoretical framework for analyzing the expressive power of GNNs to capture different graph structures. Our results characterize the discriminative power of popular GNN variants, such as Graph Convolutional Networks and GraphSAGE, and show that they cannot learn to distinguish certain simple graph structures. We then develop a simple architecture that is provably the most expressive among the class of GNNs and is as powerful as the Weisfeiler-Lehman graph isomorphism test. We empirically validate our theoretical findings on a number of graph classification benchmarks, and demonstrate that our model achieves state-of-the-art performance.


The two decoders also follows the general line of the original Transformer #REFR (see Fig.

We only retain the RNN output at the last time index T , since it contains all information on the complete sequence s1:T .

The sampled value of w is then replicated T times and concatenated to s1:T to produce the input of the Transformer encoder.

This latter follows the line of the original Transformer #OTHEREFR (see Fig.

1 ): input embedding followed by positional encoding, followed by a Multi-Head Attention (MHA) module, normalization layer (NL) with skip connection with the MHA input, feed-forward (FF) layer and another NL with skip connection with the FF layer input.

Finally, a linear layer produces the parameters of (6), that is µ φz,t and the logarithm of v φz,t .

1 ; here we do not detail the different modules as we did for the encoder, because of room limitation; see #OTHEREFR for details).

The sampled value of z1:T is concatenated with the replicated sampled value of w to produce the "main" input, which is the source information for the query of the second MHA module.

In parallel, s1:T is used as the second input, i.e.

the source information for the key and value of the second MHA module.

A causal mask in the MHA module ensures that the dependency at time t is on s1:t−1.

Speech Modeling With A Hierarchical Transformer Dynamical Vae

Abstract: The dynamical variational autoencoders (DVAEs) are a family of latent-variable deep generative models that extends the VAE to model a sequence of observed data and a corresponding sequence of latent vectors. In almost all the DVAEs of the literature, the temporal dependencies within each sequence and across the two sequences are modeled with recurrent neural networks. In this paper, we propose to model speech signals with the Hierarchical Transformer DVAE (HiT-DVAE), which is a DVAE with two levels of latent variable (sequence-wise and frame-wise) and in which the temporal dependencies are implemented with the Transformer architecture. We show that HiT-DVAE outperforms several other DVAEs for speech spectrogram modeling, while enabling a simpler training procedure, revealing its high potential for downstream low-level speech processing tasks such as speech enhancement.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


We trained our appearance-based I3D model #REFR on the ASL Citizen dataset to establish an initial baseline, which achieves a top-1 accuracy (recall-at-1) of 62.10%, with a DCG of 0.784 and MRR of 0.725 (see the third row in Table 3 ).

4 This accuracy is notable, given the di culty of the problem -our dataset has completely unseen users, and spans one of the largest vocabulary sizes in ISLR to date (2,731 signs).

The pose-based ST-GCN model performs similarly, but consistently worse by a few percentage points on all metrics, but still substantially better than any previous reported results on datasets of similar size and complexity.

For comparison, due to the number of classes, random guessing would only yield 0.04% expected accuracy.

In previous work, appearance-based and pose-based models have generally shown competitive performance.

Pose-based methods reduce information and potentially introduce errors at the keypoint extraction step, at the bene t of making features relevant for signs more accessible and standardized compared to raw pixels.

Asl Citizen: A Community-Sourced Dataset For Advancing Isolated Sign Language Recognition

Abstract: Sign languages are used as a primary language by approximately 70 million D/deaf people worldwide. However, most communication technologies operate in spoken and written languages, creating inequities in access. To help tackle this problem, we release ASL Citizen, the largest Isolated Sign Language Recognition (ISLR) dataset to date, collected with consent and containing 83,912 videos for 2,731 distinct signs lmed by 52 signers in a variety of environments. We propose that this dataset be used for sign language dictionary retrieval for American Sign Language (ASL), where a user demonstrates a sign to their own webcam with the aim of retrieving matching signs from a dictionary. We show that training supervised machine learning classi ers with our dataset greatly advances the state-of-the-art on metrics relevant for dictionary retrieval, achieving, for instance, 62% accuracy and a recall-at-10 of 90%, evaluated entirely on videos of users who are not present in the training or validation sets. Preprint. Under review.

Quo Vadis, Action Recognition? A New Model And The Kinetics Dataset

Abstract: The paucity of videos in current action classification datasets (UCF-101 and HMDB-51) has made it difficult to identify good video architectures, as most methods obtain similar performance on existing small-scale benchmarks. This paper re-evaluates state-of-the-art architectures in light of the new Kinetics Human Action Video dataset. Kinetics has two orders of magnitude more data, with 400 human action classes and over 400 clips per class, and is collected from realistic, challenging YouTube videos. We provide an analysis on how current architectures fare on the task of action classification on this dataset and how much performance improves on the smaller benchmark datasets after pre-training on Kinetics. ::: We also introduce a new Two-Stream Inflated 3D ConvNet (I3D) that is based on 2D ConvNet inflation: filters and pooling kernels of very deep image classification ConvNets are expanded into 3D, making it possible to learn seamless spatio-temporal feature extractors from video while leveraging successful ImageNet architecture designs and even their parameters. We show that, after pre-training on Kinetics, I3D models considerably improve upon the state-of-the-art in action classification, reaching 80.9% on HMDB-51 and 98.0% on UCF-101.


Similarly, the depth-separable regular convolutions of the XceptionNet architecture #REFR are put to the test.

We show that the proposed network model provides higher sensitivity and interpretability than the current COVID-Net #OTHEREFR and COVID-NET-S #OTHEREFR .

We also employ ResNet50, a ResNet variant developed by Kaiming He et al.

#OTHEREFR , with 50 layers, where we replace the output layer with a regression head with two outputs. We also tested the Swin transformer #OTHEREFR .

This is a hierarchical transformer architecture whose representation is generated by shifted windows.

It can serve as the main structural support for a regression task performed for evaluation.

We also tested the InceptionNet architecture, #OTHEREFR which emphasizes parallel processing and concurrent feature extraction.

Moreover, we tested the model proposed by Cohen et al.

#OTHEREFR which was trained using a large dataset as a feature extractor and allows score predictions.

In addition, we tested MobileNetV3, a convolutional neural network tailored to cell phone CPUs through a combination of hardware-aware network architecture search (NAS) and the NetAdapt algorithm #OTHEREFR .

The output of this model was also modified to predict the score through a regression head.

Vision Transformer-Based Model For Severity Quantification Of Lung Pneumonia Using Chest X-Ray Images

Abstract: To develop generic and reliable approaches for diagnosing and assessing the severity of COVID-19 from chest X-rays (CXR), a large number of well-maintained COVID-19 datasets are needed. Existing severity quantification architectures require expensive training calculations to achieve the best results. For healthcare professionals to quickly and automatically identify COVID-19 patients and predict associated severity indicators, computer utilities are needed. In this work, we propose a Vision Transformer (ViT)-based neural network model that relies on a small number of trainable parameters to quantify the severity of COVID-19 and other lung diseases. We present a feasible approach to quantify the severity of CXR, called Vision Transformer Regressor Infection Prediction (ViTReg-IP), derived from a ViT and a regression head. We investigate the generalization potential of our model using a variety of additional test chest radiograph datasets from different open sources. In this context, we performed a comparative study with several competing deep learning analysis methods. The experimental results show that our model can provide peak performance in quantifying severity with high generalizability at a relatively low computational cost. The source codes used in our work are publicly available at https://github.com/ bouthainas/ViTReg-IP.

Xception: Deep Learning With Depthwise Separable Convolutions

Abstract: We present an interpretation of Inception modules in convolutional neural networks as being an intermediate step in-between regular convolution and the depthwise separable convolution operation (a depthwise convolution followed by a pointwise convolution). In this light, a depthwise separable convolution can be understood as an Inception module with a maximally large number of towers. This observation leads us to propose a novel deep convolutional neural network architecture inspired by Inception, where Inception modules have been replaced with depthwise separable convolutions. We show that this architecture, dubbed Xception, slightly outperforms Inception V3 on the ImageNet dataset (which Inception V3 was designed for), and significantly outperforms Inception V3 on a larger image classification dataset comprising 350 million images and 17,000 classes. Since the Xception architecture has the same number of parameters as Inception V3, the performance gains are not due to increased capacity but rather to a more efficient use of model parameters.

XceptionNet architecture
depth-separable regular convolutions


Similar to IMP in #REFR , we adopt a progressive learning process, where each round prunes p 1/k % of the weights on top of the previous round (p = 20% by default).

Yet, different from IMP, we replace the optimization step of retraining non-zero weights with unlearning on non-zero model weights.

We term the resulting method 'AO-sparse MU' and summarize its pipeline (S1)-(S3) below.

(S1) Initialize model θ = θ o and a pruning mask m = 1. (S2) Prune p% non-zero parameters in θ per magnitude. Then, update m to a sparser mask.

(S3) Unlearn D f on the sparse model m θ using an approximate unlearning method (e.g., FT) to update the non-zero model parameters and go back to (S2).

The above procedure (S1)→(S2) (S3) repeatedly prunes and unlearns the model over multiple rounds (assuming k rounds).

However, different from IMP, AO-sparse MU is much lighter in computation since approximate unlearning in (S2) takes fewer computation overheads than retraining non-zero model weights in IMP.

Model Sparsification Can Simplify Machine Unlearning

Abstract: Recent data regulations necessitate machine unlearning (MU): The removal of the effect of specific examples from the model. While exact unlearning is possible by conducting a model retraining with the remaining data from scratch, its computational cost has led to the development of approximate but efficient unlearning schemes. Beyond data-centric MU solutions, we advance MU through a novel model-based viewpoint: sparsification via weight pruning. Our results in both theory and practice indicate that model sparsity can boost the multi-criteria unlearning performance of an approximate unlearner, closing the approximation gap, while continuing to be efficient. With this insight, we develop two new sparsity-aware unlearning meta-schemes, termed 'prune first, then unlearn' and 'sparsity-aware unlearning'. Extensive experiments show that our findings and proposals consistently benefit MU in various scenarios including class-wise data scrubbing, random data scrubbing, and backdoor data forgetting. One highlight is the 77% unlearning efficacy gain of fine-tuning (one of the simplest approximate unlearning methods) in the proposed sparsity-aware unlearning paradigm. Codes are available at https://github.com/OPTML-Group/Unlearn-Sparse.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Abstract: Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance. We find that a standard technique for pruning weights naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the"lottery ticket hypothesis": dense, randomly-initialized feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - arrive at comparable test accuracy in a comparable number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective. We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Furthermore, the winning tickets we find above that size learn faster than the original network and exhibit higher test accuracy.


For the terms z i , we will prove the following by the steps (1) and #REFR below.

By the discrete Helmholtz decomposition in the last equation in (4.9), we decompose u h in (4.9) as

u h = curlw 0 + z 0 + N i=1 (curlw i + z i ).

For the terms curlw i of u h , by Section 3 in this paper, we have w 0 ∈ N D H,0 and w i ∈

V i , which satisfy N i=0 a div (curlw i , curlw i ) ≤ C(1 + H δ )a div (curlw ⊥ h , curlw ⊥ h ). (4.11)

(1) For the coarse component z 0 , by (4.9), (4.8), (4.1) and (4.7), we have

||divz 0 || 0 = ||div Q H P h z ⊥ h || 0 ≤ C| P h z ⊥ h | 1 ≤ C||div P h z ⊥ h || 0 = C||divz ⊥ h || 0 .

By (4.9), (4.8), (4.7) and the definition Θ ⊥ , we get

On A Sharp Estimate Of Overlapping Schwarz Methods In H(Curl; Ω) And H(Div; Ω)

Abstract: The previous proved-bound is C(1 + H 2 δ 2) for the condition number of the overlapping domain decomposition H(curl; Ω) and H(div; Ω) methods, where H and δ are the sizes of subdomains and overlaps respectively. But all numerical results indicate that the best bound is C(1 + H δ). In this work, we solve this long-standing open problem by proving that C(1 + H δ) is indeed the best bound.

Domain Decomposition Preconditioning For The High-Frequency Time-Harmonic Maxwell Equations With Absorption

Abstract: This paper rigorously analyses preconditioners for the time-harmonic Maxwell equations with absorption, where the PDE is discretised using curl-conforming finite-element methods of fixed, arbitrary order and the preconditioner is constructed using additive Schwarz domain decomposition methods. The theory developed here shows that if the absorption is large enough, and if the subdomain and coarse mesh diameters and overlap are chosen appropriately, then the classical two-level overlapping additive Schwarz preconditioner (with PEC boundary conditions on the subdomains) performs optimally--in the sense that GMRES converges in a wavenumber-independent number of iterations--for the problem with absorption. An important feature of the theory is that it allows the coarse space to be built from low-order elements even if the PDE is discretised using high-order elements. It also shows that additive methods with minimal overlap can be robust. Numerical experiments are given that illustrate the theory and its dependence on various parameters. These experiments motivate some extensions of the preconditioners which have better robustness for problems with less absorption, including the propagative case. At the end of the paper we illustrate the performance of these on two substantial applications; the first (a problem with absorption arising from medical imaging) shows the empirical robustness of the preconditioner against heterogeneity, and the second (scattering by a COBRA cavity) shows good scalability of the preconditioner with up to 3,000 processors.


Model training comes at a high environmental cost, as energy is required to run it on hardware for weeks or months at a time #REFR .

They provide superior performance in these tasks compared to traditional models, and there is a trend for developing even larger and more powerful DNNs #OTHEREFR .

However, these models generally require large amounts of data and high computing power, which is associated with high energy consumption incurring high financial and environmental costs #OTHEREFR .

Furthermore, for emerging applications such as autonomous driving and the Internet of Things, models must be run on low-power devices #OTHEREFR .

Therefore, energy-efficient DL is crucial from an economic, environmental, and application perspective.

The efficiency of DL can be divided into training efficiency and inference efficiency #OTHEREFR .

However, training a DL model is only the beginning of the lifecycle.

Once the model is trained, it will be implemented and used. This process, called inference, also consumes enormous energy.

Inference does not last weeks or months, but unlike training, it is not a one-time event.

It takes place continuously and can therefore exceed the energy consumption of the training after a certain number of inference events #OTHEREFR .

Various researchers have presented approaches, methods, and techniques such as mixed-precision training, pruning or knowledge distillation that can accelerate training and inference time and reduce the energy consumption of the models.

Towards Energy-Efficient Deep Learning: An Overview Of Energyefficient Approaches Along The Deep Learning Lifecycle

Abstract: Deep Learning has enabled many advances in machine learning applications in the last few years. However, since current Deep Learning algorithms require much energy for computations, there are growing concerns about the associated environmental costs. Energy-efficient Deep Learning has received much attention from researchers and has already made much progress in the last couple of years. This paper aims to gather information about these advances from the literature and show how and at which points along the lifecycle of Deep Learning (IT-Infrastructure, Data, Modeling, Training, Deployment, Evaluation) it is possible to reduce energy consumption.

Energy And Policy Considerations For Deep Learning In Nlp

Abstract: Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.


One solution is to incorporate imitation learning with RL to improve the robustness of the policy with fine-tuning. #REFR integrates a task-specific inverse dynamic model into RL.

One-shot imitation learning means robots are able to learn from a single demonstration of a given task and then generalize to new situations of the same task #OTHEREFR . However, it is very challenging due to data scarcity.

#OTHEREFR extracts the tool trajectory from the video and aligns the simulated environment with the video to initialize a policy for RL to learn.

There are two main drawbacks of this kind of method: (1) training robots in the real world is challenging due to sample efficiency and safety concerns.

(2) learning in simulation necessitates a high level of verisimilitude with the actual world and additional techniques to deal with the sim-to-real gap.

Other researchers #OTHEREFR , #OTHEREFR formulate this problem as visual servoing.

#OTHEREFR iterates the pose aligning between the robot and the human hand in the video; #OTHEREFR obtains the approaching policy in a self-supervised manner through backward learning from the goal pose. However, the above methods hold strong taskrelated assumptions (e.g.

Efficient Robot Skill Learning With Imitation From A Single Video For Contact-Rich Fabric Manipulation

Abstract: Classical policy search algorithms for robotics typically require performing extensive explorations, which are timeconsuming and expensive to implement with real physical platforms. To facilitate the efficient learning of robot manipulation skills, in this work, we propose a new approach comprised of three modules: (1) learning of general prior knowledge with random explorations in simulation, including state representations, dynamic models, and the constrained action space of the task; (2) extraction of a state alignment-based reward function from a single demonstration video; (3) real-time optimization of the imitation policy under systematic safety constraints with sampling-based model predictive control. This solution results in an efficient one-shot imitation-from-video strategy that simplifies the learning and execution of robot skills in real applications. Specifically, we learn priors in a scene of a task family and then deploy the policy in a novel scene immediately following a single demonstration, preventing time-consuming and risky explorations in the environment. As we do not make a strong assumption of dynamic consistency between the scenes, learning priors can be conducted in simulation to avoid collecting data in real-world circumstances. We evaluate the effectiveness of our approach in the context of contact-rich fabric manipulation, which is a common scenario in industrial and domestic tasks. Detailed numerical simulations and real-world hardware experiments reveal that our method can achieve rapid skill acquisition for challenging manipulation tasks.

Ridm: Reinforced Inverse Dynamics Modeling For Learning From A Single Observed Demonstration

Abstract: Imitation learning has long been an approach to alleviate the tractability issues that arise in reinforcement learning. However, most literature makes several assumptions such as access to the expert's actions, availability of many expert demonstrations, and injection of task-specific domain knowledge into the learning process. We propose reinforced inverse dynamics modeling (RIDM), a method of combining reinforcement learning and imitation from observation (IfO) to perform imitation using a single expert demonstration, with no access to the expert's actions, and with little task-specific domain knowledge. Given only a single set of the expert's raw states, such as joint angles in a robot control task, at each time-step, we learn an inverse dynamics model to produce the necessary low-level actions, such as torques, to transition from one state to the next such that the reward from the environment is maximized. We demonstrate that RIDM outperforms other techniques when we apply the same constraints on the other methods on six domains of the MuJoCo simulator and for two different robot soccer tasks for two experts from the RoboCup 3D simulation league on the SimSpark simulator.