GANs #REFR have been one of the primary families of generative models for natural image synthesis.

While these models exhibit unprecedented image synthesis ability, they require timeconsuming iterative processes to achieve high-quality image sampling.

To accelerate the sampling, several methods propose to reduce the sampling steps #OTHEREFR or reuse pre-computed features #OTHEREFR .

Latent Diffusion Model (LDM) #OTHEREFR performs the reverse processes in lowdimensional latent space instead of pixel space.

However, consecutive reverse processes are still computationally expensive, limiting the usage of large-scale text-to-image models for interactive applications.

As the sampling quality and diversity of GANs improve [39-42, 44, 72, 84] , GANs have been deployed to various computer vision and graphics applications, such as text-to-image synthesis #OTHEREFR , image-to-image translation #OTHEREFR , and image editing #OTHEREFR .

Notably, StyleGAN-family models #OTHEREFR have shown impressive ability in image synthesis tasks for single-category domains #OTHEREFR 112] .

Other works have explored class-conditional GANs #OTHEREFR on datasets with a fixed set of object categories.

In this paper, we change the data regimes from single-or multi-categories datasets to extremely data-rich situations.

We make the first expedition toward training a large-scale GAN for text-to-image generation on a vast amount of webcrawled text and image pairs, such as LAION2B-en #OTHEREFR and COYO-700M #OTHEREFR .

Scaling Up Gans For Text-To-Image Synthesis

Abstract: The recent success of text-to-image synthesis has taken the world by storm and captured the general public's imagination. From a technical standpoint, it also marked a drastic change in the favored architecture to design generative image models. GANs used to be the de facto choice, with techniques like StyleGAN. With DALL•E 2, autoregressive and diffusion models became the new standard for largescale generative models overnight. This rapid shift raises a fundamental question: can we scale up GANs to benefit from large datasets like LAION? We find that naïvely increasing the capacity of the StyleGAN architecture quickly becomes unstable. We introduce GigaGAN, a new GAN architecture that far exceeds this limit, demonstrating GANs as a viable option for text-to-image synthesis. GigaGAN offers three major advantages. First, it is orders of magnitude faster at inference time, taking only 0.13 seconds to synthesize a 512px image. Second, it can synthesize high-resolution images, for example, 16-megapixel images in 3.66 seconds. Finally, GigaGAN supports various latent space editing applications such as latent interpolation, style mixing, and vector arithmetic operations.

Generative Adversarial Nets

Abstract: We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to ½ everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.


We use EfficientNet #REFR as the task-specific backbone to extract the patch visual features h i and we train the diffusion model with T = 300 and sample it with inference ratio r = 10.

We evaluate the performance of Positional Diffusion using the Direct Comparison Metric #OTHEREFR , a percentage that indicates the number of correctly ordered pieces over the full test set. A higher score indicates better performance. Implementation Details.

We divide an image in n × n patches, resulting a total of K = n 2 elements.

We divide a 2D space with a range of (-1,-1), (1,1) into a grid of n × n cells.

We use the centers of the cells as starting positions X for the patches.

The input data for puzzle solving are the pixel values for each patch, resized to 32x32.

We train a single model with all puzzle sizes simultaneously.

At inference, we arrange the patches by mapping each estimated patch positionx i to a cell in the grid.

We measure the distance between each patch position and cells' centers, and assign each patch to its closest cell, mapping each cell to at most one patch.

By using a greedy approach that prioritizes the assignment between cells-patch pairs starting from those with the lowest distance, we ensure that the most confident prediction will be assigned first, increasing the prediction robustness to noise. Comparisons.

We compared Positional Diffusion against a set of SOTA methods for puzzle solving.

Positional Diffusion: Ordering Unordered Sets With Diffusion Probabilistic Models

Abstract: Positional reasoning is the process of ordering unsorted parts contained in a set into a consistent structure. We present Positional Diffusion, a plug-and-play graph formulation with Diffusion Probabilistic Models to address positional reasoning. We use the forward process to map elements' positions in a set to random positions in a continuous space. Positional Diffusion learns to reverse the noising process and recover the original positions through an Attention-based Graph Neural Network. We conduct extensive experiments with benchmark datasets including two puzzle datasets, three sentence ordering datasets, and one visual storytelling dataset, demonstrating that our method outperforms long-lasting research on puzzle solving with up to +18% compared to the second-best deep learning method, and performs on par against the state-of-the-art methods on sentence ordering and visual storytelling. Our work highlights the suitability of diffusion models for ordering problems and proposes a novel formulation and method for solving various ordering tasks. Project website at https://iitpavis.github.io/Positional Diffusion/

Efficientnet: Rethinking Model Scaling For Convolutional Neural Networks

Abstract: Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet. ::: To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.4% top-1 / 97.1% top-5 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters. Source code is at this https URL.


By converting coordinates to heatmaps, we can exploit the strong feature extraction power of ResNet #REFR as ResNet takes tensor inputs, not vector inputs.

When a hand is in neutral pose and the other one is in fist pose, their 3D relative depth can be zero although the hand with fist pose takes smaller area.

Hence, pose is necessary to determine z of the 3D relative translation.

Please note that the 2D affine transformations do not affect the depths of each 2.5D pose; hence, the depths of each 2.5D pose still represent the root joint-relative depths of each hand.

We denote the transformed P r and P l by P ′ r and P ′ l , respectively.

The 2.5D pose of the right hand P ′ r and left hand P ′ l are converted to 2.5D Gaussian heatmaps by making a Gaussian blob around the coordinates.

Then, we concatenate the 2.5D Gaussian heatmap of two hands in a channel dimension, denoted by H ∈ R 2J×D×H×W .

D, H, and W represent the depth, height, and width of the 2.5D heatmap, respectively, and we set them to 64. Output: 3D relative translation between two hands.

We predict the 3D relative translation between two hands t ∈ R 3 from the 2.5D Gaussian heatmap H.

We pass H to ResNet-18 #OTHEREFR , which produces a feature map F ∈ R C×H/8×W/8 . C = 512 represents the channel dimension of F.

We use the original ResNet-18 after dropping the first convolutional block to reduce the downsampling and the last fully-connected layers.

Bringing Inputs To Shared Domains For 3D Interacting Hands Recovery In The Wild

Abstract: Despite recent achievements, existing 3D interacting hands recovery methods have shown results mainly on motion capture (MoCap) environments, not on in-the-wild (ITW) ones. This is because collecting 3D interacting hands data in the wild is extremely challenging, even for the 2D data. We present InterWild, which brings MoCap and ITW samples to shared domains for robust 3D interacting hands recovery in the wild with a limited amount of ITW 2D/3D interacting hands data. 3D interacting hands recovery consists of two sub-problems: 1) 3D recovery of each hand and 2) 3D relative translation recovery between two hands. For the first sub-problem, we bring MoCap and ITW samples to a shared 2D scale space. Although ITW datasets provide a limited amount of 2D/3D interacting hands, they contain large-scale 2D single hand data. Motivated by this, we use a single hand image as an input for the first sub-problem regardless of whether two hands are interacting. Hence, interacting hands of MoCap datasets are brought to the 2D scale space of single hands of ITW datasets. For the second sub-problem, we bring MoCap and ITW samples to a shared appearance-invariant space. Unlike the first subproblem, 2D labels of ITW datasets are not helpful for the second sub-problem due to the 3D translation's ambiguity. Hence, instead of relying on ITW samples, we amplify the generalizability of MoCap samples by taking only a geometric feature without an image as an input for the second sub-problem. As the geometric feature is invariant to appearances, MoCap and ITW samples do not suffer from a huge appearance gap between the two datasets. The code is publicly available 1 .

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


Ironically, because it can handle raw EHR data without any preprocessing or feature selection, UniHPF generates extremely long embedded data, given how it encodes entire EHR data in a text-encoding fashion #REFR .

This is a clear limitation in that extensive human labor and clinical domain knowledge are required to produce satisfactory model performance.

Universal Healthcare Predictive Framework Recently, #OTHEREFR presented UniHPF, a universal framework that can embed entire features of raw EHR regardless of schema and medical code standard used in the database.

Specifically, UniHPF views EHR data as pure text and flattened the EHR tables (e.g.

prescriptions, lab results) to feed them to Transformer-based text encoders.

This text-based approach showed comparable if not better performance compared to conventional approaches on various predictive tasks, even without relying on medical domain knowledge.

This imposes strong computational limits on the framework and necessitates an additional module for compressing the embedded data to a smaller size.

Rediscovery Of Cnn'S Versatility For Text-Based Encoding Of Raw Electronic Health Records

Abstract: Making the most use of abundant information in electronic health records (EHR) is rapidly becoming an important topic in the medical domain. Recent work presented a promising framework that embeds entire features in raw EHR data regardless of its form and medical code standards. The framework, however, only focuses on encoding EHR with minimal preprocessing and fails to consider how to learn efficient EHR representation in terms of computation and memory usage. In this paper, we search for a versatile encoder not only reducing the large data into a manageable size but also well preserving the core information of patients to perform diverse clinical tasks. We found that hierarchically structured Convolutional Neural Network (CNN) often outperforms the state-of-the-art model on diverse tasks such as reconstruction, prediction, and generation, even with fewer parameters and less training time. Moreover, it turns out that making use of the inherent hierarchy of EHR data can boost the performance of any kind of backbone models and clinical tasks performed. Through extensive experiments, we present concrete evidence to generalize our research findings into real-world practice. We give a clear guideline on building the encoder based on the research findings captured while exploring numerous settings.

Doctor Ai: Predicting Clinical Events Via Recurrent Neural Networks

Abstract: Leveraging large historical data in electronic health record (EHR), we developed Doctor AI, a generic predictive model that covers observed medical conditions and medication uses. Doctor AI is a temporal model using recurrent neural networks (RNN) and was developed and applied to longitudinal time stamped EHR data from 260K patients over 8 years. Encounter records (e.g. diagnosis codes, medication codes or procedure codes) were input to RNN to predict (all) the diagnosis and medication categories for a subsequent visit. Doctor AI assesses the history of patients to make multilabel predictions (one label for each diagnosis or medication category). Based on separate blind test set evaluation, Doctor AI can perform differential diagnosis with up to 79% recall@30, significantly higher than several baselines. Moreover, we demonstrate great generalizability of Doctor AI by adapting the resulting models from one institution to another without losing substantial accuracy.


For all models, we used a batch size of 1 image pair and the Adam optimizer #REFR We analyze the performance of baselines and our proposed KeyMorph under conditions of large initial misalignments in terms of rotation. Fig.

We train six variants of KeyMorph (KM) corresponding to six pre-determined transformations: 1 model using an affine transformation and 5 models using a TPS transformation with λ ∈ {0, 0.01, 0.1, 1, 10}. We refer to these models as KM-Aff and KM-TPS-λ.

Additionally, we train a variant using a TPS transformation where λ is sampled stochastically during training from a log-uniform distribution p(λ) = LogUni f (0, 10). We refer to this model as KM-LogUnif.

We used N = 128 keypoints throughout our experiments, and analyze the effect of changing this number in Section 6.2.

3 plots overall Dice across rotation angle of the moving image for baselines and two variants of KeyMorph: KM-Aff and KM-TPS-0.

We find that all DLIR models suffer substantially as the rotation angle increases.

Training with aggressive augmentation increases performance for test pairs with large misalignments, but reduces the accuracy for those with smaller misalignments. Using supervision (dashed DLIR lines) leads to improved accuracy.

For unimodal registration, the DLIR model that was trained with all modalities (DLIR-Multi) did not produce better accuracy than a model that was trained with each modality separately.

ANTs yields excellent results when the initial misalignment is small (e.g. near 0 degrees of rotation).

A Robust And Interpretable Deep Learning Framework For Multi-Modal Registration Via Keypoints

Abstract: We present KeyMorph, a deep learning-based image registration framework that relies on automatically detecting corresponding keypoints. State-of-the-art deep learning methods for registration often are not robust to large misalignments, are not interpretable, and do not incorporate the symmetries of the problem. In addition, most models produce only a single prediction at test-time. Our core insight which addresses these shortcomings is that corresponding keypoints between images can be used to obtain the optimal transformation via a differentiable closed-form expression. We use this observation to drive the end-to-end learning of keypoints tailored for the registration task, and without knowledge of ground-truth keypoints. This framework not only leads to substantially more robust registration but also yields better interpretability, since the keypoints reveal which parts of the image are driving the final alignment. Moreover, KeyMorph can be designed to be equivariant under image translations and/or symmetric with respect to the input image ordering. Finally, we show how multiple deformation fields can be computed efficiently and in closed-form at test time corresponding to different transformation variants. We demonstrate the proposed framework in solving 3D affine and spline-based registration of multi-modal brain MRI scans. In particular, we show registration accuracy that surpasses current state-of-the-art methods, especially in the context of large displacements. Our code is available at https://github.com/evanmy/keymorph.

Adam: A Method For Stochastic Optimization

Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.

large initial misalignments
Adam optimizer


Since there are no available reported results of existing methods on the IRR task on MIMIC-CXR #REFR , we compare MOTOR and its variants for analyzing the impact of different components in MOTOR deeply. We present the results in Table 1 .

For multi-modal pretraining in the natural image domain #OTHEREFR , #OTHEREFR , #OTHEREFR , the image-text retrieval task is a simple and effective way to evaluate the crossmodal understanding ability of pretrained models.

Therefore, we develop an image-report retrieval (IRR) task in our benchmark.

Two subtasks are included in IRR: report retrieval (RR), where images and reports are queries and targets, respectively; and image retrieval (IR), where reports are queries and images are targets.

The R@K (recall with top k predictions) metric is used for the performance evaluation for both subtasks.

We directly use the pretraining dataset MIMIC-CXR #OTHEREFR for evaluating the IRR performance under both zero-shot and finetuning settings.

Table 1 comprehensively show the effectiveness of each method component of MOTOR.

Moreover, the significant performance superiority of MOTOR over other variants under the zero-shot setting sufficiently demonstrates the excellent generalization ability of MOTOR.

Concretely, under the zero-shot setting, "MOTOR w GK/SK" outperforms "MOTOR w/o knowledge" by a large margin.

This demonstrates that both the general and specific knowledge can improve the medical multi-modal pretraining.

Moreover, the performance improvement of "MOTOR w GK/SK" over "MOTOR w GK+SK" reveals that the general and specific knowledge can complement each other well.

Towards Medical Artificial General Intelligence Via Knowledge-Enhanced Multimodal Pretraining

Abstract: Medical artificial general intelligence (MAGI) enables one foundation model to solve different medical tasks, which is very practical in the medical domain. It can significantly reduce the requirement of large amounts of task-specific data by sufficiently sharing medical knowledge among different tasks. However, due to the challenges of designing strongly generalizable models with limited and complex medical data, most existing approaches tend to develop task-specific models. To take a step towards MAGI, we propose a new paradigm called Medical-knOwledge-enhanced mulTimOdal pretRaining (MOTOR). In MOTOR, we combine two kinds of basic medical knowledge, i.e., general and specific knowledge, in a complementary manner to boost the general pretraining process. As a result, the foundation model with comprehensive basic knowledge can learn compact representations from pretraining radiographic data for better cross-modal alignment, i.e., matching the medical data of different modalities with the same semantic meanings. MOTOR unifies the understanding and generation, which are two kinds of core intelligence of an AI system, into a single medical foundation model, to flexibly handle more diverse medical tasks. To enable a comprehensive evaluation and facilitate future research, we construct a medical multimodal benchmark including a wide range of downstream tasks, such as chest x-ray report generation and medical visual question answering. Extensive experiments on our benchmark show that MOTOR obtains promising results through simple task-oriented adaptation. The visualization shows that the injected knowledge successfully highlights key information in the medical data, demonstrating the excellent interpretability of MOTOR. Our MOTOR successfully mimics the human practice of fulfilling a "medical student" to accelerate the process of becoming a "specialist". We believe that our work makes a significant stride in realizing MAGI.

Mimic-Cxr-Jpg, A Large Publicly Available Database Of Labeled Chest Radiographs

Abstract: Chest radiography is an extremely powerful imaging modality, allowing for a detailed inspection of a patient's thorax, but requiring specialized training for proper interpretation. With the advent of high performance general purpose computer vision algorithms, the accurate automated analysis of chest radiographs is becoming increasingly of interest to researchers. However, a key challenge in the development of these techniques is the lack of sufficient data. Here we describe MIMIC-CXR-JPG v2.0.0, a large dataset of 377,110 chest x-rays associated with 227,827 imaging studies sourced from the Beth Israel Deaconess Medical Center between 2011 - 2016. Images are provided with 14 labels derived from two natural language processing tools applied to the corresponding free-text radiology reports. MIMIC-CXR-JPG is derived entirely from the MIMIC-CXR database, and aims to provide a convenient processed version of MIMIC-CXR, as well as to provide a standard reference for data splits and image labels. All images have been de-identified to protect patient privacy. The dataset is made freely available to facilitate and encourage a wide range of research in medical computer vision.


The softmax form of marginal and conditional probabilities in Equation #REFR suggests a way to model V : estimate both W uK (i)s and T uis using linear functional forms as in the conditional logit model with nest observables and item observables, respectively.

The likelihood in Equation (15) decomposes into a marginal probability of choosing nest K (i) and a conditional probability of choosing i given nest K (i) is chosen.

P(i | u, s) = exp T uis /λ K (i) ∑ j∈I K (i) exp T u js /λ K (i) conditional prob.

of item exp W uK (i)s + λ K (i) I uK (i)s ∑ K k=1 exp (W uks + λ k I uks )

of nest #OTHEREFR The inclusive value of nest k (for user u in session s), denoted as I uks , is defined as log ∑ j∈I k exp(T u js /λ k ), which is the expected utility from choosing the best alternative from nest k given Gumbel error term in utility.

#OTHEREFR A set of nests is a partition of item sets.

When the model requires the user to choose only one item across all partitions each time, these partitions are called nests.

When such a requirement is removed, the model allows users to choose an item from every partition, these partitions are referred to as categories.

#OTHEREFR Our nested logit setup can be constrasted with modeling multiple categories using the multinomial logit in torch-choice, where we model choice over multiple categories simultaneously, and the user chooses one item from each category simultaneously.

In the future we aim to support modeling choice over multiple categories where each category can include a two level nested logit.

Torch-Choice: A Pytorch Package For Large-Scale Choice Modelling With Python

Abstract: The torch-choice is an open-source library for flexible, fast choice modeling with Python and PyTorch. torch-choice provides a ChoiceDataset data structure to manage databases flexibly and memory-efficiently. The paper demonstrates constructing a ChoiceDataset from databases of various formats and functionalities of ChoiceDataset. The package implements two widely used models, namely the multinomial logit and nested logit models, and supports regularization during model estimation. The package incorporates the option to take advantage of GPUs for estimation, allowing it to scale to massive datasets while being computationally efficient. Models can be initialized using either R-style formula strings or Python dictionaries. We conclude with a comparison of the computational efficiencies of torch-choice and mlogit in R as (1) the number of observations increases, (2) the number of covariates increases, and (3) the expansion of item sets. Finally, we demonstrate the scalability of torch-choice on large-scale datasets. Keywords choice modelling • Python • PyTorch • large scale dataset • GPU acceleration Recent years have seen the proliferation of various tools and software packages that go a long way in making it easy to estimate the parameters of choice models. While each of these have their particular strengths, there are important limitations of these packages when they are used with the goal of building flexible, fast choice models which scale to massive data.

Counterfactual Inference For Consumer Choice Across Many Product Categories

Abstract: This paper proposes a method for estimating consumer preferences among discrete choices, where the consumer chooses at most one product in a category, but selects from multiple categories in parallel. The consumer's utility is additive in the different categories. Her preferences about product attributes as well as her price sensitivity vary across products and are in general correlated across products. We build on techniques from the machine learning literature on probabilistic models of matrix factorization, extending the methods to account for time-varying product attributes and products going out of stock. We evaluate the performance of the model using held-out data from weeks with price changes or out of stock products. We show that our model improves over traditional modeling approaches that consider each category in isolation. One source of the improvement is the ability of the model to accurately estimate heterogeneity in preferences (by pooling information across categories); another source of improvement is its ability to estimate the preferences of consumers who have rarely or never made a purchase in a given category in the training data. Using held-out data, we show that our model can accurately distinguish which consumers are most price sensitive to a given product. We consider counterfactuals such as personally targeted price discounts, showing that using a richer model such as the one we propose substantially increases the benefits of personalization in discounts.


These risk metrics should satisfy certain axioms in order to be well-defined as well as to enable practical use in robotic applications #REFR .

In #OTHEREFR , the authors learn a speed-map representation given semantic information while in #OTHEREFR , the authors use a self-supervised learning-based terrain roughness classifier.

In #OTHEREFR , the authors proposed a neural network architecture for learning the distribution of traversability costs arising from geometric risks using the pointcloud obtained from a LiDAR in a supervised manner.

In this paper, we build upon the traversability analyses of classical geometric methods, while using semantic information from a LiDAR and incorporating the uncertainty of these methods for risk-aware planning #OTHEREFR .

Risk can be incorporated into motion planning using a variety of different methods, including chance constraints #OTHEREFR , exponential utility functions #OTHEREFR , distributional robustness #OTHEREFR , and quantile regression #OTHEREFR .

Risk measures, often used in finance and operations research, provide a mapping from a random variable (usually the cost) to a real number.

Conditional value-at-risk (CVaR) is one such risk measure that has this desirable set of properties, and is a part of a class of risk metrics known as coherent risk measures #OTHEREFR .

Coherent risk measures have been used in a variety of decision making problems, especially Markov decision processes (MDPs) #OTHEREFR .

Coherent risk measures have been used in a MPC framework when the system model is uncertain #OTHEREFR , when the uncertainty is a result of measurement noise or moving obstacles #OTHEREFR , and when the uncertainty arises from both measurement and process noise #OTHEREFR .

Our work extends CVaR risk to a risk-based planning framework which utilizes different sources of traversability risk (such as collision risk, step risk, slippage risk, etc.) Morever, this paper introduces the first field-hardened and theoretically grounded approach to traversability assessment and risk-constrained planning using CVaR metrics.

Using CVaR to assess traversability risks allows us to dynamically tune the entire system's behavior -from aggressive to highly conservative -by changing a single value, the risk level.

Step: Stochastic Traversability Evaluation And Planning For Risk-Aware Off-Road Navigation; Results From The Darpa Subterranean Challenge

Abstract: Although autonomy has gained widespread usage in structured and controlled environments, robotic autonomy in unknown and off-road terrain remains a difficult problem. Extreme, off-road, and unstructured environments such as undeveloped wilderness, caves, rubble, and other post-disaster sites pose unique and challenging problems for autonomous navigation. Based on our participation in the DARPA Subterranean Challenge, we propose an approach to improve autonomous traversal of robots in subterranean environments that are perceptually degraded and completely unknown through a traversability and planning framework called STEP (Stochastic Traversability Evaluation and Planning). We present 1) rapid uncertainty-aware mapping and traversability evaluation, 2) tail risk assessment using the Conditional Value-at-Risk (CVaR), 3) efficient risk and constraint-aware kinodynamic motion planning using sequential quadratic programming-based (SQP) model predictive control (MPC), 4) fast recovery behaviors to account for unexpected scenarios that may cause failure, and 5) risk-based gait adaptation for quadrupedal robots. We illustrate and validate extensive results from our experiments on wheeled and legged robotic platforms in field studies at the Valentine Cave, CA (cave environment), Kentucky Underground, KY (mine environment), and Louisville Mega Cavern, KY (final competition site for the DARPA Subterranean Challenge with tunnel, urban, and cave environments).

How Should A Robot Assess Risk? Towards An Axiomatic Theory Of Risk In Robotics

Abstract: Endowing robots with the capability of assessing risk and making risk-aware decisions is widely considered a key step toward ensuring safety for robots operating under uncertainty. But, how should a robot quantify risk? A natural and common approach is to consider the framework whereby costs are assigned to stochastic outcomes–an assignment captured by a cost random variable. Quantifying risk then corresponds to evaluating a risk metric, i.e., a mapping from the cost random variable to a real number. Yet, the question of what constitutes a “good” risk metric has received little attention within the robotics community. The goal of this paper is to explore and partially address this question by advocating axioms that risk metrics in robotics applications should satisfy in order to be employed as rational assessments of risk. We provide instantiations of the class of risk metrics that satisfy these axioms (referred to as distortion risk metrics) and also discuss pitfalls of commonly used risk metrics in robotics that do not satisfy these axioms. Our hope is that the ideas presented here will lead to a foundational framework for quantifying risk (and hence safety) in robotics applications.


Planning a safe, comfortable, and efficient trajectory for a self-driving vehicle (SDV) in complex urban environments is a challenging and critical task in autonomous driving #REFR .

Unlike highway driving #OTHEREFR , urban driving requires handling various road geometries, such as roundabouts and intersections, while interacting with traffic lights, pedestrians, and other vehicles.

Conventional rule-based approaches #OTHEREFR have achieved some success in industry but require extensive human engineering to deal with diverse real-world scenarios.

Recent advances in deep learning techniques have motivated researchers #OTHEREFR to employ neural networks to model complex driving policies.

Imitation learning (IL) from human drivers' demonstrations is a promising solution for learning these policies, as experienced drivers can handle even the most difficult situations, and their driving data can be easily collected at scale.

The simplest IL algorithm is the behavior cloning (BC) method, which has wide applications in autonomous driving #OTHEREFR .

Ccil: Context-Conditioned Imitation Learning For Urban Driving

Abstract: Imitation learning holds great promise for addressing the complex task of autonomous urban driving, as experienced human drivers can navigate highly challenging scenarios with ease. While behavior cloning is a widely used imitation learning approach in autonomous driving due to its exemption from risky online interactions, it suffers from the covariate shift issue. To address this limitation, we propose a context-conditioned imitation learning approach that employs a policy to map the context state into the ego vehicle's future trajectory, rather than relying on the traditional formulation of both ego and context states to predict the ego action. Additionally, to reduce the implicit ego information in the coordinate system, we design an ego-perturbed goal-oriented coordinate system. The origin of this coordinate system is the ego vehicle's position plus a zero mean Gaussian perturbation, and the x-axis direction points towards its goal position. Our experiments on the real-world large-scale Lyft and nuPlan datasets show that our method significantly outperforms state-of-the-art approaches.

A Survey Of Autonomous Driving: Common Practices And Emerging Technologies

Abstract: Automated driving systems (ADSs) promise a safe, comfortable and efficient driving experience. However, fatalities involving vehicles equipped with ADSs are on the rise. The full potential of ADSs cannot be realized unless the robustness of state-of-the-art improved further. This paper discusses unsolved problems and surveys the technical aspect of automated driving. Studies regarding present challenges, high-level system architectures, emerging methodologies and core functions: localization, mapping, perception, planning, and human machine interface, were thoroughly reviewed. Furthermore, the state-of-the-art was implemented on our own platform and various algorithms were compared in a real-world driving setting. The paper concludes with an overview of available datasets and tools for ADS development.


The results of these methods follow the findings of #REFR for camera 2D object detectors.

Their results indicate that synthetic data can replace real-world data to an extent, but proportionately more synthetic than real-world data is required.

Moreover, synthetic data can be successfully used for network pre-training leading to better performance compared to training on real-world data only #OTHEREFR .

Similar research to quantify the domain shift has also been conducted for LiDAR point clouds with 3D object detectors. Dworak et al.

#OTHEREFR train three object detectors with data generated in the simulation environment CARLA and evaluate their performance on the real-world KITTI dataset #OTHEREFR .

Although networks achieve an mAP of up to 87 % when trained and evaluated on CARLA ("sim-to-sim"), the best network only reaches an mAP of 19 % if trained on CARLA and tested on KITTI ("sim-to-real"). The authors also experimented with fine-tuning and mixed-data training.

To evaluate their methods for synthetic LiDAR data generation, Fang et al. #OTHEREFR and Manivasagam et al.

#OTHEREFR compared their generated data with the CARLA and KITTI datasets by training object detectors and evaluating on KITTI.

Both works highlight the sim-to-real domain shift between the CARLA and KITTI datasets.

#OTHEREFR conduct similar LiDAR sim-to-real comparison experiments but focus on semantic segmentation instead of object detection.

Quantifying The Lidar Sim-To-Real Domain Shift: A Detailed Investigation Using Object Detectors And Analyzing Point Clouds At Target-Level

Abstract: LiDAR object detection algorithms based on neural networks for autonomous driving require large amounts of data for training, validation, and testing. As real-world data collection and labeling are time-consuming and expensive, simulation-based synthetic data generation is a viable alternative. However, using simulated data for the training of neural networks leads to a domain shift of training and testing data due to differences in scenes, scenarios, and distributions. In this work, we quantify the sim-to-real domain shift by means of LiDAR object detectors trained with a new scenario-identical real-world and simulated dataset. In addition, we answer the questions of how well the simulated data resembles the real-world data and how well object detectors trained on simulated data perform on real-world data. Further, we analyze point clouds at the target-level by comparing real-world and simulated point clouds within the 3D bounding boxes of the targets. Our experiments show that a significant sim-to-real domain shift exists even for our scenario-identical datasets. This domain shift amounts to an average precision reduction of around 14 % for object detectors trained with simulated data. Additional experiments reveal that this domain shift can be lowered by introducing a simple noise model in simulation. We further show that a simple downsampling method to model real-world physics does not influence the performance of the object detectors.

How Much Real Data Do We Actually Need: Analyzing Object Detection Performance Using Synthetic And Real Data

Abstract: In recent years, deep learning models have resulted in a huge amount of progress in various areas, including computer vision. By nature, the supervised training of deep models requires a large amount of data to be available. This ideal case is usually not tractable as the data annotation is a tremendously exhausting and costly task to perform. An alternative is to use synthetic data. In this paper, we take a comprehensive look into the effects of replacing real data with synthetic data. We further analyze the effects of having a limited amount of real data. We use multiple synthetic and real datasets along with a simulation tool to create large amounts of cheaply annotated synthetic data. We analyze the domain similarity of each of these datasets. We provide insights about designing a methodological procedure for training deep networks using these datasets.


We follow the official split of 1,200 and 670 as the train and test set, respectively. ActivityNet Captions #REFR contains 20,000 YouTube videos.

MSRVTT #OTHEREFR contains 10,000 YouTube videos, each with 20 text descriptions.

We follow the 1k-A split #OTHEREFR with 9,000 videos for training and 1,000 for testing. LSMDC #OTHEREFR contains 118,081 video clips from 202 movies.

We follow the split of #OTHEREFR with 1,000 videos for testing. MSVD #OTHEREFR contains 1,970 videos.

We report results on the "val1" split of 10,009 and 4,917 as the train and test set. DiDeMo #OTHEREFR contains 10,464 videos annotated 40,543 text descriptions.

We follow the training and evaluation protocol in #OTHEREFR .

We choose Recall at rank K (R@K), the Sum of Recall at rank {1, 5, 10} (Rsum), Median Rank (MdR), and mean rank (MnR) to evaluate the retrieval performance.

Diffusionret: Generative Text-Video Retrieval With Diffusion Model

Abstract: Existing text-video retrieval solutions are, in essence, discriminant models focused on maximizing the conditional likelihood, i.e., p(candidates|query). While straightforward, this de facto paradigm overlooks the underlying data distribution p(query), which makes it challenging to identify outof-distribution data. To address this limitation, we creatively tackle this task from a generative viewpoint and model the correlation between the text and the video as their joint probability p(candidates, query). This is accomplished through a diffusion-based text-video retrieval framework (Diffusion-Ret), which models the retrieval task as a process of gradually generating joint distribution from noise. During training, DiffusionRet is optimized from both the generation and discrimination perspectives, with the generator being optimized by generation loss and the feature extractor trained with contrastive loss. In this way, DiffusionRet cleverly leverages the strengths of both generative and discriminative methods. Extensive experiments on five commonly used text-video retrieval benchmarks, including MSRVTT, LSMDC, MSVD, ActivityNet Captions, and DiDeMo, with superior performances, justify the efficacy of our method. More encouragingly, without any modification, DiffusionRet even performs well in out-domain retrieval settings. We believe this work brings fundamental insights into the related fields. Code will be available at https://github.com/jpthu17/DiffusionRet.

Dense-Captioning Events In Videos

Abstract: Most natural videos contain numerous events. For example, in a video of a "man playing a piano", the video might also contain "another man dancing" or "a crowd clapping". We introduce the task of dense-captioning events, which involves both detecting and describing events in a video. We propose a new model that is able to identify all events in a single pass of the video while simultaneously describing the detected events with natural language. Our model introduces a variant of an existing proposal module that is designed to capture both short as well as long events that span minutes. To capture the dependencies between the events in a video, our model introduces a new captioning module that uses contextual information from past and future events to jointly describe all events. We also introduce ActivityNet Captions, a large-scale benchmark for dense-captioning events. ActivityNet Captions contains 20k videos amounting to 849 video hours with 100k total descriptions, each with it's unique start and end time. Finally, we report performances of our model for dense-captioning events, video retrieval and localization.

20,000 YouTube videos
ActivityNet Captions


Efficient Search Space Quality Evaluation

For example, it costs more than 10 days to train a supernet on 8 V100 GPUs #REFR .

We now address the efficiency challenge caused by Q-T score evaluation.

The most accurate evaluation is to get accuracy by training a supernet (search space) from scratch and measure latency on target device.

However, it's impractical to conduct large-scale search due to the prohibitive cost.

To reduce the cost, we build an accurate INT8 latency predictor by nn-Meter #OTHEREFR , then we introduce block-wise quantization scheme.

Block-wise knowledge distillation (BKD) is firstly proposed in DNA #OTHEREFR and then further improved in DONNA #OTHEREFR .

It originally uses block-wise representation of existing models (teacher) to supervise a corresponding student model block.

In our work, we extend BKD to supervise the training of all elastic stages (each contains a large amount of blocks).

Since a stage size is much smaller than search space size, the training time is greatly reduced. Fig. 4 illustrates the BKD process.

Spaceevo: Hardware-Friendly Search Space Design For Efficient Int8 Inference

Abstract: The combination of Neural Architecture Search (NAS) and quantization has proven successful in automatically designing low-FLOPs INT8 quantized neural networks (QNN). However, directly applying NAS to design accurate QNN models that achieve low latency on real-world devices leads to inferior performance. In this work, we find that the poor INT8 latency is due to the quantization-unfriendly issue: the operator and configuration (e.g., channel width) choices in prior art search spaces lead to diverse quantization efficiency and can slow down the INT8 inference speed. To address this challenge, we propose SpaceEvo, an automatic method for designing a dedicated, quantizationfriendly search space for each target hardware. The key idea of SpaceEvo is to automatically search hardwarepreferred operators and configurations to construct the search space, guided by a metric called Q-T score to quantify how quantization-friendly a candidate search space is. We further train a quantized-for-all supernet over our discovered search space, enabling the searched models to be directly deployed without extra retraining or quantization. Our discovered models establish new SOTA INT8 quantized accuracy under various latency constraints, achieving up to 10.1% accuracy improvement on ImageNet than prior art CNNs under the same latency. Extensive experiments on diverse edge devices demonstrate that SpaceEvo consistently outperforms existing manually-designed search spaces with up to 2.5× faster speed while achieving the same accuracy.

Bignas: Scaling Up Neural Architecture Search With Big Single-Stage Models

Abstract: Neural architecture search (NAS) has shown promising results discovering models that are both accurate and fast. For NAS, training a one-shot model has become a popular strategy to rank the relative quality of different architectures (child models) using a single set of shared weights. However, while one-shot model weights can effectively rank different network architectures, the absolute accuracies from these shared weights are typically far below those obtained from stand-alone training. To compensate, existing methods assume that the weights must be retrained, finetuned, or otherwise post-processed after the search is completed. These steps significantly increase the compute requirements and complexity of the architecture search and model deployment. In this work, we propose BigNAS, an approach that challenges the conventional wisdom that post-processing of the weights is necessary to get good prediction accuracies. Without extra retraining or post-processing steps, we are able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs. Our discovered model family, BigNASModels, achieve top-1 accuracies ranging from 76.5% to 80.9%, surpassing state-of-the-art models in this range including EfficientNets and Once-for-All networks without extra retraining or post-processing. We present ablative study and analysis to further understand the proposed BigNASModels.


Following #REFR and using the implementation of [Nog ] , we utilize Bayesian optimization to find weights that lead to better performance of the state-independent agents in the band environment.

In addition to the actions, the maximum number of actions that the agent is allowed to perform before the episode counts as a loss is a very important hyperparameter which we also want to optimize.

Hence, we have 3 hyperparameters: the two independent weights w ∈ R 2 and the maximum number of actions per episode, which we take to be in [10, 10000].

Then, for any value of these hyperparameters, one may play many games in the band environment and collect rewards, which together make up a set of observations {w n , R(w n )}.

To optimize the expected value of the reward, we wish to model the reward via a stochastic process.

A convenient choice that facilitates optimization via Bayesian inference is to model R(w) as a draw from a Gaussian process (GP), meaning that for any finite set of weights {w i ∈ R 3 }, the vector of random variables R(w i ) is distributed according to a multivariate Gaussian.

It is assumed that R(w) is drawn from a GP prior with a chosen mean and covariance.

Beginning with the prior, the algorithm iteratively chooses the next weight to observe w next , plays a game at w next and records the reward, computes the Bayesian posterior given the new data point {w next , R(w next )}, and records the best weight seen so far w best .

At each step of the optimization, the Bayesian posterior is Gaussian and depends on a collection of game results {w n , R(w n )} with predictive mean function µ(w, {w n , R(w n )}) and variance σ 2 (w, {w n , R(w n )}). At any given step a useful quantity is

γ(w) = R(w best ) − µ(w; {w n , R(w n )}) σ(w; {w n , R(w n )}) ,

which measures the deviation of the mean prediction at w from the current best and normalizes by the variance. More specifically, the next weight is chosen as

Searching For Ribbons With Machine Learning

Abstract: We apply Bayesian optimization and reinforcement learning to a problem in topology: the question of when a knot bounds a ribbon disk. This question is relevant in an approach to disproving the four-dimensional smooth Poincaré conjecture; using our programs, we rule out many potential counterexamples to the conjecture. We also show that the programs are successful in detecting many ribbon knots in the range of up to 70 crossings.

Practical Bayesian Optimization Of Machine Learning Algorithms

Abstract: The use of machine learning algorithms frequently involves careful tuning of learning parameters and model hyperparameters. Unfortunately, this tuning is often a "black art" requiring expert experience, rules of thumb, or sometimes brute-force search. There is therefore great appeal for automatic approaches that can optimize the performance of any given learning algorithm to the problem at hand. In this work, we consider this problem through the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). We show that certain choices for the nature of the GP, such as the type of kernel and the treatment of its hyperparameters, can play a crucial role in obtaining a good optimizer that can achieve expertlevel performance. We describe new algorithms that take into account the variable cost (duration) of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks.


IL utilizes the same loss in Eq.13 while RL is implemented based on the Asynchronous Advantage Actor-Critic (A3C) algorithm #REFR .

We employ an additional Instruction Trajectory Matching (ITM) task following previous efforts #OTHEREFR during pretraining, which predicts whether a pair of instruction and trajectory is aligned.

The three tasks IF (Instruction Following), IG (Instruction Generation) and ITM (Instruction Trajectory Matching) are sampled with a ratio IG:IF:ITM=4:1:2.

We present the pseudo-code of the pretraining procedure in Algorithm 1 (ITM is omitted for simplicity).

For finetuning, the instruction following task is optimized with Reinforcement Learning (RL) and Imitation Learning (IL).

During finetuning, the sampling ratio for IG and IF is set to IG:IF=2:5; the ITM task is abandoned.

Following the common practice #OTHEREFR , we concatenate the object features with the panoramic features and add an object grounding loss for the instruction following task on REVERIE #OTHEREFR .

The detailed architecture of LANA is shown in Table 7 .

Lana: A Language-Capable Navigator For Instruction Following And Generation

Abstract: Recently, visual-language navigation (VLN)-entailing robot agents to follow navigation instructions-has shown great advance. However, existing literature put most emphasis on interpreting instructions into actions, only delivering "dumb" wayfinding agents. In this article, we devise LANA, a language-capable navigation agent which is able to not only execute human-written navigation commands, but also provide route descriptions to humans. This is achieved by simultaneously learning instruction following and generation with only one single model. More specifically, two encoders, respectively for route and language encoding, are built and shared by two decoders, respectively for action prediction and instruction generation, so as to exploit cross-task knowledge and capture task-specific characteristics. Throughout pretraining and fine-tuning, both instruction following and generation are set as optimization objectives. We empirically verify that, compared with recent advanced task-specific solutions, LANA attains better performances on both instruction following and route description, with nearly half complexity. In addition, endowed with language generation capability, LANA can explain to human its behaviours and assist human's wayfinding. This work is expected to foster future efforts towards building more trustworthy and sociallyintelligent navigation robots.

Asynchronous Methods For Deep Reinforcement Learning

Abstract: We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.


The concatenation of character and word level embeddings have been employed along with bidirectional LSTMs #REFR .

Recently, employing regular expressions #OTHEREFR , the prototypical network #OTHEREFR , and extensions to CRFs #OTHEREFR have been proposed for the few-shot slot filling task.

Furthermore, several works #OTHEREFR have leveraged model agnostic meta-learning to jointly model the slot filling and intent classification tasks.

In a similar line of work, in the zero-shot setting, no in-domain labeled training data is required, which has shown encouraging results to eliminate the need for manual labeling.

Using LSTM-based models along with the natural language descriptions of the slots has been shown to overcome the challenge of different names of the slot for apparently similar slots (e.g., city vs destination in Travel domain) for the zero-shot adaptation of slot filling task #OTHEREFR .

Similarly, attention mechanisms have further improved the performance #OTHEREFR .

Moreover, the coarse-to-fine approach in #OTHEREFR was proposed to improve the robustness of the zero-shot slot filling task.

In spite of promising results, there is a huge performance gap between zero-shot learning methods and supervised learning approaches that limit their applicability.

In this work, we employ a zero-shot slot filling model to automatically generate a set of labels in the target domain and further improve the performance of the slot filling task in an open-domain setting. Unsupervised and Weak Supervision Slot Filling.

The other line of work to eliminate the need for manual labeling is weak supervision and unsupervised learning.

Unsupervised learning methods #OTHEREFR have used mining to build latent variable models #OTHEREFR .

Toward Open-Domain Slot Filling Via Self-Supervised Co-Training

Abstract: Slot filling is one of the critical tasks in modern conversational systems. The majority of existing literature employs supervised learning methods, which require labeled training data for each new domain. Zero-shot learning and weak supervision approaches, among others, have shown promise as alternatives to manual labeling. Nonetheless, these learning paradigms are significantly inferior to supervised learning approaches in terms of performance. To minimize this performance gap and demonstrate the possibility of open-domain slot filling, we propose a Self-supervised Co-training framework, called SCot, that requires zero in-domain manually labeled training examples and works in three phases. Phase one acquires two sets of complementary pseudo labels automatically. Phase two leverages the power of the pre-trained language model BERT, by adapting it for the slot filling task using these sets of pseudo labels. In phase three, we introduce a self-supervised cotraining mechanism, where both models automatically select highconfidence soft labels to further improve the performance of the other in an iterative fashion. Our thorough evaluations show that SCot outperforms state-of-the-art models by 45.57% and 37.56% on SGD and MultiWoZ datasets, respectively. Moreover, our proposed framework SCot achieves comparable performance when compared to state-of-the-art fully supervised models.

Robust Zero-Shot Cross-Domain Slot Filling With Example Values

Abstract: Task-oriented dialog systems increasingly rely on deep learning-based slot filling models, usually needing extensive labeled training data for target domains. Often, however, little to no target domain training data may be available, or the training and target domain schemas may be misaligned, as is common for web forms on similar websites. Prior zero-shot slot filling models use slot descriptions to learn concepts, but are not robust to misaligned schemas. We propose utilizing both the slot description and a small number of examples of slot values, which may be easily available, to learn semantic representations of slots which are transferable across domains and robust to misaligned schemas. Our approach outperforms state-of-the-art models on two multi-domain datasets, especially in the low-data setting.


Experiments demonstrate that the accuracy drop of BCT is less than 0.90% on most GLUE #REFR datasets without retraining.

Compressed transformer-based models have faster calculation, less memory footprint, and lower bandwidth requirements.

In this case, several compression methods are proposed to compress transformer-based models with an acceptable loss of accuracy.

However, most of them need to spend much time retraining to fine-tune the parameters.

In this paper, we investigate a framework of blockwise compression of transformers without retraining through algorithm/hardware codesign, denoted as BCT.

Unlike previous compression frameworks, BCT compresses the whole transformer, including embedding, matrix multiplication, GELU #OTHEREFR , Softmax, layer normalization #OTHEREFR , and all the intermediate results by block. Figure 1 shows the framework of BCT.

The main contributions of this paper can be summa-rized as follows: (i) We propose an efficient blockwise compression framework for transformer-based models without retraining.

(ii) We provide a concrete well-designed methodology for each transformer layer, including matrix multiplication and nonlinear operations.

(iii) Without retraining, we evaluate BCT on several GLUE datasets, and BCT achieves a less than 0.90% accuracy drop in most tasks.

In conclusion, BCT is suitable for deploying transformer-based models and does not require laborious retraining.

Blockwise Compression Of Transformer-Based Models Without Retraining

Abstract: Transformer-based models, represented by GPT-3, ChatGPT, and GPT-4, have recently attracted increasing interest, research enthusiasm, and business demand. However, their massive computation resources and huge memory footprint are inevitable challenges. To tackle this issue, we propose BCT, a framework of blockwise compression for transformers without retraining, to lower deployment thresholds. BCT achieves more fine-grained compression of the whole transformer, including embedding, matrix multiplication, GELU, Softmax, layer normalization, and all the intermediate results. As a case, we compress an efficient model with BCT and evaluate it on several General Language Understanding Evaluation (GLUE) datasets. The results show that BCT can achieve a less than 0.90% accuracy drop in most tasks.

Glue: A Multi-Task Benchmark And Analysis Platform For Natural Language Understanding

Abstract: For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one specific task or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation benchmark (GLUE), a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks. GLUE is model-agnostic, but it incentivizes sharing knowledge across tasks because certain tasks have very limited training data. We further provide a hand-crafted diagnostic test suite that enables detailed linguistic analysis of NLU models. We evaluate baselines based on current methods for multi-task and transfer learning and find that they do not immediately give substantial improvements over the aggregate performance of training a separate model per task, indicating room for improvement in developing general and robust NLU systems.


Description And Implementation Of Xai Explanation Types

As first research endeavors target the end-user of an XAI-based DSS, these interdisciplinary research outcomes must be incorporated into practical applications to design valuable explanations for end-user #REFR .

In practice, this XAI jungle is exacerbated by developers primarily designing these XAI implementations for developers without prioritizing the actual end-users (van der Waa et al., 2021).

Following the explanation theory of #OTHEREFR , a useful explanation is defined as a social and cognitive process of knowledge transfer from an XAI-based DSS to the end-user.

Thus, if an explanation is perceived as inadequate, contradicts an end-user's mental model, or does not appeal to their emotions or beliefs, trust issues can occur and user acceptance may be reduced, leading to algorithm aversion #OTHEREFR .

Following recent IS research, a mental model defines any type of mental representation used to encode beliefs, facts, and knowledge when conceptualizing cognitive processes #OTHEREFR .

In this sense, the extent to which these explanation types affect end-users' cognitive load is unknown, which is an essential factor in the design and development of appropriate XAI implementations #OTHEREFR .

Impact Of Explainable Ai On Cognitive Load: Insights From An Empirical Study

Abstract: While the emerging research field of explainable artificial intelligence (XAI) claims to address the lack of explainability in high-performance machine learning models, in practice, XAI targets developers rather than actual end-users. Unsurprisingly, end-users are often unwilling to use XAI-based decision support systems. Similarly, there is limited interdisciplinary research on end-users' behavior during XAI explanations usage, rendering it unknown how explanations may impact cognitive load and further affect end-user performance. Therefore, we conducted an empirical study with 271 prospective physicians, measuring their cognitive load, task performance, and task time for distinct implementationindependent XAI explanation types using a COVID-19 use case. We found that these explanation types strongly influence end-users' cognitive load, task performance, and task time. Further, we contextualized a mental efficiency metric, ranking local XAI explanation types best, to provide recommendations for future applications and implications for sociotechnical XAI research.

Explainable Artificial Intelligence (Xai): Concepts, Taxonomies, Opportunities And Challenges Toward Responsible Ai

Abstract: In the last few years, Artificial Intelligence (AI) has achieved a notable momentum that, if harnessed appropriately, may deliver the best of expectations over many application sectors across the field. For this to occur shortly in Machine Learning, the entire community stands in front of the barrier of explainability, an inherent problem of the latest techniques brought by sub-symbolism (e.g. ensembles or Deep Neural Networks) that were not present in the last hype of AI (namely, expert systems and rule based models). Paradigms underlying this problem fall within the so-called eXplainable AI (XAI) field, which is widely acknowledged as a crucial feature for the practical deployment of AI models. The overview presented in this article examines the existing literature and contributions already done in the field of XAI, including a prospect toward what is yet to be reached. For this purpose we summarize previous efforts made to define explainability in Machine Learning, establishing a novel definition of explainable Machine Learning that covers such prior conceptual propositions with a major focus on the audience for which the explainability is sought. Departing from this definition, we propose and discuss about a taxonomy of recent contributions related to the explainability of different Machine Learning models, including those aimed at explaining Deep Learning methods for which a second dedicated taxonomy is built and examined in detail. This critical literature analysis serves as the motivating background for a series of challenges faced by XAI, such as the interesting crossroads of data fusion and explainability. Our prospects lead toward the concept of Responsible Artificial Intelligence, namely, a methodology for the large-scale implementation of AI methods in real organizations with fairness, model explainability and accountability at its core. Our ultimate goal is to provide newcomers to the field of XAI with a thorough taxonomy that can serve as reference material in order to stimulate future research advances, but also to encourage experts and professionals from other disciplines to embrace the benefits of AI in their activity sectors, without any prior bias for its lack of interpretability.


These learning-based algorithms are known to be vulnerable to input perturbations #REFR -small (possibly adversarial) changes in their input can lead to large variations in their output.

Motivation and Problem Statement: Neural networks have become increasingly popular in control systems in recent years due to their relative ease of in-the-loop computation.

As such, runtime verification of the safety and performance of neural network controlled systems is essential in safety-critical applications.

This task is generally challenging due to the nonlinear and large-scale structure of the neural networks and their interconnection with nonlinear dynamics #OTHEREFR .

A basic ingredient for verifying control systems is the ability to overapproximate the set of reachable states from a given set of initial conditions, possibly in the presence of disturbances.

If, for example, this overapproximation avoids obstacles or reaches a goal region, then the system certifiably satisfies the corresponding safety or performance criteria.

Recently, several promising reachability-based methods have been proposed for verifying stand-alone neural networks or feedback systems with neural networks in the control loop, however, these methods either suffer from large computational complexity, large over-approximation error, or lack of generality.

Contraction-Guided Adaptive Partitioning For Reachability Analysis Of Neural Network Controlled Systems

Abstract: In this paper, we present a contraction-guided adaptive partitioning algorithm for improving interval-valued robust reachable set estimates in a nonlinear feedback loop with a neural network controller and disturbances. Based on an estimate of the contraction rate of over-approximated intervals, the algorithm chooses when and where to partition. Then, by leveraging a decoupling of the neural network verification step and reachability partitioning layers, the algorithm can provide accuracy improvements for little computational cost. This approach is applicable with any sufficiently accurate open-loop interval-valued reachability estimation technique and any method for bounding the input-output behavior of a neural network. Using contraction-based robustness analysis, we provide guarantees of the algorithm's performance with mixed monotone reachability. Finally, we demonstrate the algorithm's performance through several numerical simulations and compare it with existing methods in the literature. In particular, we report a sizable improvement in the accuracy of reachable set estimation in a fraction of the runtime as compared to state-of-the-art methods.

Intriguing Properties Of Neural Networks

Abstract: Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. ::: First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. ::: Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. We can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.

input perturbations -small
possibly adversarial) changes


In comparison, the SR results of SMORE #REFR and our proposed model recover more image details.

Although our proposed framework yields significant improvements on the simulated HCP-1200 data #OTHEREFR compared with the three baselines, it is more important to show how well the models perform on the real acquired anisotropic MR volumes.

Therefore, we also conducted a comparison experiment on two real measured anisotropic adult brain volumes (details in Table 1 ).

Here we only report the qualitative results since the isotropic GT HR images are not acquired.

As shown in Figure 5 , the SR results of cubic interpolation suffer from severe blocking artifacts due to its poor de-aliasing ability.

While IREM #OTHEREFR produces sharp but overly smooth MR images, where many high-frequency details are lost.

Moreover, the SR results of our model are better in terms of anatomical consistency.

Self-Supervised Arbitrary Scale Super-Resolution Framework For Anisotropic Mri

Abstract: In this paper, we propose an efficient self-supervised arbitraryscale super-resolution (SR) framework to reconstruct isotropic magnetic resonance (MR) images from anisotropic MRI inputs without involving external training data. The proposed framework builds a training dataset using "in-the-wild" anisotropic MR volumes with arbitrary image resolution. We then formulate the 3D volume SR task as a SR problem for 2D image slices. The anisotropic volume's high-resolution (HR) plane is used to build the HR-LR image pairs for model training. We further adapt the implicit neural representation (INR) network to implement the 2D arbitrary-scale image SR model. Finally, we leverage the welltrained proposed model to up-sample the 2D LR plane extracted from the anisotropic MR volumes to their HR views. The isotropic MR volumes thus can be reconstructed by stacking and averaging the generated HR slices. Our proposed framework has two major advantages: (1) It only involves the arbitrary-resolution anisotropic MR volumes, which greatly improves the model practicality in real MR imaging scenarios (e.g., clinical brain image acquisition); (2) The INR-based SR model enables arbitrary-scale image SR from the arbitrary-resolution input image, which significantly improves model training efficiency. We perform experiments on a simulated public adult brain dataset and a real collected 7T brain dataset. The results indicate that our current framework greatly outperforms two well-known self-supervised models for anisotropic MR image SR tasks.

Self Super-Resolution For Magnetic Resonance Images Using Deep Networks

Abstract: High resolution magnetic resonance (MR) imaging (MRI) is desirable in many clinical applications; however, there is a trade-off between resolution, speed of acquisition, and noise. It is common for MR images to have worse through-plane resolution (slice thickness) than in-plane resolution. In these MRI images, high frequency information in the through-plane direction is not acquired, and cannot be resolved through interpolation. To address this issue, super-resolution methods have been developed to enhance spatial resolution. As an ill-posed problem, state-of-the-art super-resolution methods rely on the presence of external/training atlases to learn the transform from low resolution (LR) images to high resolution (HR) images. For several reasons, such HR atlas images are often not available for MRI sequences. This paper presents a self super-resolution (SSR) algorithm, which does not use any external atlas images, yet can still resolve HR images only reliant on the acquired LR image. We use a blurred version of the input image to create training data for a state-of-the-art super-resolution deep network. The trained network is applied to the original input image to estimate the HR image. Our SSR result shows a significant improvement on through-plane resolution compared to competing SSR methods.


CorGAN #REFR is a novel work that utilizes convolutional neural networks (CNN) instead of multilayer perceptrons (MLP) to model EHR data.

Considering the obstacle of directly using GAN on generating high-dimensional binary EHR vectors, medGAN alters to a low-dimensional dense space for generation by taking advantage of pre-trained auto-encoders.

The model generates a dense EHR vector and then recovers a synthetic EHR with decoders.

medBGAN and medWGAN #OTHEREFR are two improved GAN models for realistic EHR synthesis.

medGAN is based on the conventional GAN model for EHR synthesis, and such a model is prone to mode collapse where GAN models may fail to learn the distribution of real-world data.

medBGAN and medWGAN integrate Boundary-seeking GAN (BGAN) #OTHEREFR and Wasserstein GAN (WGAN-GP) #OTHEREFR with gradient penalty respectively to improve the performance of medGAN and stabilize model training.

Specifically, they use CNN to model the autoencoder and the generative network.

They empirically elucidate through experiments that CNN can perform better than the MLP in this task.

#OTHEREFR is proposed to further refine the GAN models from several perspectives.

To avoid model collapse, the authors take advantage of WGAN.

The most prominent feature of EMR-WGAN is that it is directly trained on the discrete EHR data, while the previous works universally use an autoencoder to first transform the raw EHR data into low-dimensional dense space.

Ehrdiff : Exploring Realistic Ehr Synthesis With Diffusion Models

Abstract: Electronic health records (EHR) contain vast biomedical knowledge and are rich resources for developing precise medicine systems. However, due to privacy concerns, there are limited high-quality EHR data accessible to researchers hence hindering the advancement of methodologies. Recent research has explored using generative modelling methods to synthesize realistic EHR data, and most proposed methods are based on the generative adversarial network (GAN) and its variants for EHR synthesis. Although GAN-style methods achieved state-of-the-art performance in generating high-quality EHR data, such methods are hard to train and prone to mode collapse. Diffusion models are recently proposed generative modelling methods and set cutting-edge performance in image generation. The performance of diffusion models in realistic EHR synthesis is rarely explored. In this work, we explore whether the superior performance of diffusion models can translate to the domain of EHR synthesis and propose a novel EHR synthesis method named EHRDIFF. Through comprehensive experiments, EHRDIFF achieves new state-of-the-art performance for the quality of synthetic EHR data and can better protect private information in real training EHRs in the meanwhile. The code will be released soon in https://github.com/sczzz3/ehrdiff.

Corgan: Correlation-Capturing Convolutional Generative Adversarial Networks For Generating Synthetic Healthcare Records

Abstract: Deep learning models have demonstrated high-quality performance in areas such as image classification and speech processing. However, creating a deep learning model using electronic health record (EHR) data, requires addressing particular privacy challenges that are unique to researchers in this domain. This matter focuses attention on generating realistic synthetic data while ensuring privacy. In this paper, we propose a novel framework called correlation-capturing Generative Adversarial Network (corGAN), to generate synthetic healthcare records. In corGAN we utilize Convolutional Neural Networks to capture the correlations between adjacent medical features in the data representation space by combining Convolutional Generative Adversarial Networks and Convolutional Autoencoders. To demonstrate the model fidelity, we show that corGAN generates synthetic data with performance similar to that of real data in various Machine Learning settings such as classification and prediction. We also give a privacy assessment and report on statistical analysis regarding realistic characteristics of the synthetic data. The software of this work is open-source and is available at: this https URL.


Model-free RL has also led to significant progress in dexterous manipulation, starting with OpenAI's demonstration of finger-gaiting and finger-pivoting #REFR , trained in simulation and translated to real hardware.

#OTHEREFR build a map of valid grasps and use search methods to generate gaits for planar reorientation, while Han et al.

#OTHEREFR consider finger-gaiting of a sphere and identify the non-holonomic nature of the problem.

Some methods have also considered RRT for finger-gaiting in-hand manipulation #OTHEREFR , but limited to simulation for a spherical object. More recently, Morgan et al.

demonstrate robust finger-gaiting for object reorientation using actor-critic reinforcement learning #OTHEREFR and multi-modal motion planning #OTHEREFR , both in conjunction with a compliant, highly underactuated hand designed explicitly for this task. Bhatt et al.

#OTHEREFR also demonstrate robust fingergaiting finger-pivoting manipulation with a soft compliant hand, but these skills were not autonomously learned but rather hand-designed and executed in an open-loop fashion.

However, this approach uses extensive extrinsic sensing infeasible outside the lab, and relies on support surfaces such as the palm underneath the object. Khandate et al.

#OTHEREFR show dexterous finger-gaiting and fingerpivoting skills using only precision fingertip grasps to enable both palm-up and palm-down operation, but only on a range of simple convex shapes and in a simulated environment. Makoviychuk et al.

#OTHEREFR showed that GPU physics could be used to accelerate learning skills similar to OpenAI's. Allshire et al.

#OTHEREFR used extensive domain randomization and simto-real transfer to re-orient a cube but used table top as an external support surface. Chen et al.

#OTHEREFR demonstrated inhand re-orientation for a wide range of objects under palm-up and palm-down orientations of the hand with extrinsic sensing providing dense object feedback. Qi et al.

Sampling-Based Exploration For Reinforcement Learning Of Dexterous Manipulation

Abstract: In this paper, we present a novel method for achieving dexterous manipulation of complex objects, while simultaneously securing the object without the use of passive support surfaces. We posit that a key difficulty for training such policies in a Reinforcement Learning framework is the difficulty of exploring the problem state space, as the accessible regions of this space form a complex structure along manifolds of a high-dimensional space. To address this challenge, we use two versions of the non-holonomic Rapidly-Exploring Random Trees algorithm; one version is more general, but requires explicit use of the environment's transition function, while the second version uses manipulation-specific kinematic constraints to attain better sample efficiency. In both cases, we use states found via sampling-based exploration to generate reset distributions that enable training control policies under full dynamic constraints via model-free Reinforcement Learning. We show that these policies are effective at manipulation problems of higher difficulty than previously shown, and also transfer effectively to real robots. These videos can be found on the project website: sbrl.cs.columbia.edu

Solving Rubik'S Cube With A Robot Hand

Abstract: We demonstrate that models trained only in simulation can be used to solve a manipulation problem of unprecedented complexity on a real robot. This is made possible by two key components: a novel algorithm, which we call automatic domain randomization (ADR) and a robot platform built for machine learning. ADR automatically generates a distribution over randomized environments of ever-increasing difficulty. Control policies and vision state estimators trained with ADR exhibit vastly improved sim2real transfer. For control policies, memory-augmented models trained on an ADR-generated distribution of environments show clear signs of emergent meta-learning at test time. The combination of ADR with our custom robot platform allows us to solve a Rubik's cube with a humanoid robot hand, which involves both control and state estimation problems. Videos summarizing our results are available: this https URL


Related Work Dnn-Based Image Super-Resolution

For example, ESPCN #REFR invents the pixel-shuffle layer to obtain the HR output given the LR input.

USRNet #OTHEREFR proposes an end-to-end trainable unfolding network that leverages both learning-based methods and model-based methods.

Therefore, they can handle the SISR problem with different scaling factors, blur kernels, and noise levels under a unified framework.

SwinIR #OTHEREFR is a strong baseline model that introduces Swin Transformer #OTHEREFR to image restoration.

Their non-linear mapping module is composed of several residual Swin Transformer blocks.

To reduce the computational cost, there are also many efficient SISR methods.

LAPAR #OTHEREFR presents a method based on a linearly-assembled adaptive regression network.

Restormer #OTHEREFR proposes an efficient Transformer model by making several key designs in the building blocks.

All of those methods train one the SR model on large-scale image datasets such as DIV2K #OTHEREFR and test on the given input images.

However, they are not designed for large input as the practical scenario of SR usually needs to superresolve large input like 2K to higher resolution (4K/8K).

Cabm: Content-Aware Bit Mapping For Single Image Super-Resolution Network With Large Input

Abstract: With the development of high-definition display devices, the practical scenario of Super-Resolution (SR) usually needs to super-resolve large input like 2K to higher resolution (4K/8K). To reduce the computational and memory cost, current methods first split the large input into local patches and then merge the SR patches into the output. These methods adaptively allocate a subnet for each patch. Quantization is a very important technique for network acceleration and has been used to design the subnets. Current methods train an MLP bit selector to determine the propoer bit for each layer. However, they uniformly sample subnets for training, making simple subnets overfitted and complicated subnets underfitted. Therefore, the trained bit selector fails to determine the optimal bit. Apart from this, the introduced bit selector brings additional cost to each layer of the SR network. In this paper, we propose a novel method named Content-Aware Bit Mapping (CABM), which can remove the bit selector without any performance loss. CABM also learns a bit selector for each layer during training. After training, we analyze the relation between the edge information of an input patch and the bit of each layer. We observe that the edge information can be an effective metric for the selected bit. Therefore, we design a strategy to build an Edge-to-Bit lookup table that maps the edge score of a patch to the bit of each layer during inference. The bit configuration of SR network can be determined by the lookup tables of all layers. Our strategy can find better bit configuration, resulting in more efficient mixed precision networks. We conduct detailed experiments to demonstrate the generalization ability of our method. The code will be released.

Real-Time Single Image And Video Super-Resolution Using An Efficient Sub-Pixel Convolutional Neural Network

Abstract: Recently, several models based on deep neural networks have achieved great success in terms of both reconstruction accuracy and computational performance for single image super-resolution. In these methods, the low resolution (LR) input image is upscaled to the high resolution (HR) space using a single filter, commonly bicubic interpolation, before reconstruction. This means that the super-resolution (SR) operation is performed in HR space. We demonstrate that this is sub-optimal and adds computational complexity. In this paper, we present the first convolutional neural network (CNN) capable of real-time SR of 1080p videos on a single K2 GPU. To achieve this, we propose a novel CNN architecture where the feature maps are extracted in the LR space. In addition, we introduce an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output. By doing so, we effectively replace the handcrafted bicubic filter in the SR pipeline with more complex upscaling filters specifically trained for each feature map, whilst also reducing the computational complexity of the overall SR operation. We evaluate the proposed approach using images and videos from publicly available datasets and show that it performs significantly better (+0.15dB on Images and +0.39dB on Videos) and is an order of magnitude faster than previous CNN-based methods.


Clean-Label Untargeted Attacks Against At

This could be easily avoided by the robust learner using early stopping #REFR on the validation set.

We evaluate the performance of stickers (using 0 -norm ball that changes 3% of overall pixels) and compare it with REM (using ∞ -norm ball and = 8/255 ≈ 0.031) on robust learners with 0 ∈ [4/255, 16/255].

For each 0 , we execute B 0 -AT five times with different random seeds.

To ensure a consistent result, we fix the random seeds 2000000000-2111100000.

Throughout the training epochs, we record the best natural test accuracy on all test data x and the best robust test accuracy on adversarial datax ∈ B 0 [x].

Rather than relying on the last-checkpoint accuracy, we chose to do this because we found that the AT has a catastrophic overfitting #OTHEREFR when learning from the poisoned set S (see appendix), which gives a false sense of the effectiveness of the poisoning attacks.

In the appendix (see Figure 12) , we report the full learning curves of CIFAR-10 experiments.

Additionally, we evaluated our stickers on the ImageNet Subset dataset (see below).

Table 6 shows that as 0 → (e.g., 0 = = 8/255), the effectiveness of REM, which uses the same norm ball B as the robust learner B 0 , decreases in generating poisoned data to deter (highlighted by "−" in cyan-green) unauthorized data collection, and sometimes even has an encouragement effect (highlighted by "+" in red).

In contrast, our method using a different B remains effective in deterrence when 0 approaches , which supports the efficacy of specifying a norm ball different from the learner's.

Note that as 0 becomes larger than , both REM and our method have diminishing returns, probably due to strong invariance #OTHEREFR and low natural and robust accuracies under large perturbation radii, which calls for large radius poisons.

Assessing Vulnerabilities Of Adversarial Learning Algorithm Through Poisoning Attacks

Abstract: Adversarial training (AT) is a robust learning algorithm that can defend against adversarial attacks in the inference phase and mitigate the side effects of corrupted data in the training phase. As such, it has become an indispensable component of many artificial intelligence (AI) systems. However, in high-stake AI applications, it is crucial to understand AT's vulnerabilities to ensure reliable deployment. In this paper, we investigate AT's susceptibility to poisoning attacks, a type of malicious attack that manipulates training data to compromise the performance of the trained model. Previous work has focused on poisoning attacks against standard training, but little research has been done on their effectiveness against AT. To fill this gap, we design and test effective poisoning attacks against AT. Specifically, we investigate and design clean-label poisoning attacks, allowing attackers to imperceptibly modify a small fraction of training data to control the algorithm's behavior on a specific target data point. Additionally, we propose the clean-label untargeted attack, enabling attackers can attach tiny stickers on training data to degrade the algorithm's performance on all test data, where the stickers could serve as a signal against unauthorized data collection. Our experiments demonstrate that AT can still be poisoned, highlighting the need for caution when using vanilla AT algorithms in security-related applications. The code is at https://github.com/zjfheart/Poison-adv-training.git.

Overfitting In Adversarially Robust Deep Learning

Abstract: It is common practice in deep learning to use overparameterized networks and train for as long as possible; there are numerous studies that show, both theoretically and empirically, that such practices surprisingly do not unduly harm the generalization performance of the classifier. In this paper, we empirically study this phenomenon in the setting of adversarially trained deep networks, which are trained to minimize the loss under worst-case adversarial perturbations. We find that overfitting to the training set does in fact harm robust performance to a very large degree in adversarially robust training across multiple datasets (SVHN, CIFAR-10, CIFAR-100, and ImageNet) and perturbation models ($\ell_\infty$ and $\ell_2$). Based upon this observed effect, we show that the performance gains of virtually all recent algorithmic improvements upon adversarial training can be matched by simply using early stopping. We also show that effects such as the double descent curve do still occur in adversarially trained models, yet fail to explain the observed overfitting. Finally, we study several classical and modern deep learning remedies for overfitting, including regularization and data augmentation, and find that no approach in isolation improves significantly upon the gains achieved by early stopping. All code for reproducing the experiments as well as pretrained model weights and training logs can be found at this https URL.