On the other hand, the 1kB split uses the identical 1,000 test split of 1kA for the test, whereas the train set is a subset of 1kA's containing 6,656 samples #REFR .

The train dataset contains 1,261 samples, and the test set contains 439 samples, respectively.

The original MSRVTT-full #OTHEREFR dataset, used on video captioning task, contains 6,513 train, 497 validation, and 2,990 test samples.

However, we have observed a wide range of dataset split variations throughout research on text-to-video retrieval.

One split variant randomly samples 1,000 clip-text pairs from the test set for evaluation and uses the rest of the 9,000 samples as train data #OTHEREFR , which is commonly denoted as the 1kA split.

Another commonly used data split also uses the identical 1,000 test set, while adopting both the train and validation set from the standard MSRVTT for training.

We evaluated our method on two split protocols most prominently observed in the literature, 1kA, and 7k.

For convenience, we denote the former as MSRVTT-9k and the latter as MSRVTT-7k. MSVD-QA.

MSVD-QA #OTHEREFR contains 47k open-ended questions on 2k videos, derived from the original MSVD dataset #OTHEREFR .

We construct the answer set with 1,000 most frequently appeared answers.

Meltr: Meta Loss Transformer For Learning To Fine-Tune Video Foundation Models

Abstract: Foundation models have shown outstanding performance and generalization capabilities across domains. Since most studies on foundation models mainly focus on the pretraining phase, a naive strategy to minimize a single task-specific loss is adopted for fine-tuning. However, such fine-tuning methods do not fully leverage other losses that are potentially beneficial for the target task. Therefore, we propose MEta Loss TRansformer (MELTR), a plug-in module that automatically and non-linearly combines various loss functions to aid learning the target task via auxiliary learning. We formulate the auxiliary learning as a bi-level optimization problem and present an efficient optimization algorithm based on Approximate Implicit Differentiation (AID). For evaluation, we apply our framework to various video foundation models (UniVL, Violet and All-in-one), and show significant performance gain on all four downstream tasks: text-to-video retrieval, video question answering, video captioning, and multimodal sentiment analysis. Our qualitative analyses demonstrate that MELTR adequately 'transforms' individual loss functions and 'melts' them into an effective unified loss.

Learning A Text-Video Embedding From Incomplete And Heterogeneous Data

Abstract: Joint understanding of video and language is an active research area with many applications. Prior work in this domain typically relies on learning text-video embeddings. One difficulty with this approach, however, is the lack of large-scale annotated video-caption datasets for training. To address this issue, we aim at learning text-video embeddings from heterogeneous data sources. To this end, we propose a Mixture-of-Embedding-Experts (MEE) model with ability to handle missing input modalities during training. As a result, our framework can learn improved text-video embeddings simultaneously from image and video datasets. We also show the generalization of MEE to other input modalities such as face descriptors. We evaluate our method on the task of video retrieval and report results for the MPII Movie Description and MSR-VTT datasets. The proposed MEE model demonstrates significant improvements and outperforms previously reported methods on both text-to-video and video-to-text retrieval tasks. Code is available at: https://github.com/antoine77340/Mixture-of-Embedding-Experts


We then use PointNet #REFR architecture to form a mapping from point-cloud observation of the target object, and pose of the deformable tool to 3D representation of the contact points between the two bodies.

However, this contact reasoning concept is more suitable for the control aspect than for the planning aspect due to its ability to track the interaction in real time.

Thus, the question of how to predict the interaction between deformable and rigid objects and exploit such interactions for planning remains open.

To address the aforementioned open issues, we propose Sequence Planning with deformable-ON-rigid contact prediction from GEometric features (SPONGE), a sequence planning pipeline powered by a contact prediction model that predicts contact between deformable and rigid bodies, with the aim of providing robots with the aforementioned human-like planning skill in order to efficiently automate downstream deformable object manipulation tasks such as cleaning dishes (Fig. 1 ).

Instead of contact reasoning, in this paper we tackle the concept of contact prediction of a 3D deformable tool acting on rigid objects, which is better suited for planning purposes.

We take a data-driven approach with physics-based simulation to model the interactions between 3D deformable objects and rigid objects.

The trained contact prediction model is then used as the driving force behind the planning of a subsequent task.

Finally, we deploy SPONGE in the real world to demonstrate that the contact prediction model trained only with synthetic data from physics-based simulation can help to produce an efficient plan for a manipulation task to be successfully executed in the real world.

In summary, the main contributions of this paper are as follows:

• A deep learning-based contact prediction model that predicts the contacts between 3D deformable and rigid objects under interactions.

• A planning pipeline powered by the proposed contact prediction model to efficiently automate deformable object manipulation tasks.

Sponge: Sequence Planning With Deformable-On-Rigid Contact Prediction From Geometric Features

Abstract: Planning robotic manipulation tasks, especially those that involve interaction between deformable and rigid objects, is challenging due to the complexity in predicting such interactions. We introduce SPONGE, a sequence planning pipeline powered by a deep learning-based contact prediction model for contacts between deformable and rigid bodies under interactions. The contact prediction model is trained on synthetic data generated by a developed simulation environment to learn the mapping from point-cloud observation of a rigid target object and the pose of a deformable tool, to 3D representation of the contact points between the two bodies. We experimentally evaluated the proposed approach for a dish cleaning task both in simulation and on a real Franka Emika Panda with real-world objects. The experimental results demonstrate that in both scenarios the proposed planning pipeline is capable of generating high-quality trajectories that can accomplish the task by achieving more than 90% area coverage on different objects of varying sizes and curvatures while minimizing travel distance. Code and video are available at: https://irobotics.aalto.fi/sponge/.

Pointnet: Deep Learning On Point Sets For 3D Classification And Segmentation

Abstract: Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.


Goodfellow proposed the first generative model, Generative Adversarial Network (GAN), in 2014 #REFR . Table II shows the evolution timeline of generative algorithms.

The algorithms simulate the simple human brain, which improves the learning model through experience accumulation #OTHEREFR .

Neural network models further emulate the signal processing and thinking mechanisms of human brain nerves #OTHEREFR , #OTHEREFR .

Furthermore, generative algorithms, such as Google's Transformer architecture #OTHEREFR , draw on human attention mechanisms to enable the completion of multiple tasks by an algorithm.

They can model the probability distribution of the input data and then generate new data.

They are trained by processing large amounts of text data, then fine-tuned on specific tasks with labeled data, enabling them to interact according to contextual content and chat in a manner similar to human beings.

In most cases, the significance of GAN is a source of inspiration for many popular variations and architectures.

The transformer model has a wide range of applications in various domains (including NLP and CV).

In addition, several pre-training models, such as BERT, GPT-3, and LaMDA, have been developed based on the Transformer model.

The diffusion model is currently the most advanced image generation model because of its optimized performance.

With the development of generative models, language models have also made great progress. For example, Devlin et al.

Ai-Generated Content (Aigc): A Survey

Abstract: To address the challenges of digital intelligence in the digital economy, artificial intelligence-generated content (AIGC) has emerged. AIGC uses artificial intelligence to assist or replace manual content generation by generating content based on user-inputted keywords or requirements. The development of large model algorithms has significantly strengthened the capabilities of AIGC, which makes AIGC products a promising generative tool and adds convenience to our lives. As an upstream technology, AIGC has unlimited potential to support different downstream applications. It is important to analyze AIGC's current capabilities and shortcomings to understand how it can be best utilized in future applications. Therefore, this paper provides an extensive overview of AIGC, covering its definition, essential conditions, cutting-edge capabilities, and advanced features. Moreover, it discusses the benefits of large-scale pretrained models and the industrial chain of AIGC. Furthermore, the article explores the distinctions between auxiliary generation and automatic generation within AIGC, providing examples of text generation. The paper also examines the potential integration of AIGC with the Metaverse. Lastly, the article highlights existing issues and suggests some future directions for application. Impact Statement-It is necessary for academia and industry to take an overview of what AIGC is, how AIGC works, how AIGC changes our lifestyles, and what AIGC will be in the future. This article proposes a survey of AIGC from its definition, pros, cons, applications, current challenges, and future directions to answer these urgent questions. We summarize the existing major literature, which helps relevant researchers become familiar with and understand the existing works and unsolved problems. Based on the review of literature and the commercialization of scientific and research findings, we conduct some cutting-edge AIGC research. In particular, the challenges and future directions of AIGC can be helpful for developing AI. Relevant technologies of AIGC will boost the development of artificial intelligence, better serve human society, and achieve sustainable development.

Generative Adversarial Networks

Abstract: We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

generative algorithms
Generative Adversarial Network


B. Networks And Algorithms 1) Whistle Extraction Network:

The generators follow the U-Net #REFR architecture, which has 6 U-Net blocks with a basic width of 64. InstanceNorm layers are used in the U-Net blocks.

It contains 4 convolutional layers with a stride of 2 and a fully connected layer.

The networks are optimized by Adam optimizers (initial learning rate = 1 × 10 −4 , betas = [0.5, 0.9], batch size = 64) for 3 × 10 4 and 5 × 10 4 iterations on the reduced and full datasets, respectively.

In each WGAN training iteration, the discriminator is optimized for 5 steps while the generator is optimized for 1 step, where the network parameters are updated by applying the optimizer to one minibatch of data in each step.

For sample selection, we used T e =70, T c =0.5, T p =64.

3) CycleGAN: The GAN model that we used to add whistles on synthetic noise employs the CycleGAN architecture of #OTHEREFR .

The discriminator is a fully convolutional network with 3 convolutional layers.

We trained the generators and discriminators with Adam optimizers (learning rate = 2 × 10 −4 , betas = [0.5, 0.999], batch size = 64) for 25,120 iterations (160 epochs for 10,000 real positive samples) for the reduced dataset and 50 epochs for the full dataset.

We apply a random γ following a unified distribution between (0.5, 1.5) in Eq. 3.

Learning Stage-Wise Gans For Whistle Extraction In Time-Frequency Spectrograms

Abstract: Whistle contour extraction aims to derive animal whistles from time-frequency spectrograms as polylines. For toothed whales, whistle extraction results can serve as the basis for analyzing animal abundance, species identity, and social activities. During the last few decades, as long-term recording systems have become affordable, automated whistle extraction algorithms were proposed to process large volumes of recording data. Recently, a deep learning-based method demonstrated superior performance in extracting whistles under varying noise conditions. However, training such networks requires a large amount of labor-intensive annotation, which is not available for many species. To overcome this limitation, we present a framework of stage-wise generative adversarial networks (GANs), which compile new whistle data suitable for deep model training via three stages: generation of background noise in the spectrogram, generation of whistle contours, and generation of whistle signals. By separating the generation of different components in the samples, our framework composes visually promising whistle data and labels even when few expert annotated data are available. Regardless of the amount of human-annotated data, the proposed data augmentation framework leads to a consistent improvement in performance of the whistle extraction model, with a maximum increase of 1.69 in the whistle extraction mean F1-score. Our stage-wise GAN also surpasses one single GAN in improving whistle extraction models with augmented data.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .


We compare our loss (editing loss) with the following baselines: #REFR No Editing: the initial segmentation y init is used as the final segmentationŷ, and the overall error in this case is the distance from the CAS contours to y init . This should serve as an upper bound for error.

We use the editing evaluation metric D and report the 95 th percentile of the overall editing error, the error near the user input, and far from the user input (mm).

The near and far regions are defined by thresholding A at 0.5.

For the CV results, we report the mean and standard deviation over the five folds.

The statistical significance is computed for the difference with InterCNN.

(2) CE Loss: an editing model trained using the standard CE segmentation loss.

(3) Dice Loss #OTHEREFR : an editing model trained using Dice segmentation loss.

4InterCNN #OTHEREFR : for every training sample, simulated user edits based on the prediction are accumulated with any previous edits and re-input to the model for 10 iterations, trained using CE loss.

We report the results after a single edit (the furthest CAS contour fromŷ) in Table 1 .

A single training epoch takes ≈ 3 minutes for all the models except for InterCNN, which takes ≈ 14 minutes, on a single NVIDIA Tesla V100 GPU.

From Sparse To Precise: A Practical Editing Approach For Intracardiac Echocardiography Segmentation

Abstract: Accurate and safe catheter ablation procedures for patients with atrial fibrillation require precise segmentation of cardiac structures in Intracardiac Echocardiography (ICE) imaging. Prior studies have suggested methods that employ 3D geometry information from the ICE transducer to create a sparse ICE volume by placing 2D frames in a 3D grid, enabling training of 3D segmentation models. However, the resulting 3D masks from these models can be inaccurate and may lead to serious clinical complications due to the sparse sampling in ICE data, frames misalignment, and cardiac motion. To address this issue, we propose an interactive editing framework that allows users to edit segmentation output by drawing scribbles on a 2D frame. The user interaction is mapped to the 3D grid and utilized to execute an editing step that modifies the segmentation in the vicinity of the interaction while preserving the previous segmentation away from the interaction. Furthermore, our framework accommodates multiple edits to the segmentation output in a sequential manner without compromising previous edits. This paper presents a novel loss function and a novel evaluation metric specifically designed for editing. Results from cross-validation and testing indicate that our proposed loss function outperforms standard losses and training strategies in terms of segmentation quality and following user input. Additionally, we show quantitatively and qualitatively that subsequent edits do not compromise previous edits when using our method, as opposed to standard segmentation losses. Overall, our approach enhances the accuracy of the segmentation while avoiding undesired changes away from user interactions and without compromising the quality of previously edited regions, leading to better patient outcomes.

Iterative Interaction Training For Segmentation Editing Networks

Abstract: Automatic segmentation has great potential to facilitate morphological measurements while simultaneously increasing efficiency. Nevertheless often users want to edit the segmentation to their own needs and will need different tools for this. There has been methods developed to edit segmentations of automatic methods based on the user input, primarily for binary segmentations. Here however, we present an unique training strategy for convolutional neural networks (CNNs) trained on top of an automatic method to enable interactive segmentation editing that is not limited to binary segmentation. By utilizing a robot-user during training, we closely mimic realistic use cases to achieve optimal editing performance. In addition, we show that an increase of the iterative interactions during the training process up to ten improves the segmentation editing performance substantially. Furthermore, we compare our segmentation editing CNN (interCNN) to state-of-the-art interactive segmentation algorithms and show a superior or on par performance.

initial segmentation
final segmentationŷ


The detection error with the deep features method #REFR on WL images indicates a performance similar to that of both the VIS and NIR spectral bands.

The deep features method #OTHEREFR , which is based on facial features, indicates less variation in degraded performance, especially in the NIR bands.

However, the Hierarchical Deep Residual SLERP #OTHEREFR , which is based on texture features that indicate higher variation in the detection performance, especially in NIR spectral bands, compared to the deep features method #OTHEREFR .

The possible degradation of the Hierarchical Deep Residual SLERP #OTHEREFR can be attributed to the lack of texture information in the NIR spectral bands.

Thus, the detection performance of D-MAD in the NIR spectrum depends on the type of feature used for morphing de-tection.

• The use of wholeLight (WL) that is captured without any spectral filtering indicates the varied detection performance with respect to D-MAD techniques.

Because the deep features method #OTHEREFR is based on facial features, it is less sensitive to different spectral bands, which may be due to the backbone model that is trained only using VIS images.

However, the performance of Hierarchical Deep Residual SLERP #OTHEREFR on WL images shows degraded performance compared to the VIS spectral bands.

Table 3 lists the performance of the D-MAD algorithms on the proposed multispectral framework and visible images.

Both multispectral and visible data are collected using the same data subjects, and this can provide insights into the utility of multispectral images for reliable morphing attack detection.

Furthermore, the performance of the multispectral framework was presented by fusing the results of all individual spectral bands.

Multispectral Imaging For Differential Face Morphing Attack Detection: A Preliminary Study

Abstract: Face morphing attack detection is emerging as an increasingly challenging problem owing to advancements in high-quality and realistic morphing attack generation. Reliable detection of morphing attacks is essential because these attacks are targeted for border control applications. This paper presents a multispectral framework for differential morphing-attack detection (D-MAD). The D-MAD methods are based on using two facial images that are captured from the ePassport (also called the reference image) and the trusted device (for example, Automatic Border Control (ABC) gates) to detect whether the face image presented in ePassport is morphed. The proposed multispectral D-MAD framework introduce a multispectral image captured as a trusted capture to capture seven different spectral bands to detect morphing attacks. Extensive experiments were conducted on the newly created datasets with 143 unique data subjects that were captured using both visible and multispectral cameras in multiple sessions. The results indicate the superior performance of the proposed multispectral framework compared to visible images.

Deep Face Representations For Differential Morphing Attack Detection

Abstract: The vulnerability of facial recognition systems to face morphing attacks is well known. Many different approaches for morphing attack detection have been proposed in the scientific literature. However, the morphing attack detection algorithms proposed so far have only been trained and tested on datasets whose distributions of image characteristics are either very limited (e.g. only created with a single morphing tool) or rather unrealistic (e.g. no print-scan transformation). As a consequence, these methods easily overfit on certain image types and the results presented cannot be expected to apply to real-world scenarios. For example, the results of the latest NIST Face Recognition Vendor Test MORPH show that the submitted MAD algorithms lack robustness and performance when considering unseen and challenging datasets. In this work, subsets of the FERET and FRGCv2 face databases are used to create a large realistic database for training and testing of morphing attack detection algorithms, containing a large number of ICAO-compliant bona fide facial images, corresponding unconstrained probe images, and morphed images created with four different tools. Furthermore, multiple post-processings are applied on the reference images, e.g. print-scan and JPEG2000 compression. On this database, previously proposed differential morphing algorithms are evaluated and compared. In addition, the application of deep face representations for differential morphing attack detection algorithms is investigated. It is shown that algorithms based on deep face representations can achieve very high detection performance (less than 3\%~\mbox{D-EER}) and robustness with respect to various post-processings. Finally, the limitations of the developed methods are analyzed.


As seen in Table 5 , we found that a more expressive normalizing flow with resampled base #REFR Stimper et al., 2022) improves significantly the results over a simpler radial normalizing flow (Rezende & Mohamed, 2015) across all the metrics.

The list of core architectures used across the experiments are: ResNet18 / ResNet50 / EfficientNet / Swin #OTHEREFR Tan & Le, 2021; from the torchvision repository 2 and Wide-ResNet-28-10 (Zagoruyko & Komodakis, 2016) from the original implementation of DUE.

Except for the experiment on architecture type and size where ResNet18 has output channels for the residual blocks with size [64, 128, 256, 512] , ResNet18 has output channels for the residual blocks with size [32, 64, 128, 256] which causes small differences in final accuracy.

For DUE we use the original implementation 3 with by default we use the RBF kernel function.

For NatPN we use the original implementation 4 but change the uncertainty head with a more expressive density estimator.

For all the experiments (except toys where we use radial flow) we use NSF-R with 16 layers.

Workshop On The Pitfalls Of Limited Data And Computation For Trustworthy Ml, Iclr 2023 Training, Architecture, And Prior For Deterministic Uncertainty Methods

Abstract: Accurate and efficient uncertainty estimation is crucial to build reliable Machine Learning (ML) models capable to provide calibrated uncertainty estimates, generalize and detect Out-Of-Distribution (OOD) datasets. To this end, Deterministic Uncertainty Methods (DUMs) is a promising model family capable to perform uncertainty estimation in a single forward pass. This work investigates important design choices in DUMs: (1) we show that training schemes decoupling the core architecture and the uncertainty head schemes can significantly improve uncertainty performances. (2) we demonstrate that the core architecture expressiveness is crucial for uncertainty performance and that additional architecture constraints to avoid feature collapse can deteriorate the trade-off between OOD generalization and detection. (3) Contrary to other Bayesian models, we show that the prior defined by DUMs do not have a strong effect on the final performances.

Neural Spline Flows

Abstract: A normalizing flow models a complex probability density as an invertible transformation of a simple base density. Flows based on either coupling or autoregressive transforms both offer exact density evaluation and sampling, but rely on the parameterization of an easily invertible elementwise transformation, whose choice determines the flexibility of these models. Building upon recent work, we propose a fully-differentiable module based on monotonic rational-quadratic splines, which enhances the flexibility of both coupling and autoregressive transforms while retaining analytic invertibility. We demonstrate that neural spline flows improve density estimation, variational inference, and generative modeling of images.


However, we provide an alternative approach that, while not as fast as Algorithm 3, still provides an improvement over the algorithm of #REFR for graphs in which all (self-avoiding) paths from s to t are short.

For the more general case of st-path G (x) when G(x) is not known to only have one st-path, while it is possible that an algorithm similar to Algorithm 3 would work, we have not been able to bound the running time effectively.

This is because in the case of a single path, once you find an intermediate edge on the path, the longest paths from s and t to that edge must be shorter than the length of the longest path from s to t.

This ensures that subproblems take shorter time than the original problem. With multiple paths, we no longer have that guarantee.

Our approach does not make use of our path-edge sampling algorithm as a subroutine, and instead uses the path detection algorithm of Lemma 8 to decide whether there are paths through various subgraphs, and then uses that information to find each edge in a path in order from s to t.

In this way, we avoid the problem of subproblems being larger than the original problem, since if the longest path from s to t has length L, and the first edge we find on the path is (s, u), then longest path from u to t that doesn't go through s must have length at most L − 1. However, we lose the advantage of a divide-and-conquer approach.

To find the first edge on a path, we use a group testing approach.

We divide the neighbors of s in G into two sets, S 1 and S 2 and run path detection algorithms in parallel on two subgraphs of G(x), one with edges from s removed, except those to vertices in S 1 (that is, G − {{s,u}∈E:u∈S 1 } ), and one with edges from s removed, except those to vertices in S 2 .

We will detect which of these subgraphs contains a path, and we will know there is a path whose first edge goes from s to a vertex in the corresponding set (S 1 or S 2 ).

Quantum Algorithm For Path-Edge Sampling

Abstract: We present a quantum algorithm for sampling an edge on a path between two nodes s and t in an undirected graph given as an adjacency matrix, and show that this can be done in query complexity that is asymptotically the same, up to log factors, as the query complexity of detecting a path between s and t. We use this path sampling algorithm as a subroutine for st-path finding and st-cut-set finding algorithms in some specific cases. Our main technical contribution is an algorithm for generating a quantum state that is proportional to the positive witness vector of a span program.

Quantum Query Complexity Of Some Graph Problems

Abstract: Quantum algorithms for graph problems are considered, both in the adjacency matrix model and in an adjacency list-like array model. We give almost tight lower and upper bounds for the bounded error quantum query complexity of Connectivity, Strong Connectivity, Minimum Spanning Tree, and Single Source Shortest Paths. For example we show that the query complexity of Minimum Spanning Tree is in Θ(n 3/2) in the matrix model and in \(\Theta(\sqrt{nm})\) in the array model, while the complexity of Connectivity is also in Θ(n 3/2) in the matrix model, but in Θ(n) in the array model. The upper bounds utilize search procedures for finding minima of functions under various conditions.


In our preliminary experiments, small batches provided models with worse generalization, and sets of 256 and 512 objects produced similar results; hence, to prioritize a flat minimum #REFR , we chose 256 objects in each batch.

Each of those hyperparameters is relevant and has an impact on the final results, but focusing on generalization, it is essential to highlight the early stopping criteria.

The patience hyperparameter allows managing the overfitting; when a higher patience value is used, the overfitting is increased, and obviously, the performance in the testing set is decreased.

Another relevant pattern was observed in the batch size; the impact of the size batch on optimization results has been studied.

The evidence shows that a small batch size generates flat minima (better generalization), and on the other hand, bigger batches tend to lead to sharp minima #OTHEREFR .

However, this evidence is based on a representative training set, so each batch is a good sample; in a case considering data shift, that is not true, and because of that, small batches can provide very wrong directions for the optimizer.

Informative Regularization For A Multi-Layer Perceptron Rr Lyrae Classifier Under Data Shift

Abstract: In recent decades, machine learning has provided valuable models and algorithms for processing and extracting knowledge from time-series surveys. Different classifiers have been proposed and performed to an excellent standard. Nevertheless, few papers have tackled the data shift problem in labeled training sets, which occurs when there is a mismatch between the data distribution in the training set and the testing set. This drawback can damage the prediction performance in unseen data. Consequently, we propose a scalable and easily adaptable approach based on an informative regularization and an ad-hoc training procedure to mitigate the shift problem during the training of a multi-layer perceptron for RR Lyrae classification. We collect ranges for characteristic features to construct a symbolic representation of prior knowledge, which was used to model the informative regularizer component. Simultaneously, we design a two-step back-propagation algorithm to integrate this knowledge into the neural network, whereby one step is applied in each epoch to minimize classification error, while another is applied to ensure regularization. Our algorithm defines a subset of parameters (a mask) for each loss function. This approach handles the forgetting effect, which stems from a trade-off between these loss functions (learning from data versus learning expert knowledge) during training. Experiments were conducted using recently proposed shifted benchmark sets for RR Lyrae stars, outperforming baseline models by up to 3% through a more reliable classifier. Our method provides a new path to incorporate knowledge from characteristic features into artificial neural networks to manage the underlying data shift problem.

On Large-Batch Training For Deep Learning: Generalization Gap And Sharp Minima

Abstract: The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say $32$-$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.


Image generation has garnered significant research interest, particularly in recent years, with the advent of GANs #REFR .

GANs consist of two neural networks, a generator and a discriminator, that compete with each other to create realistic images #OTHEREFR .

While GANs demonstrate the capability to generate high resolution images with good perceptual quality #OTHEREFR , it suffers from the difficulty to train effectively #OTHEREFR .

VAEs #OTHEREFR emerged as a popular generative model, utilizing probabilistic graphical models to generate images through latent space representation, enabling efficient synthesis but with lower sample quality than GANs.

Denoising Diffusion Probabilistic Model (DDPM) #OTHEREFR marked a significant milestone in the development of diffusion-based generative models.

DDPM demonstrated the potential of these models to generate high-quality images through a series of iterative noise-removal steps.

Speed Is All You Need: On-Device Acceleration Of Large Diffusion Models Via Gpu-Aware Optimizations

Abstract: The rapid development and application of foundation models have revolutionized the field of artificial intelligence. Large diffusion models have gained significant attention for their ability to generate photorealistic images and support various tasks. On-device deployment of these models provides benefits such as lower server costs, offline functionality, and improved user privacy. However, common large diffusion models have over 1 billion parameters and pose challenges due to restricted computational and memory resources on devices. We present a series of implementation optimizations for large diffusion models that achieve the fastest reported inference latency to-date(under 12 seconds for Stable Diffusion 1.4 without INT8 quantization for a 512 × 512 image with 20 iterations) on GPUequipped mobile devices. These enhancements broaden the applicability of generative AI and improve the overall user experience across a wide range of devices.

Generative Adversarial Networks


In particular, we point out that our results are sharper than the EDI #REFR reconstruction, which assumes all events correspond to the same amount of intensity change.

We use three image quality metrics to compare DeblurSR with different baseline approaches: the Mean Squared Error (MSE, lower is better), the Peak Signal-to-Noise Ratio (PSNR, higher is better), and the Structural Similarity Index Measure (SSIM, higher is better).

Our method demonstrates an impressive ability in motion deblurring.

Specifically, on the REDS dataset, De-blurSR improves the current best-performing method by 12.3% in MSE, 4.7% in PSNR, and 4.9% in SSIM.

On HQF, DeblurSR outperforms the state-of-the-art approach by 22.2% in MSE, 10.1% in PSNR, and 14.0% in SSIM.

Qualitatively, as shown in Figure 4 and Figure 5 , De-blurSR generates smooth and sharp frames.

Meanwhile, our recon-structed frames are significantly less noisy than the eSL-Net #OTHEREFR reconstruction, which overly emphasizes texture details.

Compared with E-CIR #OTHEREFR , DeblurSR offers more realistic details around the thin edges.

The difference between E-CIR and DeblurSR is sometimes subtle and hard to notice from static images.

We encourage readers to watch the animated visualizations provided in the supplementary material, which additionally demonstrate the temporal smoothness of our results in the video format.

Deblursr: Event-Based Motion Deblurring Under The Spiking Representation

Abstract: We present DeblurSR, a novel motion deblurring approach that converts a blurry image into a sharp video. DeblurSR utilizes event data to compensate for motion ambiguities and exploits the spiking representation to parameterize the sharp output video as a mapping from time to intensity. Our key contribution, the Spiking Representation (SR), is inspired by the neuromorphic principles determining how biological neurons communicate with each other in living organisms. We discuss why the spikes can represent sharp edges and how the spiking parameters are interpreted from the neuromorphic perspective. DeblurSR has higher output quality and requires fewer computing resources than state-of-the-art event-based motion deblurring methods. We additionally show that our approach easily extends to video super-resolution when combined with recent advances in implicit neural representation. The implementation and animated visualization of DeblurSR are available at https://github.com/chensong1995/DeblurSR.

Bringing A Blurry Frame Alive At High Frame-Rate With An Event Camera

Abstract: Event-based cameras can measure intensity changes (called ‘events’) with microsecond accuracy under high-speed motion and challenging lighting conditions. With the active pixel sensor (APS), the event camera allows simultaneous output of the intensity frames. However, the output images are captured at a relatively low frame-rate and often suffer from motion blur. A blurry image can be regarded as the integral of a sequence of latent images, while the events indicate the changes between the latent images. Therefore, we are able to model the blur-generation process by associating event data to a latent image. In this paper, we propose a simple and effective approach, the Event-based Double Integral (EDI) model, to reconstruct a high frame-rate, sharp video from a single blurry frame and its event data. The video generation is based on solving a simple non-convex optimization problem in a single scalar variable. Experimental results on both synthetic and real images demonstrate the superiority of our EDI model and optimization method in comparison to the state-of-the-art.


Visual SLAM (VSLAM) systems can employ a wide range of vision sensors, such as monocular, stereo, Red Green Blue-Depth (RGB-D), omnidirectional, and eventbased cameras to estimate the environmental map while localizing the camera #REFR .

The primary advantage of vision sensors is that they need low-cost hardware to supply rich visual and semantic information from surroundings for various tasks #OTHEREFR .

Semantic data, which refers to high-level information acquired from the environment, make VSLAM tasks more robust and expand the range of applications that can employ the reconstructed maps #OTHEREFR , #OTHEREFR .

For instance, robots may need to identify objects in the scene Fig.

1 : The final reconstructed map of the environment using the proposed method in a hierarchical representation.

Accordingly, the primary map entities are detected 3D points extracted from the environment and visited ArUco markers.

Marker-Based Visual Slam Leveraging Hierarchical Representations

Abstract: Fiducial markers can encode rich information about the environment and can aid Visual SLAM (VSLAM) approaches in reconstructing maps with practical semantic information. Current marker-based VSLAM approaches mainly utilize markers for improving feature detections in low-feature environments and/or for incorporating loop closure constraints, generating only low-level geometric maps of the environment prone to inaccuracies in complex environments. To bridge this gap, this paper presents a VSLAM approach utilizing a monocular camera along with fiducial markers to generate hierarchical representations of the environment while improving the camera pose estimate. The proposed approach detects semantic entities from the surroundings, including walls, corridors, and rooms encoded within markers, and appropriately adds topological constraints among them. Experimental results on a real-world dataset collected with a robot demonstrate that the proposed approach outperforms a traditional marker-based VSLAM baseline in terms of accuracy, given the addition of new constraints while creating enhanced map representations. Furthermore, it shows satisfactory results when comparing the reconstructed map quality to the one reconstructed using a LiDAR SLAM approach.

Past, Present, And Future Of Simultaneous Localization And Mapping: Toward The Robust-Perception Age

Abstract: Simultaneous localization and mapping (SLAM) consists in the concurrent construction of a model of the environment (the map ), and the estimation of the state of the robot moving within it. The SLAM community has made astonishing progress over the last 30 years, enabling large-scale real-world applications and witnessing a steady transition of this technology to industry. We survey the current state of SLAM and consider future directions. We start by presenting what is now the de-facto standard formulation for SLAM. We then review related work, covering a broad set of topics including robustness and scalability in long-term mapping, metric and semantic representations for mapping, theoretical performance guarantees, active SLAM and exploration, and other new frontiers. This paper simultaneously serves as a position paper and tutorial to those who are users of SLAM. By looking at the published research with a critical eye, we delineate open challenges and new research issues, that still deserve careful scientific investigation. The paper also contains the authors’ take on two questions that often animate discussions during robotics conferences: Do robots need SLAM? and Is SLAM solved?


Its counterpart in structured pruning-Iterative L1-norm-based pruning (ILP) #REFR , which removes filters based on their L1-norm values, cannot effectively prune a model while maintaining its accuracy.

But these methods have to explore a large search space of all available layer-wise sparsity, which is time consuming when neural networks are large and datasets are complex.

Therefore, there is a great need for an automatic, iterative, structured pruning solution that can automatically and efficiently generate small, accurate, and hardware-efficient models. The challenges are three-fold.

First, how to effectively identify the insignificant parameters in a model to prune? Existing works have explored different mechanisms, e.g., L2-Norm in Soft Filter Pruning (SFP) #OTHEREFR , Soft Channel Pruning (SCP) [Kang and Han, 2020] and EagleEye #OTHEREFR , geometric median in FPGM #OTHEREFR , Hessian in EigenDamage , Empirical Sensitivity in Provable Filter Pruning (PFP) #OTHEREFR , adversarial knockoff features in SCOP #OTHEREFR , polarization regularizer in Neuron-level Structured Pruning (NSP) #OTHEREFR , LASSO regression in Channel Pruning (CP) #OTHEREFR , and other information considering the relationship between neighboring layers (Gate Batch Normalization (GBN) #OTHEREFR , Sparse Structure Selection (SSS) #OTHEREFR , Hinge #OTHEREFR , Pruning From Scratch (PFS) and Stripe-Wise Pruning (SWP) #OTHEREFR ).

In comparison, activation-based attention, proposed in this paper, can more effectively capture the importance of filters, and pruning based on attention values can produce much better models, as quantitatively shown in our evaluation (Section 4).

Second, how to design an effective iterative pruning process to recover the accuracy loss caused by structured pruning? LTH-based iterative pruning is a promising approach, but it has only been shown to work with unstructured pruning such as IMP.

For example, ILP can prune ResNet-50 by at most 11.5% of parameters when the maximum accuracy loss is limited to 1% on ImageNet.

So directly applying iterative pruning with existing weightmagnitude-based structured pruning methods does not produce accurate pruned models.

This paper proposes a novel LTH-based iterative, structured pruning solution using attentions, and it significantly outperforms ILP and other related structured pruning works that involve an iterative process (GDP #OTHEREFR , ACTD , Quantization and Pruning (QP) #OTHEREFR , IMP-Refill and IMP-Regroup #OTHEREFR ).

The third challenge is how to automate the pruning process so it does not require any human intervention? The existing structured pruning works all require difficult hand-tuning of many hyper-parameters, e.g., DCP #OTHEREFR and MDP need multiple hyper-parameters to balance the original taskspecific loss and the additional pruning loss; VCNNP #OTHEREFR requires careful settings of τ and θ to decide which filters to prune; DMC #OTHEREFR and DeepHoyer ] require parameters to decide the regularization strength with different settings for different datasets and models.

To address this challenge, this paper proposes a fully automated pruning solution that can automatically generate pruned models that meet users' diverse model accuracy, size, and speed requirements.

Automatic Attention Pruning: Improving And Automating Model Pruning Using Attentions

Abstract: Pruning is a promising approach to compress deep learning models in order to deploy them on resource-constrained edge devices. However, many existing pruning solutions are based on unstructured pruning, which yields models that cannot efficiently run on commodity hardware; and they often require users to manually explore and tune the pruning process, which is timeconsuming and often leads to sub-optimal results. To address these limitations, this paper presents Automatic Attention Pruning (AAP), an adaptive, attention-based, structured pruning approach to automatically generate small, accurate, and hardware-efficient models that meet user objectives. First, it proposes iterative structured pruning using activation-based attention maps to effectively identify and prune unimportant filters. Then, it proposes adaptive pruning policies for automatically meeting the pruning objectives of accuracy-critical, memory-constrained, and latency-sensitive tasks. A comprehensive evaluation shows that AAP substantially outperforms the state-of-the-art structured pruning works for a variety of model architectures. Our code is at: https://github.com/kaiqi123/ Automatic-Attention-Pruning.git.

Comparing Rewinding And Fine-Tuning In Neural Network Pruning

Abstract: Many neural network pruning algorithms proceed in three steps: train the network to completion, remove unwanted structure to compress the network, and retrain the remaining structure to recover lost accuracy. The standard retraining technique, fine-tuning, trains the unpruned weights from their final trained values using a small fixed learning rate. In this paper, we compare fine-tuning to alternative retraining techniques. Weight rewinding (as proposed by Frankle et al., (2019)), rewinds unpruned weights to their values from earlier in training and retrains them from there using the original training schedule. Learning rate rewinding (which we propose) trains the unpruned weights from their final values using the same learning rate schedule as weight rewinding. Both rewinding techniques outperform fine-tuning, forming the basis of a network-agnostic pruning algorithm that matches the accuracy and compression ratios of several more network-specific state-of-the-art techniques.


A similar conclusion has already been made in #REFR but the properties of multi-person and long video in MPEblink make it worse for landmark-based eyeblink detection methods.

• For Blink-AP, InstBlink significantly outperforms the others by large margins (i.e., 21% at least of Blink-AP 50 and 6% at least of Blink-AP 75 ), which verifies the superiority of the proposed framework.

We argue that one essential reason is that our framework can model a better long-term temporal eyeblink representation than the frame-based method #OTHEREFR and the sliding window based methods #OTHEREFR .

Moreover, under the proposed framework, eyeblink features can be facilitated via face's global context (e.g., head pose and illumination condition) with joint optimization and interaction, while previous works that utilize a sequential manner cannot.

From Table 2 , it can also be summarized that the landmark-based methods #OTHEREFR perform poorer than the region-based counterparts.

We think that one essential reason is that landmark detection is unreliable under unconstrained conditions.

Our method is region-based, but localizes eye region in an implicit way where global face context is included and thus becomes more robust.

• InstBlink also outperforms others on Inst-AP.

We think that is because our framework can better model the longterm spatio-temporal instance representations, while the counterparts achieve tracking under a tracking-by-detection framework, which contains limited spatio-temporal modeling and may suffer from heavy occlusion. Inference speed analysis.

The result is listed in Table 3 , assuming the 4 compared methods use InsightFace for face & landmark detection and InstBlink inferences within a clip length of 36.

It can be seen that InstBlink is also of high inference speed (i.e., 112 FPS for network forwarding) while the real-time capacity of other methods is not superior.

Real-Time Multi-Person Eyeblink Detection In The Wild For Untrimmed Video

Abstract: Real-time eyeblink detection in the wild can widely serve for fatigue detection, face anti-spoofing, emotion analysis, etc. The existing research efforts generally focus on singleperson cases towards trimmed video. However, multiperson scenario within untrimmed videos is also important for practical applications, which has not been well concerned yet. To address this, we shed light on this research field for the first time with essential contributions on dataset, theory, and practices. In particular, a large-scale dataset termed MPEblink that involves 686 untrimmed videos with 8748 eyeblink events is proposed under multiperson conditions. The samples are captured from unconstrained films to reveal "in the wild" characteristics. Meanwhile, a real-time multi-person eyeblink detection method is also proposed. Being different from the existing counterparts, our proposition runs in a one-stage spatio-temporal way with end-to-end learning capacity. Specifically, it simultaneously addresses the sub-tasks of face detection, face tracking, and human instance-level eyeblink detection. This paradigm holds 2 main advantages: (1) eyeblink features can be facilitated via the face's global context (e.g., head pose and illumination condition) with joint optimization and interaction, and (2) addressing these sub-tasks in parallel instead of sequential manner can save time remarkably to meet the real-time running requirement. Experiments on MPEblink verify the essential challenges of real-time multiperson eyeblink detection in the wild for untrimmed video. Our method also outperforms existing approaches by large margins and with a high inference speed.

Towards Real-Time Eyeblink Detection In The Wild:Dataset,Theory And Practices

Abstract: Effective and real-time eyeblink detection is of wide-range applications, such as deception detection, drive fatigue detection, face anti-spoofing, etc. Although numerous of efforts have already been paid, most of them focus on addressing the eyeblink detection problem under the constrained indoor conditions with the relative consistent subject and environment setup. Nevertheless, towards the practical applications eyeblink detection in the wild is more required, and of greater challenges. However, to our knowledge this has not been well studied before. In this paper, we shed the light to this research topic. A labelled eyeblink in the wild dataset (i.e., HUST-LEBW) of 673 eyeblink video samples (i.e., 381 positives, and 292 negatives) is first established by us. These samples are captured from the unconstrained movies, with the dramatic variation on human attribute, human pose, illumination condition, imaging configuration, etc. Then, we formulate eyeblink detection task as a spatial-temporal pattern recognition problem. After locating and tracking human eye using SeetaFace engine and KCF tracker respectively, a modified LSTM model able to capture the multi-scale temporal information is proposed to execute eyeblink verification. A feature extraction approach that reveals appearance and motion characteristics simultaneously is also proposed. The experiments on HUST-LEBW reveal the superiority and efficiency of our approach. It also verifies that, the existing eyeblink detection methods cannot achieve satisfactory performance in the wild.


We chose to use the R language #REFR , due to the familiarity of the authors with the language and a preference for its package architecture.

In this section, we discuss how our design principles are translated into Kaiaulu's specific design decisions.

In the following section, we fully flesh out Kaiaulu's modules and features.

Batch Mode, Interactive Mode, and Literate Programming in Kaiaulu.

Minimally, the structure of an R package consists of the package metadata and its API.

In addition, the R ecosystem encourages and promotes best practices to include documentation packages called vignettes, which leads R users to expect an API and R Notebooks when installing packages from CRAN (The Comprehensive R Archive Network).

#OTHEREFR CRAN treats R Notebooks as first class citizens in an R package #OTHEREFR showing on each package's website any R Notebooks available.

Because of R package structure, complying with familiar software abstractions (see Section 2.5) automatically brings the benefits of literate programming (see Section 2.3).

Building The Msr Tool Kaiaulu: Design Principles And Experiences

Abstract: Background: Since Alitheia Core was proposed and subsequently retired, tools that support empirical studies of software projects continue to be proposed, such as Codeface, Codeface4Smells, Grimoire-Lab and SmartSHARK, but they all make different design choices and provide overlapping functionality. Aims: We seek to understand the design decisions adopted by these tools-the good and the bad-along with their consequences, to understand why their authors reinvented functionality already present in other tools, and to help inform the design of future tools. Method: We used action research to evaluate the tools, and to determine a set of principles and anti-patterns to motivate a new tool design. Results: We identified 7 major design choices among the tools: 1) Abstraction Debt, 2) the use of Project Configuration Files, 3) the choice of Batch or Interactive Mode, 4) Minimal Paths to Data, 5) Familiar Software Abstractions, 6) Licensing and 7) the Perils of Code Reuse. Building on the observed good and bad design decisions, we created our own tool architecture and implemented it as an R package. Conclusions: Tools should not require onerous setup for users to obtain data. Authors should consider the conventions and abstractions used by their chosen language and build upon these instead of redefining them. Tools should encourage best practices in experiment reproducibility by leveraging self-contained and readable schemas that are used for tool automation, and reuse must be done with care to avoid depending on dead code.

Xnmt: The Extensible Neural Machine Translation Toolkit

Abstract: This paper describes XNMT, the eXtensible Neural Machine Translation toolkit. XNMT distin- guishes itself from other open-source NMT toolkits by its focus on modular code design, with the purpose of enabling fast iteration in research and replicable, reliable results. In this paper we describe the design of XNMT and its experiment configuration system, and demonstrate its utility on the tasks of machine translation, speech recognition, and multi-tasked machine translation/parsing. XNMT is available open-source at this https URL


-Number Of Branches In The Ladder Self-Attention Block.

The top-1 accuracy of our PSLT and DeiT-S on both the ImageNet and ImageNet-V2 validation set conforms to the linear fit in #REFR .

As described in Section 3.1, our proposed PSLT is composed of the proposed ladder self-attention block, Light FFN and the pixel-adaptive fusion module.

Table 15 shows the performance of models by stacking the proposed modules one by one, which demonstrates the effectiveness of the modules.

Compared to the ViT, although applying our ladder self-attetion (LSA) and Light FFN (LFFN) does not improve the performance, the number of parameters and FLOPs are largely reduced, where the parameters have been reduced by 9 times and 15 times respectively, and the FLOPs have been reduced by 27 times and 37 times respectively.

The pixel-adaptive fusion module (PAFM) improves the model performance by integrating information of all branches in the block.

Our PSLT (including DeiT-S) achieves smaller improvement over ViT-B on ImageNet-V2 validation (2% and 1% top-1 accuracy improvement on ImageNet and ImageNet-V2 validation set, respectively).

Such a phenomenon has been similarly observed in #OTHEREFR , #OTHEREFR .

As described in #OTHEREFR , this shows the model generalization ability is challenged on different validation set.

Pslt: A Light-Weight Vision Transformer With Ladder Self-Attention And Progressive Shift

Abstract: Vision Transformer (ViT) has shown great potential for various visual tasks due to its ability to model long-range dependency. However, ViT requires a large amount of computing resource to compute the global self-attention. In this work, we propose a ladder self-attention block with multiple branches and a progressive shift mechanism to develop a lightweight transformer backbone that requires less computing resources (e.g. a relatively small number of parameters and FLOPs), termed Progressive Shift Ladder Transformer (PSLT). First, the ladder self-attention block reduces the computational cost by modelling local self-attention in each branch. In the meanwhile, the progressive shift mechanism is proposed to enlarge the receptive field in the ladder self-attention block by modelling diverse local self-attention for each branch and interacting among these branches. Second, the input feature of the ladder self-attention block is split equally along the channel dimension for each branch, which considerably reduces the computational cost in the ladder self-attention block (with nearly 1 3 the amount of parameters and FLOPs), and the outputs of these branches are then collaborated by a pixel-adaptive fusion. Therefore, the ladder self-attention block with a relatively small number of parameters and FLOPs is capable of modelling long-range interactions. Based on the ladder self-attention block, PSLT performs well on several vision tasks, including image classification, objection detection and person re-identification. On the ImageNet-1k dataset, PSLT achieves a top-1 accuracy of 79.9% with 9.2M parameters and 1.9G FLOPs, which is comparable to several existing models with more than 20M parameters and 4G FLOPs. Code is available at https://isee-ai.cn/wugaojie/PSLT.html.

Do Imagenet Classifiers Generalize To Imagenet?

Abstract: We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.


A framework used for dataset generation was also proposed for cooperatively exploiting both visual and wireless data #REFR .

The traditional vision-based object tracking algorithms, such as the sparse representation and correlation filtering, have difficulty in accurately locating high-mobility UEs in real-time #OTHEREFR .

Furthermore, in complex environments, tracking multiple UEs in the face of blockage and uneven light, the localization accuracy of UEs tends to degrade significantly.

As a result, the traditional CV-related ML algorithms cannot satisfy the high location accuracy required by mmWave communications.

Finally, gathering abundant labelled data from real-world environments including both visual data and wireless signals to train ML models is still challenging.

Nevertheless, vision-assisted beam management methods that predict the optimal mmWave beams have been investigated in the last few years.

However, these methods may still be plagued by a number of issues:

• The robustness of the existing ML models for visionassisted beam management has to be improved #OTHEREFR .

When the classical image classification models designed for prediction are used for mmWave link blockage prediction and beam prediction, the target accuracy cannot be always satisfied, for example due to the over-fitting issues.

In this context, a preliminary study was conducted in #OTHEREFR by relying on a simple dataset, for investigating visionassisted beam management in multi-user scenarios.

• The scalability of the methods is not guaranteed in practical scenarios.

Vision-Assisted Mmwave Beam Management For Next-Generation Wireless Systems: Concepts, Solutions And Open Challenges

Abstract: Beamforming techniques have been widely used in the millimeter wave (mmWave) bands to mitigate the path loss of mmWave radio links as the narrow straight beams by directionally concentrating the signal energy. However, traditional mmWave beam management algorithms usually require excessive channel state information overhead, leading to extremely high computational and communication costs. This hinders the widespread deployment of mmWave communications. By contrast, the revolutionary vision-assisted beam management system concept employed at base stations (BSs) can select the optimal beam for the target user equipment (UE) based on its location information determined by machine learning (ML) algorithms applied to visual data, without requiring channel information. In this paper, we present a comprehensive framework for a vision-assisted mmWave beam management system, its typical deployment scenarios as well as the specifics of the framework. Then, some of the challenges faced by this system and their efficient solutions are discussed from the perspective of ML. Next, a new simulation platform is conceived to provide both visual and wireless data for model validation and performance evaluation. Our simulation results indicate that the vision-assisted beam management is indeed attractive for next-generation wireless systems.

Viwi Vision-Aided Mmwave Beam Tracking: Dataset, Task, And Baseline Solutions

Abstract: Vision-aided wireless communication is motivated by the recent advances in deep learning and computer vision as well as the increasing dependence on line-of-sight links in millimeter wave (mmWave) and terahertz systems. By leveraging vision, this new research direction enables an interesting set of new capabilities such as vision-aided mmWave beam and blockage prediction, proactive hand-off, and resource allocation among others. These capabilities have the potential of reliably supporting highly-mobile applications such as vehicular/drone communications and wireless virtual/augmented reality in mmWave and terahertz systems. Investigating these interesting applications, however, requires the development of special dataset and machine learning tasks. Based on the Vision-Wireless (ViWi) dataset generation framework [1], this paper develops an advanced and realistic scenario/dataset that features multiple base stations, mobile users, and rich dynamics. Enabled by this dataset, the paper defines the vision-wireless mmWave beam tracking task (ViWi-BT) and proposes a baseline solution that can provide an initial benchmark for the future ViWi-BT algorithms.


Additionally, it is important that function f that can learn a fruitful representation and complex data dependencies #REFR .

where F * s is the final selected feature set, J is a desired loss function, and f (x

Fs ; θ) is a classification function parameterized by θ aiming at estimating the target for the i-th sample using a subset of features x (i) Fs . Solving this optimization problem can be a challenging task.

As the choice of feature subset F s grows exponentially with increasing number of features d, solving Equation 1 is a NP-hard problem.

We choose artificial neural networks due to their high expressive power; a simple one-hidden layer feed-forward neural network is known to be a universal approximator #OTHEREFR .

Finally, as we aim to select features in a computationally efficient manner, in this paper, we choose sparse neural networks to represent the data and perform feature selection.

Supervised Feature Selection With Neuron Evolution In Sparse Neural Networks

Abstract: Feature selection that selects an informative subset of variables from data not only enhances the model interpretability and performance but also alleviates the resource demands. Recently, there has been growing attention on feature selection using neural networks. However, existing methods usually suffer from high computational costs when applied to high-dimensional datasets. In this paper, inspired by evolution processes, we propose a novel resource-efficient supervised feature selection method using sparse neural networks, named "NeuroFS". By gradually pruning the uninformative features from the input layer of a sparse neural network trained from scratch, NeuroFS derives an informative subset of features efficiently. By performing several experiments on 11 low and high-dimensional real-world benchmarks of different types, we demonstrate that NeuroFS achieves the highest ranking-based score among the considered state-of-the-art supervised feature selection models. The code is available on GitHub 1 .

Lassonet: Neural Networks With Feature Sparsity

Abstract: We propose a neural network model, with a separate linear (residual) term, that explicitly bounds the input layer weights for a feature by the linear weight for that feature. The model can be seen as a modification of so-called residual neural networks to produce a path of models that are feature-sparse, that is, use only a subset of the features. This is analogous to the solution path from the usual Lasso ($\ell_1$-regularized) linear regression. We call the proposed procedure {\tt LassoNet} and develop a projected proximal gradient algorithm for its optimization. This approach can sometimes give as low or lower test error than a standard neural network, and its feature selection provides more interpretable solutions. We illustrate the method using both simulated and real data examples, and show that it is often able to achieve competitive performance with a much smaller number of input feature


The assumption of the MDP does not hold on most realworld systems due to delays and hence those maybe only partially observable #REFR .

In this force trajectory tracking problem, the trajectories are restricted to sinusoidal waveforms between 0.05 to 0.35 Hz.

This is because (as discussed in Section III-B) the input motor current is saturated to constrain the pendulum within safe position bounds and avoid any damage to the ball screw during learning.

In addition, the initial random policy with abruptly switching high current values can induce vibrationbased damage to the system.

This problem is often countered by adding more information in the observation space of the agent.

The observation vector includes the following: the absolute encoder-based angle, q [rad], angular velocity,q

where the divisor, D = 10 6 , maintains an episodic reward between [-30, 0] .

In absence of this, the summation of reward per time step was observed to undergo catastrophic forgetting #OTHEREFR resulting in unlearning and poor performance.

Real-Time Model-Free Deep Reinforcement Learning For Force Control Of A Series Elastic Actuator

Abstract: Many state-of-the art robotic applications utilize series elastic actuators (SEAs) with closed-loop force control to achieve complex tasks such as walking, lifting, and manipulation. Model-free PID control methods are more prone to instability due to nonlinearities in the SEA where cascaded model-based robust controllers can remove these effects to achieve stable force control. However, these model-based methods require detailed investigations to characterize the system accurately. Deep reinforcement learning (DRL) has proved to be an effective model-free method for continuous control tasks, where few works deal with hardware learning. This paper describes the training process of a DRL policy on hardware of an SEA pendulum system for tracking force control trajectories from 0.05-0.35 Hz at 50 N amplitude using the Proximal Policy Optimization (PPO) algorithm. Safety mechanisms are developed and utilized for training the policy for 12 hours (overnight) without an operator present within the full 21 hours training period. The tracking performance is evaluated showing improvements of 25 N in mean absolute error when comparing the first 18 min. of training to the full 21 hours for a 50 N amplitude, 0.1 Hz sinusoid desired force trajectory. Finally, the DRL policy exhibits better tracking and stability margins when compared to a model-free PID controller for a 50 N chirp force trajectory.

Challenges Of Real-World Reinforcement Learning

Abstract: Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are often hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. We present a set of nine unique challenges that must be addressed to productionize RL to real world problems. For each of these challenges, we specify the exact meaning of the challenge, present some approaches from the literature, and specify some metrics for evaluating that challenge. An approach that addresses all nine challenges would be applicable to a large number of real world problems. We also present an example domain that has been modified to present these challenges as a testbed for practical RL research.


E 2 NAS successfully overperformed previous works on the three datasets available in NAS-Bench-201 #REFR (e.g., a +29.45 % top 1 accuracy improvement on ImageNet-16-120).

where U is a function updating the hidden state ℎ obtained by aggregating all its predecessors with function G.

where L is the Cross-Entropy loss, R is a 2 regularization term, is a value that balances between optimizing the current architecture (exploitation) or preventing other alternatives from vanishing (exploration).

#OTHEREFR proposed iDARTS, a solution that reformulates the optimization process of DARTS with a Neumann-approximation of the Implicit Theorem Function (IFT) #OTHEREFR . Concretely, the architectural parameter gradients Δ L (see Eq. 3) are calculated as follows:

where L is the validation loss, L is the training loss, and are the network weights.

However, it is computationally intensive to compute the inverse of the Hessian matrix

Efficient Automation Of Neural Network Design: A Survey On Differentiable Neural Architecture Search

Abstract: In the past few years, Differentiable Neural Architecture Search (DNAS) rapidly imposed itself as the trending approach to automate the discovery of deep neural network architectures. This rise is mainly due to the popularity of DARTS, one of the first major DNAS methods. In contrast with previous works based on Reinforcement Learning or Evolutionary Algorithms, DNAS is faster by several orders of magnitude and uses fewer computational resources. In this comprehensive survey, we focus specifically on DNAS and review recent approaches in this field. Furthermore, we propose a novel challenge-based taxonomy to classify DNAS methods. We also discuss the contributions brought to DNAS in the past few years and its impact on the global NAS field. Finally, we conclude by giving some insights into future research directions for the DNAS field. CCS Concepts: • General and reference → Surveys and overviews; • Computing methodologies → Search methodologies; Computer vision.

Nas-Bench-201: Extending The Scope Of Reproducible Neural Architecture Search

Abstract: Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years. It could be time to take a step back and analyze the good and bad aspects in the field of NAS. A variety of algorithms search architectures under different search space. These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization. This raises a comparability problem when comparing the performance of various NAS algorithms. NAS-Bench-101 has shown success to alleviate this problem. In this work, we propose an extension to NAS-Bench-101: NAS-Bench-102 with a different search space, results on multiple datasets, and more diagnostic information. NAS-Bench-102 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. The design of our search space is inspired by the one used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph. Each edge here is associated with an operation selected from a predefined operation set. For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-102 includes all possible architectures generated by 4 nodes and 5 associated operation options, which results in 15,625 neural cell candidates in total. The training log using the same setup and the performance for each architecture candidate are provided for three datasets. This allows researchers to avoid unnecessary repetitive training for selected architecture and focus solely on the search algorithm itself. The training time saved for every architecture also largely improves the efficiency of most NAS algorithms and presents a more computational cost friendly NAS community for a broader range of researchers. We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms. In further support of the proposed NAS-Bench-102, we have analyzed it from many aspects and benchmarked 10 recent NAS algorithms, which verify its applicability.


NDDR-CNN #REFR proposes a novel CNN structure to learn a discriminative feature embedding for each task.

As we will explain in the next section, directly extending the one-shot weight-sharing strategy from single-task NAS to multi-task NAS incurs interferences among tasks, leading to significant performance degradation.

AutoTaskFormer tackles this problem using a novel NAS pipeline that involves two search stages instead of one, specifically designed for multi-task learning.

Multi-task Learning (MTL). MTL aims to learn parameter sharing across multiple tasks.

For example, Cross-Stitch #OTHEREFR manually employs an additional group of shared units to merge multiple task backbones.

Sluice #OTHEREFR jointly learns a latent multi-task architecture and task-specific models.

There are several works that design task-sharing architectures through NAS.

In particular, MTL-NAS #OTHEREFR starts with two taskspecific networks and seeks optimal edges between the intertask branches.

AdaShare #OTHEREFR learns the sharing weights through a task-specific policy that selectively chooses which layers to execute for a given task.

AutoMTL #OTHEREFR proposes a source-to-source compiler that transforms the backbone CNN into a supermodel.

However, due to the bottom-up approach of combining task-specific networks, the sharing between tasks and networks needs to be meticulously designed and hence can only support a small number of tasks.

Autotaskformer: Searching Vision Transformers For Multi-Task Learning

Abstract: Vision Transformers have shown great performance in single tasks such as classification and segmentation. However, real-world problems are not isolated, which calls for vision transformers that can perform multiple tasks concurrently. Existing multi-task vision transformers are handcrafted and heavily rely on human expertise. In this work, we propose a novel one-shot neural architecture search framework, dubbed AutoTaskFormer (Automated Multi-Task Vision TransFormer), to automate this process. AutoTask-Former not only identifies the weights to share across multiple tasks automatically, but also provides thousands of well-trained vision transformers with a wide range of parameters (e.g., number of heads and network depth) for deployment under various resource constraints. Experiments on both small-scale (2-task Cityscapes and 3-task NYUv2) and large-scale (16-task Taskonomy) datasets show that Auto-TaskFormer outperforms state-of-the-art handcrafted vision transformers in multi-task learning. The entire code and models will be open-sourced.

Nddr-Cnn: Layerwise Feature Fusing In Multi-Task Cnns By Neural Discriminative Dimensionality Reduction

Abstract: In this paper, we propose a novel Convolutional Neural Network (CNN) structure for general-purpose multi-task learning (MTL), which enables automatic feature fusing at every layer from different tasks. This is in contrast with the most widely used MTL CNN structures which empirically or heuristically share features on some specific layers (e.g., share all the features except the last convolutional layer). The proposed layerwise feature fusing scheme is formulated by combining existing CNN components in a novel way, with clear mathematical interpretability as discriminative dimensionality reduction, which is referred to as Neural Discriminative Dimensionality Reduction (NDDR). Specifically, we first concatenate features with the same spatial resolution from different tasks according to their channel dimension. Then, we show that the discriminative dimensionality reduction can be fulfilled by 1x1 Convolution, Batch Normalization, and Weight Decay in one CNN. The use of existing CNN components ensures the end-to-end training and the extensibility of the proposed NDDR layer to various state-of-the-art CNN architectures in a "plug-and-play" manner. The detailed ablation analysis shows that the proposed NDDR layer is easy to train and also robust to different hyperparameters. Experiments on different task sets with various base network architectures demonstrate the promising performance and desirable generalizability of our proposed method. The code of our paper is available at https://github.com/ethanygao/NDDR-CNN.


In #REFR , the vanilla attention matrix is based on the calculation of all the query-key pairs.

According to #OTHEREFR , the computation of the vanilla attention matrix A ∈ R n×n is based on the dot-product.

It is defined as A = softmax QK √ d ; Q = TW q , K = TW k , where the query matrix Q ∈ R n×d and key matrix K ∈ R n×d are generated by the linear projection of the input token matrix T ∈ R n×dm based on the learnable weights matrices W q ∈ R dm×d and W k ∈ R dm×d . n indicates the total number of input tokens.

d represents the embedding dimension and d m denotes the dimension of an input token.

The new value matrix V new ∈ R n×d can be obtained via

V new = A V; V = TW v , where the value matrix V ∈ R n×d and W v ∈ R dm×d .

However, in the proposed Causal Semantics Extractor, only the top κ most similar keys and values for each query are used to compute the causal attention matrix.

Similar to #OTHEREFR , all the queries and keys are calculated by the dot-product.

Then, the row-wise top κ elements are used for the softmax calculation.

In the proposed Causal Semantics Extractor, the value matrix #OTHEREFR , for the input video, and VisualAtten(•) indicates a visual attention mechanism based on the element-wise multiplication.

X mul = FC(Z ta Z va ), where denotes the operation of feature concatenation and FC(•) indicates a fully connected layer.

Causalainer: Causal Explainer For Automatic Video Summarization

Abstract: The goal of video summarization is to automatically shorten videos such that it conveys the overall story without losing relevant information. In many application scenarios, improper video summarization can have a large impact. For example in forensics, the quality of the generated video summary will affect an investigator's judgment while in journalism it might yield undesired bias. Because of this, modeling explainability is a key concern. One of the best ways to address the explainability challenge is to uncover the causal relations that steer the process and lead to the result. Current machine learning-based video summarization algorithms learn optimal parameters but do not uncover causal relationships. Hence, they suffer from a relative lack of explainability. In this work, a Causal Explainer, dubbed Causalainer, is proposed to address this issue. Multiple meaningful random variables and their joint distributions are introduced to characterize the behaviors of key components in the problem of video summarization. In addition, helper distributions are introduced to enhance the effectiveness of model training. In visual-textual input scenarios, the extra input can decrease the model performance. A causal semantics extractor is designed to tackle this issue by effectively distilling the mutual information from the visual and textual inputs. Experimental results on commonly used benchmarks demonstrate that the proposed method achieves state-of-the-art performance while being more explainable.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


Time Constrained approaches have been proposed that limited the amount of time available for the AutoML system to search through the space, however, this came with a trade-off -balancing between time and pipeline accuracy #REFR . Thus the resulting pipeline was mostly sub-optimal.

While the iterative search (such as grid search or random search) has shown success, however, the search space explodes when pipelines contain many steps with an even greater number of choices.

This explosion in the number of candidate pipelines is prohibitive of compute time and cost of resources needed to support searching through large hyper-parameter search space.

The significant time cost of AutoML systems comes from evaluating the end-to-end pipeline.

As the quality of the pipeline is only measured after the completion of the last step.

Etop: Early Termination Of Pipelines For Faster Training Of Automl System

Abstract: Recent advancements in software and hardware technologies have enabled the use of AI/ML models in everyday applications has significantly improved the quality of service rendered. However, for a given application, finding the right AI/ML model is a complex and costly process, that involves the generation, training, and evaluation of multiple interlinked steps (called pipelines), such as data pre-processing, feature engineering, selection, and model tuning. These pipelines are complex (in structure) and costly (both in compute resource and time) to execute end-to-end, with a hyper-parameter associated with each step. Au-toML systems automate the search of these hyper-parameters but are slow, as they rely on optimizing the pipelines' end output. We propose "the eTOP Framework" which works on top of any AutoML system and decides whether or not to execute the pipeline to the end or terminate at an intermediate step. Experimental evaluation on 26 benchmark datasets 3 and integration of eTOPwith MLBox 4 reduces the training time of the AutoML system upto 40x than baseline MLBox.

Hyper-Parameter Tuning Under A Budget Constraint

Abstract: We study a budgeted hyper-parameter tuning problem, where we optimize the tuning result under a hard resource constraint. We propose to solve it as a sequential decision making problem, such that we can use the partial training progress of configurations to dynamically allocate the remaining budget. Our algorithm combines a Bayesian belief model which estimates the future performance of configurations, with an action-value function which balances exploration-exploitation tradeoff, to optimize the final output. It automatically adapts the tuning behaviors to different constraints, which is useful in practice. Experiment results demonstrate superior performance over existing algorithms, including the-state-of-the-art one, on real-world tuning tasks across a range of different budgets.


Both the part-id and surface-id fields are encoded with a d-dimensional Sinusoidal Positional Encoding #REFR .

There are three atomic part types: long profile, short profile and angle bracket.

2) Surface Nodes: Different to the one in #OTHEREFR , we associate each surface node

v s i ∈ V s with the features φ (v s i ) = [surface-type ∈ N, surface-id ∈ R d ].

There are two surface types (long and short) for profiles and one (lateral) for brackets.

3) Surface-to-Surface Edges: We design a fullyconnected graph for all surface nodes V s to capture the relation between untouched surfaces, which is more fine-grained than those in #OTHEREFR with only connects between touched surfaces.

These edges are assigned with a feature φ (e i ) ∈ R, indicating the relation between the two surfaces: φ (e i ) = relative distance (parallel); 1 (belong to the same part); −1 (orthogonal); 0 (same-surface loop).

4) Surface-to-Part Edges: These connect each surface and part node pair

(v s i , v p j ) ∈ V s × V p , where surface v s i belongs to the part v p

Efficient And Feasible Robotic Assembly Sequence Planning Via Graph Representation Learning

Abstract: Automatic Robotic Assembly Sequence Planning (RASP) can significantly improve productivity and resilience in modern manufacturing along with the growing need for greater product customization. One of the main challenges in realizing such automation resides in efficiently finding solutions from a growing number of potential sequences for increasingly complex assemblies. Besides, costly feasibility checks are always required for the robotic system. To address this, we propose a holistic graphical approach including a graph representation called Assembly Graph for product assemblies and a policy architecture, Graph Assembly Processing Network, dubbed GRACE for assembly sequence generation. Secondly, we use GRACE to extract meaningful information from the graph input and predict assembly sequences in a step-by-step manner. In experiments, we show that our approach can predict feasible assembly sequences across product variants of aluminum profiles based on data collected in simulation of a dual-armed robotic system. We further demonstrate that our method is capable of detecting infeasible assemblies, substantially alleviating the undesirable impacts from false predictions, and hence facilitating real-world deployment soon. Code and training data will be open-sourced. Equal Contribution.

Attention Is All You Need


All models are trained using Python 3.10.4, Pytorch 1.11.0 #REFR , and an Nvidia Tesla V100 GPU.

The learning rate for the noise estimator decreases by half every 10 epochs with 50 epochs trained, while the learning rate for the denoising network is halved every 20 epochs with 100 epochs trained.

As for the hyper-parameter in the adaptive re-visible loss, we set λ = 3 as the initial value and progressively increase it to 11.

The denoising network and noise estimation are jointly optimized during training.

The patches of size 128 × 128 are randomly cropped for training.

Note that the details of the global masker and global mask mapper remain the same as Blind2Unblind #OTHEREFR .

Datasets We consider five types of Poisson-Gaussian noise in synthetic noise estimation: (1) P G1 : α = 0.1, σ = 0.02, (2) P G2 : α = 0.1, σ = 0.0002, (3) P G3 : α = 0.05, σ = 0.02, (4) P G4 : α = 0.05, σ = 0.0002, (5) P G5 : α = 0.01, σ = 0.02.

For grayscale images, we use BSD400 #OTHEREFR for the training set, and for sRGB images, we use CBSD432 #OTHEREFR .

The noise levels are estimated on standard BSD68 #OTHEREFR and CBSD68 #OTHEREFR for grayscale and sRGB images, respectively.

For synthetic denoising, we use ILSVRC2012 #OTHEREFR validation set for sRGB image denoising and BSD400 #OTHEREFR for grayscale image denoising.

Specifically, following the setting in #OTHEREFR , we select 44328 images with sizes between 256 × 256 and 512×512 pixels from ILSVRC2012 validation set for training.

Blind2Sound: Self-Supervised Image Denoising Without Residual Noise

Abstract: Self-supervised blind denoising for Poisson-Gaussian noise remains a challenging task. Pseudo-supervised pairs constructed from single noisy images re-corrupt the signal and degrade the performance. The visible blindspots solve the information loss in masked inputs. However, without explicitly noise sensing, mean square error as an objective function cannot adjust denoising intensities for dynamic noise levels, leading to noticeable residual noise. In this paper, we propose Blind2Sound, a simple yet effective approach to overcome residual noise in denoised images. The proposed adaptive re-visible loss senses noise levels and performs personalized denoising without noise residues while retaining the signal lossless. The theoretical analysis of intermediate medium gradients guarantees stable training, while the Cramer Gaussian loss acts as a regularization to facilitate the accurate perception of noise levels and improve the performance of the denoiser. Experiments on synthetic and real-world datasets show the superior performance of our method, especially for single-channel images. The code is available in supplementary materials.

Pytorch: An Imperative Style, High-Performance Deep Learning Library

Abstract: Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. ::: In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. ::: We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.


We utilize the AdamW optimizer [20] and the CosineAnnealing learning rate scheduler #REFR with their default parameters.

The FlyingThings3D dataset #OTHEREFR is used as the training dataset, which contains 800 pairs of synthetic RGBD images.

The Middle-bury2014 dataset #OTHEREFR is used as the testing dataset, which contains 23 real-world RGBD images.

There is a significant domain gap between the synthetic and real-world images, making it suitable for evaluating the generalizability of the pre-trained DfF models.

We simulate focal stacks with the RGBD images for training and testing, using a stack size of #OTHEREFR The lens data comes from lensnet.com #OTHEREFR 10.

The focus distances are chosen linearly from the minimum (20 cm) and maximum (20 m) depth range of each image, randomly perturbed.

The training batch size is 16, and the initial learning rate is 1e −4 .

Each DfF model is trained for 400 epochs on a single A100 GPU.

Following training, we evaluate each model directly on the testing focal stacks without any fine-tuning.

In real-world experiments, we use the Canon EOS R camera and the RF 50mm F/1.8 lens discussed in Section 4.

The training procedure keeps the same as the previous experiment.

Aberration-Aware Depth-From-Focus

Abstract: Computer vision methods for depth estimation usually use simple camera models with idealized optics. For modern machine learning approaches, this creates an issue when attempting to train deep networks with simulated data, especially for focus-sensitive tasks like Depth-from-Focus. In this work, we investigate the domain gap caused by off-axis aberrations that will affect the decision of the best-focused frame in a focal stack. We then explore bridging this domain gap through aberration-aware training (AAT). Our approach involves a lightweight network that models lens aberrations at different positions and focus distances, which is then integrated into the conventional network training pipeline. We evaluate the generality of pretrained models on both synthetic and real-world data. Our experimental results demonstrate that the proposed AAT scheme can improve depth estimation accuracy without fine-tuning the model or modifying the network architecture. Our code and models will be made publicly available.

Sgdr: Stochastic Gradient Descent With Warm Restarts

Abstract: Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR