A similar discrepancy between generated data quality and downstream task performance has been reported in a prior image classification task #REFR .

A significant boost still exists when the volume of true data gets larger, which surpasses the improvement by SD para .

This indicates that a suitable data filtering can improve both training efficiency and image captioning performance when true labeled data is limited.

However, the improvement for SD base and SD para are not significant.

And the image captioning performance even decreases in some cases for the Transformer-based model (see Table 11 in Appendix).

Therefore, CLIPScore and the other two criteria are not golden standards to select highquality data that are suitable for the captioning task, even though they have been testified to perform well in image-caption quality evaluation #OTHEREFR .

Authors found that although the GAN-generated images receive high scores close to those of true images, the classification model trained on fully synthetic images has a much lower accuracy than those trained on true images.

Following this thread, we calculate the three metrics for both COCO data and synthetic data.

We have a similar finding that three quality measures under the synthetic data are close level to those under true data (see Table 5 ).

However, when completely replacing the true data with the synthetic data as the training (e.g., in Experiment 1), the performance is quite lower than that of the true data (i.e., CIDEr score: 81.0 vs. 92.4).

In this way, we extend findings in #OTHEREFR 's study to the image captioning task.

Multimodal Data Augmentation For Image Captioning Using Diffusion Models

Abstract: Image captioning, an important visionlanguage task, often requires a tremendous number of finely labeled image-caption pairs for learning the underlying alignment between images and texts. In this paper, we proposed a multimodal data augmentation method, leveraging a recent text-to-image model called Stable Diffusion, to expand the training set via high-quality generation of image-caption pairs. Extensive experiments on the MS COCO dataset demonstrate the advantages of our approach over several benchmark methods, and particularly a significant boost when having fewer training instances. In addition, models trained on our augmented datasets also outperform prior unpaired image captioning methods by a large margin. Finally, further improvement regarding the training efficiency and effectiveness can be obtained after intentionally filtering the generated data based on quality assessment.

Classification Accuracy Score For Conditional Generative Models

Abstract: Deep generative models (DGMs) of images are now sufficiently mature that they produce nearly photorealistic samples and obtain scores similar to the data distribution on heuristics such as Frechet Inception Distance (FID). These results, especially on large-scale datasets such as ImageNet, suggest that DGMs are learning the data distribution in a perceptually meaningful space and can be used in downstream tasks. To test this latter hypothesis, we use class-conditional generative models from a number of model classes—variational autoencoders, autoregressive models, and generative adversarial networks (GANs)—to infer the class labels of real data. We perform this inference by training an image classifier using only synthetic data and using the classifier to predict labels on real data. The performance on this task, which we call Classification Accuracy Score (CAS), reveals some surprising results not identified by traditional metrics and constitute our contributions. First, when using a state-of-the-art GAN (BigGAN-deep), Top-1 and Top-5 accuracy decrease by 27.9% and 41.6%, respectively, compared to the original data; and conditional generative models from other model classes, such as Vector-Quantized Variational Autoencoder-2 (VQ-VAE-2) and Hierarchical Autoregressive Models (HAMs), substantially outperform GANs on this benchmark. Second, CAS automatically surfaces particular classes for which generative models failed to capture the data distribution, and were previously unknown in the literature. Third, we find traditional GAN metrics such as Inception Score (IS) and FID neither predictive of CAS nor useful when evaluating non-GAN models. Furthermore, in order to facilitate better diagnoses of generative models, we open-source the proposed metric.


LiDAR point clouds, which contain rich spatial information, offer superior performance on 3D detection benchmarks compared to camerabased methods #REFR Caesar et al, 2020) .

Although image backbones pretrained on Ima-geNet provide semantically rich features, they fail to capture critical structural information.

The existing pre-training method #OTHEREFR introduces depth-relevant tasks, but such method excludes the view transformation module and benefits the backbone only.

View transformation module is of great significance, since it constructs 3D information and encodes 3D prior assumptions.

As the view transformation module is optimized only by the detection loss, the spatial information from the depth-pretrained backbone is not fully utilized.

Ideally, the image backbone and the view transformation should incorporate aligned geometry knowledge to enhance their performance.

Therefore, integrating LiDAR information into camera-based models is a natural step.

Recent works #OTHEREFR have explored LiDAR information in two paradigms.

The first paradigm #OTHEREFR projects LiDAR points onto the perspective view to provide additional depth supervision.

However, the task of depth estimation in 2D space is inherently ill-posed and challenging.

The second paradigm #OTHEREFR ) leverages a teacher-student knowledge distillation fashion.

Geometric-Aware Pretraining For Vision-Centric 3D Object Detection

Abstract: Multi-camera 3D object detection for autonomous driving is a challenging problem that has garnered notable attention from both academia and industry. An obstacle encountered in vision-based techniques involves the precise extraction of geometry-conscious features from RGB images. Recent approaches have utilized geometric-aware image backbones pretrained on depth-relevant tasks to acquire spatial information. However, these approaches overlook the critical aspect of view transformation, resulting in inadequate performance due to the misalignment of spatial knowledge between the image backbone and view transformation. To address this issue, we propose a novel geometricaware pretraining framework called GAPretrain. Our approach incorporates spatial and structural cues to camera networks by employing the geometric-rich modality as guidance during the pretraining phase. The transference of modal-specific attributes across different modalities is non-trivial, but we bridge this gap by using a unified bird's-eye-view (BEV) representation and structural hints derived from LiDAR point clouds to facilitate the pretraining process. GAPretrain serves as a plug-and-play solution that can be flexibly applied to multiple state-of-the-art detectors. Our experiments demonstrate the effectiveness and generalization ability of the proposed method. We achieve 46.2 mAP and 55.5 NDS on the nuScenes val set using the BEVFormer method, with a gain of 2.7 and 2.1 points, respectively. We also conduct experiments on various image backbones and view transformations to validate the efficacy of our approach.

Scalability In Perception For Autonomous Driving: Waymo Open Dataset

Abstract: The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the overall viability of the technology. In an effort to help align the research community's contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data captured across a range of urban and suburban geographies. It is 15x more diverse than the largest camera+LiDAR dataset available based on our proposed diversity metric. We exhaustively annotated this data with 2D (camera image) and 3D (LiDAR) bounding boxes, with consistent identifiers across frames. Finally, we provide strong baselines for 2D as well as 3D detection and tracking tasks. We further study the effects of dataset size and generalization across geographies on 3D detection methods. Find data, code and more up-to-date information at http://www.waymo.com/open.


For example, it is known that given a book the number of nodes and the average shortest path length of the book's word network is correlated with the style of certain literary eras #REFR .

These can be found in many writing book and generally come by way of applied psychology and accumulated wisdom of previous writers #OTHEREFR ]. Yet more books are being published than ever before.

We have data, and by careful analysis of this data we can create new tools for fiction writers so that fiction is no longer only an art, but also a science.

We've considered reader-communities and enjoyment-communities as ways to craft books readers will both choose to read and enjoy.

We've also shown the potential of our maturity-realism plane to coarsely identify books one might enjoy, and more generally the potential of PCA analysis to find untapped combinations of genres.

One can easily imagine an array of other data inspired approaches. Natural language processing represents a major area here.

If an author wishes to mimic the style of an era but their prose doesn't feel quite right they could look at the word network their book generates.

Perhaps they find that the size of their word network is too small, meaning they are using too few unique words. This can help "debug" prose.

Adapting this technique could help character sound distinct.

A common problem is that different character's dialogue sound the same simply because the same author wrote it.

If one can extract the each of the main character's dialogue (or the prose of their point of view scenes) then one can run those texts through text-author identification or natural language processing methods.

Revisiting The Relevance Of Traditional Genres: A Network Analysis Of Fiction Readers' Preferences

Abstract: We investigate how well traditional fiction genres like Fantasy, Thriller, and Literature represent readers' preferences. Using user data from Goodreads we construct a book network where two books are strongly linked if the same people tend to read or enjoy them both. We then partition this network into communities of similar books and assign each a list of subjects from The Open Library to serve as a proxy for traditional genres. Our analysis reveals that the network communities correspond to existing combinations of traditional genres, but that the exact communities differ depending on whether we consider books that people read or books that people enjoy. In addition, we apply principal component analysis to the data and find that the variance in the book communities is best explained by two factors: the maturity/childishness and realism/fantastical nature of the books. We propose using this maturity-realism plane as a coarse classification tool for stories.

Identification Of Literary Movements Using Complex Networks To Represent Texts

Abstract: The use of statistical methods to analyze large databases of text has been useful in unveiling patterns of human behavior and establishing historical links between cultures and languages. In this study, we identified literary movements by treating books published from 1590 to 1922 as complex networks, whose metrics were analyzed with multivariate techniques to generate six clusters of books. The latter correspond to time periods coinciding with relevant literary movements over the last five centuries. The most important factor contributing to the distinctions between different literary styles was the average shortest path length, in particular the asymmetry of its distribution. Furthermore, over time there has emerged a trend toward larger average shortest path lengths, which is correlated with increased syntactic complexity, and a more uniform use of the words reflected in a smaller power-law coefficient for the distribution of word frequency. Changes in literary style were also found to be driven by opposition to earlier writing styles, as revealed by the analysis performed with geometrical concepts. The approaches adopted here are generic and may be extended to analyze a number of features of languages and cultures.


XGBoost (eXtreme Gradient Boosting) #REFR is powered by a novel sparsity-aware algorithm and weighted quantile sketch. This allows it to scale to large datasets.

GBDTs have been a powerful technique to model tabular data, ever since their creation in 2001 #OTHEREFR .

GBDTs work by building an ensemble of decision trees, incrementally updated using gradient descent.

Due to their strong performance, many highperforming instantiations have been proposed.

LightGBM (Light Gradient Boosting Machine) #OTHEREFR is designed to be a lightweight gradientboosted tree implementation.

CatBoost (Categorical Boosting) #OTHEREFR combines a permutation-driven variant of boosting with a novel technique for processing categorical features.

In their survey on deep learning for tabular data, Borisov et al. described three types of tabular data #OTHEREFR .

Data transformation methods #OTHEREFR seek to encode the data into a format that is better-suited for neural nets.

When Do Neural Nets Outperform Boosted Trees On Tabular Data?

Abstract: Tabular data is one of the most commonly used types of data in machine learning. Despite recent advances in neural nets (NNs) for tabular data, there is still an active discussion on whether or not NNs generally outperform gradient-boosted decision trees (GBDTs) on tabular data, with several recent works arguing either that GBDTs consistently outperform NNs on tabular data, or vice versa. In this work, we take a step back and ask, 'does it matter?' We conduct the largest tabular data analysis to date, by comparing 19 algorithms across 176 datasets, and we find that the 'NN vs. GBDT' debate is overemphasized: for a surprisingly high number of datasets, either the performance difference between GBDTs and NNs is negligible, or light hyperparameter tuning on a GBDT is more important than selecting the best algorithm. Next, we analyze 965 metafeatures to determine what properties of a dataset make NNs or GBDTs better-suited to perform well. For example, we find that GBDTs are much better than NNs at handling skewed feature distributions, heavy-tailed feature distributions, and other forms of dataset irregularities. Our insights act as a guide for practitioners to decide whether or not they need to run a neural net to reach top performance on their dataset. Our codebase and all raw results are available at https://github.com/naszilla/tabzilla.

Xgboost: A Scalable Tree Boosting System

Abstract: Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.


Validating Hypothesis Model And Test Function

Intriguingly, we notice that both results from the hypothesis model generally show constant robustness even in a high-confidence adversarial attack #REFR fabricating unseen perturbation.

Here, we observe that the adversarial robustness of CF is inferior to that of CC, AC, and even Adv.

Intuitively, it is an obvious result since the test function violating Eq.

(7) forces feature representation to be the worst possible condition of extremely deviating from correct prediction.

For the prediction results for CC and AC, they show impressive robustness performance than Adv with large margins.

Since AC directly leverages the feature variation acquired from adversarial perturbation, they present better adversarial robustness than CC obtained from the test function outputting the worst-case counterfactuals on the feature variation.

Such robustness demonstrates the estimated causal features have ability to overcome various types of adversarial perturbation.

Demystifying Causal Features On Adversarial Examples And Causal Inoculation For Robust Network By Adversarial Instrumental Variable Regression

Abstract: The origin of adversarial examples is still inexplicable in research fields, and it arouses arguments from various viewpoints, albeit comprehensive investigations. In this paper, we propose a way of delving into the unexpected vulnerability in adversarially trained networks from a causal perspective, namely adversarial instrumental variable (IV) regression. By deploying it, we estimate the causal relation of adversarial prediction under an unbiased environment dissociated from unknown confounders. Our approach aims to demystify inherent causal features on adversarial examples by leveraging a zero-sum optimization game between a casual feature estimator (i.e., hypothesis model) and worstcase counterfactuals (i.e., test function) disturbing to find causal features. Through extensive analyses, we demonstrate that the estimated causal features are highly related to the correct prediction for adversarial robustness, and the counterfactuals exhibit extreme features significantly deviating from the correct prediction. In addition, we present how to effectively inoculate CAusal FEatures (CAFE) into defense networks for improving adversarial robustness.

Towards Evaluating The Robustness Of Neural Networks

Abstract: Neural networks provide state-of-the-art results for most machine learning tasks. Unfortunately, neural networks are vulnerable to adversarial examples: given an input $x$ and any target classification $t$, it is possible to find a new input $x'$ that is similar to $x$ but classified as $t$. This makes it difficult to apply neural networks in security-critical areas. Defensive distillation is a recently proposed approach that can take an arbitrary neural network, and increase its robustness, reducing the success rate of current attacks' ability to find adversarial examples from $95\%$ to $0.5\%$. ::: In this paper, we demonstrate that defensive distillation does not significantly increase the robustness of neural networks by introducing three new attack algorithms that are successful on both distilled and undistilled neural networks with $100\%$ probability. Our attacks are tailored to three distance metrics used previously in the literature, and when compared to previous adversarial example generation algorithms, our attacks are often much more effective (and never worse). Furthermore, we propose using high-confidence adversarial examples in a simple transferability test we show can also be used to break defensive distillation. We hope our attacks will be used as a benchmark in future defense attempts to create neural networks that resist adversarial examples.


Finally, to address the problem that only image-level labels are provided in MIL, Xu et al. #REFR design an automatic instance-level label generation method.

Thus, they propose a MIL structure involving maximum likelihood estimation to predict multiple labels, i.e., bag-level labels and diagnostic scores; instance-level labels and informativeness, simultaneously.

Similarly, when studying the classification of the retinal nerve fiber layer (RNFL), Manivannan et al.

#OTHEREFR have observed that regions that contain the RNFL generally have strong intra-class variation, making them difficult to distinguish from other regions.

Therefore, they map the instances into a discriminative subspace to increase the discrepancy for disentangled instance feature learning. Jia et al.

#OTHEREFR incorporate the multi-scale image feature into the learning process to obtain more latent information on histopathology images.

Their work has led to an interesting MIL algorithm design direction and may shed light on how to improve the performance of local detection algorithms.

Other studies on problems such as phenotype categorization #OTHEREFR , #OTHEREFR , #OTHEREFR and multi-label classification #OTHEREFR have also made promising progress with the MIL algorithm.

Label-Efficient Deep Learning In Medical Image Analysis: Challenges And Future Directions

Abstract: Deep learning has seen rapid growth in recent years and achieved state-of-the-art performance in a wide range of applications. However, training models typically requires expensive and time-consuming collection of large quantities of labeled data. This is particularly true within the scope of medical imaging analysis (MIA), where data are limited and labels are expensive to be acquired. Thus, label-efficient deep learning methods are developed to make comprehensive use of the labeled data as well as the abundance of unlabeled and weak-labeled data. In this survey, we extensively investigated over 300 recent papers to provide a comprehensive overview of recent progress on label-efficient learning strategies in MIA. We first present the background of label-efficient learning and categorize the approaches into different schemes. Next, we examine the current state-of-the-art methods in detail through each scheme. Specifically, we provide an in-depth investigation, covering not only canonical semi-supervised, self-supervised, and multi-instance learning schemes, but also recently emerged active and annotation-efficient learning strategies. Moreover, as a comprehensive contribution to the field, this survey not only elucidates the commonalities and unique features of the surveyed methods but also presents a detailed analysis of the current challenges in the field and suggests potential avenues for future research.

Camel: A Weakly Supervised Learning Framework For Histopathology Image Segmentation

Abstract: Histopathology image analysis plays a critical role in cancer diagnosis and treatment. To automatically segment the cancerous regions, fully supervised segmentation algorithms require labor-intensive and time-consuming labeling at the pixel level. In this research, we propose CAMEL, a weakly supervised learning framework for histopathology image segmentation using only image-level labels. Using multiple instance learning (MIL)-based label enrichment, CAMEL splits the image into latticed instances and automatically generates instance-level labels. After label enrichment, the instance-level labels are further assigned to the corresponding pixels, producing the approximate pixel-level labels and making fully supervised training of segmentation models possible. CAMEL achieves comparable performance with the fully supervised approaches in both instance-level classification and pixel-level segmentation on CAMELYON16 and a colorectal adenoma dataset. Moreover, the generality of the automatic labeling methodology may benefit future weakly supervised learning studies for histopathology image analysis.


Semantic encoder: The semantic encoder is composed of L e Transformer #REFR encoder layers, which aims to further encode the semantic information of speech representations.

To learn the correct acoustic boundaries, we use the source or target text length J as the supervised signal.

There are two benefits of using CIF as a boundary detector.

For offline ST model, it can address the length gap between speech and text.

It can also dynamically detect the acoustic boundaries of streaming audio to perform read/write policies for streaming inference.

Translation decoder: The translation decoder is composed of L e Transformer decoder layers, which generates the translations in an autoregressive way. The translation loss is defined as:

L ST (x, y) = − J j=1 log p (y j | y <j , x) (6)

Adapting Offline Speech Translation Models For Streaming With Future-Aware Distillation And Inference

Abstract: A popular approach to streaming speech translation is to employ a single offline model with a wait-k policy to support different latency requirements, which is simpler than training multiple online models with different latency constraints. However, there is a mismatch problem in using a model trained with complete utterances for streaming inference with partial input. We demonstrate that speech representations extracted at the end of a streaming input are significantly different from those extracted from a complete utterance. To address this issue, we propose a new approach called Future-Aware Streaming Translation (FAST) that adapts an offline ST model for streaming input. FAST includes a Future-Aware Inference (FAI) strategy that incorporates future context through a trainable masked embedding, and a Future-Aware Distillation (FAD) framework that transfers future context from an approximation of full speech to streaming input. Our experiments on the MuST-C EnDe, EnEs, and EnFr benchmarks show that FAST achieves better trade-offs between translation quality and latency than strong baselines. Extensive analyses suggest that our methods effectively alleviate the aforementioned mismatch problem between offline training and online inference. * Equal contribution. † Corresponding author. (a) Training: Full speech encoding (b) Testing: Streaming encoding

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


To validate the effectiveness of our approach, we compared it with our baselines and PDDLStream #REFR .

sampling a random node and extending the tree toward it, to TAMP problems.

To address this issue, we propose a hierarchical strategy that first samples an abstract action sequence by employing Monte Carlo tree search (MCTS) in conjunction with a symbolic planner (which can be of any off-the-shelf planner), and subsequently samples the corresponding geometric state variables. Moreover, we introduce two methods to improve planning speed.

First, the reward of MCTS is generated by evaluating the result of TP (i.e. action sequence) using MP.

Second, when performing MP, a goal state candidate is generated in advance based on the given action sequence.

We found that prechecking the feasibility of the goal candidate can significantly improve the search speed, particularly in problems with tight goal constraints, such as the kitchen domain.

A Reachability Tree-Based Algorithm For Robot Task And Motion Planning

Abstract: This paper presents a novel algorithm for robot task and motion planning (TAMP) problems by utilizing a reachability tree. While tree-based algorithms are known for their speed and simplicity in motion planning (MP), they are not well-suited for TAMP problems that involve both abstracted and geometrical state variables. To address this challenge, we propose a hierarchical sampling strategy, which first generates an abstracted task plan using Monte Carlo tree search (MCTS) and then fills in the details with a geometrically feasible motion trajectory. Moreover, we show that the performance of the proposed method can be significantly enhanced by selecting an appropriate reward for MCTS and by using a pre-generated goal state that is guaranteed to be geometrically feasible. A comparative study using TAMP benchmark problems demonstrates the effectiveness of the proposed approach.

Pddlstream: Integrating Symbolic Planners And Blackbox Samplers Via Optimistic Adaptive Planning

Abstract: Many planning applications involve complex relationships defined on high-dimensional, continuous variables. For example, robotic manipulation requires planning with kinematic, collision, visibility, and motion constraints involving robot configurations, object poses, and robot trajectories. These constraints typically require specialized procedures to sample satisfying values. We extend PDDL to support a generic, declarative specification for these procedures that treats their implementation as black boxes. We provide domain-independent algorithms that reduce PDDLStream problems to a sequence of finite PDDL problems. We also introduce an algorithm that dynamically balances exploring new candidate plans and exploiting existing ones. This enables the algorithm to greedily search the space of parameter bindings to more quickly solve tightly-constrained problems as well as locally optimize to produce low-cost solutions. We evaluate our algorithms on three simulated robotic planning domains as well as several real-world robotic tasks.


We train the source model on two clean subsets trainclean-100 and train-clean-360 of LibriTTS dataset #REFR , a multi-speaker TTS corpus, totaling 1151 speakers and 245 hours of speech.

For the architecture behind the diffusion decoder, we implemented that of AdaSpeech #OTHEREFR , including the phoneme encoder, the acoustic condition modeling module, and the variance adapter.

Our implementation is based on the open-source software 1 2 of the two models.

We use the diffusion decoder with 12 convolution blocks, 128 residual channels and 3.5M parameters proposed in #OTHEREFR .

Other model configurations follow corresponding parts of AdaSpeech and PriorGrad unless otherwise stated.

The total number parameters to be finetuned for the diffusion decoder is 0.131M, compared to 1.184M for AdaSpeech. Data.

For evaluation, we select 11 speakers (7 females and 4 males) with different accents from VCTK #OTHEREFR following the practice in #OTHEREFR .

For each speaker, 10 utterances with the same transcripts across all speakers are randomly selected as the test set.

The preprocessing of speech and text data follows AdaSpeech except using a sampling rate of 22,050 Hz. Training, adaptation and inference.

Following AdaSpeech, the training process comprises two stages in which the numbers of steps are 200K and 100K, respectively.

The models are trained on one NVIDIA RTX 3090 GPU using a batch size of 50,000 speech frames.

An Investigation Into The Adaptability Of A Diffusion-Based Tts Model

Abstract: Given the recent success of diffusion in producing naturalsounding synthetic speech, we investigate how diffusion can be used in speaker adaptive TTS. Taking cues from more traditional adaptation approaches, we show that adaptation can be included in a diffusion pipeline using conditional layer normalization with a step embedding. However, we show experimentally that, whilst the approach has merit, such adaptation alone cannot approach the performance of Transformer-based techniques. In a second experiment, we show that diffusion can be optimally combined with Transformer, with the latter taking the bulk of the adaptation load and the former contributing to improved naturalness.

Libritts: A Corpus Derived From Librispeech For Text-To-Speech

Abstract: This paper introduces a new speech corpus called "LibriTTS" designed for text-to-speech use. It is derived from the original audio and text materials of the LibriSpeech corpus, which has been used for training and evaluating automatic speech recognition systems. The new corpus inherits desired properties of the LibriSpeech corpus while addressing a number of issues which make LibriSpeech less than ideal for text-to-speech work. The released corpus consists of 585 hours of speech data at 24kHz sampling rate from 2,456 speakers and the corresponding texts. Experimental results show that neural end-to-end TTS models trained from the LibriTTS corpus achieved above 4.0 in mean opinion scores in naturalness in five out of six evaluation speakers. The corpus is freely available for download from this http URL.


Given historical data, the APC model uses the log Mel spectrograms feature as input and predicts future surface properties. The model is a 3-layer GRU #REFR .

The input audio provides critical input information to our system.

Following #OTHEREFR , we use auto-regressive predictive coding (APC) #OTHEREFR to extract structural audio stream representations.

where h u ∈ R 512 represents the posterior probability of every layer in GRUs.

Our targeted audio stream representations are latent states in the final GRU unit.

During the training process, we add a linear layer that maps the output to predict the next log Mel spectrogram.

The representation of the final GRU unit h 3 is projected to a manifold as in #OTHEREFR to extract the audio stream feature h ∈ R 512 .

Style Transfer For 2D Talking Head Animation

Abstract: Audio-driven talking head animation is a challenging research topic with many real-world applications. Recent works have focused on creating photo-realistic 2D animation, while learning different talking or singing styles remains an open problem. In this paper, we present a new method to generate talking head animation with learnable style references. Given a set of style reference frames, our framework can reconstruct 2D talking head animation based on a single input image and an audio stream. Our method first produces facial landmarks motion from the audio stream and constructs the intermediate style patterns from the style reference images. We then feed both outputs into a style-aware image generator to generate the photo-realistic and fidelity 2D animation. In practice, our framework can extract the style information of a specific character and transfer it to any new static image for talking head animation. The intensive experimental results show that our method achieves better results than recent state-of-the-art approaches qualitatively and quantitatively. Our source code can be found at: https://github.com/aioz-ai/AudioDrivenStyleTransfer.

Learning Phrase Representations Using Rnn Encoder-Decoder For Statistical Machine Translation

Abstract: In this paper, we propose a novel neural network model called RNN Encoder‐ Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixedlength vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder‐Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases.


We assess navigation performance by measuring Success Rate (SR) and Success weighted by Path-Length (SPL) #REFR .

c) Evaluation: To assess the navigation performance, we rank the difficulty of each trajectory as proposed by #OTHEREFR .

Trajectories are categorized into easy (1.5 − 3m), medium (3 − 5m), and hard (5 − 10m) based on their geodesic distance to the goal.

We add an extra category labeled as "very hard" (>10m) to evaluate the agent's ability to plan for long-horizon goals.

Each difficulty level is assessed over 1,000 trajectories using a maximum episode length of 500.

To ensure a comprehensive assessment, we sample distinct starting positions and goals for each environment and difficulty, resulting in a total of 4,000 trajectories per environment.

A navigation trial is considered successful if the agent comes to a STOP within a maximum distance of 1m from the goal.

We also use a Soft Success Rate (SSR), where a trial is successful if the agent is less than 1m away from the goal at any point during navigation.

Lastly, we monitor the ratio of Collision-Free Trajectories (CFT) across all difficulties.

A trajectory is deemed collision-free if it does not collide during the experiment.

d) Results: We present quantitative navigation results in Table I .

One-4-All: Neural Potential Fields For Embodied Navigation

Abstract: A fundamental task in robotics is to navigate between two locations. In particular, real-world navigation can require long-horizon planning using high-dimensional RGB images, which poses a substantial challenge for end-to-end learning-based approaches. Current semi-parametric methods instead achieve long-horizon navigation by combining learned modules with a topological memory of the environment, often represented as a graph over previously collected images. However, using these graphs in practice typically involves tuning a number of pruning heuristics to avoid spurious edges, limit runtime memory usage and allow reasonably fast graph queries. In this work, we present One-4-All (O4A), a method leveraging self-supervised and manifold learning to obtain a graph-free, end-to-end navigation pipeline in which the goal is specified as an image. Navigation is achieved by greedily minimizing a potential function defined continuously over the O4A latent space. Our system is trained offline on non-expert exploration sequences of RGB data and controls, and does not require any depth or pose measurements. We show that O4A can reach long-range goals in 8 simulated Gibson indoor environments, and further demonstrate successful real-world navigation using a Jackal UGV platform. a * Equal contribution. Author ordering determined by competitive duckcalling, where the winner was selected by a blind jury on their ability to recreate various duck calls.

On Evaluation Of Embodied Navigation Agents

Abstract: Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence. The past two years have seen a surge of creative work on navigation. This creative output has produced a plethora of sometimes incompatible task definitions and evaluation protocols. To coordinate ongoing and future research in this area, we have convened a working group to study empirical methodology in navigation research. The present document summarizes the consensus recommendations of this working group. We discuss different problem statements and the role of generalization, present evaluation measures, and provide standard scenarios that can be used for benchmarking.


FlowNet #REFR introduced a learning based two-frame optical flow estimation method using CNNs wherein a single flow field is estimated from a pair of image frames.

Horn and Schunck #OTHEREFR was a pioneering algorithm that introduced a variational energy minimization approach to estimate optical flow using a brightness consistency assumption (pixel intensity during pixel coordinate transformation assumed to be constant for a smaller temporal change) and a spatial smoothness constraint for the optical flow field as part of the objective function.

Other classical multi-frame methods use a bank of hand-crafted motion filters tuned to capture moving patterns and texture characteristics from each image sequence to estimate the direction and magnitude of optical flow fields #OTHEREFR .

With the recent advancement of deep learning and availability of large datasets with known ground truth, optical flow estimation is reformulated as an end-to-end learning problem without requiring assumptions regarding the characteristics of images and of motion patterns.

Motivated by Heeger's #OTHEREFR approach to use hand crafted spatiotemporal Gabor filters, Teney et al.

#OTHEREFR developed a learning-based multi-frame optical flow estimation model based on learnable 3D convolutional neural networks (CNN) and signal processing concepts.

Several other learning based two-frame methods improved upon the FlowNet model, namely the coarse-to-fine pyramid network #OTHEREFR , multiple intermediate flow estimates and warping based brightness error computation #OTHEREFR , receptive field guided motion feature extraction networks #OTHEREFR and the 4D all-pairs correlation volume with gated recurrent network #OTHEREFR .

In dynamic scenes, limited temporal information available from only two frames causes the two-frame methods to have poor generalizations in and near occluded regions as well as in out-of-boundary regions.

This is likely because in these regions the flow dynamics are often complex and cannot be fully captured using only two frames.

In regions with disappearing scene elements, learning based two-frame methods do not have access to context Figure 1 : Examples of optical flow estimates (OFEs) using our methods compared with recent state-of-the-art two-frame and multi-frame methods.

The regions bounded by red boxes in the input frames represent the regions where our method significantly outperformed the other methods.

Sstm: Spatiotemporal Recurrent Transformers For Multi-Frame Optical Flow Estimation

Abstract: Inaccurate optical flow estimates in and near occluded regions, and out-ofboundary regions are two of the current significant limitations of optical flow estimation algorithms. Recent state-of-the-art optical flow estimation algorithms are two-frame based methods where optical flow is estimated sequentially for each consecutive image pair in a sequence. While this approach gives good flow estimates, it fails to generalize optical flows in occluded regions mainly due to limited local evidence regarding moving elements in a scene. In this work, we propose a learning-based multi-frame optical flow estimation method that estimates two or more consecutive optical flows in parallel from multi-frame image sequences. Our underlying hypothesis is that by understanding temporal scene dynamics from longer sequences with more than two frames, we can characterize pixel-wise dependencies in a larger spatiotemporal domain, generalize complex motion patterns and thereby improve the accuracy of optical flow estimates in occluded regions. We present learning-based spatiotemporal recurrent transformers for multi-frame based optical flow estimation (SSTMs). Our method utilizes 3D Convolutional Gated Recurrent Units (3D-ConvGRUs) and spatiotemporal transformers to learn recurrent space-time motion dynamics and global dependencies in the scene and provide a generalized optical flow estimation. When compared with recent state-of-the-art two-frame and multi-frame methods on real world and synthetic datasets, performance of the SSTMs were significantly higher in occluded and out-of-boundary regions. Among all published state-of-the-art multi-frame methods, SSTM achieved state-of the-art results on the Sintel Final and KITTI2015 benchmark datasets.

Flownet: Learning Optical Flow With Convolutional Networks

Abstract: Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical flow estimation has not been among the tasks CNNs succeeded at. In this paper we construct CNNs which are capable of solving the optical flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations. Since existing ground truth data sets are not sufficiently large to train a CNN, we generate a large synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps.


Existing methods build the multimodal alignment by designing constraints across different modalities (called crossmodal constraints #REFR ).

In BMQA, multimodal quality alignment refers to finding the corresponding quality representation relationship between two modalities.

The image and QSD are obtained for the same scene, and the quality description can be highly consistent #OTHEREFR .

For example, the QSD keywords can directly indicate the image regions that the visual attention focuses on #OTHEREFR , thus improving the learning performance of an image feature encoder.

In the construction of training objective, it is necessary to define a special metric to measure the difference between two modalities.

If two modalities come from different perspectives of a single sensor, the metric is defined as an absolute value error, such as mean absolute error (MAE) and mean square error (MSE).

If two modalities come from different sensors, the metric is defined as a relative value error, such as cosine similarity #OTHEREFR .

The image and QSD are heterogeneous modalities.

Therefore, we adopt the cosine similarity to measure the relative difference, and design an attentive pooling for multimodal quality alignment.

Blind Multimodal Quality Assessment: A Brief Survey And A Case Study Of Low-Light Images

Abstract: Blind image quality assessment (BIQA) aims at automatically and accurately forecasting objective scores for visual signals, which has been widely used to monitor product and service quality in low-light applications, covering smartphone photography, video surveillance, autonomous driving, etc. Recent developments in this field are dominated by unimodal solutions inconsistent with human subjective rating patterns, where human visual perception is simultaneously reflected by multiple sensory information (e.g., sight and hearing). In this article, we present a unique blind multimodal quality assessment (BMQA) of low-light images from subjective evaluation to objective score. To investigate the multimodal mechanism, we first establish a multimodal low-light image quality (MLIQ) database with authentic low-light distortions, containing image and audio modality pairs. Further, we specially design the key modules of BMQA, considering multimodal quality representation, latent feature alignment and fusion, and hybrid self-supervised and supervised learning. Extensive experiments show that our BMQA yields state-of-the-art accuracy on the proposed MLIQ benchmark database. In particular, we also build an independent single-image modality Dark-4K database, which is used to verify its applicability and generalization performance in mainstream unimodal applications. Qualitative and quantitative results on Dark-4K show that BMQA achieves superior performance to existing BIQA approaches as long as a pre-trained quality semantic description model is provided. The proposed framework and two databases as well as the collected BIQA methods and evaluation metrics are made publicly available.

Multimodal Machine Learning: A Survey And Taxonomy

Abstract: Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.


Despite the merits, FL in IoT systems could also meet with plenty of challenges due to potentially heterogeneous device capabilities and data distributions, which are also known as system heterogeneity and non-IID data #REFR .

As a result, training deep learning models centrally by gathering data from user devices might be prohibited and impractical.

To alleviate the above issues, it is attractive to adopt federated learning (FL) #OTHEREFR - #OTHEREFR to collectively train models, which can take full advantages of the local data and edge computational resources in the IoT systems.

FL is a distributed computing paradigm where multiple edge devices, i.e., clients, collaboratively train a global model without disclosing local sensitive data.

Conventional FL systems usually consist of a central server and a number of clients, where each client updates the model based on local data and the server is responsible for synchronizing model parameters.

The primary goal of FL is to efficiently train a global model with the highest possible accuracy #OTHEREFR , #OTHEREFR .

The non-IID data in FL means the underlying distributions of local data are not identical across clients #OTHEREFR .

System heterogeneity could be caused by different resource constraints, including storage, computing capability, energy and network bandwidth #OTHEREFR .

In synchronous FL, aggregation is conducted when all the selected clients fully complete their local training and return their model updates.

However, the clients could have significantly different performances in real-world scenarios, which implies that some faster clients have to wait for the slower ones, i.e., stragglers, resulting in unnecessary time-consuming.

Moreover, some clients might be unresponsive due to inactive devices and/or communication failures and thus induce unavailability of some local updates, i.e., dropout.

Stabilizing And Improving Federated Learning With Non-Iid Data And Client Dropout

Abstract: The label distribution skew induced data heterogeniety has been shown to be a significant obstacle that limits the model performance in federated learning, which is particularly developed for collaborative model training over decentralized data sources while preserving user privacy. This challenge could be more serious when the participating clients are in unstable circumstances and dropout frequently. Previous work and our empirical observations demonstrate that the classifier head for classification task is more sensitive to label skew and the unstable performance of FedAvg mainly lies in the imbalanced training samples across different classes. The biased classifier head will also impact the learning of feature representations. Therefore, maintaining a balanced classifier head is of significant importance for building a better global model. To this end, we propose a simple yet effective framework by introducing a prior-calibrated softmax function for computing the cross-entropy loss and a prototype-based feature augmentation scheme to re-balance the local training, which are lightweight for edge devices and can facilitate the global model aggregation. The improved model performance over existing baselines in the presence of non-IID data and client dropout is demonstrated by conducting extensive experiments on benchmark classification tasks.

Advances And Open Problems In Federated Learning

Abstract: Federated learning (FL) is a machine learning setting where many clients (e.g. mobile devices or whole organizations) collaboratively train a model under the orchestration of a central server (e.g. service provider), while keeping the training data decentralized. FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches. Motivated by the explosive growth in FL research, this paper discusses recent advances and presents an extensive collection of open problems and challenges.


Using the AFGs and their corresponding labels (benign or vulnerable) as the training dataset, one can train a GNN model for vulnerability detection, e.g., Devign #REFR .

Example with code vulnerability detection.

Figure 1(a) shows an example source code with a "double free" vulnerability, which happens when the second free (line 12) is called after the first free (line 9).

Vulnerability detection methods firstly convert the source code to an attributed graph.

For example, we construct the attributed graph from the source code as shown in Figure 1 (b) by building the Attributed control and data Flow Graph (AFG) and encoding the syntax attributes for each node.

The node denotes the statement, the edge denotes control or data flow between two statements, and the attributes include syntax features, such as which keywords are used in a statement.

For the AFG generated from the example source code in Figure 1 , nodes 9, 12 and the keyword free should be identified in the final explanation results.

Figure 1 (c) presents the output from two recent representative works and ILLUMINATI.

GNNExplainer estimates the edge importance from the AFG by learning the soft continuous edge masks.

In this example, GNNExplainer identifies #OTHEREFR and #OTHEREFR as important and considers this subgraph as the explanation result.

This is not accurate because node 12 is missed due to none of its edges is considered important.

Illuminati: Towards Explaining Graph Neural Networks For Cybersecurity Analysis

Abstract: Graph neural networks (GNNs) have been utilized to create multi-layer graph models for a number of cybersecurity applications from fraud detection to software vulnerability analysis. Unfortunately, like traditional neural networks, GNNs also suffer from a lack of transparency, that is, it is challenging to interpret the model predictions. Prior works focused on specific factor explanations for a GNN model. In this work, we have designed and implemented ILLUMINATI, a comprehensive and accurate explanation framework for cybersecurity applications using GNN models. Given a graph and a pre-trained GNN model, ILLUMINATI is able to identify the important nodes, edges, and attributes that are contributing to the prediction while requiring no prior knowledge of GNN models. We evaluate ILLUMINATI in two cybersecurity applications, i.e., code vulnerability detection and smart contract vulnerability detection. The experiments show that ILLUMINATI achieves more accurate explanation results than state-of-theart methods, specifically, 87.6% of subgraphs identified by ILLUMINATI are able to retain their original prediction, an improvement of 10.3% over others at 77.3%. Furthermore, the explanation of ILLUMINATI can be easily understood by the domain experts, suggesting the significant usefulness for the development of cybersecurity applications.

Devign: Effective Vulnerability Identification By Learning Comprehensive Program Semantics Via Graph Neural Networks

Abstract: Vulnerability identification is crucial to protect the software systems from attacks for cyber security. It is especially important to localize the vulnerable functions among the source code to facilitate the fix. However, it is a challenging and tedious process, and also requires specialized security expertise. Inspired by the work on manually-defined patterns of vulnerabilities from various code representation graphs and the recent advance on graph neural networks, we propose Devign, a general graph neural network based model for graph-level classification through learning on a rich set of code semantic representations. It includes a novel Conv module to efficiently extract useful features in the learned rich node representations for graph-level classification. The model is trained over manually labeled datasets built on 4 diversified large-scale open-source C projects that incorporate high complexity and variety of real source code instead of synthesis code used in previous works. The results of the extensive evaluation on the datasets demonstrate that Devign outperforms the state of the arts significantly with an average of 10.51% higher accuracy and 8.68\% F1 score, increases averagely 4.66% accuracy and 6.37% F1 by the Conv module.


When retrained for enough epochs (e.g., 600 or 800 epochs), retraining from scratch is able to produce comparable performance, which is consistent with #REFR .

Comparison between rewinding and fine-tuning shows that their performance is almost the same after retraining 200 epochs.

However, in a low-epoch retraining regime, fine-tuning is more efficient than rewinding for filter pruning.

We also conduct experiments that retrain pruned subnetworks from scratch, where the parameters of a subnetwork are re-initialized after pruning. The original learning rate schedule is adopted during retraining.

As shown in Figure 1 (right, green line), retraining from scratch requires more epochs to reduce accuracy drop compared to fine-tuning and rewinding.

3 Moreover, our experimental results suggest that it is unfair to compare retraining techniques under different epochs, since all of them benefit from more epochs.

In this subsection, we empirically study existing retraining techniques, including finetuning, rewinding and retraining from scratch.

Our experimental results show that these retraining techniques can achieve similar performance if we retrain pruned subnetworks with enough epochs.

However, fine-tuning for a few epochs is a more efficient choice among them as our focus is not on recovering the original accuracy as much as possible in the following. Finally, we introduce a standard setting for further exploration.

Network Pruning Spaces

Abstract: Network pruning techniques, including weight pruning and filter pruning, reveal that most state-of-the-art neural networks can be accelerated without a significant performance drop. This work focuses on filter pruning which enables accelerated inference with any off-the-shelf deep learning library and hardware. We propose the concept of network pruning spaces that parametrize populations of subnetwork architectures. Based on this concept, we explore the structure aspect of subnetworks that result in minimal loss of accuracy in different pruning regimes and arrive at a series of observations by comparing subnetwork distributions. We conjecture through empirical studies that there exists an optimal FLOPs-to-parameterbucket ratio related to the design of original network in a pruning regime. Statistically, the structure of a winning subnetwork guarantees an approximately optimal ratio in this regime. Upon our conjectures, we further refine the initial pruning space to reduce the cost of searching a good subnetwork architecture. Our experimental results on ImageNet show that the subnetwork we found is superior to those from the state-of-the-art pruning methods under comparable FLOPs.

Rethinking The Value Of Network Pruning

Abstract: Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization.


We use the BASIL news bias prediction dataset which contains sentence-level annotations for two types of bias #REFR : lexical and informational bias.

In this section, we analyze the structural properties of biased news articles using the main sentences identified by the model.

To do so, we collect the predicted main sentences and then assess whether they capture elements of the formalized discourse structures commonly used in journalism.

By validating these from the viewpoints of both summarization and structure, we ensure the reliability of using multiple attention heads as a form of explanation mechanism

Lexical bias refers to the bias mainly caused by the word choice of the journalist, such as using polarized words (e.g.

Donald Trump is investing more in the conspiracy theories about President Obama's birth certificate as he explores his own bid for the presidency.) And informational bias refers to the biased elaboration of certain events or facts, which includes using selective quotations to strengthen their viewpoint. (e.g. The Arizona group said the call from Mr.

Trump on Wednesday came unexpectedly, and the group had spent much of the day Thursday scurrying to make travel arrangements to New York.)

Disentangling Structure And Style: Political Bias Detection In News By Inducing Document Hierarchy

Abstract: We address an important gap in detection of political bias in news articles. Previous works that perform supervised document classification can be biased towards the writing style of each news outlet, leading to overfitting and limited generalizability. Our approach overcomes this limitation by considering both the sentence-level semantics and the documentlevel rhetorical structure, resulting in a more robust and style-agnostic approach to detecting political bias in news articles. We introduce a novel multi-head hierarchical attention model that effectively encodes the structure of long documents through a diverse ensemble of attention heads. While journalism follows a formalized rhetorical structure, the writing style may vary by news outlet. We demonstrate that our method overcomes this domain dependency and outperforms previous approaches for robustness and accuracy. Further analysis demonstrates the ability of our model to capture the discourse structures commonly used in the journalism domain.

In Plain Sight: Media Bias Through The Lens Of Factual Reporting

Abstract: The increasing prevalence of political bias in news media calls for greater public awareness of it, as well as robust methods for its detection. While prior work in NLP has primarily focused on the lexical bias captured by linguistic attributes such as word choice and syntax, other types of bias stem from the actual content selected for inclusion in the text. In this work, we investigate the effects of informational bias: factual content that can nevertheless be deployed to sway reader opinion. We first produce a new dataset, BASIL, of 300 news articles annotated with 1,727 bias spans and find evidence that informational bias appears in news articles more frequently than lexical bias. We further study our annotations to observe how informational bias surfaces in news articles by different media outlets. Lastly, a baseline model for informational bias prediction is presented by fine-tuning BERT on our labeled data, indicating the challenges of the task and future directions.


For Car, we evaluate models on NuScenes #REFR , a driving dataset with 3D detection and tracking annotations.

For the additional experiment on shape reconstruction, we use Chamfer distance and F1 score, following #OTHEREFR .

We evaluate all methods on three categories of objects: Chair #OTHEREFR , Table #OTHEREFR , and Car #OTHEREFR .

For all categories, we first pre-train each method on ShapeNet #OTHEREFR , a dataset of synthetic objects, to obtain a category-level prior.

For Chair and Table, we evaluate models on ScanNet #OTHEREFR , a dataset of real-world indoor scene scans.

For more details on these datasets, refer to section C of the supplementary material.

For partial-view novel view synthesis, we compare FINV against Instant-NGP #OTHEREFR , pixelNeRF #OTHEREFR , IBRNet #OTHEREFR , IBRNet fine-tuned during test time, Au-toRF #OTHEREFR , and EG3D with pivotal tuning inversion (EG3D+PTI) #OTHEREFR .

We use open-source implementations of each method when available, or private implementations shared by the authors.

Note that AutoRF is originally proposed for reconstruction from one single view, but it supports multiple input views as well, which we find to improve reconstruction quality.

Partial-View Object View Synthesis Via Filtering Inversion

Abstract: We propose Filtering Inversion (FINV), a learning framework and optimization process that predicts a renderable 3D object representation from one or few partial views. FINV addresses the challenge of synthesizing novel views of objects from partial observations, spanning cases where the object is not entirely in view, is partially occluded, or is only observed from similar views. To achieve this, FINV learns shape priors by training a 3D generative model. At inference, given one or more views of a novel real-world object, FINV first finds a set of latent codes for the object by inverting the generative model from multiple initial seeds. Maintaining the set of latent codes, FINV filters and resamples them after receiving each new observation, akin to particle filtering. The generator is then finetuned for each latent code on the available views in order to adapt to novel objects. We show that FINV successfully synthesizes novel views of real-world objects (e.g., chairs, tables, and cars), even if the generative prior is trained only on synthetic objects. The ability to address the sim-to-real problem allows FINV to be used for object categories without real-world datasets. FINV achieves state-of-the-art performance on multiple real-world datasets, recovers object shape and texture from partial and sparse views, is robust to occlusion, and is able to incrementally improves its representation with more observations. * This work was partly done during an internship at Nvidia.

Nuscenes: A Multimodal Dataset For Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image-based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first published dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online at this http URL.


Cauchy loss #REFR , however, performed the best as compared to the other loss functions, as evident from Table 7 .

We now present the robustness of TPMCF in the presence of outliers.

Table 7 presents the performance of TPMCF with four different loss functions for various values of the outlier ratio (λ) on RT-10 dataset.

It may be noted that meansquared-error (MSE) #OTHEREFR is outlier sensitive, and hence, the performance of TPMCF with MSE as the loss function was the worst.

Mean-absolute-error (MAE) #OTHEREFR , Huber loss #OTHEREFR , and Cauchy loss are comparatively outlier resilient.

Table 8 presents the impact of the outliers on the prediction accuracy of TPMCF for all four datasets.

As observed from Table 8 , the outliers have a severe impact on the performance of TPMCF.

In Figure 6 , we report the performance Figure 6 , when we removed the first 2% outliers (λ = 0.02), we achieved maximum performance gain (PG) that is defined as follows:

where m 1 and m 2 represent the MAE of TPMCF after removing outliers with the ratio λ 1 and λ 2 , respectively (λ 1 < λ 2 ).

Tpmcf: Temporal Qos Prediction Using Multi-Source Collaborative Features

Abstract: In recent times, with the proliferation of online activities and the rapid deployment of web service APIs, personalized service recommendations have played a paramount role in the growth of the e-commerce industry. The performance of web services is one of the standard measures for service recommendation. Quality-of-Service (QoS) parameters determining the service performance fluctuate over time for a user. Thus, for a given user, the service QoS prediction over time plays an essential part in identifying a suitable service to invoke among a pool of services having the same functionality. The contemporary temporal QoS prediction methods hardly achieved the desired accuracy due to various limitations, such as data sparsity, the presence of outliers, and the inability to capture higher-order temporal relationships among user-service interactions. Even though some recent recurrent neural network-based sequential architectures can model temporal relationships among QoS data, the prediction accuracy degrades due to the absence of other features (e.g., collaborative self features, collaborative spatial features of users/services) to comprehend the relationship among the user-service interactions. In addition to the lack of effective representation of implicit features, having the same attention across all the features in every timestep may impede improving the prediction accuracy. This paper addresses the above challenges and proposes a scalable strategy for Temporal QoS Prediction using Multi-source Collaborative Features (namely, TPMCF) enabling faster responsiveness while attaining high prediction accuracy. Our work combines the collaborative features of the user/service by exploiting the user-service relationship with the spatio-temporal auto-extracted features by employing graph convolution and a variant of transformer encoder with multi-head self-attention. While the graph convolutional is responsible for automatic feature extraction exploiting the spatial information, the transformer encoder is accountable for capturing the temporal dependency among QoS data for prediction. We validated our proposed method on WS-DREAM-2 benchmark datasets. Extensive experiments showed that TPMCF outperformed the major state-of-the-art approaches in terms of prediction accuracy while ensuring high scalability and reasonably faster responsiveness.

Robust Subspace Clustering By Cauchy Loss Function

Abstract: Subspace clustering is a problem of exploring the low-dimensional subspaces of high-dimensional data. State-of-the-art approaches are designed by following the model of spectral clustering-based method. These methods pay much attention to learn the representation matrix to construct a suitable similarity matrix and overlook the influence of the noise term on subspace clustering. However, the real data are always contaminated by the noise and the noise usually has a complicated statistical distribution. To alleviate this problem, in this paper, we propose a subspace clustering method based on Cauchy loss function (CLF). Particularly, it uses CLF to penalize the noise term for suppressing the large noise mixed in the real data. This is due to that the CLF’s influence function has an upper bound that can alleviate the influence of a single sample, especially the sample with a large noise, on estimating the residuals. Furthermore, we theoretically prove the grouping effect of our proposed method, which means that highly correlated data can be grouped together. Finally, experimental results on five real data sets reveal that our proposed method outperforms several representative clustering methods.


For instance, compared with FitNets #REFR which directly mimics the teacher features, our DSKD can bring more than 2 mIoU, showing the effectiveness of the proposed relationbased distillation algorithm.

Methods mIoU SCPNet w/o DSKD 34.4 + KD #OTHEREFR 33.8 + FitNets #OTHEREFR 33.8 + PKT #OTHEREFR 34.6 + PVKD #OTHEREFR 36.2 + NST #OTHEREFR 36.3 + DSKD 37.2 two large-scale benchmarks strongly demonstrate the superiority of our SCPNet .

Besides, we use the trained weight of the segmentation sub-network as initialization to train Cylinder3D on the Se-manticKITTI semantic segmentation task.

From Table 3 , Cylinder3D initialized from trained weight of the completion task outperforms the original Cylinder3D model by 2.6 mIoU, and achieves impressive segmentation performance among various competitive LiDAR segmentation models such as 2DPASS #OTHEREFR , PVKD #OTHEREFR and RPVNet #OTHEREFR .

The encouraging results show that knowledge learned in the completion task is also beneficial to the segmentation task. Comparison with baseline KD algorithms.

From Table 4 , it is evident that the proposed DSKD method can bring more gains than conventional knowledge distillation algorithms.

The vanilla KD objective and FitNets hamper the performance of the base model, indicating that directly mimicking the logits or features can not boost the completion performance. Qualitative results.

We also provide visual comparison between JS3C-Net #OTHEREFR , SCPNet (single-frame) and SCP-Net (multi-frame). As can be seen from Fig.

6 , our SCP-Net (single-frame) make more accurate completion predictions than JS3C-Net on road and vegetation.

On long, thin objects such as poles, our single-frame model also yields high-quality completion results compared with JS3C-Net.

The predictions of our single-frame model also resemble those of the multi-frame network, demonstrating the efficacy of the proposed DSKD algorithm.

Scpnet: Semantic Scene Completion On Point Cloud

Abstract: Training deep models for semantic scene completion (SSC) is challenging due to the sparse and incomplete input, a large quantity of objects of diverse scales as well as the inherent label noise for moving objects. To address the above-mentioned problems, we propose the following three solutions: 1) Redesigning the completion sub-network. We design a novel completion sub-network, which consists of several Multi-Path Blocks (MPBs) to aggregate multi-scale features and is free from the lossy downsampling operations. 2) Distilling rich knowledge from the multi-frame model. We design a novel knowledge distillation objective, dubbed Dense-to-Sparse Knowledge Distillation (DSKD). It transfers the dense, relation-based semantic knowledge from the multi-frame teacher to the single-frame student, significantly improving the representation learning of the single-frame model. 3) Completion label rectification. We propose a simple yet effective label rectification strategy, which uses off-the-shelf panoptic segmentation labels to remove the traces of dynamic objects in completion labels, greatly improving the performance of deep models especially for those moving objects. Extensive experiments are conducted in two public SSC benchmarks, i.e., Se-manticKITTI and SemanticPOSS. Our SCPNet ranks 1st on SemanticKITTI semantic scene completion challenge and surpasses the competitive S3CNet [3] by 7.2 mIoU. SCP-Net also outperforms previous completion algorithms on the SemanticPOSS dataset. Besides, our method also achieves competitive results on SemanticKITTI semantic segmentation tasks, showing that knowledge learned in the scene completion is beneficial to the segmentation task.

Fitnets: Hints For Thin Deep Nets

Abstract: While depth tends to improve network performances, it also makes gradient-based training more difficult since deeper networks tend to be more non-linear. The recently proposed knowledge distillation approach is aimed at obtaining small and fast-to-execute models, and it has shown that a student network could imitate the soft output of a larger teacher network or ensemble of networks. In this paper, we extend this idea to allow the training of a student that is deeper and thinner than the teacher, using not only the outputs but also the intermediate representations learned by the teacher as hints to improve the training process and final performance of the student. Because the student intermediate hidden layer will generally be smaller than the teacher's intermediate hidden layer, additional parameters are introduced to map the student hidden layer to the prediction of the teacher hidden layer. This allows one to train deeper students that can generalize better or run faster, a trade-off that is controlled by the chosen student capacity. For example, on CIFAR-10, a deep student network with almost 10.4 times less parameters outperforms a larger, state-of-the-art teacher network.


Panoptic FPN #REFR also uses a semantic segmentation branch and a Mask R-CNN instance segmentation branch, but it uses a shared Feature Pyramid Network network as the backbone.

This segmentation task can generate a scene representation that contains the segmentation mask of "stuff" (non-instance such as sky and road) and "thing" (such as cars and humans which we need to identify different instances), therefore, unifying semantic segmentation and instance segmentation.

Panoptic Segmentation also separates objects' instances with different instance ID #OTHEREFR .

UPSNet #OTHEREFR proposes a unified panoptic segmentation network which combines a semantic segmentation head with a Mask R-CNN #OTHEREFR style instance segmentation head and introduces a parameter-free panoptic head to predict the panoptic label.

FPSNet #OTHEREFR solves the same problem as a dense pixel-wise classification problem, which predicts a class label or an instance ID for each pixel, which runs faster than the above methods.

Panoptic-DeepLab #OTHEREFR uses a class-agnostic instance segmentation branch that includes a simple instance center regression, which improves the accuracy of panoptic segmentation with a simple and fast design.

Our work is inspired by the design choice of Panoptic-DeepLab.

However, rather than separating the prediction of the instance edges and semantic edges, we obtain the instance edges by combining the prediction of semantic edges, the center prediction of instances, and center regression of the instances.

Finally, we fuse the semantic edges with the instance edges to produce the panoptic edges.

Penet: A Joint Panoptic Edge Detection Network

Abstract: In recent years, compact and efficient scene understanding representations have gained popularity in increasing situational awareness and autonomy of robotic systems. In this work, we illustrate the concept of a panoptic edge segmentation and propose PENet, a novel detection network called that combines semantic edge detection and instancelevel perception into a compact panoptic edge representation. This is obtained through a joint network by multi-task learning that concurrently predicts semantic edges, instance centers and offset flow map without bounding box predictions exploiting the cross-task correlations among the tasks. The proposed approach allows extending semantic edge detection to panoptic edge detection which encapsulates both category-aware and instanceaware segmentation. We validate the proposed panoptic edge segmentation method and demonstrate its effectiveness on the real-world Cityscapes dataset.

Panoptic Feature Pyramid Networks

Abstract: The recently introduced panoptic segmentation task has renewed our community's interest in unifying the tasks of instance segmentation (for thing classes) and semantic segmentation (for stuff classes). However, current state-of-the-art methods for this joint task use separate and dissimilar networks for instance and semantic segmentation, without performing any shared computation. In this work, we aim to unify these methods at the architectural level, designing a single network for both tasks. Our approach is to endow Mask R-CNN, a popular instance segmentation method, with a semantic segmentation branch using a shared Feature Pyramid Network (FPN) backbone. Surprisingly, this simple baseline not only remains effective for instance segmentation, but also yields a lightweight, top-performing method for semantic segmentation. In this work, we perform a detailed study of this minimally extended version of Mask R-CNN with FPN, which we refer to as Panoptic FPN, and show it is a robust and accurate baseline for both tasks. Given its effectiveness and conceptual simplicity, we hope our method can serve as a strong baseline and aid future research in panoptic segmentation.


Recently, deep learning techniques, particularly large deep learning models, have shown excellent reasoning capabilities #REFR .

For instance, a set query can produce answers that include images, audio, and natural language (and their combination) seamlessly.

Designing a Multimodal Neural Database presents several substantial challenges.

First, it is crucial that the system is able to reason on the modalities given in input.

For instance, if I were to look for images of cats and dogs fighting, I need to recognize both the presence of these animals and that the interactions between the two is indeed that of fighting (a poster of Mike Tyson boxing in the background is not sufficient).

Similarly, if the query mentions someone whispering or yelling, the system must understand such subtleties in an audio frame.

The tasks of Visual Question Answering and multi-hop question answering have reached near human results #OTHEREFR for natural language processing, with promising candidates in the multimodal setting as well.

However, these models are usually extremely large, with billions of parameters, leading to the next challenge, namely scale.

Given a large collection of documents, it is infeasible to run such models on every query-document pair, or even on every document for that matter.

Open domain question answering systems (ODQA), developed for answering queries from natural language text, provide a methodology for scaling to larger document collections.

ODQA answers a query by first retrieving relevant documents from the document collection and feeding them as context to a transformer along with the query.

Multimodal Neural Databases

Abstract: The rise in loosely-structured data available through text, images, and other modalities has called for new ways of querying them. Multimedia Information Retrieval has filled this gap and has witnessed exciting progress in recent years. Tasks such as search and retrieval of extensive multimedia archives have undergone massive performance improvements, driven to a large extent by recent developments in multimodal deep learning. However, methods in this field remain limited in the kinds of queries they support and, in particular, their inability to answer database-like queries. For this reason, inspired by recent work on neural databases, we propose a new framework, which we name Multimodal Neural Databases (MM-NDBs). MMNDBs can answer complex database-like queries that involve reasoning over different input modalities, such as text and images, at scale. In this paper, we present the first architecture able to fulfill this set of requirements and test it with several baselines, showing the limitations of currently available models. The results show the potential of these new techniques to process unstructured data coming from different modalities, paving the way for future research in the area. Code to replicate the experiments will be released at https://github.com/GiovanniTRA/MultimodalNeuralDatabases CCS CONCEPTS • Information systems → Information retrieval; Multimedia and multimodal retrieval.

Making The V In Vqa Matter: Elevating The Role Of Image Understanding In Visual Question Answering

Abstract: Problems at the intersection of vision and language are of significant importance both as challenging research questions and for the rich set of applications they enable. However, inherent structure in our world and bias in our language tend to be a simpler signal for learning than visual modalities, resulting in models that ignore visual information, leading to an inflated sense of their capability. ::: We propose to counter these language priors for the task of Visual Question Answering (VQA) and make vision (the V in VQA) matter! Specifically, we balance the popular VQA dataset by collecting complementary images such that every question in our balanced dataset is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question. Our dataset is by construction more balanced than the original VQA dataset and has approximately twice the number of image-question pairs. Our complete balanced dataset is publicly available at this http URL as part of the 2nd iteration of the Visual Question Answering Dataset and Challenge (VQA v2.0). ::: We further benchmark a number of state-of-art VQA models on our balanced dataset. All models perform significantly worse on our balanced dataset, suggesting that these models have indeed learned to exploit language priors. This finding provides the first concrete empirical evidence for what seems to be a qualitative sense among practitioners. ::: Finally, our data collection protocol for identifying complementary images enables us to develop a novel interpretable model, which in addition to providing an answer to the given (image, question) pair, also provides a counter-example based explanation. Specifically, it identifies an image that is similar to the original image, but it believes has a different answer to the same question. This can help in building trust for machines among their users.

deep learning techniques
excellent reasoning capabilities


To enhance the robustness of the cross-domain model, both UMT and TDD utilize the teacher-student learning scheme, in which UMT adopts Mean Teacher #REFR and TDD adopts Dual-branch detection network.

Self-training is further regularized by distribution alignment for improvement robustness.

consistency between image-level and instance-level predictions to re-weight the instance-level alignment. ii) Training from noisy labels, a.k.a. self-training, e.g., NL #OTHEREFR and WST-BSR #OTHEREFR . iii) Based on sample generation strategy.

In this line of works, DD-MRL #OTHEREFR leverages an imageto-image translation via GAN to generate various distinctive shifted domains from the source domain.

AFAN #OTHEREFR obtains the intermediate domain (fusing the source and target domains) by interpolation.

UMT #OTHEREFR and TDD #OTHEREFR utilize both the source-like and target-like images to perform the cross-domain distillation.

In this work, a momentum-updated Faster R-CNN is performed for more stability in test-time adaptation.

Although the excellent performance is reached, all UDAOD methods require access to the source domain data during the adaptation process.

When the source data is not accessible due to privacy issues or storage overhead, more challenging settings are emerged with source-free domain adaptation #OTHEREFR and test-time adaptation #OTHEREFR .

Stfar: Improving Object Detection Robustness At Test-Time By Self-Training With Feature Alignment Regularization

Abstract: Domain adaptation helps generalizing object detection models to target domain data with distribution shift. It is often achieved by adapting with access to the whole target domain data. In a more realistic scenario, target distribution is often unpredictable until inference stage. This motivates us to explore adapting an object detection model at test-time, a.k.a. test-time adaptation (TTA). In this work, we approach test-time adaptive object detection (TTAOD) from two perspective. First, we adopt a self-training paradigm to generate pseudo labeled objects with an exponential moving average model. The pseudo labels are further used to supervise adapting source domain model. As self-training is prone to incorrect pseudo labels, we further incorporate aligning feature distributions at two output levels as regularizations to self-training. To validate the performance on TTAOD, we create benchmarks based on three standard object detection datasets and adapt generic TTA methods to object detection task. Extensive evaluations suggest our proposed method sets the state-of-the-art on test-time adaptive object detection task. Type equa)on here.

Weight-Averaged Consistency Targets Improve Semi-Supervised Deep Learning Results

Abstract: The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets. To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions. As an additional benefit, Mean Teacher improves test accuracy and enables training with fewer labels than Temporal Ensembling. Without changing the network architecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250 labels, outperforming Temporal Ensembling trained with 1000 labels. We also show that a good network architecture is crucial to performance. Combining Mean Teacher and Residual Networks, we improve the state of the art on CIFAR-10 with 4000 labels from 10.55% to 6.28%, and on ImageNet 2012 with 10% of the labels from 35.24% to 9.11%.


The following lemma was used to define these useful shifts. Lemma 8 (Chan et al. #REFR ).

for each (a, b) where a ∈ Z d−1 and b ∈ Z.

As a result we get the infinite quadtree Q d ∞ where for each integer the cells of the quadtree define a subdivision of H d .

For x, y ∈ R + we define x mod y = x − y x/y . Chan et al.

#OTHEREFR observed that shifting a quadtree by certain special vectors results in a useful shifts also for levels with smaller cells.

Let n > 1 be a positive odd integer, and consider the set

Then, for any α = 2 − , where ≥ 0 is an integer, we have that X mod α = {i/n mod α | i = 0, . . .

, n − 1} is equal to the set αX = {αi/n | i = 0, . . . , n − 1}.

We will look at the projection π z (x, z) = z, or more precisely log π z (x, z).

A Quadtree For Hyperbolic Space

Abstract: We propose a data structure in d-dimensional hyperbolic space that can be considered a natural counterpart to quadtrees in Euclidean spaces. Based on this data structure we propose a so-called L-order for hyperbolic point sets, which is an extension of the Z-order defined in Euclidean spaces. We demonstrate the usefulness of our hyperbolic quadtree data structure by giving an algorithm for constant-approximate closest pair and dynamic constant-approximate nearest neighbours in hyperbolic space of constant dimension d.

On Locality-Sensitive Orderings And Their Applications

Abstract: For any constant $d$ and parameter $\varepsilon > 0$, we show the existence of (roughly) $1/\varepsilon^d$ orderings on the unit cube $[0,1)^d$, such that any two points $p,q\in [0,1)^d$ that are close together under the Euclidean metric are "close together" in one of these linear orderings in the following sense: the only points that could lie between $p$ and $q$ in the ordering are points with Euclidean distance at most $\varepsilon\| p - q \|$ from $p$ or $q$. These orderings are extensions of the $\mathcal{Z}$-order, and they can be efficiently computed. Functionally, the orderings can be thought of as a replacement to quadtrees and related structures (like well-separated pair decompositions). We use such orderings to obtain surprisingly simple algorithms for a number of basic problems in low-dimensional computational geometry, including (i) dynamic approximate bichromatic closest pair, (ii) dynamic spanners, (iii) dynamic approximate minimum spanning trees, (iv) static and dynamic fault-tolerant spanners, and (v) approximate nearest neighbor search.


Linearly scaling the learning rate in this manner was included in the original RigL source code and is further motivated by #REFR .

Our training regimen for training on the ImageNet dataset follows #OTHEREFR

(2021) with the exception of using a mini-batch size of 512 instead of 4096.

We linearly scale the learning rate ∆T to account for our smaller batch size.

Our learning rate uses a linear warm-up to reach a maximum value of 0.2 at epoch five and is reduced by a factor of 10 at epochs 30, 70, and 90.

Using a mini-batch of 512, we train the networks for 256,000 steps to match RigL's training duration.

We increase ∆T to 800 and average the dense gradients over eight mini-batch steps to ensure that SRigL has the same quality of parameter saliency information available as RigL.

We use a cosine connectivity update schedule with α = 0.3.

We initialize the sparse model weights per #OTHEREFR .

We train the networks using SGD with momentum, L2 weight decay, and label smoothing #OTHEREFR coefficients of 0.9, 1e-4 and 0.1, respectively.

We set the minimum percentage of salient weights per neuron to 30% based on our grid search presented in Fig. 9 .

Dynamic Sparse Training With Structured Sparsity

Abstract: Dynamic Sparse Training (DST) methods achieve state-of-the-art results in sparse neural network training, matching the generalization of dense models while enabling sparse training and inference. Although the resulting models are highly sparse and theoretically cheaper to train, achieving speedups with unstructured sparsity on real-world hardware is challenging. In this work we propose a DST method to learn a variant of structured N:M sparsity, the acceleration of which in general is commonly supported in commodity hardware. Furthermore, we motivate with both a theoretical analysis and empirical results, the generalization performance of our specific N:M sparsity (constant fan-in), present a condensed representation with a reduced parameter and memory footprint, and demonstrate reduced inference time compared to dense models with a naive PyTorch CPU implementation of the condensed representation Our source code is available at github.com/calgaryml/condensed-sparsity.

Accurate, Large Minibatch Sgd: Training Imagenet In 1 Hour

Abstract: Deep learning thrives with large neural networks and large datasets. However, larger networks and larger datasets result in longer training times that impede research and development progress. Distributed synchronous SGD offers a potential solution to this problem by dividing SGD minibatches over a pool of parallel workers. Yet to make this scheme efficient, the per-worker workload must be large, which implies nontrivial growth in the SGD minibatch size. In this paper, we empirically show that on the ImageNet dataset large minibatches cause optimization difficulties, but when these are addressed the trained networks exhibit good generalization. Specifically, we show no loss of accuracy when training with large minibatch sizes up to 8192 images. To achieve this result, we adopt a hyper-parameter-free linear scaling rule for adjusting learning rates as a function of minibatch size and develop a new warmup scheme that overcomes optimization challenges early in training. With these simple techniques, our Caffe2-based system trains ResNet-50 with a minibatch size of 8192 on 256 GPUs in one hour, while matching small minibatch accuracy. Using commodity hardware, our implementation achieves ~90% scaling efficiency when moving from 8 to 256 GPUs. Our findings enable training visual recognition models on internet-scale data with high efficiency.