To that end, we adapt the learn-to-grow method #REFR , but do not apply the NAS with respect to the four operations uniformly across layers in a ViT.

It has been observed in neuroscience that learning and memory are entangled together in a highly sophisticated way #OTHEREFR .

For lifelong learning, the longterm memory (LM) maintained by the hippocampal system plays an important role.

A question naturally arises: What would be the counterpart, Artificial Hippocampi (Arti-Hippo), in ViTs to facilitate resilient lifelong learning?

The L2P methods exploit external LM memory to store the learned task prompts/tokens while using frozen Transformer backbones, which may lack the plasticity needed in resilient lifelong learning, increases the cost (due to the quadratic complexity of Transformer models with respect to the number of tokens), and unnecessarily enforces all tasks encountered in lifelong learning (regardless of their underlying difficulty levels) to use the same frozen network.

In this paper, we seek more integrative memory mechanisms that introduce learnable parts into Transformers (in contrast to be entirely frozen) to induce reconfigurability, selectivity and plasticity in lifelong learning.

Instead, we aim to find the ArtiHippo inside the ViT.

As illustrated in Figure 1 , the final projection layer in the multi-head self-attention (MHSA) block of a ViT is identified and selected as the ArtiHippo (Sec. 3.1).

The learn-to-grow NAS is only applied in maintaining ArtiHippo layers, while other components are frozen to maintain the stability of core functions, as illustrated in Figure 2 .

Rather than adopting the DARTS #OTHEREFR NAS used in #OTHEREFR , the learn-to-grow NAS in this paper is built on the single-path one-shot (SPOS) NAS #OTHEREFR , in which we propose a hierarchical exploration-exploitation sampling (Figure 3 ) strategy for lifelong learning (Sec. 3.2).

In experiments, this paper considers lifelong learning with task indices available in both training and inference, which is often referred to as task-incremental setup.

Learning To Grow Artificial Hippocampi In Vision Transformers For Resilient Lifelong Learning

Abstract: Lifelong learning without catastrophic forgetting (i.e., resiliency) possessed by human intelligence is entangled with sophisticated memory mechanisms in the brain, especially the long-term memory (LM) maintained by Hippocampi. To a certain extent, Transformers have emerged as the counterpart "Brain" of Artificial Intelligence (AI), and yet leave the LM component under-explored for lifelong learning settings. This paper presents a method of learning to grow Artificial Hippocampi (ArtiHippo) in Vision Transformers (ViTs) for resilient lifelong learning. With a comprehensive ablation study, the final linear projection layer in the multi-head self-attention (MHSA) block is selected in realizing and growing ArtiHippo. ArtiHippo is represented by a mixture of experts (MoEs). Each expert component is an on-site variant of the linear projection layer, which is maintained via neural architecture search (NAS) with the search space defined by four basic growing operationsskip, reuse, adapt, and new in lifelong learning. The LM of a task consists of two parts: the dedicated expert components (as model parameters) at different layers of a ViT learned via NAS, and the mean class-tokens (as stored latent vectors for measuring task similarity) associated with the expert components. For a new task, a hierarchical task-similarity-oriented exploration-exploitation sampling based NAS is proposed to learn the expert components. The task similarity is measured based on the normalized cosine similarity between the mean class-token of the new task and those of old tasks. The proposed method is complementary to prompt-based lifelong learningwith ViTs. In experiments, the proposed method is tested on the challenging Visual Domain Decathlon (VDD) benchmark and the recently proposed 5-Dataset benchmark. It obtains consistently better performance than the prior art with sensible ArtiHippo learned continually.

Learn To Grow: A Continual Structure Learning Framework For Overcoming Catastrophic Forgetting

Abstract: Addressing catastrophic forgetting is one of the key challenges in continual learning where machine learning systems are trained with sequential or streaming tasks. Despite recent remarkable progress in state-of-the-art deep learning, deep neural networks (DNNs) are still plagued with the catastrophic forgetting problem. This paper presents a conceptually simple yet general and effective framework for handling catastrophic forgetting in continual learning with DNNs. The proposed method consists of two components: a neural structure optimization component and a parameter learning and/or fine-tuning component. By separating the explicit neural structure learning and the parameter estimation, not only is the proposed method capable of evolving neural structures in an intuitively meaningful way, but also shows strong capabilities of alleviating catastrophic forgetting in experiments. Furthermore, the proposed method outperforms all other baselines on the permuted MNIST dataset, the split CIFAR100 dataset and the Visual Domain Decathlon dataset in continual learning setting.


Therefore, we conduct message aggregation via two channels, namely edge channel and hyperedge channel, and devise a novel dual-channel aggregating mechanism. Note that some existing GNN methods #REFR ; Y.

An overview of the Dual-channel Aggregating Mechanism is shown in Fig. 2 (the middle part).

In this module, it iteratively aggregates the information from related roads to encode the structures.

The intuition is that the edge level neighborhoods in the simple graph G R capture pairwise relationships among roads and local receptive fields, and the hyperedge level neighborhoods in the hypergraph G H capture the complex high-order relationships. Both are indispensable and complementary for road modeling.

For example, the hyperedge context (e.g., highway loop) and local environment (e.g., residential area) can jointly determine the function of a ramp road.

Wang and Derr 2021] also model high-order node relationships to help expand the receptive fields and/or improve the expressiveness of GNN model. For example, -GNNs #OTHEREFR

2019] aggregates the information from high-order neighbors defined by the -order cliques, and TDGNN [Y.

#OTHEREFR aggregates the information from multi-hop neighbors defined by the tree decomposed subgraphs.

These methods define different structures on the same simple graph.

In contrast, our aggregation mechanism takes a dual graph perspective and aggregates information from both simple graph and hypergraph.

Road Network Representation Learning: A Dual Graph Based Approach

Abstract: Road network is a critical infrastructure powering many applications including transportation, mobility and logistics in real life. To leverage the input of a road network across these different applications, it is necessary to learn the representations of the roads in the form of vectors, which is named road network representation learning (RNRL). While several models have been proposed for RNRL, they capture the pairwise relationships/connections among roads only (i.e., as a simple graph), and fail to capture among roads the high-order relationships (e.g., those roads that jointly form a local region usually have similar features such as speed limit) and long-range relationships (e.g., some roads that are far apart may have similar semantics such as being roads in residential areas). Motivated by this, we propose to construct a hypergraph, where each hyperedge corresponds to a set of multiple roads forming a region. The constructed hypergraph would naturally capture the high-order relationships among roads with hyperedges. We then allow information propagation via both the edges in the simple graph and the hyperedges in the hypergraph in a graph neural network context. In addition, we introduce different pretext tasks based on both the simple graph (i.e., graph reconstruction) and the hypergraph (including hypergraph reconstruction and hyperedge classification) for optimizing the representations of roads. The graph reconstruction and hypergraph reconstruction tasks are conventional ones and can capture structural information. The hyperedge classification task can capture long-range relationships between pairs of roads that belong to hyperedges with the same label. We call the resulting model HyperRoad. We further extend HyperRoad to problem settings when additional inputs of road attributes and/or trajectories that are generated on the roads are available. We conduct extensive experiments on two real datasets, for five downstream tasks, and under four problem settings, which demonstrate that our model achieves impressive improvements compared with existing baselines across datasets, tasks, problem settings and performance metrics.

Weisfeiler And Leman Go Neural: Higher-Order Graph Neural Networks

Abstract: In recent years, graph neural networks (GNNs) have emerged as a powerful neural architecture to learn vector representations of nodes and graphs in a supervised, end-to-end fashion. Up to now, GNNs have only been evaluated empirically—showing promising results. The following work investigates GNNs from a theoretical point of view and relates them to the 1-dimensional Weisfeiler-Leman graph isomorphism heuristic (1-WL). We show that GNNs have the same expressiveness as the 1-WL in terms of distinguishing non-isomorphic (sub-)graphs. Hence, both algorithms also have the same shortcomings. Based on this, we propose a generalization of GNNs, so-called k-dimensional GNNs (k-GNNs), which can take higher-order graph structures at multiple scales into account. These higher-order structures play an essential role in the characterization of social networks and molecule graphs. Our experimental evaluation confirms our theoretical findings as well as confirms that higher-order information is useful in the task of graph classification and regression.


FVD #REFR is used to measure the generated video from captions and our fine-grained descriptions.

To compare the performance between the caption-generated and the description-generated videos, We report the quantitative results over Frechet Video Distance (FVD) #OTHEREFR metric in Table 5 .

We employ Cogvideo #OTHEREFR to generate the video based on captions and our fine-grained descriptions.

The FVD score reflects that our fine-grained description can generate videos that are more close to the referenced videos. Qualitative results. In Fig. 10 and Fig. 11 , we provide Table 5 . Video generation performance on FAVDBench.

As the text-video model is used to directly infer without fine-tuning, the score is higher than normal ranges. Besides, the smaller FVD is the better.

Fine-Grained Audible Video Description

Abstract: We explore a new task for audiovisual language modeling called fine-grained audible video description (FAVD). It aims to provide detailed textual descriptions for the given audible videos, including the appearance and spatial locations of each object, the actions of moving objects, and the sounds in videos. Existing visual-language modeling tasks often concentrate on visual cues in videos while undervaluing the language and audio modalities. On the other hand, FAVD requires not only audiovisual language modeling skills but also paragraph-level language generation abilities. We construct the first fine-grained audible video description benchmark (FAVDBench) to facilitate this research. For each video clip, we first provide a one-sentence summary of the video, i.e., the caption, followed by 4-6 sentences describing the visual details and 1-2 audio-related descriptions at the end. The descriptions are provided in both English and Chinese. We create two new metrics for this task: an EntityScore to gauge the completeness of entities in the visual descriptions, and an Au-dioScore to assess the audio descriptions. As a preliminary approach to this task, we propose an audio-visuallanguage transformer that extends existing video captioning model with an additional audio branch. We combine the masked language modeling and auto-regressive language modeling losses to optimize our model so that it can produce paragraph-level descriptions. We illustrate the efficiency of our model in audiovisual language modeling by evaluating it against the proposed benchmark using both conventional captioning metrics and our proposed metrics. We further put our benchmark to the test in video generation models, demonstrating that employing finegrained video descriptions can create more intricate videos than using captions.

Towards Accurate Generative Models Of Video: A New Metric&Challenges

Abstract: Recent advances in deep generative models have lead to remarkable progress in synthesizing high quality images. Following their successful application in image processing and representation learning, an important next step is to consider videos. Learning generative models of video is a much harder task, requiring a model to capture the temporal dynamics of a scene, in addition to the visual presentation of objects. Although recent attempts at formulating generative models of video have had some success, current progress is hampered by (1) the lack of qualitative metrics that consider visual quality, temporal coherence, and diversity of samples, and (2) the wide gap between purely synthetic video datasets and challenging real-world datasets in terms of complexity. To this extent we propose Frechet Video Distance (FVD), a new metric for generative models of video based on FID, and StarCraft 2 Videos (SCV), a collection of progressively harder datasets that challenge the capabilities of the current iteration of generative models for video. We conduct a large-scale human study, which confirms that FVD correlates well with qualitative human judgment of generated videos, and provide initial benchmark results on SCV.

fine-grained descriptions
generated video


Arxiv:2305.02350V1 [Cs.Cl] 3 May 2023 2 Related Work

Typically, a pre-trained language model is used, attaching an extra layer for projection to task-specific target variables, and fine-tuning both #REFR .

Currently, text classification is mostly tackled using neural networks.

Virtually every state-of-the-art result for text classification has been achieved using neural language models #OTHEREFR .

This either requires graphics cards with large memory, or low batch sizes.

A substantial amount of research focuses on methods to downsize these pre-trained language models #OTHEREFR .

But, while these approaches reduce the number of parameters drastically, the memory during fine-tuning can still exceed 10 GiB due to gradient calculations.

Prior to language models, the most popular neural approach was to leverage pre-trained word embeddings, like GloVe #OTHEREFR and Word2Vec #OTHEREFR , feed them to task-specific neural architectures #OTHEREFR , and then train on the task data at hand.

Although pre-trained word embeddings also range at around 100M parameters, they are computationally efficient, since no gradient calculations are required.

Using Language Models On Low-End Hardware

Abstract: This paper evaluates the viability of using fixed language models for training text classification networks on low-end hardware. We combine language models with a CNN architecture and put together a comprehensive benchmark with 8 datasets covering single-label and multi-label classification of topic, sentiment, and genre. Our observations are distilled into a list of trade-offs, concluding that there are scenarios, where not fine-tuning a language model yields competitive effectiveness at faster training, requiring only a quarter of the memory compared to fine-tuning.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


In Grad-CAM (Selvaraju et al., 2017) and its generalization Grad-CAM++ #REFR , the final score for a particular class Y c is predicted from the whole image and the algorithm Figure 2: Our proposed framework.

Our goal is to generate heat maps to indicate the important regions that have a positive influence on the output of each prediction.

(a) ODAM generates instance-specific heat maps for explaining the predictions of an object detector; (b) ODAM-Train uses auxiliary losses to encourage heat maps for predictions on the same object to be consistent, and for different objects to be distinct; (c) ODAM-NMS uses the box IoU and the normalized correlation between heat maps to determine the duplicate detections.

The bounding box shows the detection proposal corresponding to the heat map.

ignores distinguishing object instances within.

Their explanation starts from the assumption that the score to be interpreted can be written as a linear combination of its global pooled last convolutional layer feature maps

{A k } k , Y c = k w c k ij A ijk = ij k w c k A ijk , where A ijk indexes location (i, j) of A k .

Odam: Gradient-Based Instance-Specific Vi-Sual Explanations For Object Detection

Abstract: We propose the gradient-weighted Object Detector Activation Maps (ODAM), a visualized explanation technique for interpreting the predictions of object detectors. Utilizing the gradients of detector targets flowing into the intermediate feature maps, ODAM produces heat maps that show the influence of regions on the detector's decision for each predicted attribute. Compared to previous works classification activation maps (CAM), ODAM generates instance-specific explanations rather than class-specific ones. We show that ODAM is applicable to both one-stage detectors and two-stage detectors with different types of detector backbones and heads, and produces higher-quality visual explanations than the state-of-the-art both effectively and efficiently. We next propose a training scheme, Odam-Train, to improve the explanation ability on object discrimination of the detector through encouraging consistency between explanations for detections on the same object, and distinct explanations for detections on different objects. Based on the heat maps produced by ODAM with Odam-Train, we propose Odam-NMS, which considers the information of the model's explanation for each prediction to distinguish the duplicate detected objects. We present a detailed analysis of the visualized explanations of detectors and carry out extensive experiments to validate the effectiveness of the proposed ODAM.

Grad-Cam++: Improved Visual Explanations For Deep Convolutional Networks

Abstract: Over the last decade, Convolutional Neural Network (CNN) models have been highly successful in solving complex vision based problems. However, deep models are perceived as "black box" methods considering the lack of understanding of their internal functioning. There has been a significant recent interest to develop explainable deep learning models, and this paper is an effort in this direction. Building on a recently proposed method called Grad-CAM, we propose Grad-CAM++ to provide better visual explanations of CNN model predictions (when compared to Grad-CAM), in terms of better localization of objects as well as explaining occurrences of multiple objects of a class in a single image. We provide a mathematical explanation for the proposed method, Grad-CAM++, which uses a weighted combination of the positive partial derivatives of the last convolutional layer feature maps with respect to a specific class score as weights to generate a visual explanation for the class label under consideration. Our extensive experiments and evaluations, both subjective and objective, on standard datasets showed that Grad-CAM++ indeed provides better visual explanations for a given CNN architecture when compared to Grad-CAM.


Our results complement the findings by Binns #REFR , who suggest that group and individual fairness may not always be conflicting objectives.

For GAIA LS and GAIA ALS, the disparity between fairness is less pronounced since both versions perform comparatively better than random chance. Fig.

3 also illustrates the value of weighted training which considers both utility and fairness objectives (GAIA).

Even though BNN with distribution shift by itself shows competitive performance compared to the baselines, GAIA consistently outperforms the BNN in terms of utility while matching it in terms of fairness.

This consistent improvement is more pronounced in the Adult dataset, where there are more samples for GAIA to leverage the disparity between ambiguous and non-ambiguous subsets of data. Our experimental results highlight the viability of GAIA (RQ1).

Since prior works have drawn a distinction between group and individual fairness #OTHEREFR , we study how our approach compares against both (RQ2).

Our experiments suggest that ALS introduces an improvement over LS, with a more pronounced improvement in the German dataset compared to the Adult dataset.

In addition, we observe that GAIA outperforms BNN consistently in terms of utility, while the BNN has a minuscule advantage in terms of fairness.

The BNN has a coherent representation due to the regularization effect of the variational inference on the encoding space, where the encoder must output a probabilistic distribution over the latent variables that approximates the true posterior.

This encourages similar samples to have similar encodings, leading to a more organized and smoother latent space representation.

Therefore, it is not surprising that BNNs demonstrate high performance on individual fairness metrics, as these metrics evaluate the consistency in the treatment of similar covariates.

Fairness Through Aleatoric Uncertainty

Abstract: We propose a unique solution to tackle the often-competing goals of fairness and utility in machine learning classification tasks. While fairness ensures that the model's predictions are unbiased and do not discriminate against any particular group, utility focuses on maximizing the accuracy of the model's predictions. Our aim is to investigate the relationship between uncertainty and fairness. Our approach leverages this concept by employing Bayesian learning to estimate the uncertainty in sample predictions where the estimation is independent of confounding effects related to the protected attribute. Through empirical evidence, we show that samples with low classification uncertainty are modeled more accurately and fairly than those with high uncertainty, which may have biased representations and higher prediction errors. To address the challenge of balancing fairness and utility, we propose a novel fairness-utility objective that is defined based on uncertainty quantification. The weights in this objective are determined by the level of uncertainty, allowing us to optimize both fairness and utility simultaneously. Experiments on real-world datasets demonstrate the effectiveness of our approach. Our results show that our method outperforms state-of-the-art methods in terms of the fairness-utility tradeoff and this applies to both group and individual fairness metrics. This work presents a fresh perspective on the trade-off between accuracy and fairness in machine learning and highlights the potential of using uncertainty as a means to achieve optimal fairness and utility. Preprint. Under review.

On The Apparent Conflict Between Individual And Group Fairness

Abstract: A distinction has been drawn in fair machine learning research between `group' and `individual' fairness measures. Many technical research papers assume that both are important, but conflicting, and propose ways to minimise the trade-offs between these measures. This paper argues that this apparent conflict is based on a misconception. It draws on theoretical discussions from within the fair machine learning research, and from political and legal philosophy, to argue that individual and group fairness are not fundamentally in conflict. First, it outlines accounts of egalitarian fairness which encompass plausible motivations for both group and individual fairness, thereby suggesting that there need be no conflict in principle. Second, it considers the concept of individual justice, from legal philosophy and jurisprudence which seems similar but actually contradicts the notion of individual fairness as proposed in the fair machine learning literature. The conclusion is that the apparent conflict between individual and group fairness is more of an artifact of the blunt application of fairness measures, rather than a matter of conflicting principles. In practice, this conflict may be resolved by a nuanced consideration of the sources of `unfairness' in a particular deployment context, and the carefully justified application of measures to mitigate it.


Table 2 shows the hyperparameter settings used for CSL relying on the bnlearn package #REFR for R (R Core Team, 2022).

In Table 1 we provide an overview of all twelve graphs used in Section 5, the randomly sampled target, the adjacency degree of the target and the share of d-separations w.r.t. the target.

This gives further insight into the relation of graph sparsity, degree of target and share of d-separations.

The latter can be regarded as the potential relative runtime decrease for SAGE approximation.

In Algorithm 2 we explicate how we inferred the number of true positive, false positive, true negative and false negative d-separations within an estimated graph and especially how we dealt with the exponential number of potential conditioning sets for the larger graphs.

Efficient Sage Estimation Via Causal Structure Learning

Abstract: The Shapley Additive Global Importance (SAGE) value is a theoretically appealing interpretability method that fairly attributes global importance to a model's features. However, its exact calculation requires the computation of the feature's surplus performance contributions over an exponential number of feature sets. This is computationally expensive, particularly because estimating the surplus contributions requires sampling from conditional distributions. Thus, SAGE approximation algorithms only take a fraction of the feature sets into account. We propose d-SAGE, a method that accelerates SAGE approximation. d-SAGE is motivated by the observation that conditional independencies (CIs) between a feature and the model target imply zero surplus contributions, such that their computation can be skipped. To identify CIs, we leverage causal structure learning (CSL) to infer a graph that encodes (conditional) independencies in the data as d-separations. This is computationally more efficient because the expense of the one-time graph inference and the d-separation queries is negligible compared to the expense of surplus contribution evaluations. Empirically we demonstrate that d-SAGE enables the efficient and accurate estimation of SAGE values.

Learning Bayesian Networks With The Bnlearn R Package

Abstract: bnlearn is an R package (R Development Core Team 2010) which includes several algorithms for learning the structure of Bayesian networks with either discrete or continuous variables. Both constraint-based and score-based algorithms are implemented, and can use the functionality provided by the snow package (Tierney et al. 2008) to improve their performance via parallel computing. Several network scores and conditional independence algorithms are available for both the learning algorithms and independent use. Advanced plotting options are provided by the Rgraphviz package (Gentry et al. 2010).


We speculate that this could be due to the fact that NSD contains more complex natural scenes compared to the previous Generic Object Decoding ( GOD ) dataset #REFR .

The second and third lines represent the results obtained using our proposed MindDiffuser with and without stage 2, respectively.

The fourth and fifth lines represent the results obtained by reproducing the experiment described in Takagi's #OTHEREFR paper and by using Ozcelik's #OTHEREFR provided random version code, respectively.

compared to recent work, our approach produces reconstructed results on NSD that are visually more similar to the ground truth both in terms of semantic and structure.

This suggests that our approach outperforms existing methods in controllable image reconstruction on NSD.

We only have access to the random version code released by Ozcelik #OTHEREFR , and based on their paper, the quantitative results between the random and dense versions are similar. However, this model did not perform well on NSD.

Extracted instance features using only self-supervised pre-trained feature extractor SWAV #OTHEREFR on the visual modality may not contain complete semantic and structural information, leading to unsatisfactory results even if the decoding accuracy of these features is high (with an average accuracy of around 0.6 on the test set). It has been observed that although Takagi et al.

#OTHEREFR and MindDiffuser without stage 2 are able to achieve semantically similar reconstruction results to the original images, such as airplanes, plants on a table, people surfing at a beach, bathrooms, and people skiing, it is clear that these images cannot be aligned with the original images in terms of details such as shape and orientation.

This indicates that the CLIP-based modulation used in stage 2 of this paper can effectively reconstruct results that are both semantically and structurally aligned with the original images.

In order to further quantitatively compare the reconstruction performance of our method with some state-of-the-art methods, we utilize three evaluation metrics to compare from both semantic and structural aspects.

We use cosine similarity in the last layer of CLIP visual branch (512 dimensions) to measure the semantic similarity between the reconstructed results and the original images.

Minddiffuser: Controlled Image Reconstruction From Human Brain Activity With Semantic And Structural Diffusion

Abstract: Reconstructing visual stimuli from measured functional magnetic resonance imaging (fMRI) has been a meaningful and challenging task. Previous studies have successfully achieved reconstructions with structures similar to the original images, such as the outlines and size of some natural images. However, these reconstructions lack explicit semantic information and are difficult to discern. In recent years, many studies have utilized multi-modal pre-trained models with stronger generative capabilities to reconstruct images that are semantically similar to the original ones. However, these images have uncontrollable structural information such as position and orientation. To address both of the aforementioned issues simultaneously, we propose a two-stage image reconstruction model called MindDiffuser, utilizing Stable Diffusion. In Stage 1, the VQ-VAE latent representations and the CLIP text embeddings decoded from fMRI are put into the image-to-image process of Stable Diffusion, which yields a preliminary image that contains semantic and structural information. In Stage 2, we utilize the low-level CLIP visual features decoded from fMRI as supervisory information, and continually adjust the two features in Stage 1 through backpropagation to align the structural information. The results of both qualitative and quantitative analyses demonstrate that our proposed model has surpassed the current state-of-the-art models in terms of reconstruction results on Natural Scenes Dataset (NSD). Furthermore, the results of ablation experiments indicate that each component of our model is effective for image reconstruction.

Generic Decoding Of Seen And Imagined Objects Using Hierarchical Visual Features

Abstract: Object recognition is a key function in both human and machine vision. While recent studies have achieved fMRI decoding of seen and imagined contents, the prediction is limited to training examples. We present a decoding approach for arbitrary objects, using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features including those from a convolutional neural network can be predicted from fMRI patterns and that greater accuracy is achieved for low/high-level features with lower/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, the decoding of imagined objects reveals progressive recruitment of higher to lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.


In this sense, our results represent a substantial generalization of the work in #REFR while maintaining the core features of L 0/1 -SVM.

Recently in the literature, there is a resurging interest in the original SVM problem with the (0, 1) loss, abbreviated as "L 0/1 -SVM", following theoretical and algorithmic developments for optimization problems with the "ℓ 0 -norm", see e.g., #OTHEREFR and the references therein.

In particular, #OTHEREFR proposed KKT-like optimality conditions for the L 0/1 -SVM optimization problem and a efficient ADMM solver to obtain an approximate solution.

In this work, we draw inspiration from the aforementioned papers and present a kernelized version of the theory in which the ambient functional space has a richer structure than the usual Euclidean space.

More precisely, we shall formulate the L 0/1 -SVM problem in the context of Multiple Kernel Learning #OTHEREFR and describe a first-order optimality theory as well as a numerical procedure for the optimization problem via the ADMM.

Obviously, the MKL framework can offer much more flexibility than the single-kernel formulation by letting the optimization algorithm determine the best combination of different kernel functions.

The remainder of this paper is organized as follows.

Section II reviews the classic L 0/1 -SVM in the singlekernel case, while Section III discusses the MKL framework.

Section IV establishes the optimality theory for the MKL-L 0/1 -SVM problem, and in Section V we propose an ADMM algorithm to solve the optimization problem.

Finally, numerical experiments and concluding remarks are provided in Sections VI and VII, respectively.

Notation R + denotes the set of nonnegative reals, and R n + := R + × • • • × R + the n-fold Cartesian product. N m := {1, 2, . . .

Abstract: We formulate the Multiple Kernel Learning (abbreviated as MKL) problem for the support vector machine with the infamous (0, 1)-loss function. Some first-order optimality conditions are given and then exploited to develop a fast ADMM solver for the nonconvex and nonsmooth optimization problem. A simple numerical experiment on synthetic planar data shows that our MKL-L 0/1-SVM framework could be promising.

Support Vector Machine Classifier Via $L_{0/1}$ Soft-Margin Loss

Abstract: Support vector machine (SVM) has attracted great attentions for the last two decades due to its extensive applications, and thus numerous optimization models have been proposed. To distinguish all of them, in this paper, we introduce a new model equipped with an $L_{0/1}$ soft-margin loss (dubbed as $L_{0/1}$-SVM) which well captures the nature of the binary classification. Many of the existing convex/non-convex soft-margin losses can be viewed as a surrogate of the $L_{0/1}$ soft-margin loss. Despite the discrete nature of $L_{0/1}$, we manage to establish the existence of global minimizer of the new model as well as revealing the relationship among its minimizers and KKT/P-stationary points. These theoretical properties allow us to take advantage of the alternating direction method of multipliers. In addition, the $L_{0/1}$-support vector operator is introduced as a filter to prevent outliers from being support vectors during the training process. Hence, the method is expected to be relatively robust. Finally, numerical experiments demonstrate that our proposed method generates better performance in terms of much shorter computational time with much fewer number of support vectors when against with some other leading methods in areas of SVM. When the data size gets bigger, its advantage becomes more evident.


OE #REFR uses purified unlabeled OOD data, which is encouraged to produce a uniform softmax distribution during training.

The mainstream OOD detection works primarily focus on detecting samples with semantic shifts, i.e.OOD samples drawn from classes that do not overlap with known classes.

To enhance the ability to understand ID/OOD discrepancy, some methods synthesize OOD training data such as #OTHEREFR , and another series of OOD detection methods introduce extra OOD data into training.

MCD [32] maximizes the entropy discrepancy between ID and extra OOD samples outputted by two parallel classifiers.

The survey #OTHEREFR indicates that OOD detection with external data generally performs better.

However, massive unlabeled samples could be very easily obtained considering their accessibility, and ID data would inevitably exist among them.

Hence, 'purified unlabeled OOD data' means the mixed ID data need to be manually removed, which brings a high cost and is impractical.

Against the above issues, UDG #OTHEREFR proposes a realistic benchmark that retains ID samples in the unlabeled set, namely SCOOD (Semantically Coherent Out-of-Distribution Detection) benchmarks.

Uncertainty-Aware Optimal Transport For Semantically Coherent Out-Of-Distribution Detection

Abstract: Semantically coherent out-of-distribution (SCOOD) detection aims to discern outliers from the intended data distribution with access to unlabeled extra set. The coexistence of in-distribution and out-of-distribution samples will exacerbate the model overfitting when no distinction is made. To address this problem, we propose a novel uncertainty-aware optimal transport scheme. Our scheme consists of an energy-based transport (ET) mechanism that estimates the fluctuating cost of uncertainty to promote the assignment of semantic-agnostic representation, and an inter-cluster extension strategy that enhances the discrimination of semantic property among different clusters by widening the corresponding margin distance. Furthermore, a T-energy score is presented to mitigate the magnitude gap between the parallel transport and classifier branches. Extensive experiments on two standard SCOOD benchmarks demonstrate the above-par OOD detection performance, outperforming the state-of-the-art methods by a margin of 27.69% and 34.4% on FPR@95, respectively. Code is available at https://github.com/LuFan31/ET-OOD. * Co-first Author. †Corresponding Author. ‡Work done during an internship at Ant Group.

Deep Anomaly Detection With Outlier Exposure

Abstract: It is important to detect anomalous inputs when deploying machine learning systems. The use of larger and more complex inputs in deep learning magnifies the difficulty of distinguishing between anomalous and in-distribution examples. At the same time, diverse image and text data are available in enormous quantities. We propose leveraging these data to improve deep anomaly detection by training anomaly detectors against an auxiliary dataset of outliers, an approach we call Outlier Exposure (OE). This enables anomaly detectors to generalize and detect unseen anomalies. In extensive experiments on natural language processing and small- and large-scale vision tasks, we find that Outlier Exposure significantly improves detection performance. We also observe that cutting-edge generative models trained on CIFAR-10 may assign higher likelihoods to SVHN images than to CIFAR-10 images; we use OE to mitigate this issue. We also analyze the flexibility and robustness of Outlier Exposure, and identify characteristics of the auxiliary dataset that improve performance.


This approach resembles the graph U-Net architecture used in #REFR for image processing tasks such as image classification and segmentation.

This edge augmentation significantly reduces the number of hops required for the nodes to gather messages from distant nodes, thus resulting in a smaller number of GN blocks for faster message propagation.

(b) Multi Graph: This approach draws inspiration from multigrid methods for PDEs, whose foundation is a hierarchy of discretizations (or meshes) 40 .

By performing a series of graph down-sampling operations followed by up-sampling operations, we apply the multigrid concept to graphs and pass information across the computational domain in a hierarchical manner.

In practice, the down-sampling operations correspond to mesh coarsening and result in smaller graphs connecting points that are far away in the domain.

On the other hand, during up-sampling the information is redistributed across the entire graph.

For both approaches, we assume that the computational domain is a triangular mesh which undergoes a transformation into a graph as detailed in Section 4.1 and illustrated in Figure 1 .

Gnn-Based Physics Solver For Time-Independent Pdes

Abstract: Physics-based deep learning frameworks have shown to be effective in accurately modeling the dynamics of complex physical systems with generalization capability across problem inputs. However, time-independent problems pose the challenge of requiring long-range exchange of information across the computational domain for obtaining accurate predictions. In the context of graph neural networks (GNNs), this calls for deeper networks, which, in turn, may compromise or slow down the training process. In this work, we present two GNN architectures to overcome this challenge-the Edge Augmented GNN and the Multi-GNN. We show that both these networks perform significantly better (by a factor of 1.5 to 2) than baseline methods when applied to time-independent solid mechanics problems. Furthermore, the proposed architectures generalize well to unseen domains, boundary conditions, and materials. Here, the treatment of variable domains is facilitated by a novel coordinate transformation that enables rotation and translation invariance. By broadening the range of problems that neural operators based on graph neural networks can tackle, this paper provides the groundwork for their application to complex scientific and industrial settings.

Graph U-Nets

Abstract: We consider the problem of representation learning for graph data. Convolutional neural networks can naturally operate on images, but have significant challenges in dealing with graph data. Given images are special cases of graphs with nodes lie on 2D lattices, graph embedding tasks have a natural correspondence with image pixel-wise prediction tasks such as segmentation. While encoder-decoder architectures like U-Nets have been successfully applied on many image pixel-wise prediction tasks, similar methods are lacking for graph data. This is due to the fact that pooling and up-sampling operations are not natural on graph data. To address these challenges, we propose novel graph pooling (gPool) and unpooling (gUnpool) operations in this work. The gPool layer adaptively selects some nodes to form a smaller graph based on their scalar projection values on a trainable projection vector. We further propose the gUnpool layer as the inverse operation of the gPool layer. The gUnpool layer restores the graph into its original structure using the position information of nodes selected in the corresponding gPool layer. Based on our proposed gPool and gUnpool layers, we develop an encoder-decoder model on graph, known as the graph U-Nets. Our experimental results on node classification and graph classification tasks demonstrate that our methods achieve consistently better performance than previous models.


First, diffusion models often use convolutional neural networks such as U-Net #REFR to learn the reverse process to construct desired data samples from the noise.

VAEs can capture more diversity and are typically easier to scale and train than GANs, but still fall short in terms of visual sample quality and sampling efficiency #OTHEREFR .

Recently, diffusion models such as denoising diffusion probabilistic model (DDPM) #OTHEREFR have emerged as a powerful class of generative models, capable of producing highquality images comparable to those of GANs.

Importantly, they additionally offer desirable properties such as strong sample diversity, faithful distribution coverage, a stationary training objective, and easy scalability.

This implies that diffusion models are well suited for learning models of complex and diverse data, which also motivates us to explore the potential of diffusion-based generative models for graphic layout generation.

Though diffusion models have shown splendid performance in high-fidelity image generation #OTHEREFR , it is still a sparsely explored area and provides unique challenges to develop diffusion-based generative models for layout generation.

However, a layout is a non-sequential data structure consisting of varying length samples with discrete (classes) and continuous (coordinates) elements simultaneously, instead of pixels laid on a regular lattice.

Obviously, convolutional neural networks are not suitable for layout denoising, which prevents diffusion models from being directly applied to layout generation.

Second, the placement and sizing of a given element depend not only on its attributes (such as category label) but also on its relationship to other elements.

How to incorporate the attributes knowledge and model the elements' relationship in diffusion models is still an open problem.

Since diffusion models are general frameworks, they leave room for adapting the underlying neural architectures to exploit the properties of the data.

Layoutdm: Transformer-Based Diffusion Model For Layout Generation

Abstract: Automatic layout generation that can synthesize highquality layouts is an important tool for graphic design in many applications. Though existing methods based on generative models such as Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) have progressed, they still leave much room for improving the quality and diversity of the results. Inspired by the recent success of diffusion models in generating high-quality images, this paper explores their potential for conditional layout generation and proposes Transformer-based Layout Diffusion Model (LayoutDM) by instantiating the conditional denoising diffusion probabilistic model (DDPM) with a purely transformer-based architecture. Instead of using convolutional neural networks, a transformer-based conditional Layout Denoiser is proposed to learn the reverse diffusion process to generate samples from noised layout data. Benefitting from both transformer and DDPM, our Lay-outDM is of desired properties such as high-quality generation, strong sample diversity, faithful distribution coverage, and stationary training in comparison to GANs and VAEs. Quantitative and qualitative experimental results show that our method outperforms state-of-the-art generative models in terms of quality and diversity. * Corresponding author. generate design layouts given a set of design components with user-specified attributes. Though meaningful attempts are made [1, 10, 11, 15, 18, 21, 23-25, 33, 44-46], it is still challenging to generate realistic and complex layouts, because many factors need to be taken into consideration, such as design elements, their attributes, and their relationships to other elements.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .


The style transfer is similar to that of LDR #REFR , but it replaces the transposed convolutions with bilinear upsampling and convolutions.

For example, CDGA #OTHEREFR , SIM #OTHEREFR , MCSSF #OTHEREFR , and BDL+ESL #OTHEREFR use this method or the already transferred images.

In contrast to previous works, the authors of LDR #OTHEREFR train a style translation model that transfers the target domain images in order to make them look like source domain images.

They employ the general translation model of BDL #OTHEREFR but add a cycle-reconstruction loss to enforce semantic consistency between the image and the image reconstructed from the labels.

The active pseudo-labeling (APL) method #OTHEREFR first adapts the target domain images to the source domain using a style transfer.

Afterward, the style-transferred images are used to create pseudo labels that are later used for selfsupervised training in the target domain (cf. III-E).

#OTHEREFR employ a CycleGAN for style transfer from the source to the target domain in their image-level domain adaptation (ILDA) method.

They enforce the similarity of segmentation masks based on style-transferred images and unaltered synthetic images using a discriminator in the generation process to avoid artifacts and guide the synthesis.

The DISE-CT method #OTHEREFR is based on DISE #OTHEREFR but adds a cycle consistency to the generator training.

It also adapts the zero loss #OTHEREFR to a zero-style loss.

Survey On Unsupervised Domain Adaptation For Semantic Segmentation For Visual Perception In Automated Driving

Abstract: The research leading to these results is funded by the German Federal Ministry for Economic Affairs and Energy within the project "KI Delta Learning" (Förderkennzeichen 19A19013C and 19A19013K). The authors would like to thank the consortium for the successful cooperation.

Label-Driven Reconstruction For Domain Adaptation In Semantic Segmentation

Abstract: Unsupervised domain adaptation enables to alleviate the need for pixel-wise annotation in the semantic segmentation. One of the most common strategies is to translate images from the source domain to the target domain and then align their marginal distributions in the feature space using adversarial learning. However, source-to-target translation enlarges the bias in translated images, owing to the dominant data size of the source domain. Furthermore, consistency of the joint distribution in source and target domains cannot be guaranteed through global feature alignment. Here, we present an innovative framework, designed to mitigate the image translation bias and align cross-domain features with the same category. This is achieved by 1) performing the target-to-source translation and 2) reconstructing both source and target images from their predicted labels. Extensive experiments on adapting from synthetic to real urban scene understanding demonstrate that our framework competes favorably against existing state-of-the-art methods.


To increase the representation ability of our model, we introduce 3 Res-Blocks #REFR . Details of the network are shown in Fig. 10 .

Alike to SuperPoint #OTHEREFR , we adopt 8 times downsampling to reduce the resolution of high-dimension features, making the model efficient at test time.

Sfd2: Semantic-Guided Feature Detection And Description

Abstract: Visual localization is a fundamental task for various applications including autonomous driving and robotics. Prior methods focus on extracting large amounts of often redundant locally reliable features, resulting in limited efficiency and accuracy, especially in large-scale environments under challenging conditions. Instead, we propose to extract globally reliable features by implicitly embedding high-level semantics into both the detection and description processes. Specifically, our semantic-aware detector is able to detect keypoints from reliable regions (e.g. building, traffic lane) and suppress unreliable areas (e.g. sky, car) implicitly instead of relying on explicit semantic labels. This boosts the accuracy of keypoint matching by reducing the number of features sensitive to appearance changes and avoiding the need of additional segmentation networks at test time. Moreover, our descriptors are augmented with semantics and have stronger discriminative ability, providing more inliers at test time. Particularly, experiments on long-term large-scale visual localization Aachen Day-Night and RobotCar-Seasons datasets demonstrate that our model outperforms previous local features and gives competitive accuracy to advanced matchers but is about 2 and 3 times faster when using 2k and 4k keypoints, respectively. Code is available at https://github.com/feixue94/sfd2.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


This application consists of two steps: (1) background rendering with NeRF #REFR , and (2) object and mask rendering with our renderer.

While image-based rendering(IBR) can perform view interpolation excellently, the view-dependent effect of highly specular objects, such as chrome spheres, is difficult to reproduce using IBR.

Physically-based rendering(PBR) can handle this view-dependent effect realistically, but PBR requires scene material, geometry, and spatially-varying lighting that is difficult to obtain in the real-world.

Because MAIR can perform accurate inverse rendering in real-world scenes, and can be easily applied to existing view synthesis methods with multi-view images, we can take advantage of IBR and PBR.

The view synthesis result of the scene with chrome sphere inserted is in the accompanied video.

We render the shadow of an object in all images and we train NeRF with these images.

Background including shadow in novel view is rendered with NeRF, and chrome sphere in novel view is rendered with our lighting and renderer.

Among the variants of NeRF, we use DirectVoxGO #OTHEREFR for fast training.

Mair: Multi-View Attention Inverse Rendering With 3D Spatially-Varying Lighting Estimation

Abstract: We propose a scene-level inverse rendering framework that uses multi-view images to decompose the scene into geometry, a SVBRDF, and 3D spatially-varying lighting. Because multi-view images provide a variety of information about the scene, multi-view images in object-level inverse rendering have been taken for granted. However, owing to the absence of multi-view HDR synthetic dataset, scene-level inverse rendering has mainly been studied using single-view image. We were able to successfully perform scene-level inverse rendering using multi-view images by expanding OpenRooms dataset and designing efficient pipelines to handle multi-view images, and splitting spatially-varying lighting. Our experiments show that the proposed method not only achieves better performance than single-view-based methods, but also achieves robust performance on unseen real-world scene. Also, our sophisticated 3D spatially-varying lighting volume allows for photorealistic object insertion in any 3D location.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


We use off-policy Soft Actor Critic #REFR algorithm for optimizing RL objective in our experiments.

The lower primitive action a which is directly executed on the environment, is a 4 dimensional vector with every dimension a i ∈ [0, 1].

The first 3 dimensions provide offsets to be scaled and added to gripper position for moving it to the intended position.

The last dimension provides gripper control(0 implies a fully closed gripper, 0.5 implies a half closed gripper and 1 implies a fully open gripper).

We select 100 randomly generated mazes each for training, testing and validation.

For selecting train, test and validation mazes, we first randomly generate 300 distinct mazes, and then randomly divide them into 100 train, test and validation mazes each. Each experiment is run on 4 parallel workers.

We keep the regularization weight hyperparameter as Ψ = 0.0078. We use Adam #OTHEREFR optimizer.

The hyperparameter u which is the number of training iterations after which the replay buffer is flushed and re-populated is set as 10. The experiments are run for 2.93e6 timesteps.

The method for generating expert demonstrations is provided in Appendix A.3.

Crisp: Curriculum Inducing Primitive Informed Subgoal Prediction For Hierarchical Reinforcement Learning

Abstract: Hierarchical reinforcement learning is a promising approach that uses temporal abstraction to solve complex long horizon problems. However, simultaneously learning a hierarchy of policies is unstable as it is challenging to train higherlevel policy when the lower-level primitive is nonstationary. In this paper, we propose a novel hierarchical algorithm by generating a curriculum of achievable subgoals for evolving lower-level primitives using reinforcement learning and imitation learning. The lower level primitive periodically performs data relabeling on a handful of expert demonstrations using our primitive informed parsing approach. We provide expressions to bound the sub-optimality of our method and develop a practical algorithm for hierarchical reinforcement learning. Since our approach uses a handful of expert demonstrations, it is suitable for most robotic control tasks. Experimental evaluation on complex maze navigation and robotic manipulation environments show that inducing hierarchical curriculum learning significantly improves sample efficiency, and results in efficient goal conditioned policies for solving temporally extended tasks.

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning With A Stochastic Actor

Abstract: Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy - that is, succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.


Motivated by the results presented in #REFR , a Densely Connected Convolutional Network (DenseNet) is used to implement the "shared" encoder shown in Fig. 4 .

This equalization approach uses a joint CAE and CNN (a.k.a., JCAECNN) architecture similar to that used in #OTHEREFR ; however, the approach described here differs from that in #OTHEREFR in that the CAE architecture is modified to consist of multiple decoder heads instead of one.

The decoder heads decompose a received preamble-that undergoes Rayleigh fading-into its L weighted and delayed copies of the transmitted signal, x[n]. As illustrated in Fig.

4 , the JCAECNN equalization process is built on a Single Input Multiple Output (SIMO) system that includes: signal collection and detection (Sect. III-A), multipath fading and AWGN scaling (Sect.

III-C), and signal preparation-in accordance with our prior work #OTHEREFR -prior to equalization and classification.

Equalization is performed using a CAE consisting of a single encoder and L decoder heads (i.e., one for each path of the Rayleigh fading channel) using the preambles' IQ+NL representations at the selected SNR.

DenseNet is comprised of multiple Dense blocks where each block is created by connecting each convolutional layer to all preceding convolutional layers of the same size.

These dense connections grant each convolutional layer access to all previously generated feature maps to enable feature reuse.

In this work, two densely connected blocks are used to construct the encoder.

The DenseNet encoder generates compressed representation h(R[m, n]), which is the input shared with each of the following NN architectures: the L decoder heads and the classifier The resulting JCAECNN architecture is derived from the square error function of equation #OTHEREFR .

The optimization goal is to minimize the error between the received signal r(m) and its delayed and scaled copies C k = α k x(m − τ k ) generated using the TDL of equation #OTHEREFR .

Improving Rf-Dna Fingerprinting Performance In An Indoor Multipath Environment Using Semi-Supervised Learning

Abstract: The number of Internet of Things (IoT) deployments is expected to reach 75.4 billion by 2025. Roughly 70% of all IoT devices employ weak or no encryption; thus, putting them and their connected infrastructure at risk of attack by devices that are wrongly authenticated or not authenticated at all. A physical layer-based security approach-known as Specific Emitter Identification (SEI)-has been proposed and is being pursued as a viable IoT security mechanism. SEI is advantageous because it is a passive technique that exploits inherent and distinct features that are unintentionally imparted upon the signal during its formation and transmission within and by the IoT device's Radio Frequency (RF) front-end. SEI's passive exploitation of unintentional signal features removes any need to modify the IoT device, which makes it ideal for existing and future IoT deployments. Despite the amount of SEI research conducted there remains challenges that must be addressed to make SEI a viable IoT security approach. One of these challenges is the extraction of SEI features from signals collected under multipath fading conditions. Multipath corrupts the inherent SEI exploited features that are used to discriminate one IoT device from another; thus, degrading authentication performance and increasing the chance of attack. This work presents two semi-supervised Deep Learning (DL) equalization approaches and compares their performance with the current state of the art. The two approaches are the Conditional Generative Adversarial Network (CGAN) and Joint Convolutional Auto-Encoder and Convolutional Neural Network (JCAECNN). Both approaches learn the channel distribution to enable multipath correction while simultaneously preserving the SEI exploited signal features. CGAN and JCAECNN performance is assessed using a Rayleigh fading channel under degrading SNR, up to thirty-two IoT devices, as well as two publicly available signal sets. The JCAECNN improves SEI performance by 10% beyond that of the current state of the art.

Densely Connected Convolutional Networks

Abstract: Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections - one between each layer and its subsequent layer - our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less computation to achieve high performance. Code and pre-trained models are available at this https URL .

DenseNet
Densely Connected Convolutional


On the other hand, NER and CR are represented more strongly in the higher layers, which confirms the related findings in the literature #REFR .

On the other hand, our NormFormer-like architecture (Section 4) defines each layer k as:

a k (x) = x + LN(a k (x)) k =â k ( k−1 ) + ff k (â k ( k−1 ))

Then it is trivial to calculate the contribution of each layer as s k = k − k−1 .

We use this s k entities to compute the learnt convex combination of all layersŝ = 12 k=1 γ k s k .

Interpreting γ k as the amount of 'knowledge' of a particular task in layer k, we see that POS information is contained primarily in the lowest layers, followed by SRL and DP.

Trained On 100 Million Words And Still In Shape: Bert Meets British National Corpus

Abstract: While modern masked language models (LMs) are trained on ever larger corpora, we here explore the effects of down-scaling training to a modestly-sized but representative, wellbalanced, and publicly available English text source-the British National Corpus. We show that pre-training on this carefully curated corpus can reach better performance than the original BERT model. We argue that this type of corpora has great potential as a language modeling benchmark. To showcase this potential, we present fair, reproducible and data-efficient comparative studies of LMs, in which we evaluate several training objectives and model architectures and replicate previous empirical results in a systematic way. We propose an optimized LM architecture called LTG-BERT. bert#pre-training-data).

A Primer In Bertology: What We Know About How Bert Works

Abstract: Transformer-based models are now widely used in NLP, but we still do not understand a lot about their inner workings. This paper describes what is known to date about the famous BERT model (Devlin et al. 2019), synthesizing over 40 analysis studies. We also provide an overview of the proposed modifications to the model and its training regime. We then outline the directions for further research.


Real data from the Waymo #REFR dataset is used to evaluate NFL against real scans at held-out viewpoints, including real-world intensities, ray drops and secondary returns.

This includes the possibilities that there are multiple returns 1 from the emitted ray, or no return at all.

Our formulation closely adheres to the principles of the LiDAR measurement process and incorporates them into the neural field framework.

Specifically, we (i) devise volume rendering for LiDAR sensors; (ii) incorporate beam divergence and (iii) propose truncated volume rendering to account for secondary returns and improve range prediction.

We evaluate our method on both synthetic and real Li-DAR data.

To this end, we (iv) develop a LiDAR simulator for synthesizing scenes from 3D assets that serve as a test bed for viewpoints far from the original scan locations, and to study the effect of different scan patterns.

Additionally, we (v) propose a novel closed-loop evaluation protocol that leverages real data to evaluate view synthesis in challenging views.

As an end-to-end test for downstream tasks, we further evaluate the performance of state-of-the-art segmentation and registration networks when trained on real scans and tested on novel views generated by NFL.

Neural Lidar Fields For Novel View Synthesis

Abstract: We present Neural Fields for LiDAR (NFL), a method to optimise a neural field scene representation from LiDAR measurements, with the goal of synthesizing realistic LiDAR scans from novel viewpoints. NFL combines the rendering power of neural fields with a detailed, physically motivated model of the LiDAR sensing process, thus enabling it to accurately reproduce key sensor behaviors like beam divergence, secondary returns, and ray dropping. We evaluate NFL on synthetic and real LiDAR scans and show that it outperforms explicit reconstruct-then-simulate methods as well as other NeRF-style methods on LiDAR novel view synthesis task. Moreover, we show that the improved realism of the synthesized views narrows the domain gap to real scans and translates to better registration and semantic segmentation performance.

Scalability In Perception For Autonomous Driving: Waymo Open Dataset

Abstract: The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the overall viability of the technology. In an effort to help align the research community's contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data captured across a range of urban and suburban geographies. It is 15x more diverse than the largest camera+LiDAR dataset available based on our proposed diversity metric. We exhaustively annotated this data with 2D (camera image) and 3D (LiDAR) bounding boxes, with consistent identifiers across frames. Finally, we provide strong baselines for 2D as well as 3D detection and tracking tasks. We further study the effects of dataset size and generalization across geographies on 3D detection methods. Find data, code and more up-to-date information at http://www.waymo.com/open.


We designed a deep learning model based on VGG-Net #REFR to meet our goals.

Although many methods such as molecular dynamics, have been evaluated #OTHEREFR and used in microstructure analysis convolution neutral network #OTHEREFR (CNN) is a better choice in automatic image recognition since the development of image datasets (such as ImageNet #OTHEREFR ) and computational power.

Some kinds of model architectures with CNN module have been proposed.

At the same time, we applied some competing models to our dataset, including other deep learning algorithms such as Dense-Net #OTHEREFR and Res-Net #OTHEREFR and a traditional edge detection algorithm like Sobel. Details of these models are introduced as follows.

VGG-Net CNN extracts different morphological features of the image through the convolution, and then pools the image, and finally gets the predicted result. VGG-Net was proposed by Oxford's Visual Geometry Group.

VGG-Net uses two or three 2×2 or 3×3 convolution kernels to replace the larger convolution nuclei in the previous convolutional neural network.

The stacked small convolution kernel can improve the depth of the network while maintaining the same receptive field, and more complex models can be learned through multi-layer nonlinear layers.

Automatically Predict Material Properties With Microscopic Image: Example Polymer Compatibility

Abstract: Many material properties are manifested in the morphological appearance and characterized with microscopic image, such as scanning electron microscopy (SEM). Polymer compatibility is a key physical quantity of polymer material and commonly and intuitively judged by SEM images. However, human observation and judgement for the images is time-consuming, labor-intensive and hard to be quantified. Computer image recognition with machine learning method can make up the defects of artificial judging, giving accurate and quantitative judgement. We achieve automatic compatibility recognition utilizing convolution neural network and transfer learning method, and the model obtains up to 94% accuracy. We also put forward a quantitative criterion for polymer compatibility with this model. The proposed method can be widely applied to the quantitative characterization of the microstructure and properties of various materials.

Very Deep Convolutional Networks For Large-Scale Image Recognition

Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.


Potential Of Existing Point Cloud Benchmarks For Façade Segmentation

Still, even simulated total points numbers were lower than the ones of TLS datasets: the semantic3D.net TLS dataset consisted of 4 BN points #REFR .

Interestingly, our set consisted not only of real-world point clouds but also included synthetic-world point clouds.

This was the case with both SynthCity #OTHEREFR and Paris-CARLA-3D #OTHEREFR .

The former presented a completely simulated MLS point cloud based on a vector model and covering a combination of European mainland cities and New York, USA #OTHEREFR .

The latter combined acquired point clouds with simulated ones using the CARLA environment #OTHEREFR .

One distinct advantage of synthetic point clouds was that they can easily outnumber the real ones: 700 M to 60 M in the case of Paris-CARLA-3D #OTHEREFR .

On the other hand, the KITTI-360 MLS dataset featured 1 BN points, with 73.7 km of roads being measured in Karlsruhe, Germany #OTHEREFR .

It was thus clear that quantity of points was directly linked to the pace of acquisition (e.g., MLS is intuitively faster than TLS) and the total covered area.

The latter is particularly difficult to acquire and compare, since the various datasets have different ways of quantifing this measure, namely: as a number of scenes, the length of road driven, approximate area extent, or else it is unpublished.

Moreover, three obtained datasets #OTHEREFR did not reveal their total number of points, thereby this statistic is omitted in these cases.

Regarding point clouds georeferencing, ou analysis showed that the set was equally divided between those published in a local CRS and in those in a global CRS, both had a score of eight datasets.

Tum-Faç Ade: Reviewing And Enriching Point Cloud Benchmarks For Faç Ade Segmentation

Abstract: Point clouds are widely regarded as one of the best dataset types for urban mapping purposes. Hence, point cloud datasets are commonly investigated as benchmark types for various urban interpretation methods. Yet, few researchers have addressed the use of point cloud benchmarks for façade segmentation. Robust façade segmentation is becoming a key factor in various applications ranging from simulating autonomous driving functions to preserving cultural heritage. In this work, we present a method of enriching existing point cloud datasets with façade-related classes that have been designed to facilitate façade segmentation testing. We propose how to efficiently extend existing datasets and comprehensively assess their potential for façade segmentation. We use the method to create the TUM-FAÇ ADE dataset, which extends the capabilities of TUM-MLS-2016. Not only can TUM-FAÇ ADE facilitate the development of point-cloud-based façade segmentation tasks, but our procedure can also be applied to enrich further datasets.

Semantic3D.Net: A New Large-Scale Point Cloud Classification Benchmark

Abstract: This paper presents a new 3D point cloud classification benchmark data set with over four billion manually labelled points, meant as input for data-hungry (deep) learning methods. We also discuss first submissions to the benchmark that use deep convolutional neural networks (CNNs) as a work horse, which already show remarkable performance improvements over state-of-the-art. CNNs have become the de-facto standard for many tasks in computer vision and machine learning like semantic segmentation or object detection in images, but have no yet led to a true breakthrough for 3D point cloud labelling tasks due to lack of training data. With the massive data set presented in this paper, we aim at closing this data gap to help unleash the full potential of deep learning methods for 3D labelling tasks. Our semantic3D.net data set consists of dense point clouds acquired with static terrestrial laser scanners. It contains 8 semantic classes and covers a wide range of urban outdoor scenes: churches, streets, railroad tracks, squares, villages, soccer fields and castles. We describe our labelling interface and show that our data set provides more dense and complete point clouds with much higher overall number of labelled points compared to those already available to the research community. We further provide baseline method descriptions and comparison between methods submitted to our online system. We hope semantic3D.net will pave the way for deep learning methods in 3D point cloud labelling to learn richer, more general 3D representations, and first submissions after only a few months indicate that this might indeed be the case.


Specifically, the ART #REFR generates the most likely hypothesis (text) to explain what has happened between the two observations (texts).

The main idea behind the VL foundation models is to learn transferable visual representation with noisy text supervision through a two-tower structure.

We follow the current VAR pipeline and adopt the foundation model CLIP to generate initial visual & textual features.

Humans reckon plausible inferences or hypotheses from incomplete observations every day #OTHEREFR .

To teach the machine with this ability, researchers proposed several new Tasks, like ART #OTHEREFR for NLP, Sherlock #OTHEREFR for vision, and VideoVAR #OTHEREFR , VideoABC #OTHEREFR for video.

For Sherlock, Video-VAR, and VideoABC, the observations are represented by regional or whole images, while inference could be either text or middle frames.

There are several similar tasks, such as Visual Commonsense Reasoning (VCR) #OTHEREFR and Vi-sual7W #OTHEREFR .

Abductive reasoning differs from them in having non-definitely correct, plausible inferences as human beings.

Prompt Tuning comes from adapting the LLM to downstream NLP tasks by adding a few "virtual tokens" as the prefix or suffix to original sentences.

According to whether using trainable continuous or handcrafted discrete "virtual tokens", we could classify prompts into soft #OTHEREFR or hard #OTHEREFR 26] prompts.

Fine-Grained Regional Prompt Tuning For Visual Abductive Reasoning

Abstract: Visual Abductive Reasoning (VAR) is an emerging vision-language (VL) topic where the model needs to retrieve/generate a likely textual hypothesis from a visual input (image or part of an image) using backward reasoning based on prior knowledge or commonsense. Unlike in conventional VL retrieval or captioning tasks, where entities of texts appear in the image, in abductive inferences, the relevant facts about inferences are not directly visible in the input images. Besides, the inferences are causally relevant to regional visual hints and vary with the latter. Existing works highlight visual parts from a global background with specific prompt tuning techniques (e.g., colorful prompt tuning) on top of foundation models, like CLIP. However, these methods uniformly patchify "regional hints" and "global context" at the same granularity level and may lose finegrained visual details significant for abductive reasoning. To tackle this, we propose a simple yet effective Regional Prompt Tuning, which encodes "regional visual hints" and "global contexts" separately at fine and coarse-grained levels. Specifically, our model explicitly upsamples, then patchify local hints to get fine-grained regional prompts. These prompts are concatenated with coarse-grained contextual tokens from whole images. We also equip our model with a new Dual-Contrastive Loss to regress the visual feature simultaneously toward features of factual description (a.k.a. clue text) and plausible hypothesis (abductive inference text) during training. Extensive experiments on the Sherlock dataset demonstrate that our fully fine-tuned RGP/RGP s with Dual-Contrastive Loss significantly outperforms previous SOTAs, achieving the 1 st rank on abductive reasoning leaderboards among all submissions, under all metrics (e.g., P@1 i→t : RGP s 38.78 vs CPT-CLIP 33.44, higher=better). We would open-source our codes for further research. (a) Performance improvements obtained by RGP, RGPs and Dual-Contrastive Loss using two ViT backbone architectures. X axis is Image ⇆ T ext Mean Rank (i.e., observation ⇆ hypothesis; lower=better). (b) Overview of Regional Prompt Tuning and Dual-Contrastive Loss Figure 1: Effects and Methodology of Regional Prompt Tuning on the Visual Abductive Reasoning Task. (a). Performance boosts on the Sherlock testing set; (b) The RGP patchify regional observation in more fine-grained level than its colorful counterpart (CPT), and Dual-Contrasive Loss serves to shrink gaps between "Vision-Inference/-Clue" modalities simultaneously.

Abductive Commonsense Reasoning

Abstract: Abductive reasoning is inference to the most plausible explanation. For example, if Jenny finds her house in a mess when she returns from work, and remembers that she left a window open, she can hypothesize that a thief broke into her house and caused the mess, as the most plausible explanation. While abduction has long been considered to be at the core of how people interpret and read between the lines in natural language (Hobbs et al., 1988), there has been relatively little research in support of abductive natural language inference and generation. We present the first study that investigates the viability of language-based abductive reasoning. We introduce a challenge dataset, ART, that consists of over 20k commonsense narrative contexts and 200k explanations. Based on this dataset, we conceptualize two new tasks – (i) Abductive NLI: a multiple-choice question answering task for choosing the more likely explanation, and (ii) Abductive NLG: a conditional generation task for explaining given observations in natural language. On Abductive NLI, the best model achieves 68.9% accuracy, well below human performance of 91.4%. On Abductive NLG, the current best language generators struggle even more, as they lack reasoning capabilities that are trivial for humans. Our analysis leads to new insights into the types of reasoning that deep pre-trained language models fail to perform—despite their strong performance on the related but more narrowly defined task of entailment NLI—pointing to interesting avenues for future research.


The work in #REFR aims to optimize the beamformers associated with the common message and private messages to maximize the sum rate of CUs in multi-user multiple input single output (MISO) systems under imperfect channel state information.

However, the CUs in the guard zone have bad QoS experience since they are out of the coverage of the communication system.

To address this issue, the work in #OTHEREFR proposes to deploy intelligent reflecting surface (IRS) in the guard zone to enhance the network coverage and QoS of the users.

There also exist several works related to RSMA, and the readers are referred to a comprehensive survey on RSMA in #OTHEREFR . However, the existing works do not investigate CRC.

In particular, the authors in #OTHEREFR and #OTHEREFR evaluate the performance obtained by RSMA.

These works demonstrate that RSMA outperforms both NOMA and SDMA in terms of spectrum efficiency, energy efficiency, QoS, and computational complexity in different network loads and channel conditions.

The authors in #OTHEREFR analyze the data rate obtained by RSMA for two-receiver MISO broadcast channel with finite rate feedback.

Considering a multi-user single input single output (SISO), the authors in #OTHEREFR aim to allocate the common rate, transmit power of the common message, and transmit power of the private messages by accounting for successful SIC power requirements.

Different from #OTHEREFR , #OTHEREFR , and #OTHEREFR , the works in #OTHEREFR and #OTHEREFR aim to achieve max-min fairness among the users, i.e., maximizing the user capacity with the minimum data rate, in the RSMA-based network.

Recently, RSMA can also be combined with enabling technologies in 6G.

In particular, RSMA is shown to efficiently manage the inter-beam interference for satellite communications #OTHEREFR .

Joint Rate Allocation And Power Control For Rsma-Based Communication And Radar Coexistence Systems

Abstract: We consider a rate-splitting multiple access (RSMA)-based communication and radar coexistence (CRC) system. The proposed system allows an RSMA-based communication system to share spectrum with multiple radars. Furthermore, RSMA enables flexible and powerful interference management by splitting messages into common parts and private parts to partially decode interference and partially treat interference as noise. The RSMA-based CRC system thus significantly improves spectral efficiency, energy efficiency and quality of service (QoS) of communication users (CUs). However, the RSMA-based CRC system raises new challenges. Due to the spectrum sharing, the communication network and the radars cause interference to each other, which reduces the signal-to-interference-plus-noise ratio (SINR) of the radars as well as the data rate of the CUs in the communication network. Therefore, a major problem is to maximize the sum rate of the CUs while guaranteeing their QoS requirements of data transmissions and the SINR requirements of multiple radars. To achieve these objectives, we formulate a problem that optimizes i) the common rate allocation to the CUs, transmit power of common message and transmit power of private messages of the CUs, and ii) transmit power of the radars. The problem is non-convex with multiple decision parameters, which is challenging to be solved. We propose two algorithms. The first sequential quadratic programming (SQP) can quickly return a local optimal solution, and has been known to be the state-of-the-art in nonlinear programming methods. The second is an additive approximation scheme (AAS) which solves the problem globally in a reasonable amount of time, based on the technique of applying exhaustive enumeration to a modified instance. The simulation results show the improvement of the AAS compared with the SQP in terms of sum rate. Furthermore, with the AAS, the sum rate of the CUs only slightly decreases when the radars' SINR is significantly increased. This implies that the AAS supports the RSMA-based communication system which allows to well coexist with the radars.

Sum-Rate Maximization For Linearly Precoded Downlink Multiuser Miso Systems With Partial Csit: A Rate-Splitting Approach

Abstract: This paper considers the Sum-Rate (SR) maximization problem in downlink MU-MISO systems under imperfect Channel State Information at the Transmitter (CSIT). Contrary to existing works, we consider a rather unorthodox transmission scheme. In particular, the message intended to one of the users is split into two parts: a common part which can be recovered by all users, and a private part recovered by the corresponding user. On the other hand, the rest of users receive their information through private messages. This Rate-Splitting (RS) approach was shown to boost the achievable Degrees of Freedom (DoF) when CSIT errors decay with increased SNR. In this work, the RS strategy is married with linear precoder design and optimization techniques to achieve a maximized Ergodic SR (ESR) performance over the entire range of SNRs. Precoders are designed based on partial CSIT knowledge by solving a stochastic rate optimization problem using means of Sample Average Approximation (SAA) coupled with the Weighted Minimum Mean Square Error (WMMSE) approach. Numerical results show that in addition to the ESR gains, the benefits of RS also include relaxed CSIT quality requirements and enhanced achievable rate regions compared to conventional transmission with NoRS.


Equipping a model with the option to reject has also been studied in selective classification #REFR .

We argue that whether an example with covariate shift should be detected or not depends on whether the deployed classifier would misclassify it or not.

By taking a modelspecific perspective, our MS-OOD DETECTION resolves the dilemma between OOD detection #OTHEREFR and OOD generalization #OTHEREFR : a robust model should generalizes to examples with covariate shift; a weak model should reject them. OOD generalization.

Covariate shift is commonly studied in model generalization and robustness #OTHEREFR .

Instead of rejecting examples with covariate shift, the community aims to improve the robustness of a neural network model so that the model could classify them correctly.

However, given the notable accuracy gap between classifying ID examples and covariate-shift examples #OTHEREFR , we argue that it is desirable to also consider covariate shift in OOD detection. Selective classification.

Different from OOD detection, selective classification focuses on rejecting uncertain "ID" examples.

Recently, #OTHEREFR proposed to integrate selective classification with OOD detection (SCOD), aiming to detect both misclassified ID and semantic shift data (see Figure 2 (d)).

In this context, our work can be seen as a generalized version, further taking covariate shifts into account (see Figure 2 (e)).

Compared to #OTHEREFR , we provide a more comprehensive study, further emphasizing the role of models in evaluation.

OOD detection methods have roughly two categories: posthoc and training-based #OTHEREFR .

Unified Out-Of-Distribution Detection: A Model-Specific Perspective

Abstract: Out-of-distribution (OOD) detection aims to identify test examples that do not belong to the training distribution and are thus unlikely to be predicted reliably. Despite a plethora of existing works, most of them focused only on the scenario where OOD examples come from semantic shift (e.g., unseen categories), ignoring other possible causes (e.g., covariate shift). In this paper, we present a novel, unifying framework to study OOD detection in a broader scope. Instead of detecting OOD examples from a particular cause, we propose to detect examples that a deployed machine learning model (e.g., an image classifier) is unable to predict correctly. That is, whether a test example should be detected and rejected or not is "model-specific". We show that this framework unifies the detection of OOD examples caused by semantic shift and covariate shift, and closely addresses the concern of applying a machine learning model to uncontrolled environments. We provide an extensive analysis that involves a variety of models (e.g., different architectures and training strategies), sources of OOD examples, and OOD detection approaches, and reveal several insights into improving and understanding OOD detection in uncontrolled environments.

Selective Classification For Deep Neural Networks

Abstract: Selective classification techniques (also known as reject option) have not yet been considered in the context of deep neural networks (DNNs). These techniques can potentially significantly improve DNNs prediction performance by trading-off coverage. In this paper we propose a method to construct a selective classifier given a trained neural network. Our method allows a user to set a desired risk level. At test time, the classifier rejects instances as needed, to grant the desired risk (with high probability). Empirical results over CIFAR and ImageNet convincingly demonstrate the viability of our method, which opens up possibilities to operate DNNs in mission-critical applications. For example, using our method an unprecedented 2% error in top-5 ImageNet classification can be guaranteed with probability 99.9%, and almost 60% test coverage.