In our implementation, we adopt the qpth library #REFR to build the graph matching module.

After enhancing the vertex features and constructing the edge features on graph G D and G T , we meet the core component of our method: the differentiable graph matching layer. By optimizing the QP in Eq.

6 from quadratic affinity matrix M and vertex affinity matrix B, we can derive the optimal matching score vector x and reshape it back to the shape n d × n t to get the matching score map X.

Since we finally formulate the graph matching problem as a QP, we can construct the graph matching module as a differentiable QP layer in our neural network.

Since KKT conditions are the necessary and sufficient conditions for the optimal solution x * and its dual variables, we could derive the gradient in backward pass of our graph matching layer based on the KKT conditions and implicit function theorem, which is inspired by OptNet #OTHEREFR .

In the inference stage, to reduce the computational cost and accelerate the algorithm, we solve the QP using the CVXPY library #OTHEREFR only for forward operation.

For training, we use weighted binary cross entropy Loss:

whereŷ i,j denotes the matching score between detection D i and tracklet T j , and y i,j is the ground truth indicating whether the object belongs to the tracklet.

k = (n t − 1) is the weight to balance the loss between positive and negative samples.

Learnable Graph Matching: A Practical Paradigm For Data Association

Abstract: Data association is at the core of many computer vision tasks, e.g., multiple object tracking, image matching, and point cloud registration. Existing methods usually solve the data association problem by network flow optimization, bipartite matching, or end-to-end learning directly. Despite their popularity, we find some defects of the current solutions: they mostly ignore the intra-view context information; besides, they either train deep association models in an end-to-end way and hardly utilize the advantage of optimization-based assignment methods, or only use an off-the-shelf neural network to extract features. In this paper, we propose a general learnable graph matching method to address these issues. Especially, we model the intra-view relationships as an undirected graph. Then data association turns into a general graph matching problem between graphs. Furthermore, to make optimization end-to-end differentiable, we relax the original graph matching problem into continuous quadratic programming and then incorporate training into a deep graph neural network with KKT conditions and implicit function theorem. In MOT task, our method achieves state-of-the-art performance on several MOT datasets. For image matching, our method outperforms state-of-the-art methods with half training data and iterations on a popular indoor dataset, ScanNet. Code will be available at https://github.com/jiaweihe1996/GMTracker.

Optnet: Differentiable Optimization As A Layer In Neural Networks

Abstract: This paper presents OptNet, a network architecture that integrates optimization problems (here, specifically in the form of quadratic programs) as individual layers in larger end-to-end trainable deep networks. These layers allow complex dependencies between the hidden states to be captured that traditional convolutional and fully-connected layers are not able to capture. In this paper, we develop the foundations for such an architecture: we derive the equations to perform exact differentiation through these layers and with respect to layer parameters; we develop a highly efficient solver for these layers that exploits fast GPU-based batch solves within a primal-dual interior point method, and which provides backpropagation gradients with virtually no additional cost on top of the solve; and we highlight the application of these approaches in several problems. In one particularly standout example, we show that the method is capable of learning to play Sudoku given just input and output games, with no a priori information about the rules of the game; this task is virtually impossible for other neural network architectures that we have experimented with, and highlights the representation capabilities of our approach.


A. Marl Framework Of Interactive Image Segmentation

The asynchronous advantage actorcritic (A3C) #REFR scheme is employed in BS-IRIS and further extended to a fully convolutional form, where agents can collaborate and communicate through convolutional layers. The algorithm is summarized in Alg. 1.

It is impractical to directly apply typical multi-agent learning algorithms such as #OTHEREFR .

2) All voxel-type agents are neatly aligned in a 3D grid and dependent with each other for the segmentation task.

It is necessary to require them to cooperate with each other.

This is achieved by enforcing all voxel agents to share the same policy.

When one agent explores a beneficial action, other agents will simultaneously acquire that knowledge, which also significantly reduces the number of parameters.

Boundary-Aware Supervoxel-Level Iteratively Refined Interactive 3D Image Segmentation With Multi-Agent Reinforcement Learning

Abstract: Interactive segmentation has recently been explored to effectively and efficiently harvest high-quality segmentation masks by iteratively incorporating user hints. While iterative in nature, most existing interactive segmentation methods tend to ignore the dynamics of successive interactions and take each interaction independently. We here propose to model iterative interactive image segmentation with a Markov decision process (MDP) and solve it with reinforcement learning (RL) where each voxel is treated as an agent. Considering the large exploration space for voxel-wise prediction and the dependence among neighboring voxels for the segmentation tasks, multi-agent reinforcement learning is adopted, where the voxel-level policy is shared among agents. Considering that boundary voxels are more important for segmentation, we further introduce a boundary-aware reward, which consists of a global reward in the form of relative cross-entropy gain, to update the policy in a constrained direction, and a boundary reward in the form of relative weight, to emphasize the correctness of boundary predictions. To combine the advantages of different types of interactions, i.e., simple and efficient for point-clicking, and stable and robust for scribbles, we propose a supervoxel-clicking based interaction design. Experimental results on four benchmark datasets have shown that the proposed method significantly outperforms the state-of-the-arts, with the advantage of fewer interactions, higher accuracy, and enhanced robustness.

Asynchronous Methods For Deep Reinforcement Learning

Abstract: We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.

convolutional layers
asynchronous advantage actorcritic


The exponential mechanism we use at the first stage is a Gumbel noise style implementation #REFR which is analyzed with RDP via a "Bounded Range" property for tighter bound.

On the attack aspect, we have proposed three knowledge graph(KG) triple inference attacks on FKGE to expose its significant privacy vulnerability.

On the defense aspect, DP-FLames is proposed to provide rigorous differential privacy protection for FKGE, which exploits the sparse gradient property of FKGE by designing the private active gradient selection strategy.

An adaptive privacy budget allocation policy is further incorporated to dynamically adjust defense magnitude against the unbalanced privacy risks throughout the training procedure.

The experiment results demonstrate that the proposed defense can effectively defend against inference attacks with a modest utility decrease.

This implementation of exponential mechanism satisfies ( , 8 2 ) − .

We adopt a PTR variant with Gaussian noise #OTHEREFR for the second stage.

Since there is a probability bounded by that the test in PTR may fail, the algorithm satisfies -approximate-( , 2 2 )-RDP.

Thus, the private selection without subsampling satisfiesapproximate-( , ( 8 2 + 2 2 ))-RDP by composition.

Since the input of Algorithm 2 is computed from a subsampled batch of the original dataset(Line 11 in Algorithm 1), the privacy guarantee can be amplified through the privacy amplification theorem for RDP with subsampled mechanism #OTHEREFR . □

Quantifying And Defending Against Privacy Threats On Federated Knowledge Graph Embedding

Abstract: Knowledge Graph Embedding (KGE) is a fundamental technique that extracts expressive representation from knowledge graph (KG) to facilitate diverse downstream tasks. The emerging federated KGE (FKGE) collaboratively trains from distributed KGs held among clients while avoiding exchanging clients' sensitive raw KGs, which can still suffer from privacy threats as evidenced in other federated model trainings (e.g., neural networks). However, quantifying and defending against such privacy threats remain unexplored for FKGE which possesses unique properties not shared by previously studied models. In this paper, we conduct the first holistic study of the privacy threat on FKGE from both attack and defense perspectives. For the attack, we quantify the privacy threat by proposing three new inference attacks, which reveal substantial privacy risk by successfully inferring the existence of the KG triple from victim clients. For the defense, we propose DP-Flames, a novel differentially private FKGE with private selection, which offers a better privacy-utility tradeoff by exploiting the entity-binding sparse gradient property of FKGE and comes with a tight privacy accountant by incorporating the state-of-the-art private selection technique. We further propose an adaptive privacy budget allocation policy to dynamically adjust defense magnitude across the training procedure. Comprehensive evaluations demonstrate that the proposed defense can successfully mitigate the privacy threat by effectively reducing the success rate of inference attacks from 83.1% to 59.4% on average with only a modest utility decrease.

Practical Differentially Private Top-$K$ Selection With Pay-What-You-Get Composition

Abstract: We study the problem of top-k selection over a large domain universe subject to user-level differential privacy. Typically, the exponential mechanism or report noisy max are the algorithms used to solve this problem. However, these algorithms require querying the database for the count of each domain element. We focus on the setting where the data domain is unknown, which is different than the setting of frequent itemsets where an apriori type algorithm can help prune the space of domain elements to query. We design algorithms that ensures (approximate) differential privacy and only needs access to the true top-k' elements from the data for any chosen k' ≥ k. This is a highly desirable feature for making differential privacy practical, since the algorithms require no knowledge of the domain. We consider both the setting where a user's data can modify an arbitrary number of counts by at most 1, i.e. unrestricted sensitivity, and the setting where a user's data can modify at most some small, fixed number of counts by at most 1, i.e. restricted sensitivity. Additionally, we provide a pay-what-you-get privacy composition bound for our algorithms. That is, our algorithms might return fewer than k elements when the top-k elements are queried, but the overall privacy budget only decreases by the size of the outcome set.


The GAN training usually ends up with (partial) mode collapse #REFR , where generative models suffer lack of diversity.

In Section 3.3, VNE + has successfully prevented representation collapse.

As another example for collapse prevention, we consider the mode collapse in GAN.

To demonstrate that this problem can be solved by VNE + , we reproduce various GAN methods based on an open source code base, StudioGAN #OTHEREFR and train all models with CIFAR-10 for 100 epochs.

To evaluate the models, we report the Inception Score #OTHEREFR (IS, higher is better) and the Fréchet Inception Distance #OTHEREFR (FID, lower is better).

Although both IS and FID are the most popular metrics for evaluating generative models, FID is known to favor more diversified images #OTHEREFR .

Table 8 demonstrate that the overall quality of the output, especially diversity, has been improved by VNE + because FID scores have been improved. IS has also been improved.

Vne: An Effective Method For Improving Deep Representation By Manipulating Eigenvalue Distribution

Abstract: Since the introduction of deep learning, a wide scope of representation properties, such as decorrelation, whitening, disentanglement, rank, isotropy, and mutual information, have been studied to improve the quality of representation. However, manipulating such properties can be challenging in terms of implementational effectiveness and general applicability. To address these limitations, we propose to regularize von Neumann entropy (VNE) of representation. First, we demonstrate that the mathematical formulation of VNE is superior in effectively manipulating the eigenvalues of the representation autocorrelation matrix. Then, we demonstrate that it is widely applicable in improving stateof-the-art algorithms or popular benchmark algorithms by investigating domain-generalization, meta-learning, selfsupervised learning, and generative models. In addition, we formally establish theoretical connections with rank, disentanglement, and isotropy of representation. Finally, we provide discussions on the dimension control of VNE and the relationship with Shannon entropy. Code is available at: https://github.com/jaeill/CVPR23-VNE. * Corresponding author (a) Domain generalization (b) Meta-learning (c) Self-supervised learning (d) GAN

Nips 2016 Tutorial: Generative Adversarial Networks

Abstract: This report summarizes the tutorial presented by the author at NIPS 2016 on generative adversarial networks (GANs). The tutorial describes: (1) Why generative modeling is a topic worth studying, (2) how generative models work, and how GANs compare to other generative models, (3) the details of how GANs work, (4) research frontiers in GANs, and (5) state-of-the-art image models that combine GANs with other methods. Finally, the tutorial contains three exercises for readers to complete, and the solutions to these exercises.


PoseNet #REFR [15] trains a CNN to regress the 6-DOF camera pose from a single RGB image without additional engineering or graph optimization.

And #OTHEREFR extends the database volumes by generating the rendered synthetic images as the database.

Active Search #OTHEREFR implements the image retrieval process followed by the direct 2D-3D matching, so it covers both advantages.

On these bases, the applications of some excellent feature extraction #OTHEREFR , feature matching #OTHEREFR and image retrieval #OTHEREFR [3] methods can further improve the camera localization performances individually.

Metrics regression localization methods aim to regress the camera pose directly from train images with the ground truth poses.

And it has been extended to video mode using LSTM to extract temporal information #OTHEREFR .

Later on, #OTHEREFR uses a Bayesian CNN implementation to obtain an estimate of the localization uncertainty and improves the accuracy on the large-scale outdoor datasets.

AtLoc #OTHEREFR shows that the attention block can be used to force the network to focus on more geometrically robust objects and features, which can learn to reject dynamic objects and illumination conditions to achieve better performance.

MapNet #OTHEREFR exploits other sensory inputs like visual odometry and GPS in addition to images, and fuses them together for camera localization.

End-to-End Scene Coordinates Prediction.

Sgl: Structure Guidance Learning For Camera Localization

Abstract: Camera localization is a classical computer vision task that serves various Artificial Intelligence and Robotics applications. With the rapid developments of Deep Neural Networks (DNNs), end-to-end visual localization methods are prosperous in recent years. In this work, we focus on the scene coordinate prediction ones and propose a network architecture named as Structure Guidance Learning (SGL) which utilizes the receptive branch and the structure branch to extract both high-level and low-level features to estimate the 3D coordinates. We design a confidence strategy to refine and filter the predicted 3D observations, which enables us to estimate the camera poses by employing the Perspective-n-Point (PnP) with RANSAC. In the training part, we design the Bundle Adjustment trainer to help the network fit the scenes better. Comparisons with some state-of-the-art (SOTA) methods and sufficient ablation experiments confirm the validity of our proposed architecture.

Geometric Loss Functions For Camera Pose Regression With Deep Learning

Abstract: Deep learning has shown to be effective for robust and real-time monocular image relocalisation. In particular, PoseNet [22] is a deep convolutional neural network which learns to regress the 6-DOF camera pose from a single image. It learns to localize using high level features and is robust to difficult lighting, motion blur and unknown camera intrinsics, where point based SIFT registration fails. However, it was trained using a naive loss function, with hyper-parameters which require expensive tuning. In this paper, we give the problem a more fundamental theoretical treatment. We explore a number of novel loss functions for learning camera pose which are based on geometry and scene reprojection error. Additionally we show how to automatically learn an optimal weighting to simultaneously regress position and orientation. By leveraging geometry, we demonstrate that our technique significantly improves PoseNets performance across datasets ranging from indoor rooms to a small city.


Observably, on the COCO-OOD dataset, the U-AP of UnSniffer outperforms the 2 nd result by more than twice, and our U-F1 is 16.2% higher than the 2 nd result, at the cost of a 1.9% drop in mAP on VOC compared to Faster-RCNN #REFR .

In Table 2 , we show UnSniffer's result on the UOD-Benchmark, along with the results of MSP #OTHEREFR , Mahalanobis #OTHEREFR , Energy score #OTHEREFR , ORE #OTHEREFR , OW-DETR #OTHEREFR and VOS #OTHEREFR .

Note that OW-DETR is based on Deformable DETR #OTHEREFR with a stronger discriminative power, while other methods use Faster-RCNN.

Since U-PRE or U-REC cannot independently reflect the model's performance, we mainly employ U-AP, U-F1, AOSE and WI.

On the COCO-Mix dataset, the UnSniffer still holds the lead in both U-AP and U-F1, which are 1% and 11.2% higher than the 2 nd results, respectively.

Those comparisons demonstrate that UnSniffer outperforms the existing methods in unknown object detection, which owes to our GOC learning the overall confidence of objects from finite known objects.

Furthermore, UnSniffer has the smallest AOSE (398) but the largest WI (0.175), which can be explained by the inverse relationship between WI and the count of known objects misclassified as an incorrect class. More details are illustrated in the supplementary material. Qualitative Analysis. Fig.

7 visualizes the results of different methods on example images of the COCO-Mix (first two rows) and COCO-OOD dataset (last three rows).

It can be seen that VOS #OTHEREFR , MSP #OTHEREFR , Mahalanobis distance #OTHEREFR , and Energy score #OTHEREFR miss many objects of the unknown class, such as the surfboards in the 1 st image, the keyboard and water cup in the 3 rd image, the CD case in the 4 th image.

Unknown Sniffer For Object Detection: Don'T Turn A Blind Eye To Unknown Objects

Abstract: The recently proposed open-world object and open-set detection achieve a breakthrough in finding never-seenbefore objects and distinguishing them from class-known ones. However, their studies on knowledge transfer from known classes to unknown ones need to be deeper, leading to the scanty capability for detecting unknowns hidden in the background. In this paper, we propose the unknown sniffer (UnSniffer) to find both unknown and known objects. Firstly, the generalized object confidence (GOC) score is introduced, which only uses class-known samples for supervision and avoids improper suppression of unknowns in the background. Significantly, such confidence score learned from class-known objects can be generalized to unknown ones. Additionally, we propose a negative energy suppression loss to further limit the non-object samples in the background. Next, the best box of each unknown is hard to obtain during inference due to lacking their semantic information in training. To solve this issue, we introduce a graphbased determination scheme to replace hand-designed nonmaximum suppression (NMS) post-processing. Finally, we present the Unknown Object Detection Benchmark, the first publicly benchmark that encompasses precision evaluation for unknown object detection to our knowledge. Experiments show that our method is far better than the existing state-of-the-art methods.

Faster R-Cnn: Towards Real-Time Object Detection With Region Proposal Networks

Abstract: State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with ’attention’ mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps ( including all steps ) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available.


To further reduce computational complexity, we can divide the tensors into smaller blocks and treat them as individual tensors instead #REFR .

L l=1 d l m=1 (n l ) 3 m ) and the memory cost is C( L l=1 d l m=1 (n l ) 2 m )

These costs are strictly larger than those of the diagonal metrics or the Monge metric, but in practice the metric is updated only periodically (typically after each epoch), and hence the computation remains manageable.

The following theorem shows the validity of the sampler, with the proof in the Supplement.

The bound depends on the EMA parameter λ in the same way as the bound for the Monge metric and otherwise follows that of Theorem 2.1.

For SGRLD in the Shampoo metric, we can bound the approximation error as defined in Theo-

rem 2.1 as E φ −φ 2 ≤ C t h 2 t S 2 T E ∆V t 2 + 1 S T + ( t=1 h 2 t ) 2 S 2 T + O(τ 2 (1 − λ) 2 ).

Scalable Stochastic Gradient Riemannian Langevin Dynamics In Non-Diagonal Metrics

Abstract: Bayesian neural network inference is often carried out using stochastic gradient sampling methods. For best performance the methods should use a Riemannian metric that improves posterior exploration by accounting for the local curvature, but the existing methods resort to simple diagonal metrics to remain computationally efficient. This loses some of the gains. We propose two non-diagonal metrics that can be used in stochastic samplers to improve convergence and exploration but that have only a minor computational overhead over diagonal metrics. We show that for neural networks with complex posteriors, caused e.g. by use of sparsity-inducing priors, using these metrics provides clear improvements. For some other choices the posterior is sufficiently easy also for the simpler metrics.

Second Order Optimization Made Practical

Abstract: Optimization in machine learning, both theoretical and applied, is presently dominated by first-order gradient methods such as stochastic gradient descent. Second-order optimization methods that involve second-order derivatives and/or second-order statistics of the data have become far less prevalent despite strong theoretical properties, due to their prohibitive computation, memory and communication costs. ::: In an attempt to bridge this gap between theoretical and practical optimization, we present a proof-of-concept distributed system implementation of a second-order preconditioned method (specifically, a variant of full-matrix Adagrad), that along with a few yet critical algorithmic and numerical improvements, provides significant practical gains in convergence on state-of-the-art deep models and gives rise to actual wall-time improvements in practice compared to conventional first-order methods. Our design effectively utilizes the prevalent heterogeneous hardware architecture for training deep models which consists of a multicore CPU coupled with multiple accelerator units. We demonstrate superior performance on very large learning problems in machine translation where our distributed implementation runs considerably faster than existing gradient-based methods.


Meanwhile, for each UE, we consider the case that only one step of stochastic gradient descent (SGD) is performed, following the same setting as #REFR .

We consider the above described MAML-based FL.

In detail, we concentrate on the situation where UEs communicate in a synchronous manner, so as to avoid using outdated parameters for global model update and make high-quality refinement in each round.

As for the concerned MAML-based FL, our goal is to optimize the initial model using only a few data points at each UE.

Hence, we only obtain an estimate of the desired gradient with SGD.

Here, the desired gradient ∇F i (w) on UE i is computed using all data points in its dataset D i , while the estimated gradient∇F i (w) on UE i is computed using SGD with the sampled dataset D

Automated Federated Learning In Mobile Edge Networks -Fast Adaptation And Convergence

Abstract: Federated Learning (FL) can be used in mobile edge networks to train machine learning models in a distributed manner. Recently, FL has been interpreted within a Model-Agnostic Meta-Learning (MAML) framework, which brings FL significant advantages in fast adaptation and convergence over heterogeneous datasets. However, existing research simply combines MAML and FL without explicitly addressing how much benefit MAML brings to FL and how to maximize such benefit over mobile edge networks. In this paper, we quantify the benefit from two aspects: optimizing FL hyperparameters (i.e., sampled data size and the number of communication rounds) and resource allocation (i.e., transmit power) in mobile edge networks. Specifically, we formulate the MAML-based FL design as an overall learning time minimization problem, under the constraints of model accuracy and energy consumption. Facilitated by the convergence analysis of MAML-based FL, we decompose the formulated problem and then solve it using analytical solutions and the coordinate descent method. With the obtained FL hyperparameters and resource allocation, we design a MAML-based FL algorithm, called Automated Federated Learning (AutoFL), that is able to conduct fast adaptation and convergence. Extensive experimental results verify that AutoFL outperforms other benchmark algorithms regarding the learning time and convergence performance.

Model-Agnostic Meta-Learning For Fast Adaptation Of Deep Networks

Abstract: We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning problems, including classification, regression, and reinforcement learning. The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. In our approach, the parameters of the model are explicitly trained such that a small number of gradient steps with a small amount of training data from a new task will produce good generalization performance on that task. In effect, our method trains the model to be easy to fine-tune. We demonstrate that this approach leads to state-of-the-art performance on two few-shot image classification benchmarks, produces good results on few-shot regression, and accelerates fine-tuning for policy gradient reinforcement learning with neural network policies.


In contrast to previous works that primarily addressed misinformation detection as a binary classification problem ( #REFR ), we address CMM as a multiclass classification task.

To investigate the prevalence of AMM, we sampled 200 misleading pairs from the COSMOS benchmark and examined their source articles on Snopes.

Following the classification taxonomy of Snopes #OTHEREFR we found that 48% of COSMOS pairs are "false claims" (41% associative imagery and 7% reinforcing captions) while 52% were classified as "miscaptioned", which we consider to be CMM because it implies a relationship between the two modalities.

After de-duplicating the images of the COSMOS benchmark, the rates were 41% miscaptioned, 35% associative imagery, 4% reinforcing captions and 20% duplicates.

We performed the same process on Fakeddit for 300 random samples and found that roughly 45% of pairs were AMM, with 41% being manipulated images and 4% with associative imagery.

Moreover, we consider that roughly 14% of Fakeddit's samples can actually be considered CMM since the remaining 40% were mostly funny memes, visual jokes, pareidolia imagery and other content that is not generally considered to be misinformation #OTHEREFR .

In this work, we introduce a taxonomy that includes three classes:

Truthful (True): an image-caption pair (I t , C t ) is considered True when the origin, content, and context of an image are accurately described in the accompanying caption.

Out-Of-Context (OOC) image-text pairs: involves a deceptive combination of a truthful caption C t and an image that is out of context I x .

Figments And Misalignments: A Framework For Fine-Grained Crossmodal Misinformation Detection

Abstract: Multimedia content has become ubiquitous on social media platforms, leading to the rise of multimodal misinformation and the urgent need for effective strategies to detect and prevent its spread. This study focuses on CrossModal Misinformation (CMM) where image-caption pairs work together to spread falsehoods. We contrast CMM with Asymmetric Multimodal Misinformation (AMM), where one dominant modality propagates falsehoods while other modalities have little or no influence. We show that AMM adds noise to the training and evaluation process while exacerbating the unimodal bias, where text-only or image-only detectors can seemingly outperform their multimodal counterparts on an inherently multimodal task. To address this issue, we collect and curate FIGMENTS, a robust evaluation benchmark for CMM, which consists of real-world cases of misinformation, excludes AMM and utilizes modality balancing to successfully alleviate unimodal bias. FIGMENTS also provides a first step towards fine-grained CMM detection by including three classes: truthful, out-ofcontext, and miscaptioned image-caption pairs. Furthermore, we introduce a method for generating realistic synthetic training data that maintains crossmodal relations between legitimate images and false human-written captions that we term Crossmodal HArd Synthetic MisAlignment (CHASMA). We conduct extensive comparative study using a Transformer-based architecture. Our results show that incorporating CHASMA in conjunction with other generated datasets consistently improved the overall performance on FIGMENTS in both binary (+6.26%) and multiclass settings (+15.

Multimodal Analytics For Real-World News Using Measures Of Cross-Modal Entity Consistency

Abstract: The World Wide Web has become a popular source for gathering information and news. Multimodal information, e.g., enriching text with photos, is typically used to convey the news more effectively or to attract attention. Photo content can range from decorative, depict additional important information, or can even contain misleading information. Therefore, automatic approaches to quantify cross-modal consistency of entity representation can support human assessors to evaluate the overall multimodal message, for instance, with regard to bias or sentiment. In some cases such measures could give hints to detect fake news, which is an increasingly important topic in today's society. In this paper, we introduce a novel task of cross-modal consistency verification in real-world news and present a multimodal approach to quantify the entity coherence between image and text. Named entity linking is applied to extract persons, locations, and events from news texts. Several measures are suggested to calculate cross-modal similarity for these entities using state of the art approaches. In contrast to previous work, our system automatically gathers example data from the Web and is applicable to real-world news. Results on two novel datasets that cover different languages, topics, and domains demonstrate the feasibility of our approach. Datasets and code are publicly available to foster research towards this new direction.


First, we generated face annotation for RGB frames using FaceAlignment #REFR , an open-source tool for face analysis 2 .

Using the synthetic data from the simulator, we generated an annotated dataset in the event spectrum to train a face detector.

We then bound the face labels with the corresponding synthetic event frames obtained with ESIM.

This allowed us to train a YOLOv2 #OTHEREFR on the synthetic version of NEFER.

We found the detector to have good generalization capabilities from synthetic to real event data, which yielded high-quality annotations at a slight cost of manual validation using CVAT #OTHEREFR .

Neuromorphic Event-Based Facial Expression Recognition

Abstract: Recently, event cameras have shown large applicability in several computer vision fields especially concerning tasks that require high temporal resolution. In this work, we investigate the usage of such kind of data for emotion recognition by presenting NEFER, a dataset for Neuromorphic Event-based Facial Expression Recognition. NEFER is composed of paired RGB and event videos representing human faces labeled with the respective emotions and also annotated with face bounding boxes and facial landmarks. We detail the data acquisition process as well as providing a baseline method for RGB and event data. The collected data captures subtle micro-expressions, which are hard to spot with RGB data, yet emerge in the event domain. We report a double recognition accuracy for the event-based approach, proving the effectiveness of a neuromorphic approach for analyzing fast and hardly detectable expressions and the emotions they conceal.

How Far Are We From Solving The 2D&3D Face Alignment Problem? (And A Dataset Of 230,000 3D Facial Landmarks)

Abstract: This paper investigates how far a very deep neural network is from attaining close to saturating performance on existing 2D and 3D face alignment datasets. To this end, we make the following 5 contributions: (a) we construct, for the first time, a very strong baseline by combining a state-of-the-art architecture for landmark localization with a state-of-the-art residual block, train it on a very large yet synthetically expanded 2D facial landmark dataset and finally evaluate it on all other 2D facial landmark datasets. (b) We create a guided by 2D landmarks network which converts 2D landmark annotations to 3D and unifies all existing datasets, leading to the creation of LS3D-W, the largest and most challenging 3D facial landmark dataset to date ~230,000 images. (c) Following that, we train a neural network for 3D face alignment and evaluate it on the newly introduced LS3D-W. (d) We further look into the effect of all"traditional"factors affecting face alignment performance like large pose, initialization and resolution, and introduce a"new"one, namely the size of the network. (e) We show that both 2D and 3D face alignment networks achieve performance of remarkable accuracy which is probably close to saturating the datasets used. Training and testing code as well as the dataset can be downloaded from https://www.adrianbulat.com/face-alignment/


The visual front-end module is comprised of a 3D convolutional layer with a kernel size of 5 × 7 × 7 followed by a ResNet18 #REFR .

We adopt the visual front-end and the audio front-end from #OTHEREFR .

Then the output features are squeezed along the spatial dimension by a global average pooling layer.

The audio front-end module consists of a 1D convolutional layer with blocks of ResNet18.

Both visual and audio front-ends are initialized using a pre-trained model on LRW #OTHEREFR .

For the multimodal attention with Conformer encoder #OTHEREFR , we use hidden dimensions of 256, feed-forward dimensions of 2048, 12 layers, 8 attention heads, and a convolution kernel size of 31. We utilize Transformer decoder #OTHEREFR

Watch Or Listen: Robust Audio-Visual Speech Recognition With Visual Corruption Modeling And Reliability Scoring

Abstract: This paper deals with AudioVisual Speech Recognition (AVSR) under multimodal input corruption situations where audio inputs and visual inputs are both corrupted, which is not well addressed in previous research directions. Previous studies have focused on how to complement the corrupted audio inputs with the clean visual inputs with the assumption of the availability of clean visual inputs. However, in real life, clean visual inputs are not always accessible and can even be corrupted by occluded lip regions or noises. Thus, we firstly analyze that the previous AVSR models are not indeed robust to the corruption of multimodal input streams, the audio and the visual inputs, compared to uni-modal models. Then, we design multimodal input corruption modeling to develop robust AVSR models. Lastly, we propose a novel AVSR framework, namely AudioVisual Reliability Scoring module (AV-RelScore), that is robust to the corrupted multimodal inputs. The AV-RelScore can determine which input modal stream is reliable or not for the prediction and also can exploit the more reliable streams in prediction. The effectiveness of the proposed method is evaluated with comprehensive experiments on popular benchmark databases, LRS2 and LRS3. We also show that the reliability scores obtained by AV-RelScore well reflect the degree of corruption and make the proposed model focus on the reliable multimodal representations.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


It can generalize to unseen generation models to some extent. Marra et al. #REFR and Yu et al.

However, their strong generalization capability relies on their large-scale training and 20 different models each trained on a different LSUN #OTHEREFR object category.

Besides detection by spatial artifacts, there are also frequency-based methods #OTHEREFR . Frank et al.

#OTHEREFR present that in the frequency domain, GAN-generated images are more likely to expose severe artifacts mainly caused by upsampling operations in previous GAN architectures. Zhang et al.

[51] propose a GAN simulator, AutoGAN, to simulate the artifacts produced by standard GAN pipelines.

Then they train a detector on the spectrum input on the synthesized images.

#OTHEREFR suggest detecting generated images by fingerprints that are often produced during GAN generation.

A recent work #OTHEREFR proposes a detector based on an ensemble of EfficientNet-B4 #OTHEREFR to alleviate the generalization problem.

However, with the boosting development of diffusion models, a general and robust detector for detecting images generated by diffusion models has not been explored.

We note that some recent works also notice the diffusiongenerated image detection problem #OTHEREFR .

Different from them, the focus of our work is exploring a generalizable detector for wide-range diffusion models.

Dire For Diffusion-Generated Image Detection

Abstract: Diffusion models have shown remarkable success in visual synthesis, but have also raised concerns about potential abuse for malicious purposes. In this paper, we seek to build a detector for telling apart real images from diffusiongenerated images. We find that existing detectors struggle to detect images generated by diffusion models, even if we include generated images from a specific diffusion model in their training data. To address this issue, we propose a novel image representation called DIffusion Reconstruction Error (DIRE), which measures the error between an input image and its reconstruction counterpart by a pre-trained diffusion model. We observe that diffusion-generated images can be approximately reconstructed by a diffusion model while real images cannot. It provides a hint that DIRE can serve as a bridge to distinguish generated and real images. DIRE provides an effective way to detect images generated by most diffusion models, and it is general for detecting generated images from unseen diffusion models and robust to various perturbations. Furthermore, we establish a comprehensive diffusion-generated benchmark including images generated by eight diffusion models to evaluate the performance of diffusion-generated image detectors. Extensive experiments on our collected benchmark demonstrate that DIRE exhibits superiority over previous generatedimage detectors. The code and dataset are available at https://github.com/ZhendongWang6/DIRE.

Do Gans Leave Artificial Fingerprints?

Abstract: In the last few years, generative adversarial networks (GAN) have shown tremendous potential for a number of applications in computer vision and related fields. With the current pace of progress, it is a sure bet they will soon be able to generate high-quality images and videos, virtually indistinguishable from real ones. Unfortunately, realistic GAN-generated images pose serious threats to security, to begin with a possible flood of fake multimedia, and multimedia forensic countermeasures are in urgent need. In this work, we show that each GAN leaves its specific fingerprint in the images it generates, just like real-world cameras mark acquired images with traces of their photo-response non-uniformity pattern. Source identification experiments with several popular GANs show such fingerprints to represent a precious asset for forensic analyses.


On the global level, TCAV #REFR constructs a set of explanatory concepts represented by vectors that are able to separate positive/negative examples in a hidden layer.

In general, research efforts devoted to visual image interpretation can be divided into two groups, post-hoc and self-interpretable.

Post-hoc methods #OTHEREFR open the black-box models with salient maps or key image parts after the deep network is finished training.

Most of the interpretative models #OTHEREFR we discussed in the previous paragraph fall into this group, locally explaining a model's prediction for each individual sample.

Extending these gradient-based methods, some works #OTHEREFR include various additional information to achieve more reasonable explanations.

On the other hand, self-interpretable #OTHEREFR methods generate explainable representations in an end-to-end training process.

Recently, ProtoPNet #OTHEREFR proposed a self-interpretable global interpretation method that extracts category-specific prototype vectors associated with image patches of training samples, which serve as an example-based explanation on both global and local levels.

Later, various methods #OTHEREFR extend ProtoPNet from different aspects, including refined prototype learning module #OTHEREFR and pruning strategies #OTHEREFR to reduce the number of prototypes.

To align the heterogeneous distributions in source and target domains, previous works focusing on domain adaptation can be roughly separated into two straits.

Visualizing Transferred Knowledge: An Interpretive Model Of Unsupervised Domain Adaptation

Abstract: Many research efforts have been committed to unsupervised domain adaptation (DA) problems that transfer knowledge learned from a labeled source domain to an unlabeled target domain. Various DA methods have achieved remarkable results recently in terms of predicting ability, which implies the effectiveness of the aforementioned knowledge transferring. However, state-of-the-art methods rarely probe deeper into the transferred mechanism, leaving the true essence of such knowledge obscure. Recognizing its importance in the adaptation process, we propose an interpretive model of unsupervised domain adaptation, as the first attempt to visually unveil the mystery of transferred knowledge. Adapting the existing concept of the prototype from visual image interpretation to the DA task, our model similarly extracts shared information from the domain-invariant representations as prototype vectors. Furthermore, we extend the current prototype method with our novel prediction calibration and knowledge fidelity preservation modules, to orientate the learned prototypes to the actual transferred knowledge. By visualizing these prototypes, our method not only provides an intuitive explanation for the base model's predictions but also unveils transfer knowledge by matching the image patches with the same semantics across both source and target domains. Comprehensive experiments and in-depth explorations demonstrate the efficacy of our method in understanding the transferred mechanism and its potential in downstream tasks including model diagnosis.

Interpretability Beyond Feature Attribution: Quantitative Testing With Concept Activation Vectors (Tcav)

Abstract: The interpretation of deep learning models is a challenge due to their size, complexity, and often opaque internal state. In addition, many systems, such as image classifiers, operate on low-level features rather than high-level concepts. To address these challenges, we introduce Concept Activation Vectors (CAVs), which provide an interpretation of a neural net's internal state in terms of human-friendly concepts. The key idea is to view the high-dimensional internal state of a neural net as an aid, not an obstacle. We show how to use CAVs as part of a technique, Testing with CAVs (TCAV), that uses directional derivatives to quantify the degree to which a user-defined concept is important to a classification result--for example, how sensitive a prediction of "zebra" is to the presence of stripes. Using the domain of image classification as a testing ground, we describe how CAVs may be used to explore hypotheses and generate insights for a standard image classification network as well as a medical application.


A model trained with NLL provides predictions that deviate from the accuracy, leaving the model poorly calibrated #REFR .

To improve post-hoc calibration under out-domain scenarios, #OTHEREFR transforms the validation set prior to performing the post-hoc approach.

In #OTHEREFR , a regression model is used to predict temperature parameter.

Post-hoc calibration methods are simple and effective, however, they require hold-out validation data, and are dependent on architecture #OTHEREFR .

Train-time calibration methods: Models trained with zero-entropy supervision tend to give over-confident predictions.

An example is negative log-likelihood (NLL), which is a widely-used task-specific loss.

Train-time calibration methods are typically based on auxiliary loss functions, which are used in-tandem with task-specific losses.

In #OTHEREFR , an auxiliary loss term DCA is proposed to calibrate the model.

It is combined with a task-specific loss to penalize when it reduces but the accuracy remains unchanged.

Likewise, #OTHEREFR proposed an auxiliary loss function that is based on a reproducing kernel in a Hilbert space #OTHEREFR .

#OTHEREFR calibrated uncertainty based on the relationship between accuracy and uncertainty.

Bridging Precision And Confidence: A Train-Time Loss For Calibrating Object Detection

Abstract: Deep neural networks (DNNs) have enabled astounding progress in several vision-based problems. Despite showing high predictive accuracy, recently, several works have revealed that they tend to provide overconfident predictions and thus are poorly calibrated. The majority of the works addressing the miscalibration of DNNs fall under the scope of classification and consider only in-domain predictions. However, there is little to no progress in studying the calibration of DNN-based object detection models, which are central to many vision-based safety-critical applications. In this paper, inspired by the train-time calibration methods, we propose a novel auxiliary loss formulation that explicitly aims to align the class confidence of bounding boxes with the accurateness of predictions (i.e. precision). Since the original formulation of our loss depends on the counts of true positives and false positives in a minibatch, we develop a differentiable proxy of our loss that can be used during training with other application-specific loss functions. We perform extensive experiments on challenging in-domain and out-domain scenarios with six benchmark datasets including MS-COCO, Cityscapes, Sim10k, and BDD100k. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios. Our source code and pre-trained models are available at https:// github.com/akhtarvision/bpc_calibration

On Calibration Of Modern Neural Networks

Abstract: Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.


We use the binary cross-entropy loss for classification and focal loss #REFR to handle class imbalance in the localization task.

To identify missing signs, the model must attend to the traffic sign cues in the environment. Similar to Han et al.

#OTHEREFR , we add CueCAn at the end of the third, fourth, and fifth blocks of the VGG-19 #OTHEREFR encoder to highlight and classify the cues.

The next task is to localize where the sign could be placed using the segmentation model with the pre-trained VGG-19 encoder and FCN-8 #OTHEREFR decoder.

For localization, optimal results and GradCAM visualizations are observed when the entire network is fine-tuned end-to-end.

Cuecan: Cue-Driven Contextual Attention For Identifying Missing Traffic Signs On Unconstrained Roads

Abstract: Missing Traffic Sign Scene with Curve Cue Fig. 1: Left: Scenes with real and inpainted traffic signs (chevron-left). Middle: Intermediary GradCAM visualizations of the cue classifier (encoder) with and without CueCAn. Right: Segmentation model with CueCAn-based encoder detects missing signs (green masks overlayed over the scene on the right for CueCAn and yellow mask by the baseline) on the scene without signs (follow pink arrows) by effectively attending to the context cues, compared to weak attention without CueCAn. Segementation GradCAMs are obtained from the centroid of the predicted sign (red dot).

Focal Loss For Dense Object Detection

Abstract: The highest accuracy object detectors to date are based on a two-stage approach popularized by R-CNN, where a classifier is applied to a sparse set of candidate object locations. In contrast, one-stage detectors that are applied over a regular, dense sampling of possible object locations have the potential to be faster and simpler, but have trailed the accuracy of two-stage detectors thus far. In this paper, we investigate why this is the case. We discover that the extreme foreground-background class imbalance encountered during training of dense detectors is the central cause. We propose to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples. Our novel Focal Loss focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training. To evaluate the effectiveness of our loss, we design and train a simple dense detector we call RetinaNet. Our results show that when trained with the focal loss, RetinaNet is able to match the speed of previous one-stage detectors while surpassing the accuracy of all existing state-of-the-art two-stage detectors. Code is at: https://github.com/facebookresearch/Detectron .


Currently, the best performing DL based detector is Faster Region−based Convolutional Neural Network (RCNN) from #REFR .

The survey specifies that most MOT algorithms that are developed to be used with a single camera have four steps/stages in common: detection, feature extraction / motion prediction, affinity, and association.

The implied aim is to implement DL at every stage and to evaluate the given algorithms as a whole on a MOTChallenge dataset #OTHEREFR . The datasets mostly consist of benchmarks for pedestrian tracking.

Deep learning is mostly used for the first two stages, while only a few contributions implement DL approaches for affinity and association.

From this survey #OTHEREFR , the authors emphasize three important parameters to deploy MOT algorithms: (i) the detection quality, (ii) Convolutional Neural Network (CNN) for feature extraction, and (iii) Single Object Tracking (SOT) trackers.

In terms of detection quality, appropriate detectors must be thoroughly selected to reduce the number of False Negatives (FN) in the Multi−Object Tracking Accuracy (MOTA) score.

In contrast, Single−Shot Detector (SSD) performs worse, as presented in #OTHEREFR .

However, SSD was almost able to work in real-time (4.5 FPS), including the detection step.

For the feature extraction stage #OTHEREFR , the best-performing method, GoogLeNet #OTHEREFR , is applied to the datasets of MOT15 #OTHEREFR , MOT16 and MOT17 #OTHEREFR .

Approaches that do not use appearance (whether they are deep or conventional methods) typically perform worse.

Visual features alone, however, are insufficient to compute affinity; many of the better-performing algorithms additionally include other characteristics, particularly motion features.

Semi-Automated Computer Vision Based Tracking Of Multiple Industrial Entities -A Framework And Dataset Creation Approach

Abstract: This contribution presents the TOMIE framework (Tracking Of Multiple Industrial Entities), a framework for the continuous tracking of industrial entities (e.g., pallets, crates, barrels) over a network of, in this example, six RGB cameras. This framework, makes use of multiple sensors, data pipelines and data annotation procedures, and is described in detail in this contribution. With the vision of a fully automated tracking system for industrial entities in mind, it enables researchers to efficiently capture high quality data in an industrial setting. Using this framework, an image dataset, the TOMIE dataset, is created, which at the same time is used to gauge the framework's validity. This dataset contains annotation files for 112,860 frames and 640,936 entity instances that are captured from a set of six cameras that perceive a large indoor space. This dataset out-scales comparable datasets by a factor of four and is made up of scenarios, drawn from industrial applications from the sector of warehousing. Three tracking algorithms, namely ByteTrack, Bot-Sort and SiamMOT are applied to this dataset, serving as a proof-of-concept and providing tracking results that are comparable to the state of the art.

Poi: Multiple Object Tracking With High Performance Detection And Appearance Feature

Abstract: Detection and learning based appearance feature play the central role in data association based multiple object tracking (MOT), but most recent MOT works usually ignore them and only focus on the hand-crafted feature and association algorithms. In this paper, we explore the high-performance detection and deep learning based appearance feature, and show that they lead to significantly better MOT results in both online and offline setting. We make our detection and appearance feature publicly available. In the following part, we first summarize the detection and appearance feature, and then introduce our tracker named Person of Interest (POI), which has both online and offline version.


Toward Models Of Human Collaboration As Lenses For Studying And Designing Co-Creative Systems

This result is in line with studies on human-AI collaboration in decision-making, suggesting that users learn to better predict the machine's behavior through inductive mechanisms (i.e., via concrete examples and hands-on testing) than via general, declarative information about internal processes #REFR .

Participants had trouble learning to predict how the AI might behave in response to the specified parameters.

They struggled to make sense of the AI system's reasoning and struggled to correct unwanted design issues.

Prior literature on group cognition suggests that to achieve effective collaboration group members should be able to interpret each other's reasoning and predict roughly how their partner might behave in response to their own actions #OTHEREFR .

Similarly, from a team learning perspective, our findings suggest that designers who systematically explored the AI's limitations and capabilities early on were better at predicting the tool's actions in response to their own and produced more satisfactory results.

While explainable AI research focuses primarily on directly communicating information about the AI system to the user, recent research has suggested that more engaging and longer forms of learning and deliberate practice might improve human-AI collaboration #OTHEREFR .

However, in addition to supporting honing the user's mental model of the AI's capabilities and limitations, it is equally important for the AI system to have an understanding of the user's capabilities, limitations, and task context to enable more effective human-AI collaboration.

Hence, this would require the AI system to have better contextual awareness of the user and the current task at hand.

We further discuss the resulting design opportunities in section 8.3.3.

Most designers felt the tools were uncollaborative and had more control over the design process than they would have preferred.

Exploring Challenges And Opportunities To Support Designers In Learning To Co-Create With Ai-Based Manufacturing Design Tools

Abstract: AI-based design tools are proliferating in professional software to assist engineering and industrial designers in complex manufacturing and design tasks. These tools take on more agentic roles than traditional computer-aided design tools and are often portrayed as "co-creators. " Yet, working effectively with such systems requires different skills than working with complex CAD tools alone. To date, we know little about how engineering designers learn to work with AI-based design tools. In this study, we observed trained designers as they learned to work with two AI-based tools on a realistic design task. We find that designers face many challenges in learning to effectively co-create with current systems, including challenges in understanding and adjusting AI outputs and in communicating their design goals. Based on our findings, we highlight several design opportunities to better support designer-AI co-creation. CCS CONCEPTS • Human-centered computing → Empirical studies in HCI; • Applied computing → Computer-aided design.

It Takes Two To Tango: Towards Theory Of Ai'S Mind

Abstract: Theory of Mind is the ability to attribute mental states (beliefs, intents, knowledge, perspectives, etc.) to others and recognize that these mental states may differ from one's own. Theory of Mind is critical to effective communication and to teams demonstrating higher collective performance. To effectively leverage the progress in Artificial Intelligence (AI) to make our lives more productive, it is important for humans and AI to work well together in a team. Traditionally, there has been much emphasis on research to make AI more accurate, and (to a lesser extent) on having it better understand human intentions, tendencies, beliefs, and contexts. The latter involves making AI more human-like and having it develop a theory of our minds. In this work, we argue that for human-AI teams to be effective, humans must also develop a theory of AI's mind (ToAIM) - get to know its strengths, weaknesses, beliefs, and quirks. We instantiate these ideas within the domain of Visual Question Answering (VQA). We find that using just a few examples (50), lay people can be trained to better predict responses and oncoming failures of a complex VQA model. We further evaluate the role existing explanation (or interpretability) modalities play in helping humans build ToAIM. Explainable AI has received considerable scientific and popular attention in recent times. Surprisingly, we find that having access to the model's internal states - its confidence in its top-k predictions, explicit or implicit attention maps which highlight regions in the image (and words in the question) the model is looking at (and listening to) while answering a question about an image - do not help people better predict its behavior.


Unlike Received Signal Strength Indicator (RSSI) based solutions #REFR , 802.11az combines mmWavebased angle estimations with the Time of Flight (ToF) to achieve high accuracy positioning.

Legacy positioning in 802.11 only knew the distance between devices, thus, there are many feasible locations (blue markers).

802.11az uses mmWave beamforming to obtain the azimuth and elevation between devices, hence, the exact location (red marker).

Although 802.11az provides enhancements for sub-6 GHz and Millimiter Wave (mmWave) operation at 60 GHz, the best positioning accuracy is achieved in the latter band.

Specifically, 802.11az obtains accurate positioning information using 802.11ay Enhanced Directional Multi-Gigabit (EDMG) beamforming #OTHEREFR , along with its accurate Channel State Information (CSI) reports.

With the orientation of the mmWave beam and CSI reports, 802.11az elaborates angle estimations funded on the ranging accuracy of multi-GHz bandwidths.

Both the ToF and angle estimations are exchanged using the Fine Timing Measurement (FTM) procedureintroduced in 802.11mc #OTHEREFR -, and Station (STA)s use the exchanged estimations to determine the distance, azimuth and elevation between STAssee Fig. 1 .

Since mmWave promises to increase positioning accuracy from meter level #OTHEREFR to centimeterlevel #OTHEREFR , #OTHEREFR ; in this article we explain the main contributions of 802.11az for mmWave based positioning:

In the legacy FTM procedure STAs only exchange ToF estima-arXiv:2303.05996v1 [cs.NI] 10 Mar 2023 tions, thus, obtaining just an approximation of the distance.

802.11az performs the FTM procedure over EDMG to also exchange angle estimations obtained through CSI reports.

Ieee 802.11Az Indoor Positioning With Mmwave

Abstract: Last years we have witnessed the uprising of location based applications, which depend on the devices ability to accurately obtain their position. IEEE 802.11, foretelling the need for such applications, started the IEEE 802.11az work on Next Generation Positioning. Although this standard provides positioning enhancements for sub-6 GHz and mmWave bands, high accuracy in the order of centimeters can only be obtained in the latter band, thanks to the beamforming information available at mmWave operation. This work presents a detailed analysis on the new techniques provided by IEEE 802.11az for enhanced secured positioning in the mmWave band, assessing them through experimentation.

A Survey Of Indoor Localization Systems And Technologies

Abstract: Indoor localization has recently witnessed an increase in interest, due to the potential wide range of services it can provide by leveraging Internet of Things (IoT), and ubiquitous connectivity. Different techniques, wireless technologies and mechanisms have been proposed in the literature to provide indoor localization services in order to improve the services provided to the users. However, there is a lack of an up-to-date survey paper that incorporates some of the recently proposed accurate and reliable localization systems. In this paper, we aim to provide a detailed survey of different indoor localization techniques, such as angle of arrival (AoA), time of flight (ToF), return time of flight (RTOF), and received signal strength (RSS); based on technologies, such as WiFi, radio frequency identification device (RFID), ultra wideband (UWB), Bluetooth, and systems that have been proposed in the literature. This paper primarily discusses localization and positioning of human users and their devices. We highlight the strengths of the existing systems proposed in the literature. In contrast with the existing surveys, we also evaluate different systems from the perspective of energy efficiency, availability, cost, reception range, latency, scalability, and tracking accuracy. Rather than comparing the technologies or techniques, we compare the localization systems and summarize their working principle. We also discuss remaining challenges to accurate indoor localization.

mmWavebased angle estimations
high accuracy positioning


Masuda and Holme #REFR used a graph distance measure and hierarchical clustering to detect and cluster evolving states in social temporal networks.

The authors illustrate the usefulness of their approach by applying it to synthetic datasets such as the Barabasi-Albert graphs and small world graphs.

(3) Change Point Detection in Social Networks.

In the field of change point detection in social networks, Wang et al.

#OTHEREFR proposed an algorithm based on a Markov generative process to analyze graph snapshots of dynamic social networks.

The algorithm was tested on real-world networks, including political voting networks, but it fails to account for long-term dependence and assumes rare changes.

Their approach assumes the entire network system is described by a single system state, which could be relaxed to multi-state setup for social networks with community structure. Zhao et al.

#OTHEREFR introduced a model-free change point detection method for dynamic social networks that uses neighborhood smoothing to estimate edge probabilities.

However, the algorithm is not applicable to directed networks or networks with an evolving number of nodes. Grattarola et al.

#OTHEREFR proposed a data-driven method for detecting changes in stationarity in a stream of attributed graphs.

They used an adversarial autoencoder to embed graphs on constant-curvature manifolds, and employed the Fréchet mean to represent the average of networks.

Fréchet Statistics Based Change Point Detection In Dynamic Social Networks

Abstract: This paper proposes a method to detect change points in dynamic social networks using Fréchet statistics. We address two main questions: (1) what metric can quantify the distances between graph Laplacians in a dynamic network and enable efficient computation, and (2) how can the Fréchet statistics be extended to detect multiple change points while maintaining the significance level of the hypothesis test? Our solution defines a metric space for graph Laplacians using the Log-Euclidean metric, enabling a closed-form formula for Fréchet mean and variance. We present a framework for change point detection using Fréchet statistics and extend it to multiple change points with binary segmentation. The proposed algorithm uses incremental computation for Fréchet mean and variance to improve efficiency and is validated on simulated and two real-world datasets, namely the UCI message dataset and the Enron email dataset.

Detecting Sequences Of System States In Temporal Networks

Abstract: Many time-evolving systems in nature, society and technology leave traces of the interactions within them. These interactions form temporal networks that reflect the states of the systems. In this work, we pursue a coarse-grained description of these systems by proposing a method to assign discrete states to the systems and inferring the sequence of such states from the data. Such states could, for example, correspond to a mental state (as inferred from neuroimaging data) or the operational state of an organization (as inferred by interpersonal communication). Our method combines a graph distance measure and hierarchical clustering. Using several empirical data sets of social temporal networks, we show that our method is capable of inferring the system's states such as distinct activities in a school and a weekday state as opposed to a weekend state. We expect the methods to be equally useful in other settings such as temporally varying protein interactions, ecological interspecific interactions, functional connectivity in the brain and adaptive social networks.


We use Firedrake #REFR to discretize (4.1) with linear finite elements on tetrahedra, and select an average of 25 points per cluster.

Indeed, this is reflected in the convergence shown in Figure 11 , where we observe a dramatic reduction in the number of iterations for solvers with these clusters.

It is, of course, important to note that not every clustering generated by standard Lloyd exhibits similarly poor performance.

The quality of the initial clustering used to seed the algorithm plays an important role in determining the multigrid performance.

Results seen here are for a representative, randomly generated, initial seeding. rotated about the z-axis, (see Table 1 ).

A 3D tetrahedral mesh with 16 921 elements is generated with Gmsh #OTHEREFR through pygmsh #OTHEREFR .

2D restricted channel: The 2D domain Ω is defined by [−2, 2] × [−1, 1] \ C with C = C + ∪ C − , for C ± representing discs of radius 0.8 at (0, ±1) (see Table 1 ).

As for the 3D restricted channel, we use Gmsh to generate a graded, triangular mesh with 5832 elements, with a characteristic length of 0.012 at the center and growing to 0.12 at the left/right edges.

This forces tighter clustering toward the center, as shown in Table 1 .

The discretization matrix for (4.1) is constructed with linear finite elements, and we target clusters of size 8.

2D anisotropic diffusion: The 2D domain is defined by the unit square, and we consider the problem −∇ • K∇u = f with pure Dirchlet conditions.

Generalizing Lloyd'S Algorithm For Graph Clustering *

Abstract: Clustering is a commonplace problem in many areas of data science, with applications in biology and bioinformatics, understanding chemical structure, image segmentation, building recommender systems, and many more fields. While there are many different clustering variants (based on given distance or graph structure, probability distributions, or data density), we consider here the problem of clustering nodes in a graph, motivated by the problem of aggregating discrete degrees of freedom in multigrid and domain decomposition methods for solving sparse linear systems. Specifically, we consider the challenge of forming balanced clusters in the graph of a sparse matrix for use in algebraic multigrid, although the algorithm has general applicability. Based on an extension of the Bellman-Ford algorithm, we generalize Lloyd's algorithm for partitioning subsets of R n to balance the number of nodes in each cluster; this is accompanied by a rebalancing algorithm that reduces the overall energy in the system. The algorithm provides control over the number of clusters and leads to "well centered" partitions of the graph. Theoretical results are provided to establish linear complexity and numerical results in the context of algebraic multigrid highlight the benefits of improved clustering.

Firedrake: Automating The Finite Element Method By Composing Abstractions

Abstract: Firedrake is a new tool for automating the numerical solution of partial differential equations. Firedrake adopts the domain-specific language for the finite element method of the FEniCS project, but with a pure Python runtime-only implementation centered on the composition of several existing and new abstractions for particular aspects of scientific computing. The result is a more complete separation of concerns that eases the incorporation of separate contributions from computer scientists, numerical analysts, and application specialists. These contributions may add functionality or improve performance. Firedrake benefits from automatically applying new optimizations. This includes factorizing mixed function spaces, transforming and vectorizing inner loops, and intrinsically supporting block matrix operations. Importantly, Firedrake presents a simple public API for escaping the UFL abstraction. This allows users to implement common operations that fall outside of pure variational formulations, such as flux limiters.


Prioritized Experience Replay (PER) #REFR is a technique for experience replay which involves replaying transitions with a high expected learning progress, as determined by the size of their temporal-difference (TD) error, more frequently than others.

Directly replacing the tabular Q functions with nonlinear function approximators such neural networks sometimes makes the training of RL agents generally hard to converge #OTHEREFR .

Potential reasons are correlated states or observations and optimization towards an unstable target. Experience replay is a successful method for addressing non-i.i.d. sampling.

The original setting of experience replay #OTHEREFR is to sample a batch of transitions uniformly from the replay memory and optimize the model.

Although this approach is viable, it may not be the most efficient method since the model may acquire more knowledge from certain transitions and less from others.

For example, experiences that led to large rewards or that resulted in the agent making a significant error are likely to be more informative than experiences that had little impact on the agent's behavior.

The probability of sampling the transition i is defined to be

where p i > 0 is the priority of transition i.

The parameter α controls how much the prioritization is used.

It means sampling transitions uniformly from the replay memory when α = 0.

Quantum Deep Q Learning With Distributed Prioritized Experience Replay

Abstract: This paper introduces the QDQN-DPER framework to enhance the efficiency of quantum reinforcement learning (QRL) in solving sequential decision tasks. The framework incorporates prioritized experience replay and asynchronous training into the training algorithm to reduce the high sampling complexities. Numerical simulations demonstrate that QDQN-DPER outperforms the baseline distributed quantum Q learning with the same model architecture. The proposed framework holds potential for more complex tasks while maintaining training efficiency.

Prioritized Experience Replay

Abstract: Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games.


Compared to the network with vanilla convolution (fifth row), the network with DCN #REFR in the last two blocks (2xDCN) increases the computation by only 0.1 GFLOPs and the running time by only 0.8 ms.

Feature Extraction: For accurate visual measurements, the extracted image feature should have a good localization performance and a large respective field on the image for a robust descriptor extraction.

We expect to achieve these goals by improving the baseline network ALIKE-N #OTHEREFR .

As shown in the first two rows of Table IX, we start by changing max-pooling to average-pooling (AVG) and ReLU to SELU #OTHEREFR .

These changes improve the mAA(10°) on the IMW-validation #OTHEREFR by 3.43% but have limited improvements on the other metrics.

Furthermore, for geometric invariance feature extraction, we replace the vanilla convolutions in the last two blocks with DCN #OTHEREFR as shown in the sixth and seventh rows of Table IX.

Besides improving the MS@3 on the Hpatches [50] by 2.81%, this change also improves the mAA(10°) and MS@3 on the IMW-validation #OTHEREFR by 4.36% and 3.28%, respectively.

Since the first two blocks have higher feature resolution, using DCN #OTHEREFR would significantly increase the computational cost.

Moreover, since the front blocks are responsible for low-level feature extraction, using DCN #OTHEREFR may degrade performance.

Therefore, we use DCN #OTHEREFR only in the last two blocks.

Score Head: Due to computational cost considerations, the baseline network has a score head of only one 1 × 1 convolutional layer.

Aliked: A Lighter Keypoint And Descriptor Extraction Network Via Deformable Transformation

Abstract: Image keypoints and descriptors play a crucial role in many visual measurement tasks. In recent years, deep neural networks have been widely used to improve the performance of keypoint and descriptor extraction. However, the conventional convolution operations do not provide the geometric invariance required for the descriptor. To address this issue, we propose the Sparse Deformable Descriptor Head (SDDH), which learns the deformable positions of supporting features for each keypoint and constructs deformable descriptors. Furthermore, SDDH extracts descriptors at sparse keypoints instead of a dense descriptor map, which enables efficient extraction of descriptors with strong expressiveness. In addition, we relax the neural reprojection error (NRE) loss from dense to sparse to train the extracted sparse descriptors. Experimental results show that the proposed network is both efficient and powerful in various visual measurement tasks, including image matching, 3D reconstruction, and visual relocalization.

Deformable Convnets V2: More Deformable, Better Results

Abstract: The superior performance of Deformable Convolutional Networks arises from its ability to adapt to the geometric variations of objects. Through an examination of its adaptive behavior, we observe that while the spatial support for its neural features conforms more closely than regular ConvNets to object structure, this support may nevertheless extend well beyond the region of interest, causing features to be influenced by irrelevant image content. To address this problem, we present a reformulation of Deformable ConvNets that improves its ability to focus on pertinent image regions, through increased modeling power and stronger training. The modeling power is enhanced through a more comprehensive integration of deformable convolution within the network, and by introducing a modulation mechanism that expands the scope of deformation modeling. To effectively harness this enriched modeling capability, we guide network training via a proposed feature mimicking scheme that helps the network to learn features that reflect the object focus and classification power of R-CNN features. With the proposed contributions, this new version of Deformable ConvNets yields significant performance gains over the original model and produces leading results on the COCO benchmark for object detection and instance segmentation.


Reference #REFR first reconstructs a harvesting function from samples by deep learning regression, and then, it optimizes constellation points for modulation by autoencoders based on neural networks.

In particular in #OTHEREFR , a nonlinear analytic model for a harvesting circuit is derived by approximating an ideal diode's characteristics via the Taylor series, and based on the model, it proposes to use multisine waveforms to deliver the largest energy.

However, this approach has limitations because individual components' laws may not even follow ideal circuit laws due to coupling effect, parasitic capacitors, etc., and then such approximation errors could accumulate.

For example, the parasitic components resulting from interrelationships make it difficult to analytically describe an equivalent end-to-end model for a circuit containing diodes.

Indeed, the true end-to-end energy harvesting function may only be available through experimental samples or circuit simulations #OTHEREFR - #OTHEREFR .

Inspired by this, recent literature attempts to take experimental samples and use a deep learning approach.

Reference #OTHEREFR focuses on the memory effect of harvesting circuits.

To capture the memory effect, it proposes a Markov decision process (MDP) model for the harvesting circuit and uses neural networks to estimate the MDP's parameters from data created by a circuit simulator.

Despite their success in practice, the deep learning approach has a critical drawback in that it generally provides no theoretical performance guarantees.

In addition, since optimization via deep learning is a black-box algorithm, it generally provides limited insights into SIET.

For instance, it cannot answer how many experimental samples are needed to attain a particular performance or how close our SIET design is to the optimal SIET.

Information And Energy Transmission With Wavelet-Reconstructed Harvesting Functions

Abstract: In practical simultaneous information and energy transmission (SIET), the exact energy harvesting function is usually unavailable because an energy harvesting circuit is nonlinear and nonideal. In this work, we consider a SIET problem where the harvesting function is accessible only at experimentally-taken sample points and study how close we can design SIET to the optimal system with such sampled knowledge. Assuming that the harvesting function is of bounded variation that may have discontinuities, we separately consider two settings where samples are taken without and with additive noise. For these settings, we propose to design a SIET system as if a waveletreconstructed harvesting function is the true one and study its asymptotic performance loss of energy and information delivery from the true optimal one. Specifically, for noiseless samples, it is shown that designing SIET as if the wavelet-reconstructed harvesting function is the truth incurs asymptotically vanishing energy and information delivery loss with the number of samples. For noisy samples, we propose to reconstruct wavelet coefficients via soft-thresholding estimation. Then, we not only obtain similar asymptotic losses to the noiseless case but also show that the energy loss by wavelets is asymptotically optimal up to a logarithmic factor.

Learning To Communicate And Energize: Modulation, Coding And Multiple Access Designs For Wireless Information-Power Transmission

Abstract: The explosion of the number of low-power devices in the next decades calls for a re-thinking of wireless network design, namely, unifying wireless transmission of information and power so as to make the best use of the RF spectrum, radiation, and infrastructure for the dual purpose of communicating and energizing. This paper provides a novel learning-based approach towards such wireless network design. To that end, a parametric model of a practical energy harvester, accounting for various sources of nonlinearities, is proposed using a nonlinear regression algorithm applied over collected real data. Relying on the proposed model, the learning problem of modulation design for Simultaneous Wireless Information-Power Transmission (SWIPT) over a point-to-point link is studied. Joint optimization of the transmitter and the receiver is implemented using Neural Network (NN)-based autoencoders. The results reveal that by increasing the receiver power demand, the baseband transmit modulation constellation converges to an On-Off keying signalling. Utilizing the observations obtained via learning, an algorithmic SWIPT modulation design is proposed. It is observed via numerical results that the performance loss of the proposed modulations are negligible compared to the ones obtained from learning. Extension of the studied problem to learning modulation design for multi-user SWIPT scenarios and coded modulation design for point-to-point SWIPT are considered. The major conclusion of this work is to utilize learning-based results to design non learning-based algorithms, which perform as well. In particular, inspired by the results obtained via learning, an algorithmic approach for coded modulation design is proposed, which performs very close to its learning counterparts, and is significantly superior due to its high real-time adaptability to new system design parameters.


In Section IV we compare our result with the converse bound which follows from #REFR , and with another code construction that does not allow for efficient encoding. Section V concludes the main part of the paper.

In Section II, we introduce important definitions and notations that are used throughout the paper.

We present our code construction and explain all its components in Section III.

The main result is formulated in Theorem 1 and its proof is divided into Lemmas 1-6.

Codes Correcting A Single Long Duplication Error

Abstract: We consider the problem of constructing a code capable of correcting a single long tandem duplication error of variable length. As the main contribution of this paper, we present a q-ary efficiently encodable code of length n + 1 and redundancy 1 that can correct a single duplication of length at least K = 4 • log q n + 1. The complexity of encoding is O(n 2 log n) and the complexity of decoding is O(n). We also present a q-ary non-efficient code of length n + 1 correcting single long duplication of length at least K = log q n + φ(n), where φ(n) → ∞ as n → ∞. This code has redundancy less than 1 for sufficiently large n. Moreover, we show that in the class of codes correcting a single long duplication with redundancy 1, the value K in our constructions is order-optimal.

Bounds And Constructions For Multi-Symbol Duplication Error Correcting Codes

Abstract: In this paper, we study codes correcting $t$ duplications of $\ell$ consecutive symbols. These errors are known as tandem duplication errors, where a sequence of symbols is repeated and inserted directly after its original occurrence. Using sphere packing arguments, we derive non-asymptotic upper bounds on the cardinality of codes that correct such errors for any choice of parameters. Based on the fact that a code correcting insertions of $t$ zero-blocks can be used to correct $t$ tandem duplications, we construct codes for tandem duplication errors. We compare the cardinalities of these codes with their sphere packing upper bounds. Finally, we discuss the asymptotic behavior of the derived codes and bounds, which yields insights about the tandem duplication channel.


To improve the model performance in 360 • IQA task, we utilize tangent image representation #REFR to reduce the distortion of the viewports extracted from the ERP image.

Our sampling module then extracts tangent viewports from salient regions of the distorted image in ERP format.

The main VQA module resembles the one used in #OTHEREFR , but the patch encoding block is re-modeled from the ground up to deal with the special structure of 360 • images.

To effectively encode input viewports into a sequence of tokens, we incorporate positional, geometric and source embeddings into the extracted sequence of tokens, and add a learnable classification token (CLS) to capture the global representation for the image.

The corresponding 360°image quality score is predicted by the output of a fully connected layer on top of the final CLS token representation at the output of the Transformer encoder. Sampling Module.

Most 360 • IQA datasets store images in the ERP format, which is the most popular spherical image representation, but is known to have significant distortions.

Moreover, to distinguish visually important parts of a panoramic image, we designed a sampling strategy motivated by the human visual attention mechanism.

In particular, we employ the 360 • image saliency prediction model named ATSal #OTHEREFR to predict salient regions of the panorama.

In this way, one can assign a saliency score for each patch extracted from the omnidirectional image, which we utilize in our sampling module.

Our main motivation behind our saliency-guided sampling scheme is to combine neural attention (self-attention) mechanism with human visual attention in a simple and intuitive manner.

At the first step of the sampling module, an input image in ERP format is fed to the ATSal saliency prediction model to predict a saliency map showing which regions are most likely to attract attention.

St360Iq: No-Reference Omnidirectional Image Quality Assessment With Spherical Vision Transformers

Abstract: Omnidirectional images, aka 360 • images, can deliver immersive and interactive visual experiences. As their popularity has increased dramatically in recent years, evaluating the quality of 360 • images has become a problem of interest since it provides insights for capturing, transmitting, and consuming this new media. However, directly adapting quality assessment methods proposed for standard natural images for omnidirectional data poses certain challenges. These models need to deal with very high-resolution data and implicit distortions due to the spherical form of the images. In this study, we present a method for noreference 360 • image quality assessment. Our proposed ST360IQ model extracts tangent viewports from the salient parts of the input omnidirectional image and employs a vision-transformers based module processing saliency selective patches/tokens that estimates a quality score from each viewport. Then, it aggregates these scores to give a final quality score. Our experiments on two benchmark datasets, namely OIQA and CVIQ datasets, demonstrate that as compared to the state-of-the-art, our approach predicts the quality of an omnidirectional image correlated with the human-perceived image quality. The code has been available on https://github.com/Nafiseh-Tofighi/ST360IQ

Convolutions On Spherical Images

Abstract: Applying convolutional neural networks to spherical images requires particular considerations. We look to the millennia of work on cartographic map projections to provide the tools to define an optimal representation of spherical images for the convolution operation. We propose a representation for deep spherical image inference based on the icosahedral Snyder equal-area (ISEA) projection, a projection onto a geodesic grid, and show that it vastly exceeds the state-of-the-art for convolution on spherical images, improving semantic segmentation results by 12.6%.


These findings improve upon those of #REFR , where a pre-trained BigGAN showed little to no correlation between FID and classification accuracy in a similar setting, strengthening the position of instance-conditioned models such as (CC-)IC-GAN. We observe similar trends for DeiT-B (see Appendix B).

In this analysis, we shed some light on the problematic (CC-)IC-GAN modeling of certain classes.

We believe that computing stratified results for generative models might be a good practice to be adopted by the community, as also supported by #OTHEREFR .

Nevertheless, the observed positive correlation between high classification accuracy and (CC-)IC-GAN's generation quality -studied through the lens of per-class FID and NN corruption-constitutes a promising result to improve the effectiveness of DA IC-GAN .

To this end, we ran an additional experiment where we avoid applying DA IC-GAN on classes having very high FID (>= 150), i.e., where (CC-)IC-GAN has very low generation quality. We report the results in Appendix B.

Notably, the impact of leveraging DA IC-GAN could be potentially improved by increasing the generation quality of the (CC-)IC-GAN's poorly modeled classes.

Instance-Conditioned Gan Data Augmentation For Representation Learning

Abstract: Data augmentation has become a crucial component to train state-of-the-art visual representation models. However, handcrafting combinations of transformations that lead to improved performances is a laborious task, which can result in visually unrealistic samples. To overcome these limitations, recent works have explored the use of generative models as learnable data augmentation tools, showing promising results in narrow application domains, e.g., few-shot learning and low-data medical imaging. In this paper, we introduce a data augmentation module, called DA IC-GAN , which leverages instance-conditioned GAN generations and can be used off-the-shelf in conjunction with most state-of-the-art training recipes. We showcase the benefits of DA IC-GAN by plugging it out-of-the-box into the supervised training of ResNets and DeiT models on the ImageNet dataset, and achieving accuracy boosts up to between 1%p and 2%p with the highest capacity models. Moreover, the learnt representations are shown to be more robust than the baselines when transferred to a handful of out-of-distribution datasets, and exhibit increased invariance to variations of instance and viewpoints. We additionally couple DA IC-GAN with a self-supervised training recipe and show that we can also achieve an improvement of 1%p in accuracy in some settings. With this work, we strengthen the evidence on the potential of learnable data augmentations to improve visual representation learning, paving the road towards non-handcrafted augmentations in model training.

Classification Accuracy Score For Conditional Generative Models

Abstract: Deep generative models (DGMs) of images are now sufficiently mature that they produce nearly photorealistic samples and obtain scores similar to the data distribution on heuristics such as Frechet Inception Distance (FID). These results, especially on large-scale datasets such as ImageNet, suggest that DGMs are learning the data distribution in a perceptually meaningful space and can be used in downstream tasks. To test this latter hypothesis, we use class-conditional generative models from a number of model classes—variational autoencoders, autoregressive models, and generative adversarial networks (GANs)—to infer the class labels of real data. We perform this inference by training an image classifier using only synthetic data and using the classifier to predict labels on real data. The performance on this task, which we call Classification Accuracy Score (CAS), reveals some surprising results not identified by traditional metrics and constitute our contributions. First, when using a state-of-the-art GAN (BigGAN-deep), Top-1 and Top-5 accuracy decrease by 27.9% and 41.6%, respectively, compared to the original data; and conditional generative models from other model classes, such as Vector-Quantized Variational Autoencoder-2 (VQ-VAE-2) and Hierarchical Autoregressive Models (HAMs), substantially outperform GANs on this benchmark. Second, CAS automatically surfaces particular classes for which generative models failed to capture the data distribution, and were previously unknown in the literature. Third, we find traditional GAN metrics such as Inception Score (IS) and FID neither predictive of CAS nor useful when evaluating non-GAN models. Furthermore, in order to facilitate better diagnoses of generative models, we open-source the proposed metric.