Following this study #REFR , we review four Non-IID scenarios that widely exist in real-world applications. They are summarized below.

FedAvg (McMahan et al., 2017) could solve the decentralized data optimization problem when these decentralized data are IID.

However, in real-world federated learning, the data distributions are usually Non-IID.

In this case, FedAvg would take more rounds to iterate the FL model under the Non-IID data setting.

The mathematical analysis of this issue in FL is given by work #OTHEREFR , which reveals that the local updates from Non-IID clients are divergent. It consequently damages the convergence performance of FedAvg.

• Feature distribution skew : The marginal distribution of label P(x) varies across clients.

In a handwriting recognition task, the same words written by different users may differ in stroke width, slant, etc.

• Label distribution skew : The marginal distribution of label P(y) may vary across clients.

For instance, clients' data samples are tied to geo-regions -kangaroos are only in Australia or zoos; for an online shopping website, customers only buy certain items with specific preferences.

• Feature concept skew : The condition distributions P(x|y) varies across clients, and P(y) is the same.

Stochastic Clustered Federated Learning

Abstract: Federated learning is a distributed learning framework that takes full advantage of private data samples kept on edge devices. In real-world federated learning systems, these data samples are often decentralized and Non-Independently Identically Distributed (Non-IID), causing divergence and performance degradation in the federated learning process. As a new solution, clustered federated learning groups federated clients with similar data distributions to impair the Non-IID effects and train a better model for every cluster. This paper proposes StoCFL, a novel clustered federated learning approach for generic Non-IID issues. In detail, StoCFL implements a flexible CFL framework that supports an arbitrary proportion of client participation and newly joined clients for a varying FL system, while maintaining a great improvement in model performance. The intensive experiments are conducted by using four basic Non-IID settings and a real-world dataset. The results show that StoCFL could obtain promising cluster results even when the number of clusters is unknown. Based on the client clustering results, models trained with StoCFL outperform baseline approaches in a variety of contexts.

The Non-Iid Data Quagmire Of Decentralized Machine Learning

Abstract: Many large-scale machine learning (ML) applications need to train ML models over decentralized datasets that are generated at different devices and locations. These decentralized datasets pose a fundamental challenge to ML because they are typically generated in very different contexts, which leads to significant differences in data distribution across devices/locations (i.e., they are not independent and identically distributed (IID)). In this work, we take a step toward better understanding this challenge, by presenting the first detailed experimental study of the impact of such non-IID data on the decentralized training of deep neural networks (DNNs). Our study shows that: (i) the problem of non-IID data partitions is fundamental and pervasive, as it exists in all ML applications, DNN models, training datasets, and decentralized learning algorithms in our study; (ii) this problem is particularly difficult for DNN models with batch normalization layers; and (iii) the degree of deviation from IID (the skewness) is a key determinant of the difficulty level of the problem. With these findings in mind, we present SkewScout, a system-level approach that adapts the communication frequency of decentralized learning algorithms to the (skew-induced) accuracy loss between data partitions. We also show that group normalization can recover much of the skew-induced accuracy loss of batch normalization.

four Non-IID scenarios
real-world applications


Imitation Learning Through Behavior Cloning

Interestingly, VICRegL is pre-trained purely on ImageNet #REFR , without robotic manipulation tasks in mind, while R3M is pre-trained on diverse human videos.

We now thoroughly evaluate the large suite of pre-trained vision models using BC.

We train BC policies with varying numbers of expert demonstrations to study how dataset size impacts performance in the low data regime.

The results in Figure 7 demonstrate that no individual pre-trained vision model can dominate all three environments. However, considering 21 tasks across all environments ( Fig. 7 rightmost) , R3M is the best-performing model. This echoes the finding of #OTHEREFR

(2022) that R3M enables data-efficient behavior cloning for robotic manipulation.

Furthermore, there is a model that follows R3M closely: VICRegL.

This lends credence to a dominant paradigm that advancements in vision can potentially be transferred directly to visuomotor control without extensive control-specific adaptation.

With respect to architectures, iBOT is clearly the bestperforming model among ViTs, highlighting the benefit of combining masked image modeling with self-distillation.

Another general observation is that ResNet-based models tend to outperform ViT-based models.

This is in spite of the fact that model size and computation complexity tend to favor ViTs.

We hypothesize this is because transformer-based architectures contain less visual prior knowledge.

For Pre-Trained Vision Models In Motor Control, Not All Policy Learning Methods Are Created Equal

Abstract: In recent years, increasing attention has been directed to leveraging pre-trained vision models for motor control. While existing works mainly emphasize the importance of this pre-training phase, the arguably equally important role played by downstream policy learning during controlspecific fine-tuning is often neglected. It thus remains unclear if pre-trained vision models are consistent in their effectiveness under different control policies. To bridge this gap in understanding, we conduct a comprehensive study on 14 pretrained vision models using 3 distinct classes of policy learning methods, including reinforcement learning (RL), imitation learning through behavior cloning (BC), and imitation learning with a visual reward function (VRF). Our study yields a series of intriguing results, including the discovery that the effectiveness of pre-training is highly dependent on the choice of the downstream policy learning algorithm. We show that conventionally accepted evaluation based on RL methods is highly variable and therefore unreliable, and further advocate for using more robust methods like VRF and BC. To facilitate more universal evaluations of pre-trained models and their policy learning methods in the future, we also release a benchmark of 21 tasks across 3 different environments alongside our work.

Imagenet Large Scale Visual Recognition Challenge

Abstract: The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements.


Comparisons With State-Of-The-Art Methods

Consistent with the performance on VG, PE-Net also achieves competitive results on Models PredCls SGCls SGDet zs-R@50 / 100 zs-R@50 / 100 zs-R@50 / 100 Motifs #REFR 3 Table 3 . Comparison with the state-of-the-art methods on Open-Images V6.

2, our model outperforms the vanilla Motifs and VCTree by 15.53%, 5.40%, 3.49%, and 15.38%, 4.45%, 2.91% at zs-R@100 on PredCls, SGCls, and SGDet.

Although TDE significantly improves the zero-shot performance by removing the effect of context bias, Motifs-TDE and VCTree-TDE are still surpassed by our PE-Net with 2.69%, 2.03%, 0.7%, and 3.29%, 2.53%, 0.4% on three tasks, respectively.

We owe the strength to the Prototype-based Modeling of our PE-Net, which models entity and predicate in the semantic space, significantly improving the model's analogical reasoning capability on unseen relation triplets.

To verify the generality of our method on different datasets, we conduct experiments on Open Images and present the results in Tab. 3.

We adopt the same evaluation metric as in #OTHEREFR .

The best and second best methods under each setting are marked according to formats.

Specifically, our method exceeds the BGNN with a large margin of 2.7% on average at four metrics, and outperforms RU-Net by 1.2%, 2.5%, and 1.4% at wmAP rel , wmAP phr , and score wtd , respectively.

It powerfully confirms PE-Net's generalization on handling relation recognition under different data distributions.

Prototype-Based Embedding Network For Scene Graph Generation

Abstract: Current Scene Graph Generation (SGG) methods explore contextual information to predict relationships among entity pairs. However, due to the diverse visual appearance of numerous possible subject-object combinations, there is a large intra-class variation within each predicate category, e.g., "man-eating-pizza, giraffe-eating-leaf", and the severe inter-class similarity between different classes, e.g., "man-holding-plate, man-eating-pizza", in model's latent space. The above challenges prevent current SGG methods from acquiring robust features for reliable relation prediction. In this paper, we claim that the predicate's category-inherent semantics can serve as class-wise prototypes in the semantic space for relieving the challenges. To the end, we propose the Prototype-based Embedding Network (PE-Net), which models entities/predicates with prototype-aligned compact and distinctive representations and thereby establishes matching between entity pairs and predicates in a common embedding space for relation recognition. Moreover, Prototype-guided Learning (PL) is introduced to help PE-Net efficiently learn such entitypredicate matching, and Prototype Regularization (PR) is devised to relieve the ambiguous entity-predicate matching caused by the predicate's semantic overlap. Extensive experiments demonstrate that our method gains superior relation recognition capability on SGG, achieving new state-of-the-art performances on both Visual Genome and Open Images datasets. The codes are available at https://github.com/VL-Group/PENET.

Neural Motifs: Scene Graph Parsing With Global Context

Abstract: We investigate the problem of producing structured graph representations of visual scenes. Our work analyzes the role of motifs: regularly appearing substructures in scene graphs. We present new quantitative insights on such repeated structures in the Visual Genome dataset. Our analysis shows that object labels are highly predictive of relation labels but not vice-versa. We also find there are recurring patterns even in larger subgraphs: more than 50% of graphs contain motifs involving at least two relations. This analysis leads to a new baseline that is simple, yet strikingly powerful. While hardly considering the overall visual context of an image, it outperforms previous approaches. We then introduce Stacked Motif Networks, a new architecture for encoding global context that is crucial for capturing higher order motifs in scene graphs. Our best model for scene graph detection achieves a 7.3% absolute improvement in recall@50 (41% relative gain) over prior state-of-the-art.


We use a triplet loss #REFR to contrastively optimize the gap between positive and negative pairs based on the cosine distance over the learned deep features.

Ideally, a machine learning system can make use of all data that is available.

As a related first step in this direction, we explore the feasibility of predicting information gleaned from 3D geometry using 2D projection images.

Being able to do so would allow a) pooling datasets that drastically differ in image types or b) relating information from a cheaper 2D screening to more readily interpretable 3D quantities that are difficult for a human observer.

We propose an image synthesis method for diverse modalities based on multimodal metric learning and k-NN regression.

To learn the metric, we use image retrieval as the target task, which aims at embedding images such that matching pairs of different modalities are close in the embedding space.

In contrast to the typical learning process, we carefully design the training scheme to avoid interference when training with longitudinal image data.

Given the learned embedding, we can synthesize images between diverse image types by k-NN regression through a weighted average based on their distances measured in the embedding space.

Given a large database, this strategy allows for a quick and simple estimation of one image type from another.

We use knee osteoarthritis as the driving medical problem and evaluate our proposed approach using the OAI image data.

Specifically, we predict cartilage thickness maps obtained from 3D MR images using 2D radiographs.

Mris: A Multi-Modal Retrieval Approach For Image Synthesis On Diverse Modalities

Abstract: Multiple imaging modalities are often used for disease diagnosis, prediction, or population-based analyses. However, not all modalities might be available due to cost, different study designs, or changes in imaging technology. If the differences between the types of imaging are small, data harmonization approaches can be used; for larger changes, direct image synthesis approaches have been explored. In this paper, we develop an approach based on multi-modal metric learning to synthesize images of diverse modalities. We use metric learning via multi-modal image retrieval, resulting in embeddings that can relate images of different modalities. Given a large image database, the learned image embeddings allow us to use k-nearest neighbor (k-NN) regression for image synthesis. Our driving medical problem is knee osteoarthritis (KOA), but our developed method is general after proper image alignment. We test our approach by synthesizing cartilage thickness maps obtained from 3D magnetic resonance (MR) images using 2D radiographs. Our experiments show that the proposed method outperforms direct image synthesis and that the synthesized thickness maps retain information relevant to downstream tasks such as progression prediction and Kellgren-Lawrence grading (KLG). Our results suggest that retrieval approaches can be used to obtain high-quality and meaningful image synthesis results given large image databases.

Facenet: A Unified Embedding For Face Recognition And Clustering

Abstract: Despite significant recent advances in the field of face recognition [10, 14, 15, 17], implementing face verification and recognition efficiently at scale presents serious challenges to current approaches. In this paper we present a system, called FaceNet, that directly learns a mapping from face images to a compact Euclidean space where distances directly correspond to a measure of face similarity. Once this space has been produced, tasks such as face recognition, verification and clustering can be easily implemented using standard techniques with FaceNet embeddings as feature vectors.


Moreover, in contrast with Shazeer and Stern #REFR we find that an out-of-date second moment estimator is primarily an issue for the (patch) embedding layer, and measure how well loss spikes are predicted by this event.

In our work we investigate instabilities which arise during CLIP training.

Unlike the instabilities observed in #OTHEREFR , we find these are not caused by attention entropy collapse.

Instead, our results indicate that spikes arise when the second moment estimator is out of date for the networks early layers.

While our analysis and methods build directly on Shazeer and Stern #OTHEREFR (AdaFactor), there are important differences.

In contrast with Shazeer and Stern #OTHEREFR , who only observe instabilities without warmup, we observe instabilities despite a long warmup period.

Finally, we note that reserachers have moved away from AdaFactor in its original formulation for large-scale training #OTHEREFR , finding AdaFactor to under-perform AdamW #OTHEREFR .

We believe this is due to the factored second moment or absence of first moment.

This is why our focus is AdamW #OTHEREFR which is the de facto standard optimizer for transformers.

Stable And Low-Precision Training For Large-Scale Vision-Language Models

Abstract: We introduce new methods for 1) accelerating and 2) stabilizing training for large language-vision models. 1) Towards accelerating training, we introduce SwitchBack , a linear layer for int8 quantized training which provides a speed-up of 13-25% while matching the performance of bfloat16 training within 0.1 percentage points for the 1B parameter CLIP ViT-Huge-the largest int8 training to date. Our main focus is int8 as GPU support for float8 is rare, though we also analyze float8 training through simulation. While SwitchBack proves effective for float8, we show that standard techniques are also successful if the network is trained and initialized so that large feature magnitudes are discouraged, which we accomplish via layer-scale initialized with zeros. 2) Towards stable training, we analyze loss spikes and find they consistently occur 1-8 iterations after the squared gradients become underestimated by their AdamW second moment estimator. As a result, we recommend an AdamW-Adafactor hybrid, which we refer to as StableAdamW because it avoids loss spikes when training a CLIP ViT-Huge model and outperforms gradient clipping.

Adafactor: Adaptive Learning Rates With Sublinear Memory Cost

Abstract: In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay rate of the second moment accumulator is too slow. We propose update clipping and a gradually increasing decay rate scheme as remedies. Combining these methods and dropping momentum, we achieve comparable results to the published Adam regime in training the Transformer model on the WMT 2014 English-German machine translation task, while using very little auxiliary storage in the optimizer. Finally, we propose scaling the parameter updates based on the scale of the parameters themselves.


The process is simple: capture a collection of images of a scene from varying viewpoints, reconstruct their camera parameters, and use the posed images to optimize a Neural Radiance Field #REFR .

With the emergence of efficient neural 3D reconstruction techniques, capturing a realistic digital representation of a real-world 3D scene has never been easier.

Due to its ease of use, we expect captured 3D content to gradually replace the traditional processes of manually-generated assets.

Unfortunately, while the pipelines for turning a real scene into a 3D representation are relatively mature and accessible, many of the other necessary tools for the creation of 3D assets (e.g., those needed for editing 3D scenes) remain underdeveloped.

Traditional processes for editing 3D models involve specialized tools and years of training in order to manually sculpt, extrude, and re-texture a given object.

This process is made even more involved with the advent of neural representations, which often do not have explicit surfaces.

This further motivates the need for 3D editing approaches designed for the modern era of 3D representations, particularly approaches that are similarly as accessible as the capture techniques themselves.

Instruct-Nerf2Nerf: Editing 3D Scenes With Instructions

Abstract: NeRF Scene Instruct NeRF2NeRF "Give him a cowboy hat" "Give him a mustache" "Make him bald" "Turn him into a clown" "As a bronze bust" "Turn him into Albert Einstein" "Turn his face into a skull" "Turn him into a Modigliani painting" "Turn him into Batman" Edited NeRF Figure 1: Editing 3D scenes with Instructions. We propose Instruct-NeRF2NeRF, a method for consistent 3D editing of a NeRF scene using text-based instructions. Our method can accomplish a diverse collection of local and global scene edits.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


One of the closest works to ours is #REFR comprised of a two-system, a fuzzy rule controller that takes the represented human knowledge constraints and returns a preferred action and a refined module that tunes the suboptimal knowledge.

another regularizer), and the design is such that it is equivalent to evaluating some Boolean constraint formula using Weighted Model Counting (WMC) which counts the weights of the solutions to a propositional logic formula #OTHEREFR .

Similar to #OTHEREFR , #OTHEREFR defines a non-negative loss function using fuzzy logic to incorporate logical constraints.

This loss measures how far the output of the network is from the nearest satisfying solution. Our approach differs from both of these.

We compile constraints and add them to the training process of an RL agent; our constraints are motivated by real-world physics, which are crucial for safety-critical domains.

Unlike #OTHEREFR we do not rely on WMC to evaluate the constraints as they rely on SAT solvers, instead we designed a more suitable metric for constraint evaluation in sequential decision-making tasks.

The constraints are represented in fuzzy logic and allow for imprecise policy selection.

During the constraint generation, they assume to have perfect knowledge of the state-space, and the constraints fully capture all aspects of the state-space, which is a strong assumption to hold.

Our work differs from this: we do not assume to know the full-state space dynamics and do not use constraints to warm-start the policy; further, our constraints are much more general and expressive.

To illustrate this generality, consider the rule from #OTHEREFR Rule: IF S 1 is M l 1 and S 2 is M l 2 and...

and S k is M l k THEN Action is a j : where S i are variables that describe different parts of the state, M l i is the fuzzy rules corresponding to each S i and a j is the action taken.

System Iii: Learning With Domain Knowledge For Safety Constraints

Abstract: Reinforcement learning agents naturally learn from extensive exploration. Exploration is costly and can be unsafe in safety-critical domains. This paper proposes a novel framework for incorporating domain knowledge to help guide safe exploration and boost sample efficiency. Previous approaches impose constraints, such as regularisation parameters in neural networks, that rely on large sample sets and often are not suitable for safety-critical domains where agents should almost always avoid unsafe actions. In our approach, called System III, which is inspired by psychologists' notions of the brain's System I and System II, we represent domain expert knowledge of safety in form of first-order logic. We evaluate the satisfaction of these constraints via p-norms in state vector space. In our formulation, constraints are analogous to hazards, objects, and regions of state that have to be avoided during exploration. We evaluated the effectiveness of the proposed method on OpenAI's Gym and Safety-Gym environments. In all tasks, including classic Control and Safety Games, we show that our approach results in safer exploration and sample efficiency.

Kogun: Accelerating Deep Reinforcement Learning Via Integrating Human Suboptimal Knowledge

Abstract: Reinforcement learning agents usually learn from scratch, which requires a large number of interactions with the environment. This is quite different from the learning process of human. When faced with a new task, human naturally have the common sense and use the prior knowledge to derive an initial policy and guide the learning process afterwards. Although the prior knowledge may be not fully applicable to the new task, the learning process is significantly sped up since the initial policy ensures a quick-start of learning and intermediate guidance allows to avoid unnecessary exploration. Taking this inspiration, we propose knowledge guided policy network (KoGuN), a novel framework that combines human prior suboptimal knowledge with reinforcement learning. Our framework consists of a fuzzy rule controller to represent human knowledge and a refine module to fine-tune suboptimal prior knowledge. The proposed framework is end-to-end and can be combined with existing policy-based reinforcement learning algorithm. We conduct experiments on both discrete and continuous control tasks. The empirical results show that our approach, which combines human suboptimal knowledge and RL, achieves significant improvement on learning efficiency of flat RL algorithms, even with very low-performance human prior knowledge.

fuzzy rule controller
suboptimal knowledge


Here, we revisit MAS #REFR and replay to test how they influence the general representation, see Figure 3b .

Crucially, we also find that the benefit of dropout is different in nature from that of using augmentations.

Dropout improves the current task performance, but not the general representation.

This is also found in #OTHEREFR , where ImageNet models trained without dropout transfer better.

Considering that dropout also results in faster forgetting (see Figure 4a) , we conclude that optimizing with augmentations, but without dropout is the preferable choice.

Methods Continual learning methods can prevent forgetting to some extent, but only after a task is learned.

While MAS remembers early task representations (Section 5.1), we find that its final representation quality is of the same level as that of finetuning.

This is in line with what is reported in both #OTHEREFR . Instead, Hu et al. report an advantage of MAS #OTHEREFR .

This is possibly an effect of their trained tasks (ImageNet) overlapping more with their downstream tasks (e.g.

classifying planes, cars and pets), as we found that MAS can indeed remember some overlapping information.

Also shown in this plot are two baselines, single task and multi task training, to put these results in perspective.

Knowledge Accumulation In Continually Learned Representations And The Issue Of Feature Forgetting

Abstract: By default, neural networks learn on all training data at once. When such a model is trained on sequential chunks of new data, it tends to catastrophically forget how to handle old data. In this work we investigate how continual learners learn and forget representations. We observe two phenomena: knowledge accumulation, i.e. the improvement of a representation over time, and feature forgetting, i.e. the loss of task-specific representations. To better understand both phenomena, we introduce a new analysis technique called task exclusion comparison. If a model has seen a task and it has not forgotten all the task-specific features, then its representation for that task should be better than that of a model that was trained on similar tasks, but not that exact one. Our image classification experiments show that most task-specific features are quickly forgotten, in contrast to what has been suggested in the past. Further, we demonstrate how some continual learning methods, like replay, and ideas from representation learning affect a continually learned representation. We conclude by observing that representation quality is tightly correlated with continual learning performance.

Memory Aware Synapses: Learning What (Not) To Forget

Abstract: Humans can learn in a continuous manner. Old rarely utilized knowledge can be overwritten by new incoming information while important, frequently used knowledge is prevented from being erased. In artificial learning systems, lifelong learning so far has focused mainly on accumulating knowledge over tasks and overcoming catastrophic forgetting. In this paper, we argue that, given the limited model capacity and the unlimited new information to be learned, knowledge has to be preserved or erased selectively. Inspired by neuroplasticity, we propose an online method to compute the importance of the parameters of a neural network, based on the data that the network is actively applied to, in an unsupervised manner. After learning a task, whenever a sample is fed to the network, we accumulate an importance measure for each parameter of the network, based on how sensitive the predicted output is to a change in this parameter. When learning a new task, changes to important parameters are penalized. We show that a local version of our method is a direct application of Hebb's rule in identifying the important connections between neurons. We test our method on a sequence of object recognition tasks and on the challenging problem of learning an embedding in a continuous manner. We show state of the art performance and the ability to adapt the importance of the parameters towards what the network needs (not) to forget, which may be different for different test conditions.


Progress in this field has been driven by two major concepts: the Transformer neural network architecture and the application of transfer learning in training #REFR .

Our vision is inspired by recent advances in the domain of natural language processing (NLP).

The Transformer has now become the standard architecture in NLP.

One of its benefits is its suitability for parallelization w.r.t.

previous approaches, which has enabled the creation of massive pre-trained LLMs #OTHEREFR .

These models are pre-trained on tasks, such as predicting the next word in a sentence, for which large amounts of data are easily accessible.

Although pre-training is costly, the models can then be adapted to a number of new tasks.

Querying Large Language Models With Sql [Vision

Abstract: In many use-cases, information is stored in text but not available in structured data. However, extracting data from natural language text to precisely fit a schema, and thus enable querying, is a challenging task. With the rise of pre-trained Large Language Models (LLMs), there is now an effective solution to store and use information extracted from massive corpora of text documents. Thus, we envision the use of SQL queries to cover a broad range of data that is not captured by traditional databases by tapping the information in LLMs. To ground this vision, we present Galois, a prototype based on a traditional database architecture, but with new physical operators for querying the underlying LLM. The main idea is to execute some operators of the the query plan with prompts that retrieve data from the LLM. For a large class of SQL queries, querying LLMs returns well structured relations, with encouraging qualitative results. Preliminary experimental results make pre-trained LLMs a promising addition to the field of database systems, introducing a new direction for hybrid query processing. However, we pinpoint several research challenges that must be addressed to build a DBMS that exploits LLMs. While some of these challenges necessitate integrating concepts from the NLP literature, others offer novel research avenues for the DB community.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


Kim, Jeong, and Ko #REFR tried to improve the explainability by simplifying a trained deep random forest model.

Empirical successes have attracted a line of work to the theoretical analysis of deep forest.

To start theoretical analysis of deep forests, Lyu, Yang, and Zhou #OTHEREFR prove a margin-based generalization bound for the additive cascade forest.

For the consistency of deep forests, Arnould, Boyer, and Scornet #OTHEREFR prove a tight bound on the excess risk of two-layer centered random trees, which regards the second layer as a variance reducer.

Assuming that the raw and new features are separated and independently used in two stages, Lyu, Chen, and Zhou #OTHEREFR prove that new features are easy to cause overfitting risk.

Although deep forest has been widely used and studied, very little work has been done to help users understand the specific predictions made by deep forest models.

Although they select the most important paths in the component forests, going through all the selected paths layer by layer is still a big workload, hence the interpretability is limited.

In this paper, we aim to explain the predictions made by the original trained deep forest, and the explanation is by directly telling how much the considered feature enlarges or reduces the predicted value for a given class.

Explaining random forests Random forest has long been a successful machine learning method, especially for tabular data #OTHEREFR .

A wealth of work has focused on explaining how each input feature functions in the model.

Interpreting Deep Forest Through Feature Contribution And Mdi Feature Importance

Abstract: Deep forest is a non-differentiable deep model which has achieved impressive empirical success across a wide variety of applications, especially on categorical/symbolic or mixed modeling tasks. Many of the application fields prefer explainable models, such as random forests with feature contributions that can provide local explanation for each prediction, and Mean Decrease Impurity (MDI) that can provide global feature importance. However, deep forest, as a cascade of random forests, possesses interpretability only at the first layer. From the second layer on, many of the tree splits occur on the new features generated by the previous layer, which makes existing explanatory tools for random forests inapplicable. To disclose the impact of the original features in the deep layers, we design a calculation method with an estimation step followed by a calibration step for each layer, and propose our feature contribution and MDI feature importance calculation tools for deep forest. Experimental results on both simulated data and real world data verify the effectiveness of our methods.

Interpretation And Simplification Of Deep Forest

Abstract: This paper proposes a new method for interpreting and simplifying a black box model of a deep random forest (RF) using a proposed rule elimination. In deep RF, a large number of decision trees are connected to multiple layers, thereby making an analysis difficult. It has a high performance similar to that of a deep neural network (DNN), but achieves a better generalizability. Therefore, in this study, we consider quantifying the feature contributions and frequency of the fully trained deep RF in the form of a decision rule set. The feature contributions provide a basis for determining how features affect the decision process in a rule set. Model simplification is achieved by eliminating unnecessary rules by measuring the feature contributions. Consequently, the simplified model has fewer parameters and rules than before. Experiment results have shown that a feature contribution analysis allows a black box model to be decomposed for quantitatively interpreting a rule set. The proposed method was successfully applied to various deep RF models and benchmark datasets while maintaining a robust performance despite the elimination of a large number of rules.


A Quantum Switch With Ghz Entangled Control.

We compare it to the local fraction for a GHZ experiment with the same measurements (right), as originally shown by Figure 1 At the angular resolution of 100 used by the plots, the minimum causally separable fraction of 41.2% is achieved at 0 = #REFR 25 and 1 = 3 5 .

The empirical model is causally inseparable for the space right above, with a causally separable fraction of~41.2% over its two causal completions.

The two completions are induced by the causal orders total ( , , ) ∨ discrete ( ) ∨ discrete ( ) and total ( , , ) ∨ discrete ( ) ∨ discrete ( ), and their causaltopes are 386-dimensional:

Causal fraction:~20.6% Causal fraction:~20.6%

The empirical model becomes causally separable if the no-signalling constraint to Diane and Eve is dropped: the model is 100% supported by the spaces of input histories induced by causal orders total ( , , , , ) and total ( , , , , ), the two causal completions of total ({A, B} , , , ), with each causal completion supporting exactly 50% of the empirical model.

Below we plot the causally separable fraction (left) and the local fraction (middle) for this empirical model as a function of the 0 and 1 measurement angles used by Charlie, Diane and Eve.

For comparison, the minimum local fraction for the empirical model is 0%, and it coincides with the minimum local fraction for the corresponding GHZ experiment.

The local fraction for the empirical model, computed over all 1048576 causal functions (reduced to 32768 compatible with the empirical model support), coincides with the causally separable local fraction, computed over the 458752 causally separable functions (reduced to 16384 compatible with the empirical model support).

To better understand the differences between the three landscapes above, below we plot the difference between causally separable fraction and local fraction for the empirical model (left), as well as the difference between local fraction for the GHZ experiment and local fraction for the empirical model (right).

The Geometry Of Causality

Abstract: We provide a unified operational framework for the study of causality, non-locality and contextuality, in a fully device-independent and theory-independent setting. This paper is the final instalment in a trilogy: spaces of input histories, our dynamical generalisation of causal orders, were introduced in "The Combinatorics of Causality"; the sheaf-theoretic treatment of causal distributions, known to us as empirical models, was detailed in "The Topology of Causality". We define causaltopes-our chosen portmanteau of "causal polytopes"-for arbitrary spaces of input histories and arbitrary choices of input contexts. We show that causaltopes are obtained by slicing simpler polytopes of conditional probability distributions with a set of causality equations, which we fully characterise. We provide efficient linear programs to compute the maximal component of an empirical model supported by any given sub-causaltope, as well as the associated causal fraction. We introduce a notion of causal separability relative to arbitrary causal constraints. We provide efficient linear programs to compute the maximal causally separable component of an empirical model, and hence its causally separable fraction, as the component jointly supported by certain sub-causaltopes. We study causal fractions and causal separability for several novel examples, including a selection of quantum switches with entangled or contextual control. In the process, we demonstrate the existence of "causal contextuality", a phenomenon where causal inseparability is clearly correlated to, or even directly implied by, non-locality and contextuality.

The Sheaf-Theoretic Structure Of Non-Locality And Contextuality

Abstract: We use the mathematical language of sheaf theory to give a unified treatment of non-locality and contextuality, in a setting that generalizes the familiar probability tables used in non-locality theory to arbitrary measurement covers; this includes Kochen–Specker configurations and more. We show that contextuality, and non-locality as a special case, correspond exactly to obstructions to the existence of global sections. We describe a linear algebraic approach to computing these obstructions, which allows a systematic treatment of arguments for non-locality and contextuality. We distinguish a proper hierarchy of strengths of no-go theorems, and show that three leading examples—due to Bell, Hardy and Greenberger, Horne and Zeilinger, respectively—occupy successively higher levels of this hierarchy. A general correspondence is shown between the existence of local hidden-variable realizations using negative probabilities, and no-signalling; this is based on a result showing that the linear subspaces generated by the non-contextual and no-signalling models, over an arbitrary measurement cover, coincide. Maximal non-locality is generalized to maximal contextuality, and characterized in purely qualitative terms, as the non-existence of global sections in the support. A general setting is developed for the Kochen–Specker-type results, as generic, model-independent proofs of maximal contextuality, and a new combinatorial condition is given, which generalizes the ‘parity proofs’ commonly found in the literature. We also show how our abstract setting can be represented in quantum mechanics. This leads to a strengthening of the usual no-signalling theorem, which shows that quantum mechanics obeys no-signalling for arbitrary families of commuting observables, not just those represented on different factors of a tensor product.


It is urgently required that law enforcement and counter-terrorism departments can realize the real-time tracking on the public threats #REFR .

In major public safety events (e.g., terrorist attacks in public transportation scenarios), law enforcement may want to track down the identified perpetrators #OTHEREFR .

Across modern cities, a large amount of cameras are installed in public areas around us, including underground transportation networks, ground transportation networks, and air transportation networks.

By placing cameras in public places such as roads, public transport, retail stores, parks and libraries, relevant departments can help prevent, track and solve crime problems via video analytics technologies #OTHEREFR . Many works focused on the video analytics-based surveillance applications. For example, Yi et al.

#OTHEREFR highlighted the low-latency video analytics requirements for public safety applications, such as counter-terrorism. Jain et al.

#OTHEREFR focused on applications based on multiple cameras, like crowd control and spotlight search.

To sum up, the "real-time" characteristic is one of the most essentially focused metrics in surveillance applications, where the edge computing technology can help boost the performance.

2) Rescue applications: Different from surveillance applications, rescue applications need to not only detect, but also identify and track the corresponding objects.

Edge-Based Video Analytics: A Survey

Abstract: Edge computing has been getting a momentum with ever-increasing data at the edge of the network. In particular, huge amounts of video data and their real-time processing requirements have been increasingly hindering the traditional cloud computing approach due to high bandwidth consumption and high latency. Edge computing in essence aims to overcome this hindrance by processing most video data making use of edge servers, such as small-scale on-premises server clusters, server-grade computing resources at mobile base stations and even mobile devices like smartphones and tablets; hence, the term edge-based video analytics. However, the actual realization of such analytics requires more than the simple, collective use of edge servers. In this paper, we survey state-of-the-art works on edge-based video analytics with respect to applications, architectures, techniques, resource management, security and privacy. We provide a comprehensive and detailed review on what works, what doesn't work and why. These findings give insights and suggestions for next generation edge-based video analytics. We also identify open issues and research directions.

Large Scale Audio-Visual Video Analytics Platform For Forensic Investigations Of Terroristic Attacks

Abstract: The forensic investigation of a terrorist attack poses a huge challenge to the investigative authorities, as several thousand hours of video footage need to be spotted. To assist law enforcement agencies (LEA) in identifying suspects and securing evidences, we present a platform which fuses information of surveillance cameras and video uploads from eyewitnesses. The platform integrates analytical modules for different input-modalities on a scalable architecture. Videos are analyzed according their acoustic and visual content. Specifically, Audio Event Detection is applied to index the content according to attack-specific acoustic concepts. Audio similarity search is utilized to identify similar video sequences recorded from different perspectives. Visual object detection and tracking are used to index the content according to relevant concepts. The heterogeneous results of the analytical modules are fused into a distributed index of visual and acoustic concepts to facilitate rapid start of investigations, following traits and investigating witness reports.


We fine-tune a MobileNet v2 #REFR using the annotated 70,124 frames extracted from raw video data.

We implement our methods on the RoCoG-v2 dataset in a preinstalled Ubuntu Linux 18.04 using Robot Operating System 2(ROS2) Galactics.

The fine-tuned MobileNet v2 #OTHEREFR is operated as the core detector in our auto zoom algorithm.

We evaluate our method with Mobile Video Networks(MoViNets) #OTHEREFR .

Specifically, we test our method with MoViNet A0, A2 and A3 on RB5.

MoViNet A0 is the smallest model in MoViNets and has input size 172×172. A2 uses deeper network design with input 224×224. A3 is the larger network, with input 256×256.

Also, we choose the stream version of the MoViNets due to lack of support for 3D conv on edge devices, as shown in Table I .

Aztr: Aerial Video Action Recognition With Auto Zoom And Temporal Reasoning

Abstract: We propose a novel approach for aerial video action recognition. Our method is designed for videos captured using UAVs and can run on edge or mobile devices. We present a learning-based approach that uses customized auto zoom to automatically identify the human target and scale it appropriately. This makes it easier to extract the key features and reduces the computational overhead. We also present an efficient temporal reasoning algorithm to capture the action information along the spatial and temporal domains within a controllable computational cost. Our approach has been implemented and evaluated both on the desktop with high-end GPUs and on the low power Robotics RB5 Platform for robots and drones. In practice, we achieve 6.1-7.4% improvement over SOTA in Top-1 accuracy on the RoCoG-v2 dataset, 8.3-10.4% improvement on the UAV-Human dataset and 3.2% improvement on the Drone Action dataset.

Mobilenetv2: Inverted Residuals And Linear Bottlenecks

Abstract: In this paper we describe a new mobile architecture, MobileNetV2, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We also describe efficient ways of applying these mobile models to object detection in a novel framework we call SSDLite. Additionally, we demonstrate how to build mobile semantic segmentation models through a reduced form of DeepLabv3 which we call Mobile DeepLabv3. The MobileNetV2 architecture is based on an inverted residual structure where the input and output of the residual block are thin bottleneck layers opposite to traditional residual models which use expanded representations in the input an MobileNetV2 uses lightweight depthwise convolutions to filter features in the intermediate expansion layer. Additionally, we find that it is important to remove non-linearities in the narrow layers in order to maintain representational power. We demonstrate that this improves performance and provide an intuition that led to this design. Finally, our approach allows decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis. We measure our performance on Imagenet classification, COCO object detection, VOC image segmentation. We evaluate the trade-offs between accuracy, and number of operations measured by multiply-adds (MAdd), as well as the number of parameters


R(Α, C ) ⊆ R(Α, C), For Any Arc Α ∈ C ∩ C .

If the insertion of α i splits an arc ω ∈ C i−1 into ω 1 and ω 2 , but ω 2 ∈ C, then eliminate R(ω 2 , C i ) by calling #REFR .

Given V l (C i ), 1 ≤ i ≤ n, we can delete R(α, C i ), if R(α, C i ) ⊆ VR(s α , S α )

, where S α ⊆ S C is the set of sites that define ∂R(α, C i ), in expected time linear on |S α |.

There are two ways to use Proposition 23, if applicable: 1.

Use it when necessary to maintain the invariant that C i encloses C (by deleting #OTHEREFR any auxiliary arc in C i−1 that blocks the insertion of α i , thus, eliminating the case C i = C i−1 ). 2.

Eliminate any auxiliary arc at the time of its creation.

The advantage of the latter is that Voronoi-like cycles become order-independent, therefore, backwards analysis becomes possible to establish the algorithm's time complexity.

We give the backwards analysis argument on the concrete case of Section 6; the same type of argument, only more technical, can be derived for this abstract formulation as well.

Abstract Voronoi-Like Graphs: Extending Delaunay'S Theorem And Applications

Abstract: Any system of bisectors (in the sense of abstract Voronoi diagrams) defines an arrangement of simple curves in the plane. We define Voronoi-like graphs on such an arrangement, which are graphs whose vertices are locally Voronoi. A vertex v is called locally Voronoi, if v and its incident edges appear in the Voronoi diagram of three sites. In a so-called admissible bisector system, where Voronoi regions are connected and cover the plane, we prove that any Voronoi-like graph is indeed an abstract Voronoi diagram. The result can be seen as an abstract dual version of Delaunay's theorem on (locally) empty circles. Further, we define Voronoi-like cycles in an admissible bisector system, and show that the Voronoi-like graph induced by such a cycle C is a unique tree (or a forest, if C is unbounded). In the special case where C is the boundary of an abstract Voronoi region, the induced Voronoi-like graph can be computed in expected linear time following the technique of [Junginger and Papadopoulou SOCG'18]. Otherwise, within the same time, the algorithm constructs the Voronoi-like graph of a cycle C on the same set (or subset) of sites, which may equal C or be enclosed by C. Overall, the technique computes abstract Voronoi (or Voronoi-like) trees and forests in linear expected time, given the order of their leaves along a Voronoi-like cycle. We show a direct application in updating a constraint Delaunay triangulation in linear expected time, after the insertion of a new segment constraint, simplifying upon the result of [Shewchuk and Brown CGTA 2015].

Deletion In Abstract Voronoi Diagrams In Expected Linear Time

Abstract: Updating an abstract Voronoi diagram in linear time, after deletion of one site, has been an open problem for a long time. Similarly for various concrete Voronoi diagrams of generalized sites, other than points. In this paper we present a simple, expected linear-time algorithm to update an abstract Voronoi diagram after deletion. We introduce the concept of a Voronoi-like diagram, a relaxed version of a Voronoi construct that has a structure similar to an abstract Voronoi diagram, without however being one. Voronoi-like diagrams serve as intermediate structures, which are considerably simpler to compute, thus, making an expected linear-time construction possible. We formalize the concept and prove that it is robust under an insertion operation, thus, enabling its use in incremental constructions.


We obtain the part segmentation mask obtained by rendering the ground truth MANO hand mesh using a differentiable neural renderer #REFR .

For adaptive global representation learning, we generate heatmaps by adjusting the Gaussian kernel size K according to the bounding box size of the hand in data preparation for supervision (details in Supplementary Material).

As the first representation of ACR, this map explicitly mitigates inter-dependencies between hands and serves as an attention mask for better global representation learning.

Part Segmentation map: A p ∈ R 33×H×W is learnt as a probabilistic segmentation volume.

Each pixel on the volume is a channel of probability logits over 33 classes which consists of 1 background and 16 hand part classes for each hand corresponding to MANO model.

As the second representation of ACR, this map serves as an attention mask for part representation learning.

M c ∈ R 218×H×W contains two maps, M h c ∈ R 109×H×W

It is split into two sets of parameters which are MANO parameter θ ∈ R 16×6 , β ∈ R 10 and 3 camera parameters for cross hand inverse feature query.

Acr: Attention Collaboration-Based Regressor For Arbitrary Two-Hand Reconstruction

Abstract: Reconstructing two hands from monocular RGB images is challenging due to frequent occlusion and mutual confusion. Existing methods mainly learn an entangled representation to encode two interacting hands, which are incredibly fragile to impaired interaction, such as truncated hands, separate hands, or external occlusion. This paper presents ACR (Attention Collaborationbased Regressor), which makes the first attempt to reconstruct hands in arbitrary scenarios. To achieve this, ACR explicitly mitigates interdependencies between hands and between parts by leveraging center and part-based attention for feature extraction. However, reducing interdependence helps release the input constraint while weakening the mutual reasoning about reconstructing the interacting hands. Thus, based on center attention, ACR also learns cross-hand prior that handle the interacting hands better. We evaluate our method on various types of hand reconstruction datasets. Our method significantly outperforms the best interacting-hand approaches on the Inter-Hand2.6M dataset while yielding comparable performance with the state-of-the-art single-hand methods on the Frei-Hand dataset. More qualitative results on in-the-wild and hand-object interaction datasets and web images/videos further demonstrate the effectiveness of our approach for arbitrary hand reconstruction. Our code is available at https://github.com/ZhengdiYu/Arbitrary-Hands-3D-Reconstruction.

Neural 3D Mesh Renderer

Abstract: For modeling the 3D world behind 2D images, which 3D representation is most appropriate? A polygon mesh is a promising candidate for its compactness and geometric properties. However, it is not straightforward to model a polygon mesh from 2D images using neural networks because the conversion from a mesh to an image, or rendering, involves a discrete operation called rasterization, which prevents back-propagation. Therefore, in this work, we propose an approximate gradient for rasterization that enables the integration of rendering into neural networks. Using this renderer, we perform single-image 3D mesh reconstruction with silhouette image supervision and our system outperforms the existing voxel-based approach. Additionally, we perform gradient-based 3D mesh editing operations, such as 2D-to-3D style transfer and 3D DeepDream, with 2D supervision for the first time. These applications demonstrate the potential of the integration of a mesh renderer into neural networks and the effectiveness of our proposed renderer.

hand mesh
differentiable neural renderer


However, it has been observed #REFR ) that this big-M formulation is not strong in the sense that its LP relaxation does not, in general, capture the convex hull of the graph of a given unit; see Figure 9 for an illustration.

Here, M l i,− and M l i,+ are data which must satisfy the inequalities

M l i,− ≤ min h l−1 ∈D l−1 w l i h l−1 + b l i M l i,+ ≥ max h l−1 ∈D l−1 w l i h l−1 + b l i .

This big-M formulation for ReLU-based networks has been used extensively in the literature #OTHEREFR .

The big-M formulation is compact, with one binary variable and O(1) general inequality constraints for each neuron.

Applied for each unit in the network, this leads to a MILP formulation with O( l∈L n l ) = O(Ln max ) binary variables and general inequality constraints, where n max = max l∈L n L .

In fact, (Anderson et al., 2019, Example 2) observe that this LP relaxation can be arbitrarily bad, even in fixed input dimension.

As MILP solvers often bound the objective function between the best feasible point and its tightest optimal continuous relaxation, a weak formulation can negatively impact performance, often substantially.

It is worth dwelling on where this lack of strength comes from.

If the input h l−1 is one dimensional, the big-M formulation is locally ideal #OTHEREFR : the extreme points of the linear programming relaxation (4a-4d) naturally satisfy the integrality constraints (4e).

However, this fails to hold in the general Figure 9 : Left: The convex hull of a ReLU neuron (5), and Right: the convex relaxation offered by the big-M formulation (4) Adapted from #OTHEREFR multivariate input case.

When Deep Learning Meets Polyhedral Theory: A Survey

Abstract: In the past decade, deep learning became the prevalent methodology for predictive modeling thanks to the remarkable accuracy of deep neural networks in tasks such as computer vision and natural language processing. Meanwhile, the structure of neural networks converged back to simpler representations based on piecewise constant and piecewise linear functions such as the Rectified Linear Unit (ReLU), which became the most commonly used type of activation function in neural networks. That made certain types of network structure-such as the typical fully-connected feedforward neural network-amenable to analysis through polyhedral theory and to the application of methodologies such as Linear Programming (LP) and Mixed-Integer Linear Programming (MILP) for a variety of purposes. In this paper, we survey the main topics emerging from this fast-paced area of work, which bring a fresh perspective to understanding neural networks in more detail as well as to applying linear optimization techniques to train, verify, and reduce the size of such networks.

Strong Mixed-Integer Programming Formulations For Trained Neural Networks

Abstract: We present an ideal mixed-integer programming (MIP) formulation for a rectified linear unit (ReLU) appearing in a trained neural network. Our formulation requires a single binary variable and no additional continuous variables beyond the input and output variables of the ReLU. We contrast it with an ideal “extended” formulation with a linear number of additional continuous variables, derived through standard techniques. An apparent drawback of our formulation is that it requires an exponential number of inequality constraints, but we provide a routine to separate the inequalities in linear time. We also prove that these exponentially-many constraints are facet-defining under mild conditions. Finally, we study network verification problems and observe that dynamically separating from the exponential inequalities (1) is much more computationally efficient and scalable than the extended formulation, (2) decreases the solve time of a state-of-the-art MIP solver by a factor of 7 on smaller instances, and (3) nearly matches the dual bounds of a state-of-the-art MIP solver on harder instances, after just a few rounds of separation and in orders of magnitude less time.


It is observed that when a deeper auxiliary network is used, the final accuracy loss for the entire model is small #REFR .

Existing research incorporates local error signals to train the device-side model using an auxiliary network #OTHEREFR , #OTHEREFR .

The auxiliary network consists of a few lightweight hidden layers that map the output of the device-side model to the same dimensions of the ground truth labels.

Although the local error signals eliminate the need to transfer global gradients from the server to devices, this approach adversely impacts accuracy since the device-side model trained by local error signals may be sub-optimal. We will empirically demonstrate this in Section IV-C.

A similar accuracy loss is reported in the literature for greedy layer-wise learning.

In this method, the use of a simple one-hidden layer auxiliary network results in accuracy loss when compared to end-to-end training that uses global loss calculated with the entire model #OTHEREFR .

However, adding more layers to the auxiliary network will increase the computational burden on resource-constrained devices, thereby offsetting the performance gains obtained from partitioning the DNN.

To address the above, ActionFed adopts pre-trained initialization of the device-side model (w C ).

The device-side model is initialized with pre-trained weights (w C,k ), which are the partial weights of the entire model trained on a large dataset (e.g. ImageNet #OTHEREFR ).

In addition, we empirically study the impact of local loss on generating activation outputs, which are used to train the server-side model.

We identify that local loss is not a satisfactory criterion for representing a 'good' activation output for training the server-side model with pretrained initialization (Section IV-C).

Communication Efficient Dnn Partitioning-Based Federated Learning

Abstract: Efficiently running federated learning (FL) on resource-constrained devices is challenging since they are required to train computationally intensive deep neural networks (DNN) independently. DNN partitioning-based FL (DPFL) has been proposed as one mechanism to accelerate training where the layers of a DNN (or computation) are offloaded from the device to an edge server. However, this creates significant communication overheads since the activation and gradient need to be transferred between the device and the edge server during training. Current techniques reduce the communication introduced by DNN partitioning using local loss-based methods. We demonstrate that these methods adversely impact accuracy and ignore the communication costs incurred when transmitting the activation from the device to the server. This paper proposes ActionFed-a communication efficient framework for DPFL to accelerate training on resource-constrained devices. ActionFed eliminates the transmission of the gradient by developing pretrained initialization of the DNN model on the device for the first time. This reduces the accuracy degradation seen in local loss-based methods. In addition, ActionFed proposes a novel replay buffer mechanism and implements a quantizationbased compression technique to reduce the transmission of the activation. It is experimentally demonstrated that ActionFed can reduce the communication cost by up to 15.77x and accelerates training by up to 3.87x when compared to vanilla DPFL.

Greedy Layerwise Learning Can Scale To Imagenet

Abstract: Shallow supervised 1-hidden layer neural networks have a number of favorable properties that make them easier to interpret, analyze, and optimize than their deep counterparts, but lack their representational power. Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks. Contrary to previous approaches using shallow networks, we focus on problems where deep learning is reported as critical for success. We thus study CNNs on image classification tasks using the large-scale ImageNet dataset and the CIFAR-10 dataset. Using a simple set of ideas for architecture and training we find that solving sequential 1-hidden-layer auxiliary problems lead to a CNN that exceeds AlexNet performance on ImageNet. Extending this training methodology to construct individual layers by solving 2-and-3-hidden layer auxiliary problems, we obtain an 11-layer network that exceeds several members of the VGG model family on ImageNet, and can train a VGG-11 model to the same accuracy as end-to-end learning. To our knowledge, this is the first competitive alternative to end-to-end training of CNNs that can scale to ImageNet. We illustrate several interesting properties of these models theoretically and conduct a range of experiments to study the properties this training induces on the intermediate layers.


We use 14 noise sources from ImageNet-C #REFR , a rich and diverse library commonly used for evaluating model robustness.

where loss weight β controls the trade-off between stego quality and secret recovery. Training for robustness.

Being robust to noises is a desirable property of both steganography and watermarking.

We insert a noise model between the image decoder G and the secret decoder D.

These noises can be split into 3 groups: differentiable including most additive and linear noises (e.g. brightness, saturation, contrast, ...), approximatable with differentiable transforms (e.g. jpeg compression #OTHEREFR ), non-differentiable (e.g. spatter).

To ensure gradient can be flown back to the preceding layers, we convert any non-differentiable noise n(.) to additive noise via

is treated as an additive constant (gradient disconnected from computation graph). Additionally, we apply multiple common noises (e.g.

contrast, brightness, jpeg compression) concurrently at a certain rate (p = 0.5), along with random individual noises.

Rosteals: Robust Steganography Using Autoencoder Latent Space

Abstract: Data hiding such as steganography and invisible watermarking has important applications in copyright protection, privacy-preserved communication and content provenance. Existing works often fall short in either preserving image quality, or robustness against perturbations or are too complex to train. We propose RoSteALS, a practical steganography technique leveraging frozen pretrained autoencoders to free the payload embedding from learning the distribution of cover images. RoSteALS has a lightweight secret encoder of just 300k parameters, is easy to train, has perfect secret recovery performance and comparable image quality on three benchmarks. Additionally, RoSteALS can be adapted for novel cover-less steganography applications in which the cover image can be sampled from noise or conditioned on text prompts via a denoising diffusion process. Our model and code are available at https://github.com/TuBui/RoSteALS.

Benchmarking Neural Network Robustness To Common Corruptions And Surface Variations

Abstract: In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize.


To perform the data generation, we use an opensource tool 2 to convert the 3D mesh semantic labels in Matterport3D #REFR into 194,400 pinhole images with semantic labels.

Then, every 18 semantic label pairs are concatenated via a corresponding rotation-translation matrix, yielding 10,800 panoramic semantic ground truth, which is referred to as 360FV-Matterport by us.

These panoramic semantic images are originally annotated with 40 object categories.

Because many of them are only a small percentage ( 0.1%), we merges some uncommon classes and maintains the 20 most common object categories: wall, floor, chair, door, table, picture, furniture, objects, window, sofa, bed, sink, stairs, ceiling, toilet, mirror, shower, bathtub, counter, and shelving.

For another front-view semantic segmentation dataset, Stan-ford2D3D #OTHEREFR , we keep the original object classes: beam, board, bookcase, ceiling, chair, clutter, column, door, floor, sofa, table, wall, window.

For the presented 360BEV-Stanford dataset, we follow the data split method of Fold-1 of the Stanford2D3D #OTHEREFR dataset.

360Bev: Panoramic Semantic Mapping For Indoor Bird'S-Eye View

Abstract: Seeing only a tiny part of the whole is not knowing the full circumstance. Bird's-eye-view (BEV) perception, a process of obtaining allocentric maps from egocentric views, is restricted when using a narrow Field of View (FoV) alone. In this work, mapping from 360°panoramas to BEV semantics, the 360BEV task, is established for the first time to achieve holistic representations of indoor scenes in a top-down view. Instead of relying on narrow-FoV image sequences, a panoramic image with depth information is sufficient to generate a holistic BEV semantic map. To benchmark 360BEV, we present two indoor datasets, 360BEV-Matterport and 360BEV-Stanford, both of which include egocentric panoramic images and semantic segmentation labels, as well as allocentric semantic maps. Besides delving deep into different mapping paradigms, we propose a dedicated solution for panoramic semantic mapping, namely 360Mapper. Through extensive experiments, our methods achieve 44.32% and 45.78% in mIoU on both datasets respectively, surpassing previous counterparts with gains of +7.60% and +9.70% in mIoU. 1

Matterport3D: Learning From Rgb-D Data In Indoor Environments

Abstract: Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms. However, existing datasets still cover only a limited number of views or a restricted scale of spaces. In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. Annotations are provided with surface reconstructions, camera poses, and 2D and 3D semantic segmentations. The precise global alignment and comprehensive, diverse panoramic set of views over entire buildings enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification.


We use Resnet50 #REFR as the secret decoder D, replacing the last fully connected layer to output a L-bit secrets.

In total, our secret encoder F has just over 300K parameters for a 100-bit secret length.

Interestingly, we empirically verify that conditioning F on both the cover image and secret (e.g. δ = F (x, s)) is not necessary (c.f. Figure 1 , see also Sup.Mat.).

The stego image is then constructed asx = G(z + δ) and regulated using a combination of pixel and perceptual losses;

L M SE = ||γ(x) − γ(x)|| 2 (2) L quality = L LP IP S (x, x) + αL M SE (3)

where γ(.) is a differentiable non-parametric mapping function from RGB to the more perceptually uniform YUV space; L LP IP S refers to the LPIPS loss #OTHEREFR commonly used as evaluator for image quality and α is a loss weight constant.

We use BCE to compute the bit recovery loss between the predicted and groundtruth secret,

where loss weight β controls the trade-off between stego quality and secret recovery. Training for robustness.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


Massive MIMO systems and high frequency communications at millimeter wave (mmWave) and sub-Thz bands are expected to play a key role in future sixth-generation (6G) networks #REFR .

These technologies are naturally supportive of each other, as massive MIMO using large transmit and receive antenna arrays facilitates generating highly focused beams that are essential for reliable communications at high frequencies, while short wavelength signaling enables packing MIMO configurations with a massive number of elements in a limited aperture.

However, implementing such massive MIMO transceivers gives rise to several challenges.

One of these core challenges is associated with the notable cost and power consumption of RF chains operating at high frequencies, which in conventional fully digital MIMO arrays separately connect each antenna element to the digital signal processing unit.

Hybrid beamforming is considered to be a leading solution for coping with the above challenge, enabling high frequency massive MIMO communications with a limited number of RF chains #OTHEREFR .

This is achieved by delegating part of the signal processing to the analog domain, thus, dividing the beamforming task into digital and analog counterparts.

Ai-Empowered Hybrid Mimo Beamforming

Abstract: Hybrid multiple-input multiple-output (MIMO) is an attractive technology for realizing extreme massive MIMO systems envisioned for future wireless communications in a scalable and power-efficient manner. However, the fact that hybrid MIMO systems implement part of their beamforming in analog and part in digital makes the optimization of their beampattern notably more challenging compared with conventional fully digital MIMO. Consequently, recent years have witnessed a growing interest in using data-aided artificial intelligence (AI) tools for hybrid beamforming design. This article reviews candidate strategies to leverage data to improve real-time hybrid beamforming design. We discuss the architectural constraints and characterize the core challenges associated with hybrid beamforming optimization. We then present how these challenges are treated via conventional optimization, and identify different AI-aided design approaches. These can be roughly divided into purely data-driven deep learning models and different forms of deep unfolding techniques for combining AI with classical optimization. We provide a systematic comparative study between existing approaches including both numerical evaluations and qualitative measures. We conclude by presenting future research opportunities associated with the incorporation of AI in hybrid MIMO systems.

Toward 6G Networks: Use Cases And Technologies

Abstract: Reliable data connectivity is vital for the ever increasingly intelligent, automated, and ubiquitous digital world. Mobile networks are the data highways and, in a fully connected, intelligent digital world, will need to connect everything, including people to vehicles, sensors, data, cloud resources, and even robotic agents. Fifth generation (5G) wireless networks, which are currently being deployed, offer significant advances beyond LTE, but may be unable to meet the full connectivity demands of the future digital society. Therefore, this article discusses technologies that will evolve wireless networks toward a sixth generation (6G) and which we consider as enablers for several potential 6G use cases. We provide a fullstack, system-level perspective on 6G scenarios and requirements, and select 6G technologies that can satisfy them either by improving the 5G design or by introducing completely new communication paradigms.

Massive MIMO systems
future sixth-generation (6G)


We also leverage a variation of Sentence-BERT #REFR architecture to learn sentence-level representations.

The unsupervised learning technique enabled the development of more robust NLP systems due to the abundance of textual data.

We need to optimize the proposed framework using unsupervised representation learning techniques to handle distributional shifts #OTHEREFR , #OTHEREFR in a robust manner so that it can handle new unseen app issues accurately.

The language models (e.g., GPT-2 #OTHEREFR , BERT #OTHEREFR , T5 #OTHEREFR ) are trained using unsupervised techniques by utilizing vast amounts of text data.

These models can capture both general semantic and syntactic information effectively, due to the size of the model and training dataset.

In this work, we employ T5, a pre-trained language model, as the backbone model in our framework.

Instead of BERT, we use a more powerful pre-trained language model T5 for learning sentence embeddings.

The T5 model follows transformers #OTHEREFR architecture, contains both encoder and decoder stacks, and has produced state-of-the-art results on several NLP benchmarks, including SuperGLUE. T5 Model.

Inspired by the idea of casting every NLP task as question answering #OTHEREFR , T5 treats every task as a text-totext problem.

With a text-to-text framework, virtually every task can be addressed with the same model, objective, training process, and decoding process.

T5 Adapts the original transformers architecture for multi-task learning, as it has both encoder and decoder stack as opposed to BERT (i.e., only encoder stack is more suitable for natural language understanding tasks) and GPT (i.e., only decoder stack is more appropriate for natural language generation tasks).

Proactive Prioritization Of App Issues Via Contrastive Learning

Abstract: Mobile app stores produce a tremendous amount of data in the form of user reviews, which is a huge source of user requirements and sentiments; such reviews allow app developers to proactively address issues in their apps. However, only a small number of reviews capture common issues and sentiments which creates a need for automatically identifying prominent reviews. Unfortunately, most existing work in text ranking and popularity prediction focuses on social contexts where other signals are available, which renders such works ineffective in the context of app reviews. In this work, we propose a new framework, PPrior, that enables proactive prioritization of app issues through identifying prominent reviews (ones predicted to receive a large number of votes in a given time window). Predicting highlyvoted reviews is challenging given that, unlike social posts, social network features of users are not available. Moreover, there is an issue of class imbalance, since a large number of user reviews receive little to no votes. PPrior employs a pre-trained T5 model and works in three phases. Phase one adapts the pretrained T5 model to the user reviews data in a self-supervised fashion. In phase two, we leverage contrastive training to learn a generic and task-independent representation of user reviews. Phase three uses radius neighbors classifier to make the final predictions. This phase also uses FAISS index for scalability and efficient search. To conduct extensive experiments, we acquired a large dataset of over 2.1 million user reviews from Google Play. Our experimental results demonstrate the effectiveness of the proposed framework when compared against several state-of-theart approaches. Moreover, the accuracy of PPrior in predicting prominent reviews is comparable to that of experienced app developers. Index Terms-App reviews, app analysis, automatic app issues prioritization.

Sentence-Bert: Sentence Embeddings Using Siamese Bert-Networks

Abstract: BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. ::: In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. ::: We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.

sentence-level representations
Sentence-BERT architecture


Consistent with the results reported in #REFR , the algebraic hybridization solver greatly outperforms the ADS solver on all cases considered.

In particular, hybrid Gauss-Seidel smoothing is used on the CPU, while 1 -Jacobi smoothing is used on the GPU.

This appears to have a more significant detrimental effect on the ADS solver; the iteration counts for the saddle-point solver are largely similar to the CPU case.

The saddle-point solver consistently achieves 9-15× speed-up compared with the LOR-ADS solver, with greater speed-up at higher orders.

At p = 6, the total solve time using the saddle-point solver using the GPU on one node of Lassen is over twice as fast as on the CPU using 4 nodes of Quartz.

For comparison, we also include CPU results for the matrix-based algebraic hybridization and ADS solvers in Table 4 ; GPU results are not reported because GPU-accelerated versions of these algorithms are not available.

Similarly, the LOR-ADS solver outperforms the ADS solver for p ≥ 2, as reported in #OTHEREFR , and consequently the saddle-point solver also compares favorably to matrix-based ADS for these test cases. The comparison with algebraic hybridization is more interesting.

The hybridization solver (which performs a conjugate gradient iteration directly on the hybridized Schur complement system) requires significantly fewer iterations than either ADS or the saddle-point solver.

For p = 2, hybridization results in the fastest overall runtime, and for p = 3, we obtained roughly equal runtimes with the hybridization and saddle-point solvers.

However, for p > 3, the time required to assemble and set up the hybridized solver becomes prohibitive, and the saddle-point solver results in significant speed-ups.

For p = 4, the saddle-point solver is almost twice as fast, and for p = 6, the saddle-point solver results in an almost 30× speedup.

Matrix-Free Gpu-Accelerated Saddle-Point Solvers For High-Order Problems In H(Div)

Abstract: This work describes the development of matrix-free GPU-accelerated solvers for high-order finite element problems in H(div). The solvers are applicable to grad-div and Darcy problems in saddle-point formulation, and have applications in radiation diffusion and porous media flow problems, among others. Using the interpolation-histopolation basis (cf. SIAM J. Sci. Comput., 45 (2023), A675-A702), efficient matrix-free preconditioners can be constructed for the (1, 1)-block and Schur complement of the block system. With these approximations, block-preconditioned MINRES converges in a number of iterations that is independent of the mesh size and polynomial degree. The approximate Schur complement takes the form of an M-matrix graph Laplacian, and therefore can be well-preconditioned by highly scalable algebraic multigrid methods. High-performance GPU-accelerated algorithms for all components of the solution algorithm are developed, discussed, and benchmarked. Numerical results are presented on a number of challenging test cases, including the "crooked pipe" grad-div problem, the SPE10 reservoir modeling benchmark problem, and a nonlinear radiation diffusion test case.

Algebraic Hybridization And Static Condensation With Application To Scalable H(Div) Preconditioning

Abstract: We propose a unified algebraic approach for the practical application and preconditioning of static condensation and hybridization, two popular techniques in finite element discretizations. We demo...


However, this enormous search space includes redundant expressions; the evolutionary process often generates unnecessarily large models with poor interpretability #REFR .

However, the inefficiency of GP evolutionary search approach highdimensional problems, which causes the above problem from the root, remains to be addressed.

Another issue is that the interpretability of GP solutions suffers substantially on high-dimensional problems.

GP often leads to very large expressions or chaining of nonlinear functions, which makes the expressions much harder to analyze #OTHEREFR .

This problem is related to "bloating" in GP (where trees grow unnecessarily large without a corresponding increase in fitness), which is a common side effect of the evolutionary-based search approach in GP.

The flexible representation of GP means it can theoretically discover the entire set of possible function forms.

Bloating is a common problem in GP-based SR and has been researched for decades.

Previous solutions to this problem include incorporating a parsimony measure to the fitness function #OTHEREFR , #OTHEREFR or utilising online program simplification #OTHEREFR . For instance, Kinzett et al.

#OTHEREFR explored the use of two online program simplification approaches in the evolution process: algebraic simplification #OTHEREFR and numerical simplification.

This helps to reduce bloating in the evolution of GP.

However, it needs predefined parameters, i.e., simplification rules and threshold.

Differentiable Genetic Programming For High-Dimensional Symbolic Regression

Abstract: Symbolic regression (SR) is the process of discovering hidden relationships from data with mathematical expressions, which is typically considered an effective way to reach interpretable machine learning. Genetic programming (GP) has been the dominator in solving various SR problems. However, as the scale of SR problems increases, GP often poorly demonstrates and cannot effectively address the real-world high-dimensional problems. This limitation is mainly caused by the stochastic evolutionary nature of traditional GP in constructing the proper trees. In this paper, we propose a differentiable approach named DGP to construct GP trees towards high-dimensional SR for the first time. Specifically, a new data structure called differentiable symbolic tree is proposed to relax the discrete structure to be continuous, thus a gradient-based optimizer can be easily presented for the efficient optimization. In addition, a sampling method is proposed to eliminate the discrepancy caused by the above relaxation for valid symbolic expressions. Furthermore, a diversification mechanism is introduced to promote the optimizer escaping from local optima for globally optimal expressions. With these designs, the proposed DGP method can efficiently search for the GP trees with higher performance, thus being capable of dealing with high-dimensional SR. To demonstrate the effectiveness of DGP, we conducted various experiments against the state of the arts based on both GP and deep neural networks. The experiment results reveal that DGP can outperform these chosen peer competitors on real-world high-dimensional regression benchmarks with dimensions varying from hundreds to thousands. In addition, on the challenging synthetic SR problems, the proposed DGP method can also achieve the best recovery rate even with different noisy levels. It is believed this work can facilitate SR being a powerful alternative to interpretable machine learning for a broader range of real-world problems.

Interaction-Transformation Evolutionary Algorithm For Symbolic Regression

Abstract: The Interaction-Transformation (IT) is a new representation for Symbolic Regression that restricts the search space into simpler, but expressive, function forms. This representation has the advantage of creating a smoother search space unlike the space generated by Expression Trees, the common representation used in Genetic Programming. This paper introduces an Evolutionary Algorithm capable of evolving a population of IT expressions supported only by the mutation operator. The results show that this representation is capable of finding better approximations to real-world data sets when compared to traditional approaches and a state-of-the-art Genetic Programming algorithm.

unnecessarily large models
evolutionary process


This is similar to findings in #REFR where they found ImageNet trained CNN's are biased towards texture.

To better understand model failures, we categorized each attribute into 11 categories with results shown in Figure 5 (left).

For example, "visibility" contains attributes like "dark" and "bright" while "expression" contains "smiley" and "sad". All models struggle when "visibility" related compositions are used.

The highest performance was within the "material" category with attributes such as "cloth" and "tin".

To compare backbone models, we average the image scores over CLIP backbone architectures in Figure 5 (middle).

Some noticeable patterns are that the CNN backbone models are better with "material", "pattern" and "texture" related compositions while ViT's are better at "color" and "shape".

Summary (1) Models struggle with compositionality but are better with those most associated with objects such as "materials".

(2) CNN based backbones may be better at recognizing texture and patterns while ViT backbones with color and shape.

Surprisingly, (3) these models are typically better at matching captions given the image rather than text.

Probing Conceptual Understanding Of Large Visual-Language Models

Abstract: We present a novel framework for probing and improving relational, compositional and contextual understanding of large visual-language models (V+L). While large V+L models have achieved success in various downstream tasks, it is not clear if they have a conceptual grasp of the content. We propose a novel benchmarking dataset for probing three aspects of content understanding. Our probes are grounded in cognitive science and help determine if a V+L model can, for example, determine if snow garnished with a man is implausible, or if it can identify beach furniture by knowing it is located on a beach. We have experimented with 5 well known models, such as CLIP and ViLT, and found that they mostly fail to demonstrate a conceptual understanding. That said, we find interesting insights such as cross-attention helps learning conceptual understandingWe use these insights to propose a new finetuning technique that rewards the three conceptual understanding measures we proposed. We hope that the presented benchmarks will help the community assess and improve the conceptual understanding capabilities of large V+L models. Visual-language (V+L) models are recently and greatly successful deep learning models that learn representations of image and text in a shared space. These representations are useful for downstream tasks like image classification, visual-question answering, image retrieval and more [42, 37, 27, 26, 39, 1]. However, for use in real-world applications, it is also vital that models "understand" rather than memorize to perform on more general tasks [24]. While large-language models have been shown to have a moderate amount of "theory of mind", as measured by conceptual consistency [29], V+L models have not

Imagenet-Trained Cnns Are Biased Towards Texture; Increasing Shape Bias Improves Accuracy And Robustness

Abstract: Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes. Some recent studies suggest a more important role of image textures. We here put these conflicting hypotheses to a quantitative test by evaluating CNNs and human observers on images with a texture-shape cue conflict. We show that ImageNet-trained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence and reveals fundamentally different classification strategies. We then demonstrate that the same standard architecture (ResNet-50) that learns a texture-based representation on ImageNet is able to learn a shape-based representation instead when trained on "Stylized-ImageNet", a stylized version of ImageNet. This provides a much better fit for human behavioural performance in our well-controlled psychophysical lab setting (nine experiments totalling 48,560 psychophysical trials across 97 observers) and comes with a number of unexpected emergent benefits such as improved object detection performance and previously unseen robustness towards a wide range of image distortions, highlighting advantages of a shape-based representation.