Noisy scene graph annotations in Action Genome #REFR increase the uncertainty of predicted scene graphs.

The high recall values in prior works suggest that they may have a tendency to overfit on popular predicate classes (e.g.

in front of / not looking at), without considering how the performances on rare classes (e.g. eating/wiping) are getting impacted #OTHEREFR .

Predicates lying in the tails often provide more informative depictions of underlying actions and activities in the video.

Thus, it is important to be able to measure a model's long-term performance not only on the frequently occurring relationships but also on the infrequently occurring ones.

Data imbalance is, however, not the only challenge in dynamic SGG. As shown in Fig. 2 Figure 2 .

factors, including noisy annotations, motion blur, temporal fluctuations of predictions, and a need to focus on only active objects that are involved in an action contribute to the bias in training dynamic SGG models #OTHEREFR .

As a result, the visual relationship predictions have high uncertainty, thereby increasing the challenge of dynamic SGG manyfold.

In this paper, we address these sources of bias in dynamic SGG and propose methods to compensate for them.

We identify missing annotations, multi-label mapping, and triplet (<subject−predicate−object>) variability (Fig 2) as labeling noise, which coupled with the inherent temporal fluctuations in a video can be attributed as data noise that can be modeled as the aleatoric uncertainty #OTHEREFR .

Another form of uncertainty called the epistemic uncertainty, relates to misleading model predictions due to a lack of sufficient observations #OTHEREFR and is more prevalent for long-tailed data #OTHEREFR .

Unbiased Scene Graph Generation In Videos

Abstract: The task of dynamic scene graph generation (SGG) from videos is complicated and challenging due to the inherent dynamics of a scene, temporal fluctuation of model predictions, and the long-tailed distribution of the visual relationships in addition to the already existing challenges in image-based SGG. Existing methods for dynamic SGG have primarily focused on capturing spatio-temporal context using complex architectures without addressing the challenges mentioned above, especially the long-tailed distribution of relationships. This often leads to the generation of biased scene graphs. To address these challenges, we introduce a new framework called TEMPURA: TEmporal consistency and Memory Prototype guided UnceRtainty Attenuation for unbiased dynamic SGG. TEMPURA employs object-level temporal consistencies via transformerbased sequence modeling, learns to synthesize unbiased relationship representations using memory-guided training, and attenuates the predictive uncertainty of visual relations using a Gaussian Mixture Model (GMM). Extensive experiments demonstrate that our method achieves significant (up to 10% in some cases) performance gain over existing methods highlighting its superiority in generating more unbiased scene graphs.

Action Genome: Actions As Composition Of Spatio-Temporal Scene Graphs

Abstract: Action recognition has typically treated actions and activities as monolithic events that occur in videos. However, there is evidence from Cognitive Science and Neuroscience that people actively encode activities into consistent hierarchical part structures. However in Computer Vision, few explorations on representations encoding event partonomies have been made. Inspired by evidence that the prototypical unit of an event is an action-object interaction, we introduce Action Genome, a representation that decomposes actions into spatio-temporal scene graphs. Action Genome captures changes between objects and their pairwise relationships while an action occurs. It contains 10K videos with 0.4M objects and 1.7M visual relationships annotated. With Action Genome, we extend an existing action recognition model by incorporating scene graphs as spatio-temporal feature banks to achieve better performance on the Charades dataset. Next, by decomposing and learning the temporal changes in visual relationships that result in an action, we demonstrate the utility of a hierarchical event decomposition by enabling few-shot action recognition, achieving 42.7% mAP using as few as 10 examples. Finally, we benchmark existing scene graph models on the new task of spatio-temporal scene graph prediction.


Instance segmentation is the task of detecting, segmenting, and classifying each object instance in an image #REFR .

Many biomedical applications require the detection and segmentation of individual cells in microscopy imagery #OTHEREFR .

For example, analyzing the cellular processes of living cells in time-lapse fluorescence microscopy (TFLM) experiments, requires accurate pixel-level segmentations of individual cells #OTHEREFR - #OTHEREFR .

Most applications require each cell to be segmented and identified as an unique entity or instance #OTHEREFR .

While powerful cell segmentation algorithms have been proposed recently (e.g.

#OTHEREFR , #OTHEREFR ), segmenting cells in microstructured environments remains challenging, due to the perceptual similarity of microstructurs and cells (cf. Fig. 1 ) #OTHEREFR , #OTHEREFR , #OTHEREFR .

The vast majority of current state-of-the-art segmentation algorithms utilize deep neural networks #OTHEREFR , #OTHEREFR - #OTHEREFR .

A key factor driving the development of deep learning-based segmentation algorithms is the widespread availability of pixel-wise annotated datasets.

Examples include Microsoft COCO #OTHEREFR , Cityscapes #OTHEREFR , ADE20K #OTHEREFR , and the 2018 Data Science Bowl dataset #OTHEREFR . While general cell segmentation datasets are available (e.g.

An Instance Segmentation Dataset Of Yeast Cells In Microstructures

Abstract: Extracting single-cell information from microscopy data requires accurate instance-wise segmentations. Obtaining pixel-wise segmentations from microscopy imagery remains a challenging task, especially with the added complexity of microstructured environments. This paper presents a novel dataset for segmenting yeast cells in microstructures. We offer pixel-wise instance segmentation labels for both cells and trap microstructures. In total, we release 493 densely annotated microscopy images. To facilitate a unified comparison between novel segmentation algorithms, we propose a standardized evaluation strategy for our dataset. The aim of the dataset and evaluation strategy is to facilitate the development of new cell segmentation approaches. The dataset is publicly available at https://christophreich1996.github.io/yeast_ in_microstructures_dataset/.

The Cityscapes Dataset For Semantic Urban Scene Understanding

Abstract: Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotations, 20 000 additional images have coarse annotations to enable methods that leverage large volumes of weakly-labeled data. Crucially, our effort exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity. Our accompanying empirical study provides an in-depth analysis of the dataset characteristics, as well as a performance evaluation of several state-of-the-art approaches based on our benchmark.


Game theory combined with reinforcement learning has been used to model human decision-making and determine an optimal action sequence for intelligent agents in #REFR .

Emerging vehicular technologies in automation and communication have generated new opportunities to enhance traffic safety and efficiency #OTHEREFR .

Prior research efforts have shown that in environments consisting solely of connected automated vehicles (CAVs), the traffic throughput and energy consumption can be significantly improved under different vehicle coordination strategies such as optimal control #OTHEREFR , #OTHEREFR , virtual platooning #OTHEREFR , model predictive control #OTHEREFR , and learning-based control #OTHEREFR .

It is expected that CAVs will gradually penetrate the market and interact with human-driven vehicles (HDVs) over the next several years.

The presence of HDVs, however, complicates vehicle coordination, given the existing limitations on predicting precise driving behavior.

Addressing coordination of CAVs in a mixed-traffic environment, where CAVs and HDVs co-exist, has attracted considerable attention recently.

Mixed-integer linear programming for centralized scheduling and coordinating CAVs interacting with HDVs was proposed in #OTHEREFR .

Other approaches include dynamic programming #OTHEREFR , statistical modeling of human uncertainties #OTHEREFR , and reachability analysis #OTHEREFR .

In mixed autonomy, traffic bottlenecks such as merging roadways and intersections pose significant challenges to the decision-making and control of CAVs.

A multi-agent reinforcement learning framework for CAVs was presented in #OTHEREFR for microscopic simulation where the human drivers are simulated by a car-following model.

A hierarchical on-ramp merging control strategy for CAVs was presented in #OTHEREFR , where an upper-level planner computes an expected merging position and a low-level trajectory controller guarantees constraints under uncertainties of HDVs.

Coordination For Connected Automated Vehicles At Merging Roadways In Mixed Traffic Environment

Abstract: In this paper, we present a two-level optimal control framework to address motion coordination of connected automated vehicles (CAVs) in the presence of human-driven vehicles (HDVs) in merging scenarios. Our framework combines an unconstrained trajectory solution of a low-level energyoptimal control problem with an upper-level optimization problem that yields the minimum travel time for CAVs. We predict the future trajectories of the HDVs using Newell's car-following model. To handle potential deviations of HDVs' actual behavior from the one predicted, we provide a risktriggered re-planning mechanism for the CAVs based on timeto-conflict. The effectiveness of the proposed control framework is demonstrated via simulations with heterogeneous human driving behaviors and via experiments in a scaled environment.

Modeling Cyber-Physical Human Systems Via An Interplay Between Reinforcement Learning And Game Theory

Abstract: Predicting the outcomes of cyber-physical systems with multiple human interactions is a challenging problem. This article reviews a game theoretical approach to address this issue, where reinforcement learning is employed to predict the time-extended interaction dynamics. We explain that the most attractive feature of the method is proposing a computationally feasible approach to simultaneously model multiple humans as decision makers, instead of determining the decision dynamics of the intelligent agent of interest and forcing the others to obey certain kinematic and dynamic constraints imposed by the environment. We present two recent exploitations of the method to model 1) unmanned aircraft integration into the National Airspace System and 2) highway traffic. We conclude the article by providing ongoing and future work about employing, improving and validating the method. We also provide related open problems and research opportunities.


When compared to the results presented here the clear advantage of #REFR is that it deals with deep networks, while we only consider single-layered networks.

Also, the kind of parameter initialization they privilege (standard Gaussian initialization with zero mean) leads to deviations as powers of n −1 , where n determines the width size.

In contrast, in the present work we take a simpler approach using the Edgeworth series technique in order to explicitly compute the corrections to Gaussianity in terms of an asymptotic series in n − 1 2 , with coefficients determined by the statistics of parameter initialization.

The paper #OTHEREFR also concerns deep neural networks.

In that setting it is shown, in particular, that at initialization the difference between the infinite limit NTK ∞ and the finite width NTK (n) can be made arbitrarily small by increasing the width of all hidden layers.

They also show that after training the difference between the outputs of the finite and the corresponding infinite width networks can be made arbitrarily small by increasing the width.

Nonetheless, their bounds do not have an explicit quantitative dependence on the width of the layers and their results only apply to ReLu non-linearities; a situation that is largely fixed in our paper.

The results in #OTHEREFR provide various estimates, in terms of the width n, controlling the behavior of the NTK, and higher order generalization of this object, and of the outputs during training.

Once again their results apply to deep neural networks, but unfortunately the estimates they provide depend on increasing functions of training time t, a shortfall that, in particular, does not allow to control the t → ∞ limit; nonetheless the estimates provided in #OTHEREFR are sufficient to show that outputs of training data can be made arbitrarily close to their labels, by appropriately increasing the width n.

In the, already mentioned, extensive work carried out in #OTHEREFR one can also find results concerning the behavior of outputs of wide neural networks, where it is shown that after training the difference between outputs and labels can be controlled up to an error of the order of n −2 , where n determines the width of the networks.

Concerning the behavior of outputs during training the results in #OTHEREFR are, to the best of our knowledge, the ones with biggest overlap with the ones presented here.

Wide Neural Networks: From Non-Gaussian Random Fields At Initialization To The Ntk Geometry Of Training

Abstract: Recent developments in applications of artificial neural networks with over n = 10 14 parameters make it extremely important to study the large n behaviour of such networks. Most works studying wide neural networks have focused on the infinite width n → +∞ limit of such networks and have shown that, at initialization, they correspond to Gaussian processes [12, 20]. In this work we will study their behavior for large, but finite n. Our main contributions are the following: • The computation of the corrections to Gaussianity in terms of an asymptotic series in n − 1 2. The coefficients in this expansion are determined by the statistics of parameter initialization and by the activation function. • Controlling the evolution of the outputs of finite width n networks, during training, by computing deviations from the limiting infinite width case (in which the network evolves through a linear flow). This improves previous estimates [1, 16, 10, 18] and along the way we also obtain sharper decay rates for the (finite width) NTK in terms of n, valid during the entire training procedure. As a corollary, we also prove that, with arbitrarily high probability, the training of sufficiently wide neural networks converges to a global minimum of the corresponding quadratic loss function. • Estimating how the deviations from Gaussianity evolve with training in terms of n. In particular, using a certain metric in the space of measures we find that, along training, the resulting measure is within n − 1 2 (log n) 1+ of the time dependent Gaussian process corresponding to the infinite width network (which is explicitly given by precomposing the initial Gaussian process with the linear flow corresponding to training in the infinite width limit). Contents

On Exact Computation With An Infinitely Wide Neural Net

Abstract: How well does a classic deep net architecture like AlexNet or VGG19 classify on a standard dataset such as CIFAR-10 when its width --- namely, number of channels in convolutional layers, and number of nodes in fully-connected internal layers --- is allowed to increase to infinity? Such questions have come to the forefront in the quest to theoretically understand deep learning and its mysteries about optimization and generalization. They also connect deep learning to notions such as Gaussian processes and kernels. A recent paper [Jacot et al., 2018] introduced the Neural Tangent Kernel (NTK) which captures the behavior of fully-connected deep nets in the infinite width limit trained by gradient descent; this object was implicit in some other recent papers. An attraction of such ideas is that a pure kernel-based method is used to capture the power of a fully-trained deep net of infinite width. ::: The current paper gives the first efficient exact algorithm for computing the extension of NTK to convolutional neural nets, which we call Convolutional NTK (CNTK), as well as an efficient GPU implementation of this algorithm. This results in a significant new benchmark for the performance of a pure kernel-based method on CIFAR-10, being $10\%$ higher than the methods reported in [Novak et al., 2019], and only $6\%$ lower than the performance of the corresponding finite deep net architecture (once batch normalization, etc. are turned off). Theoretically, we also give the first non-asymptotic proof showing that a fully-trained sufficiently wide net is indeed equivalent to the kernel regression predictor using NTK.


The pre-trained language model BERT #REFR has gained significant prominence in recent times for its demonstration of efficacy in comparison to traditional one-way language models and shallow splicing methods.

The masked language model (MLM) approach utilized by BERT has been shown to provide more semantic information within generated language representations.

Our work harnesses the capabilities of BERT by incorporating sentence vectors as inputs to the model, yielding contextual features that serve as word embeddings.

The representation of each token in sentence S is obtained through the amalgamation of its respective token embedding, segment embedding, and position embedding.

This is achieved by utilizing the sentence encoding of the form [CLS, E S , SEP ], where CLS signifies the initiation of the sentence and SEP serves as an indicator of the sentence's conclusion.

Tc-Gat: Graph Attention Network For Temporal Causality Discovery

Abstract: The present study explores the intricacies of causal relationship extraction, a vital component in the pursuit of causality knowledge. Causality is frequently intertwined with temporal elements, as the progression from cause to effect is not instantaneous but rather ensconced in a temporal dimension. Thus, the extraction of temporal causality holds paramount significance in the field. In light of this, we propose a method for extracting causality from the text that integrates both temporal and causal relations, with a particular focus on the time aspect. To this end, we first compile a dataset that encompasses temporal relationships. Subsequently, we present a novel model, TC-GAT, which employs a graph attention mechanism to assign weights to the temporal relationships and leverages a causal knowledge graph to determine the adjacency matrix. Additionally, we implement an equilibrium mechanism to regulate the interplay between temporal and causal relations. Our experiments demonstrate that our proposed method significantly surpasses baseline models in the task of causality extraction.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


To resolve this issue, #REFR recently proposed a method to jointly learn a stable uncontrolled dynamics model with a Lyapunov function.

However, polynomial approximation can be a significant restriction on the class of function approximators used.

Related Work To this end, there has been substantial growth in literature on data-driven learning of stability certificates for dynamical systems, particularly those modeled with complex parametric representations.

One of the earliest of such works fits a Lyapunov function for a known uncontrolled dynamical system by penalizing violations of the corresponding Lyapunov decrease condition #OTHEREFR .

Learning certificate functions such as Lyapunov, barrier, and contraction metric functions for dynamical systems with sampled point penalties or constraints on stability violations is a ubiquitous theme in the literature #OTHEREFR .

However, when coupled with regression on unknown dynamics, such approaches do not even guarantee the learned model is stable.

It guarantees stability of the learned dynamics model by restricting it to the stable halfspace described by the Lyapunov decrease condition. arXiv:2303.03157v1 [eess.SY] 6 Mar 2023

However, its naive extension to controlled dynamics would restrict the learned model to be stabilizable by any control input, thereby hindering its ability to model general controlled nonlinear systems.

For controlled dynamics, the paradigm of sampled pointwise constraints persists. #OTHEREFR

(2020) jointly learn a dynamics model and certificate to regularize the dynamics model to perform well over long time horizons with sampled linear matrix inequality (LMI) constraints, yet they do not learn any specific controller. #OTHEREFR

(2020) assume the dynamics are known, and jointly learn a controller and a contraction metric certifying the stability of the closedloop system with loss terms corresponding to sampled point violations of a stability inequality. For Lyapunov-based approaches, #OTHEREFR

Data-Driven Control With Inherent Lyapunov Stability

Abstract: Recent advances in learning-based control leverage deep function approximators, such as neural networks, to model the evolution of controlled dynamical systems over time. However, the problem of learning a dynamics model and a stabilizing controller persists, since the synthesis of a stabilizing feedback law for known nonlinear systems is a difficult task, let alone for complex parametric representations that must be fit to data. To this end, we propose a method for jointly learning parametric representations of a nonlinear dynamics model and a stabilizing controller from data. To do this, our approach simultaneously learns a parametric Lyapunov function which intrinsically constrains the dynamics model to be stabilizable by the learned controller. In addition to the stabilizability of the learned dynamics guaranteed by our novel construction, we show that the learned controller stabilizes the true dynamics under certain assumptions on the fidelity of the learned dynamics. Finally, we demonstrate the efficacy of our method on a variety of simulated nonlinear dynamical systems.

Learning Stable Deep Dynamics Models

Abstract: Deep networks are commonly used to model dynamical systems, predicting how the state of a system will evolve over time (either autonomously or in response to control inputs). Despite the predictive power of these systems, it has been difficult to make formal claims about the basic properties of the learned systems. In this paper, we propose an approach for learning dynamical systems that are guaranteed to be stable over the entire state space. The approach works by jointly learning a dynamics model and Lyapunov function that guarantees non-expansiveness of the dynamics under the learned Lyapunov function. We show that such learning systems are able to model simple dynamical systems and can be combined with additional deep generative models to learn complex dynamics, such as video textures, in a fully end-to-end fashion.


It is a simplified version of MAML #REFR that fine-tunes the last few layers which can achieve comparable performance as the original algorithm.

We train the embedding network and classifier jointly with a mini-dataset during the meta-learning phase, which is the same as embedding learning.

After obtaining the embedding, we further optimize the classifier with a few samples while testing.

In contrast to embedding learning, we believe fine-tuning can provide better flexibility and granularity, which is more suitable for a dynamic system.

Different from meta-learning, we only retrain the simple classifier instead of the whole structure, which can reduce the computation, enabling a faster adaption to new tasks.

This design is inspired by #OTHEREFR , which shows that the effectiveness of meta-learning is mainly due to feature reuse.

Simwisense: Simultaneous Multi-Subject Activity Classification Through Wi-Fi Signals

Abstract: Recent advances in Wi-Fi sensing have ushered in a plethora of pervasive applications in home surveillance, remote healthcare, road safety, and home entertainment, among others. Most of the existing works are limited to the activity classification of a single human subject at a given time. Conversely, a more realistic scenario is to achieve simultaneous, multi-subject activity classification. The first key challenge in that context is that the number of classes grows exponentially with the number of subjects and activities. Moreover, it is known that Wi-Fi sensing systems struggle to adapt to new environments and subjects. To address both issues, we propose SiMWiSense, the first framework for simultaneous multi-subject activity classification based on Wi-Fi that generalizes to multiple environments and subjects. We address the scalability issue by using the Channel State Information (CSI) computed from the device positioned closest to the subject. We experimentally prove this intuition by confirming that the best accuracy is experienced when the CSI computed by the transceiver positioned closest to the subject is used for classification. To address the generalization issue, we develop a brand-new few-shot learning algorithm named Feature Reusable Embedding Learning (FREL). Through an extensive data collection campaign in 3 different environments and 3 subjects performing 20 different activities simultaneously, we demonstrate that SiMWiSense achieves classification accuracy of up to 97%, while FREL improves the accuracy by 85% in comparison to a traditional Convolutional Neural Network (CNN) and up to 20% when compared to the state-of-the-art few-shot embedding learning (FSEL), by using only 15 seconds of additional data for each class. For reproducibility purposes, we share our 1TB dataset and code repository 1 [1].

Model-Agnostic Meta-Learning For Fast Adaptation Of Deep Networks

Abstract: We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning problems, including classification, regression, and reinforcement learning. The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. In our approach, the parameters of the model are explicitly trained such that a small number of gradient steps with a small amount of training data from a new task will produce good generalization performance on that task. In effect, our method trains the model to be easy to fine-tune. We demonstrate that this approach leads to state-of-the-art performance on two few-shot image classification benchmarks, produces good results on few-shot regression, and accelerates fine-tuning for policy gradient reinforcement learning with neural network policies.


Probing methods #REFR ) train an external model -a probe -to predict some properties of interest from the latent representations.

(2018) attempt to intervene in an unsupervised way on the hidden space by constructing several even-spaced Variational Autoencoders (VAEs) throughout a CNN, but they only train with a reconstruction loss instead of explicitly optimizing for impact.

To further investigate causal effects of the features learned from probing, #OTHEREFR

(2021) assess the influence of a causal intervention by removing a feature.

However, subsequent work #OTHEREFR shows that such methods generalize poorly to unseen samples.

Moreover, as #OTHEREFR points out, the disconnect between the probing model and the original model may result in the properties not being utilized in the original model's prediction task.

Causal Mediation Analysis (CMA) #OTHEREFR measures the change in an output following a counterfactual intervention in an intermediate variable, or mediator. NLP According to Feder et al.

Explaining Language Models' Predictions With High-Impact Concepts

Abstract: The emergence of large-scale pretrained language models has posed unprecedented challenges in deriving explanations of why the model has made some predictions. Stemmed from the compositional nature of languages, spurious correlations have further undermined the trustworthiness of NLP systems, leading to unreliable model explanations that are merely correlated with the output predictions. To encourage fairness and transparency, there exists an urgent demand for reliable explanations that allow users to consistently understand the model's behavior. In this work, we propose a complete framework for extending conceptbased interpretability methods to NLP. Specifically, we propose a post-hoc interpretability method for extracting predictive high-level features (concepts) from the pretrained model's hidden layer activations. We optimize for features whose existence causes the output predictions to change substantially, i.e. generates a high impact. Moreover, we devise several evaluation metrics that can be universally applied. Extensive experiments on real and synthetic tasks demonstrate that our method achieves superior results on predictive impact, usability, and faithfulness compared to the baselines.

What You Can Cram Into A Single Vector: Probing Sentence Embeddings For Linguistic Properties

Abstract: Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing. "Downstream" tasks, often based on sentence classification, are commonly used to evaluate the quality of sentence representations. The complexity of the tasks makes it however difficult to infer what kind of information is present in the representations. We introduce here 10 probing tasks designed to capture simple linguistic features of sentences, and we use them to study embeddings generated by three different encoders trained in eight distinct ways, uncovering intriguing properties of both encoders and training methods.


More recently, the Differentiable Inductive Logic Programming ( ILP) approach #REFR was proposed, addressing cycles through negation.

Other exceptions would then be added as needed without changing the original rule.

Unfortunately, this approach does not adapt well to exceptions to the exceptions such as an abnormal penguins (a hypothetical super-penguin that is capable of flying).

At present, there is a tension between the above attempts to formalizing non-monotonicity and large-scale data-driven approaches based on neural networks and natural language that are efficient but lack any formalization.

In this paper we seek to investigate approaches to solving PET and other simple examples that can be formalized but that work using the same tools as the large-scale network models.

Work has been conducted to formalize NMR in neural networks starting with the Connectionist Inductive Learning and Logic Programming System (CILP) #OTHEREFR , later developed into a system for statistical relational learning.

Probabilistic approaches have also been developed which can implement a form of nonmonotonicity or at least avoid the problems of classical logic by assigning probabilities to beliefs expressed as Horn clauses, e.g. DeepProbLog #OTHEREFR .

In this paper, rather than mapping symbolic representations into neural networks and vice-versa, we are interested in the interplay between learning and reasoning as part of a curriculum.

We focus on the Logic Tensor Network (LTN) #OTHEREFR because it is a highly modular NeSy framework applicable in principle to any underlying neural network model and based on the canonical, highly expressive FOL language.

Additionally, the LTN has shown promise for learning in continual mode #OTHEREFR .

The LTN relies on two main ideas, the grounding of predicates and logical axioms into vectors and Real Logic which maps the satisfiability of the logical axioms to a real number in the interval {0,1} thus enabling viewing satisfiability as optimization.

Continual Reasoning: Non-Monotonic Reasoning In Neurosymbolic Ai Using Continual Learning

Abstract: Despite the extensive investment and impressive recent progress at reasoning by similarity, deep learning continues to struggle with more complex forms of reasoning such as non-monotonic and commonsense reasoning. Non-monotonicity is a property of non-classical reasoning typically seen in commonsense reasoning, whereby a reasoning system is allowed (differently from classical logic) to jump to conclusions which may be retracted later, when new information becomes available. Neural-symbolic systems such as Logic Tensor Networks (LTN) have been shown to be effective at enabling deep neural networks to achieve reasoning capabilities. In this paper, we show that by combining a neural-symbolic system with methods from continual learning, LTN can obtain a higher level of accuracy when addressing nonmonotonic reasoning tasks. Continual learning is added to LTNs by adopting a curriculum of learning from knowledge and data with recall. We call this process Continual Reasoning, a new methodology for the application of neural-symbolic systems to reasoning tasks. Continual Reasoning is applied to a prototypical non-monotonic reasoning problem as well as other reasoning examples. Experimentation is conducted to compare and analyze the effects that different curriculum choices may have on overall learning and reasoning results. Results indicate significant improvement on the prototypical nonmonotonic reasoning problem and a promising outlook for the proposed approach on statistical relational learning examples.

Learning Explanatory Rules From Noisy Data

Abstract: Artificial Neural Networks are powerful function approximators capable of modelling solutions to a wide variety of problems, both supervised and unsupervised. As their size and expressivity increases, so too does the variance of the model, yielding a nearly ubiquitous overfitting problem. Although mitigated by a variety of model regularisation methods, the common cure is to seek large amounts of training data---which is not necessarily easily obtained---that sufficiently approximates the data distribution of the domain we wish to test on. In contrast, logic programming methods such as Inductive Logic Programming offer an extremely data-efficient process by which models can be trained to reason on symbolic domains. However, these methods are unable to deal with the variety of domains neural networks can be applied to: they are not robust to noise in or mislabelling of inputs, and perhaps more importantly, cannot be applied to non-symbolic domains where the data is ambiguous, such as operating on raw pixels. In this paper, we propose a Differentiable Inductive Logic framework, which can not only solve tasks which traditional ILP systems are suited for, but shows a robustness to noise and error in the training data which ILP cannot cope with. Furthermore, as it is trained by backpropagation against a likelihood objective, it can be hybridised by connecting it with neural networks over ambiguous data in order to be applied to domains which ILP cannot address, while providing data efficiency and generalisation beyond what neural networks on their own can achieve.


Limoyo #REFR proposed a eyeto-hand self-calibration method, which estimates robot-sensor relation by aligning a contact-based point cloud generated by manipulator and vision-based point cloud generated by the sensor.

Hand-eye calibration techniques for 3-D sensor specifically were also put forward, whereas calibration rig with high manufacturing accuracy and known dimensions is a must in most of relevant literatures.

Typical instances include calibration board #OTHEREFR , standard sphere #OTHEREFR and free-form surface with design CAD model #OTHEREFR , #OTHEREFR .

In addition to the costly high-precision machining, these methods can be unapplicable due to unavailability and accuracy degeneration of calibration rig.

Method using common geometry like plane and its normal vector to make calibration rig dispensable was reported in #OTHEREFR .

Nevertheless, auxiliary tool is needed to measure points coordinates on the plane beforehand, which brings extra operational complexity.

Although experiments were conducted successfuly with two rectangular boxes and on-board hardware only, the contact-based point cloud generation using manipulator under impedance control is rather time consuming, and this method can not be applied facing SOR and simple plane, as the registration is realized by pair-wise ICP.

To the best of our knowledge, RegHEC is the first 3-D sensor hand-eye calibration technique that 1) uses arbitrary available object with little requirement(can be SOR or plane), 2) relies on 3-D measurements only, and 3) works in both eye-in-hand and eye-to-hand cases.

It returns the calibrated hand-eye relation together with multi-view point clouds registration, thus is more beneficial for robotic 3-D reconstruction than for other 3-D vision-guided tasks.

Reghec: Hand-Eye Calibration Via Simultaneous Multi-View Point Clouds Registration Of Arbitrary Object

Abstract: RegHEC is a registration-based hand-eye calibration technique with no need for accurate calibration rig but arbitrary available objects, applicable for both eye-in-hand and eye-to-hand cases. It tries to find the hand-eye relation which brings multi-view point clouds of arbitrary scene into simultaneous registration under a common reference frame. RegHEC first achieves initial alignment of multi-view point clouds via Bayesian optimization, where registration problem is modeled as a Gaussian process over hand-eye relation and the covariance function is modified to be compatible with distance metric in 3-D motion space SE(3), then passes the initial guess of hand-eye relation to an Anderson Accelerated ICP variant for later fine registration and accurate calibration. RegHEC has little requirement on calibration object, it is applicable with sphere, cone, cylinder and even simple plane, which can be quite challenging for correct point cloud registration and sensor motion estimation using existing methods. While suitable for most 3-D vision guided tasks, RegHEC is especially favorable for robotic 3-D reconstruction, as calibration and multi-view point clouds registration of reconstruction target are unified into a single process. Our technique is verified with extensive experiments using varieties of arbitrary objects and real hand-eye system. We release an open-source C++ implementation of RegHEC.

Self-Calibration Of Mobile Manipulator Kinematic And Sensor Extrinsic Parameters Through Contact-Based Interaction

Abstract: We present a novel approach for mobile manipulator self-calibration using contact information. Our method, based on point cloud registration, is applied to estimate the extrinsic transform between a fixed vision sensor mounted on a mobile base and an end effector. Beyond sensor calibration, we demonstrate that the method can be extended to include manipulator kinematic model parameters, which involves a non-rigid registration process. Our procedure uses on-board sensing exclusively and does not rely on any external measurement devices, fiducial markers, or calibration rigs. Further, it is fully automatic in the general case. We experimentally validate the proposed method on a custom mobile manipulator platform, and demonstrate centimetre-level post-calibration accuracy in positioning of the end effector using visual guidance only. We also discuss the stability properties of the registration algorithm, in order to determine the conditions under which calibration is possible.


A polar code of length N = 2 n is constructed by selecting a set A of indices i ∈ [0, N − 1] with high reliability #REFR .

The generator matrix of a polar code is formed by selecting a set of rows of G N .

We use A to denote the set of indices of these rows and C(A) to denote the linear code generated by the set of rows of G N indexed by A.

The characterization of the information set A for polar codes relies on the channel polarization theorem #OTHEREFR and the concept of synthetic channel reliability.

The indices in A are dedicated to information bits, while the rest of the synthetic channels with indices in A c [0, N − 1] \ A are used to transmit a known value, '0' by default, which are called frozen bits.

Regardless of the method we use for forming the set A for a polar code, every synthetic channel represented by index i, denoted by W i n where i ∈ A must be more reliable than any synthetic channels in A c .

Polar codes with the information set A can also be considered as monomial codes #OTHEREFR where the relation between the generating set of monomials I ⊂ M n discussed in Section II-B and the information set A is as follows:

∀f ∈ I, ∃ i ∈ A, where supp(i) = [n]\ ind(f ). (2)

Accordingly, we can define set A m−r equivalent to I r , collecting the row indices with | supp(i)| = m − r for i ∈ A m−r where we have m = n for polar codes while m is used for decreasing monomial codes in general throughout this paper.

On The Closed-Form Weight Enumeration Of Polar Codes: 1.5D-Weight Codewords

Abstract: The weight distribution of error correction codes is a critical determinant of their error-correcting performance, making enumeration of utmost importance. In the case of polar codes, the minimum weight wmin (which is equal to minimum distance d) is the only weight for which an explicit enumerator formula is currently available. Having closed-form weight enumerators for polar codewords with weights greater than the minimum weight not only simplifies the enumeration process but also provides valuable insights towards constructing better polar-like codes. In this paper, we contribute towards understanding the algebraic structure underlying higher weights by analyzing Minkowski sums of orbits. Our approach builds upon the lower triangular affine (LTA) group of decreasing monomial codes. Specifically, we propose a closed-form expression for the enumeration of codewords with weight 1.5 wmin. Our simulations demonstrate the potential for extending this method to higher weights.

Channel Polarization: A Method For Constructing Capacity-Achieving Codes For Symmetric Binary-Input Memoryless Channels

Abstract: A method is proposed, called channel polarization, to construct code sequences that achieve the symmetric capacity $I(W)$ of any given binary-input discrete memoryless channel (B-DMC) $W$. The symmetric capacity is the highest rate achievable subject to using the input letters of the channel with equal probability. Channel polarization refers to the fact that it is possible to synthesize, out of $N$ independent copies of a given B-DMC $W$, a second set of $N$ binary-input channels $\{W_N^{(i)}:1\le i\le N\}$ such that, as $N$ becomes large, the fraction of indices $i$ for which $I(W_N^{(i)})$ is near 1 approaches $I(W)$ and the fraction for which $I(W_N^{(i)})$ is near 0 approaches $1-I(W)$. The polarized channels $\{W_N^{(i)}\}$ are well-conditioned for channel coding: one need only send data at rate 1 through those with capacity near 1 and at rate 0 through the remaining. Codes constructed on the basis of this idea are called polar codes. The paper proves that, given any B-DMC $W$ with $I(W)>0$ and any target rate $R<I(W)$, there exists a sequence of polar codes $\{{\mathscr C}_n;n\ge 1\}$ such that ${\mathscr C}_n$ has block-length $N=2^n$, rate $\ge R$, and probability of block error under successive cancellation decoding bounded as $P_{e}(N,R) \le \bigoh(N^{-\frac14})$ independently of the code rate. This performance is achievable by encoders and decoders with complexity $O(N\log N)$ for each.


Specifically, the global branch utilizes the standard MoCo #REFR style CL method to discriminate global representations, while the local branch aligns the local semantics by enforcing assignment consistency between position-matched pairs over a set of prototypes to learn object-separable representations and avoid semantic incompleteness and misalignment.

The teacher and the student share the same architecture (CNN or ViT).

The parameters of the student are exponentially moving averaged (EMA) to the parameters of teacher.

The student maps the masked image to latent embedding, while the teacher encodes the augmented image to preserve semantic integrity and provide contrastive supervision to guide the student.

The MIM branch applies a MIM head to reconstruct the masked image using the latent embedding output by the student in both spatial and frequency domains.

The other two branches project the latent embedding to different feature spaces to avoid semantics confusion and leverage CL to learn separable representations by aligning the embedding of the student and teacher at both global and local scales.

After pre-training using CMID with a large amount of unlabeled data, only the student is used for downstream tasks to evaluate the model performance.

In the following subsections, we will provide a more detailed explanation of the different components of CMID.

Cmid: A Unified Self-Supervised Learning Framework For Remote Sensing Image Understanding

Abstract: Self-supervised learning (SSL) has gained widespread attention in the remote sensing (RS) and earth observation (EO) communities owing to its ability to learn task-agnostic representations without human-annotated labels. Nevertheless, most existing RS SSL methods are limited to learning either global semantic separable or local spatial perceptible representations. We argue that this learning strategy is suboptimal in the realm of RS, since the required representations for different RS downstream tasks are often varied and complex. In this study, we proposed a unified SSL framework that is better suited for RS images representation learning. The proposed SSL framework, Contrastive Mask Image Distillation (CMID), is capable of learning representations with both global semantic separability and local spatial perceptibility by combining contrastive learning (CL) with masked image modeling (MIM) in a self-distillation way. Furthermore, our CMID learning framework is architecture-agnostic, which is compatible with both convolutional neural networks (CNN) and vision transformers (ViT), allowing CMID to be easily adapted to a variety of deep learning (DL) applications for RS understanding. Comprehensive experiments have been carried out on four downstream tasks (i.e. scene classification, semantic segmentation, object-detection, and change detection) and the results show that models pre-trained using CMID achieve better performance than other state-of-the-art SSL methods on multiple downstream tasks. The code and pre-trained models will be made available at https://github.com/NJU-LHRS/official-CMID to facilitate SSL research and speed up the development of RS images DL applications.

Improved Baselines With Momentum Contrastive Learning

Abstract: Contrastive unsupervised learning has recently shown encouraging progress, e.g., in Momentum Contrast (MoCo) and SimCLR. In this note, we verify the effectiveness of two of SimCLR's design improvements by implementing them in the MoCo framework. With simple modifications to MoCo---namely, using an MLP projection head and more data augmentation---we establish stronger baselines that outperform SimCLR and do not require large training batches. We hope this will make state-of-the-art unsupervised learning research more accessible. Code will be made public.


Experiment Settings And Evaluation Metrics

To implement our model, we choose uncased BERT-base #REFR as the encoder on DocRED and set the embedding dimension to 768.

For CDR dataset, we pick up BioBERT-Base v1.1 #OTHEREFR , which re-trained the BERT-base-cased model on biomedical corpora. All hyper-parameters are tuned based on the development set.

Other parameters in the network are all obtained by random orthogonal initialization #OTHEREFR and updated during training.

For a fair comparison, we follow the same experimental settings from previous works.

We apply F 1 and Ign F 1 as the evaluation metrics on DocRED.

F 1 scores can be obtained by calculation through an online interface.

Document-Level Relation Extraction With Cross-Sentence Reasoning Graph

Abstract: Relation extraction (RE) has recently moved from the sentence-level to document-level, which requires aggregating document information and using entities and mentions for reasoning. Existing works put entity nodes and mention nodes with similar representations in a document-level graph, whose complex edges may incur redundant information. Furthermore, existing studies only focus on entity-level reasoning paths without considering global interactions among entities cross-sentence. To these ends, we propose a novel document-level RE model with a GRaph information Aggregation and Cross-sentence Reasoning network (GRACR). Specifically, a simplified document-level graph is constructed to model the semantic information of all mentions and sentences in a document, and an entity-level graph is designed to explore relations of long-distance crosssentence entity pairs. Experimental results show that GRACR achieves excellent performance on two public datasets of document-level RE. It is especially effective in extracting potential relations of cross-sentence entity pairs. Our code is available at https://github.com/UESTC-LHF/GRACR.


Note that our knowledge graph has many entities with relatively sparse connections, making it hard to get metrics as good as those from benchmarking #REFR .

Still, the relative performance between the three models is similar to that observed in Table 8 .

This suggests that our comparison results are consistent even though the knowledge graph grows over time.

Through comparison between the embedding models, we determine that TransE performs well for the task at hand.

The metrics for the TransE model are sufficient for good prediction, as validated by our experiments in Section 5.

We therefore selected TransE as the embedding model for the prediction of unknown associations.

Fine-tuning training parameters of the embedding model may help further improve these metrics.

Last, we investigate the influence of optimizing the knowledge graph under the TransE model, specifically removing unconnected CPE and CVE entities. The results are shown in Table 10 .

After optimizing the knowledge graph, the MRR increases from 0.232 to 0.290.

The Hist@N scores also increase, indicating that positive triples are more likely to be found in top-N results.

We conclude that removing unconnected CPE and CVE entries indeed improves the quality of our knowledge graph.

Uncovering Cwe-Cve-Cpe Relations With Threat Knowledge Graphs

Abstract: Security assessment relies on public information about products, vulnerabilities, and weaknesses. So far, databases in these categories have rarely been analyzed in combination. Yet, doing so could help predict unreported vulnerabilities and identify common threat patterns. In this paper, we propose a methodology for producing and optimizing a knowledge graph that aggregates knowledge from common threat databases (CVE, CWE, and CPE). We apply the threat knowledge graph to predict associations between threat databases, specifically between products, vulnerabilities, and weaknesses. We evaluate the prediction performance both in closed world with associations from the knowledge graph, and in open world with associations revealed afterward. Using rank-based metrics (i.e., Mean Rank, Mean Reciprocal Rank, and Hits@N scores), we demonstrate the ability of the threat knowledge graph to uncover many associations that are currently unknown but will be revealed in the future, which remains useful over different time periods. We propose approaches to optimize the knowledge graph, and show that they indeed help in further uncovering associations. CCS Concepts: • Security and privacy → Software security engineering; • Computing methodologies → Knowledge representation and reasoning.

Realistic Re-Evaluation Of Knowledge Graph Completion Methods: An Experimental Study

Abstract: In the active research area of employing embedding models for knowledge graph completion, particularly for the task of link prediction, most prior studies used two benchmark datasets FB15k and WN18 in evaluating such models. Most triples in these and other datasets in such studies belong to reverse and duplicate relations which exhibit high data redundancy due to semantic duplication, correlation or data incompleteness. This is a case of excessive data leakage---a model is trained using features that otherwise would not be available when the model needs to be applied for real prediction. There are also Cartesian product relations for which every triple formed by the Cartesian product of applicable subjects and objects is a true fact. Link prediction on the aforementioned relations is easy and can be achieved with even better accuracy using straightforward rules instead of sophisticated embedding models. A more fundamental defect of these models is that the link prediction scenario, given such data, is non-existent in the real-world. This paper is the first systematic study with the main objective of assessing the true effectiveness of embedding models when the unrealistic triples are removed. Our experiment results show these models are much less accurate than what we used to perceive. Their poor accuracy renders link prediction a task without truly effective automated solution. Hence, we call for re-investigation of possible effective approaches.


A. Detection Model Degradation Under Noise

The severity of the noise impact, from greatest to least, is Gaussian N oise > F rost N oise > M otion Blur #REFR In most cases, for the same class and interference level of noise, the Depth model has higher detection accuracy.

We list the mAP of the proposed model under different noise settings.

Among them, the prediction accuracy of the RGB model shows a nonlinear decrease for the three types of noise, with mAP decreasing from 76.34 to 0.28(Gaussian), 6.30(Motion), and 7.23(Frost), respectively.

Compared with the detection results of clean data, the accuracy of all 15 experiments decreases.

Similarly, the prediction accuracy of the Depth model also decreases nonlinearly with increasing noise intensity, with mAP decreasing from 76.34 to 1.69(Gaussian), 31.62(Motion), and 24.86(Frost), respectively. The accuracy decreases in 14/15 experiments.

In addition, the sensitivity of both modal data to these noises is consistent.

In all 15 sets of experiments, the Depth model outperformed the RGB model in terms of accuracy, which is the same as the comparison results on clean data.

The average accuracy increments are, 7.74+ for Car, 20.57+ for Pedestrian, 22.26+ for Cyclist, and 16.86+ for mAP.

Informative Data Selection With Uncertainty For Multi-Modal Object Detection

Abstract: Noise has always been nonnegligible trouble in object detection by creating confusion in model reasoning, thereby reducing the informativeness of the data. It can lead to inaccurate recognition due to the shift in the observed pattern, that requires a robust generalization of the models. To implement a general vision model, we need to develop deep learning models that can adaptively select valid information from multi-modal data. This is mainly based on two reasons. Multi-modal learning can break through the inherent defects of single-modal data, and adaptive information selection can reduce chaos in multi-modal data. To tackle this problem, we propose a universal uncertainty-aware multi-modal fusion model. It adopts a multi-pipeline loosely coupled architecture to combine the features and results from point clouds and images. To quantify the correlation in multimodal information, we model the uncertainty, as the inverse of data information, in different modalities and embed it in the bounding box generation. In this way, our model reduces the randomness in fusion and generates reliable output. Moreover, we conducted a completed investigation on the KITTI 2D object detection dataset and its derived dirty data. Our fusion model is proven to resist severe noise interference like Gaussian, motion blur, and frost, with only slight degradation. The experiment results demonstrate the benefits of our adaptive fusion. Our analysis on the robustness of multi-modal fusion will provide further insights for future research.

Nuscenes: A Multimodal Dataset For Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image-based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first published dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online at this http URL.


SuperGlue #REFR uses a module based on attention graphs that alternately stacks 'self-attention' and 'cross-attention' layers.

where W s E ∈ R d×d is a learned projection matrix.

3) Attention mechanism: A key point of recent descriptors of points cloud is the introduction of an attention mechanism that highlights some features dynamically.

The former links all the nodes of a point cloud to each other, while the latter links each point of set X to all points of set Y .

Contrary to SuperGlue #OTHEREFR or MDGAT #OTHEREFR , which compute the attention weights on the encoding vectors, some methods propose adding information on local inter-points geometry at the entry of the mechanism.

For instance, #OTHEREFR associates the 3D coordinates of each point with the descriptor, while GeoTransformer #OTHEREFR proposes to use the distances and angles between each point and its k nearest neighbours.

Alternatively, in our approach, we propose the use of four attention heads with geometric self-attention inside each set X and Y integrating the associated normals embeddings e X and e Y , respectively, followed by cross-attention between the two sets of points and then alternate between them for L times.

4) Self-attention: This type of layer predicts an attentionbased featuref i for each point of a point cloud (X or Y ), paying attention to all the other points of the same cloud.

Rocnet: 3D Robust Registration Of Point-Clouds Using Deep Learning

Abstract: This paper introduces a new method for 3D point cloud registration based on deep learning. The architecture is composed of three distinct blocs: (i) an encoder composed of a convolutional graph-based descriptor that encodes the immediate neighbourhood of each point and an attention mechanism that encodes the variations of the surface normals. Such descriptors are refined by highlighting attention between the points of the same set and then between the points of the two sets. (ii) a matching process that estimates a matrix of correspondences using the Sinkhorn algorithm. (iii) Finally, the rigid transformation between the two point clouds is calculated by RANSAC using the K c best scores from the correspondence matrix. We conduct experiments on the ModelNet40 dataset, and our proposed architecture shows very promising results, outperforming state-of-the-art methods in most of the simulated configurations, including partial overlap and data augmentation with Gaussian noise.

Superglue: Learning Feature Matching With Graph Neural Networks

Abstract: This paper introduces SuperGlue, a neural network that matches two sets of local features by jointly finding correspondences and rejecting non-matchable points. Assignments are estimated by solving a differentiable optimal transport problem, whose costs are predicted by a graph neural network. We introduce a flexible context aggregation mechanism based on attention, enabling SuperGlue to reason about the underlying 3D scene and feature assignments jointly. Compared to traditional, hand-designed heuristics, our technique learns priors over geometric transformations and regularities of the 3D world through end-to-end training from image pairs. SuperGlue outperforms other learned approaches and achieves state-of-the-art results on the task of pose estimation in challenging real-world indoor and outdoor environments. The proposed method performs matching in real-time on a modern GPU and can be readily integrated into modern SfM or SLAM systems. The code and trained weights are publicly available at https://github.com/magicleap/SuperGluePretrainedNetwork.


Federated Learning (FL) is a well-established training algorithm that addresses this by allowing a model to be trained decentrally without physically sharing the data but instead sharing the model information only #REFR .

Data analytics hold the key to ensuring food security and sustainability.

Machine learning has been widely adapted to provide technical solutions to analytical problems in agriculture and food sectors, such as crop yield prediction #OTHEREFR , consumption demand forecasting #OTHEREFR , crop and disease detection #OTHEREFR , quality control and intelligent scheduling #OTHEREFR , and several others.

Typically, building such statistical models requires large amounts of data collected from various sources, i.e., different farms, supply chains, and other stakeholders.

However, individually, they may not have adequate data to train competent machine learning models for the tasks.

While combining their data into a centralized silo may improve data quality, collecting it may be challenging due to commercially sensitive information and reputational risks #OTHEREFR .

Each participating device (referred to as a client) participates in training in an isolated environment and is coordinated by the central server.

As a result, FL allows models to be collaboratively trained on large datasets while preserving data privacy, and this can be leveraged within the agri-food industry when training on local dataset alone is insufficient.

However, with the progressive improvements in deep learning models, their number of parameters have increased exponentially #OTHEREFR .

Since FL requires the transmission of the model between the server and the clients in each round, the communication cost often becomes a bottleneck #OTHEREFR .

Large models also make edge device deployment challenging as they consume more memory footprint and computational power.

Model Pruning Enables Localized And Efficient Federated Learning For Yield Forecasting And Data Sharing

Abstract: Federated Learning (FL) presents a decentralized approach to model training in the agrifood sector and offers the potential for improved machine learning performance, while ensuring the safety and privacy of individual farms or data silos. However, the conventional FL approach has two major limitations. First, the heterogeneous data on individual silos can cause the global model to perform well for some clients but not all, as the update direction on some clients may hinder others after they are aggregated. Second, it is lacking with respect to the efficiency perspective concerning communication costs during FL and large model sizes. This paper proposes a new technical solution that utilizes network pruning on client models and aggregates the pruned models. This method enables local models to be tailored to their respective data distribution and mitigate the data heterogeneity present in agri-food data. Moreover, it allows for more compact models that consume less data during transmission. We experiment with a soybean yield forecasting dataset and find that this approach can improve inference performance by 15.5% to 20% compared to FedAvg, while reducing local model sizes by up to 84% and the data volume communicated between the clients and the server by 57.1% to 64.7%.

Communication-Efficient Learning Of Deep Networks From Decentralized Data

Abstract: Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.


In axis-aligned object detection, the output rectangles have four degrees of freedom, which are aligned with the image axes #REFR .

Rotated object detection requires the model to predict minimum area rectangles with five degrees of freedom, namely rotated bounding boxes, enclosing objects of interests #OTHEREFR .

The rotated boxes occupy a much smaller area when estimating the location status of diagonally positioned elongated objects as shown in Figure 1.

Rotated detection is strictly superior to axis-aligned detection as there is less background within the box, and the orientation can potentially convey object pose information.

However, there are only a handful of datasets with rotated annotations #OTHEREFR comparing to large number of readily available large-scale axis-aligned datasets #OTHEREFR .

A potential contributor to this phenomenon is the ease of annotating axis-aligned boxes by simple click-and-drag with current labelling tools.

Rotated boxes require much more effort to tightly enclose the objects with an extra degree of freedom.

Knowledge Combination To Learn Rotated Detection Without Rotated Annotation

Abstract: Rotated bounding boxes drastically reduce output ambiguity of elongated objects, making it superior to axis-aligned bounding boxes. Despite the effectiveness, rotated detectors are not widely employed. Annotating rotated bounding boxes is such a laborious process that they are not provided in many detection datasets where axis-aligned annotations are used instead. In this paper, we propose a framework that allows the model to predict precise rotated boxes only requiring cheaper axis-aligned annotation of the target dataset 1. To achieve this, we leverage the fact that neural networks are capable of learning richer representation of the target domain than what is utilized by the task. The under-utilized representation can be exploited to address a more detailed task. Our framework combines task knowledge of an out-ofdomain source dataset with stronger annotation and domain knowledge of the target dataset with weaker annotation. A novel assignment process and projection loss are used to enable the co-training on the source and target datasets. As a result, the model is able to solve the more detailed task in the target domain, without additional computation overhead during inference. We extensively evaluate the method on various target datasets including fresh-produce dataset, HRSC2016 and SSDD. Results show that the proposed method consistently performs on par with the fully supervised approach.

Microsoft Coco: Common Objects In Context

Abstract: We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.


The proposed method employs various image processing algorithms and a pre-trained UNET model #REFR to extract infected regions from CT scans.

A criterion on the volume of infection has been used as an indicator of the severity of infection #OTHEREFR and classified the CT scans into different classes such as healthy, mild, moderate, severe, and critical.

Radiologists commonly use the CT severity score (CTSS) to determine the extent of severity in COVID-19 patients #OTHEREFR .

The score ranges from 0 to 25 and is calculated based on the distribution and magnitude of abnormalities (lung involvement) seen on chest CT scans.

A higher CTSS score indicates more severe lung damage, and this information is essential in making appropriate treatment decisions #OTHEREFR .

In line with this approach, we have developed an automatic method that categorizes CT severity into four classes: mild, moderate, severe, and critical.

Various classical machine-learning models and ensemble-based models learn these representative feature vectors to predict the severity classes for chest CT scans.

A fully automatic severity analysis model can significantly reduce clinicians' time to assess a patient's condition.

The paper is organized as follows: Section 2 explains the dataset used in the study.

Section 3 presents the proposed method for segmenting relevant clinical features and classifying COVID-19 patients into different severity classes. Section 4 discusses results and inferences. Finally, Section 5 summarizes the present study.

Enhancing Covid-19 Severity Analysis Through Ensemble Methods

Abstract: Computed Tomography (CT) scans provide a detailed image of the lungs, allowing clinicians to observe the extent of damage caused by COVID-19. The CT severity score (CTSS) based scoring method is used to identify the extent of lung involvement observed on a CT scan. This paper presents a domain knowledge-based pipeline for extracting regions of infection in COVID-19 patients using a combination of imageprocessing algorithms and a pre-trained UNET model. Then, an infection rate-based feature vector is proposed for each CT scan. The severity of the infection is then classified into different categories using an ensemble of three machine-learning models: Extreme Gradient Boosting, Extremely Randomized Trees, and Support Vector Machine. The proposed system was evaluated on a validation dataset in the AI-Enabled Medical Image Analysis Workshop and COVID-19 Diagnosis Competition (AI-MIA-COV19D) and achieved a macro F1 score of 64%. These results demonstrate the potential of combining domain knowledge with machine learning techniques for accurate COVID-19 diagnosis using CT scans. The implementation of the proposed system for severity analysis is available at https://github.com/aanandt/Enhancing-COVID-19-Severity-Analysis-through-Ensemble-Methods.git

Automatic Lung Segmentation In Routine Imaging Is A Data Diversity Problem, Not A Methodology Problem

Abstract: Automated segmentation of anatomical structures is a crucial step in many medical image analysis tasks. For lung segmentation, a variety of approaches exist, involving sophisticated pipelines trained and validated on a range of different data sets. However, during translation to clinical routine the applicability of these approaches across diseases remains limited. Here, we show that the accuracy and reliability of lung segmentation algorithms on demanding cases primarily does not depend on methodology, but on the diversity of training data. We compare 4 generic deep learning approaches and 2 published lung segmentation algorithms on routine imaging data with more than 6 different disease patterns and 3 published data sets. We show that a basic approach - U-net - performs either better, or competitively with other approaches on both routine data and published data sets, and outperforms published approaches once trained on a diverse data set covering multiple diseases. Training data composition consistently has a bigger impact than algorithm choice on accuracy across test data sets. We carefully analyse the impact of data diversity, and the specifications of annotations on both training and validation sets to provide a reference for algorithms, training data, and annotation. Results on a seemingly well understood task of lung segmentation suggest the critical importance of training data diversity compared to model choice.


We found that EigenTrust algorithm can correctly estimate low trust scores for two types of simple attacks-namely, Models A and B-which is consistent with results in past studies #REFR .

This paper has examined the applicability of EigenTrust algorithm to the use case of C2C services using an agent-based model.

The algorithm was developed to calculate the trust scores of nodes in P2P communication and does not take into account the actual situation of C2C services.

Therefore, we established transaction protocols for C2C services, in which the buyer and seller rate each other.

When analysing the results of the simulations, we focused on changes in the counterparty's trust score over time.

We therefore contribute to the literature (1) by having implemented a simulator that clarifies transaction procedures in purchasing goods and performs transactions between users and (2) by analysing the trust scores resulting from EigenTrust algorithm to confirm the difference in the detection of attacks between P2P and C2C.

Therefore, the algorithm is expected to mitigate the impact of simple attackers on C2C services.

Furthermore, we found oscillatory patterns in trust scores for two types of advanced attacks-namely, Models E and F-whereas no previous studies have focused on such dynamics.

That result suggests that by detecting such oscillations, EigenTrust algorithm may be able to detect advanced attacks that were considered beyond its reach in P2P communications.

Doing so was possible because we focused on the transient dynamics of trust scores, whereas past studies have focused on trust scores at a fixed point at the end state of transactions.

Our results provide insights for the further development of Trust Management Algorithms for the benefit of consumer services.

Applicability Of Trust Management Algorithm In C2C Services

Abstract: The emergence of Consumer-to-Consumer (C2C) platforms has allowed consumers to buy and sell goods directly, but it has also created problems, such as commodity fraud and fake reviews. Trust Management Algorithms (TMAs) are expected to be a countermeasure to detect fraudulent users. However, it is unknown whether TMAs are as effective as reported as they are designed for Peer-to-Peer (P2P) communications between devices on a network. Here we examine the applicability of 'EigenTrust', a representative TMA, for the use case of C2C services using an agentbased model. First, we defined the transaction process in C2C services, assumed six types of fraudulent transactions, and then analysed the dynamics of EigenTrust in C2C systems through simulations. We found that EigenTrust could correctly estimate low trust scores for two types of simple frauds. Furthermore, we found the oscillation of trust scores for two types of advanced frauds, which previous research did not address. This suggests that by detecting such oscillations, EigenTrust may be able to detect some (but not all) advanced frauds. Our study helps increase the trustworthiness of transactions in C2C services and provides insights into further technological development for consumer services.

Decentralized Trust Management: Risk Analysis And Trust Aggregation

Abstract: Decentralized trust management is used as a referral benchmark for assisting decision making by human or intelligence machines in open collaborative systems. During any given period of time, each participant may only interact with a few other participants. Simply relying on direct trust may frequently resort to random team formation. Thus, trust aggregation becomes critical. It can leverage decentralized trust management to learn about indirect trust of every participant based on past transaction experiences. This article presents alternative designs of decentralized trust management and their efficiency and robustness from three perspectives. First, we study the risk factors and adverse effects of six common threat models. Second, we review the representative trust aggregation models and trust metrics. Third, we present an in-depth analysis and comparison of these reference trust aggregation methods with respect to effectiveness and robustness. We show our comparative study results through formal analysis and experimental evaluation. This comprehensive study advances the understanding of adverse effects of present and future threats and the robustness of different trust metrics. It may also serve as a guideline for research and development of next-generation trust aggregation algorithms and services in the anticipation of risk factors and mischievous threats.


The model utilizes a Pyramid Pooling Module (PPM) #REFR to achieve a larger receptive field and generate effective global prior representations.

Considering the hierarchical structure of the features obtained using the Swin Transformer backbone, more advanced techniques for dense prediction, such as Feature Pyramid Networks (FPN) #OTHEREFR and U-Net #OTHEREFR , can be employed for linking the hierarchical information.

In our work, we incorporate UPerNet #OTHEREFR with our proposed PCA.

3 , UPerNet features a top-down architecture with lateral connections that facilitate the fusion of high-level semantic information with lower-level details.

UPerNet has the capability to learn visual attributes for semantic segmentation and image completion at multiple levels.

{F 1 , F 2 , F 3 , F 4 } denotes the set of the resulting feature maps of each stage of the backbone.

The corresponding downsampling rate for the {F 1 , F 2 , F 3 , F 4 } are {4,8,16,32}, respectively.

In the decoder stage, the PPM is only used at the top of F 4 , where the resulting feature map can be denoted asF 4 .

Then,F 4 is upsampled and sum with F 3 to progressively fuse multilevel features.

Fishdreamer: Towards Fisheye Semantic Completion Via Unified Image Outpainting And Segmentation

Abstract: This paper raises the new task of Fisheye Semantic Completion (FSC), where dense texture, structure, and semantics of a fisheye image are inferred even beyond the sensor field-of-view (FoV). Fisheye cameras have larger FoV than ordinary pinhole cameras, yet its unique special imaging model naturally leads to a blind area at the edge of the image plane. This is suboptimal for safety-critical applications since important perception tasks, such as semantic segmentation, become very challenging within the blind zone. Previous works considered the out-FoV outpainting and in-FoV segmentation separately. However, we observe that these two tasks are actually closely coupled. To jointly estimate the tightly intertwined complete fisheye image and scene semantics, we introduce the new FishDreamer which relies on successful ViTs enhanced with a novel Polar-aware Cross Attention module (PCA) to leverage dense context and guide semantically-consistent content generation while considering different polar distributions. In addition to the contribution of the novel task and architecture, we also derive Cityscapes-BF and KITTI360-BF datasets to facilitate training and evaluation of this new track. Our experiments demonstrate that the proposed FishDreamer outperforms methods solving each task in isolation and surpasses alternative approaches on the Fisheye Semantic Completion. Code and datasets are publicly available at FishDreamer.

Pyramid Scene Parsing Network

Abstract: Scene parsing is challenging for unrestricted open vocabulary and diverse scenes. In this paper, we exploit the capability of global context information by different-region-based context aggregation through our pyramid pooling module together with the proposed pyramid scene parsing network (PSPNet). Our global prior representation is effective to produce good quality results on the scene parsing task, while PSPNet provides a superior framework for pixel-level prediction. The proposed approach achieves state-of-the-art performance on various datasets. It came first in ImageNet scene parsing challenge 2016, PASCAL VOC 2012 benchmark and Cityscapes benchmark. A single PSPNet yields the new record of mIoU accuracy 85.4% on PASCAL VOC 2012 and accuracy 80.2% on Cityscapes.

larger receptive field
Pyramid Pooling Module


We investigate whether combining iterative reasoning and residual learning with CNNs and Transformers #REFR , as well as decoupled image-level and patch-level discriminators, can lead to a promising design for creating a more robust and data-efficient method to tackle the fewshot image inpainting task.

Moreover, existing inpainting methods are not designed specifically for few-shot learning and may not work well with limited training samples.

Domain-related prior knowledge #OTHEREFR or lightweight generative models #OTHEREFR , #OTHEREFR may be employed in image inpainting to mitigate overfitting.

However, they do not perform well when there is a large domain gap between two tasks or the reduced network capacity affects the inpainting quality.

To this end, we propose a novel few-shot Generative Residual Image inpaintinG framework (GRIG), which enables highquality image inpainting on few-shot datasets.

To effectively optimize inpainting results, we use iterative reasoning to solve more accurate and generalizable algorithmic reasoning tasks #OTHEREFR along with residual learning #OTHEREFR to incrementally refine previous estimates.

In this framework, the inpainting process is carried out in several forward passes by feeding the generator with the output of the previous iteration and a corresponding mask.

This enables the generator to be welltrained and focused on the residual information between the previously predicted output and the ground truth, resulting in effective image inpainting with high quality and good generality on few-shot data.

We implemented our framework into three components: a generator, a projected discriminator, and a forgery-patch discriminator.

The generator uses CNNs to extract shallow features of edges and textures and the Transformer blocks to capture global interactions between feature contexts at each iterative step.

To accelerate network convergence and reduce overfitting, we decouple the image distribution learning by Fig. 1 .

Grig: Few-Shot Generative Residual Image Inpainting

Abstract: Image inpainting is the task of filling in missing or masked region of an image with semantically meaningful contents. Recent methods have shown significant improvement in dealing with large-scale missing regions. However, these methods usually require large training datasets to achieve satisfactory results and there has been limited research into training these models on a small number of samples. To address this, we present a novel few-shot generative residual image inpainting method that produces high-quality inpainting results. The core idea is to propose an iterative residual reasoning method that incorporates Convolutional Neural Networks (CNNs) for feature extraction and Transformers for global reasoning within generative adversarial networks, along with image-level and patch-level discriminators. We also propose a novel forgerypatch adversarial training strategy to create faithful textures and detailed appearances. Extensive evaluations show that our method outperforms previous methods on the few-shot image inpainting task, both quantitatively and qualitatively.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


Comparison With State-Of-The-Art Methods

We conduct quantitative and qualitative comparisons of the same-identity reconstruction task on the VoxCeleb dataset, where FOMM #REFR is used to generate the coarse face animation results.

We compare our FADM with five state-of-theart methods: FOMM #OTHEREFR , Face vid2vid #OTHEREFR , Bilayer #OTHEREFR , DaGAN #OTHEREFR , and PIRenderer #OTHEREFR .

For Bilayer, FOMM, DaGAN, and PIRenderer, we use their official pre-trained models for evaluation, while for Face vid2vid, we adopt a widely recognized unofficial model due to the absence of the official code. All of these models are pre-trained on VoxCeleb.

In or-der to accelerate the inference, we randomly select several short-time videos from the testing dataset of VoxCeleb for quantitative comparison, and the selected video list is available in the supplementary materials.

As shown in Table 1 , compared to other SOTA methods, it is evident that FADM achieves the best performance in all metrics, especially the reconstruction faithfulness L 1 and the visual quality LPIPS, demonstrating the superiority of our FADM in generating high-fidelity face animation.

Moreover, we show the visual results of FADM and existing SOTA methods in Fig. 4 .

FADM exhibits overall better quality with more fine details than other methods. .

On both testing datasets, our FADM can produce more identity-preserving and photo-realistic results compared with other methods.

Face Animation With An Attribute-Guided Diffusion Model

Abstract: Figure 1. Comparison with SOTA face animation methods on reconstruction and reenactment tasks. Left: FOMM [42] often produces artifacts and over-smoothing textures, while FADM is capable of rectifying the distortions and enriching the fine-grained details. Right: Face vid2vid [49] suffers from typical unnatural facial details, while FADM prominently improves the overall visual quality.

First Order Motion Model For Image Animation

Abstract: Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate. Once trained on a set of videos depicting objects of the same category (\eg \ faces, human bodies), our method can be applied to any object of this class. To achieve this, we decouple appearance and motion information using a self-supervised formulation. To support complex motions, we use a representation consisting of a set of learned keypoints along with their local affine transformations. A generator network models occlusions arising during target motions and combines the appearance extracted from the source image and the motion derived from the driving video. Our framework scores best on diverse benchmarks and on a variety of object categories. Our source code is publicly available.


The resulting OctFormer has a similar amount of trainable parameters (39M) as MinkowskiNet (38M) #REFR .

The dilations of octree attention in two consecutive OctFormer blocks are set to 1 and 4, respectively.

The downsampling module is implemented as an octree convolution with a kernel size of 2 and a stride of 2, followed by a Batch Normalization layer, which reduces the spatial resolution and increases the channel of a feature map by a factor of 2.

By default, is set to 96, the block numbers are set to {2, 2, 18, 2}, the head numbers of octree attentions are set to 1/16 of the channel numbers, and the point numbers for window partition are set to 32.

The output of OctFormer is hierarchical features with four spatial resolutions, i.e., { /4, /8, /16, /32}, which can be conveniently integrated with a feature pyramid network (FPN) [Lin et al. 2017] for segmentation and detection.

And the last feature map can be averaged as the global feature for shape classification.

4D Spatio-Temporal Convnets: Minkowski Convolutional Neural Networks

Abstract: In many robotics and VR/AR applications, 3D-videos are readily-available input sources (a sequence of depth images, or LIDAR scans). However, in many cases, the 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose generalized sparse convolutions that encompass all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and trilateral-stationary conditional random fields that enforce spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that a convolutional neural network with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise and outperform the 3D convolutional neural network.


PFNM and FedMA #REFR apply the Bayesian non-parametric mechanism to study the permutation invariance of the neural network, and match the neurons of client neural networks to the global neurons.

Then, the server collects the updated models from each client and aggregates them to obtain a new global model by averaging the model weights.

Unlike the privacy and communication challenges, the statistical heterogeneity challenge is a unique and popular issue in the federated learning paradigm #OTHEREFR .

According to the two stages of federated learning mentioned above, the contributions of these studies can be roughly divided into local update improvements and aggregation improvements.

Our work is an improvement in local update phase, so it can be combined with existing aggregation improvements without any conflict.

As for the aggregation improvements on non-IID data, there are a series of related studies.

Moreover, methods such as adaptive weights #OTHEREFR , attention mechanisms #OTHEREFR , and normalization #OTHEREFR are also used to improve the aggregation effect for statistical heterogeneity.

There are also many studies to alleviate the negative effects of statistical heterogeneity in the local update phase. #OTHEREFR

(2020) propose the FedProx algorithm, which adds a regular term to the loss function of the client.

This proximal term uses the 2 norm to explicitly constrain the local model to be close to the latest global model, limiting the local update of non-IID clients to be not too far apart.

However, this explicit constraint could inhibit FedProx from quickly finding a better model in the early stage.

Federated Learning With Classifier Shift For Class Imbalance

Abstract: Federated learning aims to learn a global model collaboratively while the training data belongs to different clients and is not allowed to be exchanged. However, the statistical heterogeneity challenge on non-IID data, such as class imbalance in classification, will cause client drift and significantly reduce the performance of the global model. This paper proposes a simple and effective approach named FedShift which adds the shift on the classifier output during the local training phase to alleviate the negative impact of class imbalance. We theoretically prove that the classifier shift in FedShift can make the local optimum consistent with the global optimum and ensure the convergence of the algorithm. Moreover, our experiments indicate that FedShift significantly outperforms the other state-of-the-art federated learning approaches on various datasets regarding accuracy and communication efficiency. * Use footnote for providing further information about author (webpage, alternative address)-not for acknowledging funding agencies. Preprint. Under review.

Federated Learning With Matched Averaging

Abstract: Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device, decoupling the ability to do model training from the need to store the data in the cloud. We propose Federated matched averaging (FedMA) algorithm designed for federated learning of modern neural network architectures e.g. convolutional neural networks (CNNs) and LSTMs. FedMA constructs the shared global model in a layer-wise manner by matching and averaging hidden elements (i.e. channels for convolution layers; hidden states for LSTM; neurons for fully connected layers) with similar feature extraction signatures. Our experiments indicate that FedMA outperforms popular state-of-the-art federated learning algorithms on deep CNN and LSTM architectures trained on real world datasets, while improving the communication efficiency.