A Common Setup For Classifier And Matching-Based Approaches

We identify two dominant families of approaches proposed in the recent few-shot action recognition literature: the classifier-based and the matching-based methods #REFR .

Unfortunately, discrepancies in setup and architecture between existing approaches make it difficult to compare them fairly.

We propose to follow the same representation learning strategy and start from a common frozen backbone.

Classifier-based approaches The classifier-based approaches #OTHEREFR train the video representation using a classifier in the form of a linear layer.

This is similar to the corresponding work on few-shot learning in the image domain #OTHEREFR .

Classifiers are trained using the whole dataset, but episodic training cannot guarantee to see all the examples because it is intractable to sample all the possible episodes. Xian et al.

Rethinking Matching-Based Few-Shot Action Recognition

Abstract: Few-shot action recognition, i.e. recognizing new action classes given only a few examples, benefits from incorporating temporal information. Prior work either encodes such information in the representation itself and learns classifiers at test time, or obtains frame-level features and performs pairwise temporal matching. We first evaluate a number of matching-based approaches using features from spatio-temporal backbones, a comparison missing from the literature, and show that the gap in performance between simple baselines and more complicated methods is significantly reduced. Inspired by this, we propose Chamfer++, a non-temporal matching function that achieves state-of-the-art results in few-shot action recognition. We show that, when starting from temporal features, our parameter-free and interpretable approach can outperform all other matching-based and classifier methods for one-shot action recognition on three common datasets without using temporal information in the matching stage. Project page: https://jbertrand89.github.io/matching-based-fsar

Few-Shot Video Classification Via Temporal Alignment

Abstract: There is a growing interest in learning a model which could recognize novel classes with only a few labeled examples. In this paper, we propose Temporal Alignment Module (TAM), a novel few-shot learning framework that can learn to classify a previous unseen video. While most previous works neglect long-term temporal ordering information, our proposed model explicitly leverages the temporal ordering information in video data through temporal alignment. This leads to strong data-efficiency for few-shot learning. In concrete, TAM calculates the distance value of query video with respect to novel class proxies by averaging the per frame distances along its alignment path. We introduce continuous relaxation to TAM so the model can be learned in an end-to-end fashion to directly optimize the few-shot learning objective. We evaluate TAM on two challenging real-world datasets, Kinetics and Something-Something-V2, and show that our model leads to significant improvement of few-shot video classification over a wide range of competitive baselines.


For our high fidelity simulation environment, we took inspiration from many industry standard simulations, such as Gazebo and AirSim #REFR .

Still, our high fidelity simulation does have several characteristics that cause it to vary from many common simulations.

Two of the key differentiators is that there is no active physics in the high fidelity simulation and agent bodies consist of simple relative sized cube objects.

In place of a physics environment, each platform type has a geometry based PID controller to actuate motion.

Agents send actuation commands via ROS Twist messages or setpoint position commands just like on the real platforms.

Comparable to other simulators, individual agents receive sensory information from ROS messages.

From Warfighting Needs To Robot Actuation: A Complete Rapid Integration Swarming Solution

Abstract: Swarm robotics systems have the potential to transform warfighting in urban environments, but until now have not seen large-scale field testing. We present the Rapid Integration Swarming Ecosystem (RISE), a platform for future multi-agent research and deployment. RISE enables rapid integration of third-party swarm tactics and behaviors, which was demonstrated using both physical and simulated swarms. Our physical testbed is composed of more than 250 networked heterogeneous agents and has been extensively tested in mock warfare scenarios at five urban combat training ranges. RISE implements live, virtual, constructive simulation capabilities to allow the use of both virtual and physical agents simultaneously, while our "fluid fidelity" simulation enables adaptive scaling between low and high fidelity simulation levels based on dynamic runtime requirements. Both virtual and physical agents are controlled with a unified gesture-based interface that enables a greater than 150:1 agent-to-operator ratio. Through this interface, we enable efficient swarm-based mission execution. RISE translates mission needs to robot actuation with rapid tactic integration, a reliable testbed, and efficient operation.

Airsim: High-Fidelity Visual And Physical Simulation For Autonomous Vehicles

Abstract: Developing and testing algorithms for autonomous vehicles in real world is an expensive and time consuming process. Also, in order to utilize recent advances in machine intelligence and deep learning we need to collect a large amount of annotated training data in a variety of conditions and environments. We present a new simulator built on Unreal Engine that offers physically and visually realistic simulations for both of these goals. Our simulator includes a physics engine that can operate at a high frequency for real-time hardware-in-the-loop (HITL) simulations with support for popular protocols (e.g. MavLink). The simulator is designed from the ground up to be extensible to accommodate new types of vehicles, hardware platforms and software protocols. In addition, the modular design enables various components to be easily usable independently in other projects. We demonstrate the simulator by first implementing a quadrotor as an autonomous vehicle and then experimentally comparing the software components with real-world flights.


Compared with supervised approaches for anomaly detection #REFR , employing autoencoder-based models for anomaly detection problems is efficient as only normal data are necessary for the training phase, and collecting various labeled anomalous data is costly and not always possible.

The goal of training an autoencoder is to capture the nonlinear correlations among input features #OTHEREFR .

For anomaly detection tasks, autoencoders model the data distribution of normal samples by minimizing the distance between the normal sample x and the reconstructed sample x .

As shown in #OTHEREFR , an autoencoder can solve the following optimization problem #OTHEREFR :

where φ and ψ are the parameters for the encoder and decoder, respectively, and l 2 -norm is usually utilized for || • ||.

Anomaly Detection With Ensemble Of Encoder And Decoder

Abstract: Hacking and false data injection from adversaries can threaten power grids' everyday operations and cause significant economic loss. Anomaly detection in power grids aims to detect and discriminate anomalies caused by cyber attacks against the power system, which is essential for keeping power grids working correctly and efficiently. Different methods have been applied for anomaly detection, such as statistical methods and machine learning-based methods. Usually, machine learningbased methods need to model the normal data distribution. In this work, we propose a novel anomaly detection method by modeling the data distribution of normal samples via multiple encoders and decoders. Specifically, the proposed method maps input samples into a latent space and then reconstructs output samples from latent vectors. The extra encoder finally maps reconstructed samples to latent representations. During the training phase, we optimize parameters by minimizing the reconstruction loss and encoding loss. Training samples are re-weighted to focus more on missed correlations between features of normal data. Furthermore, we employ the long short-term memory model as encoders and decoders to test its effectiveness. We also investigate a meta-learning-based framework for hyper-parameter tuning of our approach. Experiment results on network intrusion and power system datasets demonstrate the effectiveness of our proposed method, where our models consistently outperform all baselines.

Smart Grid Cyber Attacks Detection Using Supervised Learning And Heuristic Feature Selection

Abstract: False Data Injection (FDI) attacks are a common form of Cyber-attack targetting smart grids. Detection of stealthy FDI attacks is impossible by the current bad data detection systems. Machine learning is one of the alternative methods proposed to detect FDI attacks. This paper analyzes three various supervised learning techniques, each to be used with three different feature selection (FS) techniques. These methods are tested on the IEEE 14-bus, 57-bus, and 118-bus systems for evaluation of versatility. Accuracy of the classification is used as the main evaluation method for each detection technique. Simulation study clarify the supervised learning combined with heuristic FS methods result in an improved performance of the classification algorithms for FDI attack detection.


As for the optimization, we use Adam optimizer with an initial learning rate of 2e-4 and a cosine annealing scheme for learning rate scheduling #REFR .

In our experiments, we set the time interval of event streams to 50ms, which is same as #OTHEREFR .

For each pillar, we set the maximum number of events per pillar M to 5 and the number of nonempty pillars K to 100k.

In addition, the output channels of EventPillars are set to 16.

In order to make a fair comparison for other event processing strategies, the output of all event representations are uniformly resized to 512 × 512 in both training and inference phase. Note that no extra data augmentation is applied. Implementation of DMANet. The proposed DMANet is built based on Pytorch. We use ResNet-18 #OTHEREFR as the backbone of our network.

The time step for training adaptive ConvLSTMs is empirically set to 10 according to GPU memory.

All the experiments are conducted on a single NVIDIA RTX 2080Ti GPU. The code is available at https://github.com/wds320/AAAI Event based detection.

Dual Memory Aggregation Network For Event-Based Object Detection With Learnable Representation

Abstract: Event-based cameras are bio-inspired sensors that capture brightness change of every pixel in an asynchronous manner. Compared with frame-based sensors, event cameras have microsecond-level latency and high dynamic range, hence showing great potential for object detection under high-speed motion and poor illumination conditions. Due to sparsity and asynchronism nature with event streams, most of existing approaches resort to hand-crafted methods to convert event data into 2D grid representation. However, they are sub-optimal in aggregating information from event stream for object detection. In this work, we propose to learn an event representation optimized for event-based object detection. Specifically, event streams are divided into grids in the x-y-t coordinates for both positive and negative polarity, producing a set of pillars as 3D tensor representation. To fully exploit information with event streams to detect objects, a dual-memory aggregation network (DMANet) is proposed to leverage both long and short memory along event streams to aggregate effective information for object detection. Long memory is encoded in the hidden state of adaptive convLSTMs while short memory is modeled by computing spatial-temporal correlation between event pillars at neighboring time intervals. Extensive experiments on the recently released event-based automotive detection dataset demonstrate the effectiveness of the proposed method.

Sgdr: Stochastic Gradient Descent With Warm Restarts

Abstract: Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR


This property is similar to the dynamics of a SIR as shown in #REFR , which provides the possibility to extend the existing results for SIRs to the virtual centers of CSURs.

Substituting it to (20), we haveż(t) = 0 which denotes that the virtual center z(t) is a static point in this case.

Then, equation #OTHEREFR indicates that the robot is moving around z(t) along a circular orbit with a linear speed v 0 , an angular velocity ω 0 , and orbit radius v 0 /|ω 0 |.

Thus, z(t) can be interpreted as the center of the circular orbit of the CSUR when it is a static point, which is why it is referred to as a virtual center.

The relation between the CSUR position ζ(t) and its virtual center z(t) is shown in Fig. 2 .

Different from the CSUR position ζ(t) that has to always move at a constant linear speed, the virtual center z(t) can remain static at a certain position when the CSUR is controlled with a constant input u(t) ≡ ω 0 .

Therefore, in his paper, we refer to the CSUR virtual centers as CSUR agents and use them to conduct the coverage task.

Nevertheless, it is noticed that the dynamic model of a CSUR agent in (20) is more complicated than a SIR model #OTHEREFR , which brings up challenges to this extension. Sec. III-D explains the challenges in detail.

Distributed Coverage Control Of Constrained Constant-Speed Unicycle Multi-Agent Systems

Abstract: This paper proposes a novel distributed coverage controller for a multi-agent system with constant-speed unicycle robots (CSUR). The work is motivated by the limitation of the conventional method that does not ensure the satisfaction of hard state-and input-dependent constraints and leads to feasibility issues for multi-CSUR systems. In this paper, we solve these problems by designing a novel coverage cost function and a saturated gradient-search-based control law. Invariant set theory and Lyapunov-based techniques are used to prove the state-dependent confinement and the convergence of the system state to the optimal coverage configuration, respectively. The controller is implemented in a distributed manner based on a novel communication standard among the agents. A series of simulation case studies are conducted to validate the effectiveness of the proposed coverage controller in different initial conditions and with control parameters. A comparison study in simulation reveals the advantage of the proposed method in terms of avoiding infeasibility. The experiment study verifies the applicability of the method to real robots with uncertainties. The development procedure of the method from theoretical analysis to experimental validation provides a novel framework for multiagent system coordinate control with complex agent dynamics. Note to Practitioners-This paper gives a novel solution for multiple robots to effectively cover a polygonal area. Compared to the conventional approaches, our method allows the robots to cover a target region using circular orbits, which is suitable for constant-speed unicycle robots (CSUR) like fixed-wing unmanned aerial vechicles (fUAV). The main advantage of this method is to ensure that the coverage is always successful by preventing the robots from departing the target region. Also, the method satisfies common control saturation constraints in practice and can be implemented in a reliable decentralized scheme. The method is validated to be effective for wheeled robots in experiment studies, although it can also be applied to fUAVs in theory.

Nonsmooth Coordination And Geometric Optimization Via Distributed Dynamical Systems

Abstract: This paper discusses dynamical systems for disk-covering and sphere-packing problems. We present facility location functions from geometric optimization and characterize their differentiable properties. We design and analyze a collection of distributed control laws that are related to nonsmooth gradient systems. The resulting dynamical systems promise to be of use in coordination problems for networked robots; in this setting the distributed control laws correspond to local interactions between the robots. The technical approach relies on concepts from computational geometry, nonsmooth analysis, and the dynamical system approach to algorithms.


Different from #REFR , we rescale and translate the closed object such that it fits in the unit cube and then backwards apply that same rescaling and translation to all objects of the same instance independent of the joint state of the object.

This helps with the stability of learning and simplifies hyperparameter tuning.

This approach fails for articulated objects as their outer dimensions change depending on the joint state.

Blindly rescaling an articulated object such that it fits inside the unit cube or unit sphere results in an inconsistent part scaling across different joint states. To mitigate this problem, #OTHEREFR proposed the NAOCS-space.

Following their approach, first, we bring an object in its closed state (i.e.

the lower joint limit) and put it in a canonical orientation such that for all objects, Z is pointing upwards and X back, Y is given through a right-hand coordinate frame system.

It is important to note that rescaling an articulated object has no impact on revolute joint states (in deg), but prismatic joint states (in m) which have to be rescaled accordingly.

Carto: Category And Joint Agnostic Reconstruction Of Articulated Objects

Abstract: Figure 1. Visualization of CARTO on unseen object instances. We first use CARTO to jointly detect all objects in the scene and then articulate them while keeping the predicted shape code constant.

Category-Level Articulated Object Pose Estimation

Abstract: This paper addresses the task of category-level pose estimation for articulated objects from a single depth image. We present a novel category-level approach that correctly accommodates object instances not previously seen during training. A key aspect of the work is the new Articulation-Aware Normalized Coordinate Space Hierarchy (A-NCSH), which represents the different articulated objects for a given object category. This approach not only provides the canonical representation of each rigid part, but also normalizes the joint parameters and joint states. We developed a deep network based on PointNet++ that is capable of predicting an A-NCSH representation for unseen object instances from single depth input. The predicted A-NCSH representation is then used for global pose optimization using kinematic constraints. We demonstrate that constraints associated with joints in the kinematic chain lead to improved performance in estimating pose and relative scale for each part of the object. We also demonstrate that the approach can tolerate cases of severe occlusion in the observed data. Project webpage this https URL


Our work is based on the Citation Need model #REFR , which constructs the Wikipedia verifiability taxonomy and uses machine learning to determine whether a sentence needs a citation.

Studies have shown that editors primarily add references to fortify the information they edit and prevent their edits from being removed #OTHEREFR .

Numerous studies have released datasets for claim verification in specific domains, such as scientific claims #OTHEREFR .

Studies like #OTHEREFR target cross-domain verification and show shared characteristics among these datasets.

Our work focuses on broad topics to account for the wide variety of topics covered by Wikipedia articles.

Compared to other work, we also consider the guidelines employed by Wikipedia's verifiability policy.

We expand the model into an end-to-end inference pipeline that is usable by the Wikipedia community and examine the trends seen in a decade's worth of data.

In addition to extending previous work, we use a crowd-sourced list of domains compiled by Wikipedia editors who manually evaluated the credibility of those sources to assess the reliability #OTHEREFR .

Longitudinal Assessment Of Reference Quality On Wikipedia

Abstract: Wikipedia plays a crucial role in the integrity of the Web. This work analyzes the reliability of this global encyclopedia through the lens of its references. We operationalize the notion of reference quality by defining reference need (RN), i.e., the percentage of sentences missing a citation, and reference risk (RR), i.e., the proportion of non-authoritative references. We release Citation Detective, a tool for automatically calculating the RN score, and discover that the RN score has dropped by 20 percent point in the last decade, with more than half of verifiable statements now accompanying references. The RR score has remained below 1% over the years as a result of the efforts of the community to eliminate unreliable references. We propose pairing novice and experienced editors on the same Wikipedia article as a strategy to enhance reference quality. Our quasi-experiment indicates that such a co-editing experience can result in a lasting advantage in identifying unreliable sources in future edits. As Wikipedia is frequently used as the ground truth for numerous Web applications, our findings and suggestions on its reliability can have a far-reaching impact. We discuss the possibility of other Web services adopting Wiki-style user collaboration to eliminate unreliable content. CCS CONCEPTS • Information systems → Wikis.

Citation Needed: A Taxonomy And Algorithmic Assessment Of Wikipedia'S Verifiability

Abstract: Wikipedia is playing an increasingly central role on the web, and the policies its contributors follow when sourcing and fact-checking content affect million of readers. Among these core guiding principles, verifiability policies have a particularly important role. Verifiability requires that information included in a Wikipedia article be corroborated against reliable secondary sources. Because of the manual labor needed to curate Wikipedia at scale, however, its contents do not always evenly comply with these policies. Citations (i.e. reference to external sources) may not conform to verifiability requirements or may be missing altogether, potentially weakening the reliability of specific topic areas of the free encyclopedia. In this paper, we aim to provide an empirical characterization of the reasons why and how Wikipedia cites external sources to comply with its own verifiability guidelines. First, we construct a taxonomy of reasons why inline citations are required, by collecting labeled data from editors of multiple Wikipedia language editions. We then crowdsource a large-scale dataset of Wikipedia sentences annotated with categories derived from this taxonomy. Finally, we design algorithmic models to determine if a statement requires a citation, and to predict the citation reason . We evaluate the accuracy of such models across different classes of Wikipedia articles of varying quality, and on external datasets of claims annotated for fact-checking purposes.


The permutation LM #REFR is a recently proposed LM that takes advantage of CLMs and masked LMs.

CLMs and masked LMs have their own advantages and disadvantages.

A masked LM needs to create artificial tokens such as [mask] , which never occur in downstream tasks while CLMs only condition on preceding context.

Given an input sequence of linguistic units, permutation LMs randomize the order of input linguistic units and construct different permutations of the input sequence. Fig.

2 shows an example of different permutations given an input text sequence. Let Z be the set of all possible permutations.

Permutation LMs predict the next linguistic unit, u t , in one permutation, Z, of the sequence based on

An Overview On Language Models: Recent Developments And Outlook

Abstract: Language modeling studies the probability distributions over strings of texts. It is one of the most fundamental tasks in natural language processing (NLP). It has been widely used in text generation, speech recognition, machine translation, etc. Conventional language models (CLMs) aim to predict the probability of linguistic sequences in a causal manner. In contrast, pre-trained language models (PLMs) cover broader concepts and can be used in both causal sequential modeling and fine-tuning for downstream applications. PLMs have their own training paradigms (usually self-supervised) and serve as foundation models in modern NLP systems. This overview paper provides an introduction to both CLMs and PLMs from five aspects, i.e., linguistic units, structures, training methods, evaluation methods, and applications. Furthermore, we discuss the relationship between CLMs and PLMs and shed light on the future directions of language modeling in the pre-trained era.

Xlnet: Generalized Autoregressive Pretraining For Language Understanding

Abstract: With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment setting, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.


Formally, let D = {(x, y)} be a dataset of images x with corresponding object annotations y, such as COCO 2017 #REFR , and let C = {1, . . . , C} be the set of object categories.

In previous work #OTHEREFR , in each phase, the incremental detector is allowed to observe all images that contain a certain type of objects.

Because images often contain a mix of object classes, both old and new, this means that the same images can be observed in different training phases.

This is incompatible with the standard definition of incremental learning #OTHEREFR where, with the exception of the examples deliberately stored in the exemplar memory, the images observed in different phases do not repeat.

Thus, we provide results with our new IOD benchmark protocol in the main paper.

For completeness and comparison, here we also evaluate performance using the traditional IOD benchmark protocol used in #OTHEREFR , and provide comparison results between our method and other top-performing IOD methods with this protocol. Traditional IOD protocol.

We adopt such a dataset for benchmarking IOD as follows.

First, we partition C into M subsets C = C 1 ∪ • • • ∪ C M , one for each training phase.

For each phase i, we modify the samples (x, y) ∈ D, where y only contains annotations for objects of class C i and drop the others.

In phase i of training, the model is only allowed to observe images that contain at least one annotation for objects of types C i ⊂ C. Experiment results.

Table S1 shows that also with the traditional IOD protocol our CL-DETR consistently performs better than the state-of-the-art #OTHEREFR and other IOD methods #OTHEREFR .

Continual Detection Transformer For Incremental Object Detection

Abstract: Incremental object detection (IOD) aims to train an object detector in phases, each with annotations for new object categories. As other incremental settings, IOD is subject to catastrophic forgetting, which is often addressed by techniques such as knowledge distillation (KD) and exemplar replay (ER). However, KD and ER do not work well if applied directly to state-of-the-art transformer-based object detectors such as Deformable DETR [61] and UP-DETR [10]. In this paper, we solve these issues by proposing a ContinuaL DEtection TRansformer (CL-DETR), a new method for transformer-based IOD which enables effective usage of KD and ER in this context. First, we introduce a Detector Knowledge Distillation (DKD) loss, focusing on the most informative and reliable predictions from old versions of the model, ignoring redundant background predictions, and ensuring compatibility with the available ground-truth labels. We also improve ER by proposing a calibration strategy to preserve the label distribution of the training set, therefore better matching training and testing statistics. We conduct extensive experiments on COCO 2017 and demonstrate that CL-DETR achieves state-of-theart results in the IOD setting. 1

Microsoft Coco: Common Objects In Context

Abstract: We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.


Among these techniques, quantization #REFR has emerged as a general and primary paradigm for reducing both memory footprint and computational overhead.

Transformer language models (e.g., BERT, LLMs) have garnered significant attention due to their remarkable performance and scalable model size.

These models have evolved from hundreds of millions of parameters #OTHEREFR to hundreds of billions of parameters #OTHEREFR .

This necessitates the employment of compression techniques #OTHEREFR Le-Cun et al., 1989) for practical deployment.

However, quantization, particularly post-training quantization (PTQ) #OTHEREFR , which operates with limited data and GPU resources, has become increasingly challenging for these models (e.g., a 12% accuracy drop in BERT #OTHEREFR and catastrophic degradation in OPT-175B ).

This is caused by the presence of detrimental outliers in activation (e.g., the width of distribution can be 80 for BERT and even 140 for OPTs), which prevents quantization from accurately expressing floating-point numbers.

To combat the outlier bottleneck in quantization, researchers make in-depth investigations and find the concentration phenomenon that outliers typically appear in certain channels.

Hence, some works #OTHEREFR explored various fine-grained quantization schemes to provide additional consideration for them while some #OTHEREFR attempted to diminish them through a scaling operation, which can be migrated to subsequent modules to maintain floating-point (FP) equivalence.

However, we find that they ignore the asymmetric property of outliers that relatively small channel ranges can result in a significantly large range for the entire tensor (e.g., as shown in Fig.

Outlier Suppression+: Accurate Quantization Of Large Language Models By Equivalent And Optimal Shifting And Scaling

Abstract: Quantization of transformer language models faces significant challenges due to the existence of detrimental outliers in activations. We observe that these outliers are asymmetric and concentrated in specific channels. To address this issue, we propose the Outlier Suppres-sion+ framework. First, we introduce channelwise shifting and scaling operations to eliminate asymmetric presentation and scale down problematic channels. We demonstrate that these operations can be seamlessly migrated into subsequent modules while maintaining equivalence. Second, we quantitatively analyze the optimal values for shifting and scaling, taking into account both the asymmetric property and quantization errors of weights in the next layer. Our lightweight framework can incur minimal performance degradation under static and standard post-training quantization settings. Comprehensive results across various tasks and models reveal that our approach achieves near-floating-point performance on both small models, such as BERT, and large language models (LLMs) including OPTs, BLOOM, and BLOOMZ at 8-bit and 6bit settings. Furthermore, we establish a new state of the art for 4-bit BERT.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


Inspired by DeepVoxels #REFR , which improves the accuracy of RGB novel view synthesis with voxel features, we introduce a 3D voxel representation of the rough object shape before the decoder which ensures multi-view consistency of output depth images.

DeepDDF enables direct evaluation of the object shape and pose against observations in the image-space as it bypasses rendering (projection) of 3D geometry in the 2D view.

Two fundamental challenges, however, need to be resolved to achieve this direct 2D view-dependent representation.

The first is that the depth values for each camera ray across different viewpoints must be consistent, i.e., must represent the same rigid object shape.

The second is the large number of samples required to span the entire space of the inputs to the decoder p ray , d ray , and z.

We overcome these challenges by introducing a sparse 3D voxel grid aligned with the target object space.

This 3D voxel can be of very low-resolution as it does not encode the 3D shape itself.

2 , the latent code z is decoded into latent voxels by an object instance decoder f d , and the voxels on the camera rays, defined by its 3D viewpoint p ray and 3D direction d ray , are sampled by a camera ray sampler f mlp .

This low-resolution but three-dimensional voxel corresponding to the object instance encoded by the latent code z enforces consistency across the camera rays and makes the output depth values D multi-view consistent.

It also enables reduction in the number of samples for p ray , d ray , and z by ensuring this multi-view consistency.

Transposer: Transformer As An Optimizer For Joint Object Shape And Pose Estimation

Abstract: Transformer Decoder TransPoser momentum cross-attn Transformer Encoder Figure 1: We propose TransPoser, a novel neural optimization network for joint object pose and shape reconstruction from sequential observations (a). We also derive DeepDDF, a novel category-level neural 3D shape representation that directly outputs 2D depth images for given viewing conditions. By integrating DeepDDF and TransPoser, we realize efficient and accurate learned weighted optimization with learned momentum for joint shape and pose estimation (b).

Deepvoxels: Learning Persistent 3D Feature Embeddings

Abstract: In this work, we address the lack of 3D understanding of generative neural networks by introducing a persistent 3D feature embedding for view synthesis. To this end, we propose DeepVoxels, a learned representation that encodes the view-dependent appearance of a 3D scene without having to explicitly model its geometry. At its core, our approach is based on a Cartesian 3D grid of persistent embedded features that learn to make use of the underlying 3D scene structure. Our approach combines insights from 3D geometric computer vision with recent advances in learning image-to-image mappings based on adversarial loss functions. DeepVoxels is supervised, without requiring a 3D reconstruction of the scene, using a 2D re-rendering loss and enforces perspective and multi-view geometry in a principled manner. We apply our persistent 3D scene representation to the problem of novel view synthesis demonstrating high-quality results for a variety of challenging scenes.


Following #REFR , we split the train, validation, and test partitions, with 100K, 18K, and 13K samples.

2014) for both unconditional and conditional molecule generation.

QM9 is one of the most widely-used datasets for molecular machine learning research, which has also been adopted in previous 3D molecule generation studies #OTHEREFR .

QM9 contains 3D structures together with several quantum properties for 130k small molecules, limited to 9 heavy atoms (29 atoms including hydrogens).

For the molecule generation task, we also test GEOLDM on the GEOM-DRUG (Geometric Ensemble Of Molecules) dataset.

The DRUG dataset consists of much larger organic compounds, with up to 181 atoms and 44.2 atoms on average, in 5 different atom types.

It covers 37 million molecular conformations for around 450,000 molecules, labeled with energy and statistical weight.

We follow the common practice #OTHEREFR to select the 30 lowest energy conformations of each molecule for training.

Geometric Latent Diffusion Models For 3D Molecule Generation

Abstract: Generative models, especially diffusion models (DMs), have achieved promising results for generating feature-rich geometries and advancing foundational science problems such as molecule design. Inspired by the recent huge success of Stable (latent) Diffusion models, we propose a novel and principled method for 3D molecule generation named Geometric Latent Diffusion Models (GEOLDM). GEOLDM is the first latent DM model for the molecular geometry domain, composed of autoencoders encoding structures into continuous latent codes and DMs operating in the latent space. Our key innovation is that for modeling the 3D molecular geometries, we capture its critical roto-translational equivariance constraints by building a point-structured latent space with both invariant scalars and equivariant tensors. Extensive experiments demonstrate that GEOLDM can consistently achieve better performance on multiple molecule generation benchmarks, with up to 7% improvement for the valid percentage of large biomolecules. Results also demonstrate GEOLDM's higher capacity for controllable generation thanks to the latent modeling. Code is provided at https: //github.com/MinkaiXu/GeoLDM.

Cormorant: Covariant Molecular Neural Networks

Abstract: We propose Cormorant, a rotationally covariant neural network architecture for learning the behavior and properties of complex many-body physical systems. We apply these networks to molecular systems with two goals: learning atomic potential energy surfaces for use in Molecular Dynamics simulations, and learning ground state properties of molecules calculated by Density Functional Theory. Some of the key features of our network are that (a) each neuron explicitly corresponds to a subset of atoms; (b) the activation of each neuron is covariant to rotations, ensuring that overall the network is fully rotationally invariant. Furthermore, the non-linearity in our network is based upon tensor products and the Clebsch-Gordan decomposition, allowing the network to operate entirely in Fourier space. Cormorant significantly outperforms competing algorithms in learning molecular Potential Energy Surfaces from conformational geometries in the MD-17 dataset, and is competitive with other methods at learning geometric, energetic, electronic, and thermodynamic properties of molecules on the GDB-9 dataset.


Additionally, we rely on pseudo-rehearsal through experience replay #REFR using a small memory buffer to tackle catastrophic forgetting. Figure 2 visualizes our transformer-based architecture for multimodal CL.

Because when the model is updated to learn the current tasks, its performance on the past learned tasks will degrade due to catastrophic forgetting #OTHEREFR .

Figure 1 visualizes the high-level presentation of the solution that we propose to address multimodal CL.

To use a shared model across all tasks and benefit from knowledge transfer across all tasks, we consider a base transformer model as f θ (•, •) and make it adaptive by adding a unique task attention layer after its final layer.

Additionally, we modify the loss supervised learning loss by adding a knowledge distillation loss #OTHEREFR on the intermediate model layers.

We consider the teacher model in the knowledge distillation formulation to be a copy of f i−1 θ (•, •) when training the main student model on i th task T i .

The architecture consists of a shared pre-trained replaceable bimodal transformer, an independent task attention block, and MLP classification headers.

The task attention block receives task-specific tokens that make the model adaptive.

We provide details about these modules and the strategies that we use to train it.

Task-Attentive Transformer Architecture For Continual Learning Of Vision-And-Language Tasks Using Knowledge Distillation

Abstract: The size and the computational load of fine-tuning largescale pre-trained neural network are becoming two major obstacles in adopting machine learning in many applications. Continual learning (CL) can serve as a remedy through enabling knowledge-transfer across sequentially arriving tasks which relaxes the need to fine-tune all network weights from scratch. However, existing CL algorithms primarily consider learning unimodal vision-only or language-only tasks. We develop a transformer-based CL architecture for learning bimodal vision-and-language tasks based on increasing the number of the learnable parameters dynamically and using knowledge distillation. The new additional parameters are used to specialize the network for each task. Our approach enables sharing information between the tasks while addressing the challenge of catastrophic forgetting. Our approach is scalable learning to a large number of tasks because it requires little memory and time overhead. Our model reaches state-of-the-art performance on challenging vision-and-language tasks.

Experience Replay For Continual Learning

Abstract: Interacting with a complex world involves continual learning, in which tasks and data distributions change over time. A continual learning system should demonstrate both plasticity (acquisition of new knowledge) and stability (preservation of old knowledge). Catastrophic forgetting is the failure of stability, in which new experience overwrites previous experience. In the brain, replay of past experience is widely believed to reduce forgetting, yet it has been largely overlooked as a solution to forgetting in deep reinforcement learning. Here, we introduce CLEAR, a replay-based method that greatly reduces catastrophic forgetting in multi-task reinforcement learning. CLEAR leverages off-policy learning and behavioral cloning from replay to enhance stability, as well as on-policy learning to preserve plasticity. We show that CLEAR performs better than state-of-the-art deep learning techniques for mitigating forgetting, despite being significantly less complicated and not requiring any knowledge of the individual tasks being learned.

catastrophic forgetting
experience replay


We use a SlowFast network with a ResNet-50 backbone #REFR , which takes in 32 images of size 224 × 224 sampled with a uniform interval from the video stream. Fig.

Thus, we tilt the gripper at an angle θ = 50 • so the grasp point is visible in the camera.

For (2), we use a triangular trajectory, where the robot gripper first moves backward, then upward, and finally forward and downward back to the original position.

To prevent the deformable object from translating as a whole, we use the robot's second gripper to pin the other side. See Figure 3 for a visualization. In our implementation, the trajectory takes about 5 secs.

While the robot executes the trajectory T , the camera takes an RGB video stream of the bag.

A video classification model M takes the video and classifies the grasp into 3 categories: 0 layer, 1 layer, and 2 layers.

3 illustrates the visual differences among different layers grasped on a plastic bag.

Given the model classification, SLIP adjusts the gripper height and retries the grasp if it does not successfully grasp a single layer.

We choose to use a fixed height adjustment each time which is similar to the strategy in #OTHEREFR , with height deltas ∆h − and ∆h + .

One could also choose to let the adjustment height decay over time or use the bisection method, but we empirically find the numbers of trials these approaches take are similar while a fixed height adjustment is more robust to a long sequence of iterations than a decaying step size, and more robust to minor bag state changes between grasps and model classification errors than bisection, since once bisection enters into a wrong interval, it may not succeed. The pseudocode of SLIP is provided in Algorithm 1.

Bagging By Learning To Singulate Layers Using Interactive Perception

Abstract: Many fabric handling and 2D deformable material tasks in homes and industry require singulating layers of material such as opening a bag or arranging garments for sewing. In contrast to methods requiring specialized sensing or end effectors, we use only visual observations with ordinary parallel jaw grippers. We propose SLIP: Singulating Layers using Interactive Perception, and apply SLIP to the task of autonomous bagging. We develop SLIP-Bagging, a bagging algorithm that manipulates a plastic or fabric bag from an unstructured state, and uses SLIP to grasp the top layer of the bag to open it for object insertion. In physical experiments, a YuMi robot achieves a success rate of 67% to 81% across bags of a variety of materials, shapes, and sizes, significantly improving in success rate and generality over prior work. Experiments also suggest that SLIP can be applied to tasks such as singulating layers of folded cloth and garments. Supplementary material is available at https://sites.google.com/view/slip-bagging/.

Slowfast Networks For Video Recognition

Abstract: We present SlowFast networks for video recognition. Our model involves (i) a Slow pathway, operating at low frame rate, to capture spatial semantics, and (ii) a Fast pathway, operating at high frame rate, to capture motion at fine temporal resolution. The Fast pathway can be made very lightweight by reducing its channel capacity, yet can learn useful temporal information for video recognition. Our models achieve strong performance for both action classification and detection in video, and large improvements are pin-pointed as contributions by our SlowFast concept. We report state-of-the-art accuracy on major video recognition benchmarks, Kinetics, Charades and AVA. Code has been made available at: https://github.com/facebookresearch/SlowFast.


This is a variant of GRU4Rec that incorporates a pre-trained LLM model BERT #REFR to leverage the text descriptions of items.

We use the version of DROS implemented on GRU4Rec, provided by the authors 3 .

The default implementation of these methods relies solely on item ID information to construct the recommender model.

However, in our setting, we assume item text descriptions are available, and our method would utilize these text descriptions to generate recommendations.

To ensure fair comparisons between our approach and the baseline methods, we further consider comparing the following variants of GRU4Rec and DROS:

Specifically, we feed the item text descriptions into BERT and concatenate the resulting CLS embedding, which encodes text descriptions, with the initial item embedding of GRU4Rec.

We take the combined embedding as the item representation used by GRU4Rec to generate recommendation predictions.

This variant of DROS is similarly modified to incorporate BERT, allowing it to utilize the text information of items.

Tallrec: An Effective And Efficient Tuning Framework To Align Large Language Model With Recommendation

Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across diverse domains, thereby prompting researchers to explore their potential for use in recommendation systems. Initial attempts have leveraged the exceptional capabilities of LLMs, such as rich knowledge and strong generalization through In-context Learning, which involves phrasing the recommendation task as prompts. Nevertheless, the performance of LLMs in recommendation tasks remains suboptimal due to a substantial disparity between the training tasks for LLMs and recommendation tasks, as well as inadequate recommendation data during pre-training. To bridge the gap, we consider building a Large Recommendation Language Model by tunning LLMs with recommendation data. To this end, we propose an efficient and effective Tuning framework for Aligning LLMs with Recommendation, namely TALLRec. We have demonstrated that the proposed TALLRec framework can significantly enhance the recommendation capabilities of LLMs in the movie and book domains, even with a limited dataset of fewer than 100 samples. Additionally, the proposed framework is highly efficient and can be executed on a single RTX 3090 with LLaMA-7B. Furthermore, the fine-tuned LLM exhibits robust cross-domain generalization. Our code and data are available at https://github.com/SAI990323/TALLRec.


This variability aligns with the finding of Fawaz et al., who made InceptionTime an ensemble of Inception networks due to the high variability of a single Inception network #REFR .

For example, the tuned Incep-tionTime tremor classifier features 7499 trainable parameters, whereas the default four-class InceptionTime includes 490 564 trainable parameters, but each produces similar scores across two different sensors (GENEActiv and Shimmer).

This comparable performance on the test data before and after hyperparameter tuning may indicate that the hyperparameter search leads to overfitting.

Alternatively, hyperparameters-with the exception of window length-may not substantially impact the InceptionTime performance, as seen in Section II-C.

After all, hyperparameter optimization reliably selects long windows; the hyperparameter search might randomly set the other hyperparameters because they do not matter.

5) InceptionTime has a high score variability: InceptionTime demonstrates greater score variability between training runs compared with other approaches, independent of whether automated hyperparameter tuning has been performed.

However, we show that even ensembling cannot fully compensate for the variability of Inception networks for TSC.

InceptionTime's sensitivity to random initialization may also contribute to the futility of automated hyperparameter optimization.

If InceptionTime reacts to several thousand trainable parameters that cannot be fully optimized by training, the impact of four additional hyperparameters may be negligible.

This is evidenced by the large standard deviations of the model scores from the cross-validation (see Section II-C.3).

6) ROCKET is promising, especially for dyskinesia: For dyskinesia, ROCKET has the highest mean AP and is the only classifier that substantially outperforms a RC.

Time Series Classification For Detecting Parkinson'S Disease From Wrist Motions

Abstract: Parkinson's disease (PD) is a neurodegenerative disease with frequently changing motor symptoms where continuous symptom monitoring enables more targeted treatment. Classical time series classification (TSC) and deep learning techniques have limited performance for PD symptom monitoring using wearable accelerometer data because PD movement patterns are complex, but datasets are small. We investigate InceptionTime and RandOm Convolutional KErnel Transform (ROCKET) because they are state-of-the-art for TSC and promising for PD symptom monitoring: InceptionTime's high learning capacity is suited to modeling complex movement patterns while ROCKET is suited to small datasets. We used a random search to find the highest-scoring InceptionTime architecture and compared it to ROCKET with a ridge classifier and a multi-layer perceptron (MLP) on wrist motions of PD patients. We find that all approaches are suitable for estimating tremor severity and bradykinesia presence but struggle with detecting dyskinesia. ROCKET performs better for dyskinesia, whereas InceptionTime is slightly better for tremor and bradykinesia but has much higher variability in performance. Both outperform the MLP. In conclusion, both InceptionTime and ROCKET are suitable for continuous symptom monitoring, with the choice depending on the symptom of interest and desired robustness.

Inceptiontime: Finding Alexnet For Time Series Classification

Abstract: Time series classification (TSC) is the area of machine learning interested in learning how to assign labels to time series. The last few decades of work in this area have led to significant progress in the accuracy of classifiers, with the state of the art now represented by the HIVE-COTE algorithm. While extremely accurate, HIVE-COTE is infeasible to use in many applications because of its very high training time complexity in O(N^2*T^4) for a dataset with N time series of length T. For example, it takes HIVE-COTE more than 72,000s to learn from a small dataset with N=700 time series of short length T=46. Deep learning, on the other hand, has now received enormous attention because of its high scalability and state-of-the-art accuracy in computer vision and natural language processing tasks. Deep learning for TSC has only very recently started to be explored, with the first few architectures developed over the last 3 years only. The accuracy of deep learning for TSC has been raised to a competitive level, but has not quite reached the level of HIVE-COTE. This is what this paper achieves: outperforming HIVE-COTE's accuracy together with scalability. We take an important step towards finding the AlexNet network for TSC by presenting InceptionTime---an ensemble of deep Convolutional Neural Network (CNN) models, inspired by the Inception-v4 architecture. Our experiments show that InceptionTime slightly outperforms HIVE-COTE with a win/draw/loss on the UCR archive of 40/6/39. Not only is InceptionTime more accurate, but it is much faster: InceptionTime learns from that same dataset with 700 time series in 2,300s but can also learn from a dataset with 8M time series in 13 hours, a quantity of data that is fully out of reach of HIVE-COTE.


In this research, the MobileNetV2 algorithm in the SSDLite framework #REFR was used to detect and determine the 'image's human position in real-time.

The MobileNet neural network architecture has been designed specifically for limited hardware resources such as mobiles.

With a dramatic decrease in the number of parameters and memory required, maintaining the same accuracy has become an advanced state-of-the-art model in computer vision for mobiles #OTHEREFR . In the following section, its architecture is introduced.

Table 2 Shows the MobileNetV2 architecture, which contains the initial fully convolution layer with 32 filters, followed by 19 residual bottleneck layers described in Table 3 #OTHEREFR .

In this architecture, the ReLU6 activation function is used as the non-linearity because of its robustness when used with low-precision computation #OTHEREFR , #OTHEREFR .

Table 3 MobileNetV2: Each line describes a sequence of 1 or more identical layers, repeated n times.

Deeplt: Personalized Lighting Facilitates By Trajectory Prediction Of Recognized Residents In The Smart Home

Abstract: In recent years, the intelligence of various parts of the home has become one of the essential features of any modern home. One of these parts is the intelligence lighting system that personalizes the light for each person. This paper proposes an intelligent system based on machine learning that personalizes lighting in the instant future location of a recognized user, inferred by trajectory prediction. Our proposed system consists of the following modules: (I) human detection to detect and localize the person in each given video frame, (II) face recognition to identify the detected person, (III) human tracking to track the person in the sequence of video frames and (IV) trajectory prediction to forecast the future location of the user in the environment using Inverse Reinforcement Learning. The proposed method provides a unique profile for each person, including specifications, face images, and custom lighting settings. This profile is used in the lighting adjustment process. Unlike other methods that consider constant lighting for every person, our system can apply each 'person's desired lighting in terms of color and light intensity without direct user intervention. Therefore, the lighting is adjusted with higher speed and better efficiency. In addition, the predicted trajectory path makes the proposed system apply the desired lighting, creating more pleasant and comfortable conditions for the home residents. In the experimental results, the system applied the desired lighting in an average time of 1.4 seconds from the moment of entry, as well as a performance of 22.1mAp in

Mobilenetv2: Inverted Residuals And Linear Bottlenecks

Abstract: In this paper we describe a new mobile architecture, MobileNetV2, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We also describe efficient ways of applying these mobile models to object detection in a novel framework we call SSDLite. Additionally, we demonstrate how to build mobile semantic segmentation models through a reduced form of DeepLabv3 which we call Mobile DeepLabv3. The MobileNetV2 architecture is based on an inverted residual structure where the input and output of the residual block are thin bottleneck layers opposite to traditional residual models which use expanded representations in the input an MobileNetV2 uses lightweight depthwise convolutions to filter features in the intermediate expansion layer. Additionally, we find that it is important to remove non-linearities in the narrow layers in order to maintain representational power. We demonstrate that this improves performance and provide an intuition that led to this design. Finally, our approach allows decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis. We measure our performance on Imagenet classification, COCO object detection, VOC image segmentation. We evaluate the trade-offs between accuracy, and number of operations measured by multiply-adds (MAdd), as well as the number of parameters


Federated Learning #REFR is a technique that allows for a central model to be trained over distributed data that originate and are stored locally.

This approach was originally proposed within the setting of large-scale distributed learning on mobile devices and has been widely used and researched in the medical sector given the privacy enhancing aspect as well a the regulatory restrictions on data sharing within the sector.

Each local data source, e.g., a mobile phone or a server hosted at a hospital, is referred to as a client.

The group of clients that contribute to training a central model, including the central server is referred to as the federation.

The central server orchestrates the learning process over the different clients following the predefined FL algorithm.

In that sense, the central server is holistically responsible for (i) initiating the model; (ii) providing a copy of the model to all of the clients; (iii) aggregating over the received model parameters and (iv) send back the updated model to the clients.

Client Recruitment For Federated Learning In Icu Length Of Stay Prediction

Abstract: Background: Machine and deep learning methods for medical and healthcare applications have shown significant progress and performance improvement in recent years. These methods require vast amounts of training data which are available in the medical sector, albeit decentralized. Medical institutions generate vast amounts of data for which sharing and centralizing remains a challenge as the result of data and privacy regulations. The federated learning technique is well-suited to tackle these challenges. However, federated learning comes with a new set of open problems related to communication overhead, efficient parameter aggregation, client selection strategies and more. Methods: In this work, we address the step prior to the initiation of a federated network for model training, client recruitment. By intelligently recruiting clients, communication overhead and overall cost of training can be reduced without sacrificing predictive performance. Client recruitment aims at pre-excluding potential clients from partaking in the federation based on a set of criteria indicative of their eventual contributions to the federation. In this work, we propose a client recruitment approach using only the output distribution and sample size at the client site. Results: We show how a subset of clients can be recruited without sacrificing model performance whilst, at the same time, significantly improving computation time. By applying the recruitment approach to the training of federated models for accurate patient Length of Stay prediction using data from 189 Intensive Care Units, we show how the models trained in federations made up from recruited clients significantly outperform federated models trained with the standard procedure in terms of predictive power and training time.

Communication-Efficient Learning Of Deep Networks From Decentralized Data

Abstract: Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.


For example, we could generate a graph using -nearest neighbours methods and define a criterion using the degrees of nodes #REFR .

It must be noted that hierarchical clustering methods may also be based on graphs.

The single-linkage criterion is relatively naive and often does not produce suitable partitions. The remaining ones have higher time complexity.

One way to bypass the complexity problem is to add additional requirements.

For example, the Genie algorithm #OTHEREFR uses a single-linkage criterion but simultaneously monitors the partition's Gini Index value, i.e., a measure of inequality of cluster sizes.

Only the cluster of the smallest cardinality can be merged when the algorithm reaches a pre-set threshold of the Gini Index value.

Community Detection In Complex Networks Via Node Similarity, Graph Representation Learning, And Hierarchical Clustering

Abstract: Community detection is a critical challenge in the analysis of real-world graphs and complex networks, including social, transportation, citation, cybersecurity networks, and food webs. Motivated by many similarities between community detection and clustering in Euclidean spaces, we propose three algorithm frameworks to apply hierarchical clustering methods for community detection in graphs. We show that using our methods, it is possible to apply various linkagebased (single-, complete-, average-linkage, Ward, Genie) clustering algorithms to find communities based on vertex similarity matrices, eigenvector matrices thereof, and Euclidean vector representations of nodes. We convey a comprehensive analysis of choices for each framework, including state-of-the-art graph representation learning algorithms, such as Deep Neural Graph Representation, and a vertex proximity matrix known to yield high-quality results in machine learning-Positive Pointwise Mutual Information. Overall, we test over a hundred combinations of framework components and show that some-including Wasserman-Faust and PPMI proximity, DNGR representation-can compete with algorithms such as state-of-the-art Leiden and Louvain and easily outperform other known community detection algorithms. Notably, our algorithms remain hierarchical and allow the user to specify any number of clusters a priori.

Graph Degree Linkage: Agglomerative Clustering On A Directed Graph

Abstract: This paper proposes a simple but effective graph-based agglomerative algorithm, for clustering high-dimensional data. We explore the different roles of two fundamental concepts in graph theory, indegree and outdegree, in the context of clustering. The average indegree reflects the density near a sample, and the average outdegree characterizes the local geometry around a sample. Based on such insights, we define the affinity measure of clusters via the product of average indegree and average outdegree. The product-based affinity makes our algorithm robust to noise. The algorithm has three main advantages: good performance, easy implementation, and high computational efficiency. We test the algorithm on two fundamental computer vision problems: image clustering and object matching. Extensive experiments demonstrate that it outperforms the state-of-the-arts in both applications.


In the next step, we extract the features using a Wavelet Scattering Network (WSN) #REFR independently on each filtered image.

We perform the Laplacian filtering independently on color channels that will results in

L Y s , L B s , L R s and L Y t , L B t , L R t

Figure 3 shows the qualitative results of the Laplacian filtering results of the example face image.

In this work, we construct a two-layer image scatter network with a 250 × 250 pixels with invariant scale such that two wavelets per octave in the first layer and one wavelet per octave in the second layer. Furthermore, we use six rotations of wavelets per layer.

Therefore, the WSN used in this work has 577 paths.

Let the WSN computed for Laplacian filtered images be denoted as:

F D Y = W Y s − W Y t , F D Y = W B s − W B t and F D Y = W R s − W R t .

Differential Newborn Face Morphing Attack Detection Using Wavelet Scatter Network

Abstract: Face Recognition System (FRS) are shown to be vulnerable to morphed images of newborns. Detecting morphing attacks stemming from face images of newborn is important to avoid unwanted consequences, both for security and society. In this paper, we present a new reference-based/Differential Morphing Attack Detection (MAD) method to detect newborn morphing images using Wavelet Scattering Network (WSN). We propose a two-layer WSN with 250 × 250 pixels and six rotations of wavelets per layer, resulting in 577 paths. The proposed approach is validated on a dataset of 852 bona fide images and 2460 morphing images constructed using face images of 42 unique newborns. The obtained results indicate a gain of over 10% in detection accuracy over other existing D-MAD techniques.

Deep Scattering Spectrum

Abstract: A scattering transform defines a locally translation invariant representation which is stable to time-warping deformation. It extends MFCC representations by computing modulation spectrum coefficients of multiple orders, through cascades of wavelet convolutions and modulus operators. Second-order scattering coefficients characterize transient phenomena such as attacks and amplitude modulation. A frequency transposition invariant representation is obtained by applying a scattering transform along log-frequency. State-the-of-art classification results are obtained for musical genre and phone classification on GTZAN and TIMIT databases, respectively.


For CivilComments we report the average and worst accuracy in Table 3 , where we compare with ERM and groupDRO #REFR . While performing almost on par w.r.t.

Our claim is that a feature space that satisfies sufficiency and minimality is more robust to spurious correlations which may affect minority groups, and should transfer better to new distributions.

To validate this, we test on two benchmarks Waterbirds #OTHEREFR , and CivilComments #OTHEREFR ) (see Appendix C.1).

For both, we use the train and test split of the original dataset.

In Table 5 we report the results on the test set of Waterbirds for the different groups in the dataset (landbirds on land, landbirds on water, waterbirds on land, and waterbirds on water, respectively).

We fit the linear head on a random subset of the training domain, balanced by class, repeat 10 times and report accuracy and standard deviation on test.

ERM, our method is more robust to spurious correlation in the dataset, showing the higher worst group accuracy.

Importantly, we outperform GroupDRO, which uses information on the subdomain statistics, while we do not assume any prior knowledge about them.

Results per group are reported in the Appendix (Table 9) .

We evaluate the domain generalization performance on the PACS, VLCS and OfficeHome datasets from the DomainBed #OTHEREFR ) test suite (see Appendix C.1 for more details).

Leveraging Sparse And Shared Feature Activations For Disentangled Representation Learning

Abstract: Recovering the latent factors of variation of high dimensional data has so far focused on simple synthetic settings. Mostly building on unsupervised and weakly-supervised objectives, prior work missed out on the positive implications for representation learning on real world data. In this work, we propose to leverage knowledge extracted from a diversified set of supervised tasks to learn a common disentangled representation. Assuming each supervised task only depends on an unknown subset of the factors of variation, we disentangle the feature space of a supervised multi-task model, with features activating sparsely across different tasks and information being shared as appropriate. Importantly, we never directly observe the factors of variations but establish that access to multiple tasks is sufficient for identifiability under sufficiency and minimality assumptions. We validate our approach on six real world distribution shift benchmarks, and different data modalities (images, text), demonstrating how disentangled representations can be transferred to real settings.

Distributionally Robust Neural Networks For Group Shifts: On The Importance Of Regularization For Worst-Case Generalization

Abstract: Overparameterized neural networks can be highly accurate on average on an i.i.d. test set yet consistently fail on atypical groups of the data (e.g., by learning spurious correlations that hold on average but not in such groups). Distributionally robust optimization (DRO) allows us to learn models that instead minimize the worst-case training loss over a set of pre-defined groups. However, we find that naively applying group DRO to overparameterized neural networks fails: these models can perfectly fit the training data, and any model with vanishing average training loss also already has vanishing worst-case training loss. Instead, the poor worst-case performance arises from poor generalization on some groups. By coupling group DRO models with increased regularization---a stronger-than-typical L2 penalty or early stopping---we achieve substantially higher worst-group accuracies, with 10-40 percentage point improvements on a natural language inference task and two image tasks, while maintaining high average accuracies. Our results suggest that regularization is important for worst-group generalization in the overparameterized regime, even if it is not needed for average generalization. Finally, we introduce a stochastic optimization algorithm, with convergence guarantees, to efficiently train group DRO models.


Reference #REFR input text and using extra layers to conduct classification has gradually become a popular approach to solve multilingual QA task.

There are massive datasets available, and these datasets pose different challenges to QA algorithms #OTHEREFR , #OTHEREFR , #OTHEREFR .

One of these challenges is to find the answer span when the context and the problem are in different languages, and this task is called generalized crosslingual transfer(G-XLT).

More formally, cross-lingual transfer problem requires a model to identify answer a x in context c x according to problem q x where x is the language used.

Meanwhile, generalized cross-lingual transfer requires a model to find the answer span a z in context c z according to question q y where z and y are languages used.

Extensive experiments using state-of-the-art PLMs are reported in #OTHEREFR .

Their results show that there are quite large room for improvement on both XLT and G-XLT tasks.

One of the major challenges in G-XLT task is to enhance the connection between different languages.

Reference #OTHEREFR uses hyperlink prediction task to firstly enhance the connection between different representations and then finetune the model on the target task data in English. Finally, they evaluate the performance on all languages.

It is worth noting that this method creatively uses hyperlink as the bridge of different languages.

Bridging The Language Gap: Knowledge Injected Multilingual Question Answering

Abstract: Question Answering (QA) is the task of automatically answering questions posed by humans in natural languages. There are different settings to answer a question, such as abstractive, extractive, boolean, and multiple-choice QA. As a popular topic in natural language processing tasks, extractive question answering task (extractive QA) has gained extensive attention in the past few years. With the continuous evolvement of the world, generalized cross-lingual transfer (G-XLT), where question and answer context are in different languages, poses some unique challenges over cross-lingual transfer (XLT), where question and answer context are in the same language. With the boost of corresponding development of related benchmarks, many works have been done to improve the performance of various language QA tasks. However, only a few works are dedicated to the G-XLT task. In this work, we propose a generalized crosslingual transfer framework to enhance the model's ability to understand different languages. Specifically, we first assemble triples from different languages to form multilingual knowledge. Since the lack of knowledge between different languages greatly limits models' reasoning ability, we further design a knowledge injection strategy via leveraging link prediction techniques to enrich the model storage of multilingual knowledge. In this way, we can profoundly exploit rich semantic knowledge. Experiment results on real-world datasets MLQA demonstrate that the proposed method can improve the performance by a large margin, outperforming the baseline method by 13.18%/12.00% F1/EM on average. Our code and models will be open-sourced.

Mlqa: Evaluating Cross-Lingual Extractive Question Answering

Abstract: Question answering (QA) models have shown rapid progress enabled by the availability of large, high-quality benchmark datasets. Such annotated datasets are difficult and costly to collect, and rarely exist in languages other than English, making training QA systems in other languages challenging. An alternative to building large monolingual training datasets is to develop cross-lingual systems which can transfer to a target language without requiring training data in that language. In order to develop such systems, it is crucial to invest in high quality multilingual evaluation benchmarks to measure progress. We present MLQA, a multi-way aligned extractive QA evaluation benchmark intended to spur research in this area. MLQA contains QA instances in 7 languages, namely English, Arabic, German, Spanish, Hindi, Vietnamese and Simplified Chinese. It consists of over 12K QA instances in English and 5K in each other language, with each QA instance being parallel between 4 languages on average. MLQA is built using a novel alignment context strategy on Wikipedia articles, and serves as a cross-lingual extension to existing extractive QA datasets. We evaluate current state-of-the-art cross-lingual representations on MLQA, and also provide machine-translation-based baselines. In all cases, transfer results are shown to be significantly behind training-language performance.


Most DL architectures are based on Convolutional Neural Networks (CNNs) and more recently, CNNs have been hybridized with transformer networks #REFR .

Another paradigm of LLIE makes use of Retinex theory, where the assumption is that a color image can be separated into reflectance and illumination #OTHEREFR .

Non-learning Retinexbased methods such as Low-light Image Enhancement via Illumination Map Estimation (LIME) #OTHEREFR provide an effective image enhancement approach; however, post-processing denoising is typically still necessary using algorithms such as BM3D #OTHEREFR which often blurs high-frequency details.

Alternative Retinex-based methods reformulate the traditional Retinex model to incorporate an added noise term in order to cater for noise #OTHEREFR .

DL methods have recently achieved state-of-the-art LLIE performance.

Some complexities of catering for a large variety of realistic low-light scenes are abstracted away by a datadriven approach.

DL methods either opt for incorporating denoising into a single model, or apply denoising as a post-processing step.

S-LLNet #OTHEREFR makes use of a learned denoiser which operates sequentially after contrast enhancement.

Retinex-net #OTHEREFR incorporates Retinex theory into a DL model with an optional illumination-varying BM3D denoiser used as post-processing.

Raw sensor data are enhanced and denoised via CNN in #OTHEREFR .

A variety of loss functions have also been proposed (in addition to the typical l 1 and l 2 losses) which further penalize networks based on color, smoothness, brightness and perceptual interpretation #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR - #OTHEREFR .

Denoising Diffusion Post-Processing For Low-Light Image Enhancement

Abstract: Low-light image enhancement (LLIE) techniques attempt to increase the visibility of images captured in lowlight scenarios. However, as a result of enhancement, a variety of image degradations such as noise and color bias are revealed. Furthermore, each particular LLIE approach may introduce a different form of flaw within its enhanced results. To combat these image degradations, post-processing denoisers have widely been used, which often yield oversmoothed results lacking detail. We propose using a diffusion model as a post-processing approach, and we introduce Low-light Post-processing Diffusion Model (LPDM) in order to model the conditional distribution between underexposed and normally-exposed images. We apply LPDM in a manner which avoids the computationally expensive generative reverse process of typical diffusion models, and post-process images in one pass through LPDM. Extensive experiments demonstrate that our approach outperforms competing post-processing denoisers by increasing the perceptual quality of enhanced lowlight images on a variety of challenging low-light datasets.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


For CIFAR-10 dataset, we train the All-CNN model #REFR from scratch for 30 epochs using SGD with a fixed learning rate of 0.1, momentum of 0.9 and batch-size of 64.

Amnesiac Unlearning #OTHEREFR : amnesiac unlearning is a typical method of rapidly retrain.

We need save the updates of parameters during several batches on the original training phase.

Then, we can unlearn the forgetting data by subtracting the corresponding updates of parameters. Implementations.

Our methods and other baselines are implemented in Python 3.8 and use the PyTorch library #OTHEREFR .

All experiments are conducted on a workstation with one NVIDIA GeForce RTX 2070 GPU.

For Vggface2 dataset, we randomly select face images of 10 celebrities to conduct experiments.

We obtain the original model by finetuning the pretrained ResNet50 model #OTHEREFR with the training images of the 10 celebrities.

The original training parameters are similar to those on CIFAR-10.

For the fine-tune process in Boundary Unlearning we use a learning rate of 10 −5 for 10 epochs.

Note that we use the same forgetting class as the forgetting data for all unlearning methods and the rest of classes are the remaining data. Metrics.

Boundary Unlearning

Abstract: The practical needs of the "right to be forgotten" and poisoned data removal call for efficient machine unlearning techniques, which enable machine learning models to unlearn, or to forget a fraction of training data and its lineage. Recent studies on machine unlearning for deep neural networks (DNNs) attempt to destroy the influence of the forgetting data by scrubbing the model parameters. However, it is prohibitively expensive due to the large dimension of the parameter space. In this paper, we refocus our attention from the parameter space to the decision space of the DNN model, and propose Boundary Unlearning, a rapid yet effective way to unlearn an entire class from a trained DNN model. The key idea is to shift the decision boundary of the original DNN model to imitate the decision behavior of the model retrained from scratch. We develop two novel boundary shift methods, namely Boundary Shrink and Boundary Expanding, both of which can rapidly achieve the utility and privacy guarantees. We extensively evaluate Boundary Unlearning on CIFAR-10 and Vggface2 datasets, and the results show that Boundary Unlearning can effectively forget the forgetting class on image classification and face recognition tasks, with an expected speed-up of 17× and 19×, respectively, compared with retraining from the scratch.

Striving For Simplicity: The All Convolutional Net

Abstract: Most modern convolutional neural networks (CNNs) used for object recognition are built using the same principles: Alternating convolution and max-pooling layers followed by a small number of fully connected layers. We re-evaluate the state of the art for object recognition from small images with convolutional networks, questioning the necessity of different components in the pipeline. We find that max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks. Following this finding -- and building on other recent work for finding simple network structures -- we propose a new architecture that consists solely of convolutional layers and yields competitive or state of the art performance on several object recognition datasets (CIFAR-10, CIFAR-100, ImageNet). To analyze the network we introduce a new variant of the "deconvolution approach" for visualizing features learned by CNNs, which can be applied to a broader range of network structures than existing approaches.


Phase gadgets have been characterized thus far in two graphical calculi: in the qubit ZX-calculus #REFR in Ref.

Our approach to controlling the gate is to control each component and phase gate in Equation #OTHEREFR .

We apply strategies from past work: generalizing controlled phase gates from qubits to qutrits #OTHEREFR and controlling the qutrit gate #OTHEREFR .

In future work we will consider generalizing to higher dimensions, adding any number of controls to qutrit gates #OTHEREFR .

The key structural insight leverages the generalization to arbitrary dimensions of phase gadgets, a symmetric multi-qudit interaction which applies a phase dependent on the sum of all computational basis states modulo .

#OTHEREFR , and in the qutrit ZX-calculus #OTHEREFR in Ref. #OTHEREFR .

We remark that phase gadgets are broadly applicable to all qudit dimensions as a graphical construct to relate diagonal gates in the circuit model and polynomial functions.

Hence, this gives us an idea of what sort of circuit structure may be a good place to start.

Keeping minimal on introducing formalism of the qudit -calculus #OTHEREFR , we introduce non-unitary circuit elements which respectively act as addition modulo and copying of basis states, labelled | ⟩ and | ⟩ below:

Scaling W State Circuits In The Qudit Clifford Hierarchy

Abstract: We identify a novel qudit gate which we call the √ gate. This is an alternate generalization of the qutrit gate to any odd prime dimension , in the th level of the Clifford hierarchy. Using this gate which is efficiently realizable fault-tolerantly should a certain conjecture hold, we deterministically construct in the Clifford+ √ gate set,-qubit states in the qudit {|0⟩ , |1⟩} subspace. For qutrits, this gives deterministic and fault-tolerant constructions for the qubit state of sizes three with count 3, six, and powers of three. Furthermore, we adapt these constructions to recursively scale the state size to arbitrary size , in () gate count and (log) depth. This is moreover deterministic for any size qubit state, and for any prime-dimensional qudit state, size a power of. For these purposes, we devise constructions of the |0⟩-controlled Pauli gate and the controlled Hadamard gate in any prime qudit dimension. These decompositions, for which exact synthesis is unknown in Clifford+ for > 3, may be of independent interest. CCS CONCEPTS • Hardware → Quantum communication and cryptography; Quantum error correction and fault tolerance; • Computer systems organization → Quantum computing.

Interacting Quantum Observables: Categorical Algebra And Diagrammatics

Abstract: This paper has two tightly intertwined aims: (i) to introduce an intuitive and universal graphical calculus for multi-qubit systems, the ZX-calculus, which greatly simplifies derivations in the area of quantum computation and information. (ii) To axiomatize complementarity of quantum observables within a general framework for physical theories in terms of dagger symmetric monoidal categories. We also axiomatize phase shifts within this framework. Using the well-studied canonical correspondence between graphical calculi and dagger symmetric monoidal categories, our results provide a purely graphical formalisation of complementarity for quantum observables. Each individual observable, represented by a commutative special dagger Frobenius algebra, gives rise to an Abelian group of phase shifts, which we call the phase group. We also identify a strong form of complementarity, satisfied by the Z- and X-spin observables, which yields a scaled variant of a bialgebra.


In causal discovery, larger graphs are typically more challenging to learn, as the number of possible graphs grows super exponentially with the number of nodes #REFR . The results are shown in Figure 6 .

Specifically, we addressed the following questions: 1) What is the impact of changing the number of replicates K n ? 2) What is the impact of changing the number of genes N ?

To answer the first question, we varied the number of replicates per perturbation from the default setting of 500 to 100, 200, 300, and 400.

The results, shown in Figure 5 , indicate that reducing the number of replicates from 500 to 300 only slightly decreases performance, while a more significant reduction is observed with only 100 replicates.

These findings suggest that having more replicates generally improves performance, although the benefits are marginal beyond 300 replicates.

To answer the second question, we varied the number of genes from 50 to 100.

We can see that DiscoGen's performance worsen only marginally as the number of genes increases.

This gives a strong indication that DiscoGen is reasonably robust to different number of genes.

Discogen: Learning To Discover Gene Regulatory Networks

Abstract: Accurately inferring Gene Regulatory Networks (GRNs) is a critical and challenging task in biology. GRNs model the activatory and inhibitory interactions between genes and are inherently causal in nature. To accurately identify GRNs, perturbational data is required. However, most GRN discovery methods only operate on observational data. Recent advances in neural network-based causal discovery methods have significantly improved causal discovery, including handling interventional data, improvements in performance and scalability. However, applying state-of-the-art (SOTA) causal discovery methods in biology poses challenges, such as noisy data and a large number of samples. Thus, adapting the causal discovery methods is necessary to handle these challenges. In this paper, we introduce DiscoGen, a neural network-based GRN discovery method that can denoise gene expression measurements and handle interventional data. We demonstrate that our model outperforms SOTA neural network-based causal discovery methods.

Causal Structure Learning

Abstract: Graphical models can represent a multivariate distribution in a convenient and accessible form as a graph. Causal models can be viewed as a special class of graphical models that represent not only the distribution of the observed system but also the distributions under external interventions. They hence enable predictions under hypothetical interventions, which is important for decision making. The challenging task of learning causal models from data always relies on some underlying assumptions. We discuss several recently proposed structure learning algorithms and their assumptions, and we compare their empirical performance under various scenarios.


B.3) go through a self-attention module, then deformable attention is conducted with the image features, where we aggregate multi-scale image features from FPN #REFR through deformable attention.

We extend Deformable-DETR #OTHEREFR to the 3D object detection task. The initial camera queries Q 0 C (Sec.

In deformable attention, each query only interacts with its corresponding single-view image features.

The output queries are fed into a feed-forward network to get the perspective view camera candidates Q P C .

As we do with the LiDAR detector, we add positional embeddings to all of the queries, keys, and values, which indicate their 2D locations on the image of the corresponding view. A perspective view prediction head (Sec.

B.2) is attached to the perspective view camera candidates Q P C to get the object category as well as the 3D bounding box in camera coordinates.

View Transformation Our view transformation module consists of two parts: feature projection and multi-view aggregation. The feature projection is already described in Eq.

Sparsefusion: Fusing Multi-Modal Sparse Representations For Multi-Sensor 3D Object Detection

Abstract: By identifying four important components of existing LiDAR-camera 3D object detection methods (LiDAR and camera candidates, transformation, and fusion outputs), we observe that all existing methods either find dense candidates or yield dense representations of scenes. However, given that objects occupy only a small part of a scene, finding dense candidates and generating dense representations is noisy and inefficient. We propose SparseFusion, a novel multi-sensor 3D detection method that exclusively uses sparse candidates and sparse representations. Specifically, SparseFusion utilizes the outputs of parallel detectors in the LiDAR and camera modalities as sparse candidates for fusion. We transform the camera candidates into the Li-DAR coordinate space by disentangling the object representations. Then, we can fuse the multi-modality candidates in a unified 3D space by a lightweight self-attention module. To mitigate negative transfer between modalities, we propose novel semantic and geometric cross-modality transfer modules that are applied prior to the modality-specific detectors. SparseFusion achieves state-of-the-art performance on the nuScenes benchmark while also running at the fastest speed, even outperforming methods with stronger backbones. We perform extensive experiments to demonstrate the effectiveness and efficiency of our modules and overall method pipeline. Our code will be made publicly available at https://github.com/yichen928/SparseFusion.

Feature Pyramid Networks For Object Detection

Abstract: Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But pyramid representations have been avoided in recent object detectors that are based on deep convolutional networks, partially because they are slow to compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using a basic Faster R-CNN system, our method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 5 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.