The basic ideas of data-free KD are to optimize noise inputs to minimize the distance to prior knowledge #REFR , and Chen et al.

In RHFL #OTHEREFR , a server calculates the weights of clients by the symmetric cross-entropy loss function, and clients distilled knowledge from the unlabeled dataset.

FCCL #OTHEREFR computed a cross-correlation matrix also based on the unlabeled public dataset.

2) Data-free knowledge distillation: However, the former methods using KD in FL acquire a public dataset.

The server may not collect sufficient data due to data availability and privacy concerns.

In contrast to the aforementioned methods, data-free KD is a novel approach to complete the knowledge distillation process without the training data.

#OTHEREFR train The process for FedIN is described as follows.

First, the extractor and the classifier are updated by the averaged weights we and wc from the server.

The second step of the clients is training their models from the local private dataset and completing the IN training for the feature inputs and outputs (s in , sout) from the server.

Generative Adversarial Networks (GANs) #OTHEREFR to generate training data for the entire KD process, utilizing the knowledge distilled from the teacher model.

To free the limitation from a public dataset, a few researches consider data-free KD in FL.

Fedin: Federated Intermediate Layers Learning For Model Heterogeneity

Abstract: Federated learning (FL) facilitates edge devices to cooperatively train a global shared model while maintaining the training data locally and privately. However, a common but impractical assumption in FL is that the participating edge devices possess the same required resources and share identical global model architecture. In this study, we propose a novel FL method called Federated Intermediate Layers Learning (FedIN), supporting heterogeneous models without utilizing any public dataset. The training models in FedIN are divided into three parts, including an extractor, the intermediate layers, and a classifier. The model architectures of the extractor and classifier are the same in all devices to maintain the consistency of the intermediate layer features, while the architectures of the intermediate layers can vary for heterogeneous devices according to their resource capacities. To exploit the knowledge from features, we propose IN training, training the intermediate layers in line with the features from other clients. Additionally, we formulate and solve a convex optimization problem to mitigate the gradient divergence problem induced by the conflicts between the IN training and the local training. The experiment results show that FedIN achieves the best performance in the heterogeneous model environment compared with the state-of-the-art algorithms. Furthermore, our ablation study demonstrates the effectiveness of IN training and the solution to the convex optimization problem.

Zero-Shot Knowledge Distillation In Deep Networks

Abstract: Knowledge distillation deals with the problem of training a smaller model (Student) from a high capacity source model (Teacher) so as to retain most of its performance. Existing approaches use either the training data or meta-data extracted from it in order to train the Student. However, accessing the dataset on which the Teacher has been trained may not always be feasible if the dataset is very large or it poses privacy or safety concerns (e.g., bio-metric or medical data). Hence, in this paper, we propose a novel data-free method to train the Student from the Teacher. Without even using any meta-data, we synthesize the Data Impressions from the complex Teacher model and utilize these as surrogates for the original training data samples to transfer its learning to Student via knowledge distillation. We, therefore, dub our method "Zero-Shot Knowledge Distillation" and demonstrate that our framework results in competitive generalization performance as achieved by distillation using the actual training data samples on multiple benchmark datasets.


We used the SDSL library #REFR to implement our corpus indexer. We index each corpus separately.

(2016) and use the FM-Index #OTHEREFR that utilises the text compressibility vi the Burrows-Wheeler transformation (BWT) #OTHEREFR of the text. The BWT is defined as, BWT

Searching for a sequence in BWT is done in reverse order and requires O(|u| log |σ|).

For more details on BWT and reverse searching, refer to #OTHEREFR .

The CSA is at the core of Koala's index and search backbone.

Once a corpus is indexed, its constructed index sits on disk and could be queried through the Koala web interface (introduced shortly).

Each query is launched into the indexed collection of corpora and returns the hit counts of the query in the corresponding corpus.

Table 1 reports the time and memory usage for construction of indexes.

Koala: An Index For Quantifying Overlaps With Pre-Training Corpora

Abstract: In very recent years more attention has been placed on probing the role of pre-training data in Large Language Models (LLMs) downstream behaviour. Despite the importance, there is no public tool that supports such analysis of pre-training corpora at large scale. To help research in this space, we launch Koala, a searchable index over large pretraining corpora using compressed suffix arrays with highly efficient compression rate and search support. In its first release we index the public proportion of OPT 175B pretraining data. Koala provides a framework to do forensic analysis on the current and future benchmarks as well as to assess the degree of memorization in the output from the LLMs. Koala is available for public use at https: //koala-index.erc.monash.edu/ * Corresponding author 1 We plan to extend our coverage of pre-training corpora.

From Theory To Practice: Plug And Play With Succinct Data Structures

Abstract: Engineering efficient implementations of compact and succinct structures is time-consuming and challenging, since there is no standard library of easy-to-use, highly optimized, and composable components. One consequence is that measuring the practical impact of new theoretical proposals is difficult, since older baseline implementations may not rely on the same basic components, and reimplementing from scratch can be time-consuming. In this paper we present a framework for experimentation with succinct data structures, providing a large set of configurable components, together with tests, benchmarks, and tools to analyze resource requirements. We demonstrate the functionality of the framework by recomposing two succinct solutions for top-k document retrieval which can operate on both character and integer alphabets.


For the text modality, we utilize WordPiece #REFR to generate a vocabulary, and randomly initialize a matrix E ∈ R n E ×d E for word embeddings, where n E denotes the length of the vocabulary, d E represents the feature dimension. E is optimized during training.

We extract features from multimodal information as well as MBTI personality information.

For V, we use the same feature extraction method as HERO #OTHEREFR .

That is, employ pre-training 2D ResNet-152 #OTHEREFR and 3D SlowFast #OTHEREFR to extract video feature expressions V 2D ∈ R n V ×d 2D and V 3D ∈ R n V ×d 3D .

To acquire the final feature V ∈ R n V ×d V , we concatenate V 2D and V 3D and lower the dimension through a linear layer, where n V represents the length of image frame sequence, and d V represents the feature dimension.

In particular, features obtained from various information are expressed as follows: D ∈ R n D ×l D ×d E for dialogue information, B ∈ R n B ×l B ×d E for behavior description, and A ∈ R n A ×l A ×d E for multiple choice option.

Here, n D denotes the number of dialogue utterances, n B expresses the number of behavior descriptions, n A represents the number of multiple choice options, l D , l B and l A signify the sentence length, respectively.

To represent P, we concatenate the name tags and the MBTI personalities of characters into phrases and use E to retrieve the personality feature P ∈ R n P ×l P ×d E .

Here, n P denotes the number of relevant characters in VC, and l P represents the phrases length.

To enhance the personality feature, we concatenate the initial personality feature with the feature through the Self-Attention module to obtain final P C by

Personality-Aware Human-Centric Multimodal Reasoning: A New Task

Abstract: Multimodal reasoning, an area of artificial intelligence that aims at make inferences from multimodal signals such as vision, language and speech, has drawn more and more attention in recent years. People with different personalities may respond differently to the same situation. However, such individual personalities were ignored in the previous studies. In this work, we introduce a new Personality-aware Human-centric Multimodal Reasoning (Personality-aware HMR) task, and accordingly construct a new dataset based on The Big Bang Theory television shows, to predict the behavior of a specific person at a specific moment, given the multimodal information of its past and future moments. The Myers-Briggs Type Indicator (MBTI) was annotated and utilized in the task to represent individuals' personalities. We benchmark the task by proposing three baseline methods, two were adapted from the related tasks and one was newly proposed for our task. The experimental results demonstrate that personality can effectively improve the performance of human-centric multimodal reasoning. To further solve the lack of personality annotation in real-life scenes, we introduce an extended task called Personality-predicted HMR, and propose the corresponding methods, to predict the MBTI personality at first, and then use the predicted personality to help multimodal reasoning. The experimental results show that our method can accurately predict personality and achieves satisfactory multimodal reasoning performance without relying on personality annotations.

Google'S Neural Machine Translation System: Bridging The Gap Between Human And Machine Translation

Abstract: Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system.


Cityscapes #REFR is a high-resolution (1024×2048 pixels) street-view dataset that contains 2975 training, 500 validation, and 1525 testing images.

We evaluate the performance of FastInst in terms of instance segmentation AP over eight semantic classes of the dataset. Training settings.

We use a batch size of 16 and train the model for 90K iterations.

We set the initial learning rate as 0.0001 and drop it by multiplying 0.1 at 0.9 and 0.95 fractions of the total number of training steps.

During training, we randomly resize the image to a shorter edge from 800 to 1024 pixels with a step of 32 pixels, followed by a crop size of 512×1024.

During inference, we operate on the full image with a resolution of 1024×2048. Results.

Fastinst: A Simple Query-Based Model For Real-Time Instance Segmentation

Abstract: Recent attention in instance segmentation has focused on query-based models. Despite being non-maximum suppression (NMS)-free and end-to-end, the superiority of these models on high-accuracy real-time benchmarks has not been well demonstrated. In this paper, we show the strong potential of query-based models on efficient instance segmentation algorithm designs. We present FastInst, a simple, effective query-based framework for real-time instance segmentation. FastInst can execute at a real-time speed (i.e., 32.5 FPS) while yielding an AP of more than 40 (i.e., 40.5 AP) on COCO test-dev without bells and whistles. Specifically, FastInst follows the meta-architecture of recently introduced Mask2Former. Its key designs include instance activation-guided queries, dual-path update strategy, and ground truth mask-guided learning, which enable us to use lighter pixel decoders, fewer Transformer decoder layers, while achieving better performance. The experiments show that FastInst outperforms most state-ofthe-art real-time counterparts, including strong fully convolutional baselines, in both speed and accuracy. Code can be found at https://github.com/junjiehe96/ FastInst. Recently, with the success of DETR [4] in object detection, query-based single-stage instance segmentation methods [9, 10, 26, 44] have emerged. Instead of convolution, they exploit the versatile and powerful attention mechanism [40] combined with a sequence of learnable queries to infer the object class and segmentation mask. For example, Mask2Former [9] simplifies the workflow of instance segmentation by adding a pixel decoder and a maskedattention Transformer decoder on top of a backbone. Unlike previous methods [16, 43], Mask2Former does not require additional handcrafted components, such as training target assignment and NMS post-processing. While being simple, Mask2Former has its own issues: (1) it requires a large number of decoder layers to decode the object queries since its queries are learned static and need a lengthy process to refine; (2) It relies upon a heavy pixel decoder, e.g., multi-scale deformable attention Transformer (MSDefor-mAttn) [51], because its object segmentation mask straightforwardly depends on the output of the pixel decoder, which 1

The Cityscapes Dataset For Semantic Urban Scene Understanding

Abstract: Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotations, 20 000 additional images have coarse annotations to enable methods that leverage large volumes of weakly-labeled data. Crucially, our effort exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity. Our accompanying empirical study provides an in-depth analysis of the dataset characteristics, as well as a performance evaluation of several state-of-the-art approaches based on our benchmark.


The zero-shot transfer results on various indoor and outdoor benchmarks validate the ability of CLIP #REFR for 3D open-world understanding.

In this paper, we present a novel contrastive languageimage-point cloud pretraining framework, CLIP 2 , which consists of a triplet proxy collection scheme and a crossmodal contrastive learning mechanism.

Based on the observation that realistic scenarios contain a massive amount of open-world objects, we innovatively propose to collect triplet proxies from realistic scenes as pretraining data.

We then conduct cross-modal contrastive alignment across language, image and point cloud feature space to learn transferable 3D representation.

Clip 2 : Contrastive Language-Image-Point Pretraining From Real-World Point Cloud Data

Abstract: Indoor Scene Outdoor Scene Figure 1. Illustration of our open-world recognition results. Benefiting from our CLIP 2 , the 3D representation is aligned to the openworld language representation, which enables flexible zero-shot transfer. Best viewed in colors.

Nuscenes: A Multimodal Dataset For Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image-based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first published dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online at this http URL.


We study the effectiveness of span-extractor models, which are commonly used in QA systems to extract the information that is relevant to a user-query from a provided document context #REFR .

Given the ground truth attributes, we evaluate the ability of these models to extract their values from the relevant paragraphs.

We evaluate the DebertaV3 Large model fine-tuned on the Squad 2.0 dataset, which achieves 90.8 F1 on the Squad 2.0 dev set in Table 6 .

We find our EVAPORATE function generation approach (Table 1 ) significantly outperforms this pre-trained QA model on ClosedIE in all settings, over text and HTML documents.

Language Models Enable Simple Systems For Generating Structured Views Of Heterogeneous Data Lakes

Abstract: A long standing goal of the data management community is to develop general, automated systems that ingest semi-structured documents and output queryable tables without human effort or domain specific customization. Given the sheer variety of potential documents, state-of-the art systems make simplifying assumptions and use domain specific training. In this work, we ask whether we can maintain generality by using large language models (LLMs). LLMs, which are pretrained on broad data, can perform diverse downstream tasks simply conditioned on natural language task descriptions. We propose and evaluate EVAPORATE, a simple, prototype system powered by LLMs. We identify two fundamentally different strategies for implementing this system: prompt the LLM to directly extract values from documents or prompt the LLM to synthesize code that performs the extraction. Our evaluations show a cost-quality tradeoff between these two approaches. Code synthesis is cheap, but far less accurate than directly processing each document with the LLM. To improve quality while maintaining low cost, we propose an extended code synthesis implementation, EVAPORATE-CODE+, which achieves better quality than direct extraction. Our key insight is to generate many candidate functions and ensemble their extractions using weak supervision. EVAPORATE-CODE+ not only outperforms the state-of-the art systems, but does so using a sublinear pass over the documents with the LLM. This equates to a 110× reduction in the number of tokens the LLM needs to process, averaged across 16 real-world evaluation settings of 10k documents each.

Electra: Pre-Training Text Encoders As Discriminators Rather Than Generators

Abstract: Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.


The authors in #REFR proposed a method that includes two sub-networks, a spatial network, and a temporal network.

#OTHEREFR proposed a novel deep learning framework for oneclass classification, called Deep SVDD (Support Vector Data Description), which learns a high-dimensional data representation using a deep neural network. Zee et al.

#OTHEREFR presented in their work the interpretability potential of a Siamese CNN to assist humans in hard prediction tasks i.e.

authors used (CAM) to primarily know where to look in the subject images to correctly understand the decisions made by black box CNN models.

In #OTHEREFR , the authors used the depth map and the Remote Photoplethysmography (rPPG) signal as the auxiliary supervision to enhance the effectiveness of the face anti-spoofing method using a Residual Neural Network.

In recent years, authors are trying to exploit the methodologies of explainability and interpretability of a ML models in the field of biometrics, in particular, face PAD by making attempts such as depth map #OTHEREFR [39], #OTHEREFR , producing saliency maps for CNN models #OTHEREFR , #OTHEREFR or some relevant works on estimating the patterns that define a spoofed sample #OTHEREFR , #OTHEREFR .

The spatial network processes the input image to extract spatial features, while the temporal network processes a sequence of images to capture temporal features.

In #OTHEREFR , the authors proposed a Central Difference Convolutional Network (CDCN) architecture that is specifically designed for this task.

The CDCN is a deep neural network that accepts a face image as input and outputs a probability score indicating whether the input image is genuine or spoofed.

The authors in #OTHEREFR proposed an approach consisting of two stages: the metatraining stage and the meta-testing stage.

In the meta-training stage, a meta-learner is trained to learn a good initialization for the feature extractor network that is used to extract features from face images.

An Efficient Ensemble Explainable Ai (Xai) Approach For Morphed Face Detection

Abstract: The extensive utilization of biometric authentication systems have emanated attackers / imposters to forge user identity based on morphed images. In this attack, a synthetic image is produced and merged with genuine. Next, the resultant image is user for authentication. Numerous deep neural convolutional architectures have been proposed in literature for face Morphing Attack Detection (MADs) to prevent such attacks and lessen the risks associated with them. Although, deep learning models achieved optimal results in terms of performance, it is difficult to understand and analyse these networks since they are black box/opaque in nature. As a consequence, incorrect judgments may be made. There is, however, a dearth of literature that explains decision-making methods of black box deep learning models for biometric Presentation Attack Detection (PADs) or MADs that can aid the biometric community to have trust in deep learning-based biometric systems for identification and authentication in various security applications such as border control, criminal database establishment etc. In this work, we present a novel visual explanation approach named Ensemble XAI integrating Saliency maps, Class Activation Maps (CAM) and Gradient-CAM (Grad-CAM) to provide a more comprehensive visual explanation for a deep learning prognostic model (EfficientNet-B1) that we have employed to predict whether the input presented to a biometric authentication system is morphed or genuine. The experimentations have been performed on three publicly available datasets namely Face Research Lab London Set, Wide Multi-Channel Presentation Attack (WMCA), and Makeup Induced Face Spoofing (MIFS). The experimental evaluations affirms that the resultant visual explanations highlight more fine-grained details of image features/areas focused by EfficientNet-B1 to reach decisions along with appropriate reasoning.

Deep Spatial Gradient And Temporal Depth Learning For Face Anti-Spoofing

Abstract: Face anti-spoofing is critical to the security of face recognition systems. Depth supervised learning has been proven as one of the most effective methods for face anti-spoofing. Despite the great success, most previous works still formulate the problem as a single-frame multi-task one by simply augmenting the loss with depth, while neglecting the detailed fine-grained information and the interplay between facial depths and moving patterns. In contrast, we design a new approach to detect presentation attacks from multiple frames based on two insights: 1) detailed discriminative clues (e.g., spatial gradient magnitude) between living and spoofing face may be discarded through stacked vanilla convolutions, and 2) the dynamics of 3D moving faces provide important clues in detecting the spoofing faces. The proposed method is able to capture discriminative details via Residual Spatial Gradient Block (RSGB) and encode spatio-temporal information from Spatio-Temporal Propagation Module (STPM) efficiently. Moreover, a novel Contrastive Depth Loss is presented for more accurate depth supervision. To assess the efficacy of our method, we also collect a Double-modal Anti-spoofing Dataset (DMAD) which provides actual depth for each sample. The experiments demonstrate that the proposed approach achieves state-of-the-art results on five benchmark datasets including OULU-NPU, SiW, CASIA-MFSD, Replay-Attack, and the new DMAD. Codes will be available at https://github.com/clks-wzz/FAS-SGTD.


By conducting large-scale experiments on diverse 3D object detection models under corruptions, we draw some important findings, as summarized below: 1) In general, the corruption robustness of 3D object detection models is largely correlated with their clean performance, similar to the observation in #REFR .

In this paper, we systematically design 27 types of common corruptions in 3D object detection to benchmark corruption robustness of existing 3D object detectors.

We establish three corruption robustness benchmarks-KITTI-C, nuScenes-C, and Waymo-C by synthesizing the corruptions on public datasets.

2) Among all corruption types, motion-level ones degrade the model performance most, which pose a significant threat to autonomous driving.

Weather-level corruptions are also influential to models trained on normal weather.

3) Among all 3D detectors, LiDAR-camera fusion models have better corruption robustness, especially under those that apply distortions to only one modality.

However, they are also exposed to corruptions from both sensors, leading to degraded performance in this case.

Besides, there is a trade-off between robustness under image corruptions and point cloud corruptions of fusion models.

Benchmarking Robustness Of 3D Object Detection To Common Corruptions In Autonomous Driving

Abstract: 3D object detection is an important task in autonomous driving to perceive the surroundings. Despite the excellent performance, the existing 3D detectors lack the robustness to real-world corruptions caused by adverse weathers, sensor noises, etc., provoking concerns about the safety and reliability of autonomous driving systems. To comprehensively and rigorously benchmark the corruption robustness of 3D detectors, in this paper we design 27 types of common corruptions for both LiDAR and camera inputs considering real-world driving scenarios. By synthesizing these corruptions on public datasets, we establish three corruption robustness benchmarks-KITTI-C, nuScenes-C, and Waymo-C. Then, we conduct large-scale experiments on 24 diverse 3D object detection models to evaluate their corruption robustness. Based on the evaluation results, we draw several important findings, including: 1) motion-level corruptions are the most threatening ones that lead to significant performance drop of all models; 2) LiDAR-camera fusion models demonstrate better robustness; 3) camera-only models are extremely vulnerable to image corruptions, showing the indispensability of LiDAR point clouds. We release the benchmarks and codes at https://github.com/kkkcx/ 3D_Corruptions_AD. We hope that our benchmarks and findings can provide insights for future research on developing robust 3D object detection models.

Benchmarking Neural Network Robustness To Common Corruptions And Surface Variations

Abstract: In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize.

3D object detection
corruption robustness


One popular backbone of the deep learning model for segmentation is U-Net #REFR , which is a general CNN model with an encoder and decoder structure ( Fig. 1(a) ).

Medical image segmentation aims to use machine learning models (e.g., Convolutional Neural Networks or CNNs for short) to automatically segment the target regions (organs or lesions) from the input medical images with different modalities #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR .

The encoder path decomposes the input image from local to global deep features where the spatial size is gradually reduced by using the max-pooling operation.

As the layer (Sheng He and Yangming Ou are the corresponding authors.) S. He, R. Bao, P. Grant and Y.

Ou are with the Boston Children's Hospital and Harvard Medical School, Harvard University, 300 Longwood Ave., Boston, MA, USA.

E-mail: heshengxgd@gmail.com; rina.bao@childrens.harvard.edu, ellen.grant@childrens.harvard.edu, yangming.ou@childrens.harvard.edu goes to deep, it extracts the high-level contextual information by discarding the detailed information on each local pixel to remove noise and irrelevant information #OTHEREFR .

To recover the lost detailed spatial information, the decoder path hierarchically fuses global features from the output of the encoder and local features from the intermediate output of the encoder #OTHEREFR for computing the final segmentation map. In summary, as shown in Fig.

U-Netmer: U-Net Meets Transformer For Medical Image Segmentation

Abstract: The combination of the U-Net based deep learning models and Transformer is a new trend for medical image segmentation. U-Net can extract the detailed local semantic and texture information and Transformer can learn the long-rang dependencies among pixels in the input image. However, directly adapting the Transformer for segmentation has "token-flatten" problem (flattens the local patches into 1D tokens which losses the interaction among pixels within local patches) and "scalesensitivity" problem (uses a fixed scale to split the input image into local patches). Compared to directly combining U-Net and Transformer, we propose a new global-local fashion combination of U-Net and Transformer, named U-Netmer, to solve the two problems. The proposed U-Netmer splits an input image into local patches. The global-context information among local patches is learnt by the self-attention mechanism in Transformer and U-Net segments each local patch instead of flattening into tokens to solve the 'token-flatten" problem. The U-Netmer can segment the input image with different patch sizes with the identical structure and the same parameter. Thus, the U-Netmer can be trained with different patch sizes to solve the "scale-sensitivity" problem. We conduct extensive experiments in 7 public datasets on 7 organs (brain, heart, breast, lung, polyp, pancreas and prostate) and 4 imaging modalities (MRI, CT, ultrasound, and endoscopy) to show that the proposed U-Netmer can be generally applied to improve accuracy of medical image segmentation. These experimental results show that U-Netmer provides stateof-the-art performance compared to baselines and other models. In addition, the discrepancy among the outputs of U-Netmer with different scales is linearly correlated to the segmentation accuracy which can be considered as a confidence score to rank test images by difficulty without ground-truth. The code will be available on GitHub.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .


For multiscale feature aggregation, Deeplab #REFR uses various dilation convolutions to capture contextual information at multiple scales.

We selected pixel features of the train(olive points) and road(green points) for tsne visualization.

the pixel feature representation #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR .

The other is to obtain contextual information to enhance the pixel representation #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR .

This paper investigates the same direction as the latter, with the aim of how to obtain richer contextual/semantic information to improve the performance of segmentation.

The methods obtaining contextual information are broadly divided into two types, multi-scale feature aggregation and relational contextual aggregation.

PSPNet #OTHEREFR introduces pyramid spatial pooling to aggregate contextual information.

For relational contextual aggregation, ACFNet #OTHEREFR and OCRNet #OTHEREFR divide pixels in an image into multiple regions and then increase the pixel representation by weighting the aggregated region representation, the weights are determined by the relationship between the pixels and the regions.

Although the above methods are effective, they ignore the potential contextual information between the input images.

In other words,there is no consideration of using class-level features beyond image to enhance the pixel representations.

In order to obtain class-level features beyond the input image, MCIBI #OTHEREFR proposes to use simulated annealing to find a semantic feature on each class.

Class-Level Multiple Distributions Representation Are Necessary For Semantic Segmentation

Abstract: Existing approaches focus on using class-level features to improve semantic segmentation performance. How to characterize the relationships of intra-class pixels and inter-class pixels is the key to extract the discriminative representative class-level features. In this paper, we introduce for the first time to describe intra-class variations by multiple distributions. Then, multiple distributions representation learning(MDRL) is proposed to augment the pixel representations for semantic segmentation. Meanwhile, we design a class multiple distributions consistency strategy to construct discriminative multiple distribution representations of embedded pixels. Moreover, we put forward a multiple distribution semantic aggregation module to aggregate multiple distributions of the corresponding class to enhance pixel semantic information. Our approach can be seamlessly integrated into popular segmentation frameworks FCN/PSPNet/CCNet and achieve 5.61%/1.75%/0.75% mIoU improvements on ADE20K. Extensive experiments on the Cityscapes, ADE20K datasets have proved that our method can bring significant performance improvement.

Deeplab: Semantic Image Segmentation With Deep Convolutional Nets, Atrous Convolution, And Fully Connected Crfs

Abstract: In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First , we highlight convolution with upsampled filters, or ‘atrous convolution’, as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second , we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third , we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed “DeepLab” system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7 percent mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online.


Case (a) is our baseline which contains ResBlock #REFR with a similar number of parameters as OCT module.

In this part, we conduct ablation studies on Set11 dataset for our OCTUF whose iteration number is 10.

We first conduct a break-down ablation experiment in the case of CS ratio = 50% to investigate the effect of each component towards higher performance. The results are listed in Tab. 3.

When we successively apply our FFN and Dual-CA sub-modules respectively, namely Cases (b) and (c), the model achieves 0.71 dB and 2.91 dB improvements.

And the model can greatly enhance 3.09 dB gains with little storage place when both sub-modules are used together.

We also discuss the effect of the LayerNorm (LN) function in Case (d), which addresses that our OCTUF achieves better performance with the LayerNorm function.

Note that without "LayerNorm" represents removing all LN from our OCTUF.

What is more, we train our models with different learning rates as seen from Cases (e), (f), and (g). Table 4 .

Optimization-Inspired Cross-Attention Transformer For Compressive Sensing

Abstract: By integrating certain optimization solvers with deep neural networks, deep unfolding network (DUN) with good interpretability and high performance has attracted growing attention in compressive sensing (CS). However, existing DUNs often improve the visual quality at the price of a large number of parameters and have the problem of feature information loss during iteration. In this paper, we propose an Optimization-inspired Cross-attention Transformer (OCT) module as an iterative process, leading to a lightweight OCT-based Unfolding Framework (OCTUF) for image CS. Specifically, we design a novel Dual Cross Attention (Dual-CA) sub-module, which consists of an Inertia-Supplied Cross Attention (ISCA) block and a Projection-Guided Cross Attention (PGCA) block. ISCA block introduces multi-channel inertia forces and increases the memory effect by a cross attention mechanism between adjacent iterations. And, PGCA block achieves an enhanced information interaction, which introduces the inertia force into the gradient descent step through a cross attention block. Extensive CS experiments manifest that our OCTUF achieves superior performance compared to state-of-the-art methods while training lower complexity. Codes are available at https : / / github. com / songjiechong / OCTUF.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


Ref-COCO #REFR 118k / 5k RIS mIoU Ref-COCO is a Reference Segmentation dataset based on the COCO.

#OTHEREFR 4,998 / 5,105 SS mIoU PASCAL Context dataset is an extension of PASCAL VOC containing 400+ classes (usually 59 most frequently).

COCO #OTHEREFR 118k / 5k SS / IS / PS mIoU / mAP / PQ MS COCO dataset is a large-scale dataset with 80 thing categories and 91 stuff categories.

ADE20k #OTHEREFR 20,210 / 2,000 SS / IS / PS mIoU / mAP / PQ ADE20k dataset is a large-scale dataset exhaustively annotated with pixel-level objects and object part labels.

Cityscapes #OTHEREFR 2,975 / 500 SS / IS / PS mIoU / mAP / PQ Cityscapes dataset focuses on semantic understanding of urban street scenes, captured in 50 cities.

Mapillary #OTHEREFR 18k / 2k SS / PS mIoU / PQ Mapillary dataset is a large-scale dataset with accurate high-resolution annotations.

VSPW #OTHEREFR 198k / 25k VSS mIoU VPSW is a larg-scale high-resolution dataset with long videos focusing on VSS.

Youtube-VIS-2019 #OTHEREFR 95k / 14k VIS AP Extending from Youtube-VOS, Youtube-VIS is with exhaustive instance labels.

VIP-Seg #OTHEREFR 67k / 8k VPS VPQ & STQ Extending from VSPW, VIP-Seg adds extra instance labels for VPS task.

Cityscape-VPS #OTHEREFR 2,400 / 300 VPS VPQ Cityscapes-VPS dataset extracts from the val split of Cityscapes dataset, adding temporal annotations.

KITTI-STEP #OTHEREFR 5,027 / 2,981 VPS STQ KITTI-STEP focuses on the long videos in the urban scenes.

Transformer-Based Visual Segmentation: A Survey

Abstract: Visual segmentation seeks to partition images, video frames, or point clouds into multiple segments or groups. This technique has numerous real-world applications, such as autonomous driving, image editing, robot sensing, and medical analysis. Over the past decade, deep learning-based methods have made remarkable strides in this area. Recently, transformers, a type of neural network based on self-attention originally designed for natural language processing, have considerably surpassed previous convolutional or recurrent approaches in various vision processing tasks. Specifically, vision transformers offer robust, unified, and even simpler solutions for various segmentation tasks. This survey provides a thorough overview of transformer-based visual segmentation, summarizing recent advancements. We first review the background, encompassing problem definitions, datasets, and prior convolutional methods. Next, we summarize a meta-architecture that unifies all recent transformer-based approaches. Based on this meta-architecture, we examine various method designs, including modifications to the meta-architecture and associated applications. We also present several closely related settings, including 3D point cloud segmentation, foundation model tuning, domain-aware segmentation, efficient segmentation, and medical segmentation. Additionally, we compile and re-evaluate the reviewed methods on several well-established datasets. Finally, we identify open challenges in this field and propose directions for future research. The project page can be found at https://github.com/lxtGH/Awesome-Segmenation-With-Transformer. We will also continually monitor developments in this rapidly evolving field.

Modeling Context In Referring Expressions

Abstract: Humans refer to objects in their environments all the time, especially in dialogue with other people. We explore generating and comprehending natural language referring expressions for objects in images. In particular, we focus on incorporating better measures of visual context into referring expression models and find that visual comparison to other objects within an image helps improve performance significantly. We also develop methods to tie the language generation process together, so that we generate expressions for all objects of a particular category jointly. Evaluation on three recent datasets - RefCOCO, RefCOCO+, and RefCOCOg (Datasets and toolbox can be downloaded from https://github.com/lichengunc/refer), shows the advantages of our methods for both referring expression generation and comprehension.


These results have since been confirmed for other monolingual LMs such as T5 #REFR and multilingual LMs such as XGLM , PALM (Chowdhery et al., 2022) , and ALEXATM (Soltan et al., 2022) .

Since the early attempts at using language models (LMs) as multi-task learners #OTHEREFR , MT has been a task of choice to gauge LMs' multilingual ability.

Results for the zeroand few-shot ability of LMs were discussed for both GPT-2 and GPT-3 #OTHEREFR , which is especially intriguing as they were trained primarily on monolingual (English) data.

However, the focus has mainly been on global multi-task performance; often only a small part of the discussion is devoted to MT.

Moreover, results are often only reported for a few well-resourced language pairs (e.g.

English-French and English-German), and the scores reported (mostly BLEU), are hard to compare due to a non-systematic use of standardised evaluation protocols and metrics.

3 There are however some in-depth analyses of MT performance of LLMs, each focusing on a specific LM's performance in a true multilingual setting with respect to prompt design and number of few-shots. For instance, #OTHEREFR

(2022) reevaluate the MT performance of the multilingual PALM (Chowdhery et al., 2022) , focusing notably on the selection of few-shot examples.

Investigating The Translation Performance Of A Large Multilingual Language Model: The Case Of Bloom

Abstract: The NLP community recently saw the release of a new large open-access multilingual language model, BLOOM (BigScience et al., 2022) covering 46 languages. We focus on BLOOM's multilingual ability by evaluating its machine translation performance across several datasets (WMT, Flores-101 and DiaBLa) and language pairs (high-and low-resourced). Our results show that 0-shot performance suffers from overgeneration and generating in the wrong language, but this is greatly improved in the few-shot setting, with very good results for a number of language pairs. We study several aspects including prompt design, model sizes, cross-lingual transfer and the use of discursive context.

Exploring The Limits Of Transfer Learning With A Unified Text-To-Text Transformer

Abstract: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.


The weights of both the teacher and the student are initialized by training the networks on the ImageNet #REFR .

We perform experiments on ENet #OTHEREFR , PSPNet #OTHEREFR and BiseNet #OTHEREFR to evaluate the efficiency of the proposed method as they are three typical segmentation models of different structures.

ENet is a classical Encoder-Decoder model, PSP-Net is a model without a decoder and BiseNet utilizes a compound structure.

We adopt the Stochastic Gradient Descent (SGD) update rule #OTHEREFR with the aforementioned loss function in Equation 3 to optimize the parameters of the network. Loss coefficient is set to 0.5.

We adopt the 'poly' policy widely used in a lot of segmentation models to set the learning rate for each iteration, where the learning rate in an iteration equals to initial learning rate multiplied by (1 − ) .

The power is set to 0.9 and the initial learning rate is set to 0.025 for student network and 0.007 for teacher network.

All of the teacher networks are trained by 50 epochs.

The student networks are trained by 150 epochs for TuSimple and 100 epochs for CULane.

Label-Guided Attention Distillation For Lane Segmentation

Abstract: Contemporary segmentation methods are usually based on deep fully convolutional networks (FCNs). However, the layer-by-layer convolutions with a growing receptive field is not good at capturing longrange contexts such as lane markers in the scene. In this paper, we address this issue by designing a distillation method that exploits label structure when training segmentation network. The intuition is that the ground-truth lane annotations themselves exhibit internal structure. We broadcast the structure hints throughout a teacher network, i.e., we train a teacher network that consumes a lane label map as input and attempts to replicate it as output. Then, the attention maps of the teacher network are adopted as supervisors of the student segmentation network. The teacher network, with label structure information embedded, knows distinctly where the convolution layers should pay visual attention into. The proposed method is named as Label-guided Attention Distillation (LGAD). It turns out that the student network learns significantly better with LGAD than when learning alone. As the teacher network is deprecated after training, our method do not increase the inference time. Note that LGAD can be easily incorporated in any lane segmentation network. To validate the effectiveness of the proposed LGAD method, extensive experiments have been conducted on two popular lane detection benchmarks: TuSimple and CULane. The results show consistent improvement across a variety of convolutional neural network architectures. Specifically, we demonstrate the accuracy boost of LGAD on the lightweight model ENet. It turns out that the ENet-LGAD surpasses existing lane segmentation algorithms.

Imagenet Large Scale Visual Recognition Challenge

Abstract: The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements.


A. Nonlinear Filtering For Discrete-Time Systems

The above derivation converts the problem into a fully observed controlled Markov process (Π, u) via an enlargement of the state space, where control policies and even optimal control policies can be synthesized accordingly for the (hypothetically) fully observed Π #REFR .

1 {F (πt,yt,ut)∈D} • n(dy t ), D ∈ B(P(X )).

We also use Π u to emphasize the marginal behavior of the process (Π, u).

Given the observations and the adaptively generated control signal, the optimal estimation of the conditional probability of satisfying any ω-regular formula Ψ is given by

Note that it is difficult to obtain the full knowledge of Y , our goal is to generate control policies such that the optimal estimation P µ0,u [X Ψ | Y ] possesses certain confidence of satisfying the probabilistic requirement given any realization of observation.

The policy fulfilling the goal mentioned above is thereby decidable.

The construction of the optimal filter process (or the function F in (27)) can be decomposed into a two-step differential equation 8 .

The approximation of such a solution already suffers from the curse of dimensionality.

Using formal abstractions to enlarge the partially observed processes into the filter processes with full observations, based on which control policies can be determined and utilized back to the partially observed cases, seems tedious and impractical.

Besides the theoretical formal guarantee of a confidence of a satisfaction probability (i.e., a probabilistic requirement of the probabilistic specification), the abstraction essentially solves the continuous probability law of a continuous conditional expectation (or a random measure) upon some process with discrete labels using discrete inclusions.

Robustly Complete Finite-State Abstractions For Control Synthesis Of Stochastic Systems

Abstract: The essential step of abstraction-based control synthesis for nonlinear systems to satisfy a given specification is to obtain a finite-state abstraction of the original systems. The complexity of the abstraction is usually the dominating factor that determines the efficiency of the algorithm. For the control synthesis of discrete-time nonlinear stochastic systems modelled by nonlinear stochastic difference equations, recent literature has demonstrated the soundness of abstractions in preserving robust probabilistic satisfaction of ω-regular lineartime properties. However, unnecessary transitions exist within the abstractions, which are difficult to quantify, and the completeness of abstraction-based control synthesis in the stochastic setting remains an open theoretical question. In this paper, we address this fundamental question from the topological view of metrizable space of probability measures, and propose constructive finite-state abstractions for control synthesis of probabilistic linear temporal specifications. Such abstractions are both sound and approximately complete. That is, given a concrete discrete-time stochastic system and an arbitrarily small L 1-perturbation of this system, there exists a family of finite-state controlled Markov chains that both abstracts the concrete system and is abstracted by the slightly perturbed system. In other words, given an arbitrarily small prescribed precision, an abstraction always exists to decide whether a control strategy exists for the concrete system to satisfy the probabilistic specification.

Asymptotic Optimality Of Finite Model Approximations For Partially Observed Markov Decision Processes With Discounted Cost

Abstract: We consider finite model approximations of discrete-time partially observed Markov decision processes (POMDPs) under the discounted cost criterion. After converting the original partially observed stochastic control problem to a fully observed one on the belief space, the finite models are obtained through the uniform quantization of the state and action spaces of the belief space Markov decision process (MDP). Under mild assumptions on the components of the original model, it is established that the policies obtained from these finite models are nearly optimal for the belief space MDP, and so, for the original partially observed problem. The assumptions essentially require that the belief space MDP satisfies a mild weak continuity condition. We provide an example and introduce explicit approximation procedures for the quantization of the set of probability measures on the state space of POMDP (i.e., belief space).


Table 6 , we compare KGTransformer to recently proposed QA methods and report their accuracy on inhouse valid and test sets. All baselines are using RoBERTa-Large #REFR as language encoder.

We apply RoBERTa-large #OTHEREFR as language encoder with the first 1024 dimensional hidden state in the sequence as .

We transform into a 768-dimensional one through a transformation matrix.

We regard P and a triple in G containing words in W as related during the construction of .

We tune the model with batch size set to 128 and Adam #OTHEREFR whose initial rate set as 0.00001.

In the first 4 tuning epochs, the RoBERTa-large is frozen.

Structure Pretraining And Prompt Tuning For Knowledge Graph Transfer

Abstract: Knowledge graphs (KG) are essential background knowledge providers in many tasks. When designing models for KG-related tasks, one of the key tasks is to devise the Knowledge Representation and Fusion (KRF) module that learns the representation of elements from KGs and fuses them with task representations. While due to the difference of KGs and perspectives to be considered during fusion across tasks, duplicate and ad hoc KRF modules design are conducted among tasks. In this paper, we propose a novel knowledge graph pretraining model KGTransformer that could serve as a uniform KRF module in diverse KG-related tasks. We pretrain KGTransformer with three self-supervised tasks with sampled sub-graphs as input. For utilization, we propose a general prompttuning mechanism regarding task data as a triple prompt to allow flexible interactions between task KGs and task data. We evaluate pretrained KGTransformer on three tasks, triple classification, zeroshot image classification, and question answering. KGTransformer consistently achieves better results than specifically designed task models. Through experiments, we justify that the pretrained KG-Transformer could be used off the shelf as a general and effective KRF module across KG-related tasks. The code and datasets are available at https://github.com/zjukg/KGTransformer. CCS CONCEPTS • Computing methodologies → Knowledge representation and reasoning.

Roberta: A Robustly Optimized Bert Pretraining Approach

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.


To alleviate the need for large annotated datasets, weakly supervised techniques for video segmentation involve using transcripts (ordered list of the actions occurring in the video), visual similarities, and audio information to generate pseudo-labels for training #REFR .

Encoder-Decoder Temporal Convolutional Networks (ED-TCNs) #OTHEREFR use a hierarchy of temporal convolutions to perform fine-grained action segmentation, but they can act solely on low-temporal resolution videos.

Instead, Multi-Stage Temporal Convolutional Network (MS-TCN and its improved version MS-TCN++) can act on the full temporal resolution of the videos and achieves increased performance #OTHEREFR .

Spatio-temporal convolutional layers #OTHEREFR have shown promising results in capturing temporal dependencies while being easier to train than previous methods.

The main drawback of traditional supervised approaches to action segmentation is the requirement of a large amount of quality labelled data for training, which limits their applicability to large-scale domains outside of existing presegmented datasets #OTHEREFR .

In #OTHEREFR , a Gaussian Mixture Models + Convolutional Neural Networks (GMM+CNN) is first initialized and used to infer the segments of a video given a transcription of it.

The new segmentation is used to re-estimate and update the model parameters until convergence.

In #OTHEREFR , a recurrent neural network is used to model a discriminative representation of subactions, and a coarse probabilistic model to allow for temporal alignment and inference over long sequences.

Some approaches use machine learning models to infer the segments of the video #OTHEREFR .

Other approaches, such as those based on frame-to-frame visual similarities #OTHEREFR , self-attentions mechanism #OTHEREFR or iterative soft boundary assignment #OTHEREFR , enforce consistency between the video and labels without the need for temporal supervision.

Leveraging Triplet Loss For Unsupervised Action Segmentation

Abstract: In this paper, we propose a novel fully unsupervised framework that learns action representations suitable for the action segmentation task from the single input video itself, without requiring any training data. Our method is a deep metric learning approach rooted in a shallow network with a triplet loss operating on similarity distributions and a novel triplet selection strategy that effectively models temporal and semantic priors to discover actions in the new representational space. Under these circumstances, we successfully recover temporal boundaries in the learned action representations with higher quality compared with existing unsupervised approaches. The proposed method is evaluated on two widely used benchmark datasets for the action segmentation task and it achieves competitive performance by applying a generic clustering algorithm on the learned representations. 1 * Work done during an internship at the IRI.

Sct: Set Constrained Temporal Transformer For Set Supervised Action Segmentation

Abstract: Temporal action segmentation is a topic of increasing interest, however, annotating each frame in a video is cumbersome and costly. Weakly supervised approaches therefore aim at learning temporal action segmentation from videos that are only weakly labeled. In this work, we assume that for each training video only the list of actions is given that occur in the video, but not when, how often, and in which order they occur. In order to address this task, we propose an approach that can be trained end-to-end on such data. The approach divides the video into smaller temporal regions and predicts for each region the action label and its length. In addition, the network estimates the action labels for each frame. By measuring how consistent the frame-wise predictions are with respect to the temporal regions and the annotated action labels, the network learns to divide a video into class-consistent regions. We evaluate our approach on three datasets where the approach achieves state-of-the-art results.


Interestingly, our reduction of k-cycles essentially mirrors the construction in the proof of Theorem 3.1 in #REFR .

Fine-Grained Complexity The study of fine-grained complexity aims to show (conditional) hardness of easy problems.

Recent years have witnessed a bloom of development into this fascinating subject, resulting in many tight lower bounds which match exactly, or up to poly log factors, the running time of best-known algorithms #OTHEREFR 24, #OTHEREFR .

Among many others, popular hardness assumptions include the Strong Exponential Time Hypothesis (SETH), Boolean Matrix Multiplication (BMM), and All-Pairs Shortest Paths (APSP).

Our work can be seen as a particular instance under this framework, i.e.

using Boolean or Min-Weight k-Clique Conjecture to show conditional lower bounds for BCQs.

Functional Aggregate Queries (FAQ) FAQ #OTHEREFR provides a Sum-of-Product framework to define the semantics of conjunctive queries over arbitrary semirings.

The semiring point-ofview originated from the seminal paper #OTHEREFR .

We show that the embedding from a k-clique into a hypergraph holds for arbitrary semirings, which enables one to transfer the hardness of k-clique to FAQ independent of the underlying semiring.

To the best of our knowledge, this is the first semiring-oblivious reduction.

Enumeration and Preprocessing #OTHEREFR characterized for which self-join-free conjunctive queries a linear or constant delay and linear preprocessing algorithm is possible.

The Fine-Grained Complexity Of Boolean Conjunctive Queries And Sum-Product Problems

Abstract: We study the fine-grained complexity of evaluating Boolean Conjunctive Queries and their generalization to sum-of-product problems over an arbitrary semiring. For these problems, we present a general semiring-oblivious reduction from the k-clique problem to any query structure (hypergraph). Our reduction uses the notion of embedding a graph to a hypergraph, first introduced by Marx [18]. As a consequence of our reduction, we can show tight conditional lower bounds for many classes of hypergraphs, including cycles, Loomis-Whitney joins, some bipartite graphs, and chordal graphs. These lower bounds have a dependence on what we call the clique embedding power of a hypergraph H, which we believe is a quantity of independent interest. We show that the clique embedding power is always less than the submodular width of the hypergraph, and present a decidable algorithm for computing it. We conclude with many open problems for future research. 2012 ACM Subject Classification Database theory; Complexity theory and logic Keywords and phrases Fine-grained complexity, conjunctive queries, semiring-oblivious reduction Digital Object Identifier 10.4230/LIPIcs.CVIT.2016.23 1 Technically, the PANDA algorithm works for Boolean or full CQs. 2 Informally speaking, this requires the algorithm does not leverage the fast matrix multiplication.

Tight Hardness For Shortest Cycles And Paths In Sparse Graphs

Abstract: Fine-grained reductions have established equivalences between many core problems with $\tilde{O}(n^3)$-time algorithms on $n$-node weighted graphs, such as Shortest Cycle, All-Pairs Shortest Paths (APSP), Radius, Replacement Paths, Second Shortest Paths, and so on. These problems also have $\tilde{O}(mn)$-time algorithms on $m$-edge $n$-node weighted graphs, and such algorithms have wider applicability. Are these $mn$ bounds optimal when $m \ll n^2$? Starting from the hypothesis that the minimum weight $(2\ell+1)$-Clique problem in edge weighted graphs requires $n^{2\ell+1-o(1)}$ time, we prove that for all sparsities of the form $m = \Theta(n^{1+1/\ell})$, there is no $O(n^2 + mn^{1-\epsilon})$ time algorithm for $\epsilon>0$ for \emph{any} of the below problems: Minimum Weight $(2\ell+1)$-Cycle in a directed weighted graph, Shortest Cycle in a directed weighted graph, APSP in a directed or undirected weighted graph, Radius (or Eccentricities) in a directed or undirected weighted graph, Wiener index of a directed or undirected weighted graph, Replacement Paths in a directed weighted graph, Second Shortest Path in a directed weighted graph, Betweenness Centrality of a given node in a directed weighted graph. That is, we prove hardness for a variety of sparse graph problems from the hardness of a dense graph problem. Our results also lead to new conditional lower bounds from several related hypothesis for unweighted sparse graph problems including $k$-cycle, shortest cycle, Radius, Wiener index and APSP.


More than a decade later, Fefferman and Kimmel #REFR proved a second black-box separation using a distributional in-place oracle, which is a non-standard type of oracles.

The first (but often false) thought is that phases and magnitudes are continuous, and a piece of quantum information may be able to store exponentially or infinitely more information than classical ones; which is always not true 1 .

Since classical and quantum information present distinct and unique natures, the community studies their differences under different contexts and directions, including advice-aided quantum computation [ One way to understand their differences is by studying one-way communication complexity: i.e., Alice and Bob want to jointly compute a function with their private inputs, but only one-time quantum/classical communication from Alice to Bob is allowed.

Among many works, Bar-Yossef, Jayram, and Kerenidis #OTHEREFR showed an exponential separation between quantum and classical one-way communication complexity, for the so-called hidden matching problem.

The other approach is by looking at QMA v.s. QCMA.

In 2007, Aaronson and Kuperberg #OTHEREFR showed a black-box separation with respect to a black-box quantum unitary and left the same separation with respect to a classical oracle as an open question.

Recently, Natarajan and Nirkhe #OTHEREFR moved a step closer to the goal by presenting a black-box separation with respect to a distributional oracle 2 .

Therefore, we would like to further investigate the difference between quantum and classical proofs, i.e., the separation between QMA v.s. QCMA.

In the work, we address the question by demonstrating a separation relative to classically accessible classical oracle.

A language L is said to be in QMA if there exists a quantum polynomialtime machine V together with a polynomial p(•) such that,

Classical Vs Quantum Advice And Proofs Under Classically-Accessible Oracle

Abstract: It is a long-standing open question to construct a classical oracle relative to which BQP/qpoly = BQP/poly or QMA = QCMA. In this paper, we construct classically-accessible classical oracles relative to which BQP/qpoly = BQP/poly and QMA = QCMA. Here, classically-accessible classical oracles are oracles that can be accessed only classically even for quantum algorithms. Based on a similar technique, we also show an alternative proof for the separation of QMA and QCMA relative to a distributional quantumly-accessible classical oracle, which was recently shown by Natarajan and Nirkhe.

Quantum Vs Classical Proofs And Subset Verification

Abstract: We study the ability of efficient quantum verifiers to decide properties of exponentially large subsets given either a classical or quantum witness. We develop a general framework that can be used to prove that QCMA machines, with only classical witnesses, cannot verify certain properties of subsets given implicitly via an oracle. We use this framework to prove an oracle separation between QCMA and QMA using an ``in-place'' permutation oracle, making the first progress on this question since Aaronson and Kuperberg in 2007. We also use the framework to prove a particularly simple standard oracle separation between QCMA and AM.


Combining the two lower bounds above, we can have that #REFR where the first two terms encourage larger inter-class hyperspherical uniformity, and the last term promotes smaller intra-class hyperspherical uniformity.

where λ can be chosen such that both Q 1 (w) and Q 2 (w) become convex functions with respect to w.

Taking advantage of the convexity, we can separately set the gradient of Q 1 (w) and Q 2 (w) with respect to w as 0 and compute their minima. Specifically, we end up with

where l ic = exp( wc,wi ) j exp( wj ,xi ) denotes the softmax confidence.

L CE ≥ Q 1 (w * Q1 ) + Q 2 (w * Q2 ) = n i=1 log C c=1 exp 1 λn n j=1 l jc x i , x j − n 2λ C c=1 1 n n i=1 l ic x i 2 − 1 2λn n i=1 j∈Ay i x i , x j

Abstract: The neural collapse (NC) phenomenon describes an underlying geometric symmetry for deep neural networks, where both deeply learned features and classifiers converge to a simplex equiangular tight frame. It has been shown that both crossentropy loss and mean square error can provably lead to NC. We remove NC's key assumption on the feature dimension and the number of classes, and then present a generalized neural collapse (GNC) hypothesis that effectively subsumes the original NC. Inspired by how NC characterizes the training target of neural networks, we decouple GNC into two objectives: minimal intra-class variability and maximal inter-class separability. We then use hyperspherical uniformity (which characterizes the degree of uniformity on the unit hypersphere) as a unified framework to quantify these two objectives. Finally, we propose a general objective-hyperspherical uniformity gap (HUG), which is defined by the difference between inter-class and intra-class hyperspherical uniformity. HUG not only provably converges to GNC, but also decouples GNC into two separate objectives. Unlike cross-entropy loss that couples intra-class compactness and inter-class separability, HUG enjoys more flexibility and serves as a good alternative loss function. Empirical results show that HUG works well in terms of generalization and robustness.

Learning Towards Minimum Hyperspherical Energy

Abstract: Neural networks are a powerful class of nonlinear functions that can be trained end-to-end on various applications. While the over-parametrization nature in many neural networks renders the ability to fit complex functions and the strong representation power to handle challenging tasks, it also leads to highly correlated neurons that can hurt the generalization ability and incur unnecessary computation cost. As a result, how to regularize the network to avoid undesired representation redundancy becomes an important issue. To this end, we draw inspiration from a well-known problem in physics -- Thomson problem, where one seeks to find a state that distributes N electrons on a unit sphere as evenly as possible with minimum potential energy. In light of this intuition, we reduce the redundancy regularization problem to generic energy minimization, and propose a minimum hyperspherical energy (MHE) objective as generic regularization for neural networks. We also propose a few novel variants of MHE, and provide some insights from a theoretical point of view. Finally, we apply neural networks with MHE regularization to several challenging tasks. Extensive experiments demonstrate the effectiveness of our intuition, by showing the superior performance with MHE regularization.


The Adam optimizer is utilized to adjust the learning rate #REFR . The learning rate is set to 0.0001.

To determine the optimal configuration of the LSTM predictor, its performance on the validation dataset and training time is assessed.

The flow chart of the proposed network layers is shown in Fig.2 . It consists of eight layers each has 200 neurons.

A sequence input layer inputs sequence data to a network.

A fully connected layer multiplies the input by a weight matrix and then adds a bias vector.The ReLU layer performs a threshold operation on each element of the input, where any value less than zero is set to zero.An LSTM layer learns long-term dependencies between time steps in time series and sequence data.The layer performs additive interactions, which can help improve gradient flow over long sequences during training and the regression layer computes the halfmean-squared-error loss for regression tasks.

The training process is stopped when the validation accuracy does not show improvement over five consecutive iterations/epochs to prevent overfitting.

The performance of the vehicle acceleration prediction is evaluated in this section based on root mean square error (RMSE) as an indicator of the model prediction accuracy which is formulated as the following equation 1and 2 #OTHEREFR :

   N 1 t t y 1 N y (1) N t y t y 1 t 2 RMSE              N (2)

where y is the mean of the measured acceleration, t y is the measured acceleration, t y  is the predicted acceleration, and N is the number of elements in output. Fig.

3 shows the RMSE values during the LSTM training process. As depicted in Fig.

Lstm-Based Preceding Vehicle Behaviour Prediction During Aggressive Lane Change For Acc Application

Abstract: The development of Adaptive Cruise Control (ACC) systems aims to enhance the safety and comfort of vehicles by automatically regulating the speed of the vehicle to ensure a safe gap from the preceding vehicle.However, conventional ACC systems are unable to adapt themselves to changing driving conditions and drivers' behavior. To address this limitation, we propose a Long Short-Term Memory (LSTM)based ACC system that can learn from past driving experiences and adapt and predict new situations in realtime.The model is constructed based on the real-world highD dataset, acquired from German highways with the assistance of camera-equipped drones. We evaluated the ACC system under aggressive lane changes when the side lane preceding vehicle cut off, forcing the targeted driver to reduce speed. To this end, the proposed system was assessed on a simulated driving environment and compared with a feedforward Artificial Neural Network (ANN) model and Model Predictive Control (MPC) model. The results show that the LSTM-based system is 19.25 % more accurate than the ANN model and 5.9 % more accurate than the MPC model in terms of predicting future values of subject vehicle acceleration. The simulation is done in Matlab/Simulink environment. I.

Adam: A Method For Stochastic Optimization

Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.


Convergence-Optimized Power Control Under Dp Guarantee

Then, we substitute the objective function and the DP constraint in P1 with the convergence upper bound #REFR w.r.t.

Under Assumption 1, given a local update ∆ t i ∈ R d and rand k sparsification with random projection matrix A t , we have

Proof: Taking the expectation on the rand k , we have:

E A t ∆ t i 2 2 = d j=1 k d [∆ t i ] 2 j (35a) = k d θ t,τ i − θ t 2 2 (35b) = k d η 2 τ s=1 g t,s−1 i 2 2 (35c) ≤ k d η 2 τ 2 C 2 1 (35d)

where (35c) holds due to #OTHEREFR , (35d) follows from Assumption 1 and Cauchy-Schwarz inequality.

{β t } t∈[T −1] in Theorem 4 and the client-level DP result #OTHEREFR in Theorem 3, respectively. Therefore, we can approximate Problem P1 as follows:

Note that P2 can be readily solved as shown in the following result:

Theorem 5. The optimal solution to Problem P2 is given by:

Communication And Energy Efficient Wireless Federated Learning With Intrinsic Privacy

Abstract: Federated Learning (FL) is a collaborative learning framework that enables edge devices to collaboratively learn a global model while keeping raw data locally. Although FL avoids leaking direct information from local datasets, sensitive information can still be inferred from the shared models. To address the privacy issue in FL, differential privacy (DP) mechanisms are leveraged to provide formal privacy guarantee. However, when deploying FL at the wireless edge with over-the-air computation, ensuring client-level DP faces significant challenges. In this paper, we propose a novel wireless FL scheme called private federated edge learning with sparsification (PFELS) to provide client-level DP guarantee with intrinsic channel noise while reducing communication and energy overhead and improving model accuracy. The key idea of PFELS is for each device to first compress its model update and then adaptively design the transmit power of the compressed model update according to the wireless channel status without any artificial noise addition. We provide a privacy analysis for PFELS and prove the convergence of PFELS under general non-convex and non-IID settings. Experimental results show that compared with prior work, PFELS can improve the accuracy with the same DP guarantee and save communication and energy costs simultaneously.

Differentially Private Federated Learning: A Client Level Perspective

Abstract: Federated learning is a recent advance in privacy protection. In this context, a trusted curator aggregates parameters optimized in decentralized fashion by multiple clients. The resulting model is then distributed back to all clients, ultimately converging to a joint representative model without explicitly having to share the data. However, the protocol is vulnerable to differential attacks, which could originate from any party contributing during federated optimization. In such an attack, a client's contribution during training and information about their data set is revealed through analyzing the distributed model. We tackle this problem and propose an algorithm for client sided differential privacy preserving federated optimization. The aim is to hide clients' contributions during training, balancing the trade-off between privacy loss and model performance. Empirical studies suggest that given a sufficiently large number of participating clients, our proposed procedure can maintain client-level differential privacy at only a minor cost in model performance.


Using an optimized implementation of Dijkstra's algorithm (e.g., the one provided by scipy #REFR ), the search for a shortest path is also very fast.

As noted above, this curve is safe because each of its line segments is contained in at least a safe box.

This shortest path step determines whether or not our path planning problem is feasible.

If there is no path in the augmented line graph between the vertices associated with p init and p term , i.e., the distance between them is ∞, then the path planning problem (4) is infeasible.

Conversely, if there is a path between these two vertices, the original path planning problem is feasible, since a feasible trajectory can be constructed as in §II-C.

The problem of identifying the safe boxes that contain the initial and terminal points is known as stabbing problem and, given the precomputations done to construct the line graph, this takes negligible time #OTHEREFR .

Fast Path Planning Through Large Collections Of Safe Boxes

Abstract: We present a fast algorithm for the design of smooth paths (or trajectories) that are constrained to lie in a collection of axis-aligned boxes. We consider the case where the number of these safe boxes is large, and basic preprocessing of them (such as finding their intersections) can be done offline. At runtime we quickly generate a smooth path between given initial and terminal positions. Our algorithm designs trajectories that are guaranteed to be safe at all times, and it detects infeasibility whenever such a trajectory does not exist. Our algorithm is based on two subproblems that we can solve very efficiently: finding a shortest path in a weighted graph, and solving (multiple) convex optimal control problems. We demonstrate the proposed path planner on large-scale numerical examples, and we provide an efficient open-source software implementation, fastpathplanning.

Scipy 1.0: Fundamental Algorithms For Scientific Computing In Python

Abstract: SciPy is an open-source scientific computing library for the Python programming language. Since its initial release in 2001, SciPy has become a de facto standard for leveraging scientific algorithms in Python, with over 600 unique code contributors, thousands of dependent packages, over 100,000 dependent repositories and millions of downloads per year. In this work, we provide an overview of the capabilities and development practices of SciPy 1.0 and highlight some recent technical developments. This Perspective describes the development and capabilities of SciPy 1.0, an open source scientific computing library for the Python programming language.


The fairly comparable scores of ELMo and the traditional baselines in the null context supports the observation of #REFR that is, when it comes to contextual embeddings, there is only a small improvement in learning semantics over traditional ML methods.

If the models were also pretrained from scratch on the same corpus, it would be ensured that they utilize the same knowledge to learn the context.

And this would enable a fairer comparison.

Recently, it was shown that ELMo and BERT make no significant difference in semantic analysis #OTHEREFR .

Here it is observed that although they are close-by in the null context, DistilBERT is more robust than ELMo in the cross-context in text classification.

The findings of this study are in line with prior work.

DistilBERT is on par with or exceeding ELMo on a binary text classification task #OTHEREFR .

DistilBERT, as a transformerbased model, is better in capturing long-term dependencies in an input sequence #OTHEREFR .

DistilBERT is lighter than ELMo and has a shorter training time #OTHEREFR .

Here it should be noted that the experimental settings of the previous work and In this study, ELMo and DistilBERT are compared on their fine-tuning performance on two binary text classification tasks.

The main focus was to see how much can these models be benefited in a practical way without any modification to the pretraining outputs.

Abstract: I am grateful to my family for their unconditional love and patience. I am grateful to ArzucanÖzgür, for being such an inspiring figure by her selfless devotion to research the most righteous way with the passion to contribute to the community. I am grateful to Ali Hürriyetoglu, for being such a role model, who could somehow always find a way to turn the mist of research questions into a structured path to create practical solutions by combining creativity and technique. I cannot thank enough my dear friends who put up with my whims throughout this journey. I thank fellows from TabiLAB for inspiring me with their brilliance, invaluable insights and recommendations. I thank Koç University EMW research team for their generosity in sharing the data which was created with blood, sweat and tears. I feel lucky that I got to meet fellows in EMW project engineering team who invested their precious time and energy to support me in this study from the very beginning. Lastly, I owe the deepest gratitude to our professors and staff members in our department who taught us how to form such a great community and made it feel like the dearest home from the day one.

What Do You Learn From Context? Probing For Sentence Structure In Contextualized Word Representations

Abstract: Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.


A distancedbased formation control algorithm for a team of quadrotors transporting a heavy object is presented in #REFR , which measures and resists the acceleration due to disturbances and rope tension using incremental nonlinear dynamic inversion control.

Although the precise attitude and position control of the payload can be realized, the dynamic information of the payload is required for real-time feedback control, which is hard to obtain in engineering practice.

In contrast, in formation-based design, only the state information of the aerial vehicles is needed.

When the vehicle group reaches its destination, the payload is also supposed to reach the target area.

The validity and feasibility of such approach has been established via simulation #OTHEREFR and experiment #OTHEREFR , but the cable forces on the quadrotors are ignored.

To implement formation-based robust collaborative transportation, several control algorithms have been developed.

In #OTHEREFR and #OTHEREFR , a passivity-based formation control strategy is proposed with adaptive compensation terms to eliminate the wind disturbance and the cable tension.

The energy passivity property of the quadrotors-payload system is established in #OTHEREFR , where an adaptive damping term is used to dissipate the energy injected by the sudden perturbations.

The studies mentioned above are all designed based on the rigid formation.

As a matter of fact, maintaining a fixed formation for payload transportation is not necessary and it is better to employ a flexible formation, which can adapt the vehicles to the complex and uncertain environment and tasks #OTHEREFR .

Force control-based approaches have been explored for collaborative payload transportation with flexible formation, e.g., force amplification #OTHEREFR and contact force regulation #OTHEREFR .

Force-Coordination Control For Aerial Collaborative Transportation Based On Lumped Disturbance Separation And Estimation

Abstract: This article studies the collaborative transportation of a cable-suspended pipe by two quadrotors. A force-coordination control scheme is proposed, where a force-consensus term is introduced to average the load distribution between the quadrotors. Since thrust uncertainty and cable force are coupled together in the acceleration channel, disturbance observer can only obtain the lumped disturbance estimate. Under the quasi-static condition, a disturbance separation strategy is developed to remove the thrust uncertainty estimate for precise cable force estimation. The stability of the overall system is analyzed using Lyapunov theory. Both numerical simulations and indoor experiments using heterogeneous quadrotors validate the effectiveness of thrust uncertainty separation and force-consensus algorithm.

Flexible Collaborative Transportation By A Team Of Rotorcraft

Abstract: We propose a combined method for the collaborative transportation of a suspended payload by a team of rotorcraft. A recent distance-based formation-motion control algorithm based on assigning distance disagreements among robots generates the acceleration signals to be tracked by the vehicles. In particular, the proposed method does not need global positions nor tracking prescribed trajectories for the motion of the members of the team. The acceleration signals are followed accurately by an Incremental Nonlinear Dynamic Inversion controller designed for rotorcraft that measures and resists the tensions from the payload. Our approach allows us to analyze the involved accelerations and forces in the system so that we can calculate the worst case conditions explicitly to guarantee a nominal performance, provided that the payload starts at rest in the 2D centroid of the formation, and it is not under significant disturbances. For example, we can calculate the maximum safe deformation of the team with respect to its desired shape. We demonstrate our method with a team of four rotorcraft carrying a suspended object two times heavier than the maximum payload for an individual. Last but not least, our proposed algorithm is available for the community in the open-source autopilot Paparazzi.