As shown in Tab.5, the performance of our method increases as the number of negative pairs increases, with the best performance when the number reaches 512, which is consistent with the phenomenon in #REFR .

Ablation on the number of negative pairs.

The number of negative pairs is shown to have a larger effect on the performance of contrastive learning as described in #OTHEREFR , and a Table 5 .

Ablation study on the ScanNet dataset #OTHEREFR with respect to the number of negative pairs.

larger number of negative pairs has a significant advantage over the smaller ones.

Here in our method, we conduct ablation studies with different numbers of negative pairs to see the effect of the number of negative pairs on the model performance.

We also tried another way of multi-view consistency optimization, that is, we directly optimize the similarity between positive pairs p −q + 2 , where p and q + are defined in Eq.7. The result is shown in row 1 of Tab.5.

We can see that the above optimization method performs worse than our GeoContrast, which further reflects the importance of negative pairs.

Here we present the SSIM and LPIPS under varying proportions of real and synthetic data as shown in Fig.10 .

Similar to the curve of PSNR in Fig.6 , the performance of our method continues to improve as the proportion of real data increases, but the performance saturates when the proportion of real data reaches 40%.

Contranerf: Generalizable Neural Radiance Fields For Synthetic-To-Real Novel View Synthesis Via Contrastive Learning

Abstract: Although many recent works have investigated generalizable NeRF-based novel view synthesis for unseen scenes, they seldom consider the synthetic-to-real generalization, which is desired in many practical applications. In this work, we first investigate the effects of synthetic data in synthetic-to-real novel view synthesis and surprisingly observe that models trained with synthetic data tend to produce sharper but less accurate volume densities. For pixels where the volume densities are correct, fine-grained details will be obtained. Otherwise, severe artifacts will be produced. To maintain the advantages of using synthetic data while avoiding its negative effects, we propose to introduce geometry-aware contrastive learning to learn multiview consistent features with geometric constraints. Meanwhile, we adopt cross-view attention to further enhance the geometry perception of features by querying features across input views. Experiments demonstrate that under the synthetic-to-real setting, our method can render images with higher quality and better fine-grained details, outperforming existing generalizable novel view synthesis methods in terms of PSNR, SSIM, and LPIPS. When trained on real data, our method also achieves state-of-the-art results. https://haoy945.github.io/contranerf/

A Simple Framework For Contrastive Learning Of Visual Representations

Abstract: This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.

negative pairs increases
best performance


The other parameters remained the same in EigLearn as in Section 4.3.1, which is consistent with those in #REFR .

The second assessment focuses on the comparison between EigLearn and FisherGCN #OTHEREFR that is arguably more sophisticated than LanczosNet.

In this comparison, we adopted the same settings for data splitting and training as in #OTHEREFR .

In particular, we split the data into a training set with 20 samples per class, a validation set with 500 samples, and a testing set with 1000 samples.

We ran experiments with 20 random splits and for each data split there were 10 different initializations per split.

As thoroughly studied in #OTHEREFR , a random split is a less biased evaluation setting for GCN performance, and hence is preferable over the public fixed split.

Table 4 .2 reports a comparison among the baseline GCN, FisherGCN, and EigLearn using the graph filter matrices based on S SNA and S DW , respectively.

We used both classification accuracy and loss on testing set as the evaluation metrics.

For the baselines, we report the performances of both TensorFlow-based implementations of GCN and GCN T in #OTHEREFR and our PyTorch-based implementation.

to implement and fits more naturally into the neural network architectures.

In addition, EigLearnGCN is significantly faster than FisherGCN in that the additional cost associated with EigLearn is only a small fraction of the cost for training a regular GCN, while FisherGCN is several times slower than GCN as reported in #OTHEREFR .

Many Thanks Go To Professor Keli Xiao, Professor Xiangmin Jiao, Dr. Dimitrios Katramatos, Professor Eden Figueroa And Professor Yao Ma. I Am Grateful For The Collaboration Opportunities And Benefit Significantly From Their Guidance On My Research Exploration And Technical Writing. I Am Thankful For Dr. Yufei Ren, Professor

Abstract: The world has never been more connected, led by the information technology revolution in the past decades that has fundamentally changed the way people interact with each other using social networks. Consequently, enormous human activity data are collected from the business world and machine learning techniques are widely adopted to aid our decision processes. Despite of the success of machine learning in various application scenarios, there are still many questions that need to be well answered, such as optimizing machine learning outcomes when desired knowledge cannot be extracted from the available data. This naturally drives us to ponder if one can leverage some side information to populate the knowledge domain of their interest, such that the problems within that knowledge domain can be better tackled. In this work, such problems are investigated and practical solutions are proposed. To leverage machine learning in any decision-making process, one must Simplicity is the final achievement. Frederic Chopin v ACKNOWLEDGMENT First and foremost, I would like to express my sincere gratitude to my advisor, Prof. Dantong Yu. Throughout my Ph.D. journey, Prof. Yu patiently listened my thoughts on various research topics and gave me numerous pieces of priceless advice. He gave me substantial flexibility to explore and broaden my research interests, and shaped me into a better researcher and person over the years. I could not be more thankful for his support, guidance and encouragement to complete this journey. I would like to thank my committee members, Baruch M. Schieber, Ioannis Koutis, Yi Chen and Junmin Shi, for their kindness and insightful feedback during my journey on the dissertation. I would like to thank Prof. Schieber and Prof. Koutis for the generous office discussions and making complicated stuff simple and fun.

Fisher-Bures Adversary Graph Convolutional Networks

Abstract: In a graph convolutional network, we assume that the graph $G$ is generated with respect to some observation noise. We make small random perturbations $\Delta{}G$ of the graph and try to improve generalization. Based on quantum information geometry, we can have quantitative measurements on the scale of $\Delta{}G$. We try to maximize the intrinsic scale of the permutation with a small budget while minimizing the loss based on the perturbed $G+\Delta{G}$. Our proposed model can consistently improve graph convolutional networks on semi-supervised node classification tasks with reasonable computational overhead. We present two different types of geometry on the manifold of graphs: one is for measuring the intrinsic change of a graph; the other is for measuring how such changes can affect externally a graph neural network. These new analytical tools will be useful in developing a good understanding of graph neural networks and fostering new techniques.


A fully discrete discontinuous Galerkin scheme is proposed in #REFR , where the time step as well as the mesh size are irrespective.

As one of the paradigms of nonlinear dynamical system, the Swift-Hohenberg equation has been widely applied in physics #OTHEREFR , materials science #OTHEREFR , dynamics of ecological systems #OTHEREFR , and many other nonlinear fields #OTHEREFR Bifurcation analysis #OTHEREFR and pattern selection #OTHEREFR of the equilibrium solutions for the Swift-Hohenberg equation are theoretically analyzed.

Shi and Han #OTHEREFR obtained a reversible homoclinic solution approaching to a periodic solution of the Swift-Hohenberg equation, and the existence of nontrivial periodic solutions was proved by Lai and Zhang #OTHEREFR .

In #OTHEREFR , the effects of rapid oscillations of the forcing term on the long-time behaviour of the solutions are studied.

We also refer the reader to the recent article #OTHEREFR on nonstationary case and the references therein.

Recently, the efficient numerical methods for the Swift-Hohenberg equation are considered since the analytically solutions cannot be generally obtained.

The scheme is proved uniquely solvable and unconditionally energy stable.

A second-order energy-stable time-integration method that suppresses numerical instabilities is presented by Sarmiento et al.

#OTHEREFR and a detailed proof of the unconditional energy stability is provided.

Liu #OTHEREFR considered two linear, second-order and unconditionally energy stable schemes by linear invariant energy quadratization and modified scalar auxiliary variable approaches. Dehghan et al.

#OTHEREFR combined the proper orthogonal decomposition approach and the local discontinuous Galerkin technique, and discussed the energy stability. Sun et al.

Energy Stable And L 2 Norm Convergent Bdf3 Scheme For The Swift-Hohenberg Equation

Abstract: A fully discrete implicit scheme is proposed for the Swift-Hohenberg model, combining the third-order backward differentiation formula (BDF3) for the time discretization and the second-order finite difference scheme for the space discretization. Applying the Brouwer fixed-point theorem and the positive definiteness of the convolution coefficients of BDF3, the presented numerical algorithm is proved to be uniquely solvable and unconditionally energy stable, further, the numerical solution is shown to be bounded in the maximum norm. The proposed scheme is rigorously proved to be convergent in L 2 norm by the discrete orthogonal convolution (DOC) kernel, which transfer the four-level-solution form into the three-levelgradient form for the approximation of the temporal derivative. Consequently, the error estimate for the numerical solution is established by utilization of the discrete Gronwall inequality. Numerical examples in 2D and 3D cases are provided to support the theoretical results.

Unconditionally Energy Stable Dg Schemes For The Swift-Hohenberg Equation

Abstract: The Swift–Hohenberg equation as a central nonlinear model in modern physics has a gradient flow structure. Here we introduce fully discrete discontinuous Galerkin (DG) schemes for a class of fourth order gradient flow problems, including the nonlinear Swift–Hohenberg equation, to produce free-energy-decaying discrete solutions, irrespective of the time step and the mesh size. We exploit and extend the mixed DG method introduced in Liu and Yin (J Sci Comput 77:467–501, 2018) for the spatial discretization, and the “Invariant Energy Quadratization” method for the time discretization. The resulting IEQ-DG algorithms are linear, thus they can be efficiently solved without resorting to any iteration method. We actually prove that these schemes are unconditionally energy stable. We present several numerical examples that support our theoretical results and illustrate the efficiency, accuracy and energy stability of our new algorithm. The numerical results on two dimensional pattern formation problems indicate that the method is able to deliver comparable patterns of high accuracy.


We compare our TRAVEL model with two generic machine learning models and eleven state-of-the-art GNN models: (1) XGBoost: A scalable gradient boosting tree-based approach #REFR .

(2) MLP: A multi-layer perceptron is a classic feedforward artificial neural network.

(3) GCN: Graph Convolutional Networks, which generalize CNNs from low-dimensional regular grids to graph data using neighborhood-based filters #OTHEREFR .

(4) ChebNet: Chebyshev spectral graph convolution networks, which are spectral graph convolutional architectures with fast localized spectral filtering #OTHEREFR .

(5) ARMANet: Graph neural networks with convolutional autoregressive moving average (ARMA) filters #OTHEREFR .

(6) GraphSAGE: A general inductive framework for inductive representation learning on graphs #OTHEREFR .

Tap: A Comprehensive Data Repository For Traffic Accident Prediction In Road Networks

Abstract: Road safety is a major global public health concern. Effective traffic crash prediction can play a critical role in reducing road traffic accidents. However, Existing machine learning approaches tend to focus on predicting traffic accidents in isolation, without considering the potential relationships between different accident locations within road networks. To incorporate graph structure information, graph-based approaches such as Graph Neural Networks (GNNs) can be naturally applied. However, applying GNNs to the accident prediction problem faces challenges due to the lack of suitable graph-structured traffic accident datasets. To bridge this gap, we have constructed a real-world graph-based Traffic Accident Prediction (TAP) data repository, along with two representative tasks: accident occurrence prediction and accident severity prediction. With nationwide coverage, real-world network topology, and rich geospatial features, this data repository can be used for a variety of traffic-related tasks. We further comprehensively evaluate eleven state-of-the-art GNN variants and two non-graph-based machine learning methods using the created datasets. Significantly facilitated by the proposed data, we develop a novel Traffic Accident Vulnerability Estimation via Linkage (TRAVEL) model, which is designed to capture angular and directional information from road networks. We demonstrate that the proposed model consistently outperforms the baselines. The data and code are available on GitHub 1 .

Xgboost: A Scalable Tree Boosting System

Abstract: Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.


These include teaching assembly tasks #REFR , where the robot learns from human demonstration: First the robot observes the human, then it imitates the human's movements.

Robots are employed in a wide range of applications, especially in industry.

A different way of learning through user interaction is proposed in #OTHEREFR .

In this approach, natural language is used to explain to the robot, which tasks it should perform. One such typical task is grasping objects.

Solving this challenge is part of current research such as #OTHEREFR , where an object detection approach was used to learn good grasping poses.

Data driven approaches, as stated in #OTHEREFR , are often addressed by providing training data in form of labeled examples, by trial-and-error, or through human demonstration.

This means that communication between human and robot is also important here for a flexible learning progress without prior offline data generation.

Multiperspective Teaching Of Unknown Objects Via Shared-Gaze-Based Multimodal Human-Robot Interaction

Abstract: For successful deployment of robots in multifaceted situations, an understanding of the robot for its environment is indispensable. With advancing performance of state-of-the-art object detectors, the capability of robots to detect objects within their interaction domain is also enhancing. However, it binds the robot to a few trained classes and prevents it from adapting to unfamiliar surroundings beyond predefined scenarios. In such scenarios, humans could assist robots amidst the overwhelming number of interaction entities and impart the requisite expertise by acting as teachers. We propose a novel pipeline that effectively harnesses human gaze and augmented reality in a human-robot collaboration context to teach a robot novel objects in its surrounding environment. By intertwining gaze (to guide the robot's attention to an object of interest) with augmented reality (to convey the respective class information) we enable the robot to quickly acquire a significant amount of automatically labeled training data on its own. Training in a transfer learning fashion, we demonstrate the robot's capability to detect recently learned objects and evaluate the influence of different machine learning models and learning procedures as well as the amount of training data involved. Our multimodal approach proves to be an efficient and natural way to teach the robot novel objects based on a few instances and allows it to detect classes for which no training dataset is available. In addition, we make our dataset publicly available to the research community, which consists of RGB and depth data, intrinsic and extrinsic camera parameters, along with regions of interest.

Teaching Robots To Do Object Assembly Using Multi-Modal 3D Vision

Abstract: The motivation of this paper is to develop an intelligent robot assembly system using multi-modal vision for next-generation industrial assembly. The system includes two phases where in the first phase human beings demonstrate assembly to robots and in the second phase robots detect objects, plan grasps, and assemble objects following human demonstration using AI searching. A notorious difficulty to implement such a system is the bad precision of 3D visual detection. This paper presents multi-modal approaches to overcome the difficulty: It uses AR markers in the teaching phase to detect human operation, and uses point clouds and geometric constraints in the robot execution phase to avoid unexpected occlusion and noises. The paper presents several experiments to examine the precision and correctness of the approaches. It demonstrates the applicability of the approaches by integrating them with graph model-based motion planning, and by executing the results on industrial robots in real-world scenarios.


Moreover, the instability result is not expected to hold if one picks a generic isometry B, thus randomizing over both B and C #REFR .

1.X is feasible: A(X) − y 2 ≤ τ ; 2.X is preferred to X 0 : X * ≤ X 0 * ;

3.X is far from the true solution in Frobenius norm:

Theorem 2.4 suggests that the dimension factor in the recovery guarantees (2.6) is not merely a proof artifact.

AlthoughX constructed in the proof of Theorem 2.4 is preferred to the true solution, it is not a minimizer of the SDP (2.5), see Remark 3.4. in #OTHEREFR .

Still, Theorem 2.4 provides a lower bound for possible recovery guarantees which hold for all preferred solutions.

However, Theorem 2.4 gives no information on the noise level τ 0 from which the instability is to be expected.

In particular, it does not exclude the possibility that for larger noise levels τ , stronger recovery guarantees can be shown.

Indeed, #OTHEREFR made a first step in this direction by showing near-optimal error scaling whenever the adversarial noise magnitude is not too small:

How Robust Is Randomized Blind Deconvolution Via Nuclear Norm Minimization Against Adversarial Noise?

Abstract: In this paper, we study the problem of recovering two unknown signals from their convolution, which is commonly referred to as blind deconvolution. Reformulation of blind deconvolution as a low-rank recovery problem has led to multiple theoretical recovery guarantees in the past decade due to the success of the nuclear norm minimization heuristic. In particular, in the absence of noise, exact recovery has been established for sufficiently incoherent signals contained in lower-dimensional subspaces. However, if the convolution is corrupted by additive bounded noise, the stability of the recovery problem remains much less understood. In particular, existing reconstruction bounds involve large dimension factors and therefore fail to explain the empirical evidence for dimension-independent robustness of nuclear norm minimization. Recently, theoretical evidence has emerged for ill-posed behavior of low-rank matrix recovery for sufficiently small noise levels. In this work, we develop improved recovery guarantees for blind deconvolution with adversarial noise which exhibit square-root scaling in the noise level. Hence, our results are consistent with existing counterexamples which speak against linear scaling in the noise level as demonstrated for related low-rank matrix recovery problems.

On The Convex Geometry Of Blind Deconvolution And Matrix Completion

Abstract: Low-rank matrix recovery from structured measurements has been a topic of intense study in the last decade and many important problems like matrix completion and blind deconvolution have been formulated in this framework. An important benchmark method to solve these problems is to minimize the nuclear norm, a convex proxy for the rank. A common approach to establish recovery guarantees for this convex program relies on the construction of a so-called approximate dual certificate. However, this approach provides only limited insight in various respects. Most prominently, the noise bounds exhibit seemingly suboptimal dimension factors. In this paper we take a novel, more geometric viewpoint to analyze both the matrix completion and the blind deconvolution scenario. We find that for both these applications the dimension factors in the noise bounds are not an artifact of the proof, but the problems are intrinsically badly conditioned. We show, however, that bad conditioning only arises for very small noise levels: Under mild assumptions that include many realistic noise levels we derive near-optimal error estimates for blind deconvolution under adversarial noise.


Our lower bounds are obtained for without-replacement SGD with constant step size, which is also the case in other existing results in the literature Shamir, 2020, 2021; #REFR .

Towards a complete understanding of SGD-RR and permutation-based SGD in general, we seek to close the existing gaps outlined above by developing tighter lower bounds with matching upper bounds.

We present results under two different kinds of algorithm settings: Section 3 contains lower bounds obtained from SGD-RR, and Section 4 presents lower bounds that are applicable to arbitrary permutation-based SGD algorithms.

While all lower bounds proved in the aforementioned papers are only applicable to the final iterate of the algorithm, many of our results in this paper apply to arbitrary weighted average of end-of-epoch iterates, which can be used to show tightness of matching upper bounds that employ iterate averaging.

Here we include κ = Θ (1/µ) in the convergence rates to better describe the results.

For a complete summary of our results, please refer to Table 1.

• Theorem 1 derives a lower bound of rate Ω κ 2 nK 2 on the strongly convex case for the final iterate, which matches the best-known corresponding upper boundÕ κ 3 nK 2 up to a factor of κ.

Tighter Lower Bounds For Shuffling Sgd: Random Permutations And Beyond

Abstract: We study convergence lower bounds of without-replacement stochastic gradient descent (SGD) for solving smooth (strongly-)convex finite-sum minimization problems. Unlike most existing results focusing on final iterate lower bounds in terms of the number of components n and the number of epochs K, we seek bounds for arbitrary weighted average iterates that are tight in all factors including the condition number κ. For SGD with Random Reshuffling, we present lower bounds that have tighter κ dependencies than existing bounds. Our results are the first to perfectly close the gap between lower and upper bounds for weighted average iterates in both strongly-convex and convex cases. We also prove weighted average iterate lower bounds for arbitrary permutation-based SGD, which apply to all variants that carefully choose the best permutation. Our bounds improve the existing bounds in factors of n and κ and thereby match the upper bounds shown for a recently proposed algorithm called GraB. Recent works provide evidence of designing algorithms that converge faster than SGD-RR. Concretely, Rajput et al. (2022) introduce a permutation-based SGD algorithm called FlipFlop and prove that it can outperform SGD-RR for quadratic objectives. The authors also propose a lower bound applicable to arbitrary permutation-based SGD, by proving that no algorithm can converge faster than Ω

Closing The Convergence Gap Of Sgd Without Replacement

Abstract: Stochastic gradient descent without replacement sampling is widely used in practice for model training. However, the vast majority of SGD analyses assumes data sampled with replacement, and when the function minimized is strongly convex, an $\mathcal{O}\left(\frac{1}{T}\right)$ rate can be established when SGD is run for $T$ iterations. A recent line of breakthrough work on SGD without replacement (SGDo) established an $\mathcal{O}\left(\frac{n}{T^2}\right)$ convergence rate when the function minimized is strongly convex and is a sum of $n$ smooth functions, and an $\mathcal{O}\left(\frac{1}{T^2}+\frac{n^3}{T^3}\right)$ rate for sums of quadratics. On the other hand, the tightest known lower bound postulates an $\Omega\left(\frac{1}{T^2}+\frac{n^2}{T^3}\right)$ rate, leaving open the possibility of better SGDo convergence rates in the general case. In this paper, we close this gap and show that SGD without replacement achieves a rate of $\mathcal{O}\left(\frac{1}{T^2}+\frac{n^2}{T^3}\right)$ when the sum of the functions is a quadratic, and offer a new lower bound of $\Omega\left(\frac{n}{T^2}\right)$ for strongly convex functions that are sums of smooth functions.


The CrowdPose dataset #REFR is a challenging benchmark with the goal to evaluate the robustness of methods in crowded scenes.

A. Implementation Details 1) Dataset: COCO.

The COCO dataset #OTHEREFR is a popular multi-person pose estimation benchmark containing more than 200, 000 images and 250, 000 person instances labeling with 17 keypoints.

The COCO dataset is divided into three parts: the train set includes 57K images, the val set contains 5K images, and the test-dev set consists of 20K images.

However, it contains rarely crowded scenarios, which we use to validate the generalization of our methods in unobstructed scenarios.

The CrowdPose dataset contains 20K images and 80K persons labeled with 14 keypoints.

It contains 10K, 2K and 8K images for train, val and test set.

Following #OTHEREFR , we train our model on the train and validation splits combined, and report the final performance on the test set.

2) Evaluation metrics: We use the standard evaluation metric Object Keypoint Similarity (OKS) to evaluate our models, which is defined as:

Global Relation Modeling And Refinement For Bottom-Up Human Pose Estimation

Abstract: In this paper, we concern on the bottom-up paradigm in multi-person pose estimation (MPPE). Most previous bottom-up methods try to consider the relation of instances to identify different body parts during the post processing, while ignoring to model the relation among instances or environment in the feature learning process. In addition, most existing works adopt the operations of upsampling and downsampling. During the sampling process, there will be a problem of misalignment with the source features, resulting in deviations in the keypoint features learned by the model. To overcome the above limitations, we propose a convolutional neural network for bottom-up human pose estimation. It invovles two basic modules: (i) Global Relation Modeling (GRM) module globally learns relation (e.g., environment context, instance interactive information) among region of image by fusing multiple stages features in the feature learning process. It combines with the spatial-channel attention mechanism, which focuses on achieving adaptability in spatial and channel dimensions. (ii) Multi-branch Feature Align (MFA) module aggregates features from multiple branches to align fused feature and obtain refined local keypoint representation. Our model has the ability to focus on different granularity from local to global regions, which significantly boosts the performance of the multi-person pose estimation. Our results on the COCO and CrowdPose datasets demonstrate that it is an efficient framework for multi-person pose estimation.

Crowdpose: Efficient Crowded Scenes Pose Estimation And A New Benchmark

Abstract: Multi-person pose estimation is fundamental to many computer vision tasks and has made significant progress in recent years. However, few previous methods explored the problem of pose estimation in crowded scenes while it remains challenging and inevitable in many scenarios. Moreover, current benchmarks cannot provide an appropriate evaluation for such cases. In this paper, we propose a novel and efficient method to tackle the problem of pose estimation in the crowd and a new dataset to better evaluate algorithms. Our model consists of two key components: joint-candidate single person pose estimation (SPPE) and global maximum joints association. With multi-peak prediction for each joint and global association using the graph model, our method is robust to inevitable interference in crowded scenes and very efficient in inference. The proposed method surpasses the state-of-the-art methods on CrowdPose dataset by 5.2 mAP and results on MSCOCO dataset demonstrate the generalization ability of our method.


Nonetheless, we empirically note that separating clean, but hard to classify samples from noisy label samples still remains challenging for these sample selection processes #REFR , particularly for problems of large noise rates.

proaches incorporate robust loss functions #OTHEREFR , designed specifically to operate effectively with either clean or noisy labels, as well as probabilistic modeling approaches that model the data generation process #OTHEREFR .

Furthermore, training regularization #OTHEREFR imposes a penalty term on the loss function during training, thereby reducing overfitting and ameliorating generalization.

Various techniques integrate sample selection strategies as a key algorithmic step #OTHEREFR , allowing for the identification and segregation of noisy and clean labels.

One popular criterion for this sample selection process is the loss value between the prediction of the trained classifier and its label, whereby it is typically assumed that the noisy data exhibits a large loss #OTHEREFR or greater magnitude of the gradient during training #OTHEREFR .

Additionally, feature-based sample selection techniques based on the similarity to principal components of feature representations #OTHEREFR or K nearest neighbor (KNN) classification in the feature space #OTHEREFR have also been considered for sample selection criteria.

The use of peer classifiers for noisy label learning problems have been investigated for enforcing consistency in the training of the classifiers #OTHEREFR , but not for selecting clean and noisy label samples.

We argue in this paper that the prediction agreement between peer classifiers is more effective to select clean and noisy label samples than previous approaches because intuitively, such agreement is unlikely to happen, except when the classifiers agree on the clean label.

In this paper, we propose a new sample selection cri-terion based on the predictive probability agreement between peer classifiers, which has been shown in computational linguistics to be a sample reliability measure #OTHEREFR .

In light of implementation, we train three classifiers simultaneously by using the agreement between two classifiers to select samples to train the remaining classifier, as shown in Fig. 1 .

This sample selection relies on a thresholding algorithm #OTHEREFR that distinguishes samples based on how high the agreement between the peer classification predictions is.

Pass: Peer-Agreement Based Sample Selection For Training With Noisy Labels

Abstract: Noisy labels present a significant challenge in deep learning because models are prone to overfitting. This problem has driven the development of sophisticated techniques to address the issue, with one critical component being the selection of clean and noisy label samples. Selecting noisy label samples is commonly based on the small-loss hypothesis or on feature-based sampling, but we present empirical evidence that shows that both strategies struggle to differentiate between noisy label and hard samples, resulting in relatively large proportions of samples falsely selected as clean. To address this limitation, we propose a novel peer-agreement based sample selection (PASS). An automated thresholding technique is then applied to the agreement score to select clean and noisy label samples. PASS is designed to be easily integrated into existing noisy label robust frameworks, and it involves training a set of classifiers in a round-robin fashion, with peer models used for sample selection. In the experiments, we integrate our PASS with several state-of-the-art (SOTA) models, including InstanceGM, DivideMix, SSR, FaMUS, AugDesc, and C2D, and evaluate their effectiveness on several noisy label benchmark datasets, such as CIFAR-100, CIFAR-N, Animal-10N, Red Mini-Imagenet, Clothing1M, Mini-Webvision, and Imagenet. Our results demonstrate that our new sample selection approach improves the existing SOTA results of algorithms.

Combating Noisy Labels By Agreement: A Joint Training Method With Co-Regularization

Abstract: Deep Learning with noisy labels is a practically challenging problem in weakly-supervised learning. The state-of-the-art approaches"Decoupling"and"Co-teaching+"claim that the"disagreement"strategy is crucial for alleviating the problem of learning with noisy labels. In this paper, we start from a different perspective and propose a robust learning paradigm called JoCoR, which aims to reduce the diversity of two networks during training. Specifically, we first use two networks to make predictions on the same mini-batch data and calculate a joint loss with Co-Regularization for each training example. Then we select small-loss examples to update the parameters of both two networks simultaneously. Trained by the joint loss, these two networks would be more and more similar due to the effect of Co-Regularization. Extensive experimental results on corrupted data from benchmark datasets including MNIST, CIFAR-10, CIFAR-100 and Clothing1M demonstrate that JoCoR is superior to many state-of-the-art approaches for learning with noisy labels.


Another representative one-stage method is the SSD algorithm #REFR , which focuses on improving the accuracy and recall rate.

From left to right, the images are originally from MGD #OTHEREFR , USRT #OTHEREFR , and UCF #OTHEREFR datasets, respectively.

It considered detection a regression task and used an improved GoogLeNet as the backbone.

It is later extended to YOLOv2 #OTHEREFR by drawing inspiration from Faster R-CNN and the anchor mechanism.

A further improved version, YOLOv3 #OTHEREFR , uses better backbones to enhance performance.

Later, the focal loss is introduced #OTHEREFR to solve the mismatch between positive and negative samples.

Transformers #OTHEREFR have been the de-facto choice of architecture in NLP.

Since the advent of Vision Transformer #OTHEREFR , they have achieved remarkable results in computer vision tasks such as object detection.

The first representative work in this category is DETR #OTHEREFR , which eliminates the intermediate steps like nonmaximal suppression and anchor generation in CNN-based detectors.

Cctv-Gun: Benchmarking Handgun Detection In Cctv Images

Abstract: Gun violence is a critical security problem, and it is imperative for the computer vision community to develop effective gun detection algorithms for real-world scenarios, particularly in Closed Circuit Television (CCTV) surveillance data. Despite significant progress in visual object detection, detecting guns in real-world CCTV images remains a challenging and under-explored task. Firearms, especially handguns, are typically very small in size, nonsalient in appearance, and often severely occluded or indistinguishable from other small objects. Additionally, the lack of principled benchmarks and difficulty collecting relevant datasets further hinder algorithmic development. In this paper, we present a meticulously crafted and annotated benchmark, called CCTV-Gun, which addresses the challenges of detecting handguns in real-world CCTV images. Our contribution is threefold. Firstly, we carefully select and analyze real-world CCTV images from three datasets, manually annotate handguns and their holders, and assign each image with relevant challenge factors such as blur and occlusion. Secondly, we propose a new crossdataset evaluation protocol in addition to the standard intra-dataset protocol, which is vital for gun detection in practical settings. Finally, we comprehensively evaluate both classical and state-of-the-art object detection algorithms, providing an in-depth analysis of their generalizing abilities. The benchmark will facilitate further research and development on this topic and ultimately enhance security. Code, annotations, and trained models are available at https://github.com/srikarym/CCTV-Gun.

Ssd: Single Shot Multibox Detector

Abstract: We present a method for detecting objects in images using a single deep neural network. Our approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location. At prediction time, the network generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape. Additionally, the network combines predictions from multiple feature maps with different resolutions to naturally handle objects of various sizes. SSD is simple relative to methods that require object proposals because it completely eliminates proposal generation and subsequent pixel or feature resampling stages and encapsulates all computation in a single network. This makes SSD easy to train and straightforward to integrate into systems that require a detection component. Experimental results on the PASCAL VOC, COCO, and ILSVRC datasets confirm that SSD has competitive accuracy to methods that utilize an additional object proposal step and is much faster, while providing a unified framework for both training and inference. For \(300 \times 300\) input, SSD achieves 74.3 % mAP on VOC2007 test at 59 FPS on a Nvidia Titan X and for \(512 \times 512\) input, SSD achieves 76.9 % mAP, outperforming a comparable state of the art Faster R-CNN model. Compared to other single stage methods, SSD has much better accuracy even with a smaller input image size. Code is available at https://github.com/weiliu89/caffe/tree/ssd.


Such overconfidence in highly nonlinear networks is consistent with phenomena reported in large image and natural language datasets #REFR .

By contrast, LDA, which is widely used in EMG classification, demonstrated low accuracy for all datasets.

The strong constraint of sharing covariance among classes in the LDA is likely to adversely affect the classification of these datasets.

However, accuracy and confidence metrics are not necessarily positively (or negatively) correlated.

Discriminative classifiers, including Deep MLP, which shows high accuracy, tend to have worse ECE and MCE overall.

This can be observed from the reliability diagrams, indicating that neural network-based classifiers such as MLP and Deep MLP are overconfident.

By contrast, generative classifiers, in particular LDA and SMMC, showed better results in terms of ECE and MCE.

Further analysis is required to fully understand the underlying reasons.

However, generative classifiers involve stronger assumptions regarding the data generative process; thus, if this assumption is consistent with the actual data distribution, then the confidence may approach the true predictive uncertainty.

Remarkably, SMMC demonstrated superior performance in terms of both accuracy and confidence.

In particular, ECE was less than 10% for all datasets, indicating relatively few miscalibrations.

Evaluating Classifier Confidence For Surface Emg Pattern Recognition

Abstract: Surface electromyogram (EMG) can be employed as an interface signal for various devices and software via pattern recognition. In EMG-based pattern recognition, the classifier should not only be accurate, but also output an appropriate confidence (i.e., probability of correctness) for its prediction. If the confidence accurately reflects the likelihood of true correctness, then it will be useful in various application tasks, such as motion rejection and online adaptation. The aim of this paper is to identify the types of classifiers that provide higher accuracy and better confidence in EMG pattern recognition. We evaluate the performance of various discriminative and generative classifiers on four EMG datasets, both visually and quantitatively. The analysis results show that while a discriminative classifier based on a deep neural network exhibits high accuracy, it outputs a confidence that differs from true probabilities. By contrast, a scale mixture model-based classifier, which is a generative classifier that can account for uncertainty in EMG variance, exhibits superior performance in terms of both accuracy and confidence.

On Calibration Of Modern Neural Networks

Abstract: Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.


The authors showed that FAR achieved similar performance to fine-tuning on five GLUE tasks and SQuAD 2.0 #REFR while updating 6% of the parameters.

h1 = x @ W_t h2 = x @ W_f return concat([h1, h2], dim=-1)

While this approach creates additional compute overhead during training, it provides great flexibility in terms of parameter selection on modern hardware using standard frameworks like Py-Torch.

After training, the parameters can be reconfigured back removing any inference overhead.

The original paper focused on edge scenarios and used DistilBERT (66M) for their experiments.

FAR was only applied to feed-forward layers, as these make up the majority of the DistilBERT parameters.

Scaling Down To Scale Up: A Guide To Parameter-Efficient Fine-Tuning

Abstract: This paper presents a systematic overview and comparison of parameter-efficient finetuning methods covering over 40 papers published between February 2019 and February 2023. These methods aim to resolve the infeasibility and impracticality of fine-tuning large language models by only training a small set of parameters. We provide a taxonomy that covers a broad range of methods and present a detailed method comparison with a specific focus on real-life efficiency and fine-tuning multibillion-scale language models. 1 github.com/google-research/bert 2 A Recipe for Training Neural Networks, A. Karpathy

Know What You Don'T Know: Unanswerable Questions For Squad

Abstract: Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuAD 2.0, the latest version of the Stanford Question Answering Dataset (SQuAD). SQuAD 2.0 combines existing SQuAD data with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD 2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuAD 2.0 is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD 1.1 achieves only 66% F1 on SQuAD 2.0.


Non-convergence of these optimizers is documented in literature #REFR and we can confirm with our experiments that it also extends to our optimization setting.

In a first step, we investigated mean squared error loss by setting λ = 1 in our loss function defined in equation 2for a single polynomial, which revealed a learning rate of 0.1 as a reasonable setting.

We then ran tests with available TensorFlow optimizers listed in #OTHEREFR and compared their outcomes.

We found that SGD with momentum, Adam, Adamax as well as AMSgrad show the lowest losses, with a declining tendency even after 5000 epochs.

However, the training curves of Adamax and Adam exhibit recurring phases of instability every ∼ 500 epochs.

Using the AMSGrad variant of Adam eliminates this behavior with comparable remaining loss levels.

With these results in mind, we chose SGD with Nesterov momentum as non-adaptive and AMSGrad as adaptive optimizer for all further experiments, in order to work with optimizers from both paradigms.

The AMSGrad optimizer performs better on the λ = 1 optimization target, however, SGD is competitive.

The loss curves of these optimizer candidates, as well as instabilities in the Adam loss curve are shown in figure 2.

An overview of all evaluated optimizers is given in our GitHub repository at #OTHEREFR .

C K -Continuous Spline Approximation With Tensorflow Gradient Descent Optimizers

Abstract: In this work we present an "out-of-the-box" application of Machine Learning (ML) optimizers for an industrial optimization problem. We introduce a piecewise polynomial model (spline) for fitting of C k-continuos functions, which can be deployed in a cam approximation setting. We then use the gradient descent optimization context provided by the machine learning framework TensorFlow to optimize the model parameters with respect to approximation quality and C k-continuity and evaluate available optimizers. Our experiments show that the problem solution is feasible using TensorFlow gradient tapes and that AMSGrad and SGD show the best results among available TensorFlow optimizers. Furthermore, we introduce a novel regularization approach to improve SGD convergence. Although experiments show that remaining discontinuities after optimization are small, we can eliminate these errors using a presented algorithm which has impact only on affected derivatives in the local spline segment.

On The Convergence Of Adam And Beyond

Abstract: Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit example of a simple convex optimization setting where Adam does not converge to the optimal solution, and describe the precise problems with the previous analysis of Adam algorithm. Our analysis suggests that the convergence issues can be fixed by endowing such algorithms with `long-term memory' of past gradients, and propose new variants of the Adam algorithm which not only fix the convergence issues but often also lead to improved empirical performance.


As discussed in the CWI-2018 report #REFR , the performance obtained by all teams on the News domain was generally substantially higher than the performance obtained in the two other domains.

CWI-2018 received submissions by 12 teams in multiple task and track combinations.

At the end of the competition, 10 teams wrote system description papers presented at the BEA workshop.

In Table 10 (Appendices), we present the approaches by teams who submitted systems to the CWI-2018 English binary classification task and who also wrote system description papers.

An observed trend was that more teams tried deep neural networks in CWI-2018 compared to CWI-2016, a trend also observed in other areas of AI and NLP research (Section 7).

In CWI-2018's binary classification task, being sub-task 1, the organizers reported the performance from all teams in each of the three domains, namely News, WikiNews, and Wikipedia.

Lexical Complexity Prediction: An Overview

Abstract: The occurrence of unknown words in texts significantly hinders reading comprehension. To improve accessibility for specific target populations, computational modelling has been applied to identify complex words in texts and substitute them for simpler alternatives. In this paper, we present an overview of computational approaches to lexical complexity prediction focusing on the work carried out on English data. We survey relevant approaches to this problem which include traditional machine learning classifiers (e.g. SVMs, logistic regression) and deep neural networks as well as a variety of features, such as those inspired by literature in psycholinguistics as well as word frequency, word length, and many others. Furthermore, we introduce readers to past competitions and available datasets created on this topic. Finally, we include brief sections on applications of lexical complexity prediction, such as readability and text simplification, together with related studies on languages other than English.

A Report On The Complex Word Identification Shared Task 2018

Abstract: We report the findings of the second Complex Word Identification (CWI) shared task organized as part of the BEA workshop co-located with NAACL-HLT'2018. The second CWI shared task featured multilingual and multi-genre datasets divided into four tracks: English monolingual, German monolingual, Spanish monolingual, and a multilingual track with a French test set, and two tasks: binary classification and probabilistic classification. A total of 12 teams submitted their results in different task/track combinations and 11 of them wrote system description papers that are referred to in this report and appear in the BEA workshop proceedings.


It can be seen that DETR3D-Binocular converges to a poor 3D detection performance, which can compared neither to advanced CNN-based stereo 3D object detectors nor to the surround-view detection performance #REFR of DETR3D.

However, for stereo 3D object detection in driving scenes, to the best of our knowledge, there have not been any public Transformer-based detectors in the literature.

An intuitive way to introduce vision Transformers into stereo 3D object detection is to adopt the above surround-view Transformer-based 3D detectors on the KITTI #OTHEREFR dataset.

To arXiv:2304.11906v1 [cs.CV] 24 Apr 2023 achieve that, we train a binocular revision of the surround-view 3D object detector DETR3D #OTHEREFR , which is named DETR3D-Binocular.

The validation AP curve of DETR3D-Binocular during training is shown in Fig.

1a as blue circles, and its evaluation indicators after training 320 epochs are listed in Fig. 1b .

We argue that the convergence problem is caused by a lack of image correspondence supervision in surround-view 3D object detection Transformers such as DETR3D, as well as in existing monocular-based and LiDAR-based 3D object detection Transformers.

It is a fact that monocular images and LiDAR point clouds do not involve cross-view correspondence #OTHEREFR , #OTHEREFR ; and the overlapping area across images in a surroundview system is small #OTHEREFR .

Concerning the binocular vision system, however, the overlap areas between left and right images are large.

The image correspondence information in the large overlap areas, as mentioned above, is key to decoding the 3D scene.

Therefore, extracting and exploiting the image correspondence information in a Transformer-based model is a task-specific challenge of stereo 3D object detection.

Transformer-Based Stereo-Aware 3D Object Detection From Binocular Images

Abstract: Vision Transformers have shown promising progress in various object detection tasks, including monocular 2D/3D detection and surround-view 3D detection. However, when used in essential and classic stereo 3D object detection, directly adopting those surround-view Transformers leads to slow convergence and significant precision drops. We argue that one of the causes of this defect is that the surround-view Transformers do not consider the stereo-specific image correspondence information. In a surround-view system, the overlapping areas are small, and thus correspondence is not a primary issue. In this paper, we explore the model design of vision Transformers in stereo 3D object detection, focusing particularly on extracting and encoding the task-specific image correspondence information. To achieve this goal, we present TS3D, a Transformer-based Stereo-aware 3D object detector. In the TS3D, a Disparity-Aware Positional Encoding (DAPE) model is proposed to embed the image correspondence information into stereo features. The correspondence is encoded as normalized disparity and is used in conjunction with sinusoidal 2D positional encoding to provide the location information of the 3D scene. To extract enriched multiscale stereo features, we propose a Stereo Reserving Feature Pyramid Network (SRFPN). The SRFPN is designed to reserve the correspondence information while fusing intra-scale and aggregating cross-scale stereo features. Our proposed TS3D achieves a 41.29% Moderate Car detection average precision on the KITTI test set and takes 88 ms to detect objects from each binocular image pair. It is competitive with advanced counterparts in terms of both precision and inference speed.

Nuscenes: A Multimodal Dataset For Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image-based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first published dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online at this http URL.


To this end, we introduce SuMoCo, a novel GPUfriendly framework based on its unsupervised counterpart, MoCo #REFR .

This task involves classifying a driver's actions into "normal driving" and several non-driving-related activities (NDRAs) such as talking and drinking.

Existing approaches #OTHEREFR typically employ a naive fusion method that combines the multiview multimodal data at the decision level.

However, this approach fails to exploit the complementarity of the semantic features from different views and modalities and does not consider their relative importance, leading to underperformance.

Hence, we propose DMSs based on feature-level fusion with self-attention, as depicted in Fig. 1 , to address these limitations.

Following the prior work #OTHEREFR , we also train our models with supervised contrastive learning (CL).

Given that drivers can perform indefinite non-drivingrelated activities, we evaluate our proposed methods on the Driver Anomaly Detection Dataset (DAD) #OTHEREFR , which is designed explicitly for open-set NDRA detection.

The dataset consists of only two categories, namely "normal" and "anomalous" (the class for all NDRAs).

However, identifying the specific type of NDRA is critical in practice as they pose varying degrees of risk from inattention.

Therefore, we manually annotate DAD with fine-grained labels, such as "drinking", "talking" and "texting", to enable the multi-classification of NDRAs.

Our contributions in this paper are three-fold as follows: 1.

Robust Multiview Multimodal Driver Monitoring System Using Masked Multi-Head Self-Attention

Abstract: Driver Monitoring Systems (DMSs) are crucial for safe handover actions in Level-2+ self-driving vehicles. Stateof-the-art DMSs leverage multiple sensors mounted at different locations to monitor the driver and the vehicle's interior scene and employ decision-level fusion to integrate these heterogenous data. However, this fusion method may not fully utilize the complementarity of different data sources and may overlook their relative importance. To address these limitations, we propose a novel multiview multimodal driver monitoring system based on feature-level fusion through multi-head self-attention (MHSA). We demonstrate its effectiveness by comparing it against four alternative fusion strategies (Sum, Conv, SE, and AFF). We also present a novel GPU-friendly supervised contrastive learning framework SuMoCo to learn better representations. Furthermore, We fine-grained the test split of the DAD dataset to enable the multi-class recognition of drivers' activities. Experiments on this enhanced database demonstrate that 1) the proposed MHSA-based fusion method (AUC-ROC: 97.0%) outperforms all baselines and previous approaches, and 2) training MHSA with patch masking can improve its robustness against modality/view collapses. The code and annotations are publicly available 1 .

Momentum Contrast For Unsupervised Visual Representation Learning

Abstract: We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.


Without updating the whole architecture on the on-demand data, ideally, the large backbone LLM can be kept fully frozen, with only the comparatively light VPG module being (fully or partially) updated. #REFR Motivation. However, building a VL-LLM is inevitably computational costing.

To construct a VL-LLM, a visual prompt generator (VPG) module (cf. Fig.

1 ) that produces soft prompts for the input images/videos is added #OTHEREFR at the bottom of the backbone LLM for bridging the gap between vision and language modalities.

Currently, such architecture has been frequently adopted by many popular VL-LLMs.

For example, BLIP-2 pre-train a CLIP-ViT #OTHEREFR combined with a Q-Former as VPG; PaLM-E explores several different architectures, e.g. ViT #OTHEREFR or OSRT #OTHEREFR as VPG.

To obtain the final VL-LLM, the VPG needs to be tuned.

Because even though the backbone LLM is frozen, tuning the light VPG and projector parts still requires additional huge computation resources to cover the LLM load that accounts for the main overhead.

For example, tuning the VPG and projector for BLIP2-FlanT5 XXL still needs over 600 A100-GPU hours on over 100 million image-text pairs, where the LLM takes up 70% of the GPU memory consumption.

Thus, whenever developing a new VL-LLM with any newly-emerging LLM, it can be prohibitively expensive to train the VPG and projector.

Hopefully, transferring a pre-trained VPG (which is the main body of trainable parts) from an existing VL-LLM to a novel LLM instead of training from scratch #OTHEREFR , offers a promising solution for relieving the cost issue.

Intuitively, all the VL-LLMs literally can share the same VPG infrastructure and utility, #OTHEREFR which makes the VPG transferring theoretically feasible.

Transfer Visual Prompt Generator Across Llms

Abstract: While developing a new vision-language LLM (VL-LLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a comparatively lightweight visual prompt generator (VPG) becomes a feasible paradigm. However, further tuning the VPG part of the VL-LLM still suffers from indispensable computational costs, i.e., requiring thousands of GPU hours and millions of training data. One alternative solution is to transfer an existing VPG from any existing VL-LLMs for the target VL-LLM.

Spice: Semantic Propositional Image Caption Evaluation

Abstract: There is considerable interest in the task of automatically generating image captions. However, evaluation is challenging. Existing automatic evaluation metrics are primarily sensitive to n-gram overlap, which is neither necessary nor sufficient for the task of simulating human judgment. We hypothesize that semantic propositional content is an important component of human caption evaluation, and propose a new automated caption evaluation metric defined over scene graphs coined SPICE. Extensive evaluations across a range of models and datasets indicate that SPICE captures human judgments over model-generated captions better than other automatic metrics (e.g., system-level correlation of 0.88 with human judgments on the MS COCO dataset, versus 0.43 for CIDEr and 0.53 for METEOR). Furthermore, SPICE can answer questions such as which caption-generator best understands colors? and can caption-generators count?


The pathwidth pw(G) of graph G is the minimum width over all TDs of G whose trees are paths. Proposition II.4 (LB for Pathwidth #REFR ).

The treewidth tw(G) of G is the minimum width(T ) over all TDs T of G. For QSAT, the following tractability result is known.

Proposition II.3 (Treewidth UB #OTHEREFR ).

Given any CQBF Q of quantifier depth ℓ with k=tw(G Q ).

QSAT on Q can be decided in time tow(ℓ, O(k)) • poly(|var(Q)|).

However, it is not expected that one can significantly improve this, since already for the weaker pathwidth there are limits.

Given any CQBF Q of quantifier depth ℓ with k = pw(G Q ).

Then, under ETH, QSAT on Q cannot be decided in time tow(ℓ, o(k)) • poly(|var(Q)|).

Then, a treedepth decomposition T = (V, F ) of G is a forest of rooted trees, where for every edge {u, v} ∈ E we require that u is an ancestor or descendant of v in T .

Structure-Aware Lower Bounds And Broadening The Horizon Of Tractability For Qbf

Abstract: The QSAT problem, which asks to evaluate a quantified Boolean formula (QBF), is of fundamental interest in approximation, counting, decision, and probabilistic complexity and is also considered the prototypical PSPACEcomplete problem. As such, it has previously been studied under various structural restrictions (parameters), most notably parameterizations of the primal graph representation of instances. Indeed, it is known that QSAT remains PSPACE-complete even when restricted to instances with constant treewidth of the primal graph, but the problem admits a double-exponential fixed-parameter algorithm parameterized by the vertex cover number (primal graph). However, prior works have left a gap in our understanding of the complexity of QSAT when viewed from the perspective of other natural representations of instances, most notably via incidence graphs. In this paper, we develop structure-aware reductions which allow us to obtain essentially tight lower bounds for highly restricted instances of QSAT, including instances whose incidence graphs have bounded treedepth or feedback vertex number. We complement these lower bounds with novel algorithms for QSAT which establish a nearly-complete picture of the problem's complexity under standard graph-theoretic parameterizations. We also show implications for other natural graph representations, and obtain novel upper as well as lower bounds for QSAT under more fine-grained parameterizations of the primal graph.

Te-Eth: Lower Bounds For Qbfs Of Bounded Treewidth

Abstract: The problem of deciding the validity (QSAT) of quantified Boolean formulas (QBF) is a vivid research area in both theory and practice. In the field of parameterized algorithmics, the well-studied graph measure treewidth turned out to be a successful parameter. A well-known result by Chen in parameterized complexity is that QSAT when parameterized by the treewidth of the primal graph of the input formula together with the quantifier depth of the formula is fixed-parameter tractable. More precisely, the runtime of such an algorithm is polynomial in the formula size and exponential in the treewidth, where the exponential function in the treewidth is a tower, whose height is the quantifier depth. A natural question is whether one can significantly improve these results and decrease the tower while assuming the Exponential Time Hypothesis (ETH). In the last years, there has been a growing interest in the quest of establishing lower bounds under ETH, showing mostly problem-specific lower bounds up to the third level of the polynomial hierarchy. Still, an important question is to settle this as general as possible and to cover the whole polynomial hierarchy. In this work, we show lower bounds based on the ETH for arbitrary QBFs parameterized by treewidth (and quantifier depth). More formally, we establish lower bounds for QSAT and treewidth, namely, that under ETH there cannot be an algorithm that solves QSAT of quantifier depth i in runtime significantly better than i-fold exponential in the treewidth and polynomial in the input size. In doing so, we provide a versatile reduction technique to compress treewidth that encodes the essence of dynamic programming on arbitrary tree decompositions. Further, we describe a general methodology for a more fine-grained analysis of problems parameterized by treewidth that are at higher levels of the polynomial hierarchy.


CXR to evaluate our selected models on the challenging CXR modality, we used CheXpert #REFR , a large publicly available dataset containing 224 316 chest radiographs.

We also expanded our evaluation with two additional modalities, namely Chest X-ray (CXR) and Retinal Fundus Photography (RF), in an attempt to establish a more comprehensive benchmark for UPD.

Brain-MRI We used the Cambridge Centre for Ageing and Neuroscience dataset (CamCAN) #OTHEREFR as our source of lesionfree scans for all Brain-MRI datasets.

It contains T1 and T2-weighted brain scans of 652 healthy adult subjects.

The Anatomical Tracings of Lesions After Stroke (ATLAS) dataset #OTHEREFR provided us with 220 T1-weighted scans of stroke patients, which we use for evaluation.

We also used the 2020 version of the Multimodal Brain Tumor Segmentation (BraTS) Challenge dataset for evaluation #OTHEREFR - #OTHEREFR .

Samples are labeled as positive, negative, or uncertain for 14 observations, 12 of which are pathology related.

The remaining two regard the presence of support devices or the absence of all pathologies, labeled as "no finding".

We use the last observation for our "healthy" subset but we do so with caution as the label only implies the absence of the 12 pre-defined pathologies of interest, a common limitation in CXR datasets.

We limited the task by using images annotated with exactly one out of three of the most common observations, namely "pleural effusion", "opacity" or "enlarged cardiomediastinal contour".

We avoided possible confounders and biases by discarding scans with support devices, using only radiographs of anteroposterior view, and by enforcing an equal number of samples of male and female patients on all consequent subsets.

Unsupervised Pathology Detection: A Deep Dive Into The State Of The Art

Abstract: Deep unsupervised approaches are gathering increased attention for applications such as pathology detection and segmentation in medical images since they promise to alleviate the need for large labeled datasets and are more generalizable than their supervised counterparts in detecting any kind of rare pathology. As the Unsupervised Anomaly Detection (UAD) literature continuously grows and new paradigms emerge, it is vital to continuously evaluate and benchmark new methods in a common framework, in order to reassess the state-ofthe-art (SOTA) and identify promising research directions. To this end, we evaluate a diverse selection of cutting-edge UAD methods on multiple medical datasets, comparing them against the established SOTA in UAD for brain MRI. Our experiments demonstrate that newly developed feature-modeling methods from the industrial and medical literature achieve increased performance compared to previous work and set the new SOTA in a variety of modalities and datasets. Additionally, we show that such methods are capable of benefiting from recently developed self-supervised pre-training algorithms, further increasing their performance. Finally, we perform a series of experiments in order to gain further insights into some unique characteristics of selected models and datasets. Our code can be found under https://github.com/iolag/UPD study/.

Chexpert: A Large Chest Radiograph Dataset With Uncertainty Labels And Expert Comparison

Abstract: Large, labeled datasets have driven deep learning methods to achieve expert-level performance on a variety of medical imaging tasks. We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. We investigate different approaches to using the uncertainty labels for training convolutional neural networks that output the probability of these observations given the available frontal and lateral radiographs. On a validation set of 200 chest radiographic studies which were manually annotated by 3 board-certified radiologists, we find that different uncertainty approaches are useful for different pathologies. We then evaluate our best model on a test set composed of 500 chest radiographic studies annotated by a consensus of 5 board-certified radiologists, and compare the performance of our model to that of 3 additional radiologists in the detection of 5 selected pathologies. On Cardiomegaly, Edema, and Pleural Effusion, the model ROC and PR curves lie above all 3 radiologist operating points. We release the dataset to the public as a standard benchmark to evaluate performance of chest radiograph interpretation models. ::: The dataset is freely available at this https URL .


The standard OMP algorithm is a special case of Algorithm 1 when G is the least-squares loss function, i.e., G(z) = F (z) = y − Az #REFR 2 and for λ = 0.

, K do 6: Find j (k) ∈ arg max j∈[N ] ∆(x (k−1) , S (k−1) , j), with ∆ defined as in (2) 7: Define S (k) = S (k−1) ∪ {j (k) } 8:

Compute x (k) by solving (3) return x (K) 11: end procedure Remark 1 (Standard OMP).

To demonstrate that Algorithm 1 is practically implementable, we ought to show that the loss reduction factor ∆(x, S, j) is (ideally, easily) computable.

The main technical contribution of the paper is to show that this is indeed the case for the weighted LASSO, SR-LASSO, and LAD-LASSO loss functions (referred to as " * -LASSO" below).

This is summarized in the following result, which unifies Theorems 2, 3 and 4.

The Greedy Side Of The Lasso: New Algorithms For Weighted Sparse Recovery Via Loss Function-Based Orthogonal Matching Pursuit

Abstract: We propose a class of greedy algorithms for weighted sparse recovery by considering new loss function-based generalizations of Orthogonal Matching Pursuit (OMP). Given a (regularized) loss function, the proposed algorithms alternate the iterative construction of the signal support via greedy index selection and a signal update based on solving a local data-fitting problem restricted to the current support. We show that greedy selection rules associated with popular weighted sparsity-promoting loss functions admit explicitly computable and simple formulas. Specifically, we consider ℓ 0-and ℓ 1-based versions of the weighted LASSO (Least Absolute Shrinkage and Selection Operator), the Square-Root LASSO (SR-LASSO) and the Least Absolute Deviations LASSO (LAD-LASSO). Through numerical experiments on Gaussian compressive sensing and high-dimensional function approximation, we demonstrate the effectiveness of the proposed algorithms and empirically show that they inherit desirable characteristics from the corresponding loss functions, such as SR-LASSO's noise-blind optimal parameter tuning and LAD-LASSO's fault tolerance. In doing so, our study sheds new light on the connection between greedy sparse recovery and convex relaxation.

Correcting For Unknown Errors In Sparse High-Dimensional Function Approximation

Abstract: We consider sparsity-based techniques for the approximation of high-dimensional functions from random pointwise evaluations. To date, almost all the works published in this field contain some a priori assumptions about the error corrupting the samples that are hard to verify in practice. In this paper, we instead focus on the scenario where the error is unknown. We study the performance of four sparsity-promoting optimization problems: weighted quadratically-constrained basis pursuit, weighted LASSO, weighted square-root LASSO, and weighted LAD-LASSO. From the theoretical perspective, we prove uniform recovery guarantees for these decoders, deriving recipes for the optimal choice of the respective tuning parameters. On the numerical side, we compare them in the pure function approximation case and in applications to uncertainty quantification of ODEs and PDEs with random inputs. Our main conclusion is that the lesser-known square-root LASSO is better suited for high-dimensional approximation than the other procedures in the case of bounded noise, since it avoids (both theoretically and numerically) the need for parameter tuning.


A related phenomenon has been observed on neuromorphic chips, which optimally perform in simple tasks when in the subcritical regime, whereas harder tasks require progressively more critical dynamics #REFR .

A: Environment with 50 organisms (red circles with trails) foraging for food (green dots). B: A network with 12 neurons. Four sensory, four hidden and four motor neurons. All allowed edges are displayed.

The exact topology and edge weights are subject to change by the evolutionary algorithm.

decreasing the system size, making the task less complex, or introducing noise to the system pushes the optimal regime further into the subcritical range #OTHEREFR . Similarly #OTHEREFR

(2007) observed, that whereas information propagation is maximized in critical RBNs, the optimal regime shifts slightly into the subcritical regime under the presence of noise.

In recordings from the nervous systems of different animals, slightly subcritical behaviour was observed #OTHEREFR .

The escape dynamics of schooling fish remain in subcritical state even under pharmacological manipulation that increases alertness, though more alert fish self-organize closer to critical state #OTHEREFR .

Finally, for some applications the combination of systems operating at different distances from criticality has been shown to lead to optimal results #OTHEREFR .

The disordered supercritical state has been universally observed to perform poorly (S. A. #OTHEREFR .

There are different critical transitions in the examples mentioned above, thus different definitions of what it means to be subcritical.

For example, for the order/disorder transition, subcritical refers to the ordered state.

When To Be Critical? Performance And Evolvability In Different Regimes Of Neural Ising Agents

Abstract: It has long been hypothesized that operating close to the critical state is beneficial for natural, artificial and their evolutionary systems. We put this hypothesis to test in a system of evolving foraging agents controlled by neural networks that can adapt agents' dynamical regime throughout evolution. Surprisingly, we find that all populations that discover solutions, evolve to be subcritical. By a resilience analysis, we find that there are still benefits of starting the evolution in the critical regime. Namely, initially critical agents maintain their fitness level under environmental changes (for example, in the lifespan) and degrade gracefully when their genome is perturbed. At the same time, initially subcritical agents, even when evolved to the same fitness, are often inadequate to withstand the changes in the lifespan and degrade catastrophically with genetic perturbations. Furthermore, we find the optimal distance to criticality depends on the task complexity. To test it we introduce a hard and simple task: for the hard task, agents evolve closer to criticality whereas more subcritical solutions are found for the simple task. We verify that our results are independent of the selected evolutionary mechanisms by testing them on two principally different approaches: a genetic algorithm and an evolutionary strategy. In summary, our study suggests that although optimal behaviour in the simple task is obtained in a subcritical regime, initializing near criticality is important to be efficient at finding optimal solutions for new tasks of unknown complexity.

Control Of Criticality And Computation In Spiking Neuromorphic Networks With Plasticity

Abstract: The critical state is assumed to be optimal for any computation in recurrent neural networks, because criticality maximizes a number of abstract computational properties. We challenge this assumption by evaluating the performance of a spiking recurrent neural network on a set of tasks of varying complexity at - and away from critical network dynamics. To that end, we developed a spiking network with synaptic plasticity on a neuromorphic chip. We show that the distance to criticality can be easily adapted by changing the input strength, and then demonstrate a clear relation between criticality, task-performance and information-theoretic fingerprint. Whereas the information-theoretic measures all show that network capacity is maximal at criticality, this is not the case for performance on specific tasks: Only the complex, memory-intensive tasks profit from criticality, whereas the simple tasks suffer from it. Thereby, we challenge the general assumption that criticality would be beneficial for any task, and provide instead an understanding of how the collective network state should be tuned to task requirement to achieve optimal performance.


In this study, we train and compare our models on the publicly available High-Gamma Dataset #REFR .

It contains 128-channel EEG recordings of fourteen healthy subjects executing three different movement classes and a resting condition.

The movements are executed by the left hand, the right hand, and both feet.

During each trial, the subject is asked to execute one of the movement or a rest for a duration of four seconds.

The exact number of trials varied between subjects but was balanced between the four classes.

For each subject, approximately 1000 trials were recorded in a single session structured into 13 runs.

An Embedding For Eeg Signals Learned Using A Triplet Loss

Abstract: Neurophysiological time series recordings like the electroencephalogram (EEG) or local field potentials are obtained from multiple sensors. They can be decoded by machine learning models in order to estimate the ongoing brain state of a patient or healthy user. In a brain-computer interface (BCI), this decoded brain state information can be used with minimal time delay to either control an application, e.g., for communication or for rehabilitation after stroke, or to passively monitor the ongoing brain state of the subject, e.g., in a demanding work environment. A specific challenge in such decoding tasks is posed by the small dataset sizes in BCI compared to other domains of machine learning like computer vision or natural language processing. A possibility to tackle classification or regression problems in BCI despite small training data sets is through transfer learning, which utilizes data from other sessions, subjects or even datasets to train a model. In this exploratory study, we propose novel domain-specific embeddings for neurophysiological data. Our approach is based on metric learning and builds upon the recently proposed ladder loss. Using embeddings allowed us to benefit, both from the good generalisation abilities and robustness of deep learning and from the fast training of classical machine learning models for subject-specific calibration. In offline analyses using EEG data of 14 subjects, we tested the embeddings' feasibility and compared their efficiency with state-of-the-art deep learning models and conventional machine learning pipelines. For within-subject classification of motor tasks, our data suggests that embeddings significantly improve the decoding accuracy compared to an established baseline model. In between-subjects transfer scenarios, pre-trained embeddings allow for an extremely fast fine-tuning of a classifier for a novel subject, even if only a few trials per class are available, which currently can not be achieved with regular deep learning models. And in the scenarios where one subject is completely set aside for testing without specific calibration, the embeddings were not competitive. In summary, we propose the use of metric learning to obtain pre-trained embeddings of EEG-BCI data as a means to incorporate domain knowledge and to reach competitive performance on novel subjects with minimal calibration requirements.

Deep Learning With Convolutional Neural Networks For Decoding And Visualization Of Eeg Pathology

Abstract: We apply convolutional neural networks (ConvNets) to the task of distinguishing pathological from normal EEG recordings in the Temple University Hospital EEG Abnormal Corpus. We use two basic, shallow and deep ConvNet architectures recently shown to decode task-related information from EEG at least as well as established algorithms designed for this purpose. In decoding EEG pathology, both ConvNets reached substantially better accuracies (about 6% better, ~85% vs. ~79%) than the only published result for this dataset, and were still better when using only 1 minute of each recording for training and only six seconds of each recording for testing. We used automated methods to optimize architectural hyperparameters and found intriguingly different ConvNet architectures, e.g., with max pooling as the only nonlinearity. Visualizations of the ConvNet decoding behavior showed that they used spectral power changes in the delta (0-4 Hz) and theta (4-8 Hz) frequency range, possibly alongside other features, consistent with expectations derived from spectral analysis of the EEG data and from the textual medical reports. Analysis of the textual medical reports also highlighted the potential for accuracy increases by integrating contextual information, such as the age of subjects. In summary, the ConvNets and visualization techniques used in this study constitute a next step towards clinically useful automated EEG diagnosis and establish a new baseline for future work on this topic.


The original U-Net #REFR , which is proposed with symmetrical encoder-decoder architecture, is widely used in image segmentation.

In particular, skip connections are used to combine the high-level semantic features from the decoder and low-level details from the encoder.

Based on this architecture, a number of improved network architectures have been further proposed.

Res-UNet #OTHEREFR and Dense-UNet #OTHEREFR replace each sub-module of the U-Net with a residual connection or a dense connection, respectively.

U-Net++ #OTHEREFR redesigns skip connection pathways to explore the optimal depth by fusing multiple U-Net sub-networks with varying depths, where each sub-network is connected through a series of nested, dense skip pathways.

U-Net3+ #OTHEREFR proposes full-scale skip connections to fully exploit multi-scale feature representations.

Small But Mighty: Enhancing 3D Point Clouds Semantic Segmentation With U-Next Framework

Abstract: We study the problem of semantic segmentation of largescale 3D point clouds. In recent years, significant research efforts have been directed toward local feature aggregation, improved loss functions and sampling strategies. While the fundamental framework of point cloud semantic segmentation has been largely overlooked, with most existing approaches rely on the U-Net architecture by default. In this paper, we propose U-Next, a small but mighty framework designed for point cloud semantic segmentation. The key to this framework is to learn multi-scale hierarchical representations from semantically similar feature maps. Specifically, we build our U-Next by stacking multiple U-Net L 1 codecs in a nested and densely arranged manner to minimize the semantic gap, while simultaneously fusing the feature maps across scales to effectively recover the fine-grained details. We also devised a multi-level deep supervision mechanism to further smooth gradient propagation and facilitate network optimization. Extensive experiments conducted on three large-scale benchmarks including S3DIS, Toronto3D, and SensatUrban demonstrate the superiority and the effectiveness of the proposed U-Next architecture. Our U-Next architecture shows consistent and visible performance improvements across different tasks and baseline models, indicating its great potential to serve as a general framework for future research.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .


The original Seq2Seq-based methods train the encoder and decoder with an attention mechanism to predict the most probable word that emerges next in the summary text sequence #REFR .

Among these, question summarization has gained increasing attention from the research community.

Medical question summarization (MQS) is the task of summarizing longer consumer health questions (CHQs) into shorter ones, i.e., frequently asked questions (FAQs), which contain the essential intent of original CHQs.

Compared with lengthy CHQs, FAQs focus on the key information of health questions and filter out unnecessary and redundant information, which is helpful for MQAs to retrieve correct answers.

In recent, there have been various deep learning-based approaches for medical question summarization, which can be categorized into three groups: Seq2Seq-based methods, RL (reinforcement learning)-enhanced methods, and CL (contrastive learning)-enhanced methods #OTHEREFR .

Some variants of Seq2Seq, including BART #OTHEREFR , pointer-generator network , and multi-task learning framework #OTHEREFR , have also been applied to the MQS task.

However, these models rely on maximum likelihood estimation (MLE), which prioritizes the accuracy of masked token prediction over the preservation of semantic similarity or dissimilarity of questions #OTHEREFR .

To address this limitation, RL-enhanced methods attempt to utilize reinforcement learning with question-aware semantic rewards to enhance the ability of Seq2Seq-based framework on the MQS task.

These methods create reasonable rewards, such as question-type identification and question-focus recognition, to ensure the quality of generated question summary #OTHEREFR .

However, RL-enhanced methods still suffer from noise gradient estimation problems, which results in unstable training #OTHEREFR , and require substantial annotated training data for the devised rewards.

Medical Question Summarization With Entity-Driven Contrastive Learning

Abstract: By summarizing longer consumer health questions into shorter and essential ones, medical question answering (MQA) systems can more accurately understand consumer intentions and retrieve suitable answers. However, medical question summarization is very challenging due to obvious distinctions in health trouble descriptions from patients and doctors. Although existing works have attempted to utilize Seq2Seq, reinforcement learning, or contrastive learning to solve the problem, two challenges remain: how to correctly capture question focus to model its semantic intention, and how to obtain reliable datasets to fairly evaluate performance. To address these challenges, this paper proposes a novel medical question summarization framework using entity-driven contrastive learning (ECL). ECL employs medical entities in frequently asked questions (FAQs) as focuses and devises an effective mechanism to generate hard negative samples. This approach forces models to pay attention to the crucial focus information and generate more ideal question summarization. Additionally, we find that some MQA datasets suffer from serious data leakage problems, such as the iCliniq dataset's 33% duplicate rate. To evaluate the related methods fairly, this paper carefully checks leaked samples to reorganize more reasonable datasets. Extensive experiments demonstrate that our ECL method outperforms state-of-the-art methods by accurately capturing question focus and generating medical question summaries. The code and datasets are available at https://github.com/yrbobo/MQS-ECL.

Pegasus: Pre-Training With Extracted Gap-Sentences For Abstractive Summarization

Abstract: Recent work pre-training Transformers with self-supervised objectives on large text corpora has shown great success when fine-tuned on downstream NLP tasks including text summarization. However, pre-training objectives tailored for abstractive text summarization have not been explored. Furthermore there is a lack of systematic evaluation across diverse domains. In this work, we propose pre-training large Transformer-based encoder-decoder models on massive text corpora with a new self-supervised objective. In PEGASUS, important sentences are removed/masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary. We evaluated our best PEGASUS model on 12 downstream summarization tasks spanning news, science, stories, instructions, emails, patents, and legislative bills. Experiments demonstrate it achieves state-of-the-art performance on all 12 downstream datasets measured by ROUGE scores. Our model also shows surprising performance on low-resource summarization, surpassing previous state-of-the-art results on 6 datasets with only 1000 examples.


To address this challenge, a customized backbone with FPN feature fusion #REFR is designed to extract features at three levels.

We build our text recognition module following the fashion of SAR #OTHEREFR , which provides the required attention location while having a light-weighted framework.

It is worth noting that the recognition baseline can be replaced by other attention-based recognizers to pursue better performance. In the wild, texts are varying in scale.

Given the input image X ∈ R H×W ×C , the extracted visual features at different levels are denoted as

F 1 ∈ R H 2 × W 2 ×C , F 2 ∈ R H 4 × W 4 ×C , and F 3 ∈ R H 8 × W 4

Subsequently, by training adaptive weights at nearly free cost, the fused features manage to keep the scale-invariance between different levels, represented as:

F l f use = α l • F 1→l + β l • F 2→l + γ l • F 3→l (1)

Weakly-Supervised Text Instance Segmentation

Abstract: Text segmentation is a challenging vision task with many downstream applications. Current text segmentation methods require pixel-level annotations, which are expensive in the cost of human labor and limited in application scenarios. In this paper, we take the first attempt to perform weaklysupervised text instance segmentation by bridging text recognition and text segmentation. The insight is that text recognition methods provide precise attention position of each text instance, and the attention location can feed to both a text adaptive refinement head (TAR) and a text segmentation head. Specifically, the proposed TAR generates pseudo labels by performing two-stage iterative refinement operations on the attention location to fit the accurate boundaries of the corresponding text instance. Meanwhile, the text segmentation head takes the rough attention location to predict segmentation masks which are supervised by the aforementioned pseudo labels. In addition, we design a mask-augmented contrastive learning by treating our segmentation result as an augmented version of the input text image, thus improving the visual representation and further enhancing the performance of both recognition and segmentation. The experimental results demonstrate that the proposed method significantly outperforms weakly-supervised instance segmentation methods on ICDAR13-FST (18.95% improvement) and TextSeg (17.80% improvement) benchmarks.

Learning Spatial Fusion For Single-Shot Object Detection

Abstract: Pyramidal feature representation is the common practice to address the challenge of scale variation in object detection. However, the inconsistency across different feature scales is a primary limitation for the single-shot detectors based on feature pyramid. In this work, we propose a novel and data driven strategy for pyramidal feature fusion, referred to as adaptively spatial feature fusion (ASFF). It learns the way to spatially filter conflictive information to suppress the inconsistency, thus improving the scale-invariance of features, and introduces nearly free inference overhead. With the ASFF strategy and a solid baseline of YOLOv3, we achieve the best speed-accuracy trade-off on the MS COCO dataset, reporting 38.1% AP at 60 FPS, 42.4% AP at 45 FPS and 43.9% AP at 29 FPS. The code is available at this https URL