Computational Overhead And The Number Of Parameters

When using our model with the deeper backbone denoted denoted by ours †, the number of model parameters increases, but it does not increase significantly compared to other comparison groups, where is still similar to that of LF-Net #REFR .

The right table shows the number of parameters in millions, where the first group (top) are descriptor-only models and the second group (bottom) are joint detection and description models.

Our model in the first row has a second smallest model size among those of descriptoronly models.

Learning Rotation-Equivariant Features For Visual Correspondence

Abstract: Extracting discriminative local features that are invariant to imaging variations is an integral part of establishing correspondences between images. In this work, we introduce a self-supervised learning framework to extract discriminative rotation-invariant descriptors using groupequivariant CNNs. Thanks to employing group-equivariant CNNs, our method effectively learns to obtain rotationequivariant features and their orientations explicitly, without having to perform sophisticated data augmentations. The resultant features and their orientations are further processed by group aligning, a novel invariant mapping technique that shifts the group-equivariant features by their orientations along the group dimension. Our group aligning technique achieves rotation-invariance without any collapse of the group dimension and thus eschews loss of discriminability. The proposed method is trained end-to-end in a self-supervised manner, where we use an orientation alignment loss for the orientation estimation and a contrastive descriptor loss for robust local descriptors to geometric/photometric variations. Our method demonstrates state-of-the-art matching accuracy among existing rotationinvariant descriptors under varying rotation and also shows competitive results when transferred to the task of keypoint matching and camera pose estimation.

Lf-Net: Learning Local Features From Images

Abstract: We present a novel strategy to learn a local feature pipeline from collections of images with deep networks, without the need for human supervision. To do so we frame the learning problem with a two-branch network. We posit that training both branches with a standard Siamese architecture is not feasible, as solving the correspondence problem jointly with feature learning is too challenging, and does not converge well enough to train from scratch. Instead, we propose to break differentiability on one branch and use ground-truth geometry to remove the burden of solving the correspondence problem while training, while keeping the other fully-differentiable. In order to train this setup with gradient-based methods, we optimize for the differentiable branch while using the parameters from the previous training step for the other, and demonstrate that both converge to a single, optimal solution. Our method can be trained with only the relative camera pose and depth information---furthermore, we show that this ground truth does not need to be perfect and can be easily obtained with off-the-shelf Structure-from-Motion solutions. Our models outperform the state of the art on sparse feature matching on both indoor and outdoor datasets, while running at 60+ fps for QVGA images.


Vi. Solving The Conditions Of Optimality Using A Modified Physics Informed Neural Network

IV, computing the uncontrolled PDFs for the deterministic dynamics (i.e., δ = 0) requires inverting #REFR .

Additionally, to satisfy compute constraints, we uniformly randomly sampled 35,000 samples every 40,000 epochs.

For computing the Sinkhorn losses at the endpoint boundary conditions, we use the entropic regularization parameter (see (11)) ε = 0.1. Fig.

2 depicts fifty optimally controlled state sample paths for this simulation.

These sample paths are obtained via closed-loop simulation with the optimal control policy u opt resulting from the training of the PINN. Fig.

3 shows the snaphsots of the univariate marginal PDFs under optimal control and the same without control, for the aforesaid numerical simulation. Following Sec.

We used the methodof-characteristics #OTHEREFR to solve the corresponding unforced Liouville PDE, thereby obtaining the uncontrolled joint PDF snapshots.

The marginals ρ unc i , i ∈ 3 , in Fig. 3 were obtained by numerically integrating these uncontrolled joints.

Optimal Mass Transport Over The Euler Equation

Abstract: We consider the finite horizon optimal steering of the joint state probability distribution subject to the angular velocity dynamics governed by the Euler equation. The problem and its solution amounts to controlling the spin of a rigid body via feedback, and is of practical importance, for example, in angular stabilization of a spacecraft with stochastic initial and terminal states. We clarify how this problem is an instance of the optimal mass transport (OMT) problem with bilinear prior drift. We deduce both static and dynamic versions of the Eulerian OMT, and provide analytical and numerical results for the synthesis of the optimal controller.

Wasserstein Proximal Algorithms For The Schr\"{O}Dinger Bridge Problem: Density Control With Nonlinear Drift

Abstract: We study the Schrodinger bridge problem (SBP) with nonlinear prior dynamics. In control-theoretic language, this is a problem of minimum effort steering of a given joint state probability density function (PDF) to another over a finite time horizon, subject to a controlled stochastic differential evolution of the state vector. For generic nonlinear drift, we reduce the SBP to solving a system of forward and backward Kolmogorov partial differential equations (PDEs) that are coupled through the boundary conditions, with unknowns being the "Schrodinger factors" -- so named since their product at any time yields the optimal controlled joint state PDF at that time. We show that if the drift is a gradient vector field, or is of mixed conservative-dissipative nature, then it is possible to transform these PDEs into a pair of initial value problems (IVPs) involving the same forward Kolmogorov operator. Combined with a recently proposed fixed point recursion that is contractive in the Hilbert metric, this opens up the possibility to numerically solve the SBPs in these cases by computing the Schrodinger factors via a single IVP solver for the corresponding (uncontrolled) forward Kolmogorov PDE. The flows generated by such forward Kolmogorov PDEs, for the two aforementioned types of drift, in turn, enjoy gradient descent structures on the manifold of joint PDFs with respect to suitable distance functionals. We employ a proximal algorithm developed in our prior work, that exploits this geometric viewpoint, to solve these IVPs and compute the Schrodinger factors via weighted scattered point cloud evolution in the state space. We provide the algorithmic details and illustrate the proposed framework of solving the SBPs with nonlinear prior dynamics by numerical examples.


Similarly, we show results for the TUT18 task #REFR , sampled from the HARES benchmark, in Table A2 .

Here, we review results for our primary downstream tasks from external, high-performing models.

We provide these results to contextualize performance on these tasks further, and compare to our strongest result in each case.

In Table A1 , we assemble a collection of recent top results on VGGSound #OTHEREFR .

We restrict ourselves to those with which we share an evaluation metric (Top-1 Accuracy, which is common in recent VGGSound evaluations).

These methods involve any combination of supervision during pretraining, highly optimized state-of-the-art architectures like multimodal transformers, and larger pretraining sets.

Here, we compare against all results reported in the original HARES paper #OTHEREFR .

The difference between the LibriSpeechpretrained [79] Wav2Vec 2.0 #OTHEREFR performance (which is very low) and the other methods, which are trained on Au-dioSet #OTHEREFR , illustrates the importance of pretraining data to this particular task.

Our performance is competitive, but several AudioSet approaches with tuned architectures outperform our result, suggesting that our fully self-supervised pretraining and the relatively modest amount of pretraining data that we use may play a role. Arch.

Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs For Audiovisual Representation Learning

Abstract: 2 [chihweiw, iorife, mkalayeh]@netflix.com Figure 1. (Left) Audiovisual scenes can be perceptually similar even as the words spoken in them differ, which may be a challenge to self-supervised audiovisual representation learning. (Right) We propose to leverage movie dubs during training and show that it improves the quality of learned representations on a wide range of tasks.

A Multi-Device Dataset For Urban Acoustic Scene Classification

Abstract: This paper introduces the acoustic scene classification task of DCASE 2018 Challenge and the TUT Urban Acoustic Scenes 2018 dataset provided for the task, and evaluates the performance of a baseline system in the task. As in previous years of the challenge, the task is defined for classification of short audio samples into one of predefined acoustic scene classes, using a supervised, closed-set classification setup. The newly recorded TUT Urban Acoustic Scenes 2018 dataset consists of ten different acoustic scenes and was recorded in six large European cities, therefore it has a higher acoustic variability than the previous datasets used for this task, and in addition to high-quality binaural recordings, it also includes data recorded with mobile devices. We also present the baseline system consisting of a convolutional neural network and its performance in the subtasks using the recommended cross-validation setup.


Now it is easily verified that [0 N I N ]P −1 π in (24) is the same as the matrix in #REFR . V.

Therefore, this N -sum box is feasible according to #OTHEREFR .

For the specified G, H matrices, the transfer matrix M Q is,

CONCLUSION Using the N -sum box abstraction #OTHEREFR , this work translated existing CSA coding schemes for classical MAC settings into Quantum CSA coding schemes, for entanglementassisted QMACs.

This leads to immediate applications to Quantum PIR with secure and MDS-coded storage (Q-MDS-XSTPIR), as well as Quantum SDBMM.

In both cases, the rate achieved with the QCSA scheme can be expressed as R Q = min{1, 2R C }, where R C is the rate achieved by the CSA scheme in the corresponding classical setting.

Recent results in QPIR, QTPIR, QMDSTPIR can be recovered as special cases.

An important direction for future work is to explore settings where certain qudits are erased or lost, which would extend the scope of QCSA schemes to allow stragglers.

Quantum Cross Subspace Alignment Codes Via The N -Sum Box Abstraction

Abstract: Cross-subspace alignment (CSA) codes are used in various private information retrieval (PIR) schemes (e.g., with secure storage) and in secure distributed batch matrix multiplication (SDBMM). Using a recently developed N-sum box abstraction of a quantum multiple-access channel (QMAC), we translate CSA schemes over classical multiple-access channels into efficient quantum CSA schemes over a QMAC, achieving maximal superdense coding gain. Because of the N-sum box abstraction, the underlying problem of coding to exploit quantum entanglements for CSA schemes, becomes conceptually equivalent to that of designing a channel matrix for a MIMO MAC subject to given structural constraints imposed by the N-sum box abstraction, such that the resulting MIMO MAC is able to implement the functionality of a CSA scheme (encoding/decoding) over-the-air. Applications include Quantum PIR with secure and MDS-coded storage, as well as Quantum SDBMM.

The Capacity Of Private Information Retrieval From Coded Databases

Abstract: We consider the problem of private information retrieval (PIR) over a distributed storage system. The storage system consists of $N$ non-colluding databases, each storing a coded version of $M$ messages. In the PIR problem, the user wishes to retrieve one of the available messages without revealing the message identity to any individual database. We derive the information-theoretic capacity of this problem, which is defined as the maximum number of bits of the desired message that can be privately retrieved per one bit of downloaded information. We show that the PIR capacity in this case is $C=\left(1+\frac{K}{N}+\frac{K^2}{N^2}+\cdots+\frac{K^{M-1}}{N^{M-1}}\right)^{-1}=(1+R_c+R_c^2+\cdots+R_c^{M-1})^{-1}=\frac{1-R_c}{1-R_c^M}$, where $R_c$ is the rate of the $(N,K)$ code used. The capacity is a function of the code rate and the number of messages only regardless of the explicit structure of the storage code. The result implies a fundamental tradeoff between the optimal retrieval cost and the storage cost. The result generalizes the achievability and converse results for the classical PIR with replicating databases to the case of coded databases.


To the best of our knowledge, the current state-of-the-art in image-level-based WSSS methods use class activation maps (CAMs) #REFR to generate the pixel-level masks of an object from its image-level label.

Therefore, to reduce the time and resources required for generating pixel-wise masks, a wide range of research works focus on developing approaches that focus on weaker kinds of supervision.

This is where Weakly Supervised Semantic Segmentation (WSSS) can be highly beneficial.

WSSS approaches focus on generating the required masks with minimum supervision, such as image-level labels #OTHEREFR , bounding boxes #OTHEREFR , point annotations #OTHEREFR , and scribbles #OTHEREFR .

This work focuses on generating semantic segmentation masks for medical images using the most straightforward, and least-supervised, image-level labels.

We limit the scope of our work to just using image-level labels since they are the most inexpensive form of annotation.

The central idea of CAMs is to use any model trained with classification loss to generate activation maps that highlight the image regions responsible for the prediction decision.

This results mostly in a rough localization of the objects rather than precise pixel-wise masks.

The most popular CAM approaches focus on adding regularization loss to improve the quality of the CAM prediction #OTHEREFR or utilizing refinement methods that aim to enhance the CAM afterward #OTHEREFR .

For example, adversarial erasing #OTHEREFR erases the most discriminative part of the CAM to force the model to consider different parts of an object. #OTHEREFR

(Chang et al., 2020) use clustering to automatically sub-divide every class into sub-classes, implicitly generating distinctive classes for less discriminative parts of the CAM.

Boundarycam: A Boundary-Based Refinement Framework For Weakly Supervised Semantic Segmentation Of Medical Images

Abstract: Weakly Supervised Semantic Segmentation (WSSS) with only image-level supervision is a promising approach to deal with the need for Segmentation networks, especially for generating a large number of pixel-wise masks in a given dataset. However, most stateof-the-art image-level WSSS techniques lack an understanding of the geometric features embedded in the images since the network cannot derive any object boundary information from just image-level labels. We define a boundary here as the line separating an object and its background, or two different objects. To address this drawback, we propose our novel BoundaryCAM framework, which deploys state-of-the-art class activation maps combined with various post-processing techniques in order to achieve fine-grained higher-accuracy segmentation masks. To achieve this, we investigate a state-of-the-art unsupervised semantic segmentation network that can be used to construct a boundary map, which enables BoundaryCAM to predict object locations with sharper boundaries. By applying our method to WSSS predictions, we were able to achieve up to 10% improvements even to the benefit of the current state-of-the-art WSSS methods for medical imaging. The framework is open-source and accessible online 1 .

Learning Deep Features For Discriminative Localization

Abstract: In this work, we revisit the global average pooling layer proposed in [13], and shed light on how it explicitly enables the convolutional neural network (CNN) to have remarkable localization ability despite being trained on imagelevel labels. While this technique was previously proposed as a means for regularizing training, we find that it actually builds a generic localizable deep representation that exposes the implicit attention of CNNs on an image. Despite the apparent simplicity of global average pooling, we are able to achieve 37.1% top-5 error for object localization on ILSVRC 2014 without training on any bounding box annotation. We demonstrate in a variety of experiments that our network is able to localize the discriminative image regions despite just being trained for solving classification task1.


For the image part, we use a pre-trained ResNet101 #REFR , followed by the same layers.

Besides the simulation results reported in Figure 1 , we also conduct experiments on the MSCOCO-2014 dataset #OTHEREFR using more practical models. See Figure 2 for the results.

For the text part, we use a pre-trained RoBERTa model #OTHEREFR , followed by a 3-layer fully-connected network with batch norm between layers.

In both parts, the width of the fully-connected layers and the output dimension are 768.

We freeze the pre-trained parts of the model and only train the fully-connected parts.

We measure the quality of the learned representation using its zero-shot performance on the MSCOCO-2014 validation set.

Unlike common image classification datasets, images in the MSCOCO dataset usually have multiple labels, each corresponding to an object that appears in the image, and there are 80 categories in total.

We regard a prediction to be correct if it matches one label.

On The Importance Of Contrastive Loss In Multimodal Learning

Abstract: Recently, contrastive learning approaches (e.g., CLIP (Radford et al., 2021)) have received huge success in multimodal learning, where the model tries to minimize the distance between the representations of different views (e.g., image and its caption) of the same data point while keeping the representations of different data points away from each other. However, from a theoretical perspective, it is unclear how contrastive learning can learn the representations from different views efficiently, especially when the data is not isotropic. In this work, we analyze the training dynamics of a simple multimodal contrastive learning model and show that contrastive pairs are important for the model to efficiently balance the learned representations. In particular, we show that the positive pairs will drive the model to align the representations at the cost of increasing the condition number, while the negative pairs will reduce the condition number, keeping the learned representations balanced.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


SHOT #REFR showed comparable performance to SFHarmony when trained with a batchsize of 500 (85.27%), but was highly dependent on the modified source training.

Thus, robustness to the batchsize is vital if a SFDA method is to be used for harmonisation.

We use a single source model to allow fair comparison, trained with a batchsize of 50.

We wish to maximise performance across all sites: as harmonisation is normally framed as a joint domain adaptation problem #OTHEREFR , the average performance across all sites is reported.

The results can be seen in Table 1 , alongside the baseline methods.

It can be seen that SFHarmony outperforms the existing SFDA methods, especially when a small batchsize was used for training (86.22% for batchsize 5).

Interestingly, several of the SFDA approaches outperformed the adversarial approaches despite them having access to the source data, possibly due to the instability of such approaches.

The proposed D GM M loss is clearly able to align the features across sites using only the GMM summary statistics. This is demonstrated by Fig.

2 , which shows the source and target features for each site before and after DA.

Clearly the features overlap much more after DA, which both leads to the clear improvement in performance, and shows that the approach is achieving the harmonisation goals of the model having a shared feature embedding across sites.

We tried modelling the features with K ∈ {1, 2, 3} GMM components: visual inspection of the features suggested that at least 2 components would be beneficial.

Sfharmony: Source Free Domain Adaptation For Distributed Neuroimaging Analysis

Abstract: To represent the biological variability of clinical neuroimaging populations, it is vital to be able to combine data across scanners and studies. However, different MRI scanners produce images with different characteristics, resulting in a domain shift known as the 'harmonisation problem'. Additionally, neuroimaging data is inherently personal in nature, leading to data privacy concerns when sharing the data. To overcome these barriers, we propose an Unsupervised Source-Free Domain Adaptation (SFDA) method, SFHarmony. Through modelling the imaging features as a Gaussian Mixture Model and minimising an adapted Bhattacharyya distance between the source and target features, we can create a model that performs well for the target data whilst having a shared feature representation across the data domains, without needing access to the source data for adaptation or target labels. We demonstrate the performance of our method on simulated and real domain shifts, showing that the approach is applicable to classification, segmentation and regression tasks, requiring no changes to the algorithm. Our method outperforms existing SFDA approaches across a range of realistic data scenarios, demonstrating the potential utility of our approach for MRI harmonisation and general SFDA problems. Our code is available at https://github.com/ nkdinsdale/SFHarmony.

Do We Really Need To Access The Source Data? Source Hypothesis Transfer For Unsupervised Domain Adaptation

Abstract: Unsupervised domain adaptation (UDA) aims to leverage the knowledge learned from a labeled source dataset to solve similar tasks in a new unlabeled domain. Prior UDA methods typically require to access the source data when learning to adapt the model, making them risky and inefficient for decentralized private data. In this work we tackle a novel setting where only a trained source model is available and investigate how we can effectively utilize such a model without source data to solve UDA problems. To this end, we propose a simple yet generic representation learning framework, named \emph{Source HypOthesis Transfer} (SHOT). Specifically, SHOT freezes the classifier module (hypothesis) of the source model and learns the target-specific feature extraction module by exploiting both information maximization and self-supervised pseudo-labeling to implicitly align representations from the target domains to the source hypothesis. In this way, the learned target model can directly predict the labels of target data. We further investigate several techniques to refine the network architecture to parameterize the source model for better transfer performance. To verify its versatility, we evaluate SHOT in a variety of adaptation cases including closed-set, partial-set, and open-set domain adaptation. Experiments indicate that SHOT yields state-of-the-art results among multiple domain adaptation benchmarks.


We used the support vector machines implementation in the "scikit-learn" library for Python #REFR . For binary relevance, we employed the "one vs.

rest" classification approach using support vector machines as the base learner with the usual threshold, 0.5.

For both a linear and a radial kernel, we performed a grid search to determine a suitable values for the hyperparameters C and gamma (gamma applies only to the radial kernel) to minimize 0/1 loss. The x-variables are the TF-IDF variables.

The models were trained on the training data and evaluated on the validation data.

For binary relevance, the grid search determined a linear kernel with C = 100.

Multi-Label Classification Of Open-Ended Questions With Bert

Abstract: Open-ended questions in surveys are valuable because they do not constrain the respondent's answer, thereby avoiding biases. However, answers to open-ended questions are text data which are harder to analyze. Traditionally, answers were manually classified as specified in the coding manual. In the last 10 years, researchers have tried to automate coding. Most of the effort has gone into the easier problem of single label prediction, where answers are classified into a single code. However, open-ends that require multi-label classification, i.e., that are assigned multiple codes, occur frequently. In social science surveys, such open-ends are also frequently mildly multi-label. In mildly multi-label classifications, the average number of labels per answer text is relatively low (e.g. < 1.5). For example, the data set we analyze asks "What do you think is the most important political problem in Germany at the moment?" Even though the question asks for a single problem, some answers contain multiple problems. Of course, the average number of problems (or labels) per answer is still low. This paper focuses on multi-label classification of text answers to open-ended survey questions in social science surveys. We evaluate the performance of the transformer-based architecture BERT for the German language in comparison to traditional multi-label algorithms (Binary Relevance, Label Powerset, ECC) in a German social science survey, the GLES Panel (N=17,584, 55 labels). Because our data set requires at least one label per text answer, we also propose a modification in case the classification methods fail to predict any labels. We evaluate the algorithms on 0/1 loss: zero loss occurs only when all labels are predicted correctly; a mistake on one label incurs the full loss (1). This loss corresponds to the reality of manual text classification: you code the whole text answer with all labels, even if only a single suspicious label requires review. We find that classification with BERT (forcing at least one label) has the smallest 0/1 loss (13.1%) among methods considered (18.9%-21.6%). As expected, it is much easier to correctly predict answer texts that correspond to a single label (7.1% loss) than those that correspond to multiple labels (∼50% loss). Because BERT predicts zero labels for only 1.5% of the answers, forcing at least one label, while successful and recommended, ultimately does not lower the 0/1 loss by much. Our work has important implications for social scientists: 1) We have shown multi-label classification with BERT works in the German language for open-ends. 2) For mildly multi-label classification tasks, the loss now appears small enough to allow for fully automatic classification. Previously, the loss was more substantial, usually requiring semi-automatic approaches. 3) Multilabel classification with BERT requires only a single model. The leading competitor, ECC, is an iterative approach that iterates through individual single label predictions.

Scikit-Learn: Machine Learning In Python

Abstract: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.sourceforge.net.


In the report, they note several of the same observations regarding race and age. Similar findings were made in #REFR .

They do see a performance increase by fine-tuning a system on their child database.

These results are comparable with the results observed in this work, but this dataset has the benefit of being synthetic.

It was also observed how subjects of Black and Asian race in general perform worse than ones of White and Latino-Hispanic.

It was further seen that all races have a performance decrease as they get younger. In #OTHEREFR , Grother et al.

have performed a vendor test with a specific focus on the performance and bias of commercial face recognition systems concerning demographics.

Overall the results indicate that facial recognition systems are not robust to younger subjects and that racial and gender bias is a general problem across age groups.

In figure 15 , pairs of subjects from ages 7-10 with a high non-mated score can be seen.

As seen, the pairs of subjects has the same gender and race.

Similarly, pairs of subjects from the youngest age group (ages 1-4) with a high non-mated score can be seen in figure 16 .

In this particular age group, false matches across gender and race have also been observed.

Child Face Recognition At Scale: Synthetic Data Generation And Performance Benchmark

Abstract: We address the need for a large-scale database of children's faces by using generative adversarial networks (GANs) and face age progression (FAP) models to synthesize a realistic dataset referred to as HDA-SynChildFaces. To this end, we proposed a processing pipeline that initially utilizes StyleGAN3 to sample adult subjects, which are subsequently progressed to children of varying ages using InterFaceGAN. Intra-subject variations, such as facial expression and pose, are created by further manipulating the subjects in their latent space. Additionally, the presented pipeline allows to evenly distribute the races of subjects, allowing to generate a balanced and fair dataset with respect to race distribution. The created HDA-SynChildFaces consists of 1,652 subjects and a total of 188,832 images, each subject being present at various ages and with many different intra-subject variations. Subsequently, we evaluates the performance of various facial recognition systems on the generated database and compare the results of adults and children at different ages. The study reveals that children consistently perform worse than adults, on all tested systems, and the degradation in performance is proportional to age. Additionally, our study uncovers some biases in the recognition systems, with Asian and Black subjects and females performing worse than White and Latino Hispanic subjects and males.

Racial Faces In The Wild: Reducing Racial Bias By Information Maximization Adaptation Network

Abstract: Racial bias is an important issue in biometric, but has not been thoroughly studied in deep face recognition. In this paper, we first contribute a dedicated dataset called Racial Faces in-the-Wild (RFW) database, on which we firmly validated the racial bias of four commercial APIs and four state-of-the-art (SOTA) algorithms. Then, we further present the solution using deep unsupervised domain adaptation and propose a deep information maximization adaptation network (IMAN) to alleviate this bias by using Caucasian as source domain and other races as target domains. This unsupervised method simultaneously aligns global distribution to decrease race gap at domain-level, and learns the discriminative target representations at cluster level. A novel mutual information loss is proposed to further enhance the discriminative ability of network output without label information. Extensive experiments on RFW, GBU, and IJB-A databases show that IMAN successfully learns features that generalize well across different races and across different databases.


Approaches that exploit online-sampling of the uncertainties for the scenario approximation of SMPC problems are known as Scenario MPC #REFR .

By introducing chance constraints, i.e., requiring state or output constraints to be satisfied with a prespecified probability level, SMPC allows for a systematic trade-off between control performance and constraint satisfaction.

This is particularly important for MPC of uncertain systems when optimal performance requires operation near constraint boundaries in applications where rare or transient constraint violations are acceptable, such as in electric grids #OTHEREFR or finance #OTHEREFR .

For safety-critical applications, safety guarantees are enabled by employing failsafe or robust backup plans #OTHEREFR .

A major challenge in SMPC is to reformulate the chance constraint into a deterministic expression for tractability of the OCP.

Sampling-based approaches provide an appealing remedy, as they are easy to implement, independent of the underlying probability distribution, and also allow for nonlinearity of the uncertainties in the system dynamics (Lorenzen et al., 2017) .

While online-sampling comes with reduced sample complexity, offline-sampling allows for a reduced online computational load by removing redundant constraints offline, as well as for a guarantee of recursive feasibility by introducing a constraint on the first predicted step, as proposed by Lorenzen et al. (2017) for known system matrices subject to parametric uncertainties.

On the other hand, sampling-based approaches can only guarantee chance constraint satisfaction with confidence, and the computational load increases drastically with the dimension of the system #OTHEREFR .

In the data-driven setting, a reformulation of the chance constraint may be achieved by leveraging polynomial chaos expansion #OTHEREFR or employing stochastic tubes #OTHEREFR .

However, both works consider systems subject to additive stochastic disturbances, and assume the available (measured) data to be exact.

Offline Uncertainty Sampling In Data-Driven Stochastic Mpc

Abstract: In this work, we exploit an offline-sampling based strategy for the constrained data-driven predictive control of an unknown linear system subject to random measurement noise. The strategy uses only past measured, potentially noisy data in a non-parametric system representation and does not require any prior model identification. The approximation of chance constraints using uncertainty sampling leads to efficient constraint tightening. Under mild assumptions, robust recursive feasibility and closed-loop constraint satisfaction is shown. In a simulation example, we provide evidence for the improved control performance of the proposed control scheme in comparison to a purely robust data-driven predictive control approach.

The Scenario Approach For Stochastic Model Predictive Control With Bounds On Closed-Loop Constraint Violations

Abstract: Many practical applications in control require that constraints on the inputs and states of the system are respected, while some performance criterion is optimized. In the presence of model uncertainties or disturbances, it is often sufficient to satisfy the state constraints for at least a prescribed share of the time, such as in building climate control or load mitigation for wind turbines. For such systems, this paper presents a new method of Scenario-Based Model Predictive Control (SCMPC). The basic idea is to optimize the control inputs over a finite horizon, subject to robust constraint satisfaction under a finite number of random scenarios of the uncertainty and/or disturbances. Previous SCMPC approaches have suffered from a substantial gap between the rate of constraint violations specified in the optimal control problem and that actually observed in closed-loop operation of the controlled system. This paper identifies the two theoretical explanations for this gap. First, accounting for the special structure of the optimal control problem leads to a substantial reduction of the problem dimension. Second, the probabilistic constraints have to be interpreted as average-in-time, rather than pointwise-in-time. Based on these insights, a novel SCMPC method can be devised for general linear systems with additive and multiplicative disturbances, for which the number of scenarios is significantly reduced. The presented method retains the essential advantages of the general SCMPC approach, namely a low computational complexity and the ability to handle arbitrary probability distributions. Moreover, the computational complexity can be adjusted by a sample-and-remove strategy.


While our analysis and methods build directly on Shazeer and Stern #REFR (AdaFactor), there are important differences.

These instabilities may slow learning, or even destabilize training completely.

Various solutions have been proposed, including freezing the embedding layer #OTHEREFR , adding additional layer normalization #OTHEREFR , or reparametrizing the weights #OTHEREFR .

In our work we investigate instabilities which arise during CLIP training.

Unlike the instabilities observed in #OTHEREFR , we find these are not caused by attention entropy collapse.

Instead, our results indicate that spikes arise when the second moment estimator is out of date for the networks early layers.

In contrast with Shazeer and Stern #OTHEREFR , who only observe instabilities without warmup, we observe instabilities despite a long warmup period.

Moreover, in contrast with Shazeer and Stern #OTHEREFR we find that an out-of-date second moment estimator is primarily an issue for the (patch) embedding layer, and measure how well loss spikes are predicted by this event.

Finally, we note that reserachers have moved away from AdaFactor in its original formulation for large-scale training #OTHEREFR , finding AdaFactor to under-perform AdamW #OTHEREFR .

We believe this is due to the factored second moment or absence of first moment.

This is why our focus is AdamW #OTHEREFR which is the de facto standard optimizer for transformers.

Stable And Low-Precision Training For Large-Scale Vision-Language Models

Abstract: We introduce new methods for 1) accelerating and 2) stabilizing training for large language-vision models. 1) Towards accelerating training, we introduce SwitchBack , a linear layer for int8 quantized training which provides a speed-up of 13-25% while matching the performance of bfloat16 training within 0.1 percentage points for the 1B parameter CLIP ViT-Huge-the largest int8 training to date. Our main focus is int8 as GPU support for float8 is rare, though we also analyze float8 training through simulation. While SwitchBack proves effective for float8, we show that standard techniques are also successful if the network is trained and initialized so that large feature magnitudes are discouraged, which we accomplish via layer-scale initialized with zeros. 2) Towards stable training, we analyze loss spikes and find they consistently occur 1-8 iterations after the squared gradients become underestimated by their AdamW second moment estimator. As a result, we recommend an AdamW-Adafactor hybrid, which we refer to as StableAdamW because it avoids loss spikes when training a CLIP ViT-Huge model and outperforms gradient clipping.

Adafactor: Adaptive Learning Rates With Sublinear Memory Cost

Abstract: In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay rate of the second moment accumulator is too slow. We propose update clipping and a gradually increasing decay rate scheme as remedies. Combining these methods and dropping momentum, we achieve comparable results to the published Adam regime in training the Transformer model on the WMT 2014 English-German machine translation task, while using very little auxiliary storage in the optimizer. Finally, we propose scaling the parameter updates based on the scale of the parameters themselves.


In addition, we also compare the semi-supervised settings following #REFR upon the same network, and the results are reported in Table 6 .

Ground Truth Ours Full Fully-Supervised APB w/o. Knowledge Enhancement Table 6 .

Comparing the different training paradigms, i.e., supervised, semi-supervised and unsupervised settings.

Table 5 , the result indicates that using the operation in Eq. 3 works best. Semi-supervised setting.

Specifically, we conduct two semi-supervised training schemes: 1) Semi-supervised v1 refers to training the APB using 1 4 of randomly sampled labeled data on BDD-A and then training the entire network using pseudo-labels generated from the remaining raw images; 2) Semi-supervised v2 refers to the reversed process.

However, as is shown in Table 6 , we observe drastic drops in the result of the network in both Semi-supervised v1 and v2 compared with fully-supervised APB and are even inferior to our model trained in an unsupervised way.

The poor performance can be explained by only using a small portion of the dataset tend to fool the model into learning a more restricted central bias, especially in self-driving.

Our unsupervised method can leverage the information transferred from natural scenes by uncertainty mining, which is able to include more generalized information from non-traffic scenes to reduce bias.

Figure 5 shows visual comparisons of our model's variants on the BDD-A test set.

Unsupervised Self-Driving Attention Prediction Via Uncertainty Mining And Knowledge Embedding

Abstract: Predicting attention regions of interest is an important yet challenging task for self-driving systems. Existing methodologies rely on large-scale labeled traffic datasets that are labor-intensive to obtain. Besides, the huge domain gap between natural scenes and traffic scenes in current datasets also limits the potential for model training. To address these challenges, we are the first to introduce an unsupervised way to predict self-driving attention by uncertainty modeling and driving knowledge integration. Our approach's Uncertainty Mining Branch (UMB) discovers commonalities and differences from multiple generated pseudo-labels achieved from models pre-trained on natural scenes by actively measuring the uncertainty. Meanwhile, our Knowledge Embedding Block (KEB) bridges the domain gap by incorporating driving knowledge to adaptively refine the generated pseudo-labels. Quantitative and qualitative results with equivalent or even more impressive performance compared to fully-supervised state-of-the-art approaches across all three public datasets demonstrate the effectiveness of the proposed method and the potential of this direction. The code will be made publicly available.

Distilling The Knowledge In A Neural Network

Abstract: A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.


As a further research, we plan to find a theoretical methods of evaluating the expectations in #REFR and 7and evaluating the number of samples required to find these expectations reliably.

One can observe a gap of ≈ 3 dB between the achievability and converse bounds.

With a further K a increase, this gap becomes smaller, and when K a approaches n, projection-based achievability bound shows worse results.

To show this, we compare the ML-based and projection-based achievability for k = 100, n = 1000, L = 64

and P e = 10 −3 . This scenario has been taken from #OTHEREFR .

We also find that FASURA #OTHEREFR , a state-of-the-art practical scheme, demonstrates the energy efficiency very close to both ML-and projection-based bounds for K a ≤ 400.

Projection-based ML-based Single-user ach, #OTHEREFR Single-user converse, #OTHEREFR Multi-user converse, #OTHEREFR Fig. 2. Same-codebook achievability bound for no-CSI setting.

Frame length n = 1000, the number of information bits k = 100.

Base station is equipped with L = 64 antennas, Pe = 10 −3 .

Note that the channel (1) is permutation invariant, thus we can perform such type of decoding.DRAFTMarch 28, 2023

Note that the papers are devoted to a single antenna case.March 28, 2023 DRAFT

Unsourced Random Access With The Mimo Receiver: Projection Decoding Analysis

Abstract: We consider unsourced random access with MIMO receiver-a crucial communication scenario for future 5G/6G wireless networks. We perform a projection-based decoder analysis and derive energy efficiency achievability bounds when channel state information is unknown at transmitters and the receiver (no-CSI scenario). The comparison to the maximum-likelihood (ML) achievability bounds by Gao et al. (2023) is performed. We show that there is a region where the new bound outperforms the ML bound. The latter fact should not surprise the reader as both decoding criteria are suboptimal when considering per-user probability of error (PUPE). Moreover, transition to projection decoding allows for significant dimensionality reduction, which greatly reduces the computation time.

A Coupled Compressive Sensing Scheme For Unsourced Multiple Access

Abstract: This article introduces a novel paradigm for the unsourced multiple-access communication problem. This divide-and-conquer approach leverages recent advances in compressive sensing and forward error correction to produce a computationally efficient algorithm. Within the proposed framework, every active device first partitions its data into several sub-blocks, and subsequently adds redundancy using a systematic linear block code. Compressive sensing techniques are then employed to recover sub-blocks, and the original messages are obtained by connecting pieces together using a low-complexity tree-based algorithm. Numerical results suggest that the proposed scheme outperforms other existing practical coding schemes. Measured performance lies approximately $4.3$~dB away from the Polyanskiy achievability limit, which is obtained in the absence of complexity constraints.


A high number of devices attempting to connect to the network at the same time is a medium access control problem; standard random access (RA) systems are incapable of handling such a huge number of requests #REFR , and pure RA methods such as ALOHA have serious performance limitations.

In B5G, the base station (BS) needs to simultaneously support devices with a variety of capabilities and deployments as the number of active devices continues to increase as well as access requests for multiple services, e.g., vehicles, sensors, mobiles, etc., and applications in 5G and B5G, such as massive machine-type communication (mMTC) and crowded mobile broadband (cMBB) #OTHEREFR .

Hence, to support massive connections a promising candidate for the next generation of multiple access techniques, non-orthogonal multiple access (NOMA) has the ability to serve in the same resource element (RE), more than one device.

In comparison to standard orthogonal multiple access (OMA), the NOMA increases the system throughput, improves user fairness, reduces latency, and enables large connections #OTHEREFR , sharing the same orthogonal resource (time and frequency) by coding superposition in transmitter side and successive interference cancellation (SIC) in receiver side #OTHEREFR .

The adoption of NOMA and SIC considerably enhances performance, allowing two or more users per time slot #OTHEREFR , #OTHEREFR .

In (over-)crowded scenarios, the number of user connection attempts considerably outnumbers the number of available pilot sequences.

As a result, establishing collision resolution techniques became crucial for enabling efficient communication.

The strongest user collision resolution (SUCRe) protocol is a well-known decentralized grant-based random access (RA) protocol for crowded massive multiple-input multiple-output (MIMO) systems, which takes advantage of MIMO properties #OTHEREFR , giving preference to users with good channel conditions while harming edge users.

Hence, the SUCRe protocol has undergone several evolutions; e.g., in #OTHEREFR a graphbased pilot access (SUCR-GBPA) protocol is proposed, enabling all users who lost contention resolution to choose a new pilot at random.

Improving Random Access With Noma In Mmtc Xl-Mimo

Abstract: The extra-large multiple-input multiple-output (XL-MIMO) architecture has been recognized as a technology for giving support for the massive MTC (mMTC), providing very high-data rates in high-user density scenarios. However, the large dimension of the array increases the Rayleigh distance (d Rayl), in addition to obstacles and scatters causing spatial non-stationarities and distinct visibility regions (VRs) across the XL array extension. We investigate the random access (RA) problem in crowded XL-MIMO scenarios; the proposed grant-based random access (GB-RA) protocol combining the advantage of non-orthogonal multiple access (NOMA) and strongest user collision resolutions in extra-large arrays (SUCRe-XL) named NOMA-XL can allow access of two or three colliding users in the same XL subarray (SA) selecting the same pilot sequence. The received signal processing in a SA basis changes the d Rayl , enabling the far-field planar wavefront propagation condition, while improving the system performance. The proposed NOMA-XL GB-RA protocol is able to provide a reduction in the number of attempts to access the mMTC network while improving the average sum rate, as the number of SA increases.

From 5G To 6G: Has The Time For Modern Random Access Come?

Abstract: This short paper proposes the use of modern random access for IoT applications in 6G. A short overview of recent advances in uncoordinated medium access is provided, highlighting the gains that can be achieved by leveraging smart protocol design intertwined with advanced signal processing techniques at the receiver. The authors’ vision on the benefits such schemes can yield for beyond-5G systems is presented, with the aim to trigger further discussion.


We also utilize the Specaugment #REFR , which involves applying spectrogram masking with a maximum time mask length of 192 frames and a maximum frequency mask length of 48 bins.

Our audio experiments include various augmentation techniques including Mixup #OTHEREFR and Cutmix #OTHEREFR , which are com- Figure 3 : Audio-specific Cutmix which contains only a temporal axis cut.

monly used in audio data augmentation #OTHEREFR .

Additionally, we introduce an audio-specific data augmentation method called audio Cutmix as shown in Figure 3 .

Audio cutmix is similar to image Cutmix #OTHEREFR , but has an temporal axis cut because the frequency axis contains quantitative information that is not invariant as in images.

Throughout the entire experiment training process, including the process for training the baseline models, half of the audio clips are augmented using Mixup and the other half are augmented using our audio Cutmix technique.

Another augmentation technique we use is random rolling, which randomly shifts each audio clip in time by a certain number of samples.

This random shift causes the entire waveform to appear as if it was played earlier or later in time, which prevents the model from overfitting to the training data.

We adopted the identical data augmentation pipeline as MViT #OTHEREFR for our image classification experiment, which encompasses Mixup #OTHEREFR , Cutmix #OTHEREFR , Random erasing #OTHEREFR , and Randaugment #OTHEREFR .

Mmvit: Multiscale Multiview Vision Transformers

Abstract: We present Multiscale Multiview Vision Transformers (MMViT), which introduces multiscale feature maps and multiview encodings to transformer models. Our model encodes different views of the input signal and builds several channelresolution feature stages to process the multiple views of the input at different resolutions in parallel. At each scale stage, we use a cross-attention block to fuse information across different views. This enables the MMViT model to acquire complex high-dimensional representations of the input at different resolutions. The proposed model can serve as a backbone model in multiple domains. We demonstrate the effectiveness of MMViT on audio and image classification tasks, achieving state-of-theart results.

Specaugment: A Simple Data Augmentation Method For Automatic Speech Recognition

Abstract: We present SpecAugment, a simple data augmentation method for speech recognition. SpecAugment is applied directly to the feature inputs of a neural network (i.e., filter bank coefficients). The augmentation policy consists of warping the features, masking blocks of frequency channels, and masking blocks of time steps. We apply SpecAugment on Listen, Attend and Spell networks for end-to-end speech recognition tasks. We achieve state-of-the-art performance on the LibriSpeech 960h and Swichboard 300h tasks, outperforming all prior work. On LibriSpeech, we achieve 6.8% WER on test-other without the use of a language model, and 5.8% WER with shallow fusion with a language model. This compares to the previous state-of-the-art hybrid system of 7.5% WER. For Switchboard, we achieve 7.2%/14.6% on the Switchboard/CallHome portion of the Hub5'00 test set without the use of a language model, and 6.8%/14.1% with shallow fusion, which compares to the previous state-of-the-art hybrid system at 8.3%/17.3% WER.


We use best-worst-scaling (BWS) for evaluating the informativeness of the generated summaries, as this method is "a less labor-intensive alternative to paired comparisons that has been shown to produce more reliable results than rating scales" #REFR .

the percentage of summaries rated as 3-star) as the faithfulness score, and also report the distribution of summaries rated as 1, 2, and 3 stars.

Details on qualification, payment and other aspects of the evaluation can be found in Appendix A.4.

We also evaluate the generated summaries in terms of informativeness.

We consider summary to be informative if its content is important and relevant, but it does not necessarily need to be long.

Accordingly, for each dataset, we select 200 random articles with the corresponding summaries from five systems in random order.

We ask three annotators to select the most informative ("best") and the least informative ("worst") among the five.

A rating per system is computed as the percentage of times it is chosen as best minus the percentage of times it is selected as worst.

A value of 100 means that the system has been unanimously picked as "best", whereas a value of -100 means that the system has been unanimously picked as "worst".

Additional details, as well as the screenshot of the annotation interface, are in Appendix A.4.

Faithfulness-Aware Decoding Strategies For Abstractive Summarization

Abstract: Despite significant progress in understanding and improving faithfulness in abstractive summarization, the question of how decoding strategies affect faithfulness is less studied. We present a systematic study of the effect of generation techniques such as beam search and nucleus sampling on faithfulness in abstractive summarization. We find a consistent trend where beam search with large beam sizes produces the most faithful summaries while nucleus sampling generates the least faithful ones. We propose two faithfulness-aware generation methods to further improve faithfulness over current generation techniques: (1) ranking candidates generated by beam search using automatic faithfulness metrics and (2) incorporating lookahead heuristics that produce a faithfulness score on the future summary. We show that both generation methods significantly improve faithfulness across two datasets as evaluated by four automatic faithfulness metrics and human evaluation. To reduce computational cost, we demonstrate a simple distillation approach that allows the model to generate faithful summaries with just greedy decoding. 1 * * Work conducted during an internship at Amazon.

Best-Worst Scaling More Reliable Than Rating Scales: A Case Study On Sentiment Intensity Annotation

Abstract: Rating scales are a widely used method for data annotation; however, they present several challenges, such as difficulty in maintaining inter- and intra-annotator consistency. Best-worst scaling (BWS) is an alternative method of annotation that is claimed to produce high-quality annotations while keeping the required number of annotations similar to that of rating scales. However, the veracity of this claim has never been systematically established. Here for the first time, we set up an experiment that directly compares the rating scale method with BWS. We show that with the same total number of annotations, BWS produces significantly more reliable results than the rating scale.


The multi-modal fusion module leverages attention mechanism #REFR to fuse proposal features F p , gestural features F g , and word features F l , thereby generating the confidence score of each bounding-box.

To better leverage the features among language, gesture, and the scene point cloud, an attention mechanism #OTHEREFR is employed in our work.

The proposal generation module is the same as that used in 3DVG-Transformer #OTHEREFR and it generates a bounding-box from object proposal while extracting context-aware features.

The proposal features are represented by F p ∈ R M ×H , for M proposals with H-dimensional features.

The gestural encoding module uses a PointNet++ #OTHEREFR to extract the features F g ∈ R M ×H of the human agent's point cloud.

Similar to ScanRefer #OTHEREFR and 3DVG-Transformer #OTHEREFR , the language encoding module aggregates the word embeddings into the language features F l ∈ R L×H and global language features using a GRU #OTHEREFR cell and a self-attention module.

Specifically, this study centers on the combination of gestural information with the proposal and word information, aiming to disambiguate referring expressions and accurately identify the referred object.

Scaneru: Interactive 3D Visual Grounding Based On Embodied Reference Understanding

Abstract: Aiming to link natural language descriptions to specific regions in a 3D scene represented as 3D point clouds, 3D visual grounding is a very fundamental task for human-robot interaction. The recognition errors can significantly impact the overall accuracy and then degrade the operation of AI systems. Despite their effectiveness, existing methods suffer from the difficulty of low recognition accuracy in cases of multiple adjacent objects with similar appearances. To address this issue, this work intuitively introduces the humanrobot interaction as a cue to facilitate the development of 3D visual grounding. Specifically, a new task termed Embodied Reference Understanding (ERU) is first designed for this concern. Then a new dataset called ScanERU is constructed to evaluate the effectiveness of this idea. Different from existing datasets, our ScanERU is the first to cover semi-synthetic scene integration with textual, real-world visual, and synthetic gestural information. Additionally, this paper formulates a heuristic framework based on attention mechanisms and human body movements to enlighten the research of ERU. Experimental results demonstrate the superiority of the proposed method, especially in the recognition of multiple identical objects. Our codes and dataset 1 are ready to be available publicly.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

proposal features
multi-modal fusion module


Especially, contrastive learning method has shown great advances in image classification with massive unlabeled data. It aims to use similarities and differences between images. SimCLR #REFR is a prominent work for this approach.

Up until now, however, its success is mainly for upstream tasks, not for the downstream.

It requires yet another labeled data to finetune the pre-trained models.

In other words, it still depends on supervised learning that requires much manual effort.

NLP downstream tasks and their performances are limited by disadvantages of supervised learning.

In image processing, on the other hand, numerous approaches to utilize SSL have been fruitful.

As its pre-trained model applies to downstream task of classification, it outperforms even supervised learning models.

But, it also need two stage process of upstream and downstream task.

To take one step further, some works make downstream task also self-supervised, to minimize human intervention.

They include SCAN #OTHEREFR , RUC #OTHEREFR , SelfMatch #OTHEREFR , yielding tangible results.

There is not much work to extend SSL or unsupervised learning method to downstream task in natural language processing field. A few has tried contrastive learning in the field.

Shuffle & Divide: Contrastive Learning For Long Text

Abstract: We propose a self-supervised learning method for long text documents based on contrastive learning. A key to our method is Shuffle and Divide (SaD), a simple text augmentation algorithm that sets up a pretext task required for contrastive updates to BERT-based document embedding. SaD splits a document into two sub-documents containing randomly shuffled words in the entire documents. The sub-documents are considered positive examples, leaving all other documents in the corpus as negatives. After SaD, we repeat the contrastive update and clustering phases until convergence. It is naturally a time-consuming, cumbersome task to label text documents, and our method can help alleviate human efforts, which are most expensive resources in AI. We have empirically evaluated our method by performing unsupervised text classification on the 20 Newsgroups, Reuters-21578, BBC, and BBCSport datasets. In particular, our method pushes the current state-of-the-art, SS-SB-MT, on 20 Newsgroups by 20.94% in accuracy. We also achieve the state-of-the-art performance on Reuters-21578 and exceptionally-high accuracy performances (over 95%) for unsupervised classification on the BBC and BBCSport datasets.

A Simple Framework For Contrastive Learning Of Visual Representations

Abstract: This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.


Compared with the earlier information-theoretic results #REFR and 3, the bound in Corollary 2.4 is both more general and stronger.

I(X k 1 ; Y ) = H(X k 1 ) − H(X k 1 |Y ) ≤ H(X k 1 ) ≤ k log |A|,

where H(X) = − x∈B P (x) log P (x) denotes the entropy of a random variable with probability mass function P on a discrete alphabet B.

Therefore, Theorem 2.1 immediately yields: is an exchangeable vector of random variables X i taking values in a discrete alphabet A.

For every 1 ≤ k ≤ n − 1 there exists a probability measure µ = µ k,n on M 1 (A), such that:

D P X k 1 M k,µ ≤ k(k − 1) 2(n − k + 1) H(X 1 ) ≤ k(k − 1) 2(n − k + 1) log |A|.

Moreover, it can be used to recover the classical infinite version of de Finetti's theorem for compact spaces, under some conditions.

Corollary 2.5 (Classical de Finetti theorem for compact spaces) Let G be a compact metrisable space equipped with its Baire σ-algebra G.

Suppose the process {X k ; k ≥ 1} is exchangeable and the X k take values in G and are G-measurable. If for every k we have I(X k−1

A Third Information-Theoretic Approach To Finite De Finetti Theorems

Abstract: A new finite form of de Finetti's representation theorem is established using elementary information-theoretic tools. The distribution of the first k random variables in an exchangeable vector of n ≥ k random variables is close to a mixture of product distributions. Closeness is measured in terms of the relative entropy and an explicit bound is provided. This bound is tighter than those obtained via earlier information-theoretic proofs, and its utility extends to random variables taking values in general spaces. The core argument employed has its origins in the quantum information-theoretic literature.

Am With Multiple Merlins

Abstract: We introduce and study a new model of interactive proofs: AM(k), or Arthur-Merlin with k non-communicating Merlins. Unlike with the better-known MIP, here the assumption is that each Merlin receives an independent random challenge from Arthur. One motivation for this model (which we explore in detail) comes from the close analogies between it and the quantum complexity class QMA(k), but the AM(k) model is also natural in its own right. We illustrate the power of multiple Merlins by giving an AM(2) protocol for 3SAT, in which the Merlins' challenges and responses consist of only n^{1/2+o(1)} bits each. Our protocol has the consequence that, assuming the Exponential Time Hypothesis (ETH), any algorithm for approximating a dense CSP with a polynomial-size alphabet must take n^{(log n)^{1-o(1)}} time. Algorithms nearly matching this lower bound are known, but their running times had never been previously explained. Brandao and Harrow have also recently used our 3SAT protocol to show quasipolynomial hardness for approximating the values of certain entangled games. In the other direction, we give a simple quasipolynomial-time approximation algorithm for free games, and use it to prove that, assuming the ETH, our 3SAT protocol is essentially optimal. More generally, we show that multiple Merlins never provide more than a polynomial advantage over one: that is, AM(k)=AM for all k=poly(n). The key to this result is a subsampling theorem for free games, which follows from powerful results by Alon et al. and Barak et al. on subsampling dense CSPs, and which says that the value of any free game can be closely approximated by the value of a logarithmic-sized random subgame.


Revisiting Distillation For Catastrophic Forgetting

It, however, is observed in #REFR that there is tendency of classifying test samples to new classes by LwF.

where C = C o ∪ C n , y i is the i th value of the one-hot ground truth y, and p i is the i th value of predicted class probability p.

The goal of L dis is to preserve knowledge obtained from previous data, which is expressed as

where p * is the soft label of x generated by the old model.

Thus, iCaRL utilized herd selection to better approximate the class mean vector of old classes, where samples that are close to the center of old classes are selected.

Recall that our goal is to obtain a robust model trained in the continual learning manner.

To gain robustness, adversarial training is inevitable, which requires augmenting datasets with adversarial examples in every training iteration.

Following the definition of continual learning, we can derive the loss function of Robust Continual Learning (RCL).

With adversarial training, we should replace the input x in Equation 1and 2with its adversarial counterpart x adv , which is solved by

Towards Adversarially Robust Continual Learning

Abstract: Recent studies show that models trained by continual learning can achieve the comparable performances as the standard supervised learning and the learning flexibility of continual learning models enables their wide applications in the real world. Deep learning models, however, are shown to be vulnerable to adversarial attacks. Though there are many studies on the model robustness in the context of standard supervised learning, protecting continual learning from adversarial attacks has not yet been investigated. To fill in this research gap, we are the first to study adversarial robustness in continual learning and propose a novel method called Task-Aware Boundary Augmentation (TABA) to boost the robustness of continual learning models. With extensive experiments on CIFAR-10 and CIFAR-100, we show the efficacy of adversarial training and TABA in defending adversarial attacks.

Icarl: Incremental Classifier And Representation Learning

Abstract: A major open problem on the road to artificial intelligence is the development of incrementally learning systems that learn about more and more concepts over time from a stream of data. In this work, we introduce a new training strategy, iCaRL, that allows learning in such a class-incremental way: only the training data for a small number of classes has to be present at the same time and new classes can be added progressively. iCaRL learns strong classifiers and a data representation simultaneously. This distinguishes it from earlier works that were fundamentally limited to fixed data representations and therefore incompatible with deep learning architectures. We show by experiments on CIFAR-100 and ImageNet ILSVRC 2012 data that iCaRL can learn many classes incrementally over a long period of time where other strategies quickly fail.


In the future, we first aim to focus on embedding the graph information #REFR in the optimization process of the proposed approach.

In conclusion, the high-dimensional nature and imbalanced classes of hyperspectral images pose challenges for traditional machine learning algorithms. One-class classifiers are useful in cases

where the training data is from a single class only, but they still face challenges in handling the curse of dimensionality.

To address these challenges, we leverage S-SVDD for one-class classification of hyperspectral images.

Our experiments on two benchmark HSI datasets show that the proposed approach can effectively tackle the curse of dimensionality and the imbalanced nature of HSI data.

We also consider utilizing neighbouring pixels' spectral features for a future study, since they are usually correlated and give better scattering information, allowing enhanced performance.

Hyperspectral Image Analysis With Subspace Learning-Based One-Class Classification

Abstract: Hyperspectral image (HSI) classification is an important task in many applications, such as environmental monitoring, medical imaging, and land use/land cover (LULC) classification. Due to the significant amount of spectral information from recent HSI sensors, analyzing the acquired images is challenging using traditional Machine Learning (ML) methods. As the number of frequency bands increases, the required number of training samples increases exponentially to achieve a reasonable classification accuracy, also known as the curse of dimensionality. Therefore, separate band selection or dimensionality reduction techniques are often applied before performing any classification task over HSI data. In this study, we investigate recently proposed subspace learning methods for one-class classification (OCC). These methods map high-dimensional data to a lower-dimensional feature space that is optimized for one-class classification. In this way, there is no separate dimensionality reduction or feature selection procedure needed in the proposed classification framework. Moreover, one-class classifiers have the ability to learn a data description from the category of a single class only. Considering the imbalanced labels of the LULC classification problem and rich spectral information (high number of dimensions), the proposed classification approach is well-suited for HSI data. Overall, this is a pioneer study focusing on subspace learning-based one-class classification for HSI data. We analyze the performance of the proposed subspace learning one-class classifiers in the proposed pipeline. Our experiments validate that the proposed approach helps tackle the curse of dimensionality along with the imbalanced nature of HSI data.

Ellipsoidal Subspace Support Vector Data Description

Abstract: In this paper, we propose a novel method for transforming data into a low-dimensional space optimized for one-class classification. The proposed method iteratively transforms data into a new subspace optimized for ellipsoidal encapsulation of target class data. We provide both linear and non-linear formulations for the proposed method. The method takes into account the covariance of the data in the subspace; hence, it yields a more generalized solution as compared to Subspace Support Vector Data Description for a hypersphere. We propose different regularization terms expressing the class variance in the projected space. We compare the results with classic and recently proposed one-class classification methods and achieve better results in the majority of cases. The proposed method is also noticed to converge much faster than recently proposed Subspace Support Vector Data Description.


OpenAI's latest conversational agent, Chat-GPT 1 (OpenAI, 2022), a successor of Instruct-GPT (Ouyang et al., 2022) -also known as GPT-3.5models, is an instruction-following transformerbased language model, which has been further trained (aligned) with reinforcement learning from human feedback (RLHF) #REFR .

Recent advances in Large Language Models (LLMs) #OTHEREFR Chowdhery et al., 2022) , also known as Foundation Models (Bommasani et al., 2021) , have challenged the traditional supervised learning paradigm of fine-tuning by demonstrating emergent zero-shot Natural Language Understanding (NLU) capabilities #OTHEREFR ) through scaling the model's size in billions of parameters #OTHEREFR .

ChatGPT demonstrates unprecedented emergent capabilities in zero-shot Question-Answering (QA) 1 https://chat.openai.com/chat capabilities that cover common sense knowledge, but also extend to specialized domains such as problem solving, programming/debugging, and law, as presented by many users in the web.

Recently, Bommarito and Katz (2022) audited several variants of OpenAI's GPT 2/3/3.5 models in legal bar exam questions, and found that the most advanced -at the time-model ('text-davinci-003') achieves an accuracy of 50.3% on a complete practice exam, significantly in excess of the 25% baseline guessing rate, while it performs at a passing rate in two legal areas (Evidence and Torts). In a follow-up work, #OTHEREFR

(2023) assessed the model's performance in accounting certification exams, where the model significantly under-performs human capabilities with a correct rate of 14.4%.

Following the work of Bommarito and Katz, we evaluate the latest OpenAI's GPT-3.5 model (Ouyang et al., 2022 ) ('gpt-3.5-turbo', v.

March 2023 , the first available ChatGPT, on legal text classification tasks from the LexGLUE #OTHEREFR benchmark in a zeroshot fashion providing examples in a templated instruction-following format, similar to those used by Chung et al. (2022) .

Chatgpt May Pass The Bar Exam Soon, But Has A Long Way To Go For The Lexglue Benchmark

Abstract: Following the hype around OpenAI's Chat-GPT conversational agent, the last straw in the recent development of Large Language Models (LLMs) that demonstrate emergent unprecedented zero-shot capabilities, we audit the latest OpenAI's GPT-3.5 model, 'gpt-3.5-turbo', the first available ChatGPT model, in the LexGLUE benchmark in a zeroshot fashion providing examples in a templated instruction-following format. The results indicate that ChatGPT achieves an average micro-F1 score of 49.0% across LexGLUE tasks, surpassing the baseline guessing rates. Notably, the model performs exceptionally well in some datasets, achieving micro-F1 scores of 62.8% and 70.1% in the ECtHR B and LEDGAR datasets, respectively. The code base and model predictions are available for review on https://github.com/coastalcph/ zeroshot_lexglue.

Deep Reinforcement Learning From Human Preferences

Abstract: For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.

OpenAI's latest conversational
reinforcement learning


Robustness under distribution shifts Generalization capabilities of existing image classifiers have been a crucial problem #REFR , currently addressed from a few different viewpoints.

In general, transformer-based models rely on an abundance of training data to ensure proper generalization.

This requirement was relaxed in DeiT #OTHEREFR , enabling learning on medium-sized datasets.

Further development introduced novel transformer-based architectures, such as BeiT #OTHEREFR , Swin #OTHEREFR and RegNets #OTHEREFR , which realize specific refinements to boost performance.

Overall, it has been proven that ViTs are more robust compared to classic CNN image classifiers #OTHEREFR .

In our work, we verify the degree this claim holds by testing CNN and transformer-based classifiers on the uncurated fine-grained setting.

Artificial corruptions #OTHEREFR or natural shifts #OTHEREFR on curated data have already exposed biases and architectural vulnerabilities.

Adversarial robustness #OTHEREFR ] is a related field where models are tested against adversarial examples, which introduce imperceptible though influential perturbations on images.

Contrary to such attempts, we concentrated around naturally occurring distribution shifts stemming from uncurated image data.

Regarding architectural choices, many studies perform robustness tests attempting to resolve the CNN vs Transformer contest #OTHEREFR , while other ventures focus on interpreting and understanding model robustness #OTHEREFR .

In our approach, by experimenting with both CNN and transformer-based architectures we adopt such research attempts to the uncurated setting.

Fine-Grained Imagenet Classification In The Wild

Abstract: Image classification has been one of the most popular tasks in Deep Learning, seeing an abundance of impressive implementations each year. However, there is a lot of criticism tied to promoting complex architectures that continuously push performance metrics higher and higher. Robustness tests can uncover several vulnerabilities and biases which go unnoticed during the typical model evaluation stage. So far, model robustness under distribution shifts has mainly been examined within carefully curated datasets. Nevertheless, such approaches do not test the real response of classifiers in the wild, e.g. when uncurated web-crawled image data of corresponding classes are provided. In our work, we perform fine-grained classification on closely related categories, which are identified with the help of hierarchical knowledge. Extensive experimentation on a variety of convolutional and transformer-based architectures reveals model robustness in this novel setting. Finally, hierarchical knowledge is again employed to evaluate and explain misclassifications, providing an information-rich evaluation scheme adaptable to any classifier.

Do Imagenet Classifiers Generalize To Imagenet?

Abstract: We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.


MOT17 #REFR is a multiple object tracking dataset that contains 7 training sequences and 7 test sequences.

Something-Something V2 (SSv2) #OTHEREFR is another largescale action recognition dataset which focus more on temporal modeling.

The labels are like "Pulling something from left to right", so it is crucial to learn motion information.

The training set contains 168.9K training videos and the validation set contains 24.7K validation videos.

We use segment-based sampling from #OTHEREFR to sample 32 frames with 224 × 224 resolution.

The augmentation and regularization in SSv2 include random augmentation #OTHEREFR , repeated augmentation #OTHEREFR , random erasing #OTHEREFR , Mixup #OTHEREFR , and CutMix #OTHEREFR , which follow the practice in MViT #OTHEREFR .

The total frame number is only 11k, so it is not enough to train our S-ViT model.

We use the CrowdHuman #OTHEREFR dataset and the MOTSynth #OTHEREFR dataset to expand the training data.

CrowdHuman contains 19.4k images in crowd human scenarios, and MOTSynth contains 764 synthetic video sequences with 1.3m frames generated from Grand Theft Auto V.

We conduct our experiments with combinations of different data sources and discuss the influence in Sec. #OTHEREFR

Streaming Video Model

Abstract: Figure 1. Illustration of the proposed streaming video model with a comparison to conventional frame-based architecture and clip-based architecture. (a) The two-stage streaming video model gracefully serves different types of video tasks through a unified architecture. The output of the temporal-aware (T-aware) spatial encoder serves the frame-based tasks, such as MOT, while the output of the temporal decoder serves the sequence-based tasks, such as action recognition. (b) Frame-based architecture, which uses single image model to independently extract spatial features for each frame, is widely used in the frame-based video tasks. (c) Clip-based architecture, which uses video model to produce the spatiotemporal features for an entire clip, is widely used in the sequence-based video tasks.

Mot16: A Benchmark For Multi-Object Tracking

Abstract: Standardized benchmarks are crucial for the majority of computer vision applications. Although leaderboards and ranking tables should not be over-claimed, benchmarks often provide the most objective measure of performance and are therefore important guides for reseach. ::: Recently, a new benchmark for Multiple Object Tracking, MOTChallenge, was launched with the goal of collecting existing and new data and creating a framework for the standardized evaluation of multiple object tracking methods. The first release of the benchmark focuses on multiple people tracking, since pedestrians are by far the most studied object in the tracking community. This paper accompanies a new release of the MOTChallenge benchmark. Unlike the initial release, all videos of MOT16 have been carefully annotated following a consistent protocol. Moreover, it not only offers a significant increase in the number of labeled boxes, but also provides multiple object classes beside pedestrians and the level of visibility for every single object of interest.


Motivation: Approximate Sampling-And-Query Access

Consider for instance the inner product estimation from #REFR , a technique also used in several other "quantum-inspired" algorithms, which dequantizes the SWAP test (a basic quantum algorithm that can be used to estimate the inner product between two pure states).

, n} we can query the entry u(i)), for the sampling operation we can only get one sample from a distributionp u : {1, . . .

More precisely, the only guarantee is that the total variation distance betweenp u and p u is at most ε.

(Note that standard sampling-and-query access corresponds to the case ε = 0.) This variant corresponds to the setting where query access to u can be easily implemented (e.g., when the input is stored in RAM) but perfect sampling access is not available or simply too costly to be implemented.

It happens that this seemingly minor change is problematic for known dequantization techniques.

Given two unit-norm vectors u, v ∈ R n , where u is given via sampling-and-query access, the approach typically works as follows: sample an index i ∈ {1, . . .

, n} according to the distribution p u and output the value v (i) u (i) . Note that the expectation of this value is

It is easy to show that the variance of this estimator is small, and thus taking the mean for a few samples gives a good approximation of the inner product (u, v) .

Note that this test, however, is not "robust" due to the division by u(i).

Robust Dequantization Of The Quantum Singular Value Transformation And Quantum Machine Learning Algorithms

Abstract: Several quantum algorithms for linear algebra problems, and in particular quantum machine learning problems, have been "dequantized" in the past few years. These dequantization results typically hold when classical algorithms can access the data via lengthsquared sampling. This assumption, which is standard in the field of randomized linear algebra, means that for a unit-norm vector u ∈ C n , we can sample from the distribution p u : {1,. .. , n} → [0, 1] defined as p u (i) = |u(i)| 2 for each i ∈ {1,. .. , n}. Since this distribution corresponds to the distribution obtained by measuring the quantum state |u in the computational basis, length-squared sampling access gives a reasonable classical analogue to the kind of quantum access considered in many quantum algorithms for linear algebra problems. In this work we investigate how robust these dequantization results are. We introduce the notion of approximate length-squared sampling, where classical algorithms are only able to sample from a distribution close to the ideal distribution in total variation distance. While quantum algorithms are natively robust against small perturbations, current techniques in dequantization are not. Our main technical contribution is showing how many techniques from randomized linear algebra can be adapted to work under this weaker assumption as well. We then use these techniques to show that the recent low-rank dequantization framework by Chia, Gilyén, Li, Lin, Tang and Wang (JACM 2022) and the dequantization framework for sparse matrices by Gharibian and Le Gall (STOC 2022), which are both based on the Quantum Singular Value Transformation, can be generalized to the case of approximate length-squared sampling access to the input. We also apply these results to obtain a robust dequantization of many quantum machine learning algorithms, including quantum algorithms for recommendation systems, supervised clustering and low-rank matrix inversion.

A Quantum-Inspired Classical Algorithm For Recommendation Systems

Abstract: We give a classical analogue to Kerenidis and Prakash’s quantum recommendation system, previously believed to be one of the strongest candidates for provably exponential speedups in quantum machine learning. Our main result is an algorithm that, given an m × n matrix in a data structure supporting certain l2-norm sampling operations, outputs an l2-norm sample from a rank-k approximation of that matrix in time O(poly(k)log(mn)), only polynomially slower than the quantum algorithm. As a consequence, Kerenidis and Prakash’s algorithm does not in fact give an exponential speedup over classical algorithms. Further, under strong input assumptions, the classical recommendation system resulting from our algorithm produces recommendations exponentially faster than previous classical systems, which run in time linear in m and n. The main insight of this work is the use of simple routines to manipulate l2-norm sampling distributions, which play the role of quantum superpositions in the classical setting. This correspondence indicates a potentially fruitful framework for formally comparing quantum machine learning algorithms to classical machine learning algorithms.


Cityscapes #REFR is a street-view dataset with high quality annotations, containing 2975 training images and 500 validation images, with a total of 19 object classes.

We evaluate on semantic, instance, and panoptic segmentation using 3 datasets: ADE-20K #OTHEREFR is a semantic segmentation dataset containing 150 categories across 20K training images and 2K validation images.

COCO 2017 #OTHEREFR is an instance segmentation dataset, containing 118K training and 5K validation images.

Autofocusformer: Image Segmentation Off The Grid

Abstract: Real world images often have highly imbalanced content density. Some areas are very uniform, e.g., large patches of blue sky, while other areas are scattered with many small objects. Yet, the commonly used successive grid downsampling strategy in convolutional deep networks treats all areas equally. Hence, small objects are represented in very few spatial locations, leading to worse results in tasks such as segmentation. Intuitively, retaining more pixels representing small objects during downsampling helps to preserve important information. To achieve this, we propose AutoFocusFormer (AFF), a local-attention transformer image recognition backbone, which performs adaptive downsampling by learning to retain the most important pixels for the task. Since adaptive downsampling generates a set of pixels irregularly distributed on the image plane, we abandon the classic grid structure. Instead, we develop a novel point-based local attention block, facilitated by a balanced clustering module and a learnable neighborhood merging module, which yields representations for our point-based versions of state-of-the-art segmentation heads. Experiments show that our AutoFocusFormer (AFF) improves significantly over baseline models of similar sizes. * Work done while Chen Ziwen was an intern at Apple Inc. Image Remaining tokens Stage 2 Prediction AFF Swin Remaining tokens Stage 4 Comparison between on-grid model Swin and off-grid model AFF. AFF downsamples non automatically focusing on more textured, important image regions, and successfully captu the background. The red pixels indicate the locations of the remaining tokens.

The Cityscapes Dataset For Semantic Urban Scene Understanding

Abstract: Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotations, 20 000 additional images have coarse annotations to enable methods that leverage large volumes of weakly-labeled data. Crucially, our effort exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity. Our accompanying empirical study provides an in-depth analysis of the dataset characteristics, as well as a performance evaluation of several state-of-the-art approaches based on our benchmark.