For the machine translation task, the baseline is a Transformer-big model on WMT 2014 English-German dataset #REFR . This dataset has around 4.5 million sentence pairs. The results are summarized in Table IX .

For comparison, each of the attention layers was replaced with Multi-linear attention in the Encoder of the Transformer. The results are summarized in Table VIII .

Notice that tensorized transformers with Multi-linear attention achieve better performance with fewer parameters than the vanilla Transformer.

2) Tensorized Embedding Layers: The proposed TTembedding layer was tested on two language modeling tasks (PTB and WikiText-103) and a machine translation task (WMT 2014 English-German).

As shown in Table VII , Transformer-XL+TT stands for the transformers with TTembedding layers.

Compared to the Transformer with Multilinear attention, Transformer-XL+TT can not achieve that high compression ratio.

Notice that the embedding layers can be compressed significantly at the cost of a small drop in the BLEU scores.

Tensor Decomposition For Model Reduction In Neural Networks: A Review

Abstract: Modern neural networks have revolutionized the fields of computer vision (CV) and Natural Language Processing (NLP). They are widely used for solving complex CV tasks and NLP tasks such as image classification, image generation, and machine translation. Most state-of-the-art neural networks are over-parameterized and require a high computational cost. One straightforward solution is to replace the layers of the networks with their low-rank tensor approximations using different tensor decomposition methods. This paper reviews six tensor decomposition methods and illustrates their ability to compress model parameters of convolutional neural networks (CNNs), recurrent neural networks (RNNs) and Transformers. The accuracy of some compressed models can be higher than the original versions. Evaluations indicate that tensor decompositions can achieve significant reductions in model size, run-time and energy consumption, and are well suited for implementing neural networks on edge devices.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

Transformer-big model
machine translation task


These results not only confirm the previous findings about longrun stability for persistent training #REFR , but also reveal a new phenomenon that the learned energies from persistent training appear to be locally meaningful, but globally misaligned.

(ii) The long-run sample (MALA starting from real data) also resemble the training data.

(iii) The post-training sample (MALA starting from standard Gaussian noises) shows two local modes at −2 and 2, but with proportions different from 3:1 as in the training data.

(iv) The learned energy function exhibits two local modes about −2 and 2, but is globally misaligned.

The estimated energy at the local mode −2 is substantially higher than at the other mode 2, and hence is also higher than (for example) at x = 1, an OOD point near no real data.

More troubling is that the estimated energies near −2 and 2 may reverse the direction of relative magnitudes from different training runs, as shown in Appendix E.1.

Hence application of such learned energies to OOD detection is problematic.

Motivated by the 1D example, we provide some new theoretical understanding of persistent training of EBMs with MCMC sampling which enables only local mixing for mul-timodal distributions with modes separated by low-density (i.e., high-energy) barriers.

Our discussion is heuristic, but highlights the main ideas which can be exploited to develop formal analysis.

In the limit of the persistent training process (assumed to exist), letθ be a limit value of the network parameter θ, andq be a limit distribution for the synthetic data (which can be represented by the empirical distribution of the replay buffer).

Then we expect that (θ,q) satisfy the following stationarity conditions:

Persistently Trained, Diffusion-Assisted Energy-Based Models

Abstract: Maximum likelihood (ML) learning for energybased models (EBMs) is challenging, partly due to non-convergence of Markov chain Monte Carlo. Several variations of ML learning have been proposed, but existing methods all fail to achieve both post-training image generation and proper density estimation. We propose to introduce diffusion data and learn a joint EBM, called diffusion assisted-EBMs, through persistent training (i.e., using persistent contrastive divergence) with an enhanced sampling algorithm to properly sample from complex, multimodal distributions. We present results from a 2D illustrative experiment and image experiments and demonstrate that, for the first time for image data, persistently trained EBMs can simultaneously achieve long-run stability, post-training image generation, and superior out-of-distribution detection.

On The Anatomy Of Mcmc-Based Maximum Likelihood Learning Of Energy-Based Models

Abstract: This study investigates the effects of Markov Chain Monte Carlo (MCMC) sampling in unsupervised Maximum Likelihood (ML) learning. Our attention is restricted to the family of unnormalized probability densities for which the negative log density (or energy function) is a ConvNet. In general, we find that many of the techniques used to stabilize training in previous studies can have the opposite effect. Stable ML learning with a ConvNet potential can be achieved with only a few hyper-parameters and no regularization. Using this minimal framework, we identify a variety of ML learning outcomes that depend on the implementation of MCMC sampling. ::: On one hand, we show that it is easy to train an energy-based model which can sample realistic images with short-run Langevin. ML can be effective and stable even when MCMC samples have much higher energy than true steady-state samples throughout training. Based on this insight, we introduce an ML method with purely noise-initialized MCMC, high-quality short-run synthesis, and the same budget as ML with informative MCMC initialization such as CD or PCD. Unlike previous models, our model can obtain realistic high-diversity samples from a noise signal after training with no auxiliary networks. ::: On the other hand, ConvNet potentials learned with highly non-convergent MCMC do not have a valid steady-state and cannot be considered approximate unnormalized densities of the training data because long-run MCMC samples differ greatly from observed images. We show that it is much harder to train a ConvNet potential to learn a steady-state over realistic images. To our knowledge, long-run MCMC samples of all previous models lose the realism of short-run samples. With correct tuning of Langevin noise, we train the first ConvNet potentials for which long-run and steady-state MCMC samples are realistic images.


The BERT classifier for AG's News is structured by the Distil-RoBERTa-base #REFR connected with two fully connected layers, and it is trained for 10 epochs with a learning rate of 0.0001.

We apply our attack algorithm to two popular and well-performed types of victim models. The details of the models can be found below.

BERT-based Classifiers To do convincing experiments, we choose three well-performed and popular BERT-based models, which we call BERT-C models, pre-trained by Huggingface 3 .

Due to the different sizes of the datasets, the structures of BERT-based classifiers are adjusted accordingly.

For the Emotion dataset, its BERT-C adopts another version of BERT, Distil-BERT-base-uncased #OTHEREFR , and the training hyper-parameters remain the same as BERT-C for AG's News.

Since the SST2 dataset is relatively small compared with the other two models, the corresponding BERT classifier utilizes a small-size version of BERT, BERT-base-uncased #OTHEREFR .

The test accuracy of these BERT-based classifiers before they are under attack are listed in Table 1 which are publicly accessible 4 5 6 .

TextCNN-based models The other type of victim model is TextCNN #OTHEREFR , structured with a 100-dimension embedding layer followed by a 128-unit long short-term memory layer.

This classifier is trained 10 epochs by ADAM optimizer with parameters: learning rate lr = 0.005, the two coefficients used for computing running averages of gradient and its square are set to be 0.9 and 0.999 (β 1 = 0.9, β 2 = 0.999), the denominator to improve numerical stability σ = 10 −5 .

Fraud'S Bargain Attack: Generating Adversarial Text Samples Via Word Manipulation Process

Abstract: Recent studies on adversarial examples expose vulnerabilities of natural language processing (NLP) models. Existing techniques for generating adversarial examples are typically driven by deterministic heuristic rules that are agnostic to the optimal adversarial examples, a strategy that often results in attack failures. To this end, this research proposes Fraud's Bargain Attack (FBA) which utilizes a novel randomization mechanism to enlarge the search space and enables high-quality adversarial examples to be generated with high probabilities. FBA applies the Metropolis-Hasting sampler, a member of Markov Chain Monte Carlo samplers, to enhance the selection of adversarial examples from all candidates proposed by a customized stochastic process that we call Word Manipulation Process (WMP). WMP perturbs one word at a time via insertion, removal or substitution in a contextual-aware manner. Extensive experiments demonstrate that FBA outperforms the state-of-the-art methods in terms of both attack success rate and imperceptibility.

Distilbert, A Distilled Version Of Bert: Smaller, Faster, Cheaper And Lighter

Abstract: As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.


Also the columns of the bifiltration (with s fixed) have been studied earlier in #REFR .

By excision and the fact that all pairs are "good pairs" #OTHEREFR , we have isomorphisms #OTHEREFR

H n (L s , L s \ B o r (q)) ∼ = H n (L s ∩ B r (q), L s ∩ ∂B r (q)) ∼ =Hn(Ls ∩ B r (q)/(L s ∩ ∂B r (q)))

These isomorphisms imply that the relative localized persistence module corresponds point-wise to the space considered in Figure 2 (right) .

Moreover, the excision isomorphism commutes with the inclusion maps L s ⊆ L s for s ≤ s which implies that the rows of the localized relative persistence module (with fixed r) are isomorphic to the module (H n (L s ∩ B r (q), L s ∩ ∂B r (q))) s≥0 which was studied in #OTHEREFR , #OTHEREFR , #OTHEREFR .

Hence, the localized relative bifiltration encodes both types of modules of local persistent homology studied in previous work (see Appendix D for a summary of basic notions).

We also remark that the computation of the relative localized persistence module can easily be reduced to the case of absolute homology via the well-known coning construction [30, p.125] , yielding isomorphisms H n (X, A) ∼ =Hn(X ∪ ω * A) for pairs of topological spaces (X, A) with ω denoting a new vertex.

These isomorphisms are functorial, yielding an isomorphism between the relative persistence module of a pair of bifiltrations and the absolute persistence module of a bifiltration (using reduced homology).

Moreover, if the pair (X, A) is finite simplicial, so is X ∪ ω * A.

The Localized Union-Of-Balls Bifiltration

Abstract: We propose an extension of the classical union-of-balls filtration of persistent homology: fixing a point q, we focus our attention to a ball centered at q whose radius is controlled by a second scale parameter. We discuss an absolute variant, where the union is just restricted to the q-ball, and a relative variant where the homology of the q-ball relative to its boundary is considered. Interestingly, these natural constructions lead to bifiltered simplicial complexes which are not k-critical for any finite k. Nevertheless, we demonstrate that these bifiltrations can be computed exactly and efficiently, and we provide a prototypical implementation using the CGAL library. We also argue that some of the recent algorithmic advances for 2-parameter persistence (which usually assume k-criticality for some finite k) carry over to the ∞-critical case.

Approximating Local Homology From Samples

Abstract: Recently, multi-scale notions of local homology (a variant of persistent homology) have been used to study the local structure of spaces around a given point from a point cloud sample. Current reconstruction guarantees rely on constructing embedded complexes which become difficult in high dimensions. We show that the persistence diagrams used for estimating local homology, can be approximated using families of Vietoris-Rips complexes, whose simple constructions are robust in any dimension. To the best of our knowledge, our results, for the first time, make applications based on local homology, such as stratification learning, feasible in high dimensions.


S EQUENTIAL recommendation systems are designed to capture the dynamic preferences of users based on their historical behaviors, with the goal of predicting the next item that they will be interested in #REFR .

The primary supervision signal utilized for learning the parameters of these models is typically derived from the sequential interactions of users with items.

However, given the sparsity of user behavior data, sequential recommendation methods that rely solely on such data are susceptible to the problem of data sparsity, resulting in suboptimal performance.

In practice, there exists a significant amount of multimodal information associated with items (e.g., images and text descriptions), which has been employed to alleviate the data sparsity problem in building conventional recommendation systems #OTHEREFR - #OTHEREFR .

For example, #OTHEREFR , #OTHEREFR leverage item multimodal content as a regularization factor and integrate it with collaborative filtering frameworks.

Recent studies #OTHEREFR - #OTHEREFR utilize graph neural networks to uncover the hidden links between different modalities and establish an in-depth understanding of users' preferences.

Multimodal Pre-Training Framework For Sequential Recommendation Via Contrastive Learning

Abstract: Sequential recommendation systems utilize the sequential interactions of users with items as their main supervision signals in learning users' preferences. However, existing methods usually generate unsatisfactory results due to the sparsity of user behavior data. To address this issue, we propose a novel pre-training framework, named Multimodal Sequence Mixup for Sequential Recommendation (MSM4SR), which leverages both users' sequential behaviors and items' multimodal content (i.e., text and images) for effectively recommendation. Specifically, MSM4SR tokenizes each item image into multiple textual keywords and uses the pre-trained BERT model to obtain initial textual and visual features of items, for eliminating the discrepancy between the text and image modalities. A novel backbone network, i.e., Multimodal Mixup Sequence Encoder (M 2 SE), is proposed to bridge the gap between the item multimodal content and the user behavior, using a complementary sequence mixup strategy. In addition, two contrastive learning tasks are developed to assist M 2 SE in learning generalized multimodal representations of the user behavior sequence. Extensive experiments on real-world datasets demonstrate that MSM4SR outperforms state-of-the-art recommendation methods. Moreover, we further verify the effectiveness of MSM4SR on other challenging tasks including cold-start and cross-domain recommendation.

Sequential Recommender Systems: Challenges, Progress And Prospects

Abstract: The emerging topic of sequential recommender systems has attracted increasing attention in recent years.Different from the conventional recommender systems including collaborative filtering and content-based filtering, SRSs try to understand and model the sequential user behaviors, the interactions between users and items, and the evolution of users preferences and item popularity over time. SRSs involve the above aspects for more precise characterization of user contexts, intent and goals, and item consumption trend, leading to more accurate, customized and dynamic recommendations.In this paper, we provide a systematic review on SRSs.We first present the characteristics of SRSs, and then summarize and categorize the key challenges in this research area, followed by the corresponding research progress consisting of the most recent and representative developments on this topic.Finally, we discuss the important research directions in this vibrant area.


Before applying MKPC, we already have two images I 1 and I 2 , and matched key-points on both images with any model (or any combinations of models), which is defined as X #REFR 1 and X 1 2 , respectively.

The MKPC algorithm crops critical regions by clustering the matching key-points between two images outputted by arbitrary image matching models. The workflow is shown is Figure 2 .

X 1 i denotes the i th image in the first stage of the two-stage pipeline which will be proposed in the next subsection.

It consists of the following three steps: (i) Clustering the matching key-points(X 1 1 and X #OTHEREFR 2 ) of two images(I 1 and I 2 ) with DBSCAN.

(ii) Generate a bounding box by filtering and gathering those clusters (iii) Crop the area covered by the Bounding box.

The Algorithm 1 describes the MKPC algorithm flow in detail.

With the input of two images (I 1 , I 2 ) and corresponding matched key-points in stage-one (X 1 1 and X 1 2 ), the MKPC outputs the respective cropped critical regions.

Efficient And Accurate Co-Visible Region Localization With Matching Key-Points Crop (Mkpc): A Two-Stage Pipeline For Enhancing Image Matching Performance

Abstract: Image matching is a classic and fundamental task in computer vision. In this paper, under the hypothesis that the areas outside the co-visible regions carry little information, we propose a matching key-points crop (MKPC) algorithm. The MKPC locates, proposes and crops the critical regions, which are the co-visible areas with great efficiency and accuracy. Furthermore, building upon MKPC, we propose a general two-stage pipeline for image matching, which is compatible to any image matching models or combinations. We experimented with plugging SuperPoint + SuperGlue into the two-stage pipeline, whose results show that our method enhances the performance for outdoor pose estimations. What's more, in a fair comparative condition, our method outperforms the SOTA on Image Matching Challenge 2022 Benchmark, which represents the hardest outdoor benchmark of image matching currently. * denotes contributing equally to this work. Preprint. Under review.

Superpoint: Self-Supervised Interest Point Detection And Description

Abstract: This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a large number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our fully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and associated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g., synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able to repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other traditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches when compared to LIFT, SIFT and ORB.


During the training phase, all our models were optimized with the Adam algorithm #REFR and a learning rate of lr = 0.001, for a maximum of 100 epoch with an early stopping mechanism activated when the validation loss stops improving after a duration of 5 epochs.

Additionally, every dropout is performed with r Dropout = 0.5, the batch normalization uses = 0.001 and γ = 0.99, and we use a batch size of 256 in every module apart from the fixation prediction performed with high-level features.

• Fixation Prediction from High-Level Features During training, we estimate the best weight and bias parameters that minimize the loss between the predicted outputŷ and the ground truth label y, with the cross entropy function computed for every fixation time step t for every sequence s of each mini-batch b:

where S corresponds to the batch size, T corresponds to the sequence length which is set to 6 (in addition to the initial fixation point at t = 0) and H × W is the output size which is set to 160.

We set with F = 5 filters, a kernel size of K = 4 and a stride of S = 2, and, during training, we varied the batch size between 32, 64, 128 and 256 and we conducted an ablation study over theses additional hyper-parameters and settings: -Fovea size: In this work we utilized the same real-time foveation system as in #OTHEREFR , and assessed the model performance when varying the fovea size, which defines the radius of the region with highest visual acuity with the values of 50, 75 and 100 pixels.

Learning To Search For And Detect Objects In Foveal Images Using Deep Learning

Abstract: The human visual system processes images with varied degrees of resolution, with the fovea, a small portion of the retina, capturing the highest acuity region, which gradually declines toward the field of view's periphery. However, the majority of existing object localization methods rely on images acquired by image sensors with space-invariant resolution, ignoring biological attention mechanisms. As a region of interest pooling, this study employs a fixation prediction model that emulates human objective-guided attention of searching for a given class in an image. The foveated pictures at each fixation point are then classified to determine whether the target is present or absent in the scene. Throughout this twostage pipeline method, we investigate the varying results obtained by utilizing high-level or panoptic features and provide a groundtruth label function for fixation sequences that is smoother, considering in a better way the spatial structure of the problem. Finally, we present a novel dual task model capable of performing fixation prediction and detection simultaneously, allowing knowledge transfer between the two tasks. We conclude that, due to the complementary nature of both tasks, the training process benefited from the sharing of knowledge, resulting in an improvement in performance when compared to the previous approach's baseline scores.

Adam: A Method For Stochastic Optimization

Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.


FFANet #REFR introduces feature attention (FA) blocks that leverage both channel and pixel attention to improve haze removal.

They focus on designing hand-crafted priors such as the dark channel prior #OTHEREFR and color attenuation prior #OTHEREFR .

However, these priors may not be powerful enough to characterize complex scenes in practice.

Early learning-based methods #OTHEREFR use deep neural networks to predict the transmission map and atmospheric light in the physics model to obtain a latent clear image.

However, inaccuracies in the estimations may accumulate, hindering the reliable inference of the haze-free image.

With the advent of large haze datasets #OTHEREFR , data-driven methods #OTHEREFR have been developed rapidly.

DeHamer #OTHEREFR combines CNN and Transformer for image dehazing, which can aggregate long-term attention in Transformer and local attention in CNN features.

Note that these methods do not consider the physics of the hazing process. Further, Dong et al.

propose a feature dehazing unit (FDU) #OTHEREFR derived based on the physics model.

To the best of our knowledge, this work is the only one that considers the physics model in the feature space, avoiding the cumulative errors that occur in the raw space.

However, FDU uses a shared structure to predict those unknown factors without considering their different physical characteristics.

Curricular Contrastive Regularization For Physics-Aware Single Image Dehazing

Abstract: Considering the ill-posed nature, contrastive regularization has been developed for single image dehazing, introducing the information from negative images as a lower bound. However, the contrastive samples are nonconsensual, as the negatives are usually represented distantly from the clear (i.e., positive) image, leaving the solution space still under-constricted. Moreover, the interpretability of deep dehazing models is underexplored towards the physics of the hazing process. In this paper, we propose a novel curricular contrastive regularization targeted at a consensual contrastive space as opposed to a non-consensual one. Our negatives, which provide better lower-bound constraints, can be assembled from 1) the hazy image, and 2) corresponding restorations by other existing methods. Further, due to the different similarities between the embeddings of the clear image and negatives, the learning difficulty of the multiple components is intrinsically imbalanced. To tackle this issue, we customize a curriculum learning strategy to reweight the importance of different negatives. In addition, to improve the interpretability in the feature space, we build a physics-aware dual-branch unit according to the atmospheric scattering model. With the unit, as well as curricular contrastive regularization, we establish our dehazing network, named C 2 PNet. Extensive experiments demonstrate that our C 2 PNet significantly outperforms state-of-the-art methods, with extreme PSNR boosts of 3.94dB and 1.50dB, respectively, on SOTSindoor and SOTS-outdoor datasets. Code is available at https://github.com/YuZheng9/C2PNet.

Ffa-Net: Feature Fusion Attention Network For Single Image Dehazing

Abstract: In this paper, we propose an end-to-end feature fusion at-tention network (FFA-Net) to directly restore the haze-free image. The FFA-Net architecture consists of three key components: ::: 1) A novel Feature Attention (FA) module combines Channel Attention with Pixel Attention mechanism, considering that different channel-wise features contain totally different weighted information and haze distribution is uneven on the different image pixels. FA treats different features and pixels unequally, which provides additional flexibility in dealing with different types of information, expanding the representational ability of CNNs. 2) A basic block structure consists of Local Residual Learning and Feature Attention, Local Residual Learning allowing the less important information such as thin haze region or low-frequency to be bypassed through multiple local residual connections, let main network architecture focus on more effective information. 3) An Attention-based different levels Feature Fusion (FFA) structure, the feature weights are adaptively learned from the Feature Attention (FA) module, giving more weight to important features. This structure can also retain the information of shallow layers and pass it into deep layers. ::: The experimental results demonstrate that our proposed FFA-Net surpasses previous state-of-the-art single image dehazing methods by a very large margin both quantitatively and qualitatively, boosting the best published PSNR metric from 30.23db to 36.39db on the SOTS indoor test dataset. ::: Code has been made available at GitHub.


We have opted MobileNet #REFR as a feature extractor to develop an FPAD model for devices with limited computational resources.

It is advantageous over state-ofthe-art CNN architectures as it utilizes depth-wise separable convolution instead of standard convolution.

A standard convolution operation takes input as X channels of size D x × D x and produces D y × D y × Y feature maps by applying D k × D k × Y filters where the spatial height and width of the squared input image are denoted with D x .

X denotes the number of input channels while D y is the spatial height and width.

The number of output feature maps is denoted with Y .

Equation 1describes the calculation of the output feature map for standard convolution operation #OTHEREFR having stride one with padding.

Mosfpad: An End-To-End Ensemble Of Mobilenet And Support Vector Classifier For Fingerprint Presentation Attack Detection

Abstract: Automatic fingerprint recognition systems are the most extensively used systems for person authentication although they are vulnerable to Presentation attacks. Artificial artifacts created with the help of various materials are used to deceive these systems causing a threat to the security of fingerprint-based applications. This paper proposes a novel end-to-end model to detect fingerprint Presentation attacks. The proposed model incorporates MobileNet as a feature extractor and a Support Vector Classifier as a classifier to detect presentation attacks in cross-material and cross-sensor paradigms. The feature extractor's parameters are learned with the loss generated by the support vector classifier. The proposed model eliminates the need for intermediary data preparation procedures, unlike other static hybrid architectures. The performance of the proposed model has been validated on benchmark

Mobilenets: Efficient Convolutional Neural Networks For Mobile Vision Applications

Abstract: We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem. We present extensive experiments on resource and accuracy tradeoffs and show strong performance compared to other popular models on ImageNet classification. We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization.


Following the setup of #REFR , we set the scale s to 64 and the margin m to 0.5 for arcface loss.

We train networks using the standard data augmentation (i.e., flipping, translation, cropping), and mask augmentation using the tools introduced in the ICCV2021-MFR/Insightface track #OTHEREFR .

We train the CASIA-Webface and MS1MV3 datasets #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR by employing SGD with a minibatch size of 512.

Momentum and weight decay are set to 0.9 and 5e-4, respectively.

We set initial learning rate to 0.2, and employ the polynomial learning rate decay scheduler #OTHEREFR , #OTHEREFR with 2 epochs of warm restart.

We finish the training at 25 epochs and 34 epochs for MS1MV3 and CASIA-Webface datasets, respectively.

Localization Using Multi-Focal Spatial Attention For Masked Face Recognition

Abstract: Since the beginning of worldwide COVID-19 pandemic, facial masks have been recommended to limit the spread of the disease. However, these masks hide certain facial attributes. Hence, it has become difficult for existing face recognition systems to perform identity verification on masked faces. In this context, it is necessary to develop masked Face Recognition (MFR) for contactless biometric recognition systems. Thus, in this paper, we propose Complementary Attention Learning and Multi-Focal Spatial Attention that precisely removes masked region by training complementary spatial attention to focus on two distinct regions: masked regions and backgrounds. In our method, standard spatial attention and networks focus on unmasked regions, and extract maskinvariant features while minimizing the loss of the conventional Face Recognition (FR) performance. For conventional FR, we evaluate the performance on the IJB-C, Age-DB, CALFW, and CPLFW datasets. We evaluate the MFR performance on the ICCV2021-MFR/Insightface track, and demonstrate the improved performance on the both MFR and FR datasets. Additionally, we empirically verify that spatial attention of proposed method is more precisely activated in unmasked regions.

Arcface: Additive Angular Margin Loss For Deep Face Recognition

Abstract: One of the main challenges in feature learning using Deep Convolutional Neural Networks (DCNNs) for large-scale face recognition is the design of appropriate loss functions that can enhance the discriminative power. Centre loss penalises the distance between deep features and their corresponding class centres in the Euclidean space to achieve intra-class compactness. SphereFace assumes that the linear transformation matrix in the last fully connected layer can be used as a representation of the class centres in the angular space and therefore penalises the angles between deep features and their corresponding weights in a multiplicative way. Recently, a popular line of research is to incorporate margins in well-established loss functions in order to maximise face class separability. In this paper, we propose an Additive Angular Margin Loss (ArcFace) to obtain highly discriminative features for face recognition. The proposed ArcFace has a clear geometric interpretation due to its exact correspondence to geodesic distance on a hypersphere. We present arguably the most extensive experimental evaluation against all recent state-of-the-art face recognition methods on ten face recognition benchmarks which includes a new large-scale image database with trillions of pairs and a large-scale video dataset. We show that ArcFace consistently outperforms the state of the art and can be easily implemented with negligible computational overhead. To facilitate future research, the code has been made available.


As shown in TableII, for 3D object detection task using Nuscenes dataset #REFR , both DETR3D and FUTR3D exhibit comparable performance due to their similar structures.

We benchmark major Transformer-based models on an NVIDIA GPU 3090 considering indicators such as input size, runtime, accuracy, and datasets.

BEVFormer outperforms DETR3D by generating BEV features and querying 3D objects from these features.

PETR and CrossDTR transform 2D features into 3D features using CNN networks, accelerating the query process and yielding better performance than DETR3D.

ResNet101's higher precision compared to ResNet50 can be attributed to its deformable convolution mechanism and increased convolution depth, although at the cost of slower runtime speed #OTHEREFR .

On the other hand, Transformerbased road element detection research exhibits greater variation, with different models and evaluation criteria for tasks such as 2D lane (TuSimple), 3D lane (OpenLane), and local map (Nuscenes).

Lane and local map Transformer queries are faster than object detection due to fewer key-point queries and smaller CNN backbones that utilize shallower layer features.

Transformer-Based Models And Hardware Acceleration Analysis In Autonomous Driving: A Survey

Abstract: Transformer architectures have exhibited promising performance in various autonomous driving applications in recent years. On the other hand, its dedicated hardware acceleration on portable computational platforms has become the next critical step for practical deployment in real autonomous vehicles. This survey paper provides a comprehensive overview, benchmark, and analysis of Transformer-based models specifically tailored for autonomous driving tasks such as lane detection, segmentation, tracking, planning, and decision-making. We review different architectures for organizing Transformer inputs and outputs, such as encoder-decoder and encoder-only structures, and explore their respective advantages and disadvantages. Furthermore, we discuss Transformer-related operators and their hardware acceleration schemes in depth, taking into account key factors such as quantization and runtime. We specifically illustrate the operator level comparison between layers from convolutional neural network, Swin-Transformer, and Transformer with 4D encoder. The paper also highlights the challenges, trends, and current insights in Transformerbased models, addressing their hardware deployment and acceleration issues within the context of long-term autonomous driving applications.

Nuscenes: A Multimodal Dataset For Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image-based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first published dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online at this http URL.


Less discrepancy between different distributions can be observed at later stages, the same pattern authors of IBNnet #REFR have observed.

Let F denote a feature map, and µ, σ 2 denote its mean and variance values, respectively.

Then symmetric KL divergence between two distributions S and S is computed as

We use the average divergence of all channels for each image and report the mean value for the whole set.

From Figure 2(a) , it can be seen that there is less feature divergence for the modified network, suggesting that learned representations are more generalizable.

Next, we visualize mean covariance and variance matrices (Figure 2(b) ) and observe that the covariance matrix computed from intermediate outputs of the modified model has more bright spots.

This might result from suppressing sensitive covariances previously present in the baseline.

There are also fewer activations in the variance matrix for H&E, which means that the distribution shift is causing less divergence for all channels on average.

The same pattern can be observed for TRI and PAS variance matrices (Figure 10 in Appendix C), however, for the latter, some of the brighter spots appear darker compared to the baseline.

t-SNE visualization (Figure 9 in Appendix C) of learned representations from stain normalized and original image versions of the first and last encoder stages show that they are closer when we integrate our method into training scheme.

Improving Stain Invariance Of Cnns For Segmentation By Fusing Channel Attention And Domain-Adversarial Training

Abstract: Variability in staining protocols, such as different slide preparation techniques, chemicals, and scanner configurations, can result in a diverse set of whole slide images (WSIs). This distribution shift can negatively impact the performance of deep learning models on unseen samples, presenting a significant challenge for developing new computational pathology applications. In this study, we propose a method for improving the generalizability of convolutional neural networks (CNNs) to stain changes in a single-source setting for semantic segmentation. Recent studies indicate that style features mainly exist as covariances in earlier network layers. We design a channel attention mechanism based on these findings that detects stain-specific features and modify the previously proposed stain-invariant training scheme. We reweigh the outputs of earlier layers and pass them to the stain-adversarial training branch. We evaluate our method on multi-center, multi-stain datasets and demonstrate its effectiveness through interpretability analysis. Our approach achieves substantial improvements over baselines and competitive performance compared to other methods, as measured by various evaluation metrics. We also show that combining our method with stain augmentation leads to mutually beneficial results and outperforms other techniques. Overall, our study makes significant contributions to the field of computational pathology.

Two At Once: Enhancing Learning And Generalization Capacities Via Ibn-Net

Abstract: Convolutional neural networks (CNNs) have achieved great successes in many computer vision problems. Unlike existing works that designed CNN architectures to improve performance on a single task of a single domain and not generalizable, we present IBN-Net, a novel convolutional architecture, which remarkably enhances a CNN's modeling ability on one domain (e.g. Cityscapes) as well as its generalization capacity on another domain (e.g. GTA5) without finetuning. IBN-Net carefully integrates Instance Normalization (IN) and Batch Normalization (BN) as building blocks, and can be wrapped into many advanced deep networks to improve their performances. This work has three key contributions. (1) By delving into IN and BN, we disclose that IN learns features that are invariant to appearance changes, such as colors, styles, and virtuality/reality, while BN is essential for preserving content related information. (2) IBN-Net can be applied to many advanced deep architectures, such as DenseNet, ResNet, ResNeXt, and SENet, and consistently improve their performance without increasing computational cost. (3) When applying the trained networks to new domains, e.g. from GTA5 to Cityscapes, IBN-Net achieves comparable improvements as domain adaptation methods, even without using data from the target domain. With IBN-Net, we won the 1st place on the WAD 2018 Challenge Drivable Area track, with an mIoU of 86.18%.


Horse → Zebra is provided in #REFR , which contains 1,067 and 1,334 training images for horse and zebra, respectively.

SN-DCR is trained and evaluated on Horse → Zebra, Cat → Dog , Van Gogh → Photo and CityScapes datasets.

We use 120 horse images as the test images on Horse → Zebra.

Cat → Dog is from #OTHEREFR , which consists of 5,153 and 4,739 training images for cat and dog, respectively.

We used 500 images of cats as test images on Cat → Dog .

Van Gogh → Photo is a dataset of 400 Van Gogh paintings and 6287 photos extracted from #OTHEREFR . We used 400 Van Gogh images as test images.

Cityscapes contains street scenes from German cities, with 2,975 training images and 500 test images.

Spectral Normalized Dual Contrastive Regularization For Image-To-Image Translation

Abstract: Existing image-to-image(I2I) translation methods achieve state-of-the-art performance by incorporating the patch-wise contrastive learning into Generative Adversarial Networks. However, patch-wise contrastive learning only focuses on the local content similarity but neglects the global structure constraint, which affects the quality of the generated images. In this paper, we propose a new unpaired I2I translation framework based on dual contrastive regularization and spectral normalization, namely SN-DCR. To maintain consistency of the global structure and texture, we design the dual contrastive regularization using different feature spaces respectively. In order to improve the global structure information of the generated images, we formulate a semantically contrastive loss to make the global semantic structure of the generated images similar to the real images from the target domain in the semantic feature space. We use Gram Matrices to extract the style of texture from images. Similarly, we design style contrastive loss to improve the global texture information of the generated images. Moreover, to enhance the stability of model, we employ the spectral normalized convolutional network in the design of our generator. We conduct the comprehensive experiments to evaluate the effectiveness of SN-DCR, and the results prove that our method achieves SOTA in multiple tasks.

Unpaired Image-To-Image Translation Using Cycle-Consistent Adversarial Networks

Abstract: Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain $X$ to a target domain $Y$ in the absence of paired examples. Our goal is to learn a mapping $G: X \rightarrow Y$ such that the distribution of images from $G(X)$ is indistinguishable from the distribution $Y$ using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping $F: Y \rightarrow X$ and introduce a cycle consistency loss to push $F(G(X)) \approx X$ (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.


Motivated by #REFR , to understand whether visual question answering (VQA) models and humans focus on the same parts of an image to answer a question, a researcher may want to compare the saliency maps of a VQA model and human attention maps.

To ensure that a model is classifying the image based on the properties of the object, a developer can look at the number of mask pixels with high values within the bounding box of the object and compare that number to either the area of the object or the total number of mask pixels with high values.

A developer could further query for the top-images where such ratios are highest or lowest.

Concretely, a query over our toy example could count the number of pixels with values higher than 0.85 in the bounding box, which produces a count of two pixels, and compare against the area, which is 6 pixels, for a ratio of 0.33.

The query could return all images with a ratio below some threshold or could return the images with the lowest ratios.

Example 2: Comparing model saliency maps and human attention maps.

To achieve this, they can generate saliency maps for the VQA model and human attention maps for the same images, and then issue a query to retrieve images with the highest number of pixels with values above some threshold in the intersection of the two masks.

The returned images are those for which the VQA model and humans focus on the same parts of the image to answer a question.

Masksearch: Querying Image Masks At Scale

Abstract: Machine learning tasks over image databases often generate masks that annotate image content (e.g., saliency maps, segmentation maps) and enable a variety of applications (e.g., determine if a model is learning spurious correlations or if an image was maliciously modified to mislead a model). While queries that retrieve examples based on mask properties are valuable to practitioners, existing systems do not support such queries efficiently. In this paper, we formalize the problem and propose a system, MaskSearch, that focuses on accelerating queries over databases of image masks. MaskSearch leverages a novel indexing technique and an efficient filter-verification query execution framework. Experiments on realworld datasets with our prototype show that MaskSearch, using indexes approximately 5% the size of the data, accelerates individual queries by up to two orders of magnitude and consistently outperforms existing methods on various multi-query workloads that simulate dataset exploration and analysis processes.

Human Attention In Visual Question Answering: Do Humans And Deep Networks Look At The Same Regions?

Abstract: We conduct large-scale studies on ‘human attention’ in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). Our experiments show that current attention models in VQA do not seem to be looking at the same regions as humans. Finally, we train VQA models with explicit attention supervision, and find that it improves VQA performance.


Existing methods such as CLEVR and CLEVR-Change use text which has ambiguity issues making the evaluation unreliable, while CLEVR-Ref+ #REFR employs bounding boxes that are specific but require the additional ability of detection.

This is the major reason that we choose CLEVR to extend.

Another reason is that images can be synthesized using Blender #OTHEREFR with small costs. Therefore, it is practicable to create millions of samples.

CLEVR provides a good foundation on attributes and values, which are fundamental items of the atomic transformation triplet (o, a, v), as we introduced in Sec. 3.

However, the distance to defining atomic transformations well still exists unless we proceed with several modifications or designs.

The first problem is how to represent an object in the answer.

Therefore, we design to provide additional information, which is the attributes of the initial objects, including the index, color, material, and other attribute values.

In this way, an object can be referred to with its index.

Note machines still need to perform their own recognition to align objects in images with given attributes.

The second problem is available values in size and material are too few, therefore we add medium size and glass material.

The last problem is the available values of position transformation are infinite in the space of R 2 , which is not computational friendly.

Visual Reasoning: From State To Transformation

Abstract: Most existing visual reasoning tasks, such as CLEVR in VQA, ignore an important factor, i.e. transformation. They are solely defined to test how well machines understand concepts and relations within static settings, like one image. Such state driven visual reasoning has limitations in reflecting the ability to infer the dynamics between different states, which has shown to be equally important for human cognition in Piaget's theory. To tackle this problem, we propose a novel transformation driven visual reasoning (TVR) task. Given both the initial and final states, the target becomes to infer the corresponding intermediate transformation. Following this definition, a new synthetic dataset namely TRANCE is first constructed on the basis of CLEVR, including three levels of settings, i.e. Basic (single-step transformation), Event (multi-step transformation), and View (multi-step transformation with variant views). Next, we build another real dataset called TRANCO based on COIN, to cover the loss of transformation diversity on TRANCE. Inspired by human reasoning, we propose a three-staged reasoning framework called TranNet, including observing, analyzing, and concluding, to test how recent advanced techniques perform on TVR. Experimental results show that the state-of-the-art visual reasoning models perform well on Basic, but are still far from human-level intelligence on Event, View, and TRANCO. We believe the proposed new paradigm will boost the development of machine visual reasoning. More advanced methods and new problems need to be investigated in this direction. The resource of TVR is available at https://hongxin2019.github.io/TVR/.

Clevr-Ref+: Diagnosing Visual Reasoning With Referring Expressions

Abstract: Referring object detection and referring image segmentation are important tasks that require joint understanding of visual information and natural language. Yet there has been evidence that current benchmark datasets suffer from bias, and current state-of-the-art models cannot be easily evaluated on their intermediate reasoning process. To address these issues and complement similar efforts in visual question answering, we build CLEVR-Ref+, a synthetic diagnostic dataset for referring expression comprehension. The precise locations and attributes of the objects are readily available, and the referring expressions are automatically associated with functional programs. The synthetic nature allows control over dataset bias (through sampling strategy), and the modular programs enable intermediate reasoning ground truth without human annotators. In addition to evaluating several state-of-the-art models on CLEVR-Ref+, we also propose IEP-Ref, a module network approach that significantly outperforms other models on our dataset. In particular, we present two interesting and important findings using IEP-Ref: (1) the module trained to transform feature maps into segmentation masks can be attached to any intermediate module to reveal the entire reasoning process step-by-step; (2) even if all training data has at least one object referred, IEP-Ref can correctly predict no-foreground when presented with false-premise referring expressions. To the best of our knowledge, this is the first direct and quantitative proof that neural modules behave in the way they are intended. We will release data and code for CLEVR-Ref+.


A pretrained T5 model #REFR is used to fill the blanks and generate an augmented sample.

As mentioned in Section 2.2, we augment the original prompt and generate both label-preserving and label-flipping prompts #OTHEREFR ) before applying the three-level Mixup.

Both the input text and the templates are augmented separately.

We generate label-preserving and label-flipping augmented text, but only label-preserving templates are generated.

Following , we use a cloze pattern to combine both the input text (template) and the label into a single sequence, and then randomly mask a fixed percentage of the input tokens.

For model training, we adopt the experimental settings used in PET #OTHEREFR and conduct a grid search for the three-level Mixup, as indicated in Table 2 .

To evaluate the effectiveness of our DA methods, we employ PET #OTHEREFR as the backbone and augment it with MIXPRO and other DA baselines.

We utilize Albert-xxlarge-v2 #OTHEREFR as the PLM and measure the performance using the identical metrics in Table1, namely Acc., F1, F1 a , and EM.

Since few-shot learning typically exhibits significant performance fluctuations (Dodge et al., 2020; #OTHEREFR , we use 5 independent seeds and report the average performance across these seeds for each model.

All of our experiments were performed on a Linux platform equipped with NVIDIA A100 (40G).

Mixpro: Simple Yet Effective Data Augmentation For Prompt-Based Learning

Abstract: Prompt-based learning reformulates downstream tasks as cloze problems by combining the original input with a template. This technique is particularly useful in few-shot learning, where a model is trained on a limited amount of data. However, the limited templates and text used in few-shot prompt-based learning still leave significant room for performance improvement. Additionally, existing methods (Schick and Schütze, 2021c) using model ensembles can constrain the model efficiency. To address these issues, we propose an augmentation method called MIXPRO, which augments both the vanilla input text and the templates through token-level, sentence-level, and epoch-level Mixup strategies. We conduct experiments on five few-shot datasets, and the results show that MIXPRO outperforms other augmentation baselines, improving model performance by an average of 5.08% compared to before augmentation.

Exploring The Limits Of Transfer Learning With A Unified Text-To-Text Transformer

Abstract: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.


For x ∈ F k , by #REFR , if k ∈ L i , then u i ∈ F k+1 ; otherwise, u i ∈ F k . This completes the proof.

Thus, if x ∈ F 0 , prox γ r1 (x) ∈ F 1 ; if x ∈ F k for k ≥ 1, prox γ r1 (x) ∈ F k .

(ii) For i = 1, we define u i := prox γ ri (x) for simplicity. Then u i satisfies the following equation

I + nγ cγ + 1 B i B i −1 x = x − B i D i B i x cγ + 1 .

Stochastic Distributed Optimization Under Average Second-Order Similarity: Algorithms And Analysis

Abstract: We study finite-sum distributed optimization problems with n-clients under popular δ-similarity condition and µ-strong convexity. We propose two new algorithms: SVRS and AccSVRS motivated by previous works. The non-accelerated SVRS method combines the techniques of gradient-sliding and variance reduction, which achieves superior communication complexityÕ(n+ √ nδ/µ) compared to existing non-accelerated algorithms. Applying the framework proposed in Katyusha X [6], we also build a direct accelerated practical version named AccSVRS with totally smoothness-freeÕ(n+n 3/4 δ/µ) communication complexity that improves upon existing algorithms on ill-conditioning cases. Furthermore, we show a nearly matched lower bound to verify the tightness of our AccSVRS method.

An Optimal Randomized Incremental Gradient Method

Abstract: In this paper, we consider a class of finite-sum convex optimization problems whose objective function is given by the summation of $m$ ($\ge 1$) smooth components together with some other relatively simple terms. We first introduce a deterministic primal-dual gradient (PDG) method that can achieve the optimal black-box iteration complexity for solving these composite optimization problems using a primal-dual termination criterion. Our major contribution is to develop a randomized primal-dual gradient (RPDG) method, which needs to compute the gradient of only one randomly selected smooth component at each iteration, but can possibly achieve better complexity than PDG in terms of the total number of gradient evaluations. More specifically, we show that the total number of gradient evaluations performed by RPDG can be ${\cal O} (\sqrt{m})$ times smaller, both in expectation and with high probability, than those performed by deterministic optimal first-order methods under favorable situations. We also show that the complexity of the RPDG method is not improvable by developing a new lower complexity bound for a general class of randomized methods for solving large-scale finite-sum convex optimization problems. Moreover, through the development of PDG and RPDG, we introduce a novel game-theoretic interpretation for these optimal methods for convex optimization.


A. Linear Systems With Gaussian Uncertainties

Some notes regarding the synthesis: (a) It is possible to derive the Hessian as well (see #REFR ).

However, when optimizing for cases where p ϕ f 1, we want smaller steps so we do not overshoot.

We introduce p ϕ f in the gradient descent update as a normalizing factor so that the learning rate is easier to obtain and can remain relatively constant.

We also introduce a sign variable where v dir = 1 if we search for p ϕ s and v dir = −1, if we search for p ϕ f .

The gradient descent process for some parameter γ of the system, is then:

A possible improvement to the gradient descent is by incorporating the second-degree derivative for more accuracy.

(b) The lower the probability of failure is, the longer the computation (more nestings).

Keep in mind that p ϕ f = 0 is not possible due to the assumption that the uncertainty is an unbounded Gaussian.

Therefore, it is up to the designer to specify the stopping criteria at the requisite level of performance.

(c) STL-based ESS can also attempt to find the initial trajectory.

Ensuring Reliable Robot Task Performance Through Probabilistic Rare-Event Verification And Synthesis

Abstract: Providing guarantees on the safe operation of robots against edge cases is challenging as testing methods such as traditional Monte-Carlo require too many samples to provide reasonable statistics. Built upon recent advancements in rareevent sampling, we present a model-based method to verify if a robotic system satisfies a Signal Temporal Logic (STL) specification in the face of environment variations and sensor/actuator noises. Our method is efficient and applicable to both linear and nonlinear and even black-box systems with arbitrary, but known, uncertainty distributions. For linear systems with Gaussian uncertainties, we exploit a feature to find optimal parameters that minimize the probability of failure. We demonstrate illustrative examples on applying our approach to real-world autonomous robotic systems.

Integrals Over Gaussians Under Linear Domain Constraints

Abstract: Integrals of linearly constrained multivariate Gaussian densities are a frequent problem in machine learning and statistics, arising in tasks like generalized linear models and Bayesian optimization. Yet they are notoriously hard to compute, and to further complicate matters, the numerical values of such integrals may be very small. We present an efficient black-box algorithm that exploits geometry for the estimation of integrals over a small, truncated Gaussian volume, and to simulate therefrom. Our algorithm uses the Holmes-Diaconis-Ross (HDR) method combined with an analytic version of elliptical slice sampling (ESS). Adapted to the linear setting, ESS allows for efficient, rejection-free sampling, because intersections of ellipses and domain boundaries have closed-form solutions. The key idea of HDR is to decompose the integral into easier-to-compute conditional probabilities by using a sequence of nested domains. Remarkably, it allows for direct computation of the logarithm of the integral value and thus enables the computation of extremely small probability masses. We demonstrate the effectiveness of our tailored combination of HDR and ESS on high-dimensional integrals and on entropy search for Bayesian optimization.


We set the learning rate as 0.005, and decay it to 0 following the cosine strategy #REFR .

We set the maximum iterations number to 1,000, and determine the L2 regularization weights following DECLIP's hyperparameter sweeping strategy #OTHEREFR .

We do not report the results on the ImageNet-1K dataset, due to the high computational cost of conducting hyperparameter sweeping on the dataset. Non-linear Probe Task.

The downstream task head consists of a fully-connected layer with GELU activation and a fully-connected layer.

The extracted FDT features of images and questions are concatenated and then fed to the downstream task head to predict the answers. The encoders and FDT are frozen during the training.

The downstream head is optimized by the AdamW optimizer #OTHEREFR .

Revisiting Multimodal Representation In Contrastive Learning: From Patch And Token Embeddings To Finite Discrete Tokens

Abstract: Contrastive learning-based vision-language pretraining approaches, such as CLIP, have demonstrated great success in many vision-language tasks. These methods achieve cross-modal alignment by encoding a matched image-text pair with similar feature embeddings, which are generated by aggregating information from visual patches and language tokens. However, direct aligning cross-modal information using such representations is challenging, as visual patches and text tokens differ in semantic levels and granularities. To alleviate this issue, we propose a Finite Discrete Tokens (FDT) based multimodal representation. FDT is a set of learnable tokens representing certain visualsemantic concepts. Both images and texts are embedded using shared FDT by first grounding multimodal inputs to FDT space and then aggregating the activated FDT representations. The matched visual and semantic concepts are enforced to be represented by the same set of discrete tokens by a sparse activation constraint. As a result, the granularity gap between the two modalities is reduced. Through both quantitative and qualitative analyses, we demonstrate that using FDT representations in CLIP-style models improves cross-modal alignment and performance in visual recognition and vision-language downstream tasks. Furthermore, we show that our method can learn more comprehensive representations, and the learned FDT capture meaningful cross-modal correspondence, ranging from objects to actions and attributes. 1 * This work was done during a research internship at ByteDance. † Dimitris N. Metaxas has been supported by NSF IUCRC CARTA

Sgdr: Stochastic Gradient Descent With Warm Restarts

Abstract: Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR


To find the minimum power requirement for a 2-Group IC-NOMA transmission that makes R IN −N OMA #REFR (n,f ) at least as good as R IC , consider (25) and (31).

Since, l IC ≥ l m , we have P #OTHEREFR saving > 0

3)CASE III (l n > l f > l m ) : The 3-Group NOMA transmission of the scheme saves power compared to conventional IC to achieve equal information rate as given by (27).

(n,f ) denote the power per 2-Group NOMA transmission in this case. Then, (16) is modified as

From (32), we have ζ (n,f ) > 0 =⇒ P IC > P

(n,f ) , i.e., To achieve a rate at least as good as that of conventional IC, power per transmission in conventional IC system is greater when compared to power per a 2-Group NOMA transmission in this case . Then,

Let P IC n denote the power per IC transmission in this case, then (4) is modified as

Design And Analysis Of Index Codes For 3-Group Noma In Vehicular Adhoc Networks

Abstract: Index coding (IC) is a source coding technique employed to improve spectral utilisation, where the source node aims to satisfy users' demands by making minimum transmissions. Non-orthogonal multiple access (NOMA) is integral to the radio access technique used in 5G networks. Index-coded NOMA (IC-NOMA) transmission scheme in Vehicular Adhoc Networks (VANETs) involves applying NOMA principles on index-coded data to avoid network congestion and to improve spectral efficiency compared to conventional IC systems. In this work, a spectral efficient transmission scheme called 3-Group IC-NOMA is proposed, and an innovative index code design that fits with NOMA decoding principles to obtain improved spectral efficiency is developed. Through exhaustive analytical studies, we demonstrate that the proposed transmission scheme always supports higher rates than the conventional IC systems and requires less power to achieve an information rate at least as good as conventional IC systems.

Efficient 3D Road Map Data Exchange For Intelligent Vehicles In Vehicular Fog Networks

Abstract: Through connecting intelligent vehicles as well as the roadside infrastructure, the perception range of vehicles can be significantly extended, and hidden objects at blind spots can be efficiently detected and avoided. To realize this, accurate road map data must be downloaded in real time to these intelligent vehicles for navigation and localization purposes. Besides, the cloud must be updated with dynamic changes that happened in the road network. These involve the transmissions of high-definition 3D road map data for accurately representing the physical environments. In this work, we propose solutions under the fog computing architecture in a heterogeneous vehicular network to optimize data exchange among intelligent vehicles, the roadside infrastructure, as well as regional databases. Specifically, the efficiency of 3D road map data dissemination at roadside fog nodes is achieved by exploiting index coding techniques to reduce the overall data load, while opportunistic scheduling of heterogeneous transmissions can be done to judiciously manage network resources and minimize operating cost. In addition, 3D point cloud coding and hashing techniques are applied to expedite the updates of various dynamic changes in the network. We empirically evaluate the proposed solutions based on real-world mobility traces of vehicles and 3D LIght Detection And Ranging (LIDAR) data of city streets. The proposed system is also implemented in a multi-robotic testbed for practical evaluation.


Constrained Policy Optimization (CPO) #REFR is a pioneering general-purpose policy exploit algorithm for constrained reinforcement learning with guarantees for near-constraint satisfaction at each iteration.

#OTHEREFR train a DQN agent combined with DNN which outputs two discrete actions.

The safety and agility of the ego vehicle can be balanced on-the-go, indicating that the RL agent can learn an adaptive behavior. Furthermore, Ronecker et al.

#OTHEREFR propose a safer navigating method for IVs in highway scenarios by combining Deep Q-Networks from control theory.

The proposed network is trained in simulation for central decision-making by proposing targets for a trajectory planner, which shows that the value-based RL can produce efficient and safe driving behavior in highway traffic scenarios.

The security of end-to-end autonomous driving also raises significant apprehension.

Building on this, #OTHEREFR and #OTHEREFR present the Safety Gym benchmark suite and validate several constrained deep RL algorithms under constrained conditions. Li et al.

#OTHEREFR introduce a risk awareness algorithm into DRL frameworks to learn a riskaware driving decision policy for lane-changing tasks with the minimum expected risk. Chow et al.

#OTHEREFR propose safe policy optimization algorithms that employ a Lyapunov-based approach #OTHEREFR to address CMDP problems. Furthermore, Yang et al.

#OTHEREFR construct a model-free safe RL algorithm that integrates policy and neural barrier certificate learning in a stepwise state constraint scenario. Mo et al.

#OTHEREFR leverage Monte Carlo Tree Search to reduce unsafe behaviors on overtaking subtasks at highway scenarios. Fig. 7 .

Motion Planning For Autonomous Driving: The State Of The Art And Future Perspectives

Abstract: Thanks to the augmented convenience, safety advantages, and potential commercial value, Intelligent vehicles (IVs) have attracted wide attention throughout the world. Although a few autonomous driving unicorns assert that IVs will be commercially deployable by 2025, their implementation is still restricted to small-scale validation due to various issues, among which precise computation of control commands or trajectories by planning methods remains a prerequisite for IVs. This paper aims to review state-of-the-art planning methods, including pipeline planning and end-to-end planning methods. In terms of pipeline methods, a survey of selecting algorithms is provided along with a discussion of the expansion and optimization mechanisms, whereas in end-to-end methods, the training approaches and verification scenarios of driving tasks are points of concern. Experimental platforms are reviewed to facilitate readers in selecting suitable training and validation methods. Finally, the current challenges and future directions are discussed. The sideby-side comparison presented in this survey not only helps to gain insights into the strengths and limitations of the reviewed methods but also assists with system-level design choices. Index Terms-Pipeline planning, end-to-end planning, imitation learning, reinforcement learning, parallel learning. I. INTRODUCTION I NTELLIGENT vehicles (IVs) have gained considerable attention from government, industry, academia, and the

Constrained Policy Optimization

Abstract: For many applications of reinforcement learning it can be more convenient to specify both a reward function and constraints, rather than trying to design behavior through the reward function. For example, systems that physically interact with or around humans should satisfy safety constraints. Recent advances in policy search algorithms (Mnih et al., 2016, Schulman et al., 2015, Lillicrap et al., 2016, Levine et al., 2016) have enabled new capabilities in high-dimensional control, but do not consider the constrained setting. ::: We propose Constrained Policy Optimization (CPO), the first general-purpose policy search algorithm for constrained reinforcement learning with guarantees for near-constraint satisfaction at each iteration. Our method allows us to train neural network policies for high-dimensional control while making guarantees about policy behavior all throughout training. Our guarantees are based on a new theoretical result, which is of independent interest: we prove a bound relating the expected returns of two policies to an average divergence between them. We demonstrate the effectiveness of our approach on simulated robot locomotion tasks where the agent must satisfy constraints motivated by safety.


Various forms of normalisation differ along which dimensions they calculate the reference statistics (see #REFR for a comparison).

Residual connections preserve high resolution information, facilitate the propagation of gradient information to early encoding layers, and thus avoid the vanishing gradient problem and enabled the training of very deep neural networks #OTHEREFR .

As a result, residual blocks are now used in essentially all modern DL architectures.

Besides data normalisation, discussed in Section 2.3, dedicated normalisation modules between (or after) the projection and the non-linearity within a processing block may be applied #OTHEREFR .

Empirically, it has been found that normalisation leads to faster and more stable convergence (for a theoretical perspective see e.g. #OTHEREFR ).

Normalisation re-scales all intermediate activations within a layer to have zero mean and unit variance.

Among the reviewed models, P22, K22, and E21, and H22 utilise normalisation over the channel and spatial dimensions, whereas R21 use normalisation over the batch dimension and W21 omit internal normalisation entirely.

Immediately after the normalisation, an affine transformation with learned parameters is often applied to allow the activation statistics to be adapted to the data.

Recent work in #OTHEREFR and #OTHEREFR suggests that learned affine parameters may achieve the beneficial effects of normalisation by themselves.

In E21, for example, the parameters of the affine transformation are predicted by a secondary neural network based on the targeted forecast lead-time, which enables maximally effective, layer-specific lead-time encodings.

The same principle has been used in previous work #OTHEREFR to condition on various kinds of auxiliary information, an approach that could hold promise in DLWP for integrating other relevant data sources.

Inductive Biases In Deep Learning Models For Weather Prediction

Abstract: Deep learning has recently gained immense popularity in the Earth sciences as it enables us to formulate purely data-driven models of complex Earth system processes. Deep learning-based weather prediction (DLWP) models have made significant progress in the last few years, achieving forecast skills comparable to established numerical weather prediction (NWP) models with comparatively lesser computational costs. In order to train accurate, reliable, and tractable DLWP models with several millions of parameters, the model design needs to incorporate suitable inductive biases that encode structural assumptions about the data and modelled processes. When chosen appropriately, these biases enable faster learning and better generalisation to unseen data. Although inductive biases play a crucial role in successful DLWP models, they are often not stated explicitly and how they contribute to model performance remains unclear. Here, we review and analyse the inductive biases of six state-of-the-art DLWP models, involving a deeper look at five key design elements: input data, forecasting objective, loss components, layered design of the deep learning architectures, and optimisation methods. We show how the design choices made in each of the five design elements relate to structural assumptions. Given recent developments in the broader DL community, we anticipate that the future of DLWP will likely see a wider use of foundation models-large models pre-trained on big databases with self-supervised learning-combined with explicit physics-informed inductive biases that allow the models to provide competitive forecasts even at the more challenging subseasonal-to-seasonal scales.

Group Normalization

Abstract: Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems --- BN's error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN's usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN's computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform its BN-based counterparts for object detection and segmentation in COCO, and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.


Figure 2 highlights the aim of our work at large which is inline with MITRE's vision on the standardization of CVE records, an extremely valuable task to the community #REFR .

We approached the task as a ranking problem, using text similarity SBERT #OTHEREFR , optimized for sentences (vs.

the document level) and the recently released ranking T5 #OTHEREFR model on the dataset.

Our results were compared to the more general-built models BERT #OTHEREFR and RoBERTa #OTHEREFR . BM25 #OTHEREFR was used as the baseline model.

Moreover, we used T5 as Seq2Seq generation #OTHEREFR model as well as a ranker.

The rest of this paper is organized as follows: Section 2 provides a summary of related work.

Section 3 briefly summarizes the MITRE weaknesses types and how they are used.

Section 4 explains the rationale behind annotating and maintaining the released cyber-security AI dataset. Section 5 introduces our methodology. Section 6 describes our experimentation and discusses the results.

Finally, Section 7 concludes and points to possible directions for future work.

Automated Mapping Of Cve Vulnerability Records To Mitre Cwe Weaknesses

Abstract: In recent years, a proliferation of cyber-security threats and diversity has been on the rise culminating in an increase in their reporting and analysis. To counter that, many non-profit organizations have emerged in this domain, such as MITRE and OSWAP, which have been actively tracking vulnerabilities, and publishing defense recommendations in standardized formats. As producing data in such formats manually is very time-consuming, there have been some proposals to automate the process. Unfortunately, a major obstacle to adopting supervised machine learning for this problem has been the lack of publicly available specialized datasets. Here, we aim to bridge this gap. In particular, we focus on mapping CVE records into MITRE CWE Weaknesses, and we release to the research community a manually annotated dataset of 4,012 records for this task. With a human-in-the-loop framework in mind, we approach the problem as a ranking task and aim to incorporate reinforced learning to make use of the human feedback in future work. Our experimental results using fine-tuned deep learning models, namely Sentence-BERT and rankT5, show sizable performance gains over BM25, BERT, and RoBERTa, which demonstrates the need for an architecture capable of good semantic understanding for this task.

Automatic Labeling For Entity Extraction In Cyber Security

Abstract: Timely analysis of cyber-security information necessitates automated information extraction from unstructured text. While state-of-the-art extraction methods produce extremely accurate results, they require ample training data, which is generally unavailable for specialized applications, such as detecting security related entities; moreover, manual annotation of corpora is very costly and often not a viable solution. In response, we develop a very precise method to automatically label text from several data sources by leveraging related, domainspecific, structured data and provide public access to a corpus annotated with cyber-security entities. Next, we implement a Maximum Entropy Model trained with the average perceptron on a portion of our corpus ( 750,000 words) and achieve near perfect precision, recall, and accuracy, with training times under 17 seconds.


Rq2. Effects On Geometric Primitives And Concepts: How Well Does Lbs Represent Geometric Primitives And Understand Their Conceptual Relationships?

We validate whether LBS can reflect the local geometric information of each object in a synthetic photo dataset consisting of multiple objects, using the CLEVR dataset #REFR .

To evaluate whether LBS is suitable for representing geometric primitives and concepts, we perform a classification task on the Geoclidean dataset #OTHEREFR .

The Geoclidean dataset consists of realized images from a concept of Euclidean geometry, e.g., black parallel lines on a white background.

Geoclidean is divided into two categories: Geoclidean-Elements and Geoclidean-Constraints, visualized in Fig. 7 on Appendix B.

By providing a limited training set consisting of only 10 images per concept, we evaluate whether LBS can effectively learn the high-level relationships between each primitive and generalize them across different examples by classifying the concept of the test image. RQ3.

Local geometric information and spatial reasoning: How effectively does LBS reflect local geometric information and extend it to spatial reasoning tasks?

We train our model with very limited descriptions, where the label for the entire scene is provided as the rightmost object or without any descriptions at all.

We validate the effectiveness of our representation by evaluating its ability to successfully classify attributes that are not provided as labels, such as determining the color of the leftmost object.

Additionally, we test its ability to perform simple spatial reasoning, such as shifting the rightmost object and inferring the attribute of the current rightmost object. RQ4.

Domain transfer: Can the geometric concepts of LBS trained in a specific domain be extended to other domains?

To investigate whether the learned representation within a specific image domain provides meaningful geometric information across other domains, we evaluate the model by shifting the distribution from the STL-10 [11] dataset to CLEVR and vice versa.

Learning Geometry-Aware Representations By Sketching

Abstract: Understanding geometric concepts, such as distance and shape, is essential for understanding the real world and also for many vision tasks. To incorporate such information into a visual representation of a scene, we propose learning to represent the scene by sketching, inspired by human behavior. Our method, coined Learning by Sketching (LBS), learns to convert an image into a set of colored strokes that explicitly incorporate the geometric information of the scene in a single inference step without requiring a sketch dataset. A sketch is then generated from the strokes where CLIP-based perceptual loss maintains a semantic similarity between the sketch and the image. We show theoretically that sketching is equivariant with respect to arbitrary affine transformations and thus provably preserves geometric information. Experimental results show that LBS substantially improves the performance of object attribute classification on the unlabeled CLEVR dataset, domain transfer between CLEVR and STL-10 datasets, and for diverse downstream tasks, confirming that LBS provides rich geometric information.

Clevr: A Diagnostic Dataset For Compositional Language And Elementary Visual Reasoning

Abstract: When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover short-comings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.

local geometric information
synthetic photo dataset


Semantic segmentation results with different sensor resolutions on the Cityscapes dataset #REFR . For all resolutions, the learned rectangular layout performs best.

MNIST classification To illustrate the principle of our end-to-end sensor layout optimization, we start with a toy example and optimize the layout for hand-written digit recognition on MNIST #OTHEREFR with a sensor size of only 4 × 4 instead of the original 28 × 28 pixels.

The digits in MNIST are always centered, so the hypothesis is that an optimized layout puts smaller pixels in the middle in order to capture the higher information density there.

We apply the curvilinear layout φ curv whose parameters we initialize with 0, i.e., Table 2 .

Differentiable Sensor Layouts For End-To-End Learning Of Task-Specific Camera Parameters

Abstract: The success of deep learning is frequently described as the ability to train all parameters of a network on a specific application in an end-to-end fashion. Yet, several design choices on the camera level, including the pixel layout of the sensor, are considered as pre-defined and fixed, and high resolution, regular pixel layouts are considered to be the most generic ones in computer vision and graphics, treating all regions of an image as equally important. While several works have considered non-uniform, e.g., hexagonal or foveated, pixel layouts in hardware and image processing, the layout has not been integrated into the end-to-end learning paradigm so far. In this work, we present the first truly end-to-end trained imaging pipeline that optimizes the size and distribution of pixels on the imaging sensor jointly with the parameters of a given neural network on a specific task. We derive an analytic, differentiable approach for the sensor layout parameterization that allows for taskspecific, local varying pixel resolutions. We present two pixel layout parameterization functions: rectangular and curvilinear grid shapes that retain a regular topology. We provide a drop-in module that approximates sensor simulation given existing high-resolution images to directly connect our method with existing deep learning models. We show that network predictions benefit from learnable pixel layouts for two different downstream tasks, classification and semantic segmentation.

The Cityscapes Dataset For Semantic Urban Scene Understanding

Abstract: Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotations, 20 000 additional images have coarse annotations to enable methods that leverage large volumes of weakly-labeled data. Crucially, our effort exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity. Our accompanying empirical study provides an in-depth analysis of the dataset characteristics, as well as a performance evaluation of several state-of-the-art approaches based on our benchmark.

learned rectangular layout
Semantic segmentation


As discussed in previous parts, there exists an evident scaling effect in Transformer language models: larger model/data sizes and more training compute typically lead to an improved model capacity #REFR 34] .

It has been a long way that LLMs evolve into the current state: general and capable learners.

In the development process, a number of important techniques are proposed, which largely improve the capacity of LLMs.

Here, we briefly list several important techniques that (potentially) lead to the success of LLMs, as follows.

As two representative models, GPT-3 and PaLM explored the scaling limits by increasing the model size to 175B and 540B, respectively.

Furthermore, since compute budget is usually limited, scaling laws can be employed to conduct a more compute-efficient allocation of the compute resources.

For example, Chinchilla (with more training tokens) outperforms its counterpart model Gopher (with a larger model size) by increasing the data scale with the same compute budget [34] .

While, it should be noted that data scaling should be with careful cleaning process, since the quality of pre-training data plays a key role in the model capacity.

A Survey Of Large Language Models

Abstract: Ever since the Turing Test was proposed in the 1950s, humans have explored the mastering of language intelligence by machine. Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable artificial intelligence (AI) algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pretraining Transformer models over large-scale corpora, showing strong capabilities in solving various natural language processing (NLP) tasks. Since the researchers have found that model scaling can lead to an improved model capacity, they further investigate the scaling effect by increasing the parameter scale to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement, but also exhibit some special abilities (e.g., incontext learning) that are not present in small-scale language models (e.g., BERT). To discriminate the language models in different parameter scales, the research community has coined the term large language models (LLM) for the PLMs of significant size (e.g., containing tens or hundreds of billions of parameters). Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT (a powerful AI chatbot developed based on LLMs), which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. Considering this rapid technical progress, in this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions. This survey provides an up-to-date review of the literature on LLMs, which can be a useful resource for both researchers and engineers.

Scaling Laws For Neural Language Models

Abstract: We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.